betaFeedback

WorksComputer Science

ReactBench-AD: Do Driving VLAs and World-Action Models React to Imminent Hazards in Their Path?

Abstract

Vision-Language-Action models (VLAs) and World-Action Models (WAMs) have emerged in autonomous driving with the promise of strong generalization to long-tail scenarios. Although these models achieve state-of-the-art results on autonomous driving benchmarks, most existing benchmarks are dominated by nominal scenarios and therefore cannot test this promise. To test one such long-tail capability that is critical for safety, we introduce ReactBench-AD, a benchmark that measures how driving models react to objects that appear in the ego vehicle's path. Built on real NAVSIM and PhysicalAI-Autonomous-Vehicles scenes, ReactBench-AD composites a crossing pedestrian, a standing pedestrian, or a stationary car consistently across all camera views and history frames. Each edited scene thus forms an exact, pixel-level counterfactual pair with its original, which allows us to isolate the causal effect of the inserted object on the predicted trajectory. Evaluating four VLAs and two WAMs, we find that they react weakly, often barely at all. Finally, we test how easily one of the evaluated VLAs, ReCogDrive, can be taught to brake hard for imminent hazards: a short fine-tuning stage on edited scenes substantially improves its reactions, with the learned behavior generalizing to unseen objects and fully real scenes, at minimal cost to nominal driving. These results call into question the out-of-the-box long-tail capabilities of these models, but suggest that they can be a strong foundation for acquiring them with little additional training, highlighting the value of real or synthetic data that demonstrates the long-tail behaviors nominal datasets rarely contain. Code and data will be made publicly available.

argument as to why WAMs might be more adequate for embodied AI applications is that they can take advantage of the dynamic representations of how the world evolves that they inherit from their backbone video generation model. Likewise, some of these models [10, 23, 38, 45, 53] have demonstrated state-of-the-art (SOTA) performance on multiple autonomous driving benchmarks.

However, the vast majority of the samples that compose these benchmarks are nominal driving scenarios where unexpected events don’t usually occur, so how these models react to safety-critical scenarios, and in particular to imminent hazards in their path, remains understudied. Evaluating this reliably places requirements both on the inputs the models see and on how their reactions are measured. On the input side, the scenes should ideally be real rather than simulated, and they should match the input configuration each model was trained on. On the measurement side, each safety-critical scene should be paired with an otherwise identical counterpart that lacks the hazardous object, so that the object’s causal effect on the predicted trajectory can be isolated. For such a pair to be informative, the object must be placed where a reaction is actually required, i.e., on the ego vehicle’s path, and it must appear consistently in every camera view and history frame whose field of view covers its location. In addition, the metrics should describe how a model reacts, not only whether it collides: aggregate collision rates cannot distinguish a model that ignores a hazard from one that reacts, but too weakly or too late. To our knowledge no benchmark meets all of these requirements at once.

To address this gap, we introduce ReactBench-AD, a benchmark for evaluating how VLAs and WAMs react to objects inserted in their path (Figure 1). Since many VLAs and WAMs are trained on NAVSIM [6] or PhysicalAI-Autonomous-Vehicles (PAI-AV) [30], we build ReactBench-AD on these two datasets, so that these models are evaluated on inputs matching their training configuration, with the same cameras, calibrations, history length, and frame rate. Into each sample, we separately insert a crossing pedestrian, a standing pedestrian, and a stationary car at a controlled position on the ego vehicle’s path, rendering the object consistently across all camera views and history frames and modifying only the pixels of the object and its shadow. Each edited sample thus forms an exact counterfactual pair with its original, so any change in the predicted trajectory is caused by the inserted object.

Figure 1. We insert an object here, a standing pedestrian, into the ego vehicle’s path in a real driving scene, so that the edited scene SES^{E} and the original scene SOS^{O} differ only in the pixels of the inserted object. Bottom: distance travelled by each model’s front bumper over its 4 s plan in the original (yOy^{O}) and edited (yEy^{E}) scene, with the pedestrian’s extent shown in yellow. All six evaluated VLAs and WAMs shorten their plans only slightly, if at all, and drive through the pedestrian, whereas the ReCogDrive IL checkpoint stops before it after a short fine-tuning on edited scenes (IL + RT, Sec. 5).

Introduction {#sec:1}

Vision-Language-Action models (VLAs) build on top of the capabilities of Vision-Language Models (VLMs), steering them from general purpose chat models to policy networks that can output low-level actions for the manipulation of robots or vehicles [18, 22, 44, 48, 57]. The main motivation for using them in autonomous driving is that they can inherit the reasoning and generalization capabilities of VLMs to better handle long-tail and out-of-distribution (OOD) scenarios, which modular pipelines struggle with [18]. Indeed, these models show promising results, with many of them [9, 18, 22, 35, 40, 43, 56] topping open- and closed-loop benchmarks such as NAVSIM [6], NAVSIMv2 [4], Bench2Drive [19], and nuScenes [3].

A newer line of work uses a different type of foundation model to build on top of, namely, video generation models. By fine-tuning these models to denoise action tokens along with video tokens they are turned into what has recently been termed World-Action Models (WAMs) [52]. The main

We evaluate four VLAs, namely Alpamayo 1.5 [31], Alpamayo 2 [32], AutoVLA [56], and ReCoDrive [22], and two WAMs, namely DriveWAM [38] and SimWAM [53], and find that they react weakly to inserted objects and fail to avoid the collision in most cases. To understand why, we examine two potential factors: Chain-of-Thought (CoT) reasoning and the individual stages of the multi-stage training recipes that some of these models follow. For inserted pedestrians, the Alpamayo models recognize the hazard in their CoT but mostly fail to act on it. Finally, a short fine-tuning stage on edited scenes teaches ReCoDrive to brake hard, generalizing to unseen objects and fully real scenes. Together, these findings suggest that the models often recognize hazards but lack the behaviors to act on them.

Our contributions can be summarized as follows.

  • We introduce ReactBench-AD, a benchmark of exact, pixel-level counterfactual pairs built on real NAVSIM and PAI-AV scenes, with metrics that capture whether and how models react to objects in their path.

  • We evaluate four VLAs and two WAMs and show that they largely fail to handle such imminent hazards.

  • We investigate how CoT reasoning and the individual training stages affect their reactions.

  • We show that fine-tuning only ReCoDrive’s action head on edited scenes teaches it to brake for hazards, including unseen objects and fully real scenes, at almost no cost to nominal driving.

Related Work

VLAs and WAMs for Autonomous Driving. VLAs fine-tune pretrained VLMs to predict robot actions in the language space [57], an idea quickly adopted for predicting driving trajectories or low-level actions for autonomous driving [18, 40, 43, 48]. Later VLAs moved action prediction out of the language space, using special action tokens [56] or a dedicated action head, ranging from MLPs [35, 36] to generative models [9, 22, 44]. WAMs have recently emerged as an alternative to VLAs, replacing the VLM backbone with a pretrained video generation model [33, 52]. Driving WAMs either denoise action tokens conditioned on (partially) denoised video tokens [38, 45], denoise both jointly [10], or use future video prediction only as an auxiliary training objective, avoiding frame generation at inference [23, 53]. Although many of these models achieve SOTA results on open- and closed-loop benchmarks, their ability to handle safety-critical scenarios remains understudied.

Safety Evaluation of End-to-End Planners. Several benchmarks evaluate driving models in safety-critical scenarios. SafeBench [46] and Bench2Drive [19] provide such scenarios in the CARLA simulator [7], the latter as part of a broader multi-ability evaluation, while Fail2Drive [13] pairs CARLA routes containing distribution shifts, such as unseen obstacles, with in-distribution counterparts and reports consistent performance drops. However, since CARLA renders synthetic images, planners trained on real-world data, such as the VLAs and WAMs we study, cannot be evaluated there without a large sim-to-real gap. WOD-E2E [47] curates real long-tail driving scenes for open-loop evaluation, selected for rarity rather than for requiring an immediate reaction, and interPlan [17] augments nuPlan scenarios with long-tail events such as jaywalkers and accident sites, but only at the object level, so it cannot evaluate sensor-based planners.

To keep sensor inputs realistic, another line of work builds safety-critical scenarios on top of real driving scenes using neural reconstruction or generative models. NeuroNCAP [24] builds a closed-loop safety testing framework on NeRF [28, 41] reconstructions of nuScenes scenes, and HUGSIM [54] builds a closed-loop simulator on 3D Gaussian Splatting [21] reconstructions of nuScenes and other datasets, whose harder scenarios include aggressive vehicles. Challenger [49] and SMD [55] instead generate multi-view nuScenes videos of safety-critical scenarios, while Geng et al. [12] extend generative rendering to closed-loop evaluation with adversarial vehicles. Since many open-weight VLAs and WAMs, including all models we evaluate, are trained on NAVSIM or PAI-AV, evaluating these models on such benchmarks would confound their failures to react with sensor and domain shift. RealEngine [20] and NavSafe [1] reconstruct nuPlan scenes, matching the training domain of NAVSIM-based models, but imminent hazards requiring a prompt reaction are only a subset of their scenarios. Since all of these methods re-render or synthesize the entire camera image, none of them yields exact counterfactual pairs with the original sensor data, whereas ReactBench-AD modifies only the inserted object’s pixels, isolating its causal effect on the predicted trajectory.

Counterfactual Safety Evaluation. Closest to our work are two contemporaneous studies that edit real scenes to measure the causal effect of objects on predicted trajectories. Yang et al. [51] inpaint away pedestrians near the ego

corridor in nuScenes and NAVSIM frames and find that end-to-end and VLA planners rarely exhibit genuine avoidance. Luo et al. [25] insert hazards into nuScenes views via instruction-based image editing and show that planners frequently drive into them. Our work differs from these studies in four ways. First, ReactBench-AD places the object exactly on the ego vehicle’s ground-truth future trajectory, so it always obstructs the path the vehicle actually drove. Second, it uses the camera calibration to render the object into every camera view and history frame, leaving all other pixels unchanged. Third, we evaluate every model on scenes that match its training distribution, so that performance drops caused by domain shift are not mistaken for failures in safety-critical scenes. Fourth, we also evaluate WAMs.

ReactBench-AD

ReactBench-AD pairs real driving scenes with edited counterparts in which an object blocks the ego vehicle’s path at close range, so that a prompt reaction, either braking or a safe nudge around the object, is needed to avoid a collision.

Object Compositing

To keep the inserted object consistent across camera views and timesteps, control its 3D position precisely, and modify only the pixels of the object and its shadow, we render 3D assets with Blender [2] and composite them into the original scenes. For each scene, we generate three counterfactual variants: a crossing pedestrian, a standing pedestrian, and a stationary car, drawn from 42 human and seven vehicle assets. Each object is placed at the ego vehicle’s front-bumper position on its ground-truth trajectory after a look-ahead time τ∈{0.5,1.0,1.5,2.0}\tau\in\{0.5, 1.0, 1.5, 2.0\} s, so that, had the ego vehicle kept driving as logged, it would collide with the object within τ\tau. The crossing pedestrian instead walks toward the ego vehicle’s path at 1.4 m/s and reaches this point after τ\tau (Figure 2). We render the object, with a contact shadow, into the four history timesteps and up to four camera views and hide the parts occluded by existing scene geometry. The inserted objects are clearly recognizable: an off-the-shelf COCO detector (D-FINE-X [34]) finds them in 99.3% of the edited front-camera images (Appendix § 8.§ 2); further details and visualizations are provided in Appendix § 8.1.

Figure 2. Overview of ReactBench-AD. Left: we pick a target on the ego vehicle’s ground-truth trajectory (dark line), namely the ego’s front-bumper position at a look-ahead time τ∈{0.5,1.0,1.5,2.0}\tau\in\{0.5, 1.0, 1.5, 2.0\} s after the current frame tt (circles). At each history timestep, a standing object is placed at the target, whereas a moving one, such as the crossing pedestrian shown here, is placed along a path that reaches the target at t+τt + \tau (here τ=0.5\tau= 0.5 s). Middle: we composite the object at these positions into every camera view and history frame of the model’s input (top: the original scene at the current frame tt). Right: in the edited scene, we score the model’s plans for the original (yOy^{\mathrm{O}}, dashed) and edited (yEy^{\mathrm{E}}, solid) input (Eq. (1)), here for ReCogDrive.

Data and Evaluation Protocol

Sample Selection. ReactBench-AD is constructed from samples drawn from NAVSIM (ReactNAV) and PAI-AV (ReactPAI). For ReactNAV, we filter navtest, the test split of NAVSIM, to exclude samples in which the ego vehicle is nearly stationary (below 5 km/h at the current frame tt), as these require no reaction. We then keep only samples in which each of the three objects—a crossing pedestrian, a standing pedestrian, and a stationary car—can be placed at all four history timesteps up to and including tt. This filtering yields 3,228 navtest samples, which form ReactNAV. For ReactPAI, we randomly select and download 5,500 clips from PAI-AV and apply the same filters, retaining 1,045 samples. In both subsets, the ego-vehicle speed at the current frame lies between 5 and 40 km/h (Appendix § 8.§ 3).

Metrics. Let SOS^{O} denote the original scene and SES^{E} its edited, asset-augmented counterpart, and let πθ\pi_{\theta} be the evaluated model. We obtain two trajectories, yO=πθ(SO)y^{O} = \pi_{\theta}(S^{O}) and yE=πθ(SE)y^{E} = \pi_{\theta}(S^{E}), predicted from the original and edited scene, respectively. For a metric M\mathcal{M}, we write M(y;S)\mathcal{M}(y; S) for its value when trajectory yy is evaluated in scene SS, and define the edit effect as

ΔM=ME−MO=M(yE;SE)−M(yO;SE).(1)\Delta\mathcal{M} = \mathcal{M}^{E} - \mathcal{M}^{O} = \mathcal{M}(y^{E}; S^{E}) - \mathcal{M}(y^{O}; S^{E}). \tag*{(1)}

Note that both trajectories are evaluated in the edited scene SES^{E}: yOy^{O} is scored against the inserted object even though the model never observed it. Hence, MO\mathcal{M}^{O} reflects the outcome had the model ignored the hazard, and ΔM\Delta\mathcal{M} isolates the effect of the model’s reaction to it.

For ReactNAV, our primary metric is P:=NC×DACP := \mathrm{NC} \times\mathrm{DAC}, which combines the no-collision (at-fault) rate, NC, with drivable area compliance, DAC; for ReactPAI, since PAI-AV lacks the map data needed for DAC, we use P:=NCP := \mathrm{NC}. To capture reactions that improve safety without avoiding a collision, we also use the mean contact speed, CS, over the samples for which both yOy^{O} and yEy^{E} result in an at-fault collision. For both metrics, we report the edit effect (Eq. (1)), ΔP\Delta P and ΔCS\Delta\mathrm{CS}. Finally, to assess whether the models respond to the inserted actors at all, successfully or not, we report the trajectory invariance rate, TIRϵ\mathrm{TIR}_{\epsilon}, defined as the percentage of samples for which the Average Displacement Error (ADE) between yOy^{O} and yEy^{E} is below a threshold ϵ\epsilon.

We follow the at-fault criteria of NAVSIM with three small modifications that make them stricter, e.g., a lateral contact with a pedestrian or cyclist counts as at fault. Additionally, since driving models typically predict waypoints at a low temporal resolution (e.g., 2 Hz), we interpolate each predicted trajectory piecewise-linearly and check collisions at 100 Hz, so that contacts between waypoints are neither missed nor misclassified. Details on the at-fault criteria and the interpolation are provided in Appendix § 8.§ 4.

Experiments and Results

Experimental Setup

We evaluate four VLAs, Alpamayo 1.5 [31], Alpamayo 2 [32], AutoVLA [56], and ReCogDrive [22] (2B variant), and two VLMs, DriveWAM [38] and SimWAM [53], whose architectures and input configurations are detailed in Appendix § 9.1. ReCogDrive, DriveWAM, and SimWAM receive only the current front-camera frame, whereas AutoVLA and the Alpamayo models receive several cameras and history frames. AutoVLA, ReCogDrive, DriveWAM, and SimWAM were trained on NAVSIM and are evaluated on ReactNAV only, whereas the Alpamayo models were trained primarily on NVIDIA’s multi-camera driving data, including PAI-AV, and are evaluated on both subsets. On ReactNAV, the Alpamayo models receive only the three front-facing cameras, with frames 0.5 s instead of 0.1 s apart, and hence operate under a strong input distribution shift, which should be kept in mind when interpreting their results. Finally, since the models’ prediction horizons range from 4.0 to 6.4 s, we compute all metrics on only the first 4 s of each predicted trajectory to compare all models over the same horizon.

Results

Table 1 summarizes the main results, with more detailed results along with trajectory visualizations provided in Appendix § 9.§ 2. In most cases, the models fail to meaningfully improve the safety score PP when the inserted object is visible compared to when it is not (yEy^{E} vs. yOy^{O}). The main exceptions are the two Alpamayo models, which achieve substantial improvements at longer look-ahead times on both ReactNAV and ReactPAI. Nevertheless, their performance remains far from ideal: even in the best case, an inserted standing pedestrian at τ=2.0\tau= 2.0 s on ReactPAI for Alpamayo 1.5, unsafe trajectories remain the most likely outcome, accounting for 62.01% of cases. Excluding the samples in which even braking at 8 m/s2^{2} [11, 15] cannot prevent a collision barely changes ΔP\Delta P (Appendix § 9.§ 3).

ModelMetricCrossing pedestrian, τ=0.5\tau=0.51.01.52.0Standing pedestrian, τ=0.5\tau=0.51.01.52.0Car, τ=0.5\tau=0.51.01.52.0
ReactNAV
Alpamayo 1.5ΔP\Delta P+0.25+3.10+7.16+8.71+0.09+3.66+11.93+17.63+0.12+3.84+15.92+28.81
Δ\DeltaCS−1.51−2.73−2.68−2.29−1.63−4.52−5.44−4.83−1.23−4.29−7.05−7.25
Alpamayo 2ΔP\Delta P+0.06+2.11+6.75+8.36+0.03+3.25+9.42+12.33+0.12+5.36+19.39+22.43
Δ\DeltaCS−1.11−1.77−1.42−0.73−1.49−3.10−3.18−2.50−1.19−2.96−4.17−4.22
AutoVLAΔP\Delta P+0.00−0.03+0.53+0.90+0.00+0.16+0.34+0.96+0.03+0.87+2.14+3.26
Δ\DeltaCS+0.03−0.02−0.02−0.03−0.01−0.08−0.16−0.16−0.28−0.59−0.70−0.67
ReCogDriveΔP\Delta P+0.00−0.03+0.22+0.40+0.00+0.00+0.12+0.40+0.00+0.00+0.28+1.89
Δ\DeltaCS+0.31−0.45−0.57−0.45+0.51−0.26−1.06−1.20+1.42−0.07−0.96−1.08
DriveWAMΔP\Delta P+0.00+0.06+0.12+0.09+0.00+0.12+0.56+2.08+0.00+1.77+3.62+5.61
Δ\DeltaCS−0.08−0.14−0.17−0.14−0.15−0.43−0.72−0.88−0.60−0.88−0.91−0.76
SimWAMΔP\Delta P+0.00+0.06+0.12+0.22+0.00+0.00+0.15+0.71+0.00+0.06+0.99+2.26
Δ\DeltaCS−0.11−0.14−0.82−0.32−0.07−0.21−1.64−1.26+0.45−0.79−1.79−0.79
ReactPAI
Alpamayo 1.5ΔP\Delta P+0.10+5.55+25.17+35.12+0.19+5.07+21.63+35.69+0.67+12.92+20.86+25.65
Δ\DeltaCS−3.64−7.96−7.52−5.75−4.13−9.14−10.25−8.79−4.43−9.08−9.73−8.52
Alpamayo 2ΔP\Delta P+0.00+3.44+22.68+34.07+0.00+0.57+8.61+15.89+0.10+5.55+16.65+25.17
Δ\DeltaCS−1.80−6.01−6.69−5.42−2.45−5.85−6.31−5.93−3.07−8.16−8.49−6.95

Table 1. Results on ReactNAV and ReactPAI by model, inserted object, and look-ahead time τ\tau (s). ΔP\Delta P is in percentage points (higher is better); Δ\DeltaCS is in km/h (lower is better). Gray: the paired 95% bootstrap CI includes zero.

On ReactNAV, ΔP\Delta P is smallest for crossing pedestrians, larger for standing pedestrians, and largest for stationary cars. This ordering holds for almost all evaluated models, although the magnitudes differ considerably. For AutoVLA, ReCogDrive, DriveWAM, and SimWAM, improvements exceeding 1 percentage point occur only for stationary cars, with the sole exception of DriveWAM, which also improves slightly for the standing pedestrian at τ=2.0\tau= 2.0 s. When collisions are not avoided, these four models also reduce the contact speed only marginally, by less than 2 km/h in all cases, whereas Alpamayo 1.5 and Alpamayo 2 reduce it by up to 7.25 and 4.22 km/h, respectively.

On ReactPAI, the pattern changes: at τ≥1.5\tau\ge1.5 s, both Alpamayo models achieve larger safety improvements for crossing pedestrians than for stationary cars. Inspecting the models’ CoTs reveals that they describe the inserted car as stationary in only about a third of the cases for Alpamayo 1.5 (33%) and a quarter for Alpamayo 2 (24%). This suggests a possible training bias, with the models expecting leading vehicles to be in motion. When collisions are not avoided, both models also reduce the contact speed more than on ReactNAV, by up to 10.25 km/h for Alpamayo 1.5 and 8.49 km/h for Alpamayo 2. For standing pedestrians, however, the two models differ markedly: the safety improvement ΔP\Delta P of Alpamayo 2 is less than half that of Alpamayo 1.5 (+15.89 vs. +35.69 percentage points at τ=2.0\tau= 2.0 s), even though both recognize the pedestrian as standing in the lane at similar rates (60% for Alpamayo 1.5 and 56% for Alpamayo 2). The difference lies instead in how strongly they brake after recognizing it: over the final 0.5 s of the horizon, Alpamayo 1.5 is on average 13.0 km/h slower than in its original trajectory, compared with 7.7 km/h for Alpamayo 2. For crossing pedestrians, both models perform similarly, likely because milder braking often suffices, as the pedestrian may clear the path before the vehicle arrives.

Figure 3. Trajectory invariance rate, TIRϵ\mathrm{TIR}_{\epsilon}: the percentage of samples whose original and edited trajectories differ by an ADE below ϵ\epsilon, i.e., for which the model essentially ignores the inserted object (higher means less reactive). Results are aggregated across all four look-ahead times τ\tau.

Figure 3 reports TIRϵ\mathrm{TIR}_{\epsilon} for each inserted object at four thresholds, ϵ∈{0.05,0.1,0.2,0.5}\epsilon\in\{0.05, 0.1, 0.2, 0.5\} m. In many cases, the models barely react to the inserted objects at all. In the most extreme case, 34.9% of AutoVLA’s trajectory pairs have an ADE below 0.05 m when a crossing pedestrian is inserted.

Figure 4. Effect of inserted objects on the CoTs of Alpamayo 1.5 and Alpamayo 2 on ReactPAI, aggregated across all four look-ahead times τ\tau. Bars show the distribution of the CoT’s leading action: slow/stop (stop, yield, or slow down), keep distance (keep a distance from or follow a lead vehicle), or other. CoT changed: percentage of edited-scene CoTs that differ from the original-scene CoT of the same sample. Names the actor: percentage of edited-scene CoTs that mention a pedestrian or, for inserted cars, a vehicle (in parentheses: original-scene CoTs).

Failure Analysis

To understand where these failures originate, we study (a) whether the Alpamayo models reason correctly about the hazard and whether they act on this reasoning, and (b) how the individual training stages of ReCogDrive and SimWAM affect their reaction capabilities.

CoT Analysis. Of the evaluated models, only the two Alpamayo models consistently reason in natural language before predicting a trajectory. AutoVLA also supports reasoning, but in practice always defaults to its fast mode, producing the same generic reasoning output for every sample, so we exclude it from this analysis. We run all experiments in this section on ReactPAI, which is closer to the training distribution of the Alpamayo models.

Figure 4 illustrates how the inserted objects affect the CoTs of the Alpamayo models. For the vast majority of samples with an inserted pedestrian, the CoT switches to a Slow/stop instruction that explicitly refers to the inserted actor. The model thus detects the actor and correctly reasons that the vehicle should slow down or stop, yet in most cases the executed trajectory does not decelerate enough to avoid a collision. This suggests that, in this case, the main cause of failure in hazardous scenarios is not perception or reasoning, but rather a “miscommunication” between the VLM backbone and the action expert, a phenomenon also reported in other recent studies [5, 8, 26, 27]. For inserted cars, the CoT of Alpamayo 1.5 most often changes to a Keep distance instruction, consistent with the model mostly failing to recognize that the vehicle is stationary.

Figure 5. Effect of CoT swapping on NC for Alpamayo 1.5 (A1.5) and Alpamayo 2 (A2) on ReactPAI at τ=2.0 s\tau= 2.0\,\mathrm{s}. The first and second letters give the source of the visual input and of the CoT: original (O) or edited (E). Whiskers indicate 95% bootstrap CIs.

To investigate the causal effect of the CoT on the safety of the predicted trajectory, we swap the CoTs generated for the original and edited scenes, obtaining the trajectories yOE=πθ(IO,CE)\mathbf{y}^{\mathrm{OE}} = \pi_{\theta}(I^{\mathrm{O}}, C^{\mathrm{E}}) and yEO=πθ(IE,CO)\mathbf{y}^{\mathrm{EO}} = \pi_{\theta}(I^{\mathrm{E}}, C^{\mathrm{O}}), where II denotes the visual input, CC the CoT, and the superscripts O and E the original and edited versions, respectively. As Fig. 5 shows for a look-ahead time of τ=2.0 s\tau= 2.0\,\mathrm{s}, the largest safety benefit appears to come from directly observing the inserted object (IEI^{\mathrm{E}}), rather than from receiving a CoT that instructs the model to yield to a pedestrian or car absent from its visual input. Once the object is visible, however, the CoT does help: the CoT generated for the edited scene leads to safer trajectories than the one generated for the original scene (yEE\mathbf{y}^{\mathrm{EE}} vs. yEO\mathbf{y}^{\mathrm{EO}}). Even an explicit instruction to stop immediately, though, still leaves both models colliding in most samples (Appendix § 9.5).

Training Stages Analysis. ReCogDrive and SimWAM both have a released checkpoint for each training stage, which allows us to examine how each stage affects a model’s ability to avoid imminent hazards. ReCogDrive has an additional initial stage in which its VLM undergoes supervised fine-tuning (SFT) on driving scenes across a range of tasks, including trajectory prediction. Apart from this stage, both models follow a similar recipe. They are first trained with imitation learning (IL) to match human expert trajectories and then fine-tuned with reinforcement learning (RL). The RL stage uses the Predictive Driver Model Score (PDMS) [6] as the reward and variants of GRPO [37] adapted to diffusion (ReCogDrive) and flow-matching (SimWAM) models.

Table 2 reports the safety improvement ΔP\Delta P on React-NAV at τ=2.0 s\tau= 2.0\,\mathrm{s} for each training-stage checkpoint of both models. For comparison with nominal driving performance, we also include each checkpoint’s PDMS on navtest. We restrict the comparison to ego-vehicle speeds below 20 km/h, since at higher speeds ΔP\Delta P is close to zero for all checkpoints. For every inserted object, ΔP\Delta P decreases from one training stage to the next, even though PDMS increases. The largest drop occurs between ReCogDrive’s VLM SFT and IL stages, where ΔP\Delta P for stationary cars falls from +10.39 to +4.88 (though the two stages also differ in their action head), whereas for crossing pedestrians, which all checkpoints rarely avoid, the differences are small. These results suggest that further optimizing for nominal driving performance, on data in which safety-critical scenarios are rare, can in fact degrade a model’s ability to handle them.

ModelStagePDMSCross. ped.Stand. ped.Car
ReCogDriveVLM SFT84.1+2.38+4.99+10.39
ReCogDriveIL86.5+0.85+1.36+4.88
ReCogDriveRL90.8+0.51+0.62+3.29
SimWAMIL90.3+0.45+2.10+6.13
SimWAMRL91.9+0.40+1.25+4.14

Table 2. Safety improvement ΔP\Delta P (in percentage points) on ReactNAV at τ=2.0 s\tau= 2.0\ \mathrm{s} for each training-stage checkpoint of ReCogDrive and SimWAM, over samples with an initial ego-vehicle speed below 20 km/h (n=1762n = 1762). Gray: the paired 95% bootstrap CI of the difference to the previous training stage includes zero. PDMS on navtest as reported in [22, 53].

Can Models Learn to React?

Having established that current VLAs and WAMs fail to handle safety-critical scenarios reliably, and in particular to brake in an emergency, we now ask how difficult it is to teach this behavior to an existing model, ReCogDrive, using edited samples similar to those in ReactBench-AD, without degrading its nominal driving performance and in a way that generalizes to scenes not seen during training. Since our hazardous edited scenes come without ground-truth trajectories, the most straightforward approach would be to add them to the RL training mix. However, GRPO can only reinforce behavior that the policy already samples: a group of candidate trajectories that all receive the same reward yields no gradient. We test this directly on ReCogDrive’s IL checkpoint, i.e., the initialization of its RL stage. To this end, we create 300 hazardous scenes from navtrain, the training split of NAVSIM, by inserting the three ReactBench-AD objects exactly as in ReactNAV. Even with a group size of G=32G = 32, four times the one used in its RL stage, only 26.3% of the groups yield a non-zero gradient, and only 1.3% would reinforce a trajectory that is safe under ReactBench-AD’s definition (Appendix § 10.1). We therefore construct artificial target trajectories for the hazardous scenes and teach the desired behavior via supervised fine-tuning.

SFT on Artificial Targets

Training Data. Since ReactNAV is drawn from navtest, we build our training data from navtrain: we randomly select 1,957 navtrain samples whose ego speed falls within the range covered by ReactNAV and, for each of the three inserted objects, create edited variants at two of the four look-ahead times τ∈{0.5,1.0,1.5,2.0} s\tau\in\{0.5, 1.0, 1.5, 2.0\}\ \mathrm{s}, assigned such that all four are equally represented overall. This yields a total of 11,742 samples. To prevent the model from learning the shortcut that any visual artifact introduced by an inserted object is a cue to brake, we also create benign variants of the same samples, with the same objects and look-ahead times but with the objects placed off the ego vehicle’s path, yielding another 11,742 benign samples. We refer to the resulting 23,484 samples as ReactTrain and provide further details and visualizations in Appendix § 10.§ 2.

Artificial Target Trajectories. Since the hazardous ReactTrain samples come without ground-truth trajectories that react to the inserted object, we construct artificial targets for supervised fine-tuning with a simple recipe. We first fit a cubic spline to the ground-truth waypoints, which preserves the original path while allowing us to change the speed profile along it. We then place the desired stopping point on the spline at a margin dmd_m before the inserted object and denote by Δs\Delta s the arc length between the ego vehicle and this point. Given the initial ego speed v0v_0, we use the constant deceleration a=min⁡{v02/(2Δs),amax⁡}a = \min\{v_0^2/(2\Delta s), a_{\max}\}, i.e., the deceleration required to stop within Δs\Delta s, capped at a maximum value amax⁡a_{\max}. Finally, we obtain the new waypoints by sampling the spline at the arc length s(t)=v0t−12at2s(t) = v_0t - \frac{1}{2}at^2 that the ego vehicle travels up to each waypoint timestamp tt, keeping the vehicle in place once it has stopped, i.e., for t≥v0/at \ge v_0/a. We set dm=1 md_m = 1\ \mathrm{m} and amax⁡=8 m/s2a_{\max} = 8\ \mathrm{m}/\mathrm{s}^2, a common estimate of the maximum braking deceleration of a passenger car on a dry road [11, 15]. We keep the samples in which even amax⁡a_{\max} cannot prevent the collision, all with τ=0.5 s\tau= 0.5\ \mathrm{s}, since their targets still teach the model to slow down as much as possible. We illustrate this construction in Appendix § 10.§ 3.

Experimental Details and Results. We fine-tune the ReCogDrive IL checkpoint on ReactTrain with the denoising objective of its imitation-learning stage. To preserve nominal driving performance, each epoch additionally includes 11,742 original navtrain samples, drawn anew in every epoch, so that hazardous, benign, and original samples each make up one third of it. We supervise the hazardous samples with the artificial targets and the benign and original ones with the logged ground-truth trajectories. We update only the action head, keeping the VLM frozen, and train for only three epochs, i.e., 276 gradient steps (Appendix § 10.§ 3). Table 3 compares the fine-tuned checkpoint (IL + RT) with the original IL checkpoint. The fine-tuned model reacts far more strongly to the inserted hazards on ReactNAV, while its PDMS on navtest drops only marginally, mainly due to lower ego progress, while NC and TTC improve slightly. On the original ReactNAV scenes, IL + RT plans at least 20% less distance than IL in only 0.9% of the samples, never braking harder than 4.5 m/s2^2.

ReactNAVnavtest
ΔP\Delta P↑ΔCS\Delta CS↓NC↑DAC↑TTC↑EP↑C↑PDMS↑
IL+0.5−0.398.194.494.280.6100.086.2
IL + RT+41.4−9.998.794.594.979.399.986.0

Table 3. Results of ReCogDrive’s IL checkpoint (IL) and of the same checkpoint fine-tuned on ReactTrain (IL + RT) on ReactNAV, aggregated across all look-ahead times τ\tau, and on navtest (PDMS and its sub-metrics on LQR-executed trajectories, from our own evaluation). Units are as in Table 1.

Generalization. Finally, we examine whether the fine-tuned model has learned to slow down whenever its path is blocked, rather than merely reacting to the objects seen during training or to compositing artifacts. To this end, we construct two additional test sets from navtest (details in Appendix § 10.§ 4). Unseen objects: we insert standing animals, an object category absent from ReactTrain, into the ego vehicle’s path in 291 samples at the same four look-ahead times. Real scenes: we select 308 samples in which the nearest road user in the ego vehicle’s path is a nearly stationary vehicle 5–15 m ahead, and edit only the ego status, so that the ego vehicle appears to be traveling at a constant speed that would bring it into collision with this vehicle after the look-ahead time τ\tau. This produces the same kind of imminent hazard as in ReactNAV, but with fully real visual input, so any reaction cannot be attributed to compositing artifacts. Since ReCogDrive sees only the current frame, the edited ego status does not contradict its visual input.

Table 4 reports the results for both test sets, aggregated across all four look-ahead times τ\tau. The fine-tuned model generalizes to both settings. On standing animals, the safety gains are smaller than on ReactNAV (+28.1 vs. +41.4 in ΔP\Delta P), but still substantial relative to the IL checkpoint, which barely reacts (see also Figure 6). For the real scenes, which lack an original trajectory, we report absolute metrics; though not directly comparable to the deltas above, they show that the safety improvements carry over to fully real visual input (54.2 vs. 6.8 in PP), the setting that ultimately matters most.

Plans of the IL and IL + RT checkpoints for the 291 unseen-objects samples

Figure 6. Plans of the IL and IL + RT checkpoints for the 291 unseen-objects samples, each at one random τ\tau, in the original (yOy^{\mathrm{O}}) and edited (yEy^{\mathrm{E}}) scenes; each line traces the front bumper over 4 s. Although ReactTrain contains no animals, IL + RT shortens its plans in the edited scenes, while its original-scene plans stay close to those of the IL checkpoint.

Unseen objectsReal scenes
ΔP\Delta P↑ΔCS\Delta CS↓PP↑CSCS↓
IL+0.1−0.26.821.4
IL + RT+28.1−8.754.214.3

Table 4. Generalization of the IL and IL + RT checkpoints to unseen objects and real scenes, aggregated across all four look-ahead times τ\tau. Real scenes have no original trajectory, so we report absolute values: PP in percent and the mean contact speed, CS, in km/h.

Discussion. Overall, our results suggest that, in the models we probed, the generalization inherited from the foundation model often extends to recognizing hazards but not to acting on them. For inserted pedestrians, the Alpamayo CoTs call for a stop that the trajectory mostly does not execute (§ 4.3), and ReCogDrive brakes for unseen objects and in real scenes after only its action head is trained on the behavior, suggesting that its frozen VLM already encodes whether the path ahead is blocked. This points to a distinction between long-tail scenarios, which are open-ended, and the behaviors these scenarios demand, such as emergency braking, which form a much smaller set [39]. Such behaviors are rare in nominal driving but, unlike the scenarios, can be covered explicitly, even with edited data such as ReactTrain, so that a model can draw on them when it faces a scenario it has never seen.

Conclusion

We introduced ReactBench-AD, a benchmark of exact, pixel-level counterfactual pairs built from real driving scenes, and used it to show that six recent VLAs and VLMs largely fail to handle safety-critical scenarios in which hard braking or a quick but safe nudge is needed to avoid a collision, calling into question the long-tail generalization often attributed to these models. At the same time, a short fine-tuning stage on edited scenes substantially improved how ReCogDrive reacts to imminent hazards, also for unseen objects and fully real scenes, at almost no cost to nominal driving. This suggests that models such as ReCogDrive can serve as a strong foundation for acquiring safety-critical behaviors such as emergency braking, and that training data could target these comparatively few behaviors directly, relying on the model to generalize them across the many long-tail scenarios in which they are needed.

Limitations. ReactBench-AD covers mainly three object types and a single scenario (a close-range obstacle in the ego path), and its inserted objects are not fully photorealistic. Our evaluation is open-loop, so delayed reactions spanning several planning cycles are not assessed. Finally, our simple fine-tuning recipe leaves room for better uses of ReactTrain that may even avoid the small drop in nominal performance.

LLM Usage Statement

We used two coding agents, Claude Code and Codex, in both the research and the manuscript. Under the authors’ direction, the agents implemented, tested, and debugged parts of the code, and generated figures. The authors designed the benchmark, its metrics, the experiments, and the training recipe, reviewed all agent-generated figures, and validated the code and all reported numbers. The research questions, hypotheses, and conclusions are the authors’ own; the agents were used only to critique and refine them. For the manuscript, the agents drafted parts of the appendix and helped find and summarize related work; the authors checked all generated text and verified every citation. The agents were also used to improve the language of the author-written main text, make notation consistent, and propose ways to shorten sections. The authors take full responsibility for the content of this paper.

Algorithm 1 Compositing 3D assets into driving scenes

Require: Camera images {It,c}\{I_{t,c}\}, camera calibration, LiDAR point clouds, ground-truth ego trajectory, 3D asset AA, look-ahead time τ\tau

Ensure: Edited images {I~t,c}\{\widetilde{I}_{t,c}\}

1: Select an insertion location using the ego vehicle’s front-bumper position along its ground-truth trajectory at look-ahead time τ\tau

2: Define the asset’s motion or stationary placement

3: Estimate the local ground surface from LiDAR

4: for each history timestep tt do

5: Set the asset’s 3D position, orientation, and pose on the estimated ground

6: for each camera cc do

7: Render asset and contact shadow using camera calibration

8: Mask occluded asset regions using scene depth and LiDAR evidence

9: Blend the visible asset and shadow into It,cI_{t,c}

10: Harmonize the inserted foreground with PCT-Net to obtain I~t,c\widetilde{I}_{t,c}

11: end for

12: end for

13: return {I~t,c}\{\widetilde{I}_{t,c}\}

Additional ReactBench-AD Details

Object Compositing

This section complements § 3.1 with implementation details of the compositing pipeline. Algorithm 1 summarizes the pipeline, and the paragraphs below follow the order of its steps.

Asset Selection and Placement. We use 42 human assets from the Microsoft Rocketbox library [14] and seven vehicle assets from OpenX Assets [42]. Assets are assigned pseudo-randomly from the scene identifier with a fixed seed, so all four look-ahead variants of a scene use the same asset. Crossing pedestrians walk at 1.4 m/s; consecutive history frames show successive phases of the asset’s walk cycle and are rendered with motion blur. The crossing direction is also chosen pseudo-randomly per scene; if the placement is invalid in this direction at any look-ahead time, the opposite direction is used, so that all four variants of a scene share the same direction. Standing pedestrians use a natural two-foot standing pose and face the approaching ego vehicle. The stationary car is aligned with the ego vehicle’s heading at the look-ahead time τ\tau, with its rear bumper placed exactly at the ego vehicle’s front bumper at τ\tau.

Placement Validity. A placement is considered valid if (a) the object’s bounding box (see Appendix § 8.§ 4) contains fewer than five LiDAR returns higher than 0.15 m above its base at each history position and, for crossing pedestrians, also at intervals of 0.25 m along the remaining path to the interception point; (b) the ground fit described below is reliable in every history frame; (c) for the car, the ego vehicle’s footprint does not touch it at any ground-truth pose before τ\tau, so that the first contact occurs exactly at τ\tau; and (d) the object is visible in at least one camera view.

Ground Estimation. In each history frame, we fit a plane to the LiDAR returns within ±\pm4 m of the object, using the lowest returns of each cell of a coarse grid to suppress other objects, followed by iterative outlier rejection. The fit is considered reliable if at least 80 returns lie within 12 cm of the plane, at least 12 of them within 1.6 m of the object, the root-mean-square residual of these returns is at most 9 cm, and the gradient of the plane is at most 0.22. On ReactNAV, pedestrians are grounded independently in each history frame, which absorbs small registration errors between frames, whereas the car is grounded once, in the last history frame, and keeps the same rigid pose in all frames, since independent fits would make it visibly jitter. On ReactPAI, whose history frames are only 0.1 s apart, all objects use a single plane: the median of the per-frame fits, expressed in a common frame. In both cases, the object is placed on the fitted plane, and its vertical axis is tilted from gravity towards the plane normal by at most 2°, so pedestrians stand upright, as on real slopes, and a noisy fit cannot visibly lean the object.

Rendering and Lighting. Objects are rendered with an ideal pinhole camera and then warped into each camera’s lens model: the OpenCV distortion model for ReactNAV and an f-theta fisheye model for ReactPAI. On ReactPAI, we additionally motion-compensate the LiDAR returns using their individual timestamps and account for the rolling shutter when projecting them into the image. The object is lit only by an environment light whose colour is estimated from the sky regions of the camera images (or from their upper part if no sky is detected) and whose intensity is set from the median image brightness; we do not render direct sunlight or directional cast shadows. The contact shadow is obtained by rendering the object above a shadow-catcher plane aligned with the fitted ground under the same environment light.

Occlusion. Depth Anything V2 (Small) [50] predicts relative inverse depth, which is known only up to an unknown scale and shift. We recover metric depth by projecting the LiDAR points into the image and robustly fitting this scale and shift to their inverse depths, which yields a dense metric depth map with the sharp boundaries of the monocular estimate. An object pixel with rendered depth dd is is occluded if the scene depth at that pixel is smaller than d−max⁡(0.55 m,0.045d)d - \max(0.55\ \mathrm{m}, 0.045d); the margin, which grows with distance like the error of the monocular depth, prevents the background or the supporting surface from occluding the object. Pixels whose scene depth is consistent with the fitted ground plane are never treated as occluders, which protects the feet and wheels from depth errors near the road. Because monocular depth can be inaccurate, occlusions additionally require LiDAR evidence of a closer object: on ReactNAV, a predicted near-total occlusion without LiDAR support is discarded, and on ReactPAI, only connected occluding regions that contain LiDAR points in front of the object are kept. If the monocular depth cannot be aligned to the LiDAR, we skip occlusion handling for that view on ReactPAI, whereas on ReactNAV we discard the sample. Since occlusion handling can therefore miss occluders, we manually checked 100 randomly selected ReactPAI views in which at least 500 pixels of the object are visible. In none of them is the object drawn over a closer real object that should hide it, or hidden where nothing real is in front of it. Hence, with 95% confidence, fewer than 3% of these views contain such an occlusion error. Finally, pixels showing the ego vehicle’s body are never modified. On ReactNAV, the ego vehicle is not visible in the front camera and only appears at the bottom edge of the front-left and front-right cameras (CAM L0/R0). On ReactPAI, PAI-AV provides a mask of these pixels for each camera, which we use for all cameras except the front-wide one. For that camera, the provided mask is overly conservative, so we instead obtain it by projecting the hood of a 3D model of the ego vehicle (Mercedes-Benz GLS) with each clip’s calibration and refining its outline against the images.

Camera Matching and Harmonization. Before harmonization, we match the rendered object to the camera by applying a Gaussian blur with σ=0.55\sigma= 0.55 px, adding Gaussian noise whose standard deviation is estimated from the surrounding image region (capped at 1.5 intensity levels), and applying bounded corrections of white balance, highlight level and saturation towards a ring of pixels around the object. PCT-Net [16], as implemented in the libcom library [29], is then applied only to the object, excluding its shadow, and all pixels outside the object are restored from its input. On ReactPAI, PCT-Net operates on a crop around the object, padded by the object’s size; objects with fewer than 256 pixels or a bounding box smaller than 12×2412 \times24 pixels are not harmonized, the output is blended 50/50 with the input, and it is discarded if PCT-Net changes the object’s pixels by more than 24 intensity levels on average or the blend has less than half of the object’s original luminance contrast. Finally, we automatically verify that every edited image differs from the original only within the mask of the inserted object and its shadow.

Examples. Figure 7 depicts randomly selected samples from both ReactNAV and ReactPAI for each inserted object and look-ahead time τ\tau.

Randomly selected ReactNAV and ReactPAI samples for each inserted object and look-ahead time

Figure 7. Randomly selected ReactNAV (top) and ReactPAI (bottom) samples for each inserted object and look-ahead time τ\tau, shown in the edited front-camera image at the current frame tt. Each row shows a single scene, whose four look-ahead variants share the same asset.

Vehicle Asset Attribution. The seven vehicle assets come from OpenX Assets [42], which adapts Sketchfab models licensed under the Creative Commons Attribution 4.0 International licence (CC BY 4.0): Audi TT Roadster (2014), Dacia Duster (2010), Hyundai Tucson (2015) and Mini Countryman (2016) by razor24; BMW X1 (2016) by mk2design; Volvo EX30 (2024) by Mehdi Lazgouli (LagzDesign); and Volvo V60 Polestar (2013) by myedsu.

Availability. We will make the compositing code and the edited ReactNAV samples publicly available. The PAI-AV license does not permit redistributing modified data, so the edited ReactPAI samples may not be released directly; in that case, we will instead provide the identifiers of the selected clips together with the scripts that regenerate the edited samples from the original PAI-AV data.

Could Compositing Explain the Failures?

Since the inserted objects are not fully photorealistic, the models might fail to react to them because they do not recognize them or dismiss them as artifacts, rather than because they cannot handle the hazard. We address this concern in two ways: an off-the-shelf detector recognizes almost all inserted objects, and the single-frame models also fail to brake for real stationary vehicles.

Detectability. We run D-FINE-X [34], pretrained on Objects365 and fine-tuned on COCO, on the edited front-camera image at the current frame tt, at its default 640×640640 \times640 input resolution. Since each edited image differs from its original only in the object and its shadow, we take as the target box the tight box around pixels that change by more than 40 gray levels inside the object’s projected 3D bounding box. The threshold removes the faint shadow, and the projected box removes the shadow outside it. A sample counts as detected if a person detection (for pedestrians) or a car detection (for cars) with score ≥0.5\ge0.5 overlaps the target box with IoU ≥0.5\ge0.5 in the edited image, and no such detection exists in the original image. We exclude the 158 of 51,276 samples in which the object is not visible in the front camera at tt. As Table 5 shows, the detector finds the inserted object in 99.3% of the remaining samples. Detection rates drop slightly with τ\tau, i.e., for smaller and more distant objects, and are lower for ReactPAI, whose wide-angle front camera renders objects smaller and which contains more night scenes. Most remaining misses are objects that are only partially in view or poorly lit.

Object0.51.01.52.0All
ReactNAV
Crossing ped.99.999.999.599.399.7
Standing ped.100.0100.099.999.899.9
Car100.0100.099.899.699.8
ReactPAI
Crossing ped.98.598.397.395.897.5
Standing ped.99.098.697.195.797.6
Car99.098.898.598.198.6

Table 5. Share of edited samples (%) in which D-FINE-X detects the inserted object at its location, by object and look-ahead time τ\tau.

Real Stationary Vehicles. We further evaluate the models on the real-scenes test set of § 5.1, in which the hazard is a real stationary vehicle and only the ego status is edited (Appendix § 10.§ 4). We restrict this control to ReCogDrive, DriveWAM, and SimWAM, which receive only the current frame, so that the edited ego status does not contradict their visual input. As Table 6 shows, none of them avoids the vehicle at τ≤1.0\tau\le1.0 s, and at most 1.0% of the collisions are avoided at τ=1.5\tau= 1.5 s, where a deceleration of at most 3.0 m/s23.0\ \mathrm{m}/\mathrm{s}^{2} would suffice. Only at τ=2.0\tau= 2.0 s, where the edited speeds are low (median 12 km/h), do the models avoid 23–29% of the collisions. These absolute values are not directly comparable with ΔP\Delta P in Table 1, but they show that the failure to brake is not specific to composited objects.

Modelτ=0.5\tau=0.51.01.52.0Mean
ReCogDrive0.00.01.022.75.9
DriveWAM0.00.00.628.97.4
SimWAM0.00.00.023.45.8

Table 6. PP (in percent) on the 308 real scenes with a stationary vehicle ahead, by look-ahead time τ\tau (s).

Data Statistics

Table 7 shows the number of ReactNAV and ReactPAI samples in each ego-vehicle speed interval at the current frame; Figure 8 (Appendix § 9.§ 2) shows how the safety improvement varies with this speed.

km/h5–1010–1515–2020–2525–3030–3535–40Total
ReactNAV325634803680484271313,228
ReactPAI1801881991371271101041,045

Table 7. Number of samples in each ego-vehicle speed interval at the current frame for ReactNAV and ReactPAI.

Safety improvement as a function of ego-vehicle speed

Figure 8. Safety improvement as a function of ego-vehicle speed at the current frame aggregated across all four look-ahead times τ\tau. Shaded areas indicate 95% bootstrap CIs.

Metrics and Trajectory Interpolation

This section complements § 3.2 with implementation details of the evaluation metrics and the trajectory interpolation.

Footprints. Collisions are checked between rectangular footprints in the ground plane. The ego footprint is placed relative to the rear axle and oriented by the predicted heading; its dimensions are NAVSIM’s ego vehicle parameters on ReactNAV and the vehicle dimensions provided with each clip on ReactPAI. Recorded road users use their annotated bounding boxes. For each pedestrian asset, we measure a single bounding box as the union of the bounds of its meshes over its entire walk cycle, so that it encloses every pose of the asset. Its 3D extent is used for the placement validity check of Appendix § 8.1, and its ground-plane projection for collision checking. The inserted car uses the bounding box provided with the asset.

Scene. As in NAVSIM’s non-reactive evaluation, recorded road users follow their logged motion and do not react to the ego vehicle. On ReactNAV, they are annotated every 0.5 s; on ReactPAI, we use the automatically generated obstacle labels provided with the dataset, which are available at about 10 Hz and which we transform into the ego frame at the current frame tt using the logged ego motion. The inserted object moves at constant velocity over the whole horizon: the crossing pedestrian keeps walking at 1.4 m/s after crossing the ego vehicle’s path, and the standing pedestrian and the car remain stationary.

At-fault classification. We follow the at-fault criteria of NAVSIM with three modifications: (a) a lateral contact with a pedestrian or cyclist while the ego vehicle is moving is considered at fault, (b) a contact that occurs while the ego vehicle is moving backwards (i.e., its displacement is opposite to its predicted heading) is never exempted as a rear-end collision, and (c) collisions with static objects also score 0 instead of 0.5. Only the first contact with each road user is considered, as in PDMS. The ego speed at a contact is the speed of the predicted trajectory segment leading into it, and a road user’s speed is its displacement over a 1 s window around the contact; inserted objects use their known velocity, and static objects (e.g., barriers and traffic cones) are always treated as stopped. Each contact is assigned to the first of the following classes that applies:

  1. the ego vehicle is stopped (≤0.05\leq0.05 m/s): not at fault;

  2. the other road user is stopped (<0.05< 0.05 m/s): at fault;

  3. rear contact, i.e., the road user’s center lies more than 150° from the ego heading as seen from the rear axle: not at fault, unless the ego vehicle is moving backwards (modification (b));

  4. frontal contact, i.e., the front edge of the ego footprint touches the road user: at fault;

  5. lateral contact: at fault if the road user is a pedestrian or cyclist (modification (a)), if the ego vehicle occupies more than one lane, i.e., more than one lane polygon contains an ego corner and none contains all four, or if any ego corner lies outside the drivable area.

Since PAI-AV provides no map, lateral contacts with vehicles are never at fault on ReactPAI. This choice is immaterial: counting all of them as at fault instead changes the outcome of at most one of the 12,540 edited predictions per model.

Inserted-object collisions. NCinsNC_{\mathrm{ins}} is computed like NCNC but considers only at-fault collisions with the inserted object. Since such a collision also sets PP to zero, NCins−PNC_{\mathrm{ins}}-P is the fraction of trajectories that avoid the inserted object but collide with another road user or, on ReactNAV, leave the drivable area. Hence, ΔNCins\Delta NC_{\mathrm{ins}} measures the edit effect on collisions with the inserted object alone, whereas ΔP\Delta P also reflects changes in collisions with other road users and in drivable area compliance.

Contact speed and ADE. The contact speed used in ΔCS\Delta CS is the speed of the ego rear axle along the predicted segment leading into the first at-fault contact. It measures how fast the ego vehicle is driving, not the relative impact speed, and the two trajectories of a pair may first collide with different road users. The ADE used in TIRϵ\mathrm{TIR}_{\epsilon} is the mean Euclidean distance between corresponding waypoints of yOy^{\mathrm{O}} and yEy^{\mathrm{E}} over the first 4 s, at each model’s native resolution (40 waypoints for the Alpamayo models and 8 for the others); a pair is counted as invariant if this distance is strictly below ϵ\epsilon.

Trajectory interpolation. Driving models typically predict waypoints at a low temporal resolution (e.g., 2 Hz), and checking only at the waypoints can miss collisions that occur between them or misclassify the type of contact, and hence fault. Rather than tracking the waypoints with NAVSIM’s LQR controller, which not all evaluated models were trained with, we interpolate the predicted positions and (unwrapped) headings piecewise-linearly, starting from the ego vehicle’s current pose, i.e., with constant velocity between consecutive waypoints. This interpolant passes exactly through the waypoints, has no parameters, and never overshoots. We interpolate the annotated agent tracks in the same way within their annotated lifetimes. We check collisions at 100 Hz and refine each first contact to the exact instant of first touch, at which the contact is classified. Drivable area compliance is checked at 10 Hz, as in NAVSIM, whereas ΔCS\Delta\mathrm{CS} and TIRϵ\mathrm{TIR}_{\epsilon} are computed at each model’s native waypoint frequency.

Further Experimental Details and Results

Evaluated Models

This section complements § 4.1 with details of the evaluated models and of how we adapt their inputs to ReactBench-AD. Table 8 summarizes their action heads, input configurations, and outputs.

ModelAction headCamerasFrames per cameraVisual history (s)Ego history (s)Vel./acc.ReasoningTrajectory horizon (s)
Alpamayo 1.5Transformer Decoder440.31.5NoYes6.4
Alpamayo 2Transformer Decoder640.31.5NoYes6.4
AutoVLA–341.5–YesOptional5.0
ReCogDriveDiffusion Transformer1101.5YesNo4.0
DriveWAM–1101.5YesNo4.0
SimWAMDiffusion Transformer110–YesNo4.0

Table 8. Comparison of the action heads, input configurations, and outputs of the evaluated models. Visual and ego history denote the temporal spans of camera observations and past ego states, respectively; a visual history of 0 s indicates a single current frame.

Action Heads. AutoVLA and DriveWAM do not introduce a dedicated action module: AutoVLA adds 2,048 action tokens to the existing language vocabulary and generates the trajectory autoregressively as a sequence of ten such tokens, whereas DriveWAM places action tokens in the same sequence as the video tokens and denoises them with flow matching, using the same diffusion transformer that generates the future video frames.

Training Data. AutoVLA, ReCogDrive, DriveWAM, and SimWAM were all trained on NAVSIM data, and hence their expected inputs follow the NAVSIM format. The Alpamayo models, in contrast, were trained primarily on NVIDIA’s multi-camera driving data (80k hours for Alpamayo 1.5 and roughly 115k hours for Alpamayo 2), with PAI-AV listed among their training datasets, and their input format follows PAI-AV. Alpamayo 1.5 expects frames 0.1 s apart from four cameras: front-wide, front-tele, cross-left, and cross-right. Alpamayo 2 uses the same four cameras plus two rear-facing ones, rear-left and rear-right. Per its model card [31], Alpamayo 1.5 was additionally trained on a range of public driving datasets, including NAVSIM-derived data; no comparable information is publicly available for Alpamayo 2 [32]. Thus, although NAVSIM is not entirely out-of-distribution for Alpamayo 1.5, PAI-AV is considerably closer to its training distribution.

Input Adaptation and Evaluation. Because the models predict trajectories of different lengths, ranging from 4.0 to 6.4 s into the future (Table 8), we compute the metrics defined in § 3.2 using only the first 4 s of each predicted trajectory for a fair comparison; that is, we discard the last 1.0 and 2.4 s of the trajectories predicted by AutoVLA and the two Alpamayo models, respectively. For every model, the original and edited predictions of a sample use the same random seed, derived from the sample identifier, for text generation and trajectory sampling where applicable. Differences between yOy^{\mathrm{O}} and yEy^{\mathrm{E}} are thus caused by the edit rather than by independent resampling. When evaluating the Alpamayo models on ReactNAV, we use only the three front-facing NAVSIM cameras, which we map to their PAI-AV counterparts, and drop the remaining PAI-AV camera inputs, namely the front-tele camera and, for Alpamayo 2, also the two rear cameras. Moreover, because NAVSIM frames are 0.5 s apart, the visual history spans 1.5 s instead of 0.3 s, and we linearly interpolate the four NAVSIM ego poses to the 16 poses at 10 Hz that the models expect. Finally, the two subsets differ in the navigation input: on ReactNAV, we pass NAVSIM’s driving command to both models as a navigation instruction (e.g., “Turn left at the next opportunity.”), whereas on ReactPAI, for which no navigation commands are available, they receive none. On ReactNAV, AutoVLA occasionally generated a number of action tokens other than the expected ten and thus no valid trajectory; we exclude the affected 1–5 samples per inserted object and look-ahead time (at most 0.15%). Moreover, for NAVSIM, AutoVLA’s front-left and front-right inputs are the side cameras CAM_L1 and CAM_R1, which we do not render; Appendix § 9.§ 4 repeats the evaluation with the cameras described in the AutoVLA paper. We keep them unedited and, for AutoVLA only, additionally exclude the 1, 4, and 9 crossing-pedestrian samples at τ=1.0\tau= 1.0, 1.5, and 2.0 s in which the pedestrian would enter their field of view. All other models are evaluated on all 3,228 samples.

Results

Table 9 is the more detailed version of Table 1, with the original and edited version of each metric along with their differences, and additionally reports NCins\mathrm{NC}_{\mathrm{ins}} (Appendix § 8.§ 4).

ModelMetricCrossing pedestrian τ=0.5\tau=0.51.01.52.0Standing pedestrian 0.51.01.52.0Car 0.51.01.52.0
ReactNAV
Alpamayo 1.5PP O0.000.062.115.890.000.031.555.110.000.001.153.69
E0.253.169.2614.590.093.6913.4822.740.123.8417.0732.50
Δ\Delta+0.25+3.10+7.16+8.71+0.09+3.66+11.93+17.63+0.12+3.84+15.92+28.81
NCinsNC_{\mathrm{ins}} O0.000.152.949.230.000.193.2211.060.000.031.866.32
E0.253.4410.9018.680.094.2817.7231.440.124.0618.6237.08
Δ\Delta+0.25+3.28+7.96+9.45+0.09+4.09+14.50+20.38+0.12+4.03+16.76+30.76
CS O18.9318.8618.7918.6118.9318.9519.0618.8718.9118.9419.3820.13
E17.4216.1416.1116.3217.3014.4313.6214.0417.6814.6512.3312.88
Δ\Delta−1.51−2.73−2.68−2.29−1.63−4.52−5.44−4.83−1.23−4.29−7.05−7.25
Alpamayo 2PP O0.000.343.7510.530.000.464.2811.180.000.343.5910.19
E0.062.4510.5018.900.033.7213.6923.510.125.7022.9932.62
Δ\Delta+0.06+2.11+6.75+8.36+0.03+3.25+9.42+12.33+0.12+5.36+19.39+22.43
NCinsNC_{\mathrm{ins}} O0.002.7017.8735.560.004.7125.5043.430.001.8916.6435.81
E0.065.3621.4438.790.0310.3534.1752.040.159.0839.2859.05
Δ\Delta+0.06+2.66+3.56+3.22+0.03+5.64+8.67+8.61+0.15+7.19+22.65+23.23
CS O18.8218.6619.0019.5318.8518.6219.0219.6018.8018.7119.6920.72
E17.7116.8917.5818.7917.3615.5315.8417.1117.6115.7615.5216.50
Δ\Delta−1.11−1.77−1.42−0.73−1.49−3.10−3.18−2.50−1.19−2.96−4.17−4.22
AutoVLAPP O0.000.030.090.310.000.030.090.310.000.090.090.50
E0.000.000.621.210.000.190.431.270.030.962.233.75
Δ\Delta+0.00−0.03+0.53+0.90+0.00+0.16+0.34+0.96+0.03+0.87+2.14+3.26
NCinsNC_{\mathrm{ins}} O0.000.060.220.530.000.120.250.560.000.120.250.68
E0.000.090.961.710.030.340.931.830.031.152.704.56
Δ\Delta+0.00+0.03+0.74+1.18+0.03+0.22+0.68+1.27+0.03+1.02+2.45+3.88
CS O18.8618.6418.4418.2418.8618.6318.4418.2318.8318.6818.5518.47
E18.8918.6218.4218.2118.8518.5518.2718.0718.5518.0917.8517.80
Δ\Delta+0.03−0.02−0.02−0.03−0.01−0.08−0.16−0.16−0.28−0.59−0.70−0.67
RecogDrivePP O0.000.030.000.340.000.000.000.340.000.000.000.06
E0.000.000.220.740.000.000.120.740.000.000.281.95
Δ\Delta+0.00−0.03+0.22+0.40+0.00+0.00+0.12+0.40+0.00+0.00+0.28+1.89
NCinsNC_{\mathrm{ins}} O0.000.030.060.500.000.000.030.500.000.000.030.12
E0.000.000.280.900.000.000.221.050.000.000.432.23
Δ\Delta+0.00−0.03+0.22+0.40+0.00+0.00+0.19+0.56+0.00+0.00+0.40+2.11
CS O17.8219.9820.1919.7817.6119.7320.0819.8417.9720.0920.1719.90
E18.1319.5219.6119.3318.1319.4619.0218.6319.3920.0219.2118.82
Δ\Delta+0.31−0.46−0.57−0.45+0.51−0.27−1.06−1.20+1.42−0.07−0.96−1.08
DriveWAMPP O0.000.000.030.220.000.000.000.150.000.000.030.15
E0.000.060.150.310.000.120.560.230.001.773.665.76
Δ\Delta+0.00+0.06+0.12+0.09+0.00+0.12+0.56+2.08+0.00+1.77+3.62+5.61
NCinsNC_{\mathrm{ins}} O0.000.030.060.280.000.030.030.220.000.030.060.19
E0.000.090.190.340.000.150.592.350.001.773.695.82
Δ\Delta+0.00+0.06+0.12+0.06+0.00+0.12+0.56+2.14+0.00+1.73+3.62+5.64
CS O18.8918.7018.5318.3318.8918.7118.5918.5718.7818.8218.8618.92
E18.8118.5618.3618.1818.7418.2917.8717.6918.1917.9417.9618.16
Δ\Delta−0.08−0.14−0.17−0.14−0.15−0.43−0.72−0.88−0.60−0.88−0.91−0.76
SimWAMPP O0.000.000.030.340.000.000.030.400.000.000.030.12
E0.000.060.150.560.000.000.191.120.000.061.022.39
Δ\Delta+0.00+0.06+0.12+0.22+0.00+0.00+0.15+0.71+0.00+0.06+0.99+2.26
NCinsNC_{\mathrm{ins}} O0.000.000.060.400.000.030.060.430.000.000.060.15
E0.000.060.190.620.000.030.251.240.000.061.052.57
Δ\Delta+0.00+0.06+0.12+0.22+0.00+0.00+0.19+0.81+0.00+0.06+0.99+2.42
CS O19.9920.2219.7818.7919.9619.6419.7418.8219.9021.1420.2619.17
E19.8820.0718.9618.4719.8919.4318.1017.5620.3520.3418.4718.37
Δ\Delta−0.11−0.14−0.82−0.32−0.07−0.21−1.64−1.26+0.45−0.79−1.79−0.79
ReactPAI
Alpamayo 1.5PP O0.000.000.672.680.000.000.192.300.000.000.671.72
E0.105.5525.8437.800.195.0721.8237.990.6712.9221.5327.37
Δ\Delta+0.10+5.55+25.17+35.12+0.19+5.07+21.63+35.69+0.67+12.92+20.86+25.65
NCinsNC_{\mathrm{ins}} O0.000.000.672.780.000.000.192.390.000.000.671.72
E0.105.5525.8437.990.195.2621.8238.090.6713.0121.7227.66
Δ\Delta+0.10+5.55+25.17+35.22+0.19+5.26+21.63+35.69+0.67+13.01+21.05+25.93
CS O20.3320.5721.0121.6320.3520.5320.7921.3120.3521.5222.3022.59
E16.6912.6113.5015.8716.2211.3910.5312.5215.9212.4412.5714.07
Δ\Delta−3.64−7.96−7.52−5.75−4.13−9.14−10.25−8.79−4.43−9.08−9.73−8.52
Alpamayo 2PP O0.000.100.292.110.000.100.192.200.000.100.572.01
E0.003.5422.9736.170.000.678.8018.090.105.6517.2227.18
Δ\Delta+0.00+3.44+22.68+34.07+0.00+0.57+8.61+15.89+0.10+5.55+16.65+25.17
NCinsNC_{\mathrm{ins}} O0.000.100.292.110.000.100.192.090.000.100.572.01
E0.003.5422.9736.170.000.779.0918.370.105.6517.3227.56
Δ\Delta+0.00+3.44+22.58+33.88+0.00+0.67+8.80+15.98+0.10+5.55+16.75+25.45
CS O20.3420.5720.9721.7320.3420.4020.5921.0120.3020.8721.8122.38
E18.5414.5614.2816.3117.8914.5514.2815.0817.2312.7113.3215.43
Δ\Delta−1.80−6.01−6.69−5.42−2.45−5.85−6.31−5.93−3.07−8.16−8.49−6.95

Table 9. Detailed results on ReactNAV and ReactPAI by model, inserted object, and look-ahead time τ\tau (s). O and E denote the trajectories predicted from the original and edited scene, respectively, both evaluated in the edited scene, and Δ=E−O\Delta= E - O. P and NCinsNC_{\mathrm{ins}} are in percent and their deltas in percentage points (higher is better). CS is the mean contact speed in km/h over the samples for which both trajectories result in an at-fault collision (lower is better). Deltas are computed before rounding.

Additionally, Figure 8 shows the safety improvement ΔP\Delta P as a function of the ego-vehicle speed (in km/h) at the current frame, aggregated across all look-ahead times τ\tau. Finally, Figure 9 visualizes the predicted trajectories for some randomly chosen samples from ReactNAV and ReactPAI at a look-ahead time of τ=2.0\tau= 2.0 s.

Example predictions for crossing pedestrians, standing pedestrians, and cars across six models

Figure 9. Example predictions at a look-ahead time of τ=2.0\tau= 2.0 s, one for each model and inserted object: Alpamayo 1.5 and Alpamayo 2 on ReactPAI, and the other four models on ReactNAV. Each example shows the edited front-camera image at the current frame, cropped around the inserted object, next to a bird’s-eye view of the edited scene. The arrow marks the crossing pedestrian’s motion over the 4 s horizon. Dashed and solid lines show the model’s plans for the original (yO\mathbf{y}^{\mathrm{O}}) and edited (yE\mathbf{y}^{\mathrm{E}}) scenes, respectively, as front-bumper paths over the first 4 s. Examples are drawn at random from the samples whose original plan collides with the inserted object.

Unavoidable Collisions

At short look-ahead times, some collisions cannot be avoided even by emergency braking. To quantify their effect, we mark a sample as unavoidable if braking along the ego vehicle’s ground-truth path from its measured initial speed, at any constant deceleration of up to 8 m/s2^{2} [11, 15] followed by standstill, cannot prevent an at-fault collision with the inserted object within the 4 s horizon. Collisions are checked as in Appendix § 8.§ 4, considering only the inserted object, so the criterion does not depend on the evaluated model. Unavoidable samples occur only at τ=0.5\tau= 0.5 s, where they make up 22% of the ReactNAV and 28% of the ReactPAI samples, depending on the inserted object. As Table 10 shows, excluding them changes ΔP\Delta P by at most 0.2 percentage points and does not change which improvements are distinguishable from zero.

ModelCross. ped. AllCross. ped. Avoid.Stand. ped. AllStand. ped. Avoid.Car AllCar Avoid.
ReactNAV
Unavoidable (%)23.028.315.8
Alpamayo 1.5+0.25+0.32+0.09+0.13+0.12+0.15
Alpamayo 2+0.06+0.08+0.03+0.04+0.12+0.15
AutoVLA+0.00+0.00+0.00+0.00+0.03+0.00
ReCogDrive+0.00+0.00+0.00+0.00+0.00+0.00
DriveWAM+0.00+0.00+0.00+0.00+0.00+0.00
SimWAM+0.00+0.00+0.00+0.00+0.00+0.00
ReactPAI
Unavoidable (%)28.831.323.1
Alpamayo 1.5+0.10+0.13+0.19+0.28+0.67+0.87
Alpamayo 2+0.00+0.00+0.00+0.00+0.10+0.12

Table 10. Safety improvement ΔP\Delta P (in percentage points) at τ=0.5\tau= 0.5 s on all samples (All, as in Table 1) and on the avoidable samples only (Avoid.). Unavoidable: share of samples excluded. Gray: the paired 95% bootstrap CI includes zero.

AutoVLA Side Cameras

AutoVLA takes three camera views as input, and two sources disagree on which NAVSIM cameras these should be. The AutoVLA paper describes them as the front, front-left, and front-right cameras [56], which in NAVSIM are CAM F0, CAM L0, and CAM R0 (yaw ≈±56∘\approx\pm56^\circ for the latter two). The released code, however, uses the side cameras CAM L1 and CAM R1 (yaw ≈±112∘\approx\pm112^\circ) as its front-left and front-right views. Throughout the paper, we follow the released code: we keep CAM L1/R1 unedited and, for AutoVLA only, exclude the few crossing-pedestrian samples in which the inserted object would enter their field of view (Sec. B.1). Table 11 repeats the evaluation with the configuration described in the paper, i.e., with CAM L0/R0 for both the original and the edited scene. The conclusions remain largely the same, with AutoVLA reacting slightly more strongly in the released configuration.

Table 11. AutoVLA on ReactNAV with CAM L1/R1 (released code; used in all other results) or CAM L0/R0 as its front-left and front-right views, the latter for both the original and the edited scene. ΔP\Delta P is in percentage points, Δ\DeltaCS in km/h, and TIRϵ_{\epsilon} in percent at ϵ=0.05\epsilon= 0.05 m, aggregated across all four look-ahead times. Gray: the paired 95% bootstrap CI includes zero.

Side camerasMetricCrossing pedestrian, τ=0.5\tau=0.51.01.52.0Standing pedestrian, τ=0.5\tau=0.51.01.52.0Car, τ=0.5\tau=0.51.01.52.0
CAM L1/R1ΔP\Delta P+0.00−0.03+0.53+0.90+0.00+0.16+0.34+0.96+0.03+0.87+2.14+3.26
CAM L1/R1Δ\DeltaCS+0.03−0.02−0.02−0.03−0.01−0.08−0.16−0.16−0.28−0.59−0.70−0.67
CAM L1/R1TIRϵ_{\epsilon}34.919.417.3
CAM L0/R0ΔP\Delta P+0.00+0.03+0.12+0.34+0.00+0.03+0.16+0.56+0.03+0.65+1.83+2.57
CAM L0/R0Δ\DeltaCS+0.01+0.00−0.04−0.01−0.01−0.06−0.10−0.10−0.23−0.45−0.59−0.51
CAM L0/R0TIRϵ_{\epsilon}61.440.838.2

Table 11. AutoVLA on ReactNAV with CAM L1/R1 (released code; used in all other results) or CAM L0/R0 as its front-left and front-right views, the latter for both the original and the edited scene. ΔP\Delta P is in percentage points, Δ\DeltaCS in km/h, and TIRϵ_{\epsilon} in percent at ϵ=0.05\epsilon= 0.05 m, aggregated across all four look-ahead times. Gray: the paired 95% bootstrap CI includes zero.

CoT Analysis Details

This section complements the CoT analysis of § 4.3 with two further experiments: fixed CoTs, which measure the effect of the CoT when the visual input is held fixed, and attention knockout, which tests whether the smaller effect of fixed CoTs on Alpamayo 2 reflects a generally weaker reliance on its CoT.

Fixed CoTs

On the edited ReactPAI scenes, we replace each Alpamayo model’s generated CoT with one of the five predefined CoTs in Table 12. Figure 10 shows the resulting change in the no-collision rate at τ=2.0\tau= 2.0 s. For Alpamayo 1.5, instructing the model to accelerate or resume speed produces more unsafe trajectories than letting it generate its own CoT, whereas an immediate-stop instruction increases the no-collision rate by 6.9 to 15.3 percentage points. Even so, it still leaves Alpamayo 1.5 colliding in more than half of the samples. For Alpamayo 2, all changes are at most 2.6 percentage points.

Figure 10. Change in the no-collision rate, NC, of Alpamayo 1.5 and Alpamayo 2 on edited ReactPAI scenes at τ=2.0\tau= 2.0 s when their CoT is replaced by a fixed one, relative to their own generated CoT. Whiskers indicate 95% bootstrap CIs.

Table 12. Fixed CoTs used in Figure 10. Each fixed CoT replaces the model’s generated CoT verbatim, and ⟨reason⟩\langle reason\rangle is filled in according to the inserted object (bottom).

ConditionFixed CoT
AccelerateAccelerate since the lane is clear ahead
Resume speedResume speed since the lane is clear ahead
Slow downSlow down due to the ⟨reason⟩\langle reason\rangle
YieldYield due to the ⟨reason⟩\langle reason\rangle
Immediate stopCome to a full stop immediately due to the ⟨reason⟩\langle reason\rangle
Inserted object⟨reason⟩\langle reason\rangle
Crossing pedestrianpedestrian crossing ahead
Standing pedestrianpedestrian blocking the lane ahead
Carvehicle blocking the lane ahead

Table 12. Fixed CoTs used in Figure 10. Each fixed CoT replaces the model’s generated CoT verbatim, and ⟨reason⟩\langle reason\rangle is filled in according to the inserted object (bottom).

Attention Knockout

To test whether the smaller effect of fixed CoTs on Alpamayo 2 (Figure 10) reflects a generally weaker reliance on its

CoT, we conduct an attention knockout experiment, in which we hide the CoT tokens from the action expert.

CoT Pathway. In both Alpamayo models, each layer of the action expert attends over the action tokens being denoised and over the key–value cache of the corresponding VLM layer. This cache contains the prompt, which includes the camera images, followed by the tokens generated by the VLM, i.e., the CoT. Since the VLM is causal, the cache entries of the prompt do not depend on the CoT. The CoT can therefore affect the predicted trajectory only through the expert’s attention to the generated tokens.

Knockout. In the knockout condition, we mask the keys of all generated tokens in every layer of the action expert and at every denoising step. Besides the CoT text, these include the special tokens that end the reasoning and start the trajectory, since their key–value entries are computed after the CoT and therefore can also encode information about it. All prompt keys remain visible, including the token that opens the reasoning in Alpamayo 1.5, which is part of the prompt. The attention weights are renormalized over the remaining keys, and the action tokens keep their original rotary position embeddings, so the knockout is equivalent to deleting the generated tokens from the cache without shifting any positions. The model weights and all other computations are unchanged.

Samples and Rollouts. We draw 300 samples at random from the 1,045 original (unedited) ReactPAI samples and use the same samples for both models. Each model receives the same inputs as in our main ReactPAI evaluation and uses 10 Euler steps of its flow-matching sampler. For each sample, we run three rollouts: (a) the natural rollout y\mathbf{y}, with the per-sample random seeds of the main evaluation; (b) the knockout rollout yKO\mathbf{y}^{\mathrm{KO}}, which uses the same seeds, and hence the same CoT and the same initial noise as y\mathbf{y}, but hides the CoT from the action expert; and (c) the resampled rollout yRS\mathbf{y}^{\mathrm{RS}}, which draws the initial noise of the flow-matching sampler from a different seed but keeps the seed of the text generation, and hence the CoT.

sampler from a different seed but keeps the seed of the text generation, and hence the CoT. We verified that the three rollouts produce identical CoTs for every sample and that the natural CoT matches the one from the main evaluation for all 300 samples of both models. Since yKO\mathbf{y}^{\mathrm{KO}} shares the initial noise of y\mathbf{y}, any difference between the two is caused by the CoT alone, and a difference of zero would mean that the CoT has no effect.

Metrics. We compare each intervened rollout with the natural one over the first 4 s of the horizon, expressed in the ego frame at the current frame. We denote by ADEKO\mathrm{ADE}^{\mathrm{KO}} the ADE between y\mathbf{y} and yKO\mathbf{y}^{\mathrm{KO}} and by ADERS\mathrm{ADE}^{\mathrm{RS}} the ADE between y\mathbf{y} and yRS\mathbf{y}^{\mathrm{RS}}, and define FDEKO\mathrm{FDE}^{\mathrm{KO}} and FDERS\mathrm{FDE}^{\mathrm{RS}} analogously for the final displacement error at 4 s. ADERS\mathrm{ADE}^{\mathrm{RS}} measures how much the trajectories of each model change under a perturbation that does not involve the CoT, i.e., resampling the initial noise, and thus shows whether one model is generally less sensitive to perturbations than the other; it is not a significance threshold for ADEKO\mathrm{ADE}^{\mathrm{KO}}. All confidence intervals are 95% percentile bootstrap intervals over samples (10,000 resamples), and differences between the two models are computed on the same resamples, i.e., they are paired.

Results. Table 13 reports the full results. Removing the CoT clearly changes the trajectories of both models, so both use their CoT to some extent. However, it moves the trajectory of Alpamayo 2 by 0.33 m (38%) less than that of Alpamayo 1.5, and the same pattern holds at the endpoint (FDE of 1.36 m vs. 2.18 m). In contrast, resampling the initial noise changes the trajectories of both models by the same amount, so the gap does not stem from Alpamayo 2 producing less variable trajectories in general. This confirms that Alpamayo 2 depends less on its CoT, partially explaining the results in Figure 10.

Table 13. Attention knockout on 300 original ReactPAI samples, measured over the first 4 s. KO: CoT hidden from the action expert, with the same initial noise as the natural rollout. RS: initial noise resampled, CoT unchanged. All values are in meters; means with 95% bootstrap CIs.

Alpamayo 1.5Alpamayo 2Difference
ADEKO^{\mathrm{KO}}0.89 [0.81, 0.97]0.55 [0.50, 0.61]+0.33 [0.27, 0.40]
ADERS^{\mathrm{RS}}0.68 [0.61, 0.76]0.69 [0.63, 0.75]−0.01 [−0.09, 0.09]
FDEKO^{\mathrm{KO}}2.18 [1.98, 2.38]1.36 [1.22, 1.51]+0.82 [0.64, 1.00]
FDERS^{\mathrm{RS}}1.96 [1.74, 2.19]1.98 [1.80, 2.18]−0.03 [−0.28, 0.24]

Table 13. Attention knockout on 300 original ReactPAI samples, measured over the first 4 s. KO: CoT hidden from the action expert, with the same initial noise as the natural rollout. RS: initial noise resampled, CoT unchanged. All values are in meters; means with 95% bootstrap CIs.

Further Training Details

GRPO Exploration Probe

This section details the probe summarized in Sec. 5, which examines whether adding our hazardous edited scenes to ReCogDrive’s RL stage would provide a useful learning signal. GRPO can only reinforce behavior that the policy already samples: for each scene, it draws a group of GG candidate trajectories and weights each one by its reward normalized within the group, so a group in which all candidates receive the same reward yields no gradient.

Setup. Starting from ReCogDrive’s IL checkpoint, i.e., the initialization of its RL stage, we sample groups of size G∈{8,16,32}G \in\{8, 16, 32\} on 300 hazardous scenes built from nuScenes (100 per object type, τ∈{1.5,2.0}\tau\in\{1.5, 2.0\} s), keeping all other RL settings as defined by ReCogDrive [22]. G=8G = 8 is the group size used in ReCogDrive’s RL stage, and the larger groups test whether more exploration would recover a useful signal. We then measure two quantities: the fraction of groups in which the candidates do not all receive the same PDMS reward, i.e., groups that yield a non-zero gradient (Signal), and the fraction of groups that contain at least one safe trajectory, i.e., one with P=1P = 1 under the ReactBench-AD rules (§ 3), whose reward is above the group mean, i.e., groups in which the gradient increases the likelihood of a safe trajectory (Safe signal).

Results. As shown in Table 14, even with a group size of 32, only 26.3% of the groups yield a non-zero gradient.

Table 14. Percentage of GRPO groups with a non-zero gradient (Signal) and with at least one safe trajectory above the group mean reward (Safe signal) for ReCogDrive’s IL checkpoint on 300 hazardous scenes, as a function of group size GG.

ObjectSignal (%), G=8G=81632Safe signal (%), G=8G=81632
Crossing pedestrian29.043.053.01.03.03.0
Standing pedestrian9.016.026.00.00.01.0
Car0.00.00.00.00.00.0
All12.719.726.30.31.01.3

Table 14. Percentage of GRPO groups with a non-zero gradient (Signal) and with at least one safe trajectory above the group mean reward (Safe signal) for ReCogDrive’s IL checkpoint on 300 hazardous scenes, as a function of group size GG.

ReactTrain

This section complements § 5.1 with details of the construction of ReactTrain.

Source Samples. We draw the source samples only from the training logs of navtrain, following NAVSIM’s default split into training and validation logs; navtest, from which ReactNAV and the generalization sets of Appendix § 10.§ 4 are built, shares no logs with navtrain. As for ReactNAV, we keep only samples with an ego-vehicle speed of at least 5 km/h at the current frame tt. We then keep only samples in which the hazardous and the benign variant of each of the three objects pass the placement validity checks of Appendix § 8.1 at all four look-ahead times τ\tau. In addition, both variants must pass a front-camera visibility check: at least 25% of the area of the object’s projected bounding box must

ent, and only 1.3% contain a trajectory that is safe under ReactBench-AD’s definition (PP) and receives an above-average reward, i.e., a safe trajectory that the gradient would actually reinforce. The large gap between the two fractions arises because PDMS often rewards candidates whose simulated motion avoids the pedestrian or only grazes it from the side, even though the planned trajectory itself still drives into it. We therefore conclude that, due to this limited exploration, simply adding our edited scenes to the RL training mix is unlikely to teach the model the desired behavior. lie inside the front-camera image at the current frame tt, since this is the only camera image that ReCogDrive receives. The benign variants must further satisfy the clearance condition described below. This yields a pool of 8,734 navtrain samples, from which we randomly select 2,000. Of these, 1,957 are rendered successfully in all 24 variants (3 objects ×\times 2 variants ×\times 4 look-ahead times) and form the source samples of ReactTrain.

Hazardous Variants. The hazardous variants are placed and rendered exactly as in ReactNAV, using the same compositing pipeline, settings and asset pools (Appendix § 8.1). Since the asset is assigned from the scene identifier, all variants of an object in a source sample, hazardous and benign at every look-ahead time, use the same asset. By construction, the logged ground-truth trajectory collides with the inserted object in every hazardous variant.

Benign Variants. The benign variants insert the same object but place it off the ego vehicle’s path. As in the hazardous variants, the placement is defined relative to the ego vehicle’s front bumper at the look-ahead time τ\tau on the logged ground-truth trajectory, but the object is shifted by 4 m perpendicular to the ego vehicle’s heading at τ\tau. The standing pedestrian stands at the shifted position and faces the approaching ego vehicle, as in the hazardous variant. The car is the hazardous car shifted by 4 m and rotated by 180° about its center, so that it faces opposite to the ego vehicle’s heading. The walking pedestrian does not cross the ego vehicle’s path; instead, it walks at 1.4 m/s along a straight line parallel to the ego vehicle’s heading at τ\tau, in the same direction, and passes the shifted position at the current frame tt. Cars are first placed on the left of the ego vehicle and pedestrians on the right; the other side is used only if the preferred one fails a check at any look-ahead time, so all four look-ahead variants of an object share the same side. A benign placement is valid only if the footprints of the object and the ego vehicle remain at least 1 m apart along the entire logged future ego trajectory, which we check every 0.05 s by interpolating between the logged poses. The logged ground-truth trajectory therefore remains safe in every benign variant, which is why we use it as the training target for these samples.

Look-Ahead Times. Although all 24 variants are rendered for every source sample, ReactTrain contains each object at only two of the four look-ahead times per source sample. For each object independently, we randomly split the 1,957 source samples into six groups of nearly equal size and assign to each group one of the six pairs of look-ahead times, {0.5,1.0}\{0.5,1.0\}, {0.5,1.5}\{0.5,1.5\}, {0.5,2.0}\{0.5,2.0\}, {1.0,1.5}\{1.0,1.5\}, {1.0,2.0}\{1.0,2.0\} and {1.5,2.0}\{1.5,2.0\} s. Since each look-ahead time appears in three of the six pairs, each is used for about half of the source samples. The benign variants follow the hazardous ones, i.e.,

a source sample contributes a benign variant of an object exactly at the two look-ahead times of its hazardous variants. Each source sample thus contributes six hazardous and six benign samples, for a total of 11,742 hazardous and 11,742 benign samples.

Examples. Figure 11 depicts the hazardous and benign variants of randomly selected ReactTrain source samples for each inserted object and look-ahead time τ\tau.

Randomly selected ReactTrain samples showing hazardous and benign variants for crossing pedestrian, standing pedestrian, and car at different look-ahead times

Figure 11. Randomly selected ReactTrain samples for each inserted object and look-ahead time τ\tau, shown in the edited front-camera image at the current frame tt. Each column shows a different source sample, with its hazardous variant in the top row and its benign variant in the bottom row; both use the same asset.

Fine-Tuning Setup

This section complements § 5.1. Figure 12 illustrates the construction of the artificial target trajectories that supervise the hazardous ReactTrain samples. We update all parameters of ReCogDrive’s action head and keep the VLM frozen. Training runs for three epochs with a batch size of 384, which amounts to 276 gradient steps in total. We use AdamW, warming up the learning rate over the first 25 steps to a peak of 3×10−53 \times10^{-5} and then decaying it to zero with a cosine schedule. We did not tune these settings, so the reported results reflect what a minimal recipe achieves rather than the best attainable.

Figure 12. Artificial target trajectory for a ReactTrain sample with a standing pedestrian (orange box) inserted at τ=1.0\tau= 1.0 s. The ego vehicle (black box) decelerates along a cubic spline fitted to the logged ground-truth waypoints and stops (blue box) a margin dmd_m before the pose at which it would first contact the pedestrian (dashed box).

Generalization Sets

This section complements § 5.1 with details of the two test sets used to evaluate generalization.

Unseen Objects. We use five animal assets from the Microsoft Rocketbox library [14]: a camel, a horse, a cow, a roe deer, and a sheep. They range from 0.9 m (roe deer) to 2.1 m (camel) in height and from 1.25 m (sheep) to 3.24 m (camel) in length. As for the other inserted objects, each scene is assigned one animal pseudo-randomly from its identifier with a fixed seed, so all four look-ahead variants of a scene show the same animal. All animals stand in their rest pose with all four hooves on the ground. Each animal is placed at the location of the standing pedestrian, i.e., at the ego vehicle’s front-bumper position along its ground-truth trajectory at the look-ahead time τ\tau, but stands side-on, with its body perpendicular to the ego vehicle’s heading at τ\tau and its head pointing to the left or right, chosen pseudo-randomly per scene. Ground estimation, rendering, occlusion handling, and harmonization are identical to those of the other inserted objects (Appendix § 8.1). For collision checking, we use the ground-plane projection of each asset’s bounding box; since the animals are stationary, every contact with them while the ego vehicle is moving is at fault (Appendix § 8.§ 4). We pseudo-randomly select 300 ReactNAV samples and composite an animal into each of them at all four look-ahead times. Compositing fails for one to four samples per look-ahead time, mostly because the rendered animal fails a visibility check or the depth alignment is unreliable. We keep the 291 samples for which all four variants are generated successfully. Figure 13 shows a randomly selected example for each animal.

Grid of edited front-camera images showing randomly selected unseen-object test samples for Camel, Horse, Cow, Roe deer, and Sheep at look-ahead times $\tau = 0.5\,\mathrm{s}$, $\tau = 1.0\,\mathrm{s}$, $\tau = 1.5\,\mathrm{s}$, and $\tau = 2.0\,\mathrm{s}$

Figure 13. Randomly selected samples of the unseen-objects test set, one per animal, for each look-ahead time τ\tau, shown in the edited front-camera image at the current frame tt.

Real Scenes: Selection. For each navtest sample, we determine the ego lane as the lane or lane connector of the nuPlan map under the ego vehicle’s rear axle, preferring lanes on the ego vehicle’s route and then the one best aligned with its heading. Since map lanes are short segments, we append the lanes that follow it until the resulting lane chain extends at least 60 m ahead of the ego vehicle. Where a lane splits into several successors, we take the one on the ego vehicle’s route or, if the route does not single one out, the one the ego vehicle actually drives into according to its logged trajectory. Distances are measured as arc lengths along the centerline of this lane chain, from the ego vehicle’s front bumper to the nearest corner of a road user’s bounding box, and the ego vehicle’s path is the strip of the ego vehicle’s width around this centerline. The target is the nearest vehicle that moves slower than 0.5 m/s, lies entirely within the lane chain, and is 5–15 m ahead. We keep a sample if (a) such a target exists, (b) the ego vehicle moves slower than 3 m/s, (c) no other vehicle, cyclist, or pedestrian, at any speed, lies in the ego vehicle’s path closer than the target, and (d) at least half of the target’s bounding-box area lies within the path. Of the 12,146 navtest samples, 3,453 satisfy (b), 559 also satisfy (a), 347 also satisfy (c), and 308 also satisfy (d).

Real Scenes: Ego-Status Edit. Let gg be the distance from the ego vehicle’s front bumper to the target along the centerline (5.0–13.7 m, median 6.4 m). For each look-ahead time τ∈{0.5,1.0,1.5,2.0}\tau\in\{0.5, 1.0, 1.5, 2.0\} s, we set the ego velocity in all four history frames to v=g/τv = g/\tau along the ego vehicle’s heading and the acceleration to zero, and we redraw the past ego trajectory as if the ego vehicle had driven along the centerline at speed vv, i.e., with consecutive past poses v⋅0.5v \cdot0.5 s apart. The camera images and the driving command are left unchanged. The resulting speeds have medians of 46, 23, 15, and 12 km/h for τ=0.5\tau= 0.5, 1.0, 1.5, and 2.0 s, respectively (ranges 36–99, 18–49, 12–33, and 9–25 km/h).

Real Scenes: Scoring. Since these scenes have no unedited counterpart, we report absolute values instead of edit effects. CS is the contact speed (Appendix § 8.§ 4) at the first at-fault contact with the target, averaged over the predictions with such a contact. Although all targets are nearly stationary at tt, 54 of them move more than 0.5 m within the 4 s horizon, e.g., at the front of a queue that starts moving. We therefore keep the target at its pose at tt during scoring, so that a trajectory is not credited for driving towards a vehicle that later moves away; all other road users follow their logged motion. Table 4 reports the mean of each metric over the four look-ahead times. Figure 14 shows randomly selected examples.

Four randomly selected real-scenes test-set samples in unedited front-camera images

Figure 14. Randomly selected samples of the real-scenes test set, shown in the unedited front-camera image at the current frame tt. The target is the nearest vehicle ahead in the ego lane.

References

  1. [1]Yuxin Bao, Hongwei Ruan, Luobin Wang, Seth Z Zhao, Ziyang Leng, Zihan Zhang, Yu Zeng, Rowan McAllister, Henrik Christensen, and Bolei Zhou. NavSafe-∞: Benchmarking closed-loop driving safety in photorealistic environments. arXiv preprint arXiv:2609.26618, 2026.arxiv.org/abs/2609.26618
  2. [2]Blender Online Community. Blender – a 3D modelling and rendering package. https://www.blender.org, 2024. Blender Foundation, Amsterdam. Version 4.1.
  3. [3]Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11618–11628, 2020.arxiv.org/abs/1903.11027
  4. [4]Wei Cao, Marcel Hallgarten, Tianyu Li, Daniel Dauner, Xunjiang Gu, Caojun Wang, Yakov Miron, Marco Aiello, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. Pseudo-simulation for autonomous driving. In 9th Annual Conference on Robot Learning, 2025.arxiv.org/abs/2506.04218
  5. [5]Matteo Cercola, Katie Z Luo, Simone Formentin, and Marco Pavone. Perceives but doesn’t brake: A perception-action binding gap in reasoning driving VLAs. arXiv preprint arXiv:2610.28184, 2026.arxiv.org/abs/2610.28184
  6. [6]Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. NAVSIM: Data-driven non-reactive autonomous vehicle simulation and benchmarking. In Advances in Neural Information Processing Systems, pages 28706–28719. Curran Associates, Inc., 2024.arxiv.org/abs/2406.15349
  7. [7]Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16. PMLR, 2017.arxiv.org/abs/1711.03938
  8. [8]Matthew Foutter, Matteo Cercola, Lena Wild, Yunshan Wang, Michelle Li, Daniele Gammelli, and Marco Pavone. Do vision-language-action models mean what they say? on the role of faithfulness in embodied reasoning. arXiv preprint arXiv:2607.04681, 2026.arxiv.org/abs/2607.04681
  9. [9]Haoyu Fu, Diankun Zhang, Zongchuan Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, and Xiang Bai. ORION: A holistic end-to-end autonomous driving framework for vision-language instructed action generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 24823–24834, 2025.arxiv.org/abs/2503.19755
  10. [10]Jiacheng Fu, Yibo Yuan, Meng Tian, Yue Li, Jiangtong Zhu, Jianhua Han, Yueyi Zhang, Jianwu Fang, Jianru Xue, Hang Xu, et al. 4D-WAM: 4D consistent world modeling for autonomous driving. arXiv preprint arXiv:2608.10107, 2026.arxiv.org/abs/2608.10107
  11. [11]Bernd Gassmann, Fabian Oboril, Cornelius Buerkle, Shuang Liu, Shoumeng Yan, Maria Soledad Elli, Ignacio Alvarez, Naveen Aerrabotu, Suhel Jaber, Peter van Beek, Darshan Iyer, and Jack Weast. Towards standardization of AV safety: C++ library for Responsibility Sensitive Safety. In 2019 IEEE Intelligent Vehicles Symposium (IV), pages 2265–2271, 2019.
  12. [12]Jiaheng Geng, Jiatong Du, Xinyu Zhang, Ye Li, Panqu Wang, and Yanjun Huang. Driving in corner case: A real-world adversarial closed-loop evaluation platform for end-to-end autonomous driving. arXiv preprint arXiv:2512.16055, 2025.arxiv.org/abs/2512.16055
  13. [13]Simon Gerstenecker, Andreas Geiger, and Katrin Renz. Fail2Drive: Benchmarking closed-loop driving generalization. arXiv preprint arXiv:2604.08535, 2026.arxiv.org/abs/2604.08535
  14. [14]Mar Gonzalez-Franco, Eyal Ofek, Ye Pan, Angus Antley, Anthony Steed, Bernhard Spanlang, Antonella Maselli, Donna Banakou, Nuria Pelechano, Sergio Orts-Escolano, Veronica Orvalho, Laura Trutoiu, Markus Wojcik, Maria V. Sanchez-Vives, Jeremy Bailenson, Mel Slater, and Jaron Lanier. The Rocketbox library and the utility of freely available rigged avatars. Frontiers in Virtual Reality, Volume 1 - 2020, 2020.
  15. [15]Poul Greibe. Braking distance, friction and behaviour: Findings, analyses and recommendations based on braking trials. Technical report, Trafitec, 2007.
  16. [16]Julian Jorge Andrade Guerreiro, Mitsuru Nakazawa, and Björn Stenger. PCT-Net: Full resolution image harmonization using pixel-wise color transformations. In 2023 IEEE/CVF [17] Marcel Hallgarten, Julian Zapata, Martin Stoll, Katrin Renz, and Andreas Zell. Can vehicle motion planning generalize to realistic long-tail scenarios? In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5388–5395, 2024. 3
  17. [18]Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo, Dragomir Anguelov, and Mingxing Tan. EMMA: End-to-end multimodal model for autonomous driving. Transactions on Machine Learning Research, 2025. 1, 2arxiv.org/abs/2410.23262
  18. [19]Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2Drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. In Advances in Neural Information Processing Systems, pages 819–844. Curran Associates, Inc., 2024. 1, 2arxiv.org/abs/2406.03877
  19. [20]Junzhe Jiang, Nan Song, Jingyu Li, Xiatian Zhu, and Li Zhang. RealEngine: Simulating autonomous driving in realistic context. arXiv preprint arXiv:2505.16902, 2025. 3arxiv.org/abs/2505.16902
  20. [21]Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics, 42(4): 1–14, 2023. 3arxiv.org/abs/2308.04079
  21. [22]Yongkang Li, Kaixin Xiong, Xiangyu Guo, Fang Li, Sixu Yan, Gangwei Xu, Lijun Zhou, Long Chen, Haiyang Sun, Bing Wang, Kun Ma, Guang Chen, Hangjun Ye, Wenyu Liu, and Xinggang Wang. ReCogDrive: A reinforced cognitive framework for end-to-end autonomous driving. In The Fourteenth International Conference on Learning Representations, 2026. 1, 2, 4, 7, 20arxiv.org/abs/2506.08052
  22. [23]Mengmeng Liu, Diankun Zhang, Jiuming Liu, Jianfeng Cui, Hongwei Xie, Guang Chen, Hangjun Ye, Francisco Nex, Hao Cheng, and Michael Ying Yang. UNIVERSE: Unified video action models for autonomous driving with flexible mask-modulated modality generation. arXiv preprint arXiv:2607.05133, 2026. 2arxiv.org/abs/2607.05133
  23. [24]William Ljungbergh, Adam Tonderski, Joakim Johander, Holger Caesar, Kalle Åström, Michael Felsberg, and Christoffer Petersson. NeuroNCAP: Photorealistic closed-loop safety testing for autonomous driving. In Computer Vision – ECCV 2024, pages 161–177, Cham, 2025. Springer Nature Switzerland. 3arxiv.org/abs/2404.07762
  24. [25]Ziyi Luo, Zhe Sun, Yehao Lu, Lei Zhou, Lisheng Wu, Xuewei Li, Zegun Qin, and Xi Li. ExceptionDrive: A planning-oriented counterfactual corner-case benchmark for autonomous driving. arXiv preprint arXiv:2609.37871, 2026. 3arxiv.org/abs/2609.37871
  25. [26]Gabriel Manalu, Saad Wazir, Thao Do, Seongah Kim, Hyeon-taek Hwang, Tolcha Yalew Kidane, and Daeyoung Kim. Measuring the reasoning-action coupling in driving vision-language-action models through counterfactuals. arXiv preprint arXiv:2610.30953, 2026. 6arxiv.org/abs/2610.30953
  26. [27]Nicanor Mayumu, Xiaoheng Deng, and Patrick Mukala. Is VLA reasoning faithful? Probing safety of chain-of-causation in autonomous driving models. arXiv preprint arXiv:2605.17268, 2026. 6arxiv.org/abs/2605.17268
  27. [28]Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: representing scenes as neural radiance fields for view synthesis. Commun. ACM, 65(1):99–106, 2021. 3arxiv.org/abs/2003.08934
  28. [29]Li Niu, Wenyan Cong, Liu Liu, Yan Hong, Bo Zhang, Jing Liang, and Liqing Zhang. Making images real again: A comprehensive survey on deep image composition. arXiv preprint arXiv:2106.14490, 2021. 13arxiv.org/abs/2106.14490
  29. [30]NVIDIA. PhysicalAI–Autonomous-Vehicles. https://huggingface.co/datasets/nvidia/PhysicalAI–Autonomous-Vehicles, 2025. Dataset. Accessed: 2026-10-21. 2
  30. [31]NVIDIA. Alpamayo 1.5. https://huggingface.co/nvidia/Alpamayo-1.5-10B, 2026. Model card. Accessed: 2026-10-15. 2, 4, 16
  31. [32]NVIDIA. Alpamayo 2 Super. https://huggingface.co/nvidia/Alpamayo2-Super, 2026. Model card. Accessed: 2026-10-15. 2, 4, 16
  32. [33]Jonas Pai, Liam Achenbach, Oliver Sanchez, Stefanos Charalambous, Victoriano Montesinos, Benedek Forrai, Oier Mees, and Elvis Nava. mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs. In Proceedings of Robotics: Science and Systems, Sydney, Australia, 2026. 2arxiv.org/abs/2512.15692
  33. [34]Yansong Peng, Hebei Li, Peixi Wu, Yueyi Zhang, Xiaoyan Sun, and Feng Wu. D-FINE: Redefine regression task of DETRs as fine-grained distribution refinement. In International Conference on Learning Representations, pages 44015–44031, 2025. 4, 13arxiv.org/abs/2410.13842
  34. [35]Katrin Renz, Long Chen, Elahe Arani, and Oleg Sinavski. SimLingo: Vision-only closed-loop autonomous driving with language-action alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11993–12003, 2025. 1, 2arxiv.org/abs/2503.09594
  35. [36]Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L. Waslander, Yu Liu, and Hongsheng Li. LMDrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15120–15130, 2024. 2arxiv.org/abs/2312.07488
  36. [37]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 6arxiv.org/abs/2402.03300
  37. [38]Chen Shi, Jinrui Xu, Shaoshuai Shi, Kehua Sheng, Bo Zhang, and Li Jiang. DriveWAM: Video generative priors enable scalable world-action modeling for autonomous driving. arXiv preprint arXiv:2605.28544, 2026. 2, 4arxiv.org/abs/2605.28544
  38. [39]Eric Thorn, Shawn Kimmel, and Michelle Chaka. A framework for automated driving system testable cases and scenarios. Technical Report DOT HS 812 623, National Highway Traffic Safety Administration, 2018. 8
  39. [40]Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, XianPeng Lang, and Hang Zhao. DriveVLM: The convergence of autonomous driving and large vision-language models. In 8th Annual Conference on Robot Learning, 2024. 1, 2
  40. [41]Adam Tonderski, Carl Lindström, Georg Hess, William Ljungbergh, Lennart Svensson, and Christoffer Petersson. NeuRAD: Neural rendering for autonomous driving. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14895–14904, 2024. 3arxiv.org/abs/2311.15260
  41. [42]vevalabs. OpenX Assets: Vehicle and traffic simulation assets using ASAM OpenX standards. https://github.com/vevalabs/openx-assets, 2025. Accessed: 2026-10-21. 12, 13
  42. [43]Shihua Wang, Zhid ing Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. OmniDrive: A holistic vision-language dataset for autonomous driving with counterfactual reasoning. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22442–22452. IEEE, 2025. 1, 2arxiv.org/abs/2504.04348
  43. [44]Yan Wang, Wenjie Luo, Junjie Bai, Yulong Cao, Tong Che, Ke Chen, Yuxiao Chen, Jenna Diamond, Yifan Ding, Wenhao Ding, et al. Alpamayo-R1: Bridging reasoning and action prediction for generalizable autonomous driving in the long tail. arXiv preprint arXiv:2511.00088, 2025. 1, 2arxiv.org/abs/2511.00088
  44. [45]Tianze Xia, Yongkang Li, Lijun Zhou, Jingfeng Yao, Kaixin Xiong, Haiyang Sun, Bing Wang, Kun Ma, Guang Chen, Hangjun Ye, Wenyu Liu, and Xinggang Wang. DriveLaW: Unifying planning and video generation in a latent driving world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 39701–39712, 2026. 2arxiv.org/abs/2512.23421
  45. [46]Chejian Xu, Wenhao Ding, Weijie Lyu, Zuxin Liu, Shuai Wang, Yihan He, Hanjiang Hu, Ding Zhao, and Bo Li. SafeBench: A benchmarking platform for safety evaluation of autonomous vehicles. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. 2arxiv.org/abs/2206.09682
  46. [47]Runsheng Xu, Hubert Lin, Wonseok Jeon, Hao Feng, Yu-liang Zou, Liting Sun, John Gorman, Kate Tolstaya, Sarah Tang, Brandyn White, Ben Sapp, Mingxing Tan, Jyh-Jing Hwang, and Dragomir Anguelov. WOD-EX2E: Waymo Open Dataset for end-to-end driving in challenging long-tail scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3709–3718, 2026. 2DOI
  47. [48]Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. DriveGPT4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters, 9(10):8186–8193, 2024. 1, 2arxiv.org/abs/2310.01412
  48. [49]Zhiyuan Xu, Bohan Li, Huan-ang Gao, Mingju Gao, Xin Jin, Hang Zhao, Shuo Feng, Ya-Qin Zhang, and Hao Zhao. Challenger: Affordable adversarial driving video generation for safety testing. Proceedings of the IASEAI Conference, 2(1):767–780, 2026. 3DOI
  49. [50]Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything V2. In Advances in Neural Information Processing Systems, pages 21875–21911. Curran Associates, Inc., 2024. 12
  50. [51]Ruolin Yang, Zilin Huang, Buoyue Wang, Zhengyang Wan, Yuhao Luo, Zihao Sheng, and Sikai Chen. Beyond the leaderboard: Counterfactual diagnosis of end-to-end and VLA driving policies under domain shift. arXiv preprint arXiv:2609.22582, 2026. 3arxiv.org/abs/2609.22582
  51. [52]Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chun ing Zhu, Jiannan Xiang, et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922, 2026. 1, 2
  52. [53]Zongchuang Zhao, Xin Zhou, Tianyang Xu, Zhengyang Sun, Kaixuan Zhou, Yu Wu, Honglin Li, Dingkang Liang, and Xiang Bai. SimWAM: A simple world action model for end-to-end autonomous driving. arXiv preprint arXiv:2608.07468, 2026. 2, 4, 7arxiv.org/abs/2608.07468
  53. [54]Hongyu Zhou, Longzhong Lin, Jiabao Wang, Yichong Lu, Dongfeng Bai, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. HUGSIM: A real-time, photo-realistic and closed-loop simulator for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(4):4673–4691, 2026. 3DOI
  54. [55]Jiawei Zhou, Linye Lyu, Zhuotao Tian, Cheng Zhuo, and Yu Li. SMD: Multi-view safety-critical driving video generation in the real-world domain. In Proceedings of the 43rd International Conference on Machine Learning, pages 165568–165590. PMLR, 2026. 3
  55. [56]Zewei Zhou, Tianhui Cai, Seth Z. Zhao, Yun Zhang, Zhiyu Huang, Bolei Zhou, and Jiaqi Ma. AutoVLA: A vision-language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 1, 2, 4, 17arxiv.org/abs/2506.13757
  56. [57]Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Proceedings of The 7th Conference on Robot Learning, pages 2165–2183. PMLR, 2023. 1, 2arxiv.org/abs/2307.15818

Paper details

vlnc:W_4utm5qak
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Code, Codex

View statement
Looking for related papers…
Contents