betaFeedback

WorksComputer Science

Does the Action Matter? Probing Action-Conditioned Semantic Forecasting as a World Model for Autonomous Driving

Abstract

World models are increasingly used in autonomous driving to predict how a scene evolves under the ego vehicle's actions, supporting counterfactual reasoning and model-based planning. However, existing evaluations focus on prediction fidelity and planning performance, and rarely verify that the predictions depend on the conditioning action. In this paper, we study action dependence in action-conditioned semantic forecasting, which predicts future semantic segmentation from past segmentations and the ego action. We compare a forecaster with feature-wise and input-level action conditioning against an identical model without action input, and measure action dependence with test-time action interventions scored by standard segmentation metrics. The evaluation is validated in a synthetic environment with known action-dependent dynamics and applied to Cityscapes and BDD100K, both within each dataset and across the two vehicle platforms. Experiments show that the executed action improves mid-term mIoU by at most 2.5 points on Cityscapes and 0.6 points on BDD100K, whereas a wrong action reduces it by up to 23.2 and 7.0 points, below the model without action input. On real data, feature-wise conditioning depends 1.6 times more on the action than input-level conditioning at similar accuracy, whereas both depend on it equally in the synthetic environment, where the history does not determine the action. These results show that an accuracy gain indicates that a driving world model uses its action, but not how strongly its forecasts depend on it. Code is available at https://github.com/binayksingh/does-action-matter.

Introduction

World models have become a central component of autonomous driving research. By predicting future scene states conditioned on ego actions, they support counterfactual reasoning, closed-loop simulation and model-based planning. Recent driving world models synthesise future video conditioned on ego trajectories or high-level commands [7, 11, 22], and planning-oriented approaches learn latent or occupancy dynamics jointly with a driving policy [10, 26].

Existing evaluations of these models focus on the fidelity of the generated futures or on downstream planning performance, and rarely verify that the predictions depend on the conditioning action. Over short horizons, however, ego motion is strongly autocorrelated, so the observed history already encodes much of the information carried by the future action. A model trained on logged driving data can therefore attain high forecasting accuracy while making little use of its action input, a failure mode analogous to the inertia and copycat problems reported for imitation-learned driving policies [2, 4, 23]. An accuracy gain over a model without action input shows that the action is used, but not how strongly the forecasts depend on it, since on logged data the executed action and the action implied by the history largely coincide. How strongly action-conditioned forecasters trained on real driving data depend on the ego action, and whether this dependence holds across vehicle platforms, therefore remains largely unexplored.

To address these questions, we study action dependence in semantic forecasting [12], which predicts future semantic segmentation from past observations and has an established evaluation protocol on Cityscapes [3]. Conditioning this task on future vehicle odometry has been proposed before [1], but whether the forecaster uses the odometry was not examined. We condition on the ego action, represented by speed and yaw rate, and compare a U-Net forecaster with feature-wise linear modulation (FiLM) [16], input-level conditioning, and no action input. The rationale is that semantic forecasting has a compact output space and standard metrics, so each variant can be trained with several seeds at modest cost and its use of the action can be measured without new evaluation criteria. To assess whether the findings transfer across vehicle platforms, we define an equivalent protocol on BDD100K [25], which differs from Cityscapes in camera, ego-motion sensing and driving envi @endsection ronment.

To measure action dependence, we evaluate each trained model under test-time action interventions, in which the executed future action is replaced by the action of a sample from another sequence, by zero yaw rate, by a stop command, or by the training-set mean action. All conditions are scored with the standard mIoU and MO-mIoU metrics [12]. Before applying this evaluation to real driving data, we validate it in a synthetic driving environment whose dynamics depend on the action by construction.

Our main contributions are summarized as follows:

  • We present an action-conditioned semantic forecasting framework that compares FiLM, input-level and no action conditioning within a single U-Net forecaster under identical training.

  • We develop a test-time intervention protocol that replaces the executed future action with shuffled, zero-yaw, stop and mean actions, and measures action dependence with established segmentation metrics only.

  • We validate the interventions in a synthetic driving environment with action-dependent dynamics and a negative control trained on shuffled actions, which shows whether a model uses or ignores its action input.

  • We provide a cross-platform evaluation on Cityscapes and BDD100K with GPS-derived ego actions, both within each dataset and across the two platforms.

Related Work

World models and action-conditioned prediction. Predicting future observations conditioned on actions originates in action-conditional video prediction for games [15] and robot manipulation [6], and in world models that learn behaviour inside a learned simulator [8, 9]. In driving, MILE [10] learns a latent world model jointly with a driving policy, OccWorld [26] forecasts 3D occupancy together with ego motion, and GAIA-1 [11], DriveDreamer [22] and Vista [7] synthesise future video conditioned on actions, trajectories or commands. A common theme across these works is that evaluation targets the fidelity of the predicted futures or the performance of a downstream planner. In contrast, we measure directly whether the prediction depends on the action, using a model small enough to be trained with multiple seeds.

Semantic forecasting. Luc Luc et al. [12] introduced the forecasting of future semantic segmentation on Cityscapes, with short-term (0.18 s) and mid-term (0.54 s) horizons that subsequent work adopted. Follow-up methods forecast with convolutional LSTMs [14], recurrent flow-guided warping [21], or in feature space followed by a segmentation decoder [13, 18]. These methods condition on past frames only. Bhattacharyya et al. [1] additionally condition a Bayesian forecaster on past and future vehicle odometry, but report no evaluation of how the predictions depend on it. Our work extends this line of research by measuring that dependence under the standard protocol, and by defining an equivalent protocol on a second platform.

Shortcut learning in driving models. Driving policies trained by imitation are known to exploit the ego vehicle’s past motion instead of the scene, which is referred to as the inertia [2] and copycat [23] problems, and more generally to rely on spurious correlates of the expert action [4]. The forecasting counterpart of this failure mode is a world model that reproduces the observed motion and treats its action input as optional. Test-time input interventions provide a direct test for this behaviour, and we adopt them as our primary evaluation.

Method

In this section, we first formalise action-conditioned semantic forecasting (see § 3.1). We then describe the forecaster and its three action-conditioning variants (§ 3.2), the training objective (§ 3.3), and the action interventions used to measure action dependence (§ 3.4).

Problem formulation

Let St∈{1,…,C}H×WS_t \in\{1,\ldots,C\}^{H \times W} denote the semantic segmentation of the frame at time tt, with CC classes, and let at=(vt,ωt)∈R2a_t = (v_t,\omega_t) \in\mathbb{R}^2 denote the ego action, where vtv_t is the speed in m/s and ωt\omega_t the yaw rate in rad/s, positive for left turns. Frames are sampled with a fixed step Δ\Delta. The context consists of the last four segmentations St=(St−3Δ,…,St)S_t=(S_{t-3\Delta},\ldots,S_t) and the actions at the same frames At=(at−3Δ,…,at)A_t=(a_{t-3\Delta},\ldots,a_t). The future action over the next interval is summarised by its mean

aˉt+Δ=1Δ∫tt+Δaτ dτ.(1)\bar{a}_{t+\Delta} = \frac{1}{\Delta}\int_t^{t+\Delta} a_\tau\,\mathrm{d}\tau. \tag*{(1)}

Given StS_t, AtA_t and aˉt+Δ\bar{a}_{t+\Delta}, a forecaster fθf_\theta predicts class probabilities for the next frame,

Pt+Δ=fθ(St,At,aˉt+Δ)∈[0,1]C×H×W,(2)P_{t+\Delta}=f_\theta(S_t,A_t,\bar{a}_{t+\Delta})\in[0,1]^{C\times H\times W}, \tag*{(2)}

and the predicted segmentation is S^t+Δ=arg⁡max⁡cPt+Δ[c]\hat{S}_{t+\Delta}=\arg\max_c P_{t+\Delta}[c]. Longer horizons are predicted autoregressively. For step k≥1k\ge1, the context is shifted by one frame, with the prediction replacing the oldest observation and the supplied action appended to the past actions:

St+kΔ=(St+(k−1)Δ[2:4], Pt+kΔ),At+kΔ=(At+(k−1)Δ[2:4], aˉt+kΔ).(3)\begin{aligned} S_{t+k\Delta} &= (S_{t+(k-1)\Delta}[2:4],\,P_{t+k\Delta}),\\ A_{t+k\Delta} &= (A_{t+(k-1)\Delta}[2:4],\,\bar{a}_{t+k\Delta}). \tag*{(3)} \end{aligned}

Following [12], the short-term prediction is the first step (t+Δt+\Delta) and the mid-term prediction the third step (t+3Δt+3\Delta). During training and standard evaluation, aˉ\bar{a} is the action executed by the vehicle, which corresponds to the command of an ideal controller; § 3.4 replaces it at test time.

Forecaster

Architecture. Each context segmentation is one-hot encoded (predicted probabilities are used directly during rollout) and the four maps are stacked into an input Xt∈R4C×H×WX_t \in\mathbb{R}^{4C \times H \times W}. A U-Net [17] gθg_\theta with four resolution levels and 64, 128, 256 and 512 channels maps XtX_t to CC logit maps. Each block applies two 3×33 \times3 convolutions, each followed by GroupNorm and a GELU non-linearity. The output logits are

zt+Δ=gθ(Xt,ut)+λ onehot⁡(St),Pt+Δ=softmax⁡(zt+Δ),(4)\begin{aligned} z_{t+\Delta} &= g_\theta(X_t,u_t) + \lambda\,\operatorname{onehot}(S_t),\\ P_{t+\Delta} &= \operatorname{softmax}(z_{t+\Delta}), \tag*{(4)} \end{aligned}

where ut=[At,aˉt+Δ]∈R10u_t=[A_t,\bar{a}_{t+\Delta}] \in\mathbb{R}^{10} is the action vector and λ\lambda is a learned scalar initialised to 4. This residual formulation ensures that the network starts from the copy-last prediction, a strong baseline at short horizons [12], and learns only the change. Actions are standardised with the training-set mean μ\mu and standard deviation σ\sigma before entering the network.

Action conditioning. We compare three variants that share gθg_\theta and differ only in how utu_t enters the network. FiLM. An MLP maps the action vector to a code et=MLP⁡(ut)∈R128e_t=\operatorname{MLP}(u_t)\in\mathbb{R}^{128}. In every block bb, the normalised feature map hbh_b is modulated before the final non-linearity [16],

hb←hb⊙(1+γb(et))+βb(et),(5)h_b \leftarrow h_b \odot(1+\gamma_b(e_t))+\beta_b(e_t), \tag*{(5)}

where γb\gamma_b and βb\beta_b are linear projections to per-channel scales and shifts. Both projections are initialised to zero, so that training starts from the unconditioned network. Concat. The action vector is broadcast to ten constant planes and concatenated with the input, Xt′=[Xt;ut⊗1H×W]X'_t=[X_t;u_t\otimes\mathbf{1}_{H\times W}]. No action. The U-Net gθg_\theta alone, without the MLP and the FiLM layers, which receives no action information. The three variants have 13.07 M (FiLM), 12.56 M (concat) and 12.55 M (no action) parameters.

Training objective

The forecaster is trained with the per-pixel cross-entropy, averaged over K=3K=3 unrolled autoregressive steps,

L(θ)=1K∑k=1K1∣Ωk∣∑p∈Ωk−log⁡Pt+kΔ[St+kΔ(p)](p),(6)\mathcal{L}(\theta)=\frac{1}{K}\sum_{k=1}^{K}\frac{1}{|\Omega_k|}\sum_{p\in\Omega_k}-\log P_{t+k\Delta}[S_{t+k\Delta}(p)](p), \tag*{(6)}

where Ωk\Omega_k is the set of labelled pixels at step kk. Unrolling exposes the model to its own predictions, as in mid-term evaluation. For augmentation, a horizontal flip FF is applied jointly to the segmentations and actions,

(S,v,ω)↦(F(S),v,−ω),(7)(S,v,\omega)\mapsto(F(S),v,-\omega), \tag*{(7)}

so that the flipped scene and action remain physically consistent.

Action interventions

To test whether a forecaster uses its action input, we evaluate the same trained model with the future actions replaced, while the observed segmentations and past actions are kept fixed. For sample ii and step kk, the executed future action aˉk(i)=(vˉk(i),ωˉk(i))\bar{a}_{k}^{(i)} = (\bar{v}_{k}^{(i)}, \bar{\omega}_{k}^{(i)}) is replaced by

Φc(aˉk(i))={aˉk(i),c=true,aˉk(π(i)),c=shuffled,(vˉk(i),0),c=zero yaw,(0,0),c=stop,μ,c=mean,(8)\Phi_{c}\left(\bar{a}_{k}^{(i)}\right) = \begin{cases} \bar{a}_{k}^{(i)}, & c=\text{true},\\ \bar{a}_{k}^{(\pi(i))}, & c=\text{shuffled},\\ (\bar{v}_{k}^{(i)},0), & c=\text{zero yaw},\\ (0,0), & c=\text{stop},\\ \mu, & c=\text{mean}, \end{cases} \tag*{(8)}

where π\pi is a random permutation of the evaluation samples with the constraint that ii and π(i)\pi(i) belong to different driving sequences, since neighbouring samples of one sequence have nearly identical actions. The rollout of Eq. (3) is then run with Φc(aˉ)\Phi_{c}(\bar{a}) in place of aˉ\bar{a}.

Each condition cc is scored with the standard metrics of the task [12]: mIoU over all CC classes and MO-mIoU over the movable-object classes. We report the metric under each condition and its difference from the true condition. For the no-action model, all conditions give identical predictions by construction; a model that has learned to ignore its action input should likewise show differences close to zero. In addition, the response to each condition can be checked for physical plausibility; for example, a model that uses the action should predict a nearly static scene under the stop condition.

Experiments

We evaluate the three conditioning variants of § 3.2 on Cityscapes and BDD100K, measure their action dependence with the interventions of § 3.4, and validate these interventions in a synthetic environment.

Datasets

Cityscapes. Cityscapes [3] contains 30-frame video snippets recorded at approximately 17 Hz, with fine annotations (gtFine) for the 20th frame and vehicle odometry for every frame. We follow the standard forecasting protocol [12]: frames 8, 11, 14 and 17 are used to predict frame 20 (short term, Δ=0.18\Delta=0.18 s), and frames 2, 5, 8 and 11 to predict frames 14, 17 and 20 (mid term, 0.54 s). Models are trained on the 2,975 training snippets and evaluated on the 500 validation snippets against gtFine at 1024×20481024\times2048. Speed and yaw rate are taken from the vehicle odometry.

BDD100K. We use the multi-object-tracking clips of BDD100K [25], recorded at 5 fps and 720×1280720\times1280. A step of one frame (Δ=0.2\Delta=0.2 s) is the closest equivalent to the Cityscapes step, giving short-term (0.2 s) and mid-term (0.6 s) horizons with the same context structure. One sample is drawn every 2 s of each clip. Speed is taken from the GPS, and the yaw rate is the negative time derivative of the GPS course, interpolated from 1 Hz to the frame times. After removing clips with GPS gaps longer than 2 s, implausible yaw rates or an unreadable GPS file, 1,528 of the 1,600 clips remain, giving 3,658 short-term and 3,646 mid-term validation samples. On both datasets, the distance obtained by integrating speed agrees with the GPS distance (regression slope 1.03 and 1.008).

Segmentation inputs and targets. The BDD100K tracking clips have no dense annotations. We therefore segment all required frames of both datasets with SegFormer-B5 [24] trained on Cityscapes (82.2 mIoU on Cityscapes validation). These segmentations are used as inputs and training targets on both datasets and as evaluation targets on BDD100K; Cityscapes is always evaluated against gtFine.

Implementation details

Inputs are resized to 256×512256\times512 on Cityscapes and 288×512288\times512 on BDD100K, and predictions are bilinearly upsampled to the evaluation resolution. All models are trained from scratch with AdamW (learning rate 10−310^{-3}, weight decay 10−410^{-4}), a one-cycle schedule and a batch size of 16, for 60 epochs on Cityscapes and 12 epochs on BDD100K, with horizontal-flip augmentation (Eq. (7)). Each variant is trained with three random seeds, and we report the mean and standard deviation. We compare with two baselines that use no action: copy-last, which repeats the last observed segmentation, and flow warping, which extrapolates the optical flow [5] between the last two context frames to warp the last segmentation. On Cityscapes, we also report published results on the same protocol [1, 12, 18].

Synthetic validation environment

A virtual camera with a 90∘90^\circ field of view observes a horizontally wrapping label panorama with moving cars and pedestrians; the yaw rate pans the view and the speed zooms about the vanishing point. The future action differs from the past action in 70% of the samples, so that past frames alone do not determine the next view. In addition to the three variants, we train a negative control: a FiLM model trained with shuffled actions, which must learn to ignore its action input.

Results

Main results. Table 1 reports forecasting accuracy with the executed action. On Cityscapes, both action-conditioned models outperform the no-action model at both horizons, by 1.12 (FiLM) and 0.86 (concat) mIoU at short term and by 2.46 and 2.12 mIoU at mid term. On BDD100K, the improvement is smaller (0.40 and 0.05 at short term, 0.58 and 0.12 at mid term), and at mid term concat is below the no-action model in MO-mIoU (33.14 vs. 33.23). This indicates that the action adds accuracy mainly on Cityscapes and at the longer horizon. All learned forecasters outperform copy-last and flow warping; at mid term, the no-action model alone exceeds flow warping by 10.97 mIoU on Cityscapes and 5.24 on BDD100K, more than the action adds on either dataset. Figure 2 shows the per-class gains of FiLM over the no-action model on Cityscapes at mid term. The largest gains are for pole (10.1), traffic light (7.6) and traffic sign (7.1 IoU points), and on BDD100K the same three classes gain most (1.51, 1.19 and 1.24). These thin structures cover few pixels, so a small error in their predicted position, which depends on the ego motion, removes much of their IoU.

MethodCityscapes (gtFine)BDD100K (pseudo-labels)
Short termMid termShort termMid term
mIoU↑\uparrowMO↑\uparrowmIoU↑\uparrowMO↑\uparrowmIoU↑\uparrowMO↑\uparrowmIoU↑\uparrowMO↑\uparrow
Copy-last55.7752.5140.8232.6249.0638.3338.2026.37
Flow warping66.1764.8246.0539.7952.8642.4339.8028.50
S2S-dil [12]†^{\dagger}59.455.347.840.8––––
Bayes-WD-SL [1]†^{\dagger}64.8–45.9–––––
F2MF-DN121 [18]†^{\dagger}69.667.757.954.6––––
No action70.16±0.0870.16\pm0.0869.23±0.1569.23\pm0.1557.02±0.1557.02\pm0.1553.15±0.2853.15\pm0.2854.92±0.0854.92\pm0.0843.80±0.1943.80\pm0.1945.04±0.0745.04\pm0.0733.23±0.2133.23\pm0.21
Concat71.02±0.0671.02\pm0.0669.81±0.1269.81\pm0.1259.14±0.1859.14\pm0.1854.60±0.4854.60\pm0.4854.97±0.0754.97\pm0.0743.85±0.1343.85\pm0.1345.15±0.0445.15\pm0.0433.14±0.0833.14\pm0.08
FiLM71.27±0.0671.27\pm0.0670.02±0.1370.02\pm0.1359.47±0.1459.47\pm0.1454.94±0.2654.94\pm0.2655.33±0.0555.33\pm0.0544.38±0.1244.38\pm0.1245.61±0.0945.61\pm0.0933.79±0.2533.79\pm0.25

Table 1. Semantic forecasting results with the executed action (mIoU and MO-mIoU in %; mean ±\pm std over 3 seeds). Models are trained and evaluated on the same dataset. †^{\dagger}Published results use weaker single-frame segmenters than ours and are given for reference only. Best value per column among methods run in our pipeline in bold.

Figure 2. Per-class IoU change of FiLM over the no-action model on Cityscapes at mid term (executed action, mean over three seeds).

Published results on the same protocol use different

single-frame segmenters, so the comparison is indicative only.

Action dependence. Table 2 and Figure 3 report the mid-term accuracy under the action interventions; the no-action model is unaffected by construction. First, the dependence acquired by the models exceeds the accuracy the action adds. On Cityscapes, the shuffled action reduces mIoU by 23.17 (FiLM) and 14.78 (concat), to 36.31 and 44.36, well below the no-action model (57.02). Under the shuffled, stop and mean actions, both models fall below the no-action model on both datasets. This indicates that a wrong action harms the forecasts more than the absence of an action. Second, FiLM depends more on the action than concat at similar accuracy: its drop under the shuffled action is 1.6 times larger on both datasets (23.17 vs. 14.78 and 7.02 vs. 4.50), and it is larger under every intervention. At short term (Table 3), all drops are smaller, since the action affects a single forecasting step, but the ratio is larger (13.20 vs. 6.99 on Cityscapes and 3.08 vs. 1.27 on BDD100K, i.e. 1.89 and 2.43), where the history determines more of the action, and close to one in the synthetic environment (31.57 vs. 30.02, Table 5), where it does not. The difference between the mechanisms therefore appears when the action is partly redundant with the history. Third, all drops are smaller on BDD100K; in particular, zero yaw changes mIoU by at most 0.41, whereas it reduces Cityscapes mIoU by 7.09 and 8.46. This is consistent with the BDD100K actions: 90.1% of the validation samples have a future yaw rate below 0.05 rad/s, compared with 81.3% on Cityscapes.

Figure 3. Change in mid-term mIoU relative to the no-action model when the future action is replaced at test time. Models are trained and evaluated on the same dataset; mean ±\pm std over three seeds.

Train / evalModelTrue mIoU↑ShuffledZero yawStopMean
CityscapesNo action57.02±0.150.000.000.000.00
Concat59.14±0.18−14.78−7.09−11.92−9.94
FiLM59.47±0.14−23.17−8.46−17.03−17.08
BDD100KNo action45.04±0.070.000.000.000.00
Concat45.15±0.04−4.50−0.19−2.64−4.49
FiLM45.61±0.09−7.02−0.41−4.88−6.49

Table 2. Mid-term mIoU (%) with the executed action and its change when the future action is replaced (shuffled: another sequence’s action; zero yaw; stop; training-set mean). Changes are averaged over seeds; their standard deviation is at most 2.47.

Train / evalModelTrue mIoU↑ShuffledZero yawStopMean
CityscapesNo action70.16±0.080.000.000.000.00
Concat71.02±0.06−6.99−3.53−5.22−3.97
FiLM71.27±0.06−13.20−5.28−9.19−8.21
BDD100KNo action54.92±0.080.000.000.000.00
Concat54.97±0.07−1.27−0.08−1.05−0.88
FiLM55.33±0.05−3.08−0.31−2.46−2.03

Table 3. Short-term mIoU (%) with the executed action and its change under the interventions of Table 2, averaged over seeds.

ModelTrue mIoU↑ShuffledZero yawStopMean
Copy-last55.63––––
Flow warping59.40––––
No action59.990.000.000.000.00
Concat86.44−30.02−19.42−30.62−24.75
FiLM87.95−31.57−20.67−32.02−26.14
FiLM (shuffled training)60.37+0.02−0.01+0.01−0.01

Table 5. Synthetic validation environment: mid-term mIoU (%) with the executed action and its change under each intervention (one run per model). Baselines do not take an action.

Cross-platform transfer. Table 4 reports each model on both datasets. All models lose accuracy on the other platform: FiLM trained on Cityscapes reaches 41.36 mIoU on BDD100K (vs. 45.61 when trained on BDD100K), and FiLM trained on BDD100K reaches 51.65 on Cityscapes (vs. 59.47). The action still improves the transferred forecasts over the no-action model, by up to 1.52 mIoU (concat, Cityscapes to BDD100K) and 1.29 (FiLM, BDD100K to Cityscapes). The dependence follows the training platform rather than the evaluation data: FiLM trained on Cityscapes loses 13.42 under the shuffled action on BDD100K, whereas FiLM trained on BDD100K loses 7.78 on Cityscapes.

TrainModelEval: Cityscapes TrueEval: Cityscapes ShuffledEval: BDD100K TrueEval: BDD100K Shuffled
CityscapesNo action57.02±0.150.0040.59±0.180.00
Concat59.14±0.18−14.7842.11±0.18−7.94
FiLM59.47±0.14−23.1741.36±0.09−13.42
BDD100KNo action50.36±0.170.0045.04±0.070.00
Concat50.40±0.38−4.6145.15±0.04−4.50
FiLM51.65±0.03−7.7845.61±0.09−7.02

Table 4. Transfer between platforms: mid-term mIoU (%) with the executed action and its change under the shuffled action, for every training and evaluation dataset.

Synthetic validation.

Table 5 validates the interventions in the synthetic environment, where the future view depends on the action by construction. FiLM exceeds the no-action model by 27.97 mIoU (87.95 vs. 59.99), and the shuffled action reduces the mIoU of FiLM and concat by 31.57 and 30.02. The no-action model and the negative control trained with shuffled actions change by at most 0.02 under every intervention. The interventions therefore separate models that use the action from models that ignore it.

Qualitative results. The interventions panel of Figure 1 shows the mid-term forecasts of a FiLM model during a left turn. Only the forecast with the executed action moves the pedestrians toward the image centre, where they appear in the target; under the shuffled, zero-yaw, stop and mean actions, they remain near the left image border.

Overview of action-conditioned semantic forecasting

Figure 1. Overview of action-conditioned semantic forecasting. A U-Net maps the four context segmentations to logits, which are added to the scaled last context map and normalised by a softmax; the prediction is fed back for the next step. The ego action enters through FiLM, input concatenation, or not at all (bottom left; the no-action model is the U-Net alone). At test time, the future action is replaced to measure action dependence (bottom right). The thumbnails show a Cityscapes validation scene in which the vehicle turns left (vˉ=4.6\bar{v}=4.6 m/s, ωˉ=0.36\bar{\omega}=0.36 rad/s) and the mid-term predictions of a trained FiLM model; the shuffled action (6.5 m/s, 0.00 rad/s) and the training-mean action (6.3 m/s, 0.00 rad/s) both correspond to straight driving.

Model size and inference cost. Table 6 compares the three variants. Concat adds 0.01 M parameters to the no-action model and FiLM adds 0.51 M, while both improve mid-term mIoU by 2.1 and 2.5 points, respectively. The gain is therefore not explained by additional capacity. Action conditioning increases the latency of one forecasting step from 19.8 ms to at most 20.5 ms, and of the three-step rollout from 58.6 ms to 60.7 ms.

MethodParams (M)↓Δ\DeltaParams (M)GFLOPs↓Latency (ms)↓ 1 / 3 stepsmIoU↑ midΔshuf\Delta_{\mathrm{shuf}} mid
No action12.55–200.219.8 / 58.657.020.00
Concat12.56+0.01201.720.0 / 59.159.14−14.78
FiLM13.07+0.51200.220.5 / 60.759.47−23.17

Table 6. Model size and inference cost at 256×512 (fp32, batch 1, one NVIDIA V100-PCIE-32GB). Δ\DeltaParams is relative to the no-action model; GFLOPs count convolutions and matrix products of one forecasting step. mIoU and Δshuf\Delta_{\mathrm{shuf}} (change under the shuffled action) are mid-term values on Cityscapes validation.

Discussion

Across both datasets, the action-conditioned forecasters acquire a dependence on the action that far exceeds the accuracy it adds. On Cityscapes, FiLM gains 2.46 mid-term mIoU from the executed action but loses 23.17 under a shuffled one. The accuracy gain over the no-action model shows that the action is used; the interventions show how strongly the forecasts depend on it. They measure the strength of this dependence, not whether it is correct. The executed action gives the highest accuracy in every in-domain setting, and in Figure 1 only the executed turn moves the scene as observed, which indicates that the dependence acts in the right direction; a stronger dependence, however, is not desirable in itself.

The difference between FiLM and concat is a property of the data as much as of the mechanism. Both depend almost equally on the action in the synthetic environment, where the history does not determine it, and the gap is largest at short term, where the history determines it most. FiLM therefore relies on the action to a larger extent when the action is partly redundant with the history.

The shuffled and stop interventions can be inconsistent with the observed history, e.g. a high speed after a standstill, and lie partly outside the training distribution, so their drops combine the response to a different action with the effect of an implausible input. Zero yaw and the mean action stay within the range of training actions and still reduce Cityscapes mid-term mIoU by 7.09 to 17.08 points. Interventions drawn from samples with a similar history would separate these effects.

Models trained on BDD100K gain less from the action and depend less on it, although the shuffled action changes the speed more on BDD100K than on Cityscapes (7.39 vs. 4.33 m/s on average), so the weaker dependence is not caused by a weaker intervention. However, the two datasets differ in camera, driving environment, frame rate and action source, and the yaw rate interpolated from 1 Hz GPS has a lower temporal resolution than vehicle odometry; our experiments do not separate these factors. On BDD100K, the targets are SegFormer-B5 segmentations rather than human annotations, so our BDD100K conclusions are comparative between models; on Cityscapes, evaluated against gtFine, the same ordering of the variants holds.

Our study is limited to a single front camera, a compact forecaster, semantic rather than photometric prediction, horizons of up to 0.6 s and two datasets, so we do not claim that larger driving world models acquire the same dependence, nor do we relate it to planning performance. Our forecasts and metrics also operate at the class level. Object-level forecasting, for which the context among co-occurring objects has been shown to aid detection in driving scenes [19, 20], would allow the interventions to be evaluated per agent.

Conclusion

In this work, we studied how strongly action-conditioned semantic forecasters depend on the ego action. By replacing the future action at test time and scoring the forecasts with standard segmentation metrics, we measured action dependence for feature-wise, input-level and no action conditioning, after validating the interventions in a synthetic environment. Experiments on Cityscapes and BDD100K show that the executed action improves mid-term mIoU by at most 2.46 points, whereas a wrong action reduces it by up to 23.17 points, below the model without action input. On real data, feature-wise conditioning depends 1.6 times more on the action than input-level conditioning, but not in the synthetic environment, where the history does not determine the action. These findings show that an accuracy gain indicates that a driving world model uses its action but not how strongly its forecasts depend on it, so action dependence should be measured explicitly. In future work, we plan to apply history-consistent interventions to larger video and occupancy world models over longer horizons and to relate action dependence to planning performance.

References

  1. [1]Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Bayesian prediction of future street scenes using synthetic likelihoods. In International Conference on Learning Representations (ICLR), 2019. 1, 2, 4, 5arxiv.org/abs/1810.00746
  2. [2]Felipe Codevilla, Eder Santana, Antonio M. López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 1, 2arxiv.org/abs/1904.08980
  3. [3]Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 1, 4arxiv.org/abs/1604.01685
  4. [4]Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 1, 2arxiv.org/abs/1905.11979
  5. [5]Gunnar Farnebäck. Two-frame motion estimation based on polynomial expansion. In Proceedings of the Scandinavian Conference on Image Analysis (SCIA), pages 363–370, 2003. 4DOI
  6. [6]Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in Neural Information Processing Systems (NeurIPS), 2016. 2arxiv.org/abs/1605.07157
  7. [7]Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 1, 2
  8. [8]David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. 2DOI
  9. [9]Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), 2020. 2arxiv.org/abs/1912.01603
  10. [10]Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 1, 2
  11. [11]Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. GAIA-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023. 1, 2arxiv.org/abs/2309.17080
  12. [12]Pauline Luc, Natalia Neverova, Camille Couprie, Jakob Verbeek, and Yann LeCun. Predicting deeper into the future of semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017. 1, 2, 3, 4, 5arxiv.org/abs/1703.07684
  13. [13]Pauline Luc, Camille Couprie, Yann LeCun, and Jakob Verbeek. Predicting future instance segmentation by forecasting convolutional features. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. 2arxiv.org/abs/1803.11496
  14. [14]Seyed Shaboddin Nabavi, Mrigank Rochan, and Yang Wang. Future semantic segmentation with convolutional LSTM. In Proceedings of the British Machine Vision Conference (BMVC), 2018. 2arxiv.org/abs/1807.07946
  15. [15]Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in Atari games. In Advances in Neural Information Processing Systems (NeurIPS), 2015. 2
  16. [16]Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. 1, 3arxiv.org/abs/1709.07871
  17. [17]Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015. 3arxiv.org/abs/1505.04597
  18. [18]Josip Šarić, Sachа Vražić, and Siniša Šegvić. Dense semantic forecasting in video by joint regression of features and feature motion. arXiv preprint arXiv:2101.10777, 2021. 2, 4, 5DOI
  19. [19]Binay Kumar Singh and Niels Da Vitoria Lobo. Context-aware feature-fusion for co-occurring object detection in autonomous driving. In The Eighth Workshop on Precognition: Seeing through the Future, CVPR, 2026. Non-archival. arXiv:2606.12628. 7arxiv.org/abs/2606.12628
  20. [20]Binay Kumar Singh and Niels Da Vitoria Lobo. Contextually enhanced unknown aware contrastive learning-based co-occurring object detection. IEEE Access, 14:163030–163045, 2026. 7DOI
  21. [21]Adam M. Terwilliger, Garrick Brazil, and Xiaoming Liu. Recurrent flow-guided semantic forecasting. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2019. 2arxiv.org/abs/1809.08318
  22. [22]Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiageng Zhu, and Jiwen Lu. DriveDreamer: Towards real-world-driven world models for autonomous driving. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. 1, 2arxiv.org/abs/2309.09777
  23. [23]Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, and Yang Gao. Fighting copycat agents in behavioral cloning from observation histories. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 1, 2arxiv.org/abs/2010.14876
  24. [24]Enze Xie, Wenhai Wang, Zheyuang Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. SegFormer: Simple and efficient design for semantic segmentation with transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 4arxiv.org/abs/2105.15203
  25. [25]Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 1, 4arxiv.org/abs/1805.04687
  26. [26]Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. OccWorld: Learning a 3D occupancy world model for autonomous driving. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. 1, 2arxiv.org/abs/2311.16038

Paper details

vlnc:W_mpdh3smb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents