betaFeedback

WorksComputer Science

Freeforward4d: Annotation-free feed-forward 4d reconstruction of driving scenes

Abstract

Feed-forward 4D reconstruction enables efficient creation of renderable driving scenes for autonomous driving simulation. However, most existing methods rely heavily on dataset-provided cameras, LiDAR depth, or 3D box annotations, limiting their applicability to unannotated driving videos. Pretrained visual models can predict cameras, depth, and object tracks from RGB alone, but their geometry is mutually inconsistent: multi-view predictions come in short windows with independent scales and frames, while monocular depth is metric only per image. Our key insight is that, once reconciled into a single metric frame, these priors can replace dataset annotations for feed-forward 4D reconstruction. We present FreeForward4D, which calibrates overlapping prediction windows against monocular metric depth and registers them into one frame across time. On this geometry, 2D tracks are associated across cameras in 3D, and objects move through hierarchical motion that combines a shared object velocity with local Gaussian corrections, so that both training and inference use only RGB videos. On Waymo, FreeForward4D achieves 27.66 dB PSNR and 0.836 SSIM while recovering metric-scale cameras and depth without scale alignment. It surpasses STORM, SLARM-F, UFO, DGGT, and StreetForward, which rely on dataset annotations, in full-image PSNR and attains the best RGB fidelity on 8-second windows. RGB-only fine-tuning further adapts the framework to nuScenes and Argoverse 2. Code will be released.

Introduction

Reconstructing renderable dynamic driving scenes from real-world videos is essential for scalable sensor simulation and closed-loop evaluation in autonomous driving. Existing per-scene methods achieve high-fidelity reconstruction by optimizing a dedicated representation for each driving sequence [6, 45, 46]. However, their costly per-scene optimization, often requiring minutes to hours, makes scaling to large collections of driving videos impractical. This computational bottleneck has motivated feed-forward approaches, which learn transferable reconstruction priors across scenes and directly predict renderable 4D representations without scene-specific optimization [19, 26, 30, 47].

Despite these advances in efficiency, existing feed-forward methods still rely heavily on dataset annotations that are costly to acquire at scale. They typically require dataset cameras (intrinsics and poses) to establish scene geometry, LiDAR depth for geometric supervision, and often 3D box annotations or the object trajectories derived from them for dynamic modeling. Recent pose-free methods [3, 52] eliminate the need for camera poses at inference, but still rely on dataset annotations during training. Moreover, pose-free reconstruction is not guaranteed to be metric; DGGT, for instance, aligns its depth to LiDAR before evaluation [3]. Consequently, these dependencies limit the applicability of feed-forward reconstruction to unannotated driving videos, where neither dataset annotations nor metric-scale references are available. This raises a fundamental question: Can feed-forward 4D reconstruction be learned from RGB videos alone, without dataset-provided cameras, LiDAR depth, or 3D box annotations during training or inference, while recovering metric-scale geometry and coherent scene dynamics?

Pretrained visual foundation models now estimate cameras and depth from RGB images [35, 36, 40] and provide object-level observations without 3D box annotations [18, 27]. Their geometry, however, is not directly usable for reconstruction, because it is inconsistent at three levels: in scale, multi-view models predict relative depth within each temporal window while monocular models predict metric depth per image; in coordinate frame, each window has its own reference; and in image grid, predicted intrinsics refer to each model’s input resolution rather than the reconstruction grid. Image-derived object tracks additionally lack cross-camera correspondence and 3D motion, while each timestamp observes only part of a moving object’s surface. Combined naively, these priors yield cameras, depth targets, and tracks that disagree with one another.

Our key insight is that these priors, once reconciled into a single metric frame, can replace dataset cameras, LiDAR depth, and 3D boxes for both reconstruction and dynamics. Building on this insight, we present FreeForward4D, an annotation-free feed-forward framework for dynamic driving scene reconstruction using only RGB-derived conditions (Figure 1). First, metric-consistent geometric prior integration resolves all three inconsistencies: it calibrates the scale of each overlapping window against monocular metric depth, registers the windows into one scene frame, and maps the predicted intrinsics onto the reconstruction grid, yielding metric cameras and depth without ground-truth scale alignment. Second, geometry-guided dynamic reconstruction exploits this common frame to associate 2D tracks across cameras in 3D and to move each object with a shared velocity refined by local Gaussian trajectories. Source-aware temporal fusion further integrates motion-compensated observations across timestamps for novel-time rendering.

Figure 1. Annotation-free feed-forward reconstruction of dynamic driving scenes. Given only multi-view RGB videos, FreeForward4D reconciles pretrained geometric priors into a single metric frame and reconstructs metric-scale 4D Gaussians without dataset-provided cameras, LiDAR depth, or 3D box annotations during training or inference. The reconstructed scene captures object dynamics and supports novel-time rendering at interpolated and extrapolated timestamps. We show a vehicle at t0t_0–t3t_3 (center), synchronized RGB renderings, cross-view object associations, and metric depth (right).

Extensive experiments on Waymo Open [29], nuScenes [1], and Argoverse2 [43] demonstrate the effectiveness of FreeForward4D. Pose-conditioned STORM, SLARM-F, and UFO require dataset cameras as input, and pose-free methods require dataset annotations for training; without either, FreeForward4D achieves higher full-image PSNR than these methods, with the highest RGB fidelity over extended 8-second windows. On Waymo, it further recovers metric-scale cameras without any scale fitting, and RGB-only fine-tuning suffices to adapt it to new datasets.

Our contributions are summarized as follows:

  • We present FreeForward4D, a feed-forward 4D reconstruction framework trained and run on RGB driving videos alone, and show that pretrained priors, once reconciled into a single metric frame, can replace dataset-provided cameras, LiDAR depth, and 3D box annotations.

  • We propose metric-consistent geometric prior integration, which resolves inconsistencies in scale, coordinate frame, and image grid by calibrating overlapping windows against monocular metric depth, registering them into one metric frame, and aligning camera rays with the reconstruction grid, yielding metric cameras and depth without dataset cameras, LiDAR depth, or ground-truth scale alignment.

  • We propose geometry-guided dynamic reconstruction,

which associates per-camera 2D tracks across cameras through their 3D centers in this metric frame rather than 3D box annotations, couples them through a shared object velocity refined by local higher-order Gaussian motion, and fuses context times with per-object normalized weights.

  • Experiments on Waymo Open, nuScenes, and Argoverse2 show that, without any dataset annotations, FreeForward4D surpasses STORM, SLARM-F, UFO, DGGT, and StreetForward in full-image PSNR, attaining the highest RGB fidelity on Waymo 8-second windows.

Related Work

Per-scene driving scene reconstruction. Per-scene driving reconstruction has been extensively explored using neural radiance fields [21] and, more recently, 3D Gaussian Splatting [15]. These representations have also been extended to autonomous driving sensor simulation [11, 32], typically requiring scene-specific optimization for each recorded sequence. Existing methods model scene dynamics through two main paradigms. Object-centric approaches explicitly decompose driving scenes into static backgrounds and individual moving actors, whose motions are modeled using object-level transformations [6, 24, 44, 45, 49, 55, 56]. Such methods typically rely on 3D bounding boxes or tracked object poses to initialize dynamic reconstruction. Scene-level approaches, in contrast, capture dynamics through spatiotemporal fields, scene flow, or time-varying Gaussian attributes, without requiring predefined object trajectories [5, 13, 25, 33, 46].

Despite their different motion representations, both paradigms require computationally expensive scene-specific optimization, often taking hours per sequence and limiting their scalability to large-scale driving datasets.

Feed-forward driving scene reconstruction. Feed-forward methods replace per-scene optimization with networks that directly predict radiance fields or 3D Gaussians from images [2, 4, 37, 51]. In driving scenarios, sparse surround-view observations and complex scene dynamics pose challenges for both geometric reconstruction and temporal consistency. Existing methods address sparse-view geometry through depth estimation and multi-view feature aggregation [10, 20, 31], while dynamic reconstruction incorporates motion-aware aggregation, static–dynamic decomposition, and time-dependent Gaussian representations [19, 26, 47]. Other approaches explore recurrent scene updates, object-centric representations, and future-time synthesis to improve temporal modeling and rendering flexibility [8, 28, 30]. Recent pose-free methods jointly estimate cameras and dynamic representations from unposed images [3, 52], but are still trained with dataset annotations. Flux4D [34] avoids 3D box annotations but requires dataset cameras and LiDAR as input.

In contrast, FreeForward4D enables feed-forward 4D reconstruction from RGB driving videos alone, without requiring dataset-provided cameras, LiDAR depth, or 3D box annotations during training or inference.

Foundation models for geometry and object priors. Large-scale visual pretraining has enabled transferable representations for diverse geometric and semantic tasks [23]. For geometric reconstruction, monocular models provide dense depth and metric geometry estimates [12, 39, 40, 48], while multi-view models recover cross-view correspondences and scene structure from uncalibrated images [17, 35, 41, 42]. Recent advances in model scaling, recurrent inference, and chunk-wise alignment of long sequences have further improved multi-view geometric reconstruction [9, 36, 38]. Combining monocular metric depth with multi-view geometry has been explored to recover the metric scale of a single reconstruction [7]; FreeForward4D builds on such calibration to reconcile scale, coordinate frame, and image grid across overlapping temporal windows, yielding a single metric frame that conditions feed-forward reconstruction. Beyond geometry, open-vocabulary detection and promptable segmentation provide object-level cues that can be propagated across video frames [16, 18, 27].

Method

Problem formulation. We study offline 4D reconstruction of a multi-view driving video I={Itv}\mathcal{I} = \{I_t^v\}, indexed by time tt and camera vv, without dataset-provided cameras, LiDAR depth, or 3D box annotations in training or inference, which we refer to as annotation-free. Frozen visual priors process the full video to estimate camera intrinsics K^tv\hat{K}_t^v, metric camera-to-world poses T^tv\hat{T}_t^v, and 2D object tracks, and provide monocular pseudo-depth for scale calibration and supervision. A feed-forward reconstructor then predicts dynamic Gaussians and their motion from a sparse set of context frames C⊂I\mathcal{C} \subset\mathcal{I} and their derived conditions, which can be rendered at arbitrary target cameras and times. Given the precomputed priors, reconstruction and rendering run in real time (§ 4).

Overview. As illustrated in Figure 2, FreeForward4D consists of two components: metric-consistent geometric prior integration (§ 3.1), which unifies heterogeneous visual predictions into globally consistent metric geometry, and geometry-guided dynamic reconstruction (§ 3.2), which establishes cross-view object associations and models coherent dynamics through hierarchical motion modeling and source-aware temporal fusion.

Figure 2. FreeForward4D pipeline. (a) Frozen visual priors provide windowed cameras, pseudo-depth, and 2D object tracks. (b) Geometric prior integration calibrates the windows against monocular metric depth and aligns them into a single metric frame (before vs. after alignment). (c) This geometry enables cross-view object association, object motion is modeled by a shared velocity and local Gaussian trajectories, and temporal weights aggregate Gaussians from different context times. The dashed line carries 2D tracks.

Metric-Consistent Geometric Prior Integration

Pretrained visual models provide camera and depth estimates from RGB images, but their geometries are inconsistent. tent at three levels. In scale, VGGT-Ω\Omega [36] predicts relative depth within each overlapping temporal window, whereas MoGe-2 [40] provides image-wise metric depth. In coordinate frame, each VGGT-Ω\Omega window has its own reference. In image grid, the predicted intrinsics refer to each model’s input resolution. Left unresolved, the scale and frame inconsistencies misregister Gaussians from different context frames as well as target cameras; § 4.6 quantifies the effect of scale. We resolve them in turn: metric calibration fixes scale, cross-window alignment fixes coordinate frames, and intrinsic conversion aligns camera rays with the reconstruction grid.

Metric calibration. Since overlapping windows share images, the MoGe-2 depth of a shared anchor image provides a common metric reference for all windows that contain it. We therefore relate the relative depth of each window to the metric depth through a window-specific scale sws_w and an image-specific offset bib_i shared across windows:

Mi(p)≈swDwi(p)+bi,sw>0,(1)M_i(p) \approx s_w D_{wi}(p) + b_i,\qquad s_w > 0, \tag*{(1)}

where Dwi(p)D_{wi}(p) is the relative depth at pixel pp of image ii in window ww, and Mi(p)M_i(p) is the corresponding metric depth. All scales and offsets are estimated jointly by minimizing

min⁡{sw,bi}∑w1∣Aw∣∑i∈AwEwi(sw,bi)+λb∑i(biqi)2,(2)\min_{\{s_w,b_i\}} \sum_w \frac{1}{|A_w|}\sum_{i\in A_w} E_{wi}(s_w,b_i) + \lambda_b \sum_i \left(\frac{b_i}{q_i}\right)^2, \tag*{(2)}

where AwA_w is the set of anchor images selected in window ww, EwiE_{wi} is a Huber loss on normalized depth residuals averaged over sampled valid pixels, and the median metric depth qiq_i normalizes the offset penalty weighted by λb\lambda_b. Averaging within each image and window compensates for unequal numbers of valid pixels, while the shared offsets couple windows that observe the same image. The window scales are bounded by empirical depth-ratio intervals for robust fitting. The offsets absorb per-image biases of monocular depth and thus prevent them from biasing the window scales; only the fitted scales are applied to camera translations, leaving the depth supervision unchanged.

Cross-window alignment. The scaled windows are then registered into a common scene frame through rigid SE(3) transformations estimated from overlapping cameras, which preserve the metric scale. Camera estimates shared by multiple windows are fused, and the predicted intrinsics are adjusted for image resizing, cropping, and padding, aligning the camera rays with the reconstruction image grid, onto which RGB, depth, and track masks are also resampled. The resulting poses, rays, and metric depth targets serve as consistent conditions for Gaussian reconstruction and cross-view object association.

Geometry-Guided Dynamic Reconstruction

Feed-forward Gaussian prediction. Given the geometrically consistent camera estimates from § 3.1, we employ a Transformer-based reconstructor that encodes image patches, Plücker ray maps, and timestamps into tokens [47]. To aggregate information across context frames, the scene tokens are recurrently updated [30] and decoded into pixel-aligned 3D Gaussians. Each Gaussian jj originates from a context frame at time τj\tau_j, with its initial center defined as μj=oj+djrj\boldsymbol{\mu}_j = \mathbf{o}_j + d_j\mathbf{r}_j, where oj\mathbf{o}_j and rj\mathbf{r}_j are the camera ray origin and direction, and djd_j denotes the predicted camera-frame depth. The Gaussian head additionally predicts scale, orientation, opacity, color, and lifespan [5]. We retain Gaussians from all context frames to preserve complementary surface observations across views and time, providing the basis for subsequent motion modeling and temporal fusion. These components follow prior designs [5, 30, 47]; § 4.6 ablates our geometry conditions and the object-level dynamics below.

Geometry-guided cross-view object association. Grounding DINO [18] and SAM 2 [27] produce 2D object tracks independently for each camera. Since the decoded Gaussian centers lie in the common scene frame of § 3.1, we associate these tracks in 3D. Specifically, for each track at each context time, we compute its 3D center as the coordinate-wise median of the Gaussian centers within its mask. Tracks from different cameras are compared by the median distance between their 3D centers over shared context times, and mutual nearest neighbors within a distance threshold form a group, with at most one track per camera. Each Gaussian jj inherits the group ID gjg_j of the track covering its pixel, where positive IDs denote tracked foreground objects, including stationary ones, and zero denotes the background.

Hierarchical motion modeling. To predict object motion, backbone features pooled within the masks of each group across views and context times form a temporal descriptor, from which a velocity MLP predicts a shared velocity vg\mathbf{v}_g that couples the Gaussians of group gg across cameras. Cross-view identity is thus established by the predicted geometry, while motion is estimated from features.

While the shared velocity captures the overall translation of an object, it cannot represent spatially varying motion, such as that of a turning vehicle. We therefore refine the foreground Gaussian attributes, including the center μ~j\tilde{\boldsymbol{\mu}}_j, and predict a residual velocity δvj\delta\mathbf{v}_j, acceleration aj\mathbf{a}_j, and jerk jj\mathbf{j}_j with a local motion head. The head takes patch features, pixel RGB values, refined centers, shared velocities, and normalized within-patch coordinates uj\mathbf{u}_j as input. Following the higher-order motion formulation of SLARM [26], the center of Gaussian jj at target time tt is given by

μj(t)=μ~j+(vgj+δvj)Δtj+12ajΔtj2+16jjΔtj3,(3)\boldsymbol{\mu}_j(t) = \tilde{\boldsymbol{\mu}}_j + (\mathbf{v}_{g_j} + \delta\mathbf{v}_j)\Delta t_j + \frac{1}{2}\mathbf{a}_j\Delta t_j^2 + \frac{1}{6}\mathbf{j}_j\Delta t_j^3, \tag*{(3)}

where Δtj=t−τj\Delta t_j = t - \tau_j is the time offset from the source timestamp, measured in seconds. The shared velocity models object-level translation, while the local terms capture spatially varying and non-uniform motion. Only foreground Gaussian centers evolve according to Eq. (3); their scales, orientations, and colors remain fixed, while background centers remain stationary. Opacities are modulated during rendering as described below.

Source-aware temporal fusion. Gaussians reconstructed from different context frames capture complementary surfaces of dynamic objects. We propagate these Gaussians to the target time using Eq. (3) and adaptively weight their contributions according to the temporal distance from their source observations. To account for varying object visibility across context frames, the weights are normalized over the observation times of each object. Formally, let Sg\mathcal{S}_g denote the context times at which object gg is observed in at least one camera. The weight for source time s∈Sgs \in\mathcal{S}_g at target time tt is defined as

wg,s(t)=exp⁡[−(t−s)2/(2σ2)]∑u∈Sgexp⁡[−(t−u)2/(2σ2)],(4)w_{g,s}(t) = \frac{\exp\left[-(t-s)^2/(2\sigma^2)\right]}{\sum_{u \in\mathcal{S}_g}\exp\left[-(t-u)^2/(2\sigma^2)\right]}, \tag*{(4)}

where σ=0.35 s\sigma= 0.35\,\mathrm{s}. Each weight scales the opacity α~j\tilde{\alpha}_j of every Gaussian jj of object gg from source time ss, i.e. αj(t)=α~jwgj,τj(t)\alpha_j(t) = \tilde{\alpha}_j w_{g_j,\tau_j}(t), so that contributions are normalized across observation times. Background Gaussian opacities follow lifespan-based decay [5]. The target image is rendered through differentiable Gaussian rasterization [15], with uncovered pixels composited using a ray-conditioned sky color [47].

Training Objectives

The reconstruction backbone and dynamic heads are jointly trained through target-view rendering. We employ a photometric MSE loss Lrgb\mathcal{L}_{\mathrm{rgb}}, assigning five times higher weights to pixels within object track masks to emphasize object regions. A full-image perceptual loss LLPIPS\mathcal{L}_{\mathrm{LPIPS}} [54] further improves rendering fidelity. For geometric supervision, Ld\mathcal{L}_d measures the discrepancy between rendered depth and MoGe-2 pseudo-depth, with both depth maps normalized by the maximum valid pseudo-depth. We additionally introduce a motion regularization loss Lm\mathcal{L}_m that penalizes excessive Gaussian-specific residual velocity, acceleration, and jerk, encouraging local motion to remain consistent with the shared object-level motion. The overall objective is

L=Lrgb+0.05LLPIPS+0.1Ld+10−4Lm.(5)\mathcal{L} = \mathcal{L}_{\mathrm{rgb}} + 0.05\mathcal{L}_{\mathrm{LPIPS}} + 0.1\mathcal{L}_d + 10^{-4}\mathcal{L}_m. \tag*{(5)}

Experiments

Experimental Setup

Datasets and evaluation protocol. We conduct our primary experiments on the Waymo Open Dataset [29] and further examine cross-dataset adaptation through RGB-only fine-tuning on nuScenes [1] and Argoverse2 [43]. On Waymo, we evaluate all 202 validation scenes using the front-left, front, and front-right cameras at 10 Hz. Following the temporal sampling protocol of STORM [47], each 2s clip uses frames {0,5,10,15}\{0, 5, 10, 15\} as context and evaluates 12 interpolated timestamps {1:4,6:9,11:14}\{1:4, 6:9, 11:14\} and four extrapolated timestamps {16:19}\{16:19\}, yielding 48 target images per complete three-view clip. The evaluation comprises 96,360 target images at 160×240160 \times240 resolution (supplementary Sec. D). As in pose-free evaluation [14, 22], target cameras are estimated from the images: ours come from the same full-video front end as the context cameras (Sec. 3.1), so target frames inform these conditions but never enter the reconstructor (supplementary Sec. E).

Metrics and baselines. We report PSNR and SSIM on full images and on dynamic regions defined by Waymo 3D box annotations following STORM [47], averaging image-wise scores over the evaluation set. Geometric accuracy is measured by depth RMSE (D-RMSE) at pixels with valid LiDAR measurements, without scale alignment. Dataset cameras, LiDAR depth, and 3D box annotations are used only for evaluation.

We compare with pose-conditioned feed-forward methods STORM [47], SLARM-F [26], UFO [30], and PointForward [8], as well as pose-free methods DGGT [3] and StreetForward [52]. STORM and SLARM-F are locally reproduced under matched evaluation settings (supplementary Sec. B); results for other methods are taken from their respective publications unless stated otherwise.

Implementation details. We train the three-view 2s model for 100k iterations; the 8s model is initialized from the 2s checkpoint and trained for another 50k iterations on NVIDIA H200 GPUs (details in supplementary Sec. A.5). Given precomputed RGB-derived conditions, the 2s model reconstructs a three-view clip in 0.21 s and renders each target timestamp in 13 ms on a single RTX 4090.

Novel-Time Synthesis on Waymo

Quantitative comparison. Pose-conditioned STORM, SLARM-F, and UFO require dataset cameras as input; without them, FreeForward4D achieves 1.81, 0.68, and 0.40 dB higher full-image PSNR, using frozen priors reconciled into a single metric frame instead. On dynamic regions, the gains over STORM and SLARM-F increase to 2.32 and 1.41 dB in PSNR, with higher SSIM (0.729 vs. 0.623 and 0.672), consistent with our object-level motion modeling. Among pose-free methods, FreeForward4D exceeds DGGT and StreetForward by 0.25 and 0.65 dB in full-image PSNR, although both are trained with dataset annotations, and remains higher in PSNR under the evaluation protocol of the official DGGT code (supplementary Sec. E). Only PointForward, which uses ground-truth 3D box trajectories, attains higher PSNR on both full images and dynamic regions.

Qualitative comparison. Figure 3 compares extrapolated timestamps. FreeForward4D reconstructs sharper vehicle boundaries and finer appearance details than STORM and SLARM-F while preserving static structures, without dataset cameras.

Waymo qualitative comparison at extrapolated timestamps

Figure 3. Waymo qualitative comparison at extrapolated timestamps. Each example shows the left, front, and right cameras; rows correspond to ground truth, STORM, SLARM-F, and ours. Enlarged regions illustrate vehicle boundaries, temporal ghosting, and fine scene details. The baselines require dataset cameras; ours renders at cameras recovered by its own front end.

Metric Geometry and Motion from RGB

Metric geometry evaluation. FreeForward4D is the only method in Table 1 that uses no dataset cameras, LiDAR, or alignment in either training or evaluation, drawing metric scale from the frozen MoGe-2 prior, whose training data include Waymo LiDAR (supplementary Sec. A): pose-conditioned methods inherit it from dataset cameras and mostly LiDAR, DGGT aligns its depth to LiDAR before evaluation [3], and StreetForward is optionally supervised with ground-truth depth. Its D-RMSE of 7.06 m on Waymo 2s therefore jointly reflects scale, camera, and depth errors.

MethodsAnnotation-freeDynamic regionsFull imageDepth
TrainTestPSNR↑SSIM↑PSNR↑SSIM↑D-RMSE (m)↓
Pose-conditioned feed-forward methods
STORM†××22.040.62325.850.8035.47
UFO××––27.260.8255.45
SLARM-F†××22.950.67226.980.8274.78
PointForward××25.01<u>0.768</u>28.480.8613.35
Pose-free feed-forward methods
DGGT×✓22.800.65127.41<u>0.846</u>3.47
StreetForward×✓24.300.82727.010.8183.14
Ours✓✓<u>24.36</u>0.729<u>27.66</u>0.8367.06

Table 1. Waymo novel-time synthesis. Each clip contains four context timestamps from three cameras and 16 target timestamps (12 interpolated and 4 extrapolated; per-type results in supplementary Sec. D). Train/Test indicate whether the method avoids dataset-provided cameras, LiDAR depth, and 3D box annotations in the respective phase. ✓: without dataset-provided cameras, LiDAR depth, or 3D box annotations; ×: with any of them. † denotes reproduced results; other results are reported in their respective publications under their own protocols, except DGGT’s dynamic-region results, which are from PointForward [8]; supplementary Sec. E evaluates ours under the protocol of the DGGT code. Bold/underline/italic: first/second/third best in RGB metrics. –: unavailable.

Camera accuracy tests the recovered scale directly: each 2s clip is anchored at its first front-camera pose without scale adjustment or trajectory fitting. Across 1,818 clips, the absolute trajectory error is 0.58 m in median (1.08 m mean, 2.55 m P90) over a median ground-truth path length of 11.0 m, while the median relative translation error over 0.5 s is 0.27 m and the median rotation error is 1.20° (supplementary Sec. C). Figure 4 visualizes both aspects: renderings from cameras elevated by 0.5 m preserve road layout and vehicle shapes, and Gaussians on the same vehicle follow parallel, smooth trajectories.

Novel-view rendering and Gaussian trajectories on Waymo

Figure 4. Novel-view rendering and Gaussian trajectories on Waymo. Top: renderings from virtual cameras elevated by 0.5 m relative to the input views. Bottom: time-colored trajectories of selected Gaussians on moving vehicles over 0–1.5 s.

Extended 8s Reconstruction

We evaluate extended temporal windows on 8-second sequences using 16 context frames sampled at 0.5-second intervals. As shown in Table 2, our method achieves the best RGB rendering performance on Waymo, with 27.57 dB PSNR and 0.836 SSIM, surpassing UFO by 0.18 dB and STORM by 3.09 dB. Evaluated without alignment, our D-RMSE (7.11 m) is lower than those of STORM (8.11 m) and GS-LRM (7.37 m) and above that of UFO (5.10 m), all of which use dataset annotations. It also merges static Gaussians over about 20 s without per-scene optimization (supplementary Sec. F).

Rendering and depth qualityFeed-Annot.-
MethodPSNR↑\uparrowSSIM↑\uparrowD-RMSE (m)↓\downarrowforwardfree
Waymo Open
3DGS [15]19.570.51714.42××
PVG [5]22.900.61918.24××
Street Gaussians [45]21.690.60913.17××
DeformableGS [50]21.470.61113.12××
GS-LRM [53]21.810.5847.37✓×
STORM [47]24.480.7368.11✓×
UFO [30]27.390.8305.10✓×
Ours27.570.8367.11✓✓
Cross-dataset adaptation
Ours (nuScenes)25.700.803–✓✓
Ours (Argoverse2)24.920.811–✓✓

Table 2. Extended 8s reconstruction. Each Waymo window uses 16 context frames and 64 target timestamps. Baseline results are reported by UFO [30]. Results on nuScenes and Argoverse2 are obtained through RGB-only fine-tuning of our Waymo-trained 8s model.

Cross-Dataset Adaptation

Because FreeForward4D requires no annotations, adapting it to a new dataset or camera rig requires only raw videos. Whereas the baselines are trained on each target dataset with its annotations, we only fine-tune the Waymo-trained 2s model on nuScenes and Argoverse2 using RGB-derived conditions, with 700 training and 150 validation scenes per dataset and the temporal sampling used on Waymo.

As shown in Table 3, RGB-only fine-tuning alone reaches 26.90 dB PSNR on nuScenes, 0.27 dB above DGGT and 2.36 dB above STORM, both trained on nuScenes with dataset annotations. On Argoverse2, it attains 26.52 dB, 1.55 dB above STORM and within 0.44 dB of the annotation-trained DGGT. Fine-tuning the 8s model in the same way reaches 25.70 dB on nuScenes and 24.92 dB on Argoverse2 (Table 2).

MethodsAnnotation-freenuScenesArgoverse2
TrainTestPSNR↑\uparrowSSIM↑\uparrowPSNR↑\uparrowSSIM↑\uparrow
STORM [47]××24.540.78424.970.791
DGGT [3]×✓26.630.81326.960.831
Ours✓✓26.900.81326.520.829

Table 3. Cross-dataset adaptation on nuScenes and Argoverse2. Full-image novel-time synthesis after adaptation to each target dataset. Train/Test indicate whether the method avoids dataset-provided cameras, LiDAR depth, and 3D box annotations in the respective phase. Ours is fine-tuned from the Waymo model using only RGB-derived conditions, whereas the baselines are trained on each target dataset with dataset annotations; baseline results are from their publications, with STORM as reported by DGGT [3].

Ablation Studies

We ablate each component on Waymo 2s (Table 4) by disabling it in the trained full model at inference, so all variants share the same weights and differ only in the disabled component.

VariantFull image PSNR↑\uparrowFull image SSIM↑\uparrowDynamic regions PSNR↑\uparrowDynamic regions SSIM↑\uparrowDepth D-RMSE (m)↓\downarrow
w/o geom. integ.23.730.67722.570.6087.99
w/o object motion27.260.82922.770.6517.22
w/o cross-view assoc.27.620.83524.260.7267.06
w/o local motion27.550.83324.120.7197.06
w/o accel. & jerk27.640.83524.300.7277.05
w/o normalization27.590.83524.040.7247.01
Ours27.660.83624.360.7297.06

Table 4. Ablation study on Waymo 2s. w/o geom. integ. uses, for each clip, the uncalibrated VGGT-Ω\Omega cameras of the single window covering it; w/o object motion removes grouping, foreground refinement, and shared velocity; w/o local motion retains shared object velocity only; w/o accel. & jerk retains first-order residual motion; w/o normalization removes per-object normalization of temporal weights.

Geometric prior integration. Uncalibrated single-window cameras cause the largest drop in Table 4: 3.93 dB in full-image PSNR and 1.79 dB on dynamic regions. As a 2s clip lies within one window, this variant keeps a consistent coordinate frame but lacks the metric scale: camera translations in relative units no longer match the metric depths, misregistering Gaussians from different context frames and target cameras (Figure 2b). D-RMSE rises from 7.06 to 7.99 m. Cross-window alignment instead registers longer sequences, as in the 8s setting and scene-level merging (supplementary Sec. F).

Hierarchical motion modeling. Removing object motion reduces dynamic-region PSNR by 1.59 dB but full-image PSNR by only 0.40 dB. Disabling only cross-camera association lowers dynamic-region PSNR from 24.36 to 24.26 dB. Of these 1.59 dB, object-level motion (cross-view grouping with a shared velocity) contributes 1.35 dB (22.77

to 24.12 dB), as a single object motion explains most dynamics within 2 s; the local motion head adds the remaining 0.24 dB through foreground refinement with residual velocity (24.30 dB) and acceleration and jerk (24.36 dB) for spatially varying and non-uniform motion. Without any 3D annotation, grouping reaches a pairwise precision/recall of 0.867/0.647 on label-matched tracks (supplementary Sec. A.3).

Source-aware temporal fusion. Without per-object normalization, dynamic-region PSNR drops from 24.36 to 24.04 dB while D-RMSE barely changes. Without it, an object far from all its observations, e.g., at extrapolated times, fades; normalization preserves its opacity while favoring the nearest observation.

Conclusion

We presented FreeForward4D, an annotation-free feed-forward framework that reconstructs dynamic driving scenes from RGB videos. Reconciling frozen visual priors into a single metric frame enables cross-view object association and shared object motion without 3D box annotations; replacing the calibrated cameras with uncalibrated ones costs 3.93 dB in full-image PSNR. Without dataset annotations, FreeForward4D surpasses STORM, SLARM-F, UFO, DGGT, and StreetForward in full-image PSNR on Waymo and adapts to nuScenes and Argoverse2 through RGB-only fine-tuning. Its depth accuracy and dynamics are bounded by the frozen priors; needing no annotations, it can directly benefit from stronger priors and more unannotated videos.

LLM Usage Statement

We used publicly available LLM assistants (Chat-GPT/Codex, Gemini, and Kimi). LLM-based coding agents were used extensively, under the authors’ direction, to implement and debug code for data preprocessing, training, and evaluation, and to run experiments. LLM assistants were also used to edit the manuscript (wording and grammar), to check consistency among text, tables, figures, and references, and to adjust figure labels and layout. The research ideas and method design originate from the authors. The authors verified all code, results, and claims, and take full responsibility for the content of this paper.

Implementation Details

This section details the calibration, rendering, and training used by FreeForward4D.

Pretrained priors. FreeForward4D uses four publicly released models, all kept frozen and never fine-tuned on our data: VGGT-Ω\Omega [36] for relative cameras and depth, MoGe-2 [40] for monocular metric depth, Grounding DINO [18] for open-vocabulary detection, and SAM 2 [27] for mask propagation. Their training data are described in the respective papers; in particular, MoGe-2 is trained on a mixture that includes LiDAR depth from driving datasets such as Waymo and Argoverse2 [40], without a specified split. FreeForward4D itself uses no camera, LiDAR, or box annotation of any dataset for training or inference.

Shared Affine Metric Calibration

Anchors and valid support. On all datasets, VGGT-Ω\Omega processes overlapping windows of 20 consecutive timestamps with three frontal camera views per timestamp (60 images per window); the temporal stride is one timestamp on Waymo and five on nuScenes and Argoverse2. Following [7], we calibrate each window against monocular metric depth; overlapping windows are additionally coupled by shared per-image offsets. Each window selects up to three confidence-ranked anchor images. A fitting pixel must have finite relative depth Dwi(p)>10−4D_{wi}(p) > 10^{-4}, finite metric depth Mi(p)>10−4M_i(p) > 10^{-4}, a valid MoGe-2 mask, and VGGT-Ω\Omega confidence above the image median. An anchor observation with fewer than 100 such pixels is discarded. From each surviving observation we sample at most 2,048 evenly spaced entries of its valid-pixel list, forming PwiP_{wi}; AwA_w is the surviving anchor set. The reference qiq_i is the median metric depth over valid MoGe-2 pixels before window-specific confidence filtering.

Fitting error. For positive window scale sws_w and per-image offset bib_i, the main paper’s fitting term is

Ewi(sw,bi)=1∣Pwi∣∑p∈Pwiρ(swDwi(p)+bi−Mi(p)ϵi(p)),(6)E_{wi}(s_w,b_i) = \frac{1}{|P_{wi}|} \sum_{p \in P_{wi}} \rho\left(\frac{s_wD_{wi}(p) + b_i - M_i(p)}{\epsilon_i(p)}\right), \tag*{(6)}
ϵi(p)=0.05max⁡(Mi(p),0.05qi).\epsilon_i(p) = 0.05\max(M_i(p), 0.05q_i).

Here ρ\rho is the Huber loss with threshold 1.5: ρ(r)=r2/2\rho(r) = r^2/2 for ∣r∣≤1.5|r| \le1.5 and 1.5(∣r∣−0.75)1.5(|r| - 0.75) otherwise. The offset coefficient is λb=200\lambda_b = 200. Thus the penalty is 200∑i(bi/qi)2200\sum_i (b_i/q_i)^2, and the data term averages pixels within anchors and anchors within windows. All quantities are computed from frozen RGB predictions.

Scale bounds and optimization

Let Rw\mathcal{R}_w pool valid ratios Mi(p)/Dwi(p)M_i(p)/D_{wi}(p) from the surviving anchors before pixel subsampling. Each window scale is constrained to the 40th–60th percentile interval of Rw\mathcal{R}_w, which guards against degenerate fits on windows with weak depth support. The initial scale is the median of sampled ratios, projected onto this interval. Iteratively reweighted least squares uses Schur-complement elimination of the shared offsets and projected scale updates. Only the fitted scales modify camera translations, expressed relative to each window’s reference camera. The offsets are used only during calibration, and metric depth targets remain unshifted.

Camera Alignment and Image Grids

An incoming window requires at least nine images shared with the already aligned windows. The alignment rotation is the SO⁡(3)\operatorname{SO}(3) projection of the sum of relative camera rotations. Translation is the coordinate-wise median of the destination camera centers minus the rotated source centers. Pairs pass separate rotation and translation tests at the median residual plus three median absolute deviations; a 10−610^{-6} numerical offset is added to each deviation scale. We refit when at least nine pairs remain. These rigid SE⁡(3)\operatorname{SE}(3) transformations preserve the estimated metric scale. Repeated poses use rotation averaging and median translation, and repeated intrinsics use elementwise medians.

Let AwiΩA_{wi}^{\Omega} map original image coordinates to the prior grid, and AirecA_i^{\mathrm{rec}} map them to the reconstruction grid. The intrinsic matrix KwiΩK_{wi}^{\Omega} predicted on the prior grid is converted by

K^wi=Airec(AwiΩ)−1KwiΩ.(7)\hat{K}_{wi} = A_i^{\mathrm{rec}}(A_{wi}^{\Omega})^{-1}K_{wi}^{\Omega}. \tag*{(7)}

The transforms include resizing, cropping, and padding. RGB, metric depth, and object masks use the reconstruction pixel grid; track identities are resampled with nearest-neighbor interpolation.

Gaussian Construction and Object Motion

Each 8×88 \times8 image patch produces 64 pixel-aligned Gaussians through a two-layer decoder. The ray used for the initial center has camera-frame z=1z = 1 and is transformed to scene coordinates without unit-length normalization. Consequently the predicted depth is camera-frame zz-depth. Unit directions are used for the Plücker encoding. Each Gaussian retains its context time.

2D object tracks. Tracks are computed offline, independently for each camera over the full video with Grounding DINO [18] and SAM 2 [27]. Every 10 frames, Grounding DINO detects objects with the prompt “car. truck. bus. motorcycle. bicycle. pedestrian.” (box and text thresholds 0.25), and SAM 2 converts the detected boxes into masks. A new mask inherits the ID of an existing track when their mask IoU is at least 0.8 and otherwise starts a new track. Masks are propagated forward for 10 frames, and objects first detected at a keyframe are additionally propagated backward for up to 20 frames. Consequently, objects first detected at a later keyframe obtain masks at earlier context frames from frames up to 2 s later, as permitted by the offline setting of the main paper (§ 3). Only the masks and track identities at the selected context timestamps are passed to the reconstructor; target-frame masks are used only to weight the training loss.

Cross-view association. A track observation contributes a 3D center only when its mask contains at least 16 finite Gaussian centers. Coordinate-wise medians are detached before association. Cross-camera mutual nearest neighbors are accepted when their median Euclidean distance over common context times is at most 4 m and they share at least one context time. A group contains at most one track per camera. Tracks without a valid center keep distinct positive IDs.

We evaluate association against Waymo 3D labels, used for evaluation only. Each track is assigned to a labeled object by 2D box overlap (IoU ≥0.3\ge0.3) on context frames, which matches 44.6% of the 83,701 tracks over 2,020 windows. Among cross-camera pairs of matched tracks that share a context time, the 4 m threshold gives a pairwise precision of 0.867 and recall of 0.647; varying it from 3 to 6 m changes the F1 score only from 0.720 to 0.751. Centers of the same object seen by different cameras differ by a median of 1.53 m, and 83.2% of such pairs lie within 4 m, compared with 1.4% of pairs from different objects. These values exclude unmatched tracks and include errors of the label assignment.

Shared and local heads. Mask coverage weights feature pooling over views at each context time. The shared-velocity MLP receives the temporal mean and least-squares temporal slope of pooled features, the observation duration, and log⁡(1+cˉ)/5\log(1+\bar{c})/5, where cˉ\bar{c} is the mean total patch coverage. Its output is bounded componentwise to (−20,20)(-20,20) m/s. Groups observed at fewer than two context times receive zero shared velocity.

The attribute head uses patch features, input RGB, decoded color, and opacity to refine foreground centers, scales, quaternions, opacities, and colors once per reconstruction. The local head takes patch features, RGB, the refined center, shared velocity, and the normalized pixel position within the patch. Residual velocity, acceleration, and jerk have componentwise bounds 4 m/s, 6 m/s², and 12 m/s³, respectively. The temporal trajectory changes Gaussian centers; refined scale, orientation, and color remain fixed during target-time evaluation. Background centers remain fixed.

Temporal Rendering

All views of one frame share the assigned frame timestamp. For object gg, Sg\mathcal{S}_g contains context times with at least one object pixel in any view. The normalized kernel of the main paper uses σ=0.35\sigma=0.35 s and is broadcast to all Gaussians of the same group and context time. If τj\tau_j is Gaussian jj's context time, rendering uses

αjfg(t)=α~jwgj,τj(t),\alpha_j^{\mathrm{fg}}(t)=\tilde{\alpha}_j w_{g_j,\tau_j}(t),
αjbg(t)=αjexp⁡[−(t−τj)22ℓj2].(8)\alpha_j^{\mathrm{bg}}(t)=\alpha_j\exp\left[-\frac{(t-\tau_j)^2}{2\ell_j^2}\right]. \tag*{(8)}

Here α~j\tilde{\alpha}_j is refined foreground opacity, αj\alpha_j is decoded background opacity, and ℓj>0\ell_j>0 is its predicted lifespan. The lifespan is clamped below at 10−610^{-6} for numerical evaluation. Foreground temporal weights sum to one over the observed context times; all views and Gaussians at one context time receive the same weight.

Training Losses

For target pixel pp, let I^p\hat{I}_p and IpI_p be rendered and observed RGB. The binary mask mpm_p indicates a positive SAM track ID. With ωp=1+4mp\omega_p=1+4m_p,

Lrgb=∑pωp∥I^p−Ip∥223∑pωp.(9)\mathcal{L}_{\mathrm{rgb}}=\frac{\sum_p\omega_p\lVert\hat{I}_p-I_p\rVert_2^2}{3\sum_p\omega_p}. \tag*{(9)}

Full-image LPIPS supplies the perceptual loss. Let V\mathcal{V} contain valid MoGe-2 pixels with finite detached MoGe-2 depth DpM>0.01D_p^{\mathrm{M}}>0.01 m. Pseudo-depth supervision is

Ld=1∣V∣∑p∈V∣D^p−DpM∣Dmax⁡M,\mathcal{L}_d=\frac{1}{|\mathcal{V}|}\sum_{p\in\mathcal{V}}\frac{\left|\hat{D}_p-D_p^{\mathrm{M}}\right|}{D_{\max}^{\mathrm{M}}},
Dmax⁡M=max⁡p∈VDpM.(10)D_{\max}^{\mathrm{M}}=\max_{p\in\mathcal{V}}D_p^{\mathrm{M}}. \tag*{(10)}

where D^p\hat{D}_p is rendered camera-frame zz-depth and the maximum is taken over the target views supervised together in one chunk. Foreground Gaussians form F\mathcal{F}. The local coefficient penalty is

Lm=13max⁡(∣F∣,1)∑j∈F∥δvj∥2+0.1∥aj∥2+0.01∥jj∥2.(11)\mathcal{L}_m = \frac{1}{3\max(\lvert\mathcal{F}\rvert,1)}\sum_{j\in\mathcal{F}}\lVert\delta\mathbf{v}_j\rVert^2 + 0.1\lVert\mathbf{a}_j\rVert^2 + 0.01\lVert\mathbf{j}_j\rVert^2. \tag*{(11)}

If Lc\mathcal{L}_c is the main paper’s four-term objective for target chunk cc, the aggregate objective is

Ltrain=4C∑c=1CLc.(12)\mathcal{L}_{\mathrm{train}} = \frac{4}{C}\sum_{c=1}^{C}\mathcal{L}_c. \tag*{(12)}

This keeps the loss scale of a four-chunk clip across clip lengths. The reconstruction backbone and dynamic heads are trained jointly; visual-prior and perceptual models remain frozen.

Optimization. The three-view 2s model is trained for 100k iterations with a global batch size of 64 and an initial learning rate of 10−410^{-4}, using 5k warmup iterations and cosine decay to 10−610^{-6}. The 8s model is initialized from the 2s checkpoint and trained for another 50k iterations with a batch size of 32 and a learning rate of 3×10−53 \times10^{-5}.

Reproduced Baselines

STORM†^{\dagger} and SLARM-F†^{\dagger} in Table 1 of the main paper are retrained with the authors’ official open-source implementations without code modification and evaluated on the same 96,360 target images under the same protocol as ours. With our deduplicated 96,360 targets, STORM†^{\dagger} reaches 25.85 dB full-image PSNR, compared with 26.38 dB reported in [47]; FreeForward4D (27.66 dB) exceeds both.

Accuracy of the Estimated Cameras

We score the estimated cameras on all 202 Waymo validation scenes against the dataset poses, which are used only here. Each 2s clip (the nine non-overlapping 20-frame windows of each scene, starting every 20 frames) is re-based at its first front-camera pose for both trajectories; no scale, rotation, or translation is fitted. ATE is the translation RMSE over the three cameras and 20 frames. RPE is the translation RMSE of 0.5 s relative motions of the same camera. Rotation error is the geodesic angle between the re-based estimated and dataset camera rotations, averaged over the three cameras and 20 frames. Table S5 summarizes the results.

MetricMeanMedianP90
ATE, 2s clip (m)1.080.582.55
RPE, 0.5 s (m)0.530.271.20
Rotation error (°)1.521.202.47

Table S5. Camera accuracy on Waymo validation. 1,818 2s clips across 202 scenes.

The median ground-truth path of a clip is 11.0 m.

Evaluation Protocols

Waymo 2s. The 2s evaluation uses ten 20-frame windows per scene (2,020 in total): nine start every 20 frames, and the

last is aligned to the end of the scene. Target frames shared with the preceding window are counted once, which yields 72,132 interpolated and 24,228 extrapolated images (96,360 in total). Table S6 splits the Waymo 2s results of the main paper by target type. Interpolated targets are frames {1:4,6:9,11:14}\{1:4, 6:9, 11:14\} between context frames, and extrapolated targets are frames {16:19}\{16:19\} after the last context frame. All rows use the same model, cameras, and evaluation set as the main paper. Dynamic-region scores average over images containing dynamic pixels.

TargetsImagesFull image PSNR↑Full image SSIM↑Dynamic regions PSNR↑Dynamic regions SSIM↑Depth D-RMSE↓
Interp.72,13228.090.84824.880.7556.96
Extrap.24,22826.390.79822.960.6607.35
All96,36027.660.83624.360.7297.06

Table S6. Waymo 2s results by target type. Full-image and dynamic-region RGB fidelity and depth error.

Waymo 8s. Each 8s clip spans 80 frames at 10 Hz from the same three cameras at 160 × 240 resolution. Frames 0, 5, …, 75 serve as the 16 context frames, and the remaining 64 timestamps are targets: 60 interpolated and frames {76:79}\{76:79\} extrapolated. Following the protocol of UFO [30], clips start at frames 0, 20, …, 180, and a start ss with s+80≥Ns+80 \ge N, where NN is the scene length, is replaced by a random start in [0,N−80][0,N-80]. This yields 2,019 clips over the 202 validation scenes and 387,648 target images; overlapping clips are not deduplicated.

nuScenes. We use the front-left, front, and front-right cameras. The 12 Hz sweeps are resampled to a 10 Hz grid anchored at the front camera with offsets of at most 60 ms; each side camera takes its nearest exposure, and frames whose three exposures span more than 25 ms are discarded. Images are center-cropped from 1600 × 900 to 1350 × 900 and resized to 160 × 240. Fine-tuning uses the 700 training scenes and evaluation the 150 official validation scenes. The 2s protocol follows Waymo, with context frames {0,5,10,15}\{0,5,10,15\} and clips starting every 20 frames, giving 1,350 clips and 64,800 target images. The 8s protocol uses clips starting at frames 0, 20, . . . , 100, six per scene, giving 900 clips and 172,800 target images.

Argoverse2. We use the front-left, front-center, and front-right ring cameras. The 20 Hz streams are resampled to a 10 Hz grid anchored at the center camera, with offsets of at most 5 ms for the center and 25 ms for the side cameras; images are center-cropped to a 3:2 aspect ratio and resized to 160×240160 \times240. Each of the 150 validation logs forms one sequence of 139–198 frames. The 2s protocol uses non-overlapping 20-frame windows with context frames {0,5,10,15}\{0, 5, 10, 15\}, giving 1,104 windows and 52,992 target images. The 8s protocol uses 80-frame windows with stride 20, giving 654 windows and 125,568 target images.

Target Cameras

Main-paper protocol. Following the pose-free evaluation of AnySplat [14] and Off The Grid [22], target cameras are estimated from the images while the scene is reconstructed from the context frames. Our front end processes the full video in a single pass, so the context cameras, the metric calibration, and the 2D object tracks are obtained with all frames available; target RGB never enters the reconstruction network, consistent with the offline setting of the main paper (§ 3). Rendering a pose-free reconstruction at dataset cameras would first require registering the dataset frame to the frame of the reconstruction, and errors of this registration would enter the image metrics; estimating target cameras in the frame of the reconstruction keeps the evaluation on rendering quality. Unlike AnySplat, which passes the target images through its network to predict their poses, our target frames inform only the front end. Comparing such renderings with pose-conditioned methods rendered at dataset cameras follows DGGT [3].

Protocol of the DGGT code. We also evaluate with the protocol of the official DGGT code [3]. From the context frames {0,5,10,15}\{0, 5, 10, 15\} of each clip, frames 0–15 are rendered and the metrics are averaged over all 16, so the four context frames, which are inputs to the reconstruction, are scored together with the 12 intermediate frames, and the extrapolated frames {16:19}\{16 : 19\} are not evaluated. Target cameras are interpolated between the context cameras, with spherical interpolation for rotation and linear interpolation for translation, rather than estimated from the target images. We keep the clips and the 160×240160 \times240 resolution of Table 1 of the main paper. Because the evaluated frames differ, these results are not directly comparable with Table 1 of the main paper. Table S7 reports the 16 frames together and the 12 intermediate frames alone. DGGT describes its evaluation as covering all frames, with inputs 0, 5, 10, and 15 and predictions for frames 0–19, and describes its reported novel-view synthesis results as computed on interpolated frames; its public code scores frames 0–15. On both frame sets, FreeForward4D attains higher PSNR than the 27.41 dB reported by DGGT. On the same 12 intermediate frames, front-end target cameras give 28.09 dB (Table S6) and interpolated cameras 27.57 dB; the two differ only in the target cameras. Pose-conditioned baselines render at the exact dataset cameras of the target frames, which are consistent with their dataset context cameras; the counterpart for a pose-free method is to estimate the target cameras in its own frame. These estimates are not more accurate than the dataset cameras: their median trajectory error is 0.58 m over 2s clips (Sec. C). Our context cameras are estimated in the same way, so pose errors among them, which the baselines do not incur, affect only our reconstructions.

MethodFramesPSNR↑SSIM↑
DGGT [3]see text27.410.846
Ours0–1528.270.850
Oursintermediate only27.570.829

Table S7. Waymo 2s under the protocol of the official DGGT code. Frames: scored frames of each clip. DGGT as reported in its paper.

Additional Visualizations

Figs. S5 and S6 compare eight additional driving scenes with STORM and SLARM-F, Figure S7 shows predicted Gaussian trajectories, and Figs. S8–S10 show scene-level static reconstruction.

Additional novel-view renderings and Gaussian trajectories, including BEV and zoom-in views with predicted Gaussian trajectories and a time-color scale from 0 to 1.5 s

Figure S7. Additional novel-view renderings and Gaussian trajectories. Renderings from a raised camera and predicted trajectories of Gaussians on moving vehicles, with the same 2s three-view model and 0–1.5 s interval as Figure 4 of the main paper.

Scene-level static reconstruction. Our calibrated metric cameras place static Gaussians predicted by the 8s model over approximately 20 s into a common coordinate system, without per-scene optimization. Figure S8 shows road layout in bird’s-eye view and road markings, vegetation, and buildings in local views.

Scene-level static reconstruction and local views, showing BEV and zoom-in pairs

Figure S8. Scene-level static reconstruction and local views. Each BEV and zoom-in pair shares a static Gaussian representation merged from approximately 20 s of RGB. Orange arrows locate the virtual cameras.

Figure S6. Additional Waymo comparisons. Same layout as Figure S5, illustrating urban structure, road traffic, and complex visibility.

Figure S5. Additional Waymo comparisons. Rows show the ground truth (GT), STORM, SLARM-F, and ours, illustrating nearby vehicles, occlusion, intersections, and lateral traffic.

Additional static reconstructions

Figure S10. Additional static reconstructions. Zoom-ins under the same conditions as Figure S9. The bottom row is a wider view of the scene in Figure S8,Figure S8).

Wider static reconstructions

Figure S9. Wider static reconstructions. Two driving scenes and zoom-ins, each merged from approximately 20 s of static Gaussians. The top row is a wider view of the scene in Figure S8,Figure S8).

References

  1. [1]Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A Multi-modal Dataset for Autonomous Driving. In CVPR, 2020. 2, 6
  2. [2]David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. In CVPR, 2024. 3arxiv.org/abs/2312.12337
  3. [3]Xiaoxue Chen, Ziyi Xiong, Yuantao Chen, Gen Li, Nan Wang, Hongcheng Luo, Long Chen, Haiyang Sun, Bing Wang, Guang Chen, Hongyang Li, Ya-Qin Zhang, Hangjun Ye, and Hao Zhao. D-GGT: Feedforward 4D Reconstruction of Dynamic Driving Scenes using Unposed Images. In CVPR, 2026. 2, 3, 6, 8, 4arxiv.org/abs/2512.03004
  4. [4]Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jan Cham, and Jianfei Cai. MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images. In ECCV, 2024. 3DOI
  5. [5]Yurui Chen, Chun Gu, Junze Jiang, Xiatian Zhu, and Li Zhang. Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Rendering. IJCV, 134(3), 2026. 3, 5, 8arxiv.org/abs/2311.18561
  6. [6]Ziyu Chen, Jiawei Yang, Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojcic, Sanja Fidler, Marco Pavone, Li Song, and Yue Wang. OmniRe: Omni Urban Scene Reconstruction. In ICLR, 2025. 2, 3arxiv.org/abs/2408.16760
  7. [7]Zeren Chen, Xiaoya Lu, Zhijie Zheng, Pengrui Li, Lehan He, Yijin Zhou, Jing Shao, Bohan Zhuang, and Lu Sheng. Geometrically-Constrained Agent for Spatial Reasoning. In CVPR, 2026. 3, 1arxiv.org/abs/2511.22659
  8. [8]Cheng Chi, Xianqi Wang, Hongcheng Luo, Mingfei Tu, Gangwei Xu, Zehan Zhang, Bing Wang, Guang Chen, Hangjun Ye, Sida Peng, Xin Yang, and Haiyang Sun. PointForward: Feedforward Driving Reconstruction through Point-Aligned Representations. In NeurIPS, 2026. 3, 6, 7arxiv.org/abs/2605.11594
  9. [9]Kai Deng, Zexin Ti, Jiawei Xu, Jian Yang, and Jing Xie. VGGT-Long: Chunk it, loop it, align it – pushing VGGT’s limits on kilometer-scale long RGB sequences. In ICRA, 2026. 3arxiv.org/abs/2507.16443
  10. [10]Huasong Han, Kaixuan Zhou, Xiaoxiao Long, Yusen Wang, and Chunxia Xiao. GGS: Generalizable Gaussian Splatting for Lane Switching in Autonomous Driving. In AAAI, 2025. 3
  11. [11]Georg Hess, Carl Lindström, Maryam Fatemi, Christoffer Petersson, and Lennart Svensson. SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving. In CVPR, 2025. 3arxiv.org/abs/2411.16816
  12. [12]Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation. TPAMI, 46(12):10579–10596, 2024. 3DOI
  13. [13]Nan Huang, Xiaobao Wei, Wenzhao Zheng, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. S³Gaussian: Self-Supervised Street Gaussians for Autonomous Driving. In ICRA, pages 10283–10290, 2026. 3
  14. [14]Lihan Jiang, Yucheng Mao, LInning Xu, Tao Lu, Kerui Ren, Yichen Jin, Xudong Xu, Mulin Yu, Jiangmiao Pang, Feng Zhao, Dahua Lu, and Bo Dai. AnySplat: Feed-forward 3D Gaussian splatting from unconstrained views. arXiv preprint arXiv:2505.23716, 2025. 6, 4arxiv.org/abs/2505.23716
  15. [15]Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics, 42(4), 2023. 3, 5, 8arxiv.org/abs/2308.04079
  16. [16]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. In ICCV, 2023. 3arxiv.org/abs/2304.02643
  17. [17]Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding Image Matching in 3D with MASt3R. In ECCV, 2024. 3arxiv.org/abs/2406.09756
  18. [18]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. In ECCV, 2024. 2, 3, 5, 1arxiv.org/abs/2303.05499
  19. [19]Hao Lu, Tianshuo Xu, Wenzhao Zheng, Yunpeng Zhang, Wei Zhan, Dalong Du, Masayoshi Tomizuka, Kurt Keutzer, and Yingcong Chen. DrivingRecon: Large 4D Gaussian Reconstruction Model for Autonomous Driving. In NeurIPS, 2025. 2, 3arxiv.org/abs/2412.09043
  20. [20]Sheng Miao, Jiaxin Huang, Dongfeng Bai, Xu Yan, Hongyu Zhou, Yue Wang, Bingbing Liu, Andreas Geiger, and Yiyi Liao. EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis. In CVPR, 2025. 3arxiv.org/abs/2503.20168
  21. [21]Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 2020. 3arxiv.org/abs/2003.08934
  22. [22]Arthur Moreau, Richard Shaw, Michal Nazarczuk, Jisu Shin, Thomas Tanay, Zhensong Zhang, Songcen Xu, and Eduardo Pérez-Pellitero. Off the grid: Detection of primitives for feed-forward 3D Gaussian splatting. arXiv preprint arXiv:2512.15508, 2025. 6, 4arxiv.org/abs/2512.15508
  23. [23]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Isham Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. TMLR, 2024. 3arxiv.org/abs/2304.07193
  24. [24]Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural Scene Graphs for Dynamic Scenes. In CVPR, 2021. 3arxiv.org/abs/2011.10379
  25. [25]Chensheng Peng, Chengwei Zhang, Yixiao Wang, Chenfeng Xu, Yichen Xie, Wenzhao Zheng, Kurt Keutzer, Masayoshi Tomizuka, and Wei Zhan. DeSiRe-GS: 4D Street Gaussians for Static-Dynamic Decomposition and Surface Reconstruction for Urban Driving Scenes. In CVPR, 2025. 3arxiv.org/abs/2411.11921
  26. [26]Zhicheng Qiu, Jiarui Meng, Tong-an Luo, Yican Huang, Xuan Feng, Xuanfu Li, and Zhan Xu. SLARM: Streaming and Language-Aligned Reconstruction Model for Dynamic Scenes. In CVPR, 2026. 2, 3, 5, 6arxiv.org/abs/2603.22893
  27. [27]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment Anything in Images and Videos. In ICLR, 2025. 2, 3, 5, 1arxiv.org/abs/2408.00714
  28. [28]Qi Song, Yifei He, Chi Zhang, Zheng Fu, Xuhe Zhao, Mengmeng Yang, Kun Jiang, Rui Huang, and Diange Yang. Envision4D: Envisioning Visual Futures via Feed-forward 4D Gaussian Splatting for Autonomous Driving. arXiv:2606.10656, 2026. 3arxiv.org/abs/2606.10656
  29. [29]Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Vijay S. Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chen, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Sheng Zhao, Shuyang Cheng, Yu Zhang, Jonathon Shlens, Zhifei Chen, and Dragomir Anguelov. Scalability in Perception for Autonomous Driving: Waymo Open Dataset. In CVPR, 2020. 2, 6arxiv.org/abs/1912.04838
  30. [30]Kaiyuan Tan, Yingying Shen, Ziyue Zhu, Mingfei Tu, Haohui Zhu, Haiyang Sun, Bing Wang, Guang Chen, and Hangjun Ye. UFO: Unifying Feed-Forward and Optimization-based Methods for Large Driving Scene Modeling. In CVPR, 2026. 2, 3, 5, 6, 8arxiv.org/abs/2602.20943
  31. [31]Qijian Tian, Xin Tan, Yuan Xie, and Lizhuang Ma. Driving-Forward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Input. In AAAI, 2025. 3
  32. [32]Adam Tonderski, Carl Lindström, Georg Hess, William Ljungbergh, Lennart Svensson, and Christoffer Petersson. NeuRAD: Neural Rendering for Autonomous Driving. In CVPR, 2024. 3arxiv.org/abs/2311.15260
  33. [33]Haithem Turki, Jason Y. Zhang, Francesco Ferroni, and Deva Ramanan. SUDS: Scalable Urban Dynamic Scenes. In CVPR, 2023. 3arxiv.org/abs/2303.14536
  34. [34]Jingkang Wang, Henry Che, Yun Chen, Ze Yang, Lily Goli, Sivabalan Manivasagam, and Raquel Urtasun. Flux4D: Flow-based Unsupervised 4D Reconstruction. In NeurIPS, 2025. 3arxiv.org/abs/2512.03210
  35. [35]Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. VGGT: Visual Geometry Grounded Transformer. In CVPR, 2025. 2, 3arxiv.org/abs/2503.11651
  36. [36]Jianyuan Wang, Minghao Chen, Shangzhan Zhang, Nikita Karaev, Johannes Schönberger, Patrick Labatut, Piotr Bojanowski, David Novotny, Andrea Vedaldi, and Christian Rupprecht. VGGT-2. In CVPR, 2026. 2, 3, 4, 1
  37. [37]Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. IBRNet: Learning Multi-View Image-Based Rendering. In CVPR, 2021. 3arxiv.org/abs/2102.13090
  38. [38]Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. Continuous 3D Perception Model with Persistent State. In CVPR, 2025. 3arxiv.org/abs/2501.12387
  39. [39]Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision. In CVPR, 2025. 3arxiv.org/abs/2410.19115
  40. [40]Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, and Jiaolong Yang. MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details. In NeurIPS, 2025. 2, 3, 4, 1arxiv.org/abs/2507.02546
  41. [41]Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUSt3R: Geometric 3D Vision Made Easy. In CVPR, 2024. 3arxiv.org/abs/2312.14132
  42. [42]Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He. π³: Permutation-Equivariant Visual Geometry Learning. In ICLR, 2026. 3
  43. [43]Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting. In NeurIPS Datasets and Benchmarks, 2021. 2, 6
  44. [44]Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantaо Chen, Runyi Yang, Yuxin Huang, Xiaoyu Ye, Zike Yan, Yongliang Shi, Yiyi Liao, and Hao Zhao. MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving. In CICAI, 2023. 3DOI
  45. [45]Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting. In ECCV, 2024. 2, 3, 8arxiv.org/abs/2401.01339
  46. [46]Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, and Yue Wang. EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision. In ICLR, 2024. 2, 3arxiv.org/abs/2311.02077
  47. [47]Jiawei Yang, Jiahui Huang, Boris Ivanovic, Yuxiao Chen, Yan Wang, Boyi Li, Yurong You, Apoorva Sharma, Maximilian Igl, Peter Karkus, Danfei Xu, Yue Wang, and Marco Pavone. STORM: Spatio-Temporal Reconstruction Model For Large-Scale Outdoor Scenes. In ICLR, 2025. 2, 3, 5, 6, 8arxiv.org/abs/2501.00602
  48. [48]Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything V2. In NeurIPS, 2024. 3arxiv.org/abs/2406.09414
  49. [49]Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. UniSim: A Neural Closed-Loop Sensor Simulator. In CVPR, 2023. 3arxiv.org/abs/2308.01898
  50. [50]Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction. In CVPR, 2024. 8arxiv.org/abs/2309.13101
  51. [51]Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural Radiance Fields from One or Few Images. In CVPR, 2021. 3arxiv.org/abs/2012.02190
  52. [52]Zhongrui Zhu, Zhao Wang, Yida Wang, Xueyang Zhang, Yifei Zhan, and Kun Zhan. StreetForward: Perceiving Dynamic Street with Feedforward Causal Attention. In ECCV, 2026. 2, 3, 6
  53. [53]Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting. In ECCV, 2024. 8arxiv.org/abs/2404.19702
  54. [54]Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR, 2018. 5arxiv.org/abs/1801.03924
  55. [55]Hongyu Zhou, J iahao Shao, Lu Xu, Dongfeng Bai, Weichao Qiu, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting. In CVPR, 2024. 3arxiv.org/abs/2403.12722
  56. [56]Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes. In CVPR, 2024. 3arxiv.org/abs/2312.07920

Paper details

vlnc:W_wgyezgfy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Chat-GPT, Codex, Gemini (multiple versions), Kimi (multiple versions)

View statement
Looking for related papers…
Contents