betaFeedback

WorksComputer Science

Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles

Abstract

We introduce Foundation4D, a unified feed-forward model for geometry estimation, 3D detection, semantic occupancy, and Gaussian reconstruction from multi-camera driving video. A shared backbone uses factorized spatiotemporal attention to efficiently combine observations across cameras and time. Our video inference pipeline uses pose graph optimization to refine camera trajectories and assembles object tracks across input windows. We also develop a data annotation pipeline and release PAI-Foundation4D, a benchmark with dense depth and semantic occupancy labels for 1,253 driving clips. With benchmark-specific fine-tuning, Foundation4D achieves the lowest depth AbsRel on Waymo and nuScenes, the highest detection mAP on nuScenes, and the highest occupancy mIoU on OpenOccupancy among the compared methods. Our task-combination study shows benefits for both detection and geometry: co-training detection with geometry and occupancy improves top-class mAP by 1.2 points over detection-only training, while co-training geometry with detection reduces camera-to-rig translation error by 13.4%13.4\% relative to geometry-only training. We further use the predicted geometry and trajectories as priors for lidarless NuRec, supporting reconstruction of driving scenes for simulation.

RGB input, 3D detection, depth, semantic occupancy, and Gaussian rendering

Figure 1. From camera video to complementary scene representations. A shared view illustrates RGB input, 3D detection, depth, semantic occupancy, and Gaussian rendering.

Introduction

Two central goals in the development of autonomous vehicle systems are understanding the driving environment and reconstructing recorded scenes for simulation. Both require recovering scene geometry, objects, their motion, and the space they occupy. Unified perception models learn these capabilities through shared visual representations, avoiding the repeated processing of the same observations by separate networks. UniAD and PARA-Drive connect perception with prediction and planning [1, 2], while GeoUP jointly predicts depth, 3D objects, and semantic occupancy from driving video [3]. These outputs capture much of the structure and motion needed for scene reconstruction. Simulation additionally requires recovering scene appearance so that the environment and its moving actors can be rendered from novel viewpoints. Unifying perception and reconstruction would therefore enable a shared model to both understand real driving scenes and recreate them for simulation.

Recent progress in feed-forward reconstruction offers a basis for this integration. VGGT [4] jointly recovers camera parameters and dense geometry from multiple images, while Gaussian reconstruction models such as Instant NuRec [5] predict renderable driving scenes without per-scene optimization. These advances make it possible to infer scene structure and appearance directly from images, but extending them to a shared model for multi-camera video requires efficient use of observations across views and time. In particular, the global attention used in VGGT grows quadratically with the total number of image tokens, making longer sequences costly to process.

We present Foundation4D, a feed-forward model for driving perception and reconstruction. A shared backbone extracts features from multi-camera video, and task-specific decoders predict camera poses and metric depth, 3D objects, semantic occupancy, and Gaussian attributes in a single pass over an input window. Factorized spatiotemporal attention reduces the cost of combining information across cameras and time. To use these predictions over a complete driving recording, geometric estimates must remain aligned and object identities must be maintained across windows. Our video-level inference pipeline assembles the window predictions into scene geometry and camera and object trajectories over long sequences. These outputs support Gaussian scene reconstruction and also provide priors for lidarless NuRec [6], demonstrating their use in a downstream neural reconstruction workflow.

Training and evaluating these capabilities requires dense annotations that remain aligned with the changing scene. Sparse LiDAR observations can be combined to improve surface coverage, but vehicle and object motion make it difficult to align these measurements with camera images. Asynchronous sensing and rolling-shutter exposure further complicate this alignment. We provide a pipeline for the creation of accurate depth images and semantic occupancy. We release the PAI–Foundation4D benchmark [7], 1,253 clips of the Physical AI Autonomous Vehicles (PAI) dataset [8] that support geometry, detection, and occupancy evaluation on the same driving scenes. By providing dense annotations together with documented evaluation protocols, the benchmark gives the community a common basis for comparing methods and measuring progress across these tasks.

We evaluate Foundation4D on established public benchmarks and PAI–Foundation4D. With benchmark-specific fine-tuning, it achieves the lowest depth error on Waymo [9] and nuScenes [10], the highest detection accuracy on nuScenes, and the highest semantic occupancy accuracy on OpenOccupancy [11] among the compared methods. Our evaluations show that Foundation4D can be adapted effectively across driving datasets. Longer temporal context improves detection without retraining, while pose graph optimization (PGO) with a motion prior improves camera trajectory accuracy over sequential stitching on PAI (Table 4 and Table 9). Our task-combination study also reveals benefits and trade-offs of joint perception learning, with gains that depend on the tasks being combined.

Geometry results on all 1,253 PAI test clips with six cameras. † denotes internal fine-tuning. Metrics and evaluation alignment follow § 7.1.

ModelInference ModeTrajectoryTrajectoryTrajectoryRigRigRigDepthDepth
ATE↓\downarrow (cm)RTE↓\downarrow (cm)RRE↓\downarrow (°)RTE↓\downarrow (cm)RRE↓\downarrow (°)mAA@15↑\uparrow (%)AbsRel↓\downarrowδ1.25↑\delta_{1.25} \uparrow (%)
VGGT†Stitching111.128.310.12229.260.52042.500.08494.88
Foundation4DStitching78.064.690.0703.900.40488.720.08194.98
Foundation4DPGO+LM52.752.280.0663.920.40488.710.07995.03

Table 4. Geometry results on all 1,253 PAI test clips with six cameras. † denotes internal fine-tuning. Metrics and evaluation alignment follow § 7.1.

TimestampsSpan (s)mAP ↑\uparrowHOTA ↑\uparrow
20.0737.036.7
40.2039.539.1
80.4742.140.5
161.0044.942.0
322.0747.043.6
483.1347.643.3

Table 9. Temporal-context ablation with a fixed checkpoint. Metrics are five-class averages in percentages; HOTA uses center-distance matching with four-camera visibility-filtered ground truth. Temporal span is (T−1) 2/30(T - 1)\,2/30 seconds for TT timestamps at stride two in 30 Hz video.

Our reconstruction experiments show that using predicted ego poses preserves overall rendering quality close to that obtained with ground-truth ego poses (Table 15). The predicted geometry and actor trajectories also provide useful priors for lidarless NuRec [6], demonstrating their practical value in reconstructing driving scenes for simulation.

Target setRegionInstant NuRecFoundation4D with GT ego poseFoundation4D with predicted ego pose
PSNR (dB) ↑\uparrowSSIM ↑\uparrowPSNR (dB) ↑\uparrowSSIM ↑\uparrowPSNR (dB) ↑\uparrowSSIM ↑\uparrow
OverallAll22.660.7824.020.8023.940.80
Static22.830.7824.340.8124.270.81
Dynamic20.010.7220.030.7319.980.73
InterpolationAll22.860.7823.980.8023.910.80
Static23.040.7824.300.8124.220.80
Dynamic20.200.7220.050.7320.030.73
ExtrapolationAll20.340.7520.770.7620.650.76
Static20.510.7621.200.7721.090.76
Dynamic17.710.6616.000.6315.840.62

Table 15.

Our contributions are:

  • Foundation4D, a feed-forward perception and reconstruction model that shares multi-camera video features across task-specific decoders for geometry, semantics, objects, occupancy, and Gaussian reconstruction.

  • A video-level inference pipeline that aligns window-level geometry and assembles object trajectories, improving trajectory estimation over long driving sequences.

  • A data annotation pipeline that accounts for sensor timing, ego and actor motion, and visibility to generate depth and semantic occupancy labels from driving recordings.

  • The PAI–Foundation4D benchmark, with dense annotations and evaluation protocols for comparing geometry, detection, and occupancy methods on a common set of driving scenes.

Related work

Foundation4D brings geometric recovery, semantic perception, and Gaussian reconstruction into a shared model for multi-camera driving video. It builds on two closely related research directions: multi-task learning for autonomous driving, perception and reconstruction tasks.

Multi-task learning for autonomous driving

Multi-task learning allows driving models to share features across different descriptions of the same scene. PnPNet [12] and ViP3D [13] jointly model detection, tracking, and motion forecasting, while BEVFormer [14] and PETRv2 [15] explore shared representations for detection and segmentation. UniVision [16] jointly learns 3D detection and occupancy through interacting BEV and voxel features. Planning-oriented approaches further connect scene understanding to driving decisions [17, 18, 19]. UniAD [1] passes information between perception, prediction, and planning through task queries; PARA-Drive [2] studies these dependencies and co-trains parallel task heads through shared BEV features. The benefits of joint learning depend on which tasks are combined [20], making task interactions an important consideration alongside the range of outputs a model supports.

Pretraining offers a complementary route to representations that support multiple tasks. UniPAD [21] learns transferable features through volumetric rendering with image and depth reconstruction objectives, while DriveWorld [22] uses temporal scene and action prediction for downstream adaptation. Geometric foundation models such as VGGT [4] learn multi-image features for recovering cameras and dense scene structure. These reconstruction objectives provide a geometric basis for extending a shared backbone to driving perception.

Recent driving models extend geometric representations to perception and planning. DVGT-2 [23] couples streaming geometry reconstruction with trajectory planning. Closely related to our approach, GeoUP [3] adapts VGGT to calibrated multi-camera video through factorized temporal and cross-view attention, conditioning its features on known camera intrinsics and poses. It jointly predicts metric depth, 3D objects, and semantic occupancy, with auxiliary camera-pose supervision and multi-dataset training that accommodates missing task annotations. USR-Drive [24] connects object detection and renderable reconstruction by jointly denoising 3D box and Gaussian latents from multi-view video with known camera poses. Foundation4D combines geometric and semantic perception with Gaussian reconstruction in a shared feed-forward model, while explicitly estimating ego motion and camera-to-rig calibration. We train the shared model on large-scale, partially annotated driving data and examine transfer and interference between geometry, detection, and occupancy.

Perception and reconstruction tasks

Geometric reconstruction.

Feed-forward geometry models learn to recover 3D structure and camera geometry from images. DUSt3R and MASt3R establish pointmap- and correspondence-based reconstruction [25, 26], while VGGT [4] jointly predicts cameras and dense geometry from multiple images. Related models explore alternative geometric parameterizations [27, 28], optional camera and depth priors [29, 30], and efficient multi-image inference [31, 32, 33]. Temporal extensions maintain scene state or cached features for incremental reconstruction [34, 35, 36], while dynamic models recover time-varying structure and point trajectories [37, 38, 39].

Driving adaptations exploit the fixed relationships between vehicle-mounted cameras and the motion of the rig. DVGT [40] predicts metric pointmaps and ego poses from multi-camera images without supplied calibration. Rig3R [41] supports optional rig metadata and inference of missing camera geometry. With calibration available, DriveVGGT [42] uses calibration-guided attention and factorized heads for rig geometry and ego motion. TRIG [43] instead separates trajectory and rig priors in feature encoding and supervision while retaining the VGGT prediction heads. LiAuto-GeoX [44] addresses inference efficiency by distilling dense geometry and cross-view relations into a compact student.

Scaling reconstruction to long videos requires managing memory and accumulated drift. DVGT-2 [23] carries temporal information through a sliding-window cache, while Drive3R [45] combines block-wise causal attention with confidence-aware re-anchoring under known camera calibration. Complementary systems assemble local reconstructions through geometric alignment and global optimization. VGGT-Long [46] aligns overlapping chunks with confidence-weighted similarity transforms and uses loop constraints to correct drift. VGGT-SLAM [47] optimizes submap alignments on SL(4)SL(4) to account for projective ambiguity. VGGT-SLAM 2.0 [48] refines keyframe-level geometry through constraints within and between submaps, and uses VGGT attention to verify loop closures. Foundation4D adapts this local-to-global strategy to driving video, where multiple cameras move as one rig but may capture images at different times. We separate camera-to-rig calibration from ego motion so that overlapping windows can constrain a shared trajectory, whose corrections are propagated to each camera at its own capture time (§ 5).

3D Object detection and tracking.

Camera-based 3D object detection localizes and classifies surrounding objects as oriented 3D cuboids from multi-camera images, while multi-object tracking links these detections into identity-consistent trajectories over time. Query-based detectors build on the set prediction of DETR [49], decoding a sparse set of object queries that are matched one-to-one to the ground truth without anchors or non-maximum suppression. They differ mainly in how queries gather multi-view image features. Dense-attention methods make image features 3D-aware for global cross-attention. PETR [50] adds position embeddings computed from camera frustums, while 3DPPE [51] replaces them with 3D point embeddings derived from predicted depth. Sparse methods instead sample features around 3D anchors or BEV pillar queries across views and frames [52, 53]. Temporal fusion further improves localization and velocity estimation. PETRv2 [15] aligns past-frame coordinates with ego motion, StreamPETR [54] propagates object queries through a memory queue for efficient streaming detection, Stream3DPPE [51] and RoPETR [55] extend StreamPETR with point embeddings and spatio-temporal rotary position embeddings, respectively. RayDN [56] improves training by extending query denoising [57] with hard negatives sampled along camera rays, which suppresses duplicate detections at wrong depths. These detectors aggregate views and frames inside the detection head, using ego poses to align past frames. GeoUP [3] instead fuses them in a VGGT-initialized backbone conditioned on camera calibration and ego poses (§ 2.1) and decodes boxes with a RayDN-style head, reporting state-of-the-art nuScenes results. Foundation4D likewise aggregates images from all cameras across a temporal window in its geometry-pretrained backbone and decodes PETR-style queries into 3D boxes and velocities through deformable attention [58]. Unlike GeoUP, which requires ground-truth ego poses, Foundation4D predicts them directly from multi-view images.

Tracking methods aim to obtain temporally consistent object trajectories and differ in how they associate detections over time. Following the tracking by detection paradigm, SORT [59] combines Kalman-filter motion prediction with Hungarian matching, AB3DMOT [60] extends this design to 3D boxes, and CenterPoint [61] matches detections using their predicted velocities. Query-based trackers such as TrackFormer [62] and MOTR [63] instead propagate track queries from frame to frame, an approach that MUTR3D [64] and PF-Track [65] bring to multi-camera 3D tracking. These online trackers rely on past observations, whereas offline methods optimize association over many frames at once. MPNTrack [66] learns it with message passing on a detection graph. SUSHI [67] scales learned association to long clips through a hierarchy of graphs over increasingly long time spans. Foundation4D takes inspiration from the offline literature, but adopts a simple yet effective association mechanism from online methods. It utilizes Kalman-filter consuming velocities predicted by our detection head, and uses a Hungarian association for frame-by-frame association, similar to SORT and AB3DMOT. However, it processes the clip offline, reconnecting fragmented tracks in a second pass and refining each trajectory with a Rauch–Tung–Striebel (RTS) smoother that also uses later observations.

Semantic occupancy.

Semantic occupancy prediction tackles scene geometry beyond visible surfaces, completing the surrounding geometry and assigning semantic labels to occupied regions. Existing approaches can be grouped according to their latent scene representations into plane-based, dense volumetric, and sparse methods, with hybrid methods connecting these families. Plane-based approaches encode scenes using two-dimensional feature grids. BEVFormer [14] learns BEV features through spatiotemporal attention, while TPVFormer [68] uses three orthogonal feature planes. SelfOcc [69] investigates BEV and TPV representations under self-supervised training, and POP-3D [70] uses TPV features for open-vocabulary occupancy prediction. Dense volumetric approaches maintain features on regular 3D voxel grids. MonoScene [71] and SurroundOcc [72] use 3D convolutions, OccFormer [73] uses a dual-path transformer, and OccMamba [74] applies state space models. OccGen [75] uses diffusion-based refinement, PanoOcc [76] aggregates multi-view and temporal features through voxel queries, and COTR [77] learns compact geometry-aware volumetric features. Allocating features to empty regions introduces memory and computational costs, motivating sparse alternatives. SparseOcc [78] operates on sparse voxel tensors. OPUS [79] predicts occupied locations and semantic classes directly from learned queries, SPSC [80] progressively prunes unoccupied queries during coarse-to-fine refinement, and GaussianFormer [81] represents scenes with sparse semantic Gaussian primitives. Hybrid designs combine these representation choices. VoxFormer [82] expands depth-guided sparse voxel queries into a dense volume, while CONet [11] refines coarse occupied voxels at higher resolution. Symphonies [83] connects contextual instance queries with image and voxel features, and CGFormer [84] combines voxel and tri-perspective features. Foundation4D adopts a BEV representation for semantic occupancy at 20 cm resolution and up to 100 m ahead of the ego vehicle. This compact BEV representation limits memory use, allowing us to support semantic occupancy alongside the other output tasks within a unified multi-camera, multi-timestamp model.

Gaussian scene reconstruction.

3D Gaussian Splatting (3DGS) [85] represents a scene with explicit anisotropic primitives for novel-view rendering. Feed-forward methods learn to predict these primitives across scenes, reducing the need to optimize each new scene separately. Representative approaches include pixelSplat [86] and MVSplat [87] for sparse-view reconstruction, and GS-LRM [88] for transformer-based Gaussian prediction. DepthSplat [89] connects multi-view depth estimation with Gaussian reconstruction, while NoPoSplat [90] predicts Gaussians from unposed images. Geometry foundation models provide a further route to feed-forward rendering: AnySplat [91] extends a VGGT-based geometry transformer with Gaussian prediction and distills camera and geometry priors from pretrained VGGT. TokenGS [92] instead uses learnable Gaussian tokens to decouple primitive prediction from the input pixel grid.

Driving scenes additionally require wide camera coverage and a representation of moving objects. Driving-Gaussian [93], Street Gaussians [94], and OmniRe [95] reconstruct static backgrounds and dynamic actors through per-sequence optimization. Feed-forward approaches such as STORM [96] and DrivingForward [97] address sparse temporal observations and surround-view inputs, while DGGT [98] jointly estimates cameras, Gaussians, and scene motion. Camera modeling is also important for driving capture: 3DGUT [99] supports nonlinear camera projection and rolling shutter. Following the layered scene representation of Instant NuRec [5], Foundation4D integrates Gaussian reconstruction with geometry, detection, and occupancy through shared features and joint training. At inference, our Gaussian branch consumes camera poses derived from the model’s predicted ego motion and actor tracks assembled from its detections, rather than requiring ground-truth poses or tracks (§ 3.9).

Foundation4D

Problem formulation

Given a video clip with TT temporal samples from CC cameras, Foundation4D estimates camera zz-depth maps and pointmaps, ego motion, 3D bounding boxes, semantic occupancy, and 3D Gaussians for rendering. We denote each input image by It,c∈RH×W×3\mathbf{I}_{t,c} \in\mathbb{R}^{H \times W \times3}, where tt and cc index time and camera, and HH and WW are the image dimensions. Camera c=1c=1 defines the reference timestamps τt=τt,1\tau_t = \tau_{t,1}. Each image retains its capture time τt,c\tau_{t,c}, which may differ from τt\tau_t for asynchronous inputs.

Overview

Foundation4D uses a shared image encoder and a factorized spatiotemporal aggregator to combine information across cameras and time (Figure 2). The geometry and Gaussian branches decode the resulting contextual features into geometric predictions and a renderable scene representation. Detection and occupancy share a feature pyramid that combines these features with the image encoder’s finer spatial detail.

Figure 2. Foundation4D architecture. Foundation4D unifies driving perception and reconstruction in a feed-forward model with a shared representation of multi-camera video. A common image encoder and factorized spatiotemporal attention efficiently combine observations across views and time. Task-specific decoders recover camera-to-rig calibration, ego motion, and metric depth; predict 3D objects, velocities, and high-resolution semantic occupancy; and reconstruct scene appearance with 3D Gaussians.

Shared image and spatiotemporal representation

Following VGGT [4], a shared DINOv2 ViT-L/14 encoder [100] extracts patch features independently from each image, with feature width d=1024d = 1024. We use two spatial resolutions to reduce aggregation cost while retaining image detail. A 1036×10361036 \times1036 image produces a 74×7474 \times74 encoder grid, which we bilinearly downsample to 37×3737 \times37 before aggregation. The aggregator thus processes one quarter of the spatial tokens, while the original encoder features remain available to the detection and occupancy feature pyramid.

VGGT alternates within-image self-attention with global self-attention across all images. Global attention scales quadratically with the total number of tokens, making it expensive for multi-camera videos. We instead factor this operation across cameras and time. Each layer applies within-image self-attention followed by self-attention across cameras at a fixed temporal index or across time for a fixed camera. These camera and temporal groups alternate across layers, combining spatial and temporal context without attending over the full window at once (Figure 2).

Each image patch sequence also carries a rig token for relative camera calibration, an ego token for rig motion, and four register tokens for auxiliary context. All tokens participate in attention. After 16 aggregation layers, we concatenate the outputs of the final within-image and temporal attention blocks along the channel dimension. This gives a contextual token sequence Ft,cF_{t,c} of width 2d2d, whose patch features are denoted by Ft,cpatchF^{\mathrm{patch}}_{t,c}.

Geometry decoders

The geometry branch predicts camera poses and depth. We also train two auxiliary decoders: an image-semantic decoder that provides masks for video-level inference, and a pointmap decoder that supplies additional geometric supervision during training. We express camera-to-world poses Tt,c∈SE(3)T_{t,c} \in\mathrm{SE}(3) and pointmaps in the window reference frame, anchored to camera c=1c = 1 at the first reference timestamp τ1\tau_1.

Pose prediction.

We factor camera poses into a fixed rig transform TcrigT_c^{\mathrm{rig}} for each camera and a time-varying reference-camera trajectory TegoT^{\mathrm{ego}}. The rig head reads rig-token features from all cameras in the first temporal group, while the ego head reads ego-token features from the reference camera across time. Separate transformer heads process these sequences, following VGGT’s camera-head architecture [4], and regress translation and quaternion parameters through an MLP. The heads predict world-to-camera encodings, which are converted to camera-to-world transforms before pose composition.

Each rig transform maps a camera into the reference-camera frame and is reused throughout the window. We interpolate the ego trajectory at each image’s capture time, using linear interpolation for translation and shortest-arc spherical interpolation for rotation. Denoting this trajectory by Tego(τ)T^{\mathrm{ego}}(\tau), we construct the camera pose as

Tt,c=Tego(τt,c)Tcrig.(1)T_{t,c} = T^{\mathrm{ego}}(\tau_{t,c}) T_c^{\mathrm{rig}}. \tag*{(1)}

For synchronized images, the trajectory is evaluated at the shared timestamp τt\tau_t. Rig supervision uses relative poses from the first temporal group: these targets represent static calibration for synchronized inputs, but can include ego motion between exposures for asynchronous inputs. The composition above treats the predicted rig factor as fixed, and interpolating the ego trajectory does not remove this mismatch in the supervision target. Each image geometry remains associated with its own acquisition time; interpolation does not synchronize the images or compensate for independently moving objects. At inference, capture times outside the ego trajectory’s sampled interval extend the nearest segment’s motion, while a single trajectory sample yields a constant pose.

Dense prediction

Independently parameterized DPT decoders [101] transform the contextual patch features Ft,cpatchF_{t,c}^{\mathrm{patch}} into camera zz-depth Dt,cD_{t,c}, an auxiliary pointmap Pt,cauxP_{t,c}^{\mathrm{aux}} in the window reference frame, and auxiliary semantic logits St,cS_{t,c}. Each decoder reassembles tokens into spatial feature maps and fuses multiple scales through convolutional refinement, producing a 518×518518 \times518 output grid. Positive confidence maps accompany the depth and auxiliary pointmap predictions. The auxiliary semantic decoder outputs 21 class logits per pixel through a final 1×11 \times1 convolution; the highest-scoring class gives the predicted label.

The final pointmap is constructed from predicted depth and camera poses rather than taken from the auxiliary decoder. After restoring metric units, we unproject depth through the calibrated camera model, accounting for lens distortion and the output-grid resolution. For one image, let d(u)d(u) be the depth at pixel uu, r(u)∈S2\mathbf{r}(u) \in\mathbb{S}^{2} its unit camera ray, and (R,t)(\mathbf{R},\mathbf{t}) the camera-to-world rotation and translation. For valid pixels with [r(u)]z>0[\mathbf{r}(u)]_{z} > 0, the corresponding point in the window reference frame is

p(u)=Rd(u)[r(u)]zr(u)+t.(2)\mathbf{p}(u) = \mathbf{R}\frac{d(u)}{[\mathbf{r}(u)]_{z}}\mathbf{r}(u) + \mathbf{t}. \tag*{(2)}

The division by [r(u)]z[\mathbf{r}(u)]_{z} converts camera zz-depth to distance along the ray.

Supervision

The geometry objective combines dense geometry, rig and ego poses, temporal relative motion, and auxiliary image semantics:

Lgeo=λDLD+λPLP+λrigLrig+λego(Lego+λrelLrel)+msemλsemLsem,(3)\mathcal{L}_{\mathrm{geo}} = \lambda_{D}\mathcal{L}_{D} + \lambda_{P}\mathcal{L}_{P} + \lambda_{\mathrm{rig}}\mathcal{L}_{\mathrm{rig}} + \lambda_{\mathrm{ego}}\left(\mathcal{L}_{\mathrm{ego}} + \lambda_{\mathrm{rel}}\mathcal{L}_{\mathrm{rel}}\right) + m_{\mathrm{sem}}\lambda_{\mathrm{sem}}\mathcal{L}_{\mathrm{sem}}, \tag*{(3)}

We use (λD,λP,λrig,λego,λrel,λsem)=(1,1,10,10,1,1)(\lambda_{D}, \lambda_{P}, \lambda_{\mathrm{rig}}, \lambda_{\mathrm{ego}}, \lambda_{\mathrm{rel}}, \lambda_{\mathrm{sem}}) = (1, 1, 10, 10, 1, 1), with msemm_{\mathrm{sem}} enabling semantic supervision only when annotations are available. Depth, point coordinates, and translations are divided by a fixed reference length of 40 m before supervision. In the losses below, hats denote predictions and unmarked quantities denote targets.

Depth and auxiliary pointmap supervision combine direct regression with a confidence-weighted term. Over valid pixels i∈Mi \in\mathcal{M}, the residuals are eiD=∣d^i−di∣e_{i}^{D} = \lvert\hat{d}_{i} - d_{i}\rvert and eiP=∥p^iaux−pi∥2e_{i}^{P} = \lVert\hat{\mathbf{p}}_{i}^{\mathrm{aux}} - \mathbf{p}_{i}\rVert_{2}. For X∈{D,P}X \in\{D, P\} and predicted confidence qiXq_{i}^{X}, we use

LX=R0.98({eiX}i∈M)+R0.98({qiXeiX−βlog⁡qiX}i∈M),β=0.2.(4)\mathcal{L}_{X} = \mathcal{R}_{0.98}\left(\{e_{i}^{X}\}_{i\in\mathcal{M}}\right) + \mathcal{R}_{0.98}\left(\{q_{i}^{X}e_{i}^{X} - \beta\log q_{i}^{X}\}_{i\in\mathcal{M}}\right), \qquad\beta= 0.2. \tag*{(4)}

Here R0.98\mathcal{R}_{0.98} applies robust averaging independently to each term. The logarithmic penalty regularizes confidence, while the unweighted term retains direct supervision.

Rig and ego losses supervise the heads’ native world-to-camera encodings. We write the translation as tw2c\mathbf{t}^{\mathrm{w2c}} to distinguish it from the camera-to-world translation used in unprojection, and denote the quaternion by q\mathbf{q}. The error for one pose is

ℓpose=13∥t^w2c−tw2c∥1+14min⁡ϵ∈{−1,1}∥q^−ϵq∥1.(5)\ell_{\mathrm{pose}} = \frac{1}{3}\lVert\hat{\mathbf{t}}^{\mathrm{w2c}} - \mathbf{t}^{\mathrm{w2c}}\rVert_{1} + \frac{1}{4}\min_{\epsilon\in\{-1,1\}}\lVert\hat{\mathbf{q}} - \epsilon\mathbf{q}\rVert_{1}. \tag*{(5)}

The sign minimization accounts for the two quaternion representations of the same rotation. Lrig\mathcal{L}_{\mathrm{rig}} and Lego\mathcal{L}_{\mathrm{ego}} average this error over their respective valid observations.

The relative-pose term additionally constrains motion between nearby temporal samples. For each unique pair i<ji < j with j−i<5j-i < 5, we form ΔT^ji=(T^jego)−1T^iego\Delta\hat{\mathbf{T}}_{ji} = (\hat{\mathbf{T}}_{j}^{\mathrm{ego}})^{-1}\hat{\mathbf{T}}_{i}^{\mathrm{ego}}, which maps camera coordinates from time ii to time jj; targets are constructed identically from ground-truth poses. For each pair, we sum the mean absolute errors over the three translation components and four quaternion components, then average across pairs to obtain Lrel\mathcal{L}_{\mathrm{rel}}. Relative quaternions are unit quaternions with nonnegative scalar components. The loss is zero when no pair is available.

For annotated images, the auxiliary Lsem\mathcal{L}_{\mathrm{sem}} sums cross-entropy, soft Dice, and Lovász–Softmax losses with equal weights, excluding unknown pixels from all three terms. Cross-entropy uses uniform class weights and label smoothing of 0.05. Soft Dice averages overlap over the 20 known classes, adding 1 to both its numerator and denominator. Lovász–Softmax is evaluated over classes present in each image and then averaged across images.

For robust dense regression, R0.98\mathcal{R}_{0.98} caps loss values at 100 and excludes values at or above the 98th percentile when sufficient valid observations are available; small sets use the untrimmed mean. Pose supervision requires more than 100 valid pixels per observation when dense geometry labels are available.

3D object detection

The object branch predicts a sparse set of classified 3D cuboids at each timestamp in the vehicle rig coordinate system, covering a region of interest of [−108,150]×[−108,108]×[−3,5][-108, 150] \times[-108, 108] \times[-3, 5] m. Following PETR [50], we represent object hypotheses with queries anchored at learned 3D reference points. Each query predicts class scores and a cuboid described by its 3D center, dimensions, yaw, and planar velocity. The shared backbone provides multi-camera temporal context, while the object decoder produces detections for each timestamp. An offline tracker subsequently assembles these detections into trajectories over the complete clip.

The internal data detector uses 12 classes: automobile, heavy truck, bus, train or tram car, other vehicle, person, rider, trailer, protruding object, trolley bus, stroller, and animal. Public-benchmark classifiers are adapted to their target label sets.

Feature pyramid.

A feature pyramid shared with occupancy fuses the high-resolution image features with the aggregator’s contextual patch features (§ 3.3). Lateral projections align their channels, after which the contextual features are upsampled and added to the image features through a top-down pathway.

Decoder architecture.

The decoder follows the DETR3D architecture [102], alternating query self-attention, multi-scale deformable cross-attention, and feed-forward layers. Query content is initialized to zero, while positional embeddings are computed from the learned 3D reference points. In each cross-attention layer, query features predict 3D offsets around their reference points. The resulting sampling locations are projected into the camera views using the supplied intrinsics, lens model, and extrinsics. We sample the feature pyramid at these locations and combine the samples with query-dependent attention weights over cameras, scales, and sampling points. Classification and regression heads read the refined queries at each decoder layer. The reference configuration uses 900 queries and six decoder layers of width 256.

Supervision.

We use one-to-one Hungarian matching between predicted queries and annotated cuboids, with a cost combining focal classification and cuboid regression. Following PETR [50], classification uses sigmoid focal loss; matched queries receive a weighted L1L_1 regression loss. Unmatched queries are supervised as background. The regression representation contains metric center coordinates, logarithmic dimensions, sine and cosine of yaw angle, and planar velocity. The center predictions are parameterized as offsets from the normalized reference points and converted back to metric coordinates for supervision and readout. Planar velocities are later used to estimate object motion and obtain temporally consistent tracks, as explained in § 5.3.

The detection objective combines supervision at every decoder layer with query denoising and an auxiliary image-space detection loss:

Ldet=∑ℓ=16(λclsLfocal(ℓ)+λboxLbox(ℓ)+λDNLDN(ℓ))+L2D.(6)\mathcal{L}_{\mathrm{det}} = \sum_{\ell=1}^{6} \left(\lambda_{\mathrm{cls}}\mathcal{L}_{\mathrm{focal}}^{(\ell)} + \lambda_{\mathrm{box}}\mathcal{L}_{\mathrm{box}}^{(\ell)} + \lambda_{\mathrm{DN}}\mathcal{L}_{\mathrm{DN}}^{(\ell)}\right) + \mathcal{L}_{2\mathrm{D}}. \tag*{(6)}

Here Lbox\mathcal{L}_{\mathrm{box}} includes coordinate-wise regression weights. We use (λcls,λbox,λDN)=(2,0.25,1)(\lambda_{\mathrm{cls}}, \lambda_{\mathrm{box}}, \lambda_{\mathrm{DN}}) = (2, 0.25, 1); the denoising term contains the same weighted classification and regression losses. Focal loss uses α=0.25\alpha= 0.25 and γ=2\gamma= 2. Intermediate decoder outputs receive the same forms of supervision as the final output.

Query denoising follows the training principle of DN-DETR [57]. During training, we append groups of queries whose reference points are obtained by perturbing ground-truth cuboid centers in proportion to object size. Their target assignments are known, bypassing Hungarian matching for these queries; strongly perturbed queries receive background classification targets. Attention masks prevent ordinary detection queries from attending to denoising queries and isolate the denoising groups from one another.

An auxiliary YOLOX-style head [103] predicts per-camera 2D boxes, classes, objectness, and projected cuboid centers from the pyramid features. Projected cuboid annotations provide its targets. Its loss is

L2D=Lcls2D+Lobj2D+5LIoU2D+LL122D+Lcenter2D.(7)\mathcal{L}_{2\mathrm{D}} = \mathcal{L}_{\mathrm{cls}}^{2\mathrm{D}} + \mathcal{L}_{\mathrm{obj}}^{2\mathrm{D}} + 5\mathcal{L}_{\mathrm{IoU}}^{2\mathrm{D}} + \mathcal{L}_{L_1}^{2\mathrm{2D}} + \mathcal{L}_{\mathrm{center}}^{2\mathrm{D}}. \tag*{(7)}

where classification and objectness use binary cross-entropy, box localization combines squared-IoU and L1L_1 losses, and projected centers use an L1L_1 loss. Query denoising and the 2D head are used only during training.

Semantic occupancy

The occupancy branch predicts binary occupancy and semantic labels on a metric voxel grid of size Vx×Vy×VzV_x \times V_y \times V_z in the vehicle rig coordinate system, covering a region of interest of [−20,100]×[−30,30]×[−3,5] m[-20, 100] \times[-30, 30] \times[-3, 5]\,\mathrm{m}, at a voxel resolution of 0.2 m0.2\,\mathrm{m}. To limit memory usage, intermediate features remain in BEV, with the voxel height dimension introduced at the outputs.

BEV projection module.

Our BEV projector is a Deformable DETR [58] inspired module that aggregates multi-scale image features produced by the shared feature pyramid neck into a spatially grounded BEV map for each timestamp (Figure 3). For each BEV cell, we place four reference points along its vertical pillar at uniformly spaced heights within the target height range. We encode each point’s normalized 3D coordinates and use separate learned layers to initialize its query feature and positional embedding. Transformer layers refine these queries through self-attention and deformable cross-attention, while positional embeddings are used in self-attention and in predicting the deformable cross-attention weights. Cross-attention projects 3D sampling points into the camera views using camera intrinsics and static rig-to-camera extrinsics, which we assume are known and part of the vehicle rig. This projection requires no ego-motion estimates, since the reference points and BEV grid are defined in the rig coordinate system of each timestamp. A learned merging layer combines the refined pillar features into the BEV map FtBEV\mathbf{F}^{\mathrm{BEV}}_t.

Semantic occupancy BEV features showing (a) input camera frames, (b) BEV features, and (c) BEV of Foundation4D predictions

Figure 3. Foundation4D semantic occupancy BEV features. PCA of the upsampled features reveals a separation between occupied and empty regions. All panels show the same sample; PCA colors encode feature variation, while prediction colors denote semantic classes.

BEV upsampling and output heads.

We use a fully convolutional decoder and progressively upsample FtBEV\mathbf{F}^{\mathrm{BEV}}_t through 2D transposed convolutions. Let KoccK_{\mathrm{occ}} denote the number of occupancy semantic classes. Separate transposed-convolution heads output VzV_z binary channels and VzKoccV_zK_{\mathrm{occ}} semantic channels. Channel-to-height reshaping [104] converts these outputs into binary logits Otbin∈RVx×Vy×Vz\mathbf{O}^{\mathrm{bin}}_t \in\mathbb{R}^{V_x \times V_y \times V_z} and semantic logits Otsem∈RVx×Vy×Vz×Kocc\mathbf{O}^{\mathrm{sem}}_t \in\mathbb{R}^{V_x \times V_y \times V_z \times K_{\mathrm{occ}}}. At inference time, the final semantic occupancy prediction is obtained by thresholding the sigmoid probabilities of OtbinO_{t}^{\mathrm{bin}} and assigning each occupied voxel the highest-scoring semantic class from OtsemO_{t}^{\mathrm{sem}}. The remaining voxels are labeled as empty.

Supervision.

We supervise occupancy using LiDAR-derived geometry and image-derived semantic labels, with an auxiliary BEV segmentation head attached to the upsampled BEV features. The training objective is

Locc=λBCELBCE+λLovLLovaszbin+λCELCEocc+λDiceLDiceocc+λBEVLBEV.(8)\mathcal{L}_{\mathrm{occ}} = \lambda_{\mathrm{BCE}}\mathcal{L}_{\mathrm{BCE}} + \lambda_{\mathrm{Lov}}\mathcal{L}_{\mathrm{Lovasz}}^{\mathrm{bin}} + \lambda_{\mathrm{CE}}\mathcal{L}_{\mathrm{CE}}^{\mathrm{occ}} + \lambda_{\mathrm{Dice}}\mathcal{L}_{\mathrm{Dice}}^{\mathrm{occ}} + \lambda_{\mathrm{BEV}}\mathcal{L}_{\mathrm{BEV}}. \tag*{(8)}

Here, LBCE\mathcal{L}_{\mathrm{BCE}} and LLovaszbin\mathcal{L}_{\mathrm{Lovasz}}^{\mathrm{bin}} are binary cross-entropy and Lovász hinge losses, while LCEocc\mathcal{L}_{\mathrm{CE}}^{\mathrm{occ}} and LDiceocc\mathcal{L}_{\mathrm{Dice}}^{\mathrm{occ}} are cross-entropy and soft Dice losses for voxel semantics. The auxiliary BEV loss is LBEV=λCELCEBEV+λDiceLDiceBEV\mathcal{L}_{\mathrm{BEV}} = \lambda_{\mathrm{CE}}\mathcal{L}_{\mathrm{CE}}^{\mathrm{BEV}} + \lambda_{\mathrm{Dice}}\mathcal{L}_{\mathrm{Dice}}^{\mathrm{BEV}}, combining cross-entropy and soft Dice supervision with the same weights used for voxel semantics. BEV targets are obtained by selecting the topmost semantic voxel in each vertical column. We use (λBCE,λLov,λCE,λDice,λBEV)=(1,0.1,1,1,0.5)(\lambda_{\mathrm{BCE}}, \lambda_{\mathrm{Lov}}, \lambda_{\mathrm{CE}}, \lambda_{\mathrm{Dice}}, \lambda_{\mathrm{BEV}}) = (1, 0.1, 1, 1, 0.5) in our experiments. Loss terms include sums over the supervised projection-decoder layers.

Gaussian reconstruction

Following Instant NuRec [5], we represent a driving scene with static Gaussians, actor-associated Gaussians, and a sky cubemap. The reconstruction branch uses shared contextual features alongside provided camera calibrations. During training, we utilize ground-truth camera-to-world poses and actor cuboid tracks, whereas during inference, we rely on predicted poses and trajectories. The Gaussian branch does not predict actor motion itself.

For each source pixel i=(t,c,u)i = (t, c, u) on the decoder output grid, we predict a Gaussian with center μi\mu_i, RGB color cic_i, scales sis_i, unit quaternion qiq_i, and opacity αi\alpha_i, as well as a unit surface normal for auxiliary supervision. Its covariance is Σi=Ridiag⁡(si2)RiT\Sigma_i = R_i \operatorname{diag}(s_i^2)R_i^\mathsf{T}, where Ri=Rot⁡(qi)R_i = \operatorname{Rot}(q_i). Centers, orientations, and normals are expressed in the clip world frame defined by the predicted trajectory.

Gaussian decoding.

DPT decoders [101] predict depth, color residuals, normals, and Gaussian attributes from contextual patch features. The Gaussian depth decoder is initialized from the geometry depth head but has separate parameters trained with reconstruction losses; the geometry head retains its own supervision.

We obtain each unit camera ray ri\mathbf{r}_i by calibrated inverse projection, accounting for lens distortion and image resizing. Let EcE_c denote the known camera-to-rig transform. At inference, the source camera pose is T^t,c=T^rig(τt,c)Ec\hat{T}_{t,c} = \hat{T}_{\mathrm{rig}}(\tau_{t,c})E_c, where T^rig\hat{T}_{\mathrm{rig}} is the assembled predicted rig-to-world trajectory. Given predicted metric camera zz-depth did_i and the rotation and translation (R^t,c,t^t,c)(\hat{R}_{t,c}, \hat{t}_{t,c}) of this camera pose, the center of each valid Gaussian is

μi=R^t,cdi[ri]zri+t^t,c.(9)\mu_i = \widehat{\mathbf{R}}_{t,c}\frac{d_i}{[r_i]_z}r_i+\widehat{\mathbf{t}}_{t,c}. \tag*{(9)}

The factor di/[ri]zd_i/[r_i]_z converts camera depth to ray distance. Gaussian color adds a bounded learned residual to the resampled source RGB. To suppress Gaussians in sky, ego-vehicle, and unknown regions, we multiply their opacities by a mask from the geometry branch’s semantic predictions. The mask uses soft probabilities during training with gradients stopped and discrete labels at inference.

Scene rendering.

Supplied or predicted cuboid tracks associate source Gaussians with actors. To render a target timestamp, we interpolate each actor’s predicted trajectory and rigidly transform its Gaussians from their source capture times, while static Gaussians remain fixed. Differentiable splatting produces accumulated RGB cG(u)c_{\mathrm{G}}(u), accumulated opacity A(u)A(u), and ray distance at target pixel uu. A scene decoder predicts a world-space sky cubemap from contextual features, source RGB, and calibrated rays; its color csky(u)c_{\mathrm{sky}}(u) along the target ray fills the residual transmittance. A separate decoder uses contextual features to predict an affine RGB transform for each camera, with Mc∈R3×3M_c \in\mathbb{R}^{3\times3} and bc∈R3b_c \in\mathbb{R}^3, to account for appearance differences. Applying the target camera’s transform and clipping to [0,1][0,1] gives the pixel colors of the rendered image II:

c(u)=clip⁡[0,1][Mc(cG(u)+(1−A(u))csky(u))+bc].(10)c(u)=\operatorname{clip}_{[0,1]}\left[M_c\left(c_{\mathrm{G}}(u)+(1-A(u))c_{\mathrm{sky}}(u)\right)+b_c\right]. \tag*{(10)}

Supervision.

Pose and cuboid-track annotations provide conditioning during reconstruction training, while RGB, depth, and available semantic annotations supervise the corresponding losses. This training setup is distinct from the predicted-input inference configuration described above: annotated poses and actor tracks are not required when reconstructing a clip at inference. We supervise rendering at two timestamps excluded from the context: one inside the context interval and one within 0.5 s after its last sample, with camera observations aligned to the target times. The rendering objective is

Lrender=LRGB+0.2LLPIPS+0.1Linv−dist+2Lopacity,(11)\mathcal{L}_{\mathrm{render}}=\mathcal{L}_{\mathrm{RGB}}+0.2\mathcal{L}_{\mathrm{LPIPS}}+0.1\mathcal{L}_{\mathrm{inv-dist}}+2\mathcal{L}_{\mathrm{opacity}}, \tag*{(11)}

where the RGB, inverse-ray-distance, and accumulated-opacity terms use mean squared error; the opacity target is the non-sky mask. Pixel losses average over valid pixels, while LPIPS averages over valid target images after masked pixels are replaced by target colors. The full task objective adds source RGB consistency, normal supervision, sky consistency, and affine regularization:

Lgs=Lrender+0.01Lsrc−RGB+0.2Lnormal+0.01Lsky+0.01Laffine.(12)\begin{aligned} \mathcal{L}_{\mathrm{gs}}&=\mathcal{L}_{\mathrm{render}}+0.01\mathcal{L}_{\mathrm{src-RGB}}+0.2\mathcal{L}_{\mathrm{normal}}\\ &\quad+0.01\mathcal{L}_{\mathrm{sky}}+0.01\mathcal{L}_{\mathrm{affine}}. \tag*{(12)} \end{aligned}

Source RGB consistency uses an L1L_1 loss, and normal supervision uses cosine distance to normals derived from depth annotations. The sky loss is L1L_1 against observed sky colors, using a smoothed prediction as the target in unobserved regions. Affine transforms are regularized by their mean squared deviation from identity. Supervision masks come from semantic annotations on NRE and PlanNet and detached semantic predictions from the geometry branch on MLMCF; the latter are not semantic training targets. These supervision masks are distinct from the predicted mask applied to Gaussian opacity.

Training the unified model

Training data

We train on three internal driving datasets: NRE, PlanNet, and MLMCF, comprising approximately 68k, 76k, and 60k clips, respectively, with some overlap between collections. Table 1 summarizes their scene composition.

DatasetClipsUrbanResidentialHighwayRuralOther
NRE67,97016.5%9.1%64.5%7.6%2.3%
PlanNet75,92142.5%17.3%20.8%16.5%2.9%
MLMCF59,99936.5%10.9%42.9%6.9%2.8%

Table 1. Scene composition of the internal training splits. NRE is dominated by highway driving, while PlanNet emphasizes urban and residential scenes. MLMCF provides a more balanced mix of urban and highway driving, complementing the other two collections. Counts and percentages refer to unique clips within each dataset.

Clips average approximately 12 s in length, amounting to roughly 695 hours of driving in total. The sensor rigs use six polynomial ff-theta cameras [105] with different fields of view, covering both wide-angle and narrower views of the scene. Images are captured at 30 FPS, and a 128-beam LiDAR operates at 10 Hz. Training windows subsample these recordings to cover different temporal spans.

The datasets provide depth, camera motion, cuboid tracks, and, where available, semantic annotations. LiDAR measurements are used to generate depth through the annotation pipeline described in § 6. Semantic occupancy labels are obtained through an internal auto-labeling pipeline providing annotations for 10 semantic classes. Detection annotations cover 12 classes, with LiDAR-based cuboid autolabels in NRE and manually annotated cuboids in PlanNet and MLMCF.

Training strategy

We train the six-camera unified model in three stages, increasing temporal context before introducing Gaussian reconstruction. The first stage jointly trains the image encoder, spatiotemporal aggregator, and geometry, detection, and occupancy heads on short windows, reducing the cost of each update. The second stage extends the windows to provide more temporal context for pose and motion estimation. We freeze the image encoder at this point and continue training the aggregator and the three task heads.

The third stage adds Gaussian reconstruction to the trained perception model. We first warm up the Gaussian heads with the existing model frozen, then jointly optimize the aggregator and all four task heads while keeping the image encoder frozen. This schedule defers the cost of Gaussian decoding and rendering until the perception model has been trained.

At each stage, we optimize a weighted sum of the active task losses defined in § 3:

L=λgeoLgeo+λdetLdet+λoccLocc+λgsLgs.(13)\mathcal{L} = \lambda_{\mathrm{geo}}\mathcal{L}_{\mathrm{geo}} + \lambda_{\mathrm{det}}\mathcal{L}_{\mathrm{det}} + \lambda_{\mathrm{occ}}\mathcal{L}_{\mathrm{occ}} + \lambda_{\mathrm{gs}}\mathcal{L}_{\mathrm{gs}}. \tag*{(13)}

For joint fine-tuning with Gaussian reconstruction, we use (λgeo,λdet,λocc,λgs)=(1,0.1,0.2,4)(\lambda_{\mathrm{geo}}, \lambda_{\mathrm{det}}, \lambda_{\mathrm{occ}}, \lambda_{\mathrm{gs}}) = (1, 0.1, 0.2, 4). A loss contributes only when its task is active and the required supervision is available; individual losses also exclude invalid pixels, objects, or voxels. All three datasets contribute to geometry, detection, and Gaussian reconstruction, whereas occupancy uses only NRE and PlanNet. Only NRE is used to supervise the auxiliary image-semantic head. The single-task and pairwise recipes used to study task transfer are described separately in § 7.11.

Implementation details

All stages use six cameras with images resized to 1036×10361036 \times1036. During temporal refinement and final joint fine-tuning, we sample NRE, PlanNet, and MLMCF in a 1 : 2 : 1 ratio. The first stage uses four timestamps per camera and random frame strides up to eight, and runs for approximately 170k updates. Temporal refinement increases the window to eight timestamps and the maximum stride to 16 for a further 45k updates. Gaussian training retains this window configuration, with head warm-up followed by 10k joint updates.

The first two stages use AdamW with a main learning rate of 3×10−43 \times10^{-4} and weight decay of 0.01. During the first stage, the image encoder uses one tenth of this learning rate. Both stages use 500 updates of linear learning-rate warm-up. The first then keeps the learning rate constant, while the second holds it until update 10k before applying cosine decay. The Gaussian heads are warmed up with a learning rate of 10−410^{-4}. Final joint fine-tuning uses a main learning rate of 10−510^{-5} and weight decay of 0.01, with 500 warm-up updates followed by cosine learning-rate decay to zero.

We train with BF16 mixed precision, retaining FP32 computation in the geometry and Gaussian decoders and in numerically sensitive detection and occupancy operations. On NVIDIA H100 GPUs, FlashAttention-3 (FA3) [106] implements attention in the shared spatiotemporal backbone. Custom CUDA kernels apply two-dimensional rotary position embeddings (RoPE) [107] to queries and keys directly in the packed query–key–value layout, avoiding separate tensor repacking before attention. On 64 H100 GPUs, representative geometry, detection, and occupancy training windows took approximately 3.8 seconds per update with four timestamps per camera and 8.3 seconds with eight.

Multi-camera video inference system

To reconstruct videos beyond the model’s temporal context, we process overlapping windows and assemble their predicted camera trajectories and dense geometry in a common coordinate frame. The overlap provides repeated observations that connect the windows while keeping the network’s context length fixed. We consider two alternative assembly methods: sequential window stitching and global pose graph optimization (PGO). Both use the predicted geometry and known camera intrinsics without updating network parameters. This formulation is related to overlapping-window and submap reconstruction in VGGT-Long [46], VGGT-SLAM [47], and VGGT-SLAM 2.0 [48]. Figure 17 illustrates the assembled geometry alongside occupancy and detections. The resulting camera trajectories also place 3D object detections in the common reconstruction frame, where an offline multi-object tracker associates them across time to recover object trajectories over the entire clip.

Scene-level geometry, occupancy, and tracks

Figure 17. Scene-level geometry, occupancy, and tracks. Pointmaps (left) and occupancy (right) span a 20-second clip, with car geometry and occupancy accumulated in tracked object frames. Solid cuboids mark the final scene at τ=19.97\tau= 19.97 s; dashed cuboids and translucent car and pedestrian occupancy show τ=1.50\tau= 1.50 s. The ego route uses optimized camera poses and fixed camera-to-rig calibration.

Sequential window stitching

Stitching aligns each new window to the preceding, already aligned window with a single Sim(3) transform. We pool depth-derived pointmap correspondences across cameras and overlapping frames: each pair xa,yax_a, y_a describes the same camera, frame, and pixel in the new and preceding windows, respectively. These correspondences can include foreground objects because they refer to the same captured observation. Over the set Ok\mathcal{O}_k of correspondences for window kk, we estimate scale sks_k, rotation Rk\mathbf{R}_k, and translation tk\mathbf{t}_k by iteratively reweighted least squares:

Ek(sk,Rk,tk)=∑a∈OkmawaρH(∥skRkxa+tk−ya∥2),(14)\mathcal{E}_k(s_k,\mathbf{R}_k,\mathbf{t}_k)=\sum_{a\in\mathcal{O}_k}m_aw_a\rho_{\mathrm{H}}\left(\left\|s_k\mathbf{R}_kx_a+\mathbf{t}_k-y_a\right\|_2\right), \tag*{(14)}

Here, ρH\rho_{\mathrm{H}} is the Huber penalty, mam_a excludes correspondences classified as sky by the model’s 2D semantic predictions in either window, and waw_a is the geometric mean of the two depth confidences. If masks or confidences are unavailable, the corresponding masks or weights are set to one.

We apply the estimated similarity transform to all camera poses in the window and scale the corresponding depth maps by sks_k, placing both in the common reconstruction frame defined by the first window. Previously assembled predictions are retained in overlapping frames, and only new temporal samples are appended. Sequential alignment can accumulate rotation, translation, and scale errors: later windows do not revise earlier poses, and a single transform cannot correct inconsistencies within a window’s trajectory. This motivates jointly reconciling the pose estimates across all windows.

Global pose graph optimization

PGO estimates a single reference-camera trajectory for the entire video from the raw window predictions. It assumes a common metric scale across windows, supported by the fixed-scale supervision described in § 3.4. After restoring metric units, we optimize poses in SE(3)\mathrm{SE}(3) without estimating window scales or rescaling depth. Each unique reference-camera sample is a graph node, shared by all windows containing that sample. Within window kk, the predicted camera-to-window poses T^i(k)\hat{\mathbf{T}}^{(k)}_i define relative measurements Zij(k)=(T^i(k))−1T^j(k)=(Rij(k),tij(k))\mathbf{Z}^{(k)}_{ij} = (\hat{\mathbf{T}}^{(k)}_i)^{-1}\hat{\mathbf{T}}^{(k)}_j = (\mathbf{R}^{(k)}_{ij}, \mathbf{t}^{(k)}_{ij}), which map coordinates from sample jj to sample ii. These measurements remove the window’s reference rotation and translation while retaining its predicted scale. Measurements of the same pair from different windows remain separate factors, allowing the windows to constrain a shared trajectory. The graph uses predicted relative poses and timestamps, without external pose factors or loop-closure constraints.

We solve for the trajectory in two stages, first optimizing rotations and then translations. Let Ti=(Ri,ti)\mathbf{T}_i = (\mathbf{R}_i, \mathbf{t}_i) map reference-camera coordinates at node ii to the video coordinate frame, with the first pose fixed to the identity. Over the multiset E\mathcal{E} of window factors, the objectives are

{Ri⋆}=arg⁡min⁡Ri∈SO(3), R1=I31Ω∑(k,i,j)∈Eωij(k)∥RiTRj−Rij(k)∥F2,{ti⋆}=arg⁡min⁡t1=01Ω∑(k,i,j)∈Eωij(k)∥(Ri⋆)T(tj−ti)−tij(k)∥22+λmotionLmotion,(15)\begin{aligned} \{\mathbf{R}_i^\star\} &= \underset{\mathbf{R}_i\in\mathrm{SO}(3),\,\mathbf{R}_1=\mathbf{I}_3}{\arg\min}\frac{1}{\Omega}\sum_{(k,i,j)\in\mathcal{E}}\omega^{(k)}_{ij}\left\lVert\mathbf{R}_i^{\mathsf{T}}\mathbf{R}_j-\mathbf{R}^{(k)}_{ij}\right\rVert_{\mathrm{F}}^2,\\ \{\mathbf{t}_i^\star\} &= \underset{\mathbf{t}_1=\mathbf{0}}{\arg\min}\frac{1}{\Omega}\sum_{(k,i,j)\in\mathcal{E}}\omega^{(k)}_{ij}\left\lVert(\mathbf{R}_i^\star)^{\mathsf{T}}(\mathbf{t}_j-\mathbf{t}_i)-\mathbf{t}^{(k)}_{ij}\right\rVert_2^2+\lambda_{\mathrm{motion}}\mathcal{L}_{\mathrm{motion}}, \tag*{(15)} \end{aligned}

where ωij(k)\omega^{(k)}_{ij} weights each factor and Ω=∑(k,i,j)∈Eωij(k)\Omega= \sum_{(k,i,j)\in\mathcal{E}}\omega^{(k)}_{ij}. With rotations fixed, translation estimation is a linear least-squares problem. The optional motion prior Lmotion\mathcal{L}_{\mathrm{motion}} penalizes changes between incoming and outgoing translation velocities at each interior node, using the actual reference-camera timestamp intervals. Each velocity difference is multiplied by the median sampling interval before taking its squared Euclidean norm, and the penalty is averaged over interior nodes.

The optimized trajectory Ti⋆=(Ri⋆,ti⋆)\mathbf{T}_i^\star= (\mathbf{R}_i^\star, \mathbf{t}_i^\star) then provides a correction for every camera at its acquisition time. For each sample, we retain depth and semantic predictions from the window in which it lies closest to the temporal center. For an image captured at τt,c\tau_{t,c} in selected window kk, we interpolate the source and optimized reference-camera trajectories separately and apply

T^t,cclip=T⋆(τt,c)[T^(k)(τt,c)]−1T^t,c(k).(16)\hat{\mathbf{T}}^{\mathrm{clip}}_{t,c}=\mathbf{T}^\star(\tau_{t,c})\left[\hat{\mathbf{T}}^{(k)}(\tau_{t,c})\right]^{-1}\hat{\mathbf{T}}^{(k)}_{t,c}. \tag*{(16)}

This preserves each camera’s predicted relation to the source reference trajectory while transferring the optimized motion to its actual capture time. The retained depth is unchanged. For either assembly method, final pointmaps are obtained by unprojecting the retained depth with the assembled camera poses using Eq. (2). Table 6 compares the two methods under a matched checkpoint and window schedule.

VariantTrajectory ATE↓ (cm)Trajectory RTE↓ (cm)Trajectory RRE↓ (°)Rig RTE↓ (cm)Rig RRE↓ (°)Depth AbsRel↓ (%)Depth δ1.25\delta_{1.25} ↑ (%)
Stitching, no mask69.532.940.0425.770.8020.12783.62
Stitching, sky mask65.552.890.0425.750.8020.12783.68
PGO33.112.180.0415.640.8010.12584.02
PGO + LM31.261.210.0415.600.8010.12584.01

Table 6. Video assembly with stitching and pose graph optimization. All variants use checkpoint (f) and the protocol in Table 5, including one multiview Sim(3) evaluation alignment per clip with a shared pose/depth scale. Sky masks filter stitching correspondences; PGO uses capture-time corrections without pixel-alignment masks. LM denotes the linear-motion prior.

In our PGO implementation, factors connect samples separated by 1, 2, 4, or 8 sampled rows within each window. Their weights decrease linearly with the pair’s midpoint position, from 1 at the window’s beginning to 0.1 at its end. Rotations are initialized by chaining averaged adjacent-frame measurements and refined with L-BFGS using rotation-vector updates. We use λmotion=30\lambda_{\mathrm{motion}}=30; setting it to zero disables the motion prior, which is also omitted for trajectories with fewer than three nodes.

Offline multi-object tracking

We collect 3D detections over the complete clip and use the assembled ego trajectory to transform their centers, orientations, and velocities from timestamp-specific rig frames into a common world frame. We then associate detections into tracks in temporal order. A constant-velocity Kalman filter maintains each track’s planar position and velocity, using both positions and velocities predicted by the detection model as measurements. At each observation time, the filter predicts track positions, and Hungarian assignment matches detections to these predictions by minimizing planar distance subject to a distance gate. Matched detections update the filter state, unmatched detections initialize new tracks, and unmatched tracks remain eligible for association for a bounded time.

Tracking proceeds in two phases. The first uses a shorter track lifetime to form conservative trajectories; we then interpolate gaps and smooth the trajectories before a second pass with a longer lifetime reconnects remaining fragments. Final planar positions and velocities are refined with a Rauch–Tung–Striebel (RTS) backward smoother, allowing later observations to improve earlier state estimates. Height and yaw are smoothed separately, with yaw unwrapped before interpolation and smoothing, and each trajectory is assigned median cuboid dimensions.

Physical AI Autonomous Vehicles Foundation4D

Physical AI Autonomous Vehicles (PAI) [8] provides multi-sensor driving recordings for autonomous vehicle research, including multi-camera RGB videos, LiDAR point clouds, and radar detections. The dataset also provides sensor calibration, ego-motion trajectories, and automatically generated 3D cuboid tracks with object poses, dimensions, semantic classes, and track identifiers. We augment the PAI sequences used in our benchmark with LiDAR-derived depth maps and occupancy grids with per-voxel semantic and velocity labels for dense 3D perception evaluation [7]. Our geometry, detection, and semantic occupancy evaluations use the same 1,253 PAI test clips and six cameras. Task-specific protocols are described in § 7.

The annotation pipeline aggregates static LiDAR points across time and combines motion-compensated dynamic points within local temporal windows. As auxiliary inputs, it uses semantic masks precomputed by an internal semantic segmentation model and instance masks generated by SAM 2 [108] using box prompts derived from projected 3D cuboids. Camera calibration and sensor poses align LiDAR points across time and camera views, while cuboid tracks guide motion compensation and supplement image semantic predictions with cuboid class labels. The pipeline generates two outputs: depth maps rendered in each camera view and semantic occupancy grids in the vehicle frame. Figure 4 summarizes the annotation pipeline, and Figure 5 illustrates the resulting depth and semantic occupancy annotations on two PAI examples.

Figure 4. PAI depth and occupancy annotation pipeline. The pipeline combines temporally aggregated LiDAR, cuboid tracks, and image masks to generate camera-view depth maps and vehicle-frame occupancy grids with per-voxel semantic and velocity labels.

Generated depth and semantic occupancy annotations on PAI

Figure 5. Generated depth and semantic occupancy annotations on PAI. Each example shows multi-camera RGB images overlaid with depth (top), followed by unprojected depth colored by RGB, a side view of semantic occupancy, and a forward view of semantic occupancy (bottom, left to right). The shared legend indicates depth in meters and semantic classes.

Temporal LiDAR aggregation

Single LiDAR scans sparsely sample the scene, so we accumulate measurements over time to obtain more complete surface coverage. We compensate each return for ego motion using its acquisition timestamp and transform it into a common world frame. Cuboid tracks identify actors that require motion compensation: static points are accumulated across the full clip, while dynamic points are collected from a local window of up to ten LiDAR scans and warped to the actor pose at the target timestamp. This separation increases coverage while preserving the positions of moving objects.

Smoothed cuboid trajectories are used to classify tracks as static or dynamic. For dynamic tracks, we interpolate the original cuboid poses to select points within padded cuboid crops and align them across time. Ground points within these crops are excluded from object reconstruction, with only high-confidence ground returns restored to the static cloud. The accumulated static cloud is then filtered by local density while preserving sparse planar surfaces supported by multiple scans. The pipeline also supports optional intensity filtering, with separate thresholds for static and dynamic returns.

Depth annotation generation

We generate metric depth maps by rendering the accumulated static scene and motion-compensated actors into each camera view. Projection uses calibrated intrinsics and extrinsics at the camera’s native resolution, accounting for camera motion during rolling-shutter exposure. Static depth is obtained from the nearest visible projected LiDAR point; dynamic points are rendered with fixed-radius Gaussian splats. Dynamic depth is integrated into the static depth map where instance masks overlap with dynamic semantic regions, subject to occlusion checks. Objects without instance masks use direct point projection.

Visibility checks prevent occluded points from contributing to the depth map. We compare candidate depths against measured static depth and an auxiliary splatted reference, rendered from the static cloud, using the latter only where it is consistent with nearby measurements and semantic labels. Within masked dynamic regions, pixels without rendered dynamic depth are marked invalid instead of retaining background depth. Static depth is retained when a dynamic prediction is rejected by occlusion checks. Visible dynamic points from direct projection can fill missing depth. Each valid output stores distance along the camera viewing ray in meters. Missing depth, sky, and ego-vehicle pixels are set to zero.

Semantic occupancy annotation generation

We construct semantic occupancy annotations by assigning classes to the accumulated 3D points and voxelizing them in the target vehicle frame. Static points receive depth-consistent semantic votes from precomputed image semantic segmentation maps across multiple cameras and frames, with greater weight given to nearer observations. These votes are supplemented by the classes of static cuboid tracks. The highest accumulated weighted score determines each static point’s label, while dynamic points inherit the class of their associated cuboid track. Traffic-sign and traffic-light labels are then refined by propagating them from dense labeled regions to nearby connected points in 3D. Unlabeled and sky points are removed before constructing the static occupancy map.

The static cloud is pre-voxelized in the world frame and cleaned to remove isolated components. At the end timestamp of each front wide-angle camera frame, we transform the static voxel centers and motion-compensated dynamic points into the vehicle frame. The exported grid uses 0.1 m cubic voxels over x∈[−150,150)x \in[-150, 150) m, y∈[−100,100)y \in[-100, 100) m, and z∈[−3,5)z \in[-3, 5) m. Static voxel centers and dynamic points are voxelized separately, with majority voting within each set. Dynamic voxels take precedence where the two sets overlap, except where static voxels represent road, sidewalk, or terrain. Each occupied voxel also stores a 3D velocity in meters per second, representing world-relative motion expressed in vehicle coordinates. Here, static voxels have zero velocity while dynamic voxels inherit their track’s translational velocity, estimated from successive cuboid centers.

Experiments

We evaluate Foundation4D on geometry, 3D detection and tracking, semantic occupancy, and Gaussian reconstruction using internal datasets and public benchmarks2. Component and inference ablations examine the effects of model design, supervision, and video-level assembly. Finally, we compare different combinations of training tasks to assess the benefits and trade-offs of learning a shared representation.

Geometry

Our geometry heads aim to provide a consistent video-level reconstruction of the 3D scene captured by multi-camera sensors. We evaluate this capability along three axes: (i) multi-camera video reconstruction on public and PAI benchmarks (§ 7.2 and § 7.3); (ii) the contribution of core model design choices; and (iii) the impact of our video inference system (§ 7.4.1).

Evaluation protocol.

We jointly align the predicted multi-camera trajectories to their ground-truth counterparts with a single similarity transform (s,R,t)∈Sim⁡(3)(s, \mathbf{R}, t) \in\operatorname{Sim}(3) per evaluation sequence: one window for window-level evaluation or one complete clip for video-level evaluation. We compute trajectory errors from the aligned camera poses: ATE measures the mean camera-center error, while RTE and RRE measure relative translation and rotation errors between consecutive poses. We apply the same scale ss to all predicted depth maps, D~t,c=sD^t,c\widetilde{D}_{t,c} = s\widehat{D}_{t,c}, and evaluate AbsRel and δ1.25\delta_{1.25} against ground-truth depth over valid pixels. AbsRel is the mean absolute relative depth error; δ1.25\delta_{1.25} is the percentage of pixels whose predicted-to-ground-truth depth ratio lies between 11.25\frac{1}{1.25} and 1.251.25. Rig RTE and RRE compare the recovered static inter-camera transforms with calibrated extrinsics, using the same scale ss for rig translations. Rig mAA@KK averages joint rotation and translation-direction accuracy over angular thresholds from 1∘1^\circ to K∘K^\circ. Alignment to ground truth is used only for evaluation.

Public benchmarks

We compare the geometric reconstruction performance of Foundation4D with recent driving geometry and unified perception models on the public Waymo [9] and nuScenes [10] datasets, evaluating depth, trajectory, and camera-rig accuracy.

Baselines.

We consider DVGT-1, DVGT-2, LiAuto-GeoX, and GeoUP as our baselines, using their publicly released checkpoints and codebases. DVGT-1 [40] predicts metric pointmaps and ego poses from unposed multi-camera images, while DVGT-2 [23] predicts local pointmaps and relative ego motion through streaming inference. Both take images alone as model inputs; for evaluation, we use known camera-to-rig extrinsics to derive camera poses and depth from their ego-frame outputs. LiAuto-GeoX [44] is a compact model distilled from a pose-conditioned geometry teacher; we use its released student to predict depth from images alone. Although it also predicts camera poses, we lack sufficient implementation details to reproduce its reported pose performance.

GeoUP [3] jointly predicts depth, 3D objects, and semantic occupancy, assuming known camera intrinsics, camera-to-rig calibration, and ego poses. We evaluate all methods on depth estimation and compare camera pose accuracy only with DVGT-1. None explicitly estimates a shared, static camera-to-rig calibration as Foundation4D does.

Setup.

Following all four baselines, we include Waymo and nuScenes in our training domain. We therefore fine-tune the backbone and geometry decoders of Foundation4D for 20,000 iterations on a mixture of our internal NRE data and the training splits of Waymo (798 sequences, five cameras) and nuScenes (700 scenes, six cameras), sampled with probabilities 0.30, 0.35, and 0.35, respectively.

We evaluate on the official validation splits of Waymo and nuScenes, with ground-truth depth rendered from clip-aggregated LiDAR using the pipeline in § 6. We compare depth across baselines using window-level inference on all non-overlapping 48-frame windows and average metrics over all window predictions. Independent scale alignment within each window limits the effect of accumulated pose drift on depth evaluation. For ego-pose evaluation, we adopt the more practical video-level setting and process complete clips. For a fair comparison, we integrate both DVGT-1 and Foundation4D into our video inference system (§ 5), using the same 48-frame windows with a 5-frame overlap at each dataset’s native frame rate.

Results.

On both Waymo and nuScenes, the fine-tuned Foundation4D model achieves the lowest depth AbsRel and highest δ1.25\delta_{1.25} among the evaluated methods (Table 2). It also reduces video-level trajectory ATE by 29–57% relative to DVGT-1 (Table 3). Rig mAA@15 exceeds 83% on Waymo and 99% on nuScenes, showing that the model can estimate the camera arrangement alongside vehicle motion without supplied rig extrinsics or ego poses.

Depth estimation on the Waymo and nuScenes validation splits. Depth is evaluated independently in non-overlapping windows of 48 consecutive frames per camera, sampled at the native frame rates (Waymo: 10 Hz; nuScenes: ≈12\approx12 Hz).

ModelWaymoWaymonuScenesnuScenes
AbsRel↓\downarrowδ1.25↑\delta_{1.25} \uparrow (%)AbsRel↓\downarrowδ1.25↑\delta_{1.25} \uparrow (%)
DVGT-1 [40]0.2072.870.1878.97
DVGT-2 [23]0.2670.930.2275.32
LiAuto-GeoX [44]0.1975.720.2762.75
GeoUP [3]0.1387.300.1980.05
Foundation4D0.0991.970.1586.44

Table 2. Depth estimation on the Waymo and nuScenes validation splits. Depth is evaluated independently in non-overlapping windows of 48 consecutive frames per camera, sampled at the native frame rates (Waymo: 10 Hz; nuScenes: ≈12\approx12 Hz).

Video-level geometry comparison on the Waymo and nuScenes validation data described in § 7.2, at native camera rates (10 Hz and ≈12\approx12 Hz). Each clip uses one evaluation Sim(3), shared by poses and depth. Both models use our video inference system with 48-frame windows and a 5-frame overlap for stitching and PGO+LM. Rig metrics are omitted (–) because DVGT-1 does not estimate the rig.

DatasetInference ModeModelTrajectoryTrajectoryTrajectoryRigRigRigDepthDepth
ATE↓\downarrow (cm)RTE↓\downarrow (cm)RRE↓\downarrow (°)RTE↓\downarrow (cm)RRE↓\downarrow (°)mAA@15↑\uparrow (%)AbsRel↓\downarrowδ1.25↑\delta_{1.25} \uparrow (%)
WaymoStitchingDVGT-1 [40]62.333.540.06–––0.2073.04
WaymoStitchingFoundation4D32.703.200.060.961.0283.370.0992.33
WaymoPGO+LMDVGT-1 [40]54.552.380.05–––0.2073.26
WaymoPGO+LMFoundation4D23.621.560.060.901.0383.540.0992.94
nuScenesStitchingDVGT-1 [40]31.023.980.07–––0.1779.28
nuScenesStitchingFoundation4D22.132.150.052.060.1899.220.1586.58
nuScenesPGO+LMDVGT-1 [40]30.541.240.07–––0.1779.28
nuScenesPGO+LMFoundation4D17.481.190.052.020.1899.220.1586.63

Table 3. Video-level geometry comparison on the Waymo and nuScenes validation data described in § 7.2, at native camera rates (10 Hz and ≈12\approx12 Hz). Each clip uses one evaluation Sim(3), shared by poses and depth. Both models use our video inference system with 48-frame windows and a 5-frame overlap for stitching and PGO+LM. Rig metrics are omitted (–) because DVGT-1 does not estimate the rig.

PAI benchmark

We evaluate Foundation4D on the PAI benchmark introduced in § 6. The evaluation covers ego poses, camera-to-rig calibration, and depth across distance ranges, complemented by qualitative scene reconstructions.

Setup.

We evaluate the unified Foundation4D model on geometry tasks using the full PAI test split of 1,253 clips and six cameras, comparing against an internally fine-tuned VGGT baseline. We fine-tune the original VGGT architecture on the same internal data using our best geometry specialist training schedule. We retain the official VGGT-Commercial initialization, which yields slightly better geometry performance than LingBot-Map initialization in our experiments. Because VGGT lacks semantic masks and predicts geometry up to scale, we evaluate it only with stitching without sky masking. Foundation4D uses sky-masked stitching and PGO+LM. Both models use the same window sampling schedule; metrics follow the shared geometry evaluation protocol and are averaged equally across clips.

Ego and rig calibration results.

With stitching, Foundation4D reduces trajectory ATE by 29.8% relative to the internally fine-tuned VGGT baseline and improves rig calibration (Table 4). Rig translation error decreases by 86.7%, while rig mAA@15 rises from 42.50% to 88.72%. PGO+LM further reduces trajectory ATE from 78.06 to 52.75 cm.

Depth accuracy across ranges.

Both Foundation4D inference modes reduce AbsRel in every depth range relative to VGGT (Figure 6). The relative improvement is larger at long range: with stitching, it grows from 2.2% within 10 m to 7.7% beyond 60 m; PGO+LM reaches an 8.4% reduction in the far range. The example in Figure 7 shows multi-camera depth reconstruction of an overpass and the road beyond it, while highlighting residual errors around distant structures and depth discontinuities.

Figure 6. Depth error by ground-truth depth range on PAI, averaged equally over 1,253 clips in each bin. AbsRel is dimensionless; the y-axis starts at 0.05. Foundation4D stitching uses sky masks; VGGT stitching and PGO+LM do not. Each configuration uses its shared full-clip Sim⁡(3)\operatorname{Sim}(3) evaluation scale, with no per-bin alignment. {sup}†` denotes internal fine-tuning.

Figure 7. Foundation4D depth predictions with PGO+LM from the front-wide and front-tele cameras of a PAI clip. Each full view is followed by a 3.2×3.2 \times enlargement of the region boxed in the RGB image. Absolute depth error and GT-masked predictions use valid GT support; the final column excludes predicted sky and ego-vehicle pixels from the dense prediction. Gray denotes masked pixels. Depth and error colors saturate at 300 m and 30 m, respectively. Predictions retain the shared full-clip pose-derived scale without local realignment.

Qualitative results on video-level reconstruction.

Beyond per-image depth, Figure 8 shows four video-level reconstructions whose road layouts, building facades, and vegetation share a common 3D frame. The overpass detail in Figure 9 pairs one input image with accumulated geometry rendered from the corresponding predicted camera pose, showing the bridge, supporting pier, and lower roadway. The input captures one instant, while the reconstruction integrates observations across the clip.

Figure 8. Video-level point-cloud reconstructions on PAI using Foundation4D with PGO+LM. Each scene accumulates six-camera predictions over an approximately 20 s clip. Before confidence-filtered depth fusion, we remove moving objects using external annotation-assisted masks and suppress sky and ego-vehicle pixels using predicted semantics. Figure 9 shows a detailed reconstruction of the overpass scene with its RGB input.

Detailed reconstruction of the overpass scene

Figure 9. Detailed reconstruction of the overpass scene in Figure 8(b). The front-wide RGB input at τ=8.67\tau= 8.67 s is paired with RGB and semantic point-cloud renderings using the corresponding predicted camera pose and calibrated projection. Both renderings show the same geometry, accumulated from all six cameras across the full clip using depth fusion with dynamic-point restoration. White regions contain no rendered points.

Ablations

We ablate the geometry architecture and supervision to assess their contributions to depth, ego-pose, and rig-calibration accuracy. We also compare window stitching and pose-graph optimization to understand how inference choices affect video-level reconstruction.

Setup.

All variants share a reduced 16-block aggregator, the same training samples and recipe, and LingBot-Map initialization. Each is trained for 30,000 updates with 12 temporal samples per camera and a uniformly sampled stride up to 5. Starting from a VGGT [4] baseline adapted to this setup, we cumulatively add factorized global attention, rig-ego factorization with separate rig and ego losses, metric-scale training, relative-pose supervision, and auxiliary semantic supervision. Stitching without semantic masks provides a common inference protocol supported by every variant, enabling a fair model comparison. For inference ablations, we fix variant (f), which achieves the lowest ATE and depth AbsRel, and compare stitching without and with sky masks, PGO, and PGO with a linear-motion prior (LM).

We evaluate on 50 internal clips using four cameras (front-wide, front-tele, cross-left, and cross-right), with RGB and depth at 518×518518 \times518. We sample every second frame from the 15 FPS anchor timeline and process complete clips in 48-timestamp windows per camera with a 5-timestamp overlap. Metrics are averaged equally across clips. This cohort is separate from the joint-task study in § 7.11.

Impact of different model and supervision designs.

Adding factorized global attention to the baseline reduces observed end-to-end runtime by 8.5% on one H100 GPU, with a trade-off in video-level and depth accuracy (Table 5). The rig-ego variant improves trajectory, rig, and depth accuracy, yielding the lowest rig translation error and highest rig mAA@15 among the variants. These gains support explicitly modeling the static camera arrangement alongside temporal motion. Metric-scale training provides the largest gain in video-level reconstruction, approximately halving ATE from 141.72 to 70.67 cm while improving depth. This supports supervising windows in a common unit to improve their assembly. Relative-pose supervision then improves local translation and rotation accuracy, with a small increase in depth error. Auxiliary semantic supervision yields the best depth accuracy and further improves trajectory. translation, although rig accuracy does not improve uniformly. Together, the results distinguish complementary contributions to scale consistency, local motion, and dense geometry.

VariantTrajectory ATE↓ (cm)Trajectory RTE↓ (cm)Trajectory RRE↓ (°)Rig RTE↓ (cm)Rig RRE↓ (°)Rig mAA@15↑ (%)Depth AbsRel↓ (%)Depth δ1.25\delta_{1.25} ↑ (%)
(a) Baseline128.485.040.0867.750.70171.780.13383.16
(b) + factorized global attention149.884.870.0867.160.80874.240.14082.25
(c) + rig-ego factorization141.724.310.0535.000.79677.910.13682.51
(d) + metric-scale training70.673.430.0535.880.83476.980.12983.58
(e) + relative-pose loss69.813.220.0425.620.83576.910.13083.41
(f) + semantic supervision69.532.940.0425.770.80275.690.12783.62

Table 5. Cumulative geometry ablation on 50 internal clips with stitching and no semantic masks. Row (c) introduces rig-ego factorization with separate rig and ego losses. ATE uses one multiview Sim(3) alignment per clip, whose scale is also applied to depth.

Impact of different inference modes.

In Table 6, sky masking modestly improves stitching by excluding unreliable finite-depth correspondences from window alignment. PGO then nearly halves ATE relative to sky-masked stitching and improves local motion accuracy. Jointly reconciling overlapping relative-pose constraints allows later windows to revise earlier estimates, reducing error accumulation from sequential alignment. The linear-motion prior further improves translation consistency by discouraging abrupt velocity changes, with a smaller additional gain in ATE. PGO+LM achieves the lowest ATE and relative translation error. On the example sequence in Figure 10, PGO reduces the large longitudinal deviations of stitching, while LM suppresses short-interval oscillations. PGO retains the predicted depth; differences in its depth scores reflect the pose-derived scale used for evaluation.

Figure 10. Camera-position errors under different video assembly methods. In this example sequence, each method uses one full-clip multiview Sim(3) alignment shared across four cameras. Signed front-wide residuals (cm) are projected onto the ground-truth forward, left, and world-up axes and plotted against cumulative ground distance (m). Dotted lines mark window starts; all frame samples are shown without smoothing.

3D object detection and tracking

We evaluate 3D object detection and the trajectories assembled from its predictions. We study adaptation to nuScenes, performance on the full PAI test split, and the effects of detector design and offline inference through ablations.

Public benchmarks

We evaluate Foundation4D on the public nuScenes [10] 3D detection benchmark and compare it with state-of-the-art camera-only 3D detectors under the official protocol.

Setup.

We initialize from the Foundation4D detector pre-trained on our internal NRE, PlanNet, and MLMCF data without nuScenes, re-initialize its classifier for the 10 nuScenes classes, and fine-tune all weights. For a fair comparison with prior methods, we follow the standard nuScenes training and evaluation protocol, using 2 Hz annotated keyframes from all six cameras across 700 training scenes and 150 validation scenes (evaluation only).

We use the image and BEV augmentations and GridMask of StreamPETR [54] and GeoUP [3], with the same ±51.2 m\pm51.2\ \mathrm{m} detection range. We train for 10,000 iterations using AdamW with a batch size of 32 windows and a base learning rate of 4.5×10−44.5 \times10^{-4}, applying learning-rate multipliers of 0.1 and 0.03 to the aggregator and image encoder, respectively. Following prior work, we evaluate predictions on all 150 validation scenes using the official nuScenes devkit and report mAP and the nuScenes detection score (NDS), ensuring a consistent evaluation protocol for the comparisons in Table 7. At test time, Foundation4D processes consecutive eight-keyframe windows, so its predictions also use future frames within each window.

ModelArchitectureBackbone Init.mAP ↑\uparrowNDS ↑\uparrow
PETRv2ResNet101nuImages42.152.4
SparseBEVResNet101nuImages50.159.2
Stream3DPPEVoV-99FCOS3D50.058.5
RoPETRVoV-99FCOS3D52.961.4
StreamPETREVA02-LCascade R-CNN52.160.8
RayDNEVA02-LCascade R-CNN54.162.4
RayDNViT-LDINOv349.955.3
GeoUP-nuScenesViT-LVGGT56.563.0
GeoUP-JointViT-LVGGT59.265.3
Foundation4DViT-LFoundation4D60.966.5

Table 7. Camera-only 3D object detection performance on the nuScenes validation set (150 scenes). All models are trained using supervision on the 700-scene nuScenes training set.

Baselines.

We compare with PETRv2 [15], SparseBEV [53], Stream3DPPE [51], RoPETR [55], StreamPETR [54], and RayDN [56] using their published results. The most related method, GeoUP [3], combines a VGGT-initialized [4] ViT-L backbone with a RayDN-style detector and four-frame inputs. We evaluate the released GeoUP-nuScenes checkpoint, trained exclusively on nuScenes, for which the paper reports 57.9 mAP and 64.4 NDS. We also include the published results of GeoUP-Joint3, trained jointly on nuScenes, Argoverse 2, Waymo, KITTI, and DDAD with detection, depth, and occupancy supervision. For RayDN, we report GeoUP’s reproduced results using a DINOv3 [109] ViT-L backbone.

Results.

Foundation4D achieves the best results, 60.9 mAP and 66.5 NDS (Table 7). It improves over the second-best method GeoUP-Joint by +1.7 mAP and +1.2 NDS. This suggests that the Foundation4D checkpoint is a strong starting point for new camera rigs and label sets. A larger generic backbone alone does not explain the gains: with a DINOv3 ViT-L, RayDN reaches only 49.9 mAP, below its EVA02-L version, whereas both geometry-pretrained ViT-L models exceed 56 mAP.

PAI benchmark

Setup.

We evaluate the unified Foundation4D checkpoint on all 1,253 PAI test clips, covering 750,552 timestamps, using the checkpoint obtained after Gaussian-reconstruction fine-tuning. We use six physical cameras at 1036 ×\times 1036 resolution and 32-timestamp windows with spacing of two source frames during inference. Interleaved sampling covers every source timestamp. Predictions are evaluated against the officially released PAI cuboid annotations [8] (see the classes listed in § 3.7), restricted to objects visible in the selected cameras. Detection uses BEV center-distance matching at thresholds of 0.5, 1, 2, and 4 m. Tracking uses the full offline pipeline and center-distance HOTA [110], with similarity decreasing to zero at 2 m. To focus on the most representative and common classes, we report the five-class macro average over automobile, person, rider, bus, and heavy truck, together with the all-twelve-class result. Tracking statistics are combined over clips within each class before taking the unweighted class mean.

Results and distance breakdown.

Foundation4D achieves 38.00% mAP and 34.76% HOTA over the five main classes, and 29.11% mAP and 25.66% HOTA over all twelve classes in the full evaluation. Figure 11 reports detection mAP by planar distance from the ego vehicle and tracking HOTA over the full evaluation. For the five main classes, mAP is 52.23% within 64 m, 40.43% within 108 m, and 11.16% in the 108–200 m interval. The corresponding scores over all twelve classes are 38.95%, 30.87%, and 9.11%, respectively. The two near-range evaluations overlap and are scored independently. These results show the remaining difficulty of long-range detection. Since the evaluation regions and class definitions differ from nuScenes, the reported mAP scores should not be compared directly with its headline results.

Figure 11. Foundation4D detection and tracking on the PAI test split. (a) Detection mAP by planar distance from the ego vehicle, with full-evaluation scores shown separately. Each distance interval independently filters predicted and ground-truth box centers. (b) Tracking HOTA over the full evaluation, using center-distance matching. Colors distinguish averages over the five main classes and all twelve classes.

Qualitative results.

Figure 12 shows how the offline pipeline assembles detections into 3D tracks in a PAI street scene, including vehicles and pedestrians at different distances.

Foundation4D detection and tracking across temporal views

Figure 12. Foundation4D detection and tracking on the PAI dataset. Foundation4D offline tracks are projected into the front-wide camera. Top: the overview at 8 s; the dashed rectangle marks the region shown in the first temporal crop. Bottom: cropped views at 8, 10, and 12 s. Colors and numeric labels denote track identities. We display the five main classes with confidence at least 0.4.

Ablations

Setup.

We train the component variants for 80,000 updates on the internal driving datasets PlanNet and MLMCF using four fixed front cameras and detection-only supervision. Training windows contain 8 temporal samples with randomized spacing up to 8 source frames. We evaluate the final checkpoints on the 300-clip PlanNet validation split, reporting mAP and center-distance HOTA averaged over automobile, person, rider, bus, and heavy truck. Ground truth is restricted to objects visible in the four cameras. Table 8 also includes a model trained with a scaled schedule and multiple tasks.

ConfigurationmAP ↑\uparrowHOTA ↑\uparrow
VoVNet (320×800320 \times800)21.421.5
DINOv2 (532×952532 \times952)30.128.8
VGGT (518×518518 \times518)27.526.6
+ resolution 1036×10361036 \times103631.730.2
+ deformable attention32.129.3
+ QueryDN / auxiliary YOLOX34.732.5
Scaled schedule and multi-task training47.043.6

Table 8. Ablation study for 3D object detection and tracking. All models are trained on four cameras, and all except the last configuration use a reduced 80k steps training schedule. We indicate in parenthesis next to each backbone its image resolution.

Impact of model and supervision choices.

Table 8 compares dense-attention PETR detectors with VoVNet, DINOv2, and VGGT features. Starting from VGGT, we increase image resolution, add FPN features with deformable cross-attention, and introduce query denoising with auxiliary 2D supervision. Higher image resolution improves both detection and tracking while keeping the VGGT aggregator grid unchanged. FPN features and deformable attention further improve mAP, with a decrease in HOTA. Query denoising and auxiliary 2D supervision improve both metrics, reaching 34.7% mAP and 32.5% HOTA. The scaled multi-task reference reaches 47.0% mAP and 43.6% HOTA under the different evaluation settings specified in the caption.

Temporal context at inference.

We vary the number of timestamps processed jointly at inference time using a fixed checkpoint of the scaled multi-task model. Intuitively, increasing the window size enables enhancing per-view image features with further spatio-temporal context via the multi-frame aggregator transformer. Increasing the window from 2 to 32 timestamps improves mAP by 10.07 points and HOTA by 6.89 points without retraining. Extending to 48 timestamps adds 0.59 mAP points but reduces HOTA by 0.32 points. Detection therefore continues to improve over the tested range, while tracking reaches its highest score at 32 timestamps.

Tracking motion estimation and smoothing.

We hold the detector outputs, association schedule, confidence filtering, class mapping, and ground truth fixed, using the same scaled multi-task checkpoint with 32-timestamp inference windows. The baseline uses detector velocities for motion prediction, Hungarian association, gap interpolation, and Savitzky–Golay smoothing. All three configurations use the same two association passes, with maximum gaps of 24 and 50 native 30 Hz frame indices. We first introduce Kalman prediction and updates, then replace planar trajectory smoothing with a Rauch–Tung–Striebel (RTS) backward pass. RTS is applied to all classes after the final association pass and to person tracks after both passes. As shown in Table 10, Kalman filtering improves HOTA from 41.4 to 42.1, a gain of 0.7 points, and RTS smoothing adds 1.5 points, reaching 43.6 for a total gain of 2.2 points over the velocity baseline.

ConfigurationHOTA ↑
Velocity baseline41.4
+ Kalman filtering42.1
+ RTS smoothing43.6

Table 10. Tracking inference ablation.

Impact of predicted ego-motion.

As described in § 5.3, detections are transformed into a common reference frame for multi-object tracking using ego-poses. We evaluate the impact of replacing ground-truth ego-poses with predicted poses obtained through sequential window alignment. In Table 11, we first isolate pose-induced localization error using ground-truth boxes and identities, without running the tracker. Replacing ground-truth ego-poses with predicted poses reduces HOTA from 100.0 to 83.8, indicating residual localization error even with perfect detections and identities. With predicted detections and a fixed tracking configuration, the impact is more modest: HOTA decreases from 43.6 to 40.5, a drop of 3.1 points (7.1% relative). Thus, while pose accuracy affects localization, the full tracking pipeline retains most of its performance when using predicted ego-motion.

ConfigurationGround truthPredicted
GT boxes and identities (oracle)100.083.8
Predicted boxes and tracker43.640.5

Table 11. Impact of predicted ego-motion on tracking.

Occupancy

Our occupancy branch predicts occupied space and assigns semantic labels on a high-resolution voxel grid from multi-camera video. We evaluate this capability along two main directions: (i) binary occupancy and semantic prediction accuracy on public and PAI benchmarks; and (ii) the contribution of each design choice through component ablations.

Public benchmark results

Setup.

We fine-tune Foundation4D on OpenOccupancy [11], a surround-view semantic occupancy benchmark that extends nuScenes [10] with dense voxel annotations. The benchmark evaluates 16 semantic classes at a voxel resolution of 0.2 m over a region of interest of [−51.2,51.2]×[−51.2,51.2]×[−5,3][-51.2, 51.2] \times[-51.2, 51.2] \times[-5, 3] m. We train for 30,000 updates with an initial learning rate of 3×10−43 \times10^{-4} and cosine decay, a batch size of 8, temporal stride 1, 1036 × 1036 input images, and random color augmentation. Compared with pre-training, this benchmark changes the semantic classes and spatial extent while retaining the same voxel resolution.

Baselines.

We compare Foundation4D with camera-only baselines spanning plane-based, dense volumetric, and sparse 3D representations. TPVFormer [68] aggregates multi-camera features into three orthogonal feature planes. Dense volumetric approaches include MonoScene [71], OccGen [75], and OccMamba [74], while CONet [11] uses coarse-to-fine volumetric refinement. Sparse approaches include SparseOcc [78] and SPSC [80]. We also compare against SPSC-temporal, which incorporates historical frames and achieves the strongest baseline results in our comparison. For CONet, OccGen, OccMamba, and both SPSC variants, we use the camera-only OpenOccupancy scores reported in [80].

Results.

Foundation4D achieves the highest overall IoU and mIoU among the methods in Table 12: 27.4 IoU and 18.1 mIoU, compared with 25.5 and 16.2 for SPSC-temporal [80]. It also improves on SPSC-temporal in most semantic classes. The largest remaining gaps to this baseline concern barrier and traffic cone, which may reflect their absence from the pre-training label set. These results highlight the value of large-scale joint pre-training across multiple tasks: after adaptation to OpenOccupancy, our pre-trained occupancy network outperforms the TPV-based, dense volumetric, and sparse 3D baselines included in this comparison, despite the simple design of its BEV occupancy decoder.

MethodIoUmIoUbarrierbicyclebuscarconst. v.motor.ped.t. conetrailertruckdrive. surf.other flatsidewalkterrainmanmadeveg.
MonoScene [71]18.46.97.13.99.37.25.635.94.44.94.214.96.37.97.4107.6
TPVFormer [68]15.37.89.34.111.310.15.24.35.95.36.86.513.698.389.28.2
CONet [11]20.112.813.28.115.417.26.311.2108.34.712.131.418.818.716.34.88.2
OccMamba [74]21.713.213.27.415.317.66.110.310.36.56.213.332.920.519.817.44.88.7
SparseOcc [78]21.814.116.19.315.118.67.39.411.29.47.21331.821.720.718.86.110.6
OccGen [75]23.414.515.59.115.319.27.311.311.88.95.913.734.82221.819.569.9
SPSC [80]24.315.218.99.619.421.38.511.013.99.46.61434.819.222.418.85.39.6
SPSC-temporal [80]25.516.219.910.114.319.110.810.67.111.38.214.840.523.425.921.68.013.3
Foundation4D27.418.118.011.217.321.110.114.311.38.19.716.539.928.528.024.213.417.7

Table 12. Semantic occupancy results on OpenOccupancy [11]. Scores are percentages.

The qualitative examples in Figure 13 cover urban scenes, including nighttime conditions. Foundation4D reconstructs extended regions of driveable surface and sidewalk, together with surrounding vegetation and manmade structures. It also localizes individual car, bicycle, and pedestrian instances. The predictions recover both the overall scene layout and local geometric detail, including thin roadside structures.

Semantic occupancy predictions showing input camera frames, Foundation4D prediction, and GT. Legend: barrier, bicycle, bus, car, constr. veh., moto., ped., traf. cone, trailer, truck, drive. surf., other flat, sidewalk, terrain, manmade, veg.

Figure 13. Semantic occupancy predictions on OpenOccupancy. Each example pairs six camera views with the predicted occupancy and ground truth.

PAI benchmark

Setup.

We fine-tune the temporal aggregator and semantic occupancy modules of Foundation4D on 1,000 clips curated from our PlanNet dataset and annotated using the pipeline described in § 6. PlanNet shares the same camera setup as PAI. We train for 45,000 iterations using six cameras, 1036×10361036 \times1036 input images, an initial learning rate of 3×10−33 \times10^{-3} with cosine decay, batch size of 8, and a temporal stride of 2. We retain the pre-trained classifier weights for shared classes and randomly initialize the output weights for new PAI classes. The PAI occupancy annotations have a resolution of 0.1 m, which we re-voxelize to 0.2 m to match our output resolution. We use frequency-based semantic label assignment to preserve class coverage and limit the erosion of thin structures and class boundaries during re-voxelization. During evaluation, each forward pass uses 6 cameras at eight timestamps sampled with temporal stride 2, corresponding to 48 input images. We apply a binary occupancy confidence threshold of 0.5. Table 13 reports binary IoU, semantic mIoU, and per-class scores over the full ROI, [−20,100]×[−30,30]×[−3,5][-20, 100] \times[-30, 30] \times[-3, 5] m, and separate longitudinal regions. Following OpenOccupancy [11], we evaluate over the full ROI without visibility masking, including both visible and occluded regions. Due to

ROI xx (m)IoUmIoUroadsidewalkbuild.wallfencepoletraf.l.traf.s.veg.terrainpersonridercartruckbus
[−20,0][-20, 0]38.418.460.634.28.213.78.010.07.95.613.530.19.614.026.918.415.2
[0,50][0, 50]35.016.056.130.06.912.66.28.37.15.011.525.28.311.921.814.914.3
[50,100][50, 100]22.96.935.816.12.48.32.71.11.00.44.214.40.81.17.04.14.3
[−20,100][-20, 100]31.514.149.225.85.911.35.56.85.93.99.722.17.911.319.813.712.8

Table 13. Foundation4D semantic occupancy results on the PAI test set. Scores are percentages. All rows use y∈[−30,30]y \in[-30, 30] m and z∈[−3,5]z \in[-3, 5] m. In the class headers, build. denotes building, traf.l. denotes traffic light, traf.s. denotes traffic sign, and veg. denotes vegetation.

limited annotations for motorcycle, bicycle, and train, we remap motorcycle and bicycle to the existing rider class. We exclude train from semantic evaluation while retaining its voxels as occupied in binary evaluation.

Results.

Foundation4D achieves 31.5 IoU and 14.1 mIoU over the full ROI. IoU/mIoU is higher in the nearby regions, at 38.4/18.4 for [−20,0][-20, 0] m and 35.0/16.0 for [0,50][0, 50] m, and falls to 22.9/6.9 in the [50,100][50, 100] m region. The decline is particularly pronounced for thin structures, such as poles and traffic lights/signs, and small objects, such as people and riders. The daytime and nighttime examples in Figure 14 show road layouts, vehicles, and roadside structures. Distant geometry is less complete, consistent with the lower scores at longer ranges.

Semantic occupancy predictions showing input camera frames, Foundation4D predictions, and ground truth

Figure 14. Semantic occupancy predictions on PAI. Each example pairs six camera views with the predicted occupancy and ground truth.

Component ablation

Setup.

We study the occupancy architecture and training recipe of Foundation4D through seven configurations and report the results in Table 14. Each variant is trained on NRE, PlanNet, or their combination, and evaluated on the same 50 validation clips from NRE, covering different driving scenarios. These experiments use 4 input cameras at 512×512512 \times512 resolution and an ROI of [0,100]×[−25,25]×[−3,5][0, 100] \times[-25, 25] \times[-3, 5] m. Training focuses only on the semantic occupancy task, with auxiliary BEV segmentation supervision where indicated. The studied variants vary the backbone, BEV projection, auxiliary supervision, training data, training iterations, and global batch size.

Version0–100 m0–100 m0–50 m0–50 m50–100 m50–100 m
IoUmIoUIoUmIoUIoUmIoU
DINOv2 + LSS + occ. head24.0310.3727.3212.5119.687.19
DINOv2 + BEV attn. + occ. head27.0811.4429.7713.5723.448.45
VGGT + BEV attn. + occ. head27.6411.9430.3314.0124.018.96
+ BEV segmentation27.4612.2830.1614.5423.808.94
+ Dice loss27.4612.5130.3414.6223.619.36
+ additional training data28.6413.3331.4215.6824.899.85
+ longer training and larger batch32.3515.8435.2818.5128.6111.99

Table 14. Architecture and training ablations for semantic occupancy. All configurations use 4 input cameras at 512 × 512 resolution and an ROI of [0,100]×[−25,25]×[−3,5][0, 100] \times[-25, 25] \times[-3, 5] m.

Results.

Replacing LSS [111] with our BEV deformable attention module improves all reported metrics, increasing full-ROI IoU and mIoU from 24.03 and 10.37 to 27.08 and 11.44, respectively. Replacing DINOv2 with VGGT further increases performance to 27.64 IoU and 11.94 mIoU, with improvements in all regions. The auxiliary BEV segmentation task increases full-ROI mIoU from 11.94 to 12.28, and adding Dice supervision further raises it to 12.51. Adding PlanNet to the training data further improves both geometric and semantic prediction, increasing full-ROI IoU and mIoU to 28.64 and 13.33. Extending training and increasing the global batch size through additional GPUs produces the highest scores across all reported metrics and regions: 32.35 IoU and 15.84 mIoU over the full ROI, 35.28 and 18.51 within 50 m, and 28.61 and 11.99 beyond 50 m.

Gaussian reconstruction

We evaluate whether the Gaussian head of Foundation4D produces useful multi-camera 3D Gaussian reconstructions after joint fine-tuning. We compare the Foundation4D Gaussian reconstruction against Instant NuRec [5], using its released multi-view 1.1.0 checkpoint without further fine-tuning, and the recommended inference

Table 15: Gaussian reconstruction quality. Both Foundation4D variants use predicted actor tracks, constructed using their respective ego poses. The GT-ego variant uses GT source and rendering poses; the predicted-ego variant uses predicted source and rendering poses. Instant NuRec takes GT ego poses as input. Scores average five common cameras equally.

configuration. Our primary Foundation4D result uses predicted ego poses and predicted actor tracks from the same model, with rendering cameras obtained from the predicted trajectory. All Foundation4D configurations reuse the same checkpoint after 10k updates of joint fine-tuning; no per-clip optimization is performed.

Evaluation protocol

Our evaluation covers 99 clips from our Foundation4D internal reconstruction benchmark: 49 NRE and 50 PlanNet clips. Both models are scored against the same recorded target exposures and RGB references, at native-wide resolution with a height of 720 pixels and a width of 1278 or 1280 pixels, preserving the camera aspect ratio. Scores average the five common cameras equally: front-wide, cross-left, cross-right, rear-left, and rear-right. Front-tele is excluded from scoring because the Instant NuRec model was not trained for that camera. Foundation4D uses six input cameras, whereas Instant NuRec uses five; each retains its own context-window recipe.

For Foundation4D, source camera poses, actor tracks, and rendering-camera poses are all constructed from model predictions. Known camera intrinsics and fixed camera-to-rig extrinsics are retained. Target camera poses are queried from the predicted rig trajectory at each sensor timestamp and at the rolling-shutter exposure boundaries. No ground-truth ego poses, actor tracks, initial-pose anchors, or ground-truth-based trajectory or scale fits are used to construct the scene or rendering cameras.

Instant NuRec retains its original reference protocol, which requires ground-truth ego poses as input, alongside the RGB images. Consequently, the comparison measures image reconstruction under the respective inference configurations rather than an isolated Gaussian-decoder comparison.

We sample target frames at 1 Hz from the 30 Hz recordings. Each model predicts Gaussians in non-overlapping context windows. Table 15 distinguishes three target sets. Overall reconstruction includes input frames and other targets, using the closest context window for each target. The matched interpolation subset excludes input frames and retains other targets inside both models’ source intervals. The extrapolation set renders future frames from earlier context windows, rather than assigning them to the closest window.

For each camera, PSNR pools squared RGB errors and valid-value counts across clips before applying the logarithm; SSIM pools valid spatial-window statistics. We then average the five camera scores. Dynamic regions are defined by semantic masks for vehicles and other movable objects, including parked vehicles. This category does not isolate objects that are actually moving. The same mask is applied to both models. Since the models use different input-camera counts and context recipes, this benchmark compares their reconstruction systems rather than isolating the Gaussian decoder.

Quantitative comparison

As summarized in Table 15, Foundation4D demonstrates substantial improvements over Instant NuRec. Notably, even without ground-truth poses or tracks, our predicted input variant outperforms Instant NuRec (which consumes GT ego poses) by 1.28 dB in overall PSNR (23.94 vs. 22.66 dB) and 0.02 in SSIM. The performance boost is particularly pronounced in static regions across all evaluation splits (overall, interpolation, and extrapolation). While dynamic regions exhibit slightly lower overall PSNR (19.98 vs. 20.01 dB), SSIM is consistently improved (0.73 vs. 0.72). Under a matching GT-ego pose setting, Foundation4D attains the highest score of 24.02 dB. These findings highlight our method’s robustness in reconstructing static environments, while pointing to dynamic novel-view extrapolation as an avenue for future exploration. All evaluation frames are rendered from the Gaussian predictions of the relevant context windows, rather than from Gaussians merged across the full scene. Full scene merging requires ego pose and is susceptible to pose prediction errors, similar to what is observed in § 7.6.1.

Qualitative results

The multi-camera renderings in Figure 15 illustrate the broad coverage and strong multi-view geometric consistency of Foundation4D Gaussians across all six native viewpoints. In addition, the side-by-side comparisons in Figure 16 demonstrate noticeable visual improvements over Instant NuRec in road surface texture, distant background geometry, and sky appearance under diverse lighting conditions. Note that the per-frame PSNR annotations in Figure 16 reflect individual scene performance, directly corroborating the overall pooled results in Table 15.

Figure 15. Six-camera Gaussian reconstruction with Foundation4D. Each row shows approximately synchronized native-camera renderings of one example scene from the benchmark dataset. Gaussians are predicted feed-forward without per-scene optimization.

Gaussian renders from four example scenes comparing Instant NuRec and Foundation4D in front-wide and cross-left cameras

Figure 16. Gaussian renders from four example scenes. Each row compares Instant NuRec and Foundation4D at the same target exposure timestamp in the front-wide and cross-left cameras. Labels at the top right give per-image PSNR in dB.

Unified model

The scene-level visualization in Figure 17 brings together geometry, semantic occupancy, and tracked objects. We study interactions between geometry, detection, and occupancy in a four-camera setup, then profile the final six-camera unified model with all task heads enabled.

Task-combination study

We train all seven combinations of geometry (GEO), detection (DET), and occupancy (OCC), as summarized in Table 16. Each configuration is initialized from the same LingBot-Map stage-1 weights and trained in a single stage for 80,000 updates, using front-wide, front-tele, cross-left, and cross-right cameras. Data-to-task assignments and multi-task loss weights follow § 4; single-task runs use unit loss weight. OCC-only omits MLMCF, which lacks occupancy labels.

Training tasksDetectionOccupancyDepthTrajectoryRig pose
GEODETOCCTop-mAP ↑ (%)mIoU ↑ (%)AbsRel ↓ATE ↓ (cm)mAA@3 ↑ (%)RTE ↓ (cm)RRE ↓ (°)
✓\checkmark————0.1235.0037.097.530.898
—✓\checkmark—40.94——————
——✓\checkmark—11.00—————
✓\checkmark✓\checkmark—41.30—0.1274.3843.176.520.966
✓\checkmark—✓\checkmark—11.820.1244.7240.467.080.909
—✓\checkmark✓\checkmark41.6410.49—————
✓\checkmark✓\checkmark✓\checkmark42.1411.530.1274.5542.056.640.957

Table 16. Perception performance across training-task combinations. All configurations use four cameras and are trained for 80,000 updates.

All configurations share the same evaluation protocol for each task. We evaluate geometry on 301 CICD301 clips using the metrics defined in § 7.1, including rig mAA@3. Detection is evaluated on 300 class-stratified PlanNet validation clips using top-class mAP over automobile, person, rider, bus, and heavy truck, with 3D center-distance matching. For occupancy, we report empty-aware semantic mIoU on 99 NRE clips: empty voxels enter the confusion matrix, while the mean is taken over occupied semantic classes.

Findings

Detection benefits from co-training with geometry and occupancy, with top-class mAP increasing from 40.94% for DET-only to 42.14% when both tasks are included. Co-training with either task also improves detection. These consistent gains suggest that geometry and occupancy supervision help ground the shared representation in 3D structure, providing useful spatial cues for object detection.

Occupancy, in turn, benefits from co-training with geometry: mIoU rises from 11.00% to 11.82%. The two tasks describe complementary aspects of the same scene. Depth and pointmaps provide dense, pixel-aligned supervision of observed surfaces, while occupancy extends this reconstruction to volumetric scene completion. Learning these surface constraints may therefore improve the shared features used for occupancy prediction. Detection alone does not provide the same benefit: DET+OCC reaches 10.49% mIoU, whereas including geometry raises it to 11.53%.

Geometry also benefits in trajectory and rig translation accuracy. Co-training with detection reduces trajectory ATE from 5.00 to 4.38 cm and rig RTE from 7.53 to 6.52 cm, while achieving the highest rig mAA@3. The other combinations that include geometry improve these metrics as well. A plausible explanation is that

detection and occupancy supervision strengthen the shared scene features used by the pose heads. These gains come with a trade-off: GEO-only retains the lowest depth AbsRel and rig rotation error.

Runtime analysis

We profile the final unified model on one NVIDIA H100 80GB GPU with the mixed-precision settings in Sec. 4.3. All task heads run together on six-camera windows from one CICD301 scene, at 1036×10361036 \times1036 resolution and T∈{24,36,48}T \in\{24, 36, 48\} timestamps per camera. We report median forward times over five runs after two warmup iterations. Timings exclude data loading and preparation, Gaussian rasterization, losses, metrics, and output export. Peak memory refers to PyTorch allocated GPU memory.

As shown in Table 17, the 3DGS heads account for about one third of forward time, followed by the image encoder and geometry heads. Doubling the window from 24 to 48 timestamps roughly doubles total forward time, and peak allocated memory increases from 32.47 to 58.41 GiB.

ComponentT=24T = 24T=36T = 36T=48T = 48
Image encoder2.2183.3224.455
Spatiotemporal aggregator1.2482.1753.305
Geometry heads1.7862.6733.559
Shared DET/OCC neck0.0140.0210.028
Detection head0.0900.1280.168
Occupancy head41.0601.5762.104
3DGS heads (without rendering)3.2554.5786.155
Full forward (s)9.67314.48019.782
Peak allocated memory (GiB)32.4745.4458.41

Table 17. Forward runtime and peak memory of the unified model. Median times (s) are measured per six-camera window on one H100 80GB; TT counts timestamps per camera. Component medians need not sum to the independently measured full-forward median.

Application to lidarless NuRec

We apply Foundation4D to lidarless NuRec [6], using its camera trajectories, dense geometry, and actor tracks as priors for scene reconstruction. This application complements the feed-forward Gaussian decoder in § 3.9: NuRec optimizes a separate Gaussian representation for each scene against the recorded images.

Reconstruction workflow

The workflow in Figure 18 takes multi-camera video and vehicle GPS/IMU trajectories as recorded inputs. Video-level processing assembles poses and depth predicted from images by Foundation4D into camera trajectories and a fused point cloud, which initializes reconstruction geometry. Detection and tracking provide oriented cuboids with persistent actor identities, separating moving objects from the static scene and associating their observations over time. These outputs replace priors previously obtained from LiDAR or separate reconstruction and perception pipelines. NuRec uses them to optimize a dynamic Gaussian scene against RGB observations for novel-view rendering [85, 99].

Foundation4D in lidarless NuRec workflow and example depth, normal, and fused-geometry priors

Figure 18. Foundation4D in lidarless NuRec. Foundation4D and video-level processing supply the scene priors; NuRec then optimizes a Gaussian scene against the recorded RGB frames. Bottom: example depth, normal, and fused-geometry priors.

Reconstruction quality and efficiency

The quality of the input priors affects how NuRec assigns image observations to scene components: inaccurate poses or inconsistent actor tracks can lead to misplaced geometry and ghosting. The example in Figure 19 illustrates this connection between perception and reconstruction quality. Table 18 reports a reduction in end-to-end workflow turnaround from 16.0 to 3.7 hours per clip, a 4.3×4.3 \times speedup. Camera-pose and depth estimation with fusion also speed up because feed-forward prediction replaces separate iterative reconstruction stages. These measurements describe the overall workflow improvement and do not isolate individual model optimizations.

Detection and tracking priors and NuRec reconstruction, with and without Foundation4D

Figure 19. NuRec reconstruction with and without Foundation4D predictions as priors. Left: detection and tracking priors used for reconstruction; right: novel views of the optimized NuRec scene. The bottom row shows more consistent tracks and a more complete reconstruction of the crossing white vehicle. This comparison does not isolate individual model changes.

ScopeWithout Foundation4D (hours/clip)With Foundation4D (hours/clip)Speedup
End-to-end lidarless NuRec16.03.74.3×
Camera pose estimation3.50.48.8×
Depth estimation and fusion8.01.08.0×

Table 18. Workflow turnaround across releases. Times are hours per clip, including startup and teardown. Camera configurations, software, and hardware differ across releases. Rows describe separate processing scopes and are not an additive breakdown.

Limitations and open questions

Unified model.

Adding more tasks does not always improve performance. Geometry and occupancy trained together achieve 11.82% occupancy mIoU, compared with 11.53% for the three-task model that also includes detection (Table 16). Better task weighting and data sampling may help the model benefit more consistently from joint supervision, but testing new recipes is costly. For efficiency, our training schedule introduces Gaussian reconstruction after perception pretraining, deferring the cost of Gaussian decoding and rendering. Perception training took approximately 18 days, followed by another 9 days for Gaussian head training and joint fine-tuning. Although the final stage jointly optimizes the perception and Gaussian heads, we have not studied the benefits of training them together from the outset or systematically evaluated how geometry and rendering supervision affect each other.

Geometry.

Depth errors increase at long range and around object boundaries (Figure 6 and Figure 7). Distant objects and thin structures occupy few pixels, making their depth difficult to estimate reliably. Differences in camera capture times can also affect pose estimation: rig supervision can mix fixed camera calibration with ego motion between exposures. Interpolating the ego trajectory does not resolve this ambiguity.

Detection and tracking.

Detection errors in crowded scenes and at long range also affect tracking. Missed objects and temporary confidence drops interrupt the observations available for association, leading to fragmented trajectories. Reliable long-video tracking therefore requires both more stable detections and better recovery across gaps in the observations.

Semantic occupancy.

Occupancy accuracy decreases at long range, and rare classes are often missed or assigned incorrect semantic labels. Predictions also fluctuate between frames, particularly around object boundaries and thin structures. Improving the coverage of rare classes in the training data and encouraging consistent predictions across frames may help reduce these errors.

Conclusion

We presented Foundation4D, a unified model for perception and reconstruction from multi-camera driving video. Shared visual features support geometry, objects, semantic occupancy, and Gaussian reconstruction, while video-level alignment and tracking extend its predictions to long driving sequences. To support learning and evaluation, our annotation pipeline addresses sensor timing, motion, and visibility when generating dense depth and semantic occupancy labels. The released PAI–Foundation4D benchmark makes these annotations available for comparing geometry, detection, and occupancy methods on the same driving scenes.

Our evaluations show both the value of temporal observations and the limits of the current joint-training recipe. Longer input windows improve detection without retraining, and trajectory refinement improves pose estimation across windows. In our task-combination study, detection performs best with all three tasks, while occupancy performs best when paired with geometry. The predicted geometry and trajectories also provide priors for lidarless NuRec, demonstrating their use in reconstruction for simulation. Improving the balance between tasks and understanding the interaction between geometric and rendering supervision remain open directions.

Author list

Geometry: Qunjie Zhou, Shengyu Huang, Mingxuan Liu, Bingxin Ke, Fei Xue

Detection: Guillem Braso, Sérgio Agostinho, Tim Meinhardt, Ilya Fedorov, Cristiano Saltori, Mingxuan Liu

Occupancy: Cristiano Saltori, Tim Meinhardt, Bingxin Ke, Maximilian Apodaca, Gang Huang

Gaussian reconstruction: Ning Xu, Shengyu Huang, Jiahui Huang

Unified model: Shengyu Huang, Guillem Braso, Tim Meinhardt

Data: Bingxin Ke, Aljoša Ošep, Michele Fenzi, Shengyu Huang, Mingxuan Liu, Sérgio Agostinho, Jiahui Huang

Product team: Emmanuel Attia, George Dong, Jin-Bey Yu, Kefei Ren, Keqiang Wu, Kuan-Ting Lee, Naveen Kumar Rai, Ning Xu, Tian Liu

Product leadership: Lei Zhang

Leadership: Laura Leal-Taixé

Acknowledgements

We give special thanks to our product manager Martijn Tideman who worked tirelessly to help us bring this project to life. We thank Ning Xu and Jiaxin Zhang from the Dynamic autolabeling team for providing us with semantic occupancy annotations. We thank Hunter van Adelsberg and Xiangyu Chen for their contributions to the PAI dataset release. We thank Jiawei Ren, Zan Gojcic, Sanja Fidler, Nick Schneider and Matt Cragun, for the insightful discussions.

References

  1. [1]Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In CVPR, 2023. 3, 4arxiv.org/abs/2212.10156
  2. [2]Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. PARA-Drive: Parallelized architecture for real-time autonomous driving. In CVPR, 2024. 3, 4DOI
  3. [3]Longfei Xu, Xiaohui Wang, Zehao Huang, Han Li, Ya Yang, Naiyan Wang, and Si Liu. Geometry-Grounded Unified 3D Perception for Autonomous Driving. arXiv:2608.13147, 2026. 3, 4, 5, 20, 21, 27arxiv.org/abs/2608.13147
  4. [4]Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In CVPR, 2025. 3, 4, 5, 8, 25, 27arxiv.org/abs/2503.11651
  5. [5]NVIDIA. Instant NuRec: Feed-forward 3D gaussian reconstruction for driving scene simulation. Technical report, NVIDIA, 2026. arXiv:2607.14203. 3, 7, 12, 34
  6. [6]NVIDIA Corporation. NVIDIA Omniverse NuRec: How NuRec works, 2026. URL https://docs.nvidia.com/nurec/basics/how-nurec-works.html. Accessed October 15, 2026. 3, 4, 41
  7. [7]NVIDIA Corporation. Physical AI Autonomous Vehicles - Foundation4D dataset. Hugging Face, 2026. URL https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-Foundation4D. Accessed: 2026-10-19. 3, 17
  8. [8]NVIDIA Corporation. Physical AI Autonomous Vehicles dataset. Hugging Face, 2025. URL https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles. Accessed: 2026-10-15. 3, 17, 28DOI
  9. [9]Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020. 3, 20arxiv.org/abs/1912.04838
  10. [10]Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020. 3, 20, 27, 31arxiv.org/abs/1903.11027
  11. [11]Xiaofeng Wang, Zheng Zhu, Wenbo Xu, Yunpeng Zhang, Yi Wei, Xu Chi, Yun Ye, Dalong Du, Jiwen Lu, and Xingang Wang. OpenOccupancy: A large scale benchmark for surrounding semantic occupancy perception. In ICCV, 2023. 3, 6, 31, 32arxiv.org/abs/2303.03991
  12. [12]Ming Liang, Bin Yang, Wenyuan Zeng, Yun Chen, Rui Hu, Sergio Casas, and Raquel Urtasun. PnPNet: End-to-end perception and prediction with tracking in the loop. In CVPR, 2020. 4arxiv.org/abs/2005.14711
  13. [13]Junru Gu, Chenxu Hu, Tianyuan Zhang, Xuanyao Chen, Yilun Wang, Yue Wang, and Hang Zhao. ViP3D: End-to-end visual trajectory prediction via 3D agent queries. In CVPR, 2023. 4arxiv.org/abs/2208.01582
  14. [14]Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. BEVFormer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV, 2022. 4, 6arxiv.org/abs/2203.17270
  15. [15]Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. PETRv2: A unified framework for 3D perception from multi-camera images. In ICCV, 2023. 4, 5, 27arxiv.org/abs/2206.01256
  16. [16]Yu Hong, Qian Liu, Huayuan Cheng, Danjiao Ma, Hang Dai, Yu Wang, Guangzhi Cao, and Yong Ding. UniVision: A unified framework for vision-centric 3D perception. arXiv:2401.06994, 2024. 4arxiv.org/abs/2401.06994
  17. [17]Sergio Casas, Abbas Sadat, and Raquel Urtasun. MP3: A unified model to map, perceive, predict and plan. In CVPR, 2021. 4arxiv.org/abs/2101.06806
  18. [18]Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. ST-P3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In ECCV, 2022. 4arxiv.org/abs/2207.07601
  19. [19]Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. VAD: Vectorized scene representation for efficient autonomous driving. In ICCV, 2023. 4arxiv.org/abs/2303.12077
  20. [20]Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In ICML, 2020. 4arxiv.org/abs/1905.07553
  21. [21]Honghui Yang, Sha Zhang, Di Huang, Xiaoyang Wu, Haoyi Zhu, Tong He, Shixiang Tang, Hengshuang Zhao, Qibo Qiu, Binbin Lin, Xiaofei He, and Wanli Ouyang. UniPAD: A universal pre-training paradigm for autonomous driving. In CVPR, 2024. 4arxiv.org/abs/2310.08370
  22. [22]Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, Liping Jing, Yiming Nie, and Bin Dai. DriveWorld: 4D pre-trained scene understanding via world models for autonomous driving. In CVPR, 2024. 4arxiv.org/abs/2405.04390
  23. [23]Sicheng Zuo, Zixun Xie, Wenzhao Zheng, Shaoqing Xu, Fang Li, Hanbing Li, Long Chen, Zhi-Xin Yang, and Jiwen Lu. Dvgt-2: Vision geometry-action model for autonomous driving at scale. arXiv:2604.00813, 2026. 4, 5, 20, 21arxiv.org/abs/2604.00813
  24. [24]Li-Heng Chen, Haokai Pang, Chengye Su, Jiarun Liu, Qifeng Chen, Ziqian Ni, Jianxin Huang, Shi-Sheng Huang, Hongbo Fu, and Sheng Yang. USR-Drive: Unified driving scene representation via joint denoising of 3D gaussians and boxes. arXiv:2608.19036, 2026. 4arxiv.org/abs/2608.19036
  25. [25]Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUSt3R: Geometric 3D Vision Made Easy. In CVPR, 2024. 5arxiv.org/abs/2312.14132
  26. [26]Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding Image Matching in 3D with MASt3R. In ECCV, 2024. 5arxiv.org/abs/2406.09756
  27. [27]Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He. π³: Permutation-Equivariant Visual Geometry Learning. In ICLR, 2026. 5
  28. [28]Haotong Lin, Sili Chen, Jun Hao Liew, Donny Y. Chen, Zhenyu Li, Yang Zhao, Sida Peng, Hengkai Guo, Xiaowei Zhou, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth Anything 3: Recovering the Visual Space from Any Views. In ICLR, 2026. 5arxiv.org/abs/2511.10647
  29. [29]Wonbong Jang, Philippe Weinzaepfel, Vincent Leroy, Lourdes Agapito, and Jerome Revaud. Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors. In CVPR, 2025. 5arxiv.org/abs/2503.17316
  30. [30]Nikhil Keetha, Norman Müller, Johannes Schönberger, Lorenzo Porzi, Yuchen Zhang, Tobias Fischer, Arno Knaptisch, Duncan Zauss, Ethan Weber, Nelson Antunes, Jonathon Luiten, Manuel Lopez-Antequera, Samuel Rota Bulò, Christian Richardt, Deva Ramanan, Sebastian Scherer, and Peter Kontschieder. MapAnything: Universal Feed-Forward Metric 3D Reconstruction. In 3DV, 2026. 5arxiv.org/abs/2509.13414
  31. [31]Jianing Yang, Alexander Sax, Kevin J. Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass. In CVPR, 2025. 5arxiv.org/abs/2501.13928
  32. [32]Jianyuan Wang, Minghao Chen, Shangzhan Zhang, Nikita Karaev, Johannes Schönberger, Patrick Labatut, Piotr Bojanowski, David Novotny, Andrea Vedaldi, and Christian Rupprecht. VGGT-Ω. arXiv:2605.15195, 2026. 5
  33. [33]Alessandro Burzio, Tobias Fischer, Sven Elflein, Qunjie Zhou, Riccardo de Lutio, Jiawei Ren, Jiahui Huang, Shengyu Huang, Marc Pollefeys, Laura Leal-Taixé, Zan Gojcic, and Haithem Turki. Déjà View: Looping Transformers for Multi-View 3D Reconstruction. arXiv:2605.30215, 2026. 5arxiv.org/abs/2605.30215
  34. [34]Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Anjoo Kanazawa. Continuous 3D Perception Model with Persistent State. In CVPR, 2025. 5arxiv.org/abs/2501.12387
  35. [35]Dong Zhuo, Wenzhao Zheng, Jiahe Guo, Yuqi Wu, Jie Zhou, and Jiwen Lu. Streaming 4D Visual Geometry Transformer. In ICLR, 2026. 5arxiv.org/abs/2507.11539
  36. [36]Lin-Zhuo Chen, Jian Gao, Shangzhan Zhang, Yihang Chen, Ka Leong Cheng, Yipengjing Sun, Liangxiao Hu, Nan Xue, Xing Zhu, Yujun Shen, Yao Yao, and Yinghao Xu. LingBot-Map: Geometric Context Transformer for Streaming 3D Reconstruction. arXiv:2604.14141, 2026. 5arxiv.org/abs/2604.14141
  37. [37]Chuhan Zhang, Guillaume Le Moing, Skanda Koppula, Ignacio Rocco, Liliane Momeni, Junyu Xie, Shuyang Sun, Rahul Sukthankar, Joëlle K. Barral, Rajat Hadsell, Zoubin Ghahramani, Andrew Zisserman, Junlin Zhang, and Mehdi S. M. Sajjadi. Efficiently Reconstructing Dynamic Scenes One D4RT at a Time. In CVPR, 2026. 5arxiv.org/abs/2512.08924
  38. [38]Yihang Luo, Shangchen Zhou, Yushi Lan, Xingang Pan, and Chen Change Loy. 4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere. In ICML, 2026. 5arxiv.org/abs/2602.10094
  39. [39]Yanqin Jiang, Tengfei Wang, Zhengwei Wang, Chenjie Cao, Junta Wu, Wenhan Luo, Weiming Hu, Jin Gao, and Chunchao Guo. OmniX: Any-view and Any-time 4D Reconstruction via Feed-forward Trajectory Fields. In ECCV, 2026. 5arxiv.org/abs/2607.10840
  40. [40]Sicheng Zuo, Zixun Xie, Wenzhao Zheng, Shaoqing Xu, Fang Li, Shengyin Jiang, Long Chen, Zhi-Xin Yang, and Jiwen Lu. DVGT: Driving Visual Geometry Transformer. In CVPR, 2026. 5, 20, 21arxiv.org/abs/2512.16919
  41. [41]Samuel Li, Pujith Kachana, Prajwal Chidananda, Saurabh Nair, Yasutaka Furukawa, and Matthew A. Brown. Rig3R: Rig-Aware Conditioning and Discovery for 3D Reconstruction. In NeurIPS, 2025. 5DOI
  42. [42]Xiaosong Jia, Yanhao Liu, Yu Hong, Renqiu Xia, Junqi You, Bin Sun, Zhihui Hao, and Junchi Yan. DriveVGGT: Calibration-Constrained Visual Geometry Transformers for Multi-Camera Autonomous Driving. arXiv:2511.22264, 2026. 5arxiv.org/abs/2511.22264
  43. [43]Lizhou Liao, Wentao Xu, Handong Wang, Lirong Yang, Shuai Yang, Weiwei Liu, and Chang Huang. TRIG: Trajectory-Rig Decoupled Metric Geometry Learning. arXiv:2607.05801, 2026. 5arxiv.org/abs/2607.05801
  44. [44]Jiawei Lian, Haoyi Sun, Yang Wu, Lifu Mu, Siyuan Wang, Le Hui, Ning Mao, Tao Wei, Pan Zhou, Kun Zhan, and Jian Yang. LiAuto-GeoX: Efficient Grounded Driving Transformer. arXiv:2606.05774, 2026. 5, 20, 21arxiv.org/abs/2606.05774
  45. [45]Tianyi Yang, Haojun Yang, Jipeng Xiao, Xuesong Chen, Shoshuai Shi, Li Jiang, and Zhipeng Zhang. Drive3R: Block-wise streaming geometric foundation models for autonomous driving. arXiv:2610.15112, 2026. 5arxiv.org/abs/2610.15112
  46. [46]Kai Deng, Zexin Ti, Jiawei Xu, Jian Yang, and Jin Xie. VGGT-Long: Chunk it, Loop it, Align it – Pushing VGGT’s Limits on Kilometer-scale Long RGB Sequences. In ICRA, 2026. 5, 15arxiv.org/abs/2507.16443
  47. [47]Dominic Maggio, Hyungtae Lim, and Luca Carlone. VGGT-SLAM: Dense RGB SLAM Optimized on the SL(4) Manifold. In NeurIPS, 2025. 5, 15
  48. [48]Dominic Maggio and Luca Carlone. VGGT-SLAM 2.0: Real-time Dense Feed-forward Scene Reconstruction. arXiv:2601.19887, 2026. 5, 15arxiv.org/abs/2601.19887
  49. [49]Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 5arxiv.org/abs/2005.12872
  50. [50]Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In ECCV, 2022. 5, 10arxiv.org/abs/2203.05625
  51. [51]Changyong Shu, Jiajun Deng, Fisher Yu, and Yifan Liu. 3dppe: 3d point positional encoding for transformer-based multi-camera 3d object detection. In ICCV, 2023. 5, 27DOI
  52. [52]Xuewu Lin, Tianwei Lin, Zixiang Pei, Lichao Huang, and Zhizhong Su. Sparse4d: Multi-view 3d object detection with sparse spatial-temporal fusion. arXiv:2211.10581, 2022. 5arxiv.org/abs/2211.10581
  53. [53]Haisong Liu, Yao Teng, Tao Lu, Haiguang Wang, and Limin Wang. Sparsebev: High-performance sparse 3d object detection from multi-camera videos. In ICCV, 2023. 5, 27arxiv.org/abs/2308.09244
  54. [54]Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In ICCV, 2023. 5, 27arxiv.org/abs/2303.11926
  55. [55]Hang Ji, Tao Ni, Xufeng Huang, Zhan Shi, Tao Luo, Xin Zhan, and Junbo Chen. Ropetr: Improving temporal camera-only 3d detection by integrating enhanced rotary position embedding. arXiv:2504.12643, 2025. 5, 27arxiv.org/abs/2504.12643
  56. [56]Feng Liu, Tengteng Huang, Qianjing Zhang, Haotian Yao, Chi Zhang, Fang Wang, Qixiang Ye, and Yanzhao Zhou. Ray denoising: Depth-aware hard negative sampling for multi-view 3d object detection. In ECCV, 2024. 5, 27arxiv.org/abs/2402.03634
  57. [57]Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M. Ni, and Lei Zhang. DN-DETR: Accelerate DETR training by introducing query denoising. In CVPR, 2022. 5, 11
  58. [58]Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable transformers for end-to-end object detection. In ICLR, 2021. 5, 11arxiv.org/abs/2010.04159
  59. [59]Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In ICIP, 2016. 6DOI
  60. [60]Xinshuo Weng, Jianren Wang, David Held, and Kris Kitani. 3D multi-object tracking: A baseline and new evaluation metrics. In IROS, 2020. 6arxiv.org/abs/1907.03961
  61. [61]Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Center-based 3D Object Detection and Tracking. In CVPR, 2021. 6arxiv.org/abs/2006.11275
  62. [62]Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixé, and Christoph Feichtenhofer. TrackFormer: Multi-object tracking with transformers. In CVPR, 2022. 6arxiv.org/abs/2101.02702
  63. [63]Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. MOTR: End-to-end multiple object tracking with transformer. In ECCV, 2022. 6arxiv.org/abs/2105.03247
  64. [64]Tianyuan Zhang, Xuanyao Chen, Yue Wang, Yilun Wang, and Hang Zhao. MUTR3D: A multi-camera tracking framework via 3D-to-2D queries. In CVPR Workshop, 2022. 6arxiv.org/abs/2205.00613
  65. [65]Ziqi Pang, Jie Li, Pavel Tokmakov, Dian Chen, Sergey Zagoruyko, and Yu-Xiong Wang. Standing between past and future: Spatio-temporal modeling for multi-camera 3D multi-object tracking. In CVPR, 2023. 6arxiv.org/abs/2302.03802
  66. [66]Guillem Brasó and Laura Leal-Taixé. Learning a neural solver for multiple object tracking. In CVPR, 2020. 6arxiv.org/abs/1912.07515
  67. [67]Orçun Cetintas, Guillem Brasó, and Laura Leal-Taixé. Unifying short and long-term tracking with graph hierarchies. In CVPR, 2023. 6arxiv.org/abs/2212.03038
  68. [68]Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3D semantic occupancy prediction. In CVPR, 2023. 6, 32arxiv.org/abs/2302.07817
  69. [69]Yuanhui Huang, Wenzhao Zheng, Borui Zhang, Jie Zhou, and Jiwen Lu. SelfOcc: Self-supervised vision-based 3D occupancy prediction. In CVPR, 2024. 6arxiv.org/abs/2311.12754
  70. [70]Antonin Vobecky, Oriane Siméoni, David Hurych, Spyridon Gidaris, Andrei Bursuc, Patrick Pérez, and Josef Sivic. POP-3D: Open-vocabulary 3D occupancy prediction from images. In NeurIPS, 2023. 6arxiv.org/abs/2401.09413
  71. [71]Anh-Quan Cao and Raoul de Charette. MonoScene: Monocular 3D semantic scene completion. In CVPR, 2022. 6, 32arxiv.org/abs/2112.00726
  72. [72]Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. SurroundOcc: Multi-camera 3D occupancy prediction for autonomous driving. In ICCV, 2023. 6arxiv.org/abs/2303.09551
  73. [73]Yunpeng Zhang, Zheng Zhu, and Dalong Du. OccFormer: Dual-path transformer for vision-based 3D semantic occupancy prediction. In ICCV, 2023. 6arxiv.org/abs/2304.05316
  74. [74]Heng Li, Yuenan Hou, Xiaohan Xing, Yuexin Ma, Xiao Sun, and Yanyong Zhang. OccMamba: Semantic occupancy prediction with state space models. In CVPR, 2025. 6, 32arxiv.org/abs/2408.09859
  75. [75]Guoqing Wang, Zhongdao Wang, Pin Tang, Jilai Zheng, Xiangxuan Ren, Bailan Feng, and Chao Ma. OccGen: Generative multi-modal 3D occupancy prediction for autonomous driving. In ECCV, 2024. 6, 32arxiv.org/abs/2404.15014
  76. [76]Yuqi Wang, Yuntao Chen, Xingyu Liao, Lue Fan, and Zhaoxiang Zhang. PanoOcc: Unified occupancy representation for camera-based 3D panoptic segmentation. In CVPR, 2024. 6arxiv.org/abs/2306.10013
  77. [77]Qihang Ma, Xin Tan, Yanyun Qu, Lizhuang Ma, Zhizhong Zhang, and Yuan Xie. COTR: Compact occupancy Transformer for vision-based 3D occupancy prediction. In CVPR, 2024. 6arxiv.org/abs/2312.01919
  78. [78]Pin Tang, Zhongdao Wang, Guoqing Wang, Jilai Zheng, Xiangxuan Ren, Bailan Feng, and Chao Ma. SparseOcc: Rethinking sparse latent representation for vision-based semantic occupancy prediction. In CVPR, 2024. 6, 32arxiv.org/abs/2404.09502
  79. [79]Jiabaо Wang, Zhaojiang Liu, Qiang Meng, Liujiang Yan, Ke Wang, Jie Yang, Wei Liu, Qibin Hou, and Ming-Ming Cheng. OPUS: Occupancy prediction using a sparse set. In NeurIPS, 2024. 6arxiv.org/abs/2409.09350
  80. [80]Qingju Guo, Shuang Li, Binhui Xie, Jing Geng, and Wei Li. SPSC: Sparse and scalable multi-modal 3D occupancy prediction for autonomous driving. In AAAI, 2026. 6, 32DOI
  81. [81]Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. GaussianFormer: Scene as gaussians for vision-based 3D semantic occupancy prediction. In ECCV, 2024. 6arxiv.org/abs/2405.17429
  82. [82]Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M. Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. VoxFormer: Sparse voxel transformer for camera-based 3D semantic scene completion. In CVPR, 2023. 6arxiv.org/abs/2302.12251
  83. [83]Haoyi Jiang, Tianheng Cheng, Naiyu Gao, Haoyang Zhang, Tianwei Lin, Wenyu Liu, and Xinggang Wang. Symphonize 3D semantic scene completion with contextual instance queries. In CVPR, 2024. 6arxiv.org/abs/2306.15670
  84. [84]Zhu Yu, Runmin Zhang, Jiacheng Ying, Junchen Yu, Xiaohai Hu, Lun Luo, Si-Yuan Cao, and Hui-Liang Shen. Context and geometry aware voxel transformer for semantic scene completion. In NeurIPS, 2024. 6arxiv.org/abs/2405.13675
  85. [85]Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D gaussian splatting for real-time radiance field rendering. ACM TOG, 2023. 6, 41arxiv.org/abs/2308.04079
  86. [86]David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelSplat: 3D Gaussian Splats from image pairs for scalable generalizable 3D reconstruction. In CVPR, 2024. 6arxiv.org/abs/2312.12337
  87. [87]Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. MVSplat: Efficient 3D Gaussian Splatting from sparse multi-view images. In ECCV, 2024. 6DOI
  88. [88]Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. GS-LRM: Large reconstruction model for 3D Gaussian Splatting. In ECCV, 2024. 6arxiv.org/abs/2404.19702
  89. [89]Haofei Xu, Songyou Peng, Fangjinhua Wang, Hermann Blum, Daniel Barath, Andreas Geiger, and Marc Pollefeys. DepthSplat: Connecting Gaussian splatting and depth. In CVPR, 2025. 6arxiv.org/abs/2410.13862
  90. [90]Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Marc Pollefeys, Ming-Hsuan Yang, and Songyou Peng. No pose, no problem: Surprisingly simple 3D Gaussian splats from sparse unposed images. In ICLR, 2025. 6arxiv.org/abs/2410.24207
  91. [91]Lihan Jiang, Yucheng Mao, Linning Xu, Tao Lu, Kerui Ren, Yichen Jin, Xudong Xu, Mulin Yu, Jiangmiao Pang, Feng Zhao, Dahua Lin, and Bo Dai. AnySplat: Feed-forward 3D Gaussian splatting from unconstrained views. ACM TOG, 2025. 6
  92. [92]Jiawei Ren, Michal Jan Tyszkiewicz, Jiahui Huang, and Zan Gojcic. TokenGS: Decoupling 3D Gaussian prediction from pixels with learnable tokens. In CVPR, 2026. 6arxiv.org/abs/2604.15239
  93. [93]Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. DrivingGaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In CVPR, 2024. 7arxiv.org/abs/2312.07920
  94. [94]Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street Gaussians: Modeling dynamic urban scenes with Gaussian Splatting. In ECCV, 2024. 7arxiv.org/abs/2401.01339
  95. [95]Ziyu Chen, Jiawei Yang, Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojcic, Sanja Fidler, Marco Pavone, Li Song, and Yue Wang. OmniRe: Omni urban scene reconstruction. In ICLR, 2025. 7arxiv.org/abs/2408.16760
  96. [96]Jiawei Yang, Jiahui Huang, Yuxiao Chen, Yan Wang, Boyi Li, Yurong You, Maximilian Igl, Apoorva Sharma, Peter Karkus, Danfei Xu, Boris Ivanovic, Yue Wang, and Marco Pavone. STORM: Spatio-temporal reconstruction model for large-scale outdoor scenes. In ICLR, 2025. 7arxiv.org/abs/2501.00602
  97. [97]Qijian Tian, Xin Tan, Yuan Xie, and Lizhuang Ma. DrivingForward: Feed-forward 3D Gaussian splatting for driving scene reconstruction from flexible surround-view input. In AAAI, 2025. 7
  98. [98]Xiaoxue Chen, Ziyi Xiong, Yuantao Chen, Gen Li, Nan Wang, Hongcheng Luo, Long Chen, Haiyang Sun, Bing Wang, Guang Chen, Hongyang Li, Ya-Qin Zhang, Hangjun Yan, and Hao Zhao. DGGT: Feedforward 4D reconstruction of dynamic driving scenes using unposed images. In CVPR, 2026. 7arxiv.org/abs/2512.03004
  99. [99]Qi Wu, Janick Martinez Esturo, Ashkan Mirzaei, Nicolas Möenne-Loccoz, and Zan Gojcic. 3DGUT: Enabling distorted cameras and secondary rays in Gaussian Splatting. In CVPR, 2025. 7, 41arxiv.org/abs/2412.12507
  100. [100]Maxime Oquab, Timothée Darcet, Théo Moutakanni, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2024. 8arxiv.org/abs/2304.07193
  101. [101]René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, 2021. 9, 12arxiv.org/abs/2103.13413
  102. [102]Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. DETR3D: 3d object detection from multi-view images via 3d-to-2d queries. In CoRL, 2021. 10arxiv.org/abs/2110.06922
  103. [103]Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. YOLOX: Exceeding YOLO series in 2021. arXiv:2107.08430, 2021. 11arxiv.org/abs/2107.08430
  104. [104]Zichen Yu, Changyong Shu, Jiajun Deng, Kangjie Lu, Zongdai Liu, Jiangyong Yu, Dawei Yang, Hui Li, and Yan Chen. FlashOcc: Fast and memory-efficient occupancy prediction via channel-to-height plugin. arXiv:2311.12058, 2023. 11arxiv.org/abs/2311.12058
  105. [105]NVIDIA Corporation. NCore: Sensor models. Online documentation, 2026. URL https://nvidia.github.io/ncore/data/sensor_models.html. Accessed: 2026-10-20. 14
  106. [106]Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. In NeurIPS, 2024. 15arxiv.org/abs/2407.08608
  107. [107]Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024. 15arxiv.org/abs/2104.09864
  108. [108]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv:2408.00714, 2024. 17arxiv.org/abs/2408.00714
  109. [109]Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv:2508.10104, 2025. 27arxiv.org/abs/2508.10104
  110. [110]Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe. Hota: A higher order metric for evaluating multi-object tracking. IJCV, 2021. 28DOI
  111. [111]Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D. In ECCV, 2020. 34arxiv.org/abs/2008.05711

Paper details

vlnc:W_cwjsjqv5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents