betaFeedback

WorksComputer Science

OCLDA-GS: Object-Centric Local Deformation-Aware Gaussian Streaming

Abstract

Streaming 3D Gaussian Splatting (3DGS) allows for online reconstruction of dynamic scenes from multi-view videos for Free Viewpoint Video (FVV). Current time and storage efficient methods rely on selecting a subset of anchor Gaussians and neighborhood interpolation for fast streamable 3DGS. However, these methods can overlook object-specific motion and introduce inconsistencies near object boundaries. We propose OCLDA-GS, an object-centric streaming framework that dynamically reallocates object and motion-aware Gaussian anchors to allocate anchor Gaussian support for objects undergoing large motion while at the same time, restricting the gaussian neighborhood within object groups for more precise object boundaries. Furthermore, it estimates object-wise rigid transformations (e.g. a hand moving) and leverages these estimates to learn per-anchor local deformation correction for small deformations (e.g. wrinkles on a shirt). Then, we conduct experiments on three dynamic multi-view video datasets and demonstrate improvements over the previous baselines alongside improved quality between refinement timesteps. Under matched refinement budgets, our method improves PSNR over Instant Gaussian Stream (IGS) by 0.169 dB on N3DV, 0.202 dB on Meeting Room and +0.199 on Google Immersive, while reducing the mean PSNR drop between refinement timesteps by 9.2% and 13.8% in N3DV and Meeting Room, respectively.

Introduction

3D Gaussian Splatting (3DGS) [13] allows for photorealistic reconstruction from RGB images and camera poses using 3D Gaussians as primitives. 4D Gaussian Splatting (4DGS) or Dynamic Gaussian Splatting adds spatiotemporal deformation in 3DGS. This enables realtime Free Viewpoint Video (FVV) for novel view synthesis [20, 33, 37]. FVV has applications in VR, immersive communication and sports viewing [1, 35]. Current Dynamic Gaussian Splatting methods can be divided into two broad categories: offline and online [8, 27, 33]. Offline methods optimize spatiotemporal deformation by learning the motion features throughout the entire video and emphasizes quality and refinement taking significant time and compute in the process [36, 37]. In the online streaming setting, the scene representation is updated as each set of new multi-view frames arrive and the central concern is to make that update computationally cheap while maintaining image fidelity [8, 27].

Recent time and storage-efficient Gaussian streaming methods rely on selection of subset of Gaussians called anchors, whose motion features are then interpolated to update the scene representation. Methods like IGS [35] and ReconGS [6] work under fixed computational budget controlled by the number of anchors and refinement per step or keyframes making them extremely effective for real-time dynamic scene streaming under limited computation bandwidth. IGS performs farthest point sampling (FPS) and periodic refinement on pre-specified timesteps called keyframes, while ReConGS [6] organizes anchors into an adaptive hierarchy for compact motion optimization and performs per-step refinements.

These methods assume neighboring Gaussians share the 3D deformation trajectories of each other. [11, 26] However, this assumption fails for Gaussians near object boundaries. For example, in the N3DV dataset (Figure 1a), when the chef’s hand moves near a stationary table, deformation via interpolated anchor neighborhood can induce incompatible motion in Gaussians belonging to the stationary table.

Object-centric anchors for streaming reconstruction

Figure 1. Object-centric anchors for streaming reconstruction. (a) Selected N3DV renders with dataset-average PSNR (dB). Below, anchors selected by respective methods are shown, colors show object groups. (b) Quality versus reconstruction time (in seconds per frame) in the N3DV dataset.

Existing anchor selection methods like FPS or hierarchical sampling also may not reflect the motion distribution of the scene accurately. Under a limited anchor budget, these methods can allocate excessive support to stationary regions. Some related object and motion-aware control-point methods address parts of this challenge [4, 9], but maintaining accurate, object-consistent motion with limited refinement remains challenging. Furthermore, motion within an object often comprises a shared rigid component and local non-rigid deformations (e.g., a moving body with wrinkling clothes). Accounting for both components is important for accurate motion prediction under limited refinement budgets.

We introduce Object-Centric Local Deformation-Aware Gaussian Streaming (OCLDA-GS). OCLDA-GS jointly addresses anchor selection, object-aware motion propagation, and local deformation correction. First, OCLDA-GS uses optical flow [28] observed over each completed keyframe interval to redistribute a fixed anchor budget across object groups, increasing support for groups with larger observed motion. Second, it restricts motion propagation to anchors within the same group, preventing cross-group interpolation for supported Gaussians. Finally, it fits a rigid transformation to the predicted anchor correspondences of each group and predicts an additional local deformation correction.

More broadly, OCLDA-GS highlights object-aware motion modeling as a promising direction for improving the quality–efficiency trade-off in Gaussian streaming. Extensive experiments on three datasets demonstrate competitive reconstruction quality and streaming efficiency in both in-domain and cross-domain settings. Temporal analyses against online streaming method like Instant Gaussian Stream (IGS) further show reduced quality degradation between keyframes. In summary, our contributions are:

  • Flow-guided anchor selection: We dynamically redistribute a fixed anchor budget across object groups based on observed optical flow, providing greater motion support for objects undergoing larger motion.

  • Object-constrained motion propagation: We restrict anchor-based motion interpolation to within object groups, preventing incompatible motion from propagating across object boundaries.

  • Rigid-conditioned deformation correction: We leverage object-wise rigid motion estimates to learn per-anchor local deformation corrections, accounting for local non-rigid deformations within objects without additional refinement.

Related Work

Dynamic novel view synthesis. NeRF established neural radiance fields as a representation for novel view synthesis [21]. Dynamic extensions model scene changes through deformation fields, scene flow, and higher-dimensional embeddings [18, 22, 23, 24]. HexPlane and K-Planes improve efficiency through factorized spatiotemporal features [2, 5]. With an explicit Gaussian representation and differentiable rasterization, 3DGS enables high-quality real-time rendering [13]. Dynamic Gaussian methods extend this representation using learned deformation fields and spatiotemporal Gaussian formulations [19, 33, 36, 37]. These advances improve dynamic scene representation and rendering, while online reconstruction additionally requires efficient updates as new observations arrive.

Efficient Streaming. StreamRF and ReRF investigate incremental learning and residual representations for dynamic radiance fields [16, 29], while NeRFPlayer and VideoRF explore decomposed fields and feature streams for compact video representation and playback [25, 30]. For Gaussian-based online reconstruction, 3DGStream trains scene updates on the fly [27], and QUEEN uses sparse, quantized Gaussian attribute residuals [8]. Complementary compression approaches, including 4DGC and GIFStream, target compact representations and efficient video delivery [10, 15]. HiCoM and ReCon-GS use hierarchical motion representations to reduce update and storage costs [6, 7]. IGS instead employs a generalizable anchor-driven motion predictor with periodic Gaussian refinement at keyframes [35]. OCLDA-GS builds on this inference-based setting to improve motion support and Gaussian deformation.

Structured motion and object-aware deformation. Embedded deformation represents motion through blended local transformations [26], and SC-GS uses sparse control points with learned interpolation and local rigidity regularization [11]. Dynamic 3D Gaussians encourage coherent tracking through local motion constraints [20]. MotionGS incorporates optical-flow-based motion guidance, while MoDec-GS uses global-to-local motion decomposition [14, 38]. Closer to object-aware streaming, H3D-DGS combines object-wise motion processing with flow-assisted control points [9], and ComGS uses motion-sensitive keypoints and a motion-aware influence field [4]. Building on these ideas, OCLDA-GS combines motion-aware anchor allocation, explicit group-restricted propagation, and learned rigid-motion-conditioned correction within keyframe-based streaming.

Method

We introduce Object-Centric Local Deformation-Aware Gaussian Streaming (OCLDA-GS). We present a visual overview of our method in Figure 2 and describe it component by component in the following subsections.

Figure 2. Overview of our method. (a) Flow-guided anchor selection within object groups. (b) Object-wise rigid motion estimation from AGM-Net predictions. (c) Learned local deformation corrections propagated to Gaussians within each object. (d) Improved rendering quality compared to Farthest Point Sampling (FPS)-based IGS.

Preliminaries. We follow the paradigm of keyframe to target frame prediction for streaming [35]. This paradigm propagates Gaussian motion from the latest refined keyframe kk to a target frame tt using a sparse set of Gaussian anchors. Predicted Gaussian representation is refined periodically at keyframes, thereby removing computation for per-frame refinement. Let Gk={Gik}i=1N\mathcal{G}^{k} = \{G_i^k\}_{i=1}^{N} denote the Gaussian representation at keyframe kk, where each Gaussian is parameterized by its 3D center μik∈R3\boldsymbol{\mu}_i^k \in\mathbb{R}^3, orientation quaternion qik∈R4\mathbf{q}_i^k \in\mathbb{R}^4, and other geometric and appearance attributes.

Let M∈GkM \in\mathcal{G}^{k} anchor Gaussians be indexed by J={1,…,M}\mathcal{J} = \{1,\ldots,M\}. Let ρ(j)\rho(j) map anchor jj to its corresponding Gaussian index, such that its source position is ajk=μρ(j)k\mathbf{a}_j^k = \boldsymbol{\mu}_{\rho(j)}^k. Given the target frame tt, a pretrained optical flow conditioned model (Ψϕ(⋅))(\Psi_\phi(\cdot)) predicts motion features for the selected anchors, which are interpolated to obtain per-Gaussian features fik→t\mathbf{f}_i^{k\to t}. A linear motion head hh decodes the translation and rotation updates:

(Δμik→t,Δqik→t)=h(fik→t),μ^it=μik+Δμik→t,q^it=norm⁡(qik⊗Δqik→t).(1)\begin{aligned} (\Delta\boldsymbol{\mu}_i^{k\to t},\Delta\mathbf{q}_i^{k\to t}) &= h(\mathbf{f}_i^{k\to t}),\\ \hat{\boldsymbol{\mu}}_i^t &= \boldsymbol{\mu}_i^k+\Delta\boldsymbol{\mu}_i^{k\to t},\\ \hat{\mathbf{q}}_i^t &= \operatorname{norm}(\mathbf{q}_i^k\otimes\Delta\mathbf{q}_i^{k\to t}). \tag*{(1)} \end{aligned}

Here, ⊗\otimes denotes quaternion multiplication and norm denotes quaternion normalization. The predicted target position of anchor jj is a^jt=μ^ρ(j)t\hat{\mathbf{a}}_j^t=\hat{\boldsymbol{\mu}}_{\rho(j)}^t.

Object Groups. We first obtain object-level segmentation of the Gaussian representation. Let si\mathbf{s}_i denote the object semantic embedding feature of Gaussian ii, and let co\mathbf{c}_o denote the mean semantic embedding of anchors in Ao\mathcal{A}_o. Each Gaussian GikG_i^k is assigned an object-group label gi∈{−1,0,…,O−1}g_i \in\{-1,0,\ldots,O-1\}, where OO denotes the number of identified objects and −1-1 indicates an unknown group. Each anchor ajka_j^k has object-group label gρ(j)g_{\rho(j)}. The anchor indices belonging to object oo are

Ao={j∈J∣gρ(j)=o}.(2)\mathcal{A}_o=\{j\in\mathcal{J}\mid g_{\rho(j)}=o\}. \tag*{(2)}

Anchor Reselection. We address the problem of anchor allocation on static regions by periodic anchor redistribution during keyframe refinement. We redistribute the fixed anchor budget MM across object groups based on their size and observed motion. For each group oo, we estimate a normalized motion score ωok∈[0,1]\omega_o^k\in[0,1] from optical flow over the preceding refinement interval, compensating for global image motion and filtering unreliable correspondences. Specifically, let do,vkd_{o,v}^k be the 75th percentile of flow magnitudes within group oo in camera view vv, after subtracting the component-wise median flow over reliable image pixels. We compute

rok=median⁡v∈Vok[[do,vk−τ]+Dv],ωok=min⁡(1,rokQ0.95({rgk:rgk>0})).(3)\begin{aligned} r_o^k &= \operatorname{median}_{v\in\mathcal{V}_o^k}\left[\frac{[d_{o,v}^k-\tau]_+}{D_v}\right],\\ \omega_o^k &= \min\left(1,\frac{r_o^k}{Q_{0.95}(\{r_g^k:r_g^k>0\})}\right). \tag*{(3)} \end{aligned}

where DvD_v is the image diagonal, τ=0.5\tau=0.5 pixels, [x]+=max⁡(x,0)[x]_+=\max(x,0), and Q0.95Q_{0.95} denotes the 95th percentile. The set Vok\mathcal{V}_o^k includes input views with at least eight distinct reliable, visible projected pixels for the group. Let NokN_o^k denote the number of Gaussians in group oo. We assign each group an allocation weight

βok=Nok(1+ωok).(4)\beta_o^k=\sqrt{N_o^k(1+\omega_o^k)}. \tag*{(4)}

The anchor budget is distributed proportionally to these weights, subject to group population limits. Within each group, anchors are selected using 3D farthest-point sampling on the refined Gaussian centers.

Object-level Rigid Motion Estimation. For each object oo, we estimate a rigid transformation that best approximates its motion predicted by the frozen optical flow model. Let Qk\mathcal{Q}_k denote the Gaussians which participate in deformation at keyframe kk. We consider the subset of object anchors Fo={j∈Ao∣ρ(j)∈Qk}\mathcal{F}_o=\{j\in\mathcal{A}_o\mid\rho(j)\in\mathcal{Q}_k\}. Using the source and predicted anchor positions, we estimate an object-level rotation Ro∈SO(3)\mathbf{R}_o\in SO(3) and translation to∈R3\mathbf{t}_o\in\mathbb{R}^3 by solving

(Ro,to)=arg⁡min⁡R∈SO(3), t∈R3∑j∈Fo∥Rajk+t−a^jt∥22.(5)(\mathbf{R}_o,\mathbf{t}_o)=\underset{\mathbf{R}\in SO(3),\,\mathbf{t}\in\mathbb{R}^3}{\arg\min}\sum_{j\in\mathcal{F}_o}\left\|\mathbf{R}\mathbf{a}_j^k+\mathbf{t}-\hat{\mathbf{a}}_j^t\right\|_2^2. \tag*{(5)}

This least-squares problem is solved analytically using the Kabsch algorithm [12]. The resulting transformation approximates the dominant rigid motion of each object.

Local Deformation. Given the estimated rigid transformation (Ro,to)(R_o,t_o), we compute the difference between the pretrained optical flow predicted anchor positions at frame tt and those obtained by applying the fitted rigid transformation to their positions at keyframe kk. For each anchor j∈Aoj \in A_o,

djk→t=RoT(a^jt−(Roajk+to)).(6)\mathbf{d}_j^{k\to t} = R_o^{\mathsf{T}}\left(\hat{\mathbf{a}}_j^t - (R_o\mathbf{a}_j^k + \mathbf{t}_o)\right). \tag*{(6)}

The inverse rotation expresses this displacement in the source-frame axes.

We define the centered anchor position as pj=ajk−aˉok\mathbf{p}_j = \mathbf{a}_j^k - \bar{\mathbf{a}}_o^k. We condition a shared residual network on the frozen optical flow model (Ψϕ)(\Psi_\phi) features, anchor motion feature, encoded anchor position, rigid-relative displacement, semantic features, and temporal offset:

zjk→t=(fjk→t,ϕ(pj),djk→t,sρ(j),co,τk→t).(7)\mathbf{z}_j^{k\to t} = \left(\mathbf{f}_j^{k\to t},\phi(\mathbf{p}_j),\mathbf{d}_j^{k\to t},\mathbf{s}_{\rho(j)},\mathbf{c}_o,\tau^{k\to t}\right). \tag*{(7)}

where ϕ\phi denotes positional encoding and τk→t\tau^{k\to t} is the normalized temporal offset.

The residual network DθD_\theta predicts an anchor-level translation correction:

δjk→t={Dθ(zjk→t),j∈Fo,0,otherwise.(8)\boldsymbol{\delta}_j^{k\to t} = \begin{cases} D_\theta(\mathbf{z}_j^{k\to t}), & j \in F_o,\\ 0, & \text{otherwise}. \end{cases} \tag*{(8)}

Motion and Correction Propagation. To prevent cross-object motion mixing, we restrict the anchors contributing to fik→t\mathbf{f}_i^{k\to t} in Eq. (1) to those belonging to the same object group as Gaussian GikG_i^k. For each Gaussian with a known group label gig_i, we define its neighborhood as the eight nearest same-group anchors:

N8(i)=KNN⁡8(μik,{ajk∣j∈Agi}).(9)\mathcal{N}_8(i) = \operatorname{KNN}_8\left(\boldsymbol{\mu}_i^k,\{\mathbf{a}_j^k \mid j \in A_{g_i}\}\right). \tag*{(9)}

We assign larger weights to anchors closer to the Gaussian and normalize them within N8(i)\mathcal{N}_8(i):

wij=exp⁡(−∥μik−ajk∥2)∑l∈N8(i)exp⁡(−∥μik−alk∥2).(10)w_{ij} = \frac{\exp\left(-\lVert\boldsymbol{\mu}_i^k-\mathbf{a}_j^k\rVert_2\right)}{\sum_{l\in\mathcal{N}_8(i)}\exp\left(-\lVert\boldsymbol{\mu}_i^k-\mathbf{a}_l^k\rVert_2\right)}. \tag*{(10)}

These weights interpolate the anchor motion features to obtain fik→t\mathbf{f}_i^{k\to t} in Eq. (1), and propagate the learned corrections:

δik→t,loc=∑j∈N8(i)wijδjk→t.(11)\boldsymbol{\delta}_i^{k\to t,\mathrm{loc}} = \sum_{j\in\mathcal{N}_8(i)} w_{ij}\boldsymbol{\delta}_j^{k\to t}. \tag*{(11)}

For supported Gaussians i∈Qki \in Q_k, we rotate the correction into world coordinates and add it to the position predicted by Ψϕ\Psi_\phi:

μit,pred=μ^it+Rgiδik→t,loc.(12)\boldsymbol{\mu}_i^{t,\mathrm{pred}} = \hat{\boldsymbol{\mu}}_i^t + R_{g_i}\boldsymbol{\delta}_i^{k\to t,\mathrm{loc}}. \tag*{(12)}

Training Objective

We train the residual network using a photometric reconstruction loss with magnitude and smoothness regularization:

L=Lphoto+λmagLmag+λsmoothLsmooth.(13)\mathcal{L}=\mathcal{L}_{\mathrm{photo}}+\lambda_{\mathrm{mag}}\mathcal{L}_{\mathrm{mag}}+\lambda_{\mathrm{smooth}}\mathcal{L}_{\mathrm{smooth}}. \tag*{(13)}

Let I^\hat{I} denote the rendered image of the corrected Gaussian representation and II the ground-truth image. The photometric loss is

Lphoto=α∥I^−I∥1+(1−α)(1−SSIM⁡(I^,I)).(14)\mathcal{L}_{\mathrm{photo}}=\alpha\lVert\hat{I}-I\rVert_{1}+(1-\alpha)(1-\operatorname{SSIM}(\hat{I},I)). \tag*{(14)}

For the eligible anchor set F=⋃oFo\mathcal{F}=\bigcup_o\mathcal{F}_o, let uj=δjk→t/ε\mathbf{u}_j=\delta_j^{k\to t}/\varepsilon. The regularization terms are

Lmag=1∣F∣∑j∈F∥uj∥22,(15)\mathcal{L}_{\mathrm{mag}}=\frac{1}{\lvert\mathcal{F}\rvert}\sum_{j\in\mathcal{F}}\lVert\mathbf{u}_j\rVert_2^2, \tag*{(15)}
Lsmooth=1Z∑(j,l)∈Eγjl∥uj−ul∥22,\mathcal{L}_{\mathrm{smooth}}=\frac{1}{Z}\sum_{(j,l)\in\mathcal{E}}\gamma_{jl}\lVert\mathbf{u}_j-\mathbf{u}_l\rVert_2^2,

where E\mathcal{E} connects neighboring same-object anchors, γjl\gamma_{jl} denotes their spatial-semantic affinity, and Z=max⁡(∑(j,l)∈Eγjl,η)Z=\max\left(\sum_{(j,l)\in\mathcal{E}}\gamma_{jl},\eta\right) is the normalization factor.

Experiments

Configuration and Implementation Details

Datasets. We evaluate on three datasets: N3DV [17], Meeting Room [16], and Google Immersive [1]. We train the residual heads on four N3DV scenes (Coffee Martini, Cook Spinach, Flame Steak, and Flame Salmon) and two Google Immersive scenes (Dog and Truck). Cut Roasted Beef and Sear Steak from N3DV, and Welder and Alexa Meade Face Paint 1 from Google Immersive, are held out for testing. Cross-dataset evaluation uses Discussion and Trimming from Meeting Room.

Baselines. We compare against four online methods: IGS [35], ReConGS [6], 3DGStream [27], and QUEEN [8]. We reproduce these methods using their released implementations and evaluate them on common held-out scenes, cameras, and frames. For context, we also include four offline methods: Real-Time 4DGS [37], 4D-GS [33], STG [19], and Swift4D [34]. Offline rows are shaded gray and serve as contextual references.

Implementation Details. Our implementation uses PyTorch and runs on an NVIDIA A100 GPU with 80 GB memory. Ψϕ\Psi_{\phi} is the frozen AGM-Net from IGS [35]. It receives four input views, and we use M=8192M=8192 motion anchors with eight same-group neighbors per Gaussian, as defined in Eq. (9). The residual network DθD_{\theta} contains two hidden layers of width 256 with SiLU activations. We use SAGA [3] to segment the Gaussian splat at timestep 0. Details on SAGA grouping are provided in the supplementary. We optimize Eq. (13) using AdamW with a learning rate of 4×10−44\times10^{-4} and weight decay of 0.05, setting α=0.8\alpha=0.8, λmag=0.01\lambda_{\mathrm{mag}}=0.01, and λsmooth=0.001\lambda_{\mathrm{smooth}}=0.001. During streaming, Gaussian refinement is performed every five frames, using 100 updates for N3DV, 200 for Meeting Room, and 300 for Google Immersive. IGS and Ours share the same initialization and refinement budget for each scene.

Evaluation Metrics. We report PSNR (dB) [31] and structural dissimilarity, DSSIM =(1−SSIM⁡)/2=(1-\operatorname{SSIM})/2 [32], to measure reconstruction quality. Seconds per frame (s/frm) denotes the average online reconstruction time per target frame, including periodic keyframe refinement and excluding initial Gaussian reconstruction. Higher PSNR and lower DSSIM and s/frm indicate better performance.

Results

In-domain Evaluation. We evaluate on held-out scenes from N3DV and Google Immersive for the in-domain evaluation. As shown in Table 1, our method achieves 33.046 dB on N3DV and 23.798 dB on Google Immersive, improving over IGS by 0.169 dB and 0.199 dB, respectively. Our method obtains the second-highest PSNR among the online methods on both datasets, outperforming IGS, ReConGS, and 3DGStream, while QUEEN achieves the highest PSNR.

Method# Params# Online refinementsN3DV PSNR ↑N3DV DSSIM ↓N3DV s/frm ↓Meeting Room PSNR ↑Meeting Room DSSIM ↓Meeting Room s/frm ↓Google Immersive PSNR ↑Google Immersive DSSIM ↓Google Immersive s/frm ↓Average PSNR ↑Average DSSIM ↓Average s/frm ↓
Offline methods
RT 4DGS [37]032.0100.01430.4700.02526.7600.03029.7470.023
4D-GS [33]2.46M31.1500.01631.2750.02423.9900.07928.8050.040
STG [19]9032.0500.02629.5100.03429.2000.04230.2530.034
Swift4D [34]31.92M32.2300.01431.3060.02223.4820.06029.0060.032
Online methods
ReConGS [6]02,99432.2530.0217.4530.4250.0284.4421.8900.1037.9528.1890.0516.61
3DGStream [27]1.99M2,99431.7400.0228.2730.6860.0275.5422.3810.07912.6228.2690.0438.81
QUEEN [8]345.01M2,99433.5120.0185.6129.1260.0272.2525.8200.04615.2329.4860.0307.70
IGS [35]21.90M59432.8770.0202.0030.0880.0281.9623.5990.0814.5528.8550.0432.84
Ours27.27M59433.0460.0192.3030.2900.0272.1823.7980.0815.5029.0450.0423.33

Table 1. Quantitative comparison on N3DV, Meeting Room, and Google Immersive. Average is the equal-weight mean of the three displayed dataset results. Gray rows are offline methods; the remaining rows are online methods. Darker blue indicates the best, and second-best online results. Offline quality includes published reference results.

Cross-domain Evaluation. We evaluate the N3DV-trained checkpoint on Meeting Room, without further neural-weight training. Results in Table 1 demonstrate competitive transfer on Meeting Room. On Meeting Room, our method achieves 30.290 dB, improving over IGS by 0.202 dB and obtaining lower DSSIM. This shows improved cross-domain transfer of OCLDA-GS.

Reconstruction Efficiency. As discussed in § 3, we do Gaussian optimization only at keyframes and use neural prediction for intervening frames. On N3DV and Meeting Room, reconstruction requires 2.30 and 2.18 s/frm, respectively, compared with 2.00 and 1.96 s/frm for IGS. The added processing cost is approximately 0.30 and 0.22 s/frm, accompanying the quality improvements on these datasets. Across the completed comparisons in Table 1, our method has the second-lowest reported reconstruction time on every dataset. Our method achieves an average PSNR of 29.045 dB with 594 total refinements, outperforming IGS at the same refinement count and achieving the highest avThe average PSNR among online methods using the fewest refinements. QUEEN, the only online method with higher average PSNR (29.486 dB), uses 2,994 refinements, approximately 5.04× as ours.

Qualitative Results

Figure 3 compares our method with online baselines on selected examples from the five datasets. Our reconstructions show closer alignment with the ground truth around moving objects, with reduced ghosting and boundary artifacts, complementing the aggregate comparisons in Table 1.

Qualitative comparison of online methods across three datasets

Figure 3. Qualitative comparison of online methods across three datasets. Each row uses the same frame, viewpoint, and crop for all methods. Orange boxes mark regions enlarged in the insets, highlighting motion alignment and object boundaries.

Figure 4 illustrates anchor selection, group propagation, and local deformation correction on a Meeting Room sequence. Flow over the completed interval reallocates a fixed anchor budget at the current keyframe. Object group-constrained propagation preserves the recovered grouping, while the learned local deformation adjusts the propagated deformation. The final panel shows the reconstruction after standard target-keyframe refinement.

Flow-guided anchor selection and temporal propagation

Figure 4. Flow-guided anchor selection and temporal propagation. Flow guides anchor reselection within a fixed budget. Colors denote SAGA groups; white points mark selected anchors. Markers and trails visualize anchor and Gaussian motion. Ours shows the refined target keyframe from an input-camera view. Reselection and refinement occur only at keyframes.

Ablation Study

Component contributions. Table 2 compares flow-guided anchor selection, object-constrained propagation, and learned local deformation correction. Object restricts motion propagation within object groups, Flow prioritizes moving objects during anchor allocation, and LD learns per-anchor corrections. The complete method improves PSNR over IGS by 0.169 dB on N3DV and 0.202 dB on Meeting Room. Flow-containing variants consistently improve Meeting Room reconstruction, while local deformation correction provides the highest N3DV PSNR when combined with FPS. Although component gains are not uniformly additive, the complete method remains within 0.013 dB and 0.002 dB of the highest-scoring configurations on N3DV and Meeting Room, respectively.

ObjectFlowLDN3DV PSNR ↑N3DV DSSIM ↓N3DV s/frm ↓Meeting Room PSNR ↑Meeting Room DSSIM ↓Meeting Room s/frm ↓Average PSNR ↑Average DSSIM ↓Average s/frm ↓
×××32.8770.0202.0030.0880.0281.9631.4820.0241.98
✓××32.8820.0202.2030.1160.0282.1431.4990.0242.17
×✓×32.9130.0202.3730.2630.0272.3131.5880.0232.34
××✓33.0590.0202.2830.1090.0282.2431.5840.0242.26
✓✓×32.9270.0202.3330.2670.0272.2731.5970.0232.30
×✓✓33.0180.0192.2930.2920.0272.3031.6550.0232.30
✓×✓32.9210.0202.3330.1630.0282.1631.5420.0242.25
✓✓✓33.0460.0192.3030.2900.0272.1831.6680.0232.24

Table 2. Component ablation on N3DV and Meeting Room. Checkmarks indicate enabled components. The first row corresponds to IGS (FPS), and the last row to our full method. Average equally weights the two datasets. LD: Local Deformation

Table 3 compares Local Deformation (LD) and anchor reallocation. For LD configuration, No correction disables LD, while Rigid replacement uses estimated rigid motion only. LD MLP w/o Rigid learns per-anchor corrections without conditioning them on the estimated rigid rotations. For anchor reallocation, FPS selects anchors for spatial coverage, while Object based (No\sqrt{N_o}) allocates anchors by object size without motion guidance. Ours combines LD conditioned on rigid motion with motion-guided anchor reallocation, achieving the highest average PSNR of 31.668 dB.

ConfigurationPSNR ↑ N3DVPSNR ↑ MRAverage PSNR ↑Average DSSIM ↓
Local deformation configuration
No correction32.92730.26731.5970.023
Rigid replacement32.98830.20431.5960.024
LD MLP w/o Rigid32.99130.28031.6360.023
Anchor Reallocation
FPS32.88230.11631.4990.024
Object based (No\sqrt{N_o})33.02630.28031.6530.023
Ours33.04630.29031.6680.023

Table 3. Ablation of local deformation and anchor allocation on N3DV and Meeting Room (MR). Average metrics equally weight the two datasets. LD: Local Deformation

Cross-domain temporal robustness. Meeting Room is excluded from local deformation-head training, so its improvement over IGS demonstrates transfer beyond the training datasets without retraining the head. Beyond average quality, Figure 6 shows smaller PSNR degradation between keyframes. The mean drop in PSNR from frames 0 to 4 from keyframe decreases from 0.622 to 0.565 dB on N3DV and from 1.927 to 1.662 dB on Meeting Room, corresponding to reductions of 9.2% and 13.8%. Together with the full-sequence trajectories in Figure 5, these results support improved quality preservation between refinement updates, including on the unseen dataset.

Figure 5. Temporal reconstruction quality with selected online baselines. Faint traces show raw values; solid curves show a 45-frame EMA. Full comparisons appear in Table 1.

Figure 6. Mean PSNR drop relative to keyframes on N3DV and Meeting Room; lower is better.

Number of anchors. On cross-domain dataset Meeting Room evaluation, OCLDA-GS outperforms IGS by 0.113 dB using eight times fewer motion anchors. Table 4 shows consistent PSNR gains on N3DV, Meeting Room, and Google Immersive across all anchor budgets. This further demonstrates the advantage of anchor re-allocation under tighter budgets.

Dataset1,024 IGS1,024 Ours4,096 IGS4,096 Ours8,192 IGS8,192 Ours
N3DV32.74132.82232.85132.96232.87733.046
Meeting Room29.86030.20130.05730.26330.08830.290
Google Immersive23.63723.76623.59323.59823.59923.798

Table 4. Motion-anchor budget ablation across the datasets. PSNR (↑) averages evaluated targets per scene, followed by an equal-scene average within each dataset. Shading indicates the highest PSNR per dataset.

Refinement efficiency. Figure 7 examines the PSNR–runtime tradeoff across four keyframe refinement budgets. On N3DV, OCLDA-GS with 100 refinement steps achieves 32.995 dB at 2.13 s/frame, compared with 32.952 dB at 2.95 s/frame for IGS with 200 steps. This yields slightly higher PSNR with 27.8% less reconstruction time. On Meeting Room, Ours with 200 steps reaches 29.777 dB, within 0.057 dB of IGS with 400 steps, while taking 33.5% less time. These results indicate that the proposed propagation can reduce the refinement budget needed to obtain comparable PSNR.

Refinement budget and reconstruction quality

Figure 7. Refinement budget and reconstruction quality. Numbers denote keyframe refinement steps.

Limitations

OCLDA-GS depends on the quality of object groups. Inaccurate grouping can limit object-aware motion propagation. More robust object grouping could alleviate this limitation. Similarly, the rigid-motion mechanism assumes that each object group results in a useful dominant rigid transformation, which may be inaccurate for articulated objects. Finally, our learned local deformation correction updates Gaussian positions only but does not correct errors in orientation, scale, and appearance.

Conclusions

We presented OCLDA-GS, an object-centric Gaussian streaming framework that combines flow-guided anchor selection, object-constrained motion propagation, and learned local deformation correction conditioned on object-wise rigid motion. Under a fixed anchor budget, these components improve motion modeling without per-frame Gaussian refinement. Experiments across three dynamic multi-view datasets demonstrate competitive reconstruction quality and efficiency, with improvements over IGS on N3DV and Meeting Room and reduced quality degradation between keyframes. These results highlight the importance of object-aware motion modeling for efficient Gaussian streaming.

LLM Usage Statement

OpenAI ChatGPT was used to obtain reviewer-style feedback on the manuscript or major portions of it. It assisted with editing and refining author-produced text, including related work and citation presentation, and with refining author-produced material concerning experimental design, execution, analysis of results, and figures or diagrams. For code and implementation, ChatGPT generated work that the authors reviewed, edited, or approved. No LLMs or generative AI tools were used for synthetic data, dataset creation, annotation, or developing research ideas, hypotheses, claims, or conclusions. The authors take full responsibility for all submitted content, regardless of any LLM or AI-tool use.

References

  1. [1]Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew DuVall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. Immersive light field video with a layered mesh representation. ACM Transactions on Graphics, 39(4):86:1–86:15, 2020. 1, 5DOI
  2. [2]Ang Cao and Justin Johnson. HexPlane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 130–141, 2023. 2arxiv.org/abs/2301.09632
  3. [3]Jiazhong Chen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian. Segment any 3D Gaussians. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1971–1979, 2025. 5arxiv.org/abs/2312.00860
  4. [4]Jiacong Chen, Qingyu Mao, Youneng Bao, Xiandong Meng, Fanyang Meng, Ronggang Wang, and Yongsheng Liang. Motion matters: Compact gaussian streaming for free-viewpoint video reconstruction. In Advances in Neural Information Processing Systems, 2025. 2, 3arxiv.org/abs/2505.16533
  5. [5]Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-Planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479–12488, 2023. 2
  6. [6]Jiahe Fu, Qiankun Gao, Chengxiang Wen, Yanmin Wu, Siwei Ma, Jiaqi Zhang, and Jian Zhang. ReCon-GS: Continuum-preserved gaussian streaming for fast and compact reconstruction of dynamic scenes. In Advances in Neural Information Processing Systems, 2025. 2, 5
  7. [7]Qiankun Gao, Jiarui Meng, Chengxiang Wen, Jie Chen, and Jian Zhang. HiCoM: Hierarchical coherent motion for dynamic streamable scenes with 3D Gaussian Splatting. In Advances in Neural Information Processing Systems, 2024. 2DOI
  8. [8]Sharath Girish, Tianye Li, Amrita Mazumdar, Abhinav Shrivastava, David Luebke, and Shalini De Mello. QUEEN: Quantized Efficient ENcoding of dynamic gaussians for streaming free-viewpoint videos. In Advances in Neural Information Processing Systems, 2024. 2, 5arxiv.org/abs/2412.04469
  9. [9]Bing He, Yunuo Chen, Guo Lu, Qi Wang, Qunshan Gu, Rong Xie, Li Song, and Wenjun Zhang. H3D-DGS: Exploring heterogeneous 3D motion representation for deformable 3D Gaussian Splatting. In Advances in Neural Information Processing Systems, 2025. 2, 3arxiv.org/abs/2408.13036
  10. [10]Qiang Hu, Zihan Zheng, Houqiang Zhong, Sihua Fu, Li Song, Xiaoyun Zhang, Guangtao Zhai, and Yanfeng Wang. 4DGC: Rate-aware 4D Gaussian compression for efficient streamable free-viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 875–885, 2025. 2arxiv.org/abs/2503.18421
  11. [11]Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. SC-GS: Sparse-controlled gaussian splatting for editable dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4220–4230, 2024. 2, 3arxiv.org/abs/2312.14937
  12. [12]Wolfgang Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A, 32(5):922–923, 1976. 4
  13. [13]Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023. 1, 2arxiv.org/abs/2308.04079
  14. [14]Sangwoon Kwak, Joonsoo Kim, Jun Young Jeong, Won-Sik Cheong, Jihyong Oh, and Munchurl Kim. MoDec-GS: Global-to-local motion decomposition and temporal interval adjustment for compact dynamic 3D Gaussian Splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11338–11348, 2025. 3arxiv.org/abs/2501.03714
  15. [15]Hao Li, Sicheng Li, Xiang Gao, Abdououhaiti Batuer, Lu Yu, and Yiyi Liao. GIFStream: 4D Gaussian-based immersive video with feature stream. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21761–21770, 2025. 2arxiv.org/abs/2505.07539
  16. [16]Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, and Ping Tan. Streaming radiance fields for 3D video synthesis. In Advances in Neural Information Processing Systems, pages 13485–13498, 2022. 2, 5arxiv.org/abs/2210.14831
  17. [17]Tianye Li, Mira Slavcheva, Michael Zollhöfer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, and Zhaoyang Lv. Neural 3D video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5521–5531, 2022. 5
  18. [18]Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498–6508, 2021. 2arxiv.org/abs/2011.13084
  19. [19]Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8508–8520, 2024. 2, 5arxiv.org/abs/2312.16812
  20. [20]Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3D gaussians: Tracking by persistent dynamic view synthesis. In International Conference on 3D Vision, 2024. 1, 3arxiv.org/abs/2308.09713
  21. [21]Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.arxiv.org/abs/2003.08934
  22. [22]Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865–5874, 2021.arxiv.org/abs/2011.12948
  23. [23]Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. HyperNeRF: A higher-dimensional representation for topologically varying neural radiance fields. ACM Transactions on Graphics, 40(6), 2021.arxiv.org/abs/2106.13228
  24. [24]Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.arxiv.org/abs/2011.13961
  25. [25]Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. NeRFPlayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 29(5):2732–2742, 2023.DOI
  26. [26]Robert W. Sumner, Johannes Schmid, and Mark Pauly. Embedded deformation for shape manipulation. ACM Transactions on Graphics, 26(3), 2007.
  27. [27]Jiakai Sun, Han Jiao, Guangyuan Li, Zhijie Zhang, Lei Zhao, and Wei Xing. 3DGStream: On-the-fly training of 3D Gaussians for efficient streaming of photo-realistic free-viewpoint videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20675–20685, 2024.arxiv.org/abs/2403.01444
  28. [28]Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision, 2020.arxiv.org/abs/2003.12039
  29. [29]Liao Wang, Qiang Hu, Qihan He, Ziyu Wang, Jingyi Yu, Tinne Tuytelaars, Lan Xu, and Minye Wu. Neural residual radiance fields for streamably free-viewpoint videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 76–87, 2023.arxiv.org/abs/2304.04452
  30. [30]Liao Wang, Kaixin Yao, Chengcheng Guo, Zhirui Zhang, Qiang Hu, Jingyi Yu, Lan Xu, and Minye Wu. VideoRF: Rendering dynamic radiance fields as 2D feature video streams. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 470–481, 2024.arxiv.org/abs/2312.01407
  31. [31]Zhou Wang and Alan C. Bovik. Mean squared error: Love it or leave it? a new look at signal fidelity measures. IEEE Signal Processing Magazine, 26(1):98–117, 2009.DOI
  32. [32]Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.pubmed.ncbi.nlm.nih.gov/15376593
  33. [33]Guanjun Wu, Toaran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4D Gaussian Splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20310–20320, 2024.arxiv.org/abs/2310.08528
  34. [34]Jiahao Wu, Rui Peng, Zhiyan Wang, Lu Xiao, Luyang Tang, Jinbo Yan, Kaiqiang Xiong, and Ronggang Wang. Swift4D: Adaptive divide-and-conquer gaussian splatting for compact and efficient reconstruction of dynamic scene. In International Conference on Learning Representations, 2025.
  35. [35]Jinbo Yan, Rui Peng, Zhiyan Wang, Luyang Tang, Jiayu Yang, Jie Liang, Jiahao Wu, and Ronggang Wang. Instant Gaussian Stream: Fast and generalizable streaming of dynamic scene reconstruction via gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16520–16531, 2025.arxiv.org/abs/2503.16979
  36. [36]Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3D Gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20331–20341, 2024.arxiv.org/abs/2309.13101
  37. [37]Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4D Gaussian Splatting. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.10642
  38. [38]Ruijie Zhu, Yanzhe Liang, Hanzhi Chang, Jiacheng Deng, Jiahao Lu, Wenfei Yang, Tianzhu Zhang, and Yongdong Zhang. MotionGS: Exploring explicit motion guidance for deformable 3D Gaussian Splatting. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2410.07707

Paper details

vlnc:W_tqpwddw8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

OpenAI ChatGPT

View statement
Looking for related papers…
Contents