betaFeedback

WorksComputer Science

SwitchSplat: Learning Discrete Object States from Casually Captured Scenes

Abstract

3D Gaussian Splatting has made high-fidelity 3D reconstruction fast and accessible, yet existing representations assume a static world. This assumption breaks down in casual captures, where a user naturally interacts with the environment – opening doors, toggling switches, or moving objects – introducing discrete state changes that no current framework can model. Time-conditioned dynamic methods such as 4DGS conflate when a state was observed with what state the scene is in, limiting them to replaying seen sequences rather than composing arbitrary object configurations. We present SwitchSplat, the first 3DGS representation to address this gap via a base+slot decomposition: a shared static base captures the invariant scene, while each slot independently holds the two binary states (on/off) of one interactable object. Because the assignment of Gaussians to slots is fully differentiable, the model discovers the geometry of each state directly from video, without manual annotation. At inference, the user can freely combine slot states and render the resulting scene from any viewpoint. We validate SwitchSplat on a new casual-capture dataset and show that it outperforms static and dynamic baselines in novel-view synthesis quality, while uniquely supporting rendering of state combinations never seen during capture. Project page: https://switchsplat.github.io

Authors’ Contact Information: Yizheng Xie, Simon Fraser University, Burnaby, Canada, [email protected]; Yalda Foroutan, Simon Fraser University, Burnaby, Canada, [email protected]; George Shramko, Simon Fraser University, Burnaby, Canada, [email protected]; Daniel Rebain, University of British Columbia, Vancouver, Canada and Wayve Labs, Vancouver, Canada, [email protected]; Shrisudhan Govindarajan, Simon Fraser University, Burnaby, Canada and Wayve Labs, Vancouver, Canada, [email protected]; Andrea Tagliasacchi, Simon Fraser University, Burnaby, Canada and University of Toronto, Toronto, Canada and Wayve Labs, Vancouver, Canada, [email protected].

This work is licensed under a Creative Commons Attribution 4.0 International License. SA Conference Papers ’26, Kuala Lumpur, Malaysia

© 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2842-6/2026/12 https://doi.org/10.1145/3829340.3842325

ACM Reference Format:

Yizheng Xie, Yalda Foroutan, George Shramko, Daniel Rebain, Shrisudhan Govindarajan, and Andrea Tagliasacchi. 2026. SwitchSplat: Learning Discrete Object States from Casually Captured Scenes. In SIGGRAPH Asia 2026 Conference Papers (SA Conference Papers ’26), December 01–04, 2026, Kuala Lumpur, Malaysia. ACM, New York, NY, USA, 21 pages. https://doi.org/10.1145/3829340.3842325

Introduction

In inverse rendering, Gaussian Splatting (3DGS) [10] has emerged as the leading representation for real-time, high-fidelity novel-view synthesis, enabling a new generation of interactive applications. Its combination of fast training and efficient rendering has driven broad adoption across academia as well as industry, from Matterport’s digital twins and Niantic Scaniverse’s mobile reconstruction to Meta’s Hyperscape1.

While early 3DGS pipelines required meticulous data acquisition (fixed exposure and white balance, controlled lighting, and a perfectly static scene), recent works like 3DGRT [19] and FullCircle [3] have unlocked casual capture: viewpoints need not be carefully curated – made possible by 360° cameras held in a single hand – and the capturer’s pixels are automatically masked out to satisfy the static-scene assumption. This freedom naturally invites users to interact with the environment during capture – opening doors, toggling switches, or repositioning objects – turning the operator into both an active agent and a dynamic distractor within the scene.

Yet no existing reconstruction framework is equipped to handle these interactions. Many contemporary 3D representations enforce a strict static-world assumption: when an operator triggers a discrete state change – opening a door, flipping a switch, relocating an object – the model has no mechanism to account for the resulting geometric variation. As a result, even state-of-the-art pipelines can only represent a single, frozen scene configuration, and capturing multiple stable states requires re-recording the scene from scratch.

One could argue that the natural candidate to address this limitation is the family of 4D Gaussian Splatting methods [35], which extend 3DGS to dynamic scenes by conditioning the scene representation on time. However, this design choice fundamentally conflates when a state was observed with what state the scene is in: the model learns to replay a specific sequence of observations, not to understand independent object configurations.

As a result, 4D methods are largely limited to novel-viewpoint synthesis of seen sequences; composing an unseen combination of object states—e.g., door open and light on—is not directly supported, since no single timestamp in the training data corresponds to that configuration. What is needed instead is a representation that decorrelates scene state from time, allowing the user to freely specify both the viewpoint and the desired configuration of each object independently.

To address these challenges, we propose SwitchSplat, a method that learns “state-aware” representations from casual captures featuring human-object interactions; see Figure 1. SwitchSplat organizes the scene into a static base – the parts that never change throughout the capture – and a set of slots, one per interactable object. Each slot holds two 3D representations corresponding to the binary states of that object (“on” and “off”). For example, if a room was captured while a door was closed for part of the recording and open for another, a single slot would hold both the “off” state (door closed, interior hidden) and the “on” state (door open, interior visible), while the walls, floor, and furniture – unchanged throughout – are handled by the static base. To assign each Gaussian primitive to the right part of the scene, we attach a small learnable score vector that controls how strongly the primitive belongs to the base, or to either state of each slot.

Input video frames and novel view synthesis results showing switchable object states

Figure 1. SwitchSplat learns a state-aware scene representation from a casual capture (top: input video frames showing the user interacting with the car). The scene is decomposed into a static base and a set of independently switchable slots, one per interactable object (here: right door, hood, trunk). Each slot models two binary states, enabling SwitchSplat to render any combination of object configurations from novel viewpoints at inference time (*bottom: four of the 23=82^3 = 8 possible slot configurations, labeled by their on/off state vector).

This decomposition avoids data redundancy: without a shared static base, one would need to duplicate the full scene geometry for every slot, with no reusable structure across states. Instead, SwitchSplat shares the base across all configurations and only stores the geometry that actually differs per state.2 Because the full pipeline is differentiable end-to-end, the model learns simultaneously which Gaussians belong to which slot and when each slot is active – directly from the input video, without any manual annotation. At inference, the user can query any combination of slot states independently – e.g., closet door open and desk lamp on – and render the resulting scene from any viewpoint.

To summarize our contributions: (1) We identify a novel problem in the 3DGS literature: existing representations cannot model scenes with discrete, user-triggered state changes (e.g., objects that can be opened, switched, or moved), and we show that naively applying dynamic methods such as 4DGS fails to address this. (2) We propose SwitchSplat, the first 3DGS representation to address this problem via a base+slot decomposition, where a shared static base captures the invariant scene and each slot independently models the binary states of one interactable object. (3) We demonstrate on a new casual-capture dataset that SwitchSplat improves novel-view synthesis quality over static and dynamic baselines, and uniquely supports rendering arbitrary combinations of object states from unseen viewpoints.

Related Work

3D Gaussian Splatting (3DGS) [10] has become the dominant representation for novel-view synthesis, optimizing a set of anisotropic Gaussian primitives that render at real-time rates. Extensions beyond the standard perspective-camera model are particularly relevant to our setting: 3DGRT [19] reformulates rendering via ray tracing, and 3DGUT [32] adds support for highly distorted cameras and secondary rays, enabling casual capture with wide field-of-view sensors. Full-Circle [3] further shows that raw dual-fisheye 360° video suffices for high-quality reconstruction, automatically masking the camera operator. Despite these advances, all of these pipelines represent the scene as a single, fixed configuration. When multiple object states are present in the capture—a door open in some frames and closed in others—the model either conflates them into averaged geometry and ghosting artifacts, or learns to discard the minority state as a distractor [22]. Either way, only one configuration survives in the final model; there is no mechanism to retain, switch between, or compose multiple stable states at inference time.

Dynamic Gaussian Splatting. A large family of methods extends 3DGS to dynamic scenes by conditioning the representation on time. Dynamic 3D Gaussians [Luiten et al. 2024] tracks persistent primitives across frames, while Deformable 3D Gaussians [Yang et al. 2024a] and 4DGS [31] deform a canonical scene over time, with 4DGS using a learned space-time encoding. Spacetime Gaussians [12] further augments primitives with temporal opacity and parametric motion. Subsequent work improves efficiency, temporal consistency, or geometric accuracy via depth priors [2, 11, 16, 25, 35]. Differently from our method, all of these methods are parameterized by time, not by object state: recovering a specific scene configuration requires locating the right moment in the recording, and rendering a novel combination of states—e.g., two objects in configurations never co-observed—is not supported.

Pipeline from casual capture to state-conditioned rendering

Figure 2. Pipeline – From a casual capture in which the operator interacts with objects (a), we fit a differentiable slot table (b) whose learnable endpoints (tlt_l, trt_r) and temperature τk\tau_k define a per-frame interaction probability πk(t)\pi_k(t) for each object. Each 3D Gaussian primitive carries a state-logit vector ℓ\ell over a static base channel together with an off and an on channel per slot, yielding per-state probabilities (c). At render time (d), these probabilities are fused with πk(t)\pi_k(t) into a frame-conditional opacity α~(t)\tilde{\alpha}(t) that drives standard volumetric composition. All Gaussians, state logits, and slot-table parameters are optimized end-to-end under photometric losses, producing a state-aware scene that renders arbitrary object configurations from novel viewpoints.

Decomposed Gaussian Splatting. More recent dynamic Gaussian methods, such as SDD-4DGS [24], observe that applying deformation models to the entire scene is often inefficient, since most real-world scenes are largely static. This motivates methods that decompose scenes into static and dynamic components: GauFRe [13] combines a static Gaussian component with deformable dynamic Gaussians, while Swift4D [32] improves efficiency by passing only dynamic Gaussians through the deformation model. DeGauss [30] further uses dynamic-static decomposition to improve robustness to transient distractors. This observation is closely aligned with our setting, since object interactions usually affect only a small part of a 360∘360^\circ environment. In contrast to our method, existing decomposition methods still treat the dynamic component as a temporally varying foreground: they separate dynamic regions from a static background, but provide no mechanism to independently switch or compose object states at inference time.

Controllable Scene Representation. A parallel line of work studies controllable and articulated 3D representations. Pre-Gaussian methods such as Watch It Move [20] discover articulated structure from object motion; PARIS [15] goes further by decoupling static and movable parts from two multi-view captures of the same scene. Gaussian-based approaches extend this to photorealistic settings: SC-GS [8] enables motion editing via sparse control points, Template-free Articulated Gaussian Splatting [28] supports re-posing with super-points, and RigGS [36] adds rig-based editing and interpolation. ArtGS [17] and ArticulatedGS [5] further focus on part-level modeling of individual objects. Crucially, all of these methods aim to encode object motion or articulation—they model how parts move, rather than discarding motion entirely to focus on the discrete stable configurations of the scene. In contrast, our method treats object interactions as a nuisance to be factored out, retaining only the clean on/off states and ignoring the transient motion between them.

State-aware Scene Reconstruction. Closest to our setting are methods that reconstruct scenes across discrete object states. IGFuse [6] and RecurGS [7] address this by using semantic cues to detect state changes and align them via rigid transformations. GaussianCut [9] and DecoupledGaussian [29] contribute related tools for object segmentation and scene decomposition. However, all of these methods assume that state changes can be described as rigid motions of segmented object entities—an assumption that breaks down for casual capture, where interaction boundaries are fuzzy and deformations are non-rigid. In contrast, SwitchSplat learns which Gaussians belong to each state directly from photometric supervision, without requiring object masks, or rigid-body assumptions.

Method

SwitchSplat learns a state-aware 3D representation from casual captures by decomposing the scene into controllable, state-conditioned components. In the following subsections, we detail our methodology by first formulating our state-conditioned Gaussian primitives and describing their integration within volume rendering (§ 3.1). We then introduce the parameterization of the differentiable slot-table (§ 3.2), followed by our slot initialization procedure (§ 3.3). Finally, we describe our training pipeline and objectives that facilitate granular, state-controlled visualization during inference (§ 3.4 and § 3.5).

Terminology. A slot corresponds to one interactable object and holds two 3D representations for its binary states: “off” (e.g., door closed) and “on” (e.g., door open). A slot activation is a per-frame binary variable indicating which state is currently observed. All parts of the scene that remain constant throughout the capture are handled by the static base, which is shared across all configurations.

State-aware Gaussian Splat

Each 3D Gaussian primitive in our model is defined by its standard geometric and appearance attributes: mean μ\boldsymbol{\mu}, covariance Σ\boldsymbol{\Sigma}, opacity α\alpha, and view-dependent appearance c\mathbf{c}. To make our representation state-aware, we augment every Gaussian with a learnable categorical state distribution p=softmax⁡(ℓ)\mathbf{p} = \operatorname{softmax}(\boldsymbol{\ell}) over 1+2K1 + 2K channels, where KK is the total number of object slots. These channels comprise a single static base channel ℓ[0]\ell[0], together with a pair of off (ℓ[2k−1]\ell[2k-1]) and on (ℓ[2k]\ell[2k]) channels for each slot k∈{1,…,K}k \in\{1, \ldots, K\}. The two channels per slot — rather than one — let the softmax assign each Gaussian to a single configuration, so that disjoint subsets of primitives can independently model the geometry and appearance of an object in each of its two states (see Figure 3). This separation is essential because the on/off configurations of an object generally differ not just in appearance, but also in spatial extent — a closed door and an open door, for instance, occupy different parts of the scene — which a single shared primitive simply cannot represent.

Figure 3. State-aware Gaussian – (left) Each Gaussian carries a learnable logit vector ℓ\boldsymbol{\ell} that defines a categorical distribution p\mathbf{p} that softly assigns each primitive to the static base or “on”/“off” object states. (right) At render time, user specifies list of states that are active or inactive, and the probability of each gaussian being present in the scene ω(p~)\omega(\tilde{\mathbf{p}}) is computed via Eq. (1).

Test-time rendering. Assume we have trained our representation, and that we want to render an image corresponding to the slot configuration p~∈{0,1}K\tilde{\mathbf{p}} \in\{0, 1\}^{K}. This is achieved by computing the probability of each gaussian being present in the scene to be rendered:

ω(p~)=p[0]+∑k=1K(p~[k]⋅p[2k]+(1−p~[k])⋅p[2k−1])(1)\omega(\tilde{\mathbf{p}}) = \mathbf{p}[0] + \sum_{k=1}^{K} \left(\tilde{\mathbf{p}}[k] \cdot\mathbf{p}[2k] + (1-\tilde{\mathbf{p}}[k]) \cdot\mathbf{p}[2k-1]\right) \tag*{(1)}

This formulation ensures that “base” Gaussians remain present across all frames, while the presence of the on/off state Gaussians is gated by the probability of the corresponding slot being on. We then gate the opacity of the gaussian by this probability:

α~=α⋅ω(p~),(2)\tilde{\alpha} = \alpha\cdot\omega(\tilde{\mathbf{p}}), \tag*{(2)}

which is then used to compute the final RGB color for a ray r\mathbf{r} via classical volume rendering [27]:

I^(r)=∑i∈H(r)Ti(r)⋅α~i⋅ci(r),Ti(r)=∏j<i(1−α~j),\hat{\mathbf{I}}(\mathbf{r}) = \sum_{i \in\mathcal{H}(\mathbf{r})} T_i(\mathbf{r}) \cdot\tilde{\alpha}_i \cdot\mathbf{c}_i(\mathbf{r}), \qquad T_i(\mathbf{r}) = \prod_{j<i}(1-\tilde{\alpha}_j),

where H(r)\mathcal{H}(\mathbf{r}) represents the set of Gaussians intersected by the ray in depth order and ci(r)\mathbf{c}_i(\mathbf{r}) denotes the view-dependent color. While p~\tilde{\mathbf{p}} will be selected by the user at test-time, the key question in training our representation lies in how to propagate the gradients from image I^t\hat{\mathbf{I}}_t at time tt to the (per-Gaussian) state distribution p\mathbf{p}.

Differentiable slot table

To achieve this, we exploit the key assumption that, within the video, the activation state of a slot is compactly supported in time. Let us denote with the per-slot scalar πk(t)∈[0,1]\pi_k(t) \in[0,1] the probability of slot kk being in its on state at time tt. Then, ∀k≠0\forall k \ne0 we have that πk(t)\pi_k(t) is zero everywhere outside some bounded, closed region. Without loss of generality, let us drop the indexing kk, and focus on a single entry π(t)\pi(t). We can satisfy our compact support assumption by explicitly parameterizing π(t)\pi(t) as a smooth window function defined by its temporal endpoints (tl,tr)3(t_l,t_r)^{3}, and a temperature parameter τ\tau that controls the smoothness of transition; see Figure 4. By defining the midpoint as tm=(tl+tr)/2t_m = (t_l + t_r)/2, and στ(t)=σ(t/τ)\sigma_\tau(t) = \sigma(t/\tau), the probability of an interaction occurring at time tt is computed via the normalized product of two sigmoid functions σ\sigma:

Figure 4. Slot window function – The slot table πk(t)\pi_k(t) is parameterized as a normalized product of two sigmoids controlled by learnable endpoints (tl,tr)(t_l,t_r) and temperature τ\tau. (left) Sweeping τ\tau interpolates between sharp rectangular windows (low τ\tau) and broad, soft transitions (high τ\tau). (right) The resulting per-frame Bernoulli PMF over slot kk’s state, P(Sk,t)P(S_{k,t}). Green markers denote the two learnable endpoints tl,trt_l,t_r, which can shift along the time axis during optimization.

π(t)=στ(t−tl)⋅στ(tr−t)στ(tm−tl)⋅στ(tr−tm)(3)\pi(t) = \frac{\sigma_\tau(t-t_l)\cdot\sigma_\tau(t_r-t)}{\sigma_\tau(t_m-t_l)\cdot\sigma_\tau(t_r-t_m)} \tag*{(3)}

Therefore, to model the time window of each object interaction, we treat the state of the kk-th slot at time tt as a Bernoulli random variable Sk,t∈{0,1}S_{k,t} \in\{0,1\}, where Sk,t=1S_{k,t}=1 denotes the slot being in its on state and Sk,t=0S_{k,t}=0 its off state, whose probability mass function (PMF) for a specific state s∈{0,1}s \in\{0,1\} is given by:

P(Sk,t=s)=πk(t)s(1−πk(t))1−s(4)P(S_{k,t}=s) = \pi_k(t)^s(1-\pi_k(t))^{1-s} \tag*{(4)}

Train-time rendering. The quantity πk(t)\pi_k(t) defined above is our slot table, which we use to compute the probability of each Gaussian being present in the scene at time tt:

ω(t)=p[0]+∑k=1K(πk(t)⋅p[2k]+(1−πk(t))⋅p[2k−1])(5)\omega(t) = \mathbf{p}[0] + \sum_{k=1}^{K}\left(\pi_k(t)\cdot\mathbf{p}[2k] + (1-\pi_k(t))\cdot\mathbf{p}[2k-1]\right) \tag*{(5)}

from which, like before, we can compute a time-dependent effective opacity for volume rendering the predicted image I^t\hat{\mathbf{I}}_t:

α~(t)=α⋅ω(t)(6)\tilde{\alpha}(t) = \alpha\cdot\omega(t) \tag*{(6)}

This differentiable rendering framework ensures that gradients propagate seamlessly to the slot activation probabilities, Gaussian state memberships, and geometric parameters, enabling stable and efficient joint optimization of the entire scene representation.

Initialization

In a casual capture setting, neither the total number of slots KK, nor the precise slot transition timestamps are known a priori. Requiring the user to annotate them would defeat the very premise of a “casual” capture. To address this, we leverage the reasoning capabilities of Vision-Language Models (VLMs), and in particular GPT 5.6, to provide an approximate initialization for the slot-table. By prompting GPT 5.6 with a sequence of frames, we extract semantically grounded start and end timestamps for each distinct object interaction. These timestamps serve as the initial values for the endpoint parameters (tl,tr)(t_l, t_r) in our slot-table. This VLM-informed initialization allows the system to bypass exhaustive temporal searches or complex diagnostics, providing a clean and efficient configuration for final full-resolution training. To facilitate a robust optimization process, we initialize each interaction slot with a high temperature value τ\tau. This high-temperature initialization ensures that the Bernoulli random variable Sk,tS_{k,t} begins with smooth transitions at the boundaries, allowing the gradients to flow more effectively over a wider temporal range during the initial stages of optimization.

Optimization

Leveraging the initial parameters derived from the VLM-based slot discovery phase, we proceed to optimize our final state-aware scene representation. We build our state-aware framework upon the 3D Gaussian Unscented Transform (3DGUT) architecture [32]. Following the 3DGUT pipeline, we initialize the 3D Gaussian primitives using COLMAP-derived point clouds and augment each primitive with the learnable state logits ℓ\ell. To provide an unbiased starting point for optimization, these logits are initialized such that the base/on/off state channels are assigned equal membership probabilities. Throughout this stage, we jointly optimize the geometric attributes of the Gaussians, their state logits, and the temporal parameters of the slot-table (tl,tr,τ)(t_l, t_r, \tau). We strictly adhere to the optimization schedule established by 3DGUT, including its specialized densification and pruning protocols. To ensure consistent state behavior during optimization, we modify the standard densification process: whenever a Gaussian primitive is cloned or split, its state logits are inherited by the resultant primitives. This inheritance mechanism ensures that state-aware attributes remain spatially coherent even as the underlying scene geometry increases in complexity.

Training objective

Beyond the standard L1L_1 photometric and SSIM losses used to preserve perceptual fidelity, we employ two additional regularizers to mitigate float artifacts, and prevent the emergence of redundant Gaussian primitives by supervising the optimization process via:

L=Lrgb+λ1LSSIM+λ2Lsparsity+λ3Lcompactness(7)\mathcal{L} = \mathcal{L}_{\mathrm{rgb}} + \lambda_1\mathcal{L}_{\mathrm{SSIM}} + \lambda_2\mathcal{L}_{\mathrm{sparsity}} + \lambda_3\mathcal{L}_{\mathrm{compactness}} \tag*{(7)}

Sparsity loss. To ensure a parsimonious partitioning of the scene, we regularize the Gaussian primitives to favor the static base state, thereby preventing unnecessary scene regions from being categorized as interactive. This is achieved by applying a negative log-likelihood penalty on the base state probability for all NN Gaussians:

Lsparsity=−1N∑nlog⁡pn[0](8)\mathcal{L}_{\mathrm{sparsity}} = -\frac{1}{N}\sum_n \log p_n[0] \tag*{(8)}

Compactness loss. To encourage Gaussians within slots to form spatially cohesive clusters, we first compute the opacity-weighted center of mass for primitives assigned to the on state of slot kk:

μˉk=∑nαn⋅pn[2k]⋅μn∑nαn⋅pn[2k]+ϵ(9)\bar{\mu}_k = \frac{\sum_n \alpha_n \cdot p_n[2k] \cdot\mu_n}{\sum_n \alpha_n \cdot p_n[2k] + \epsilon} \tag*{(9)}

and then penalize the weighted variance [26] for both the off and on states to encourage local clustering:

Lcompactness=12K∑j={0,1}∑nαn⋅pn[2k−j]⋅∥μn−μˉk∥2∑nαn⋅pn[2k−j]+ϵ(10)\mathcal{L}_{\mathrm{compactness}} = \frac{1}{2K}\sum_{j=\{0,1\}}\frac{\sum_n \alpha_n \cdot p_n[2k-j] \cdot\lVert\mu_n-\bar{\mu}_k\rVert^2}{\sum_n \alpha_n \cdot p_n[2k-j] + \epsilon} \tag*{(10)}

Experiments

To validate the effectiveness of our proposed algorithm in the absence of an existing dataset with variable object interactions, we collected a custom dataset using a consumer-grade Insta360 X4 dual-fisheye 360° camera. This hardware choice reflects the typical setup of casual users. During capture, we emulated non-expert behaviors, moving through the scene and interacting with objects—transitioning them from resting to active states and back.

Our dataset comprises 12 scenes organized into two groups. The standard set contains 10 casually captured scenes (8 indoor and 2 outdoor), each featuring 3–9 object interactions. The golden set contains 2 additional scenes for which we acquire controlled test sequences with novel camera trajectories and object-state combinations. We preprocess the raw dual-fisheye data following FullCircle [3] to mask the capturer. The calibration is performed in COLMAP using the masked fisheye frames, and then the fisheye images are undistorted into standard pinhole images.

For evaluation, we adhere to standard NeRF procedures, holding out every 8th frame as a test set. Crucially, we exclude test samples that coincide with state transitions (i.e., frames where an object is actively being moved) to reduce noise in our metrics, though these transition frames are retained within the training set to measure whether models can learn robustly from casual capture data. For the golden set sequences, we additionally evaluate novel camera frames, which show state configurations observed during training from new camera trajectories, and novel state-combination frames, which contain object-state configurations not observed during training. This set is used to demonstrate our method’s ability to generalize beyond the training configurations, and provide true state-aware control.

Baselines

We evaluate our proposed method against several state-of-the-art scene representation baselines. For static modeling, we compare against FullCircle [3], utilizing its underlying 3DGUT [32] architecture as our primary reference.

To assess performance in dynamic contexts, we benchmark our framework against 4DGS [35] and DeGauss [30]. For all quantitative and qualitative assessments, we utilize every 8th frame of the captured sequences as our standard evaluation set.

Quantitative Results

In Table 1, we present the quantitative performance of our method relative to both static and dynamic baselines on the undistorted dataset. These results validate our framework’s ability to maintain high-fidelity visual representations while providing granular, state-controlled manipulation. More detailed metrics for all 10 scenes are available in the supplementary material.

Table 1. Quantitative comparison on the main benchmark scenes with standard held-out views. We report PSNR (dB) on ten scenes grouped by capture setting. Concurrent indoor scenes contain multiple interactions occurring simultaneously. Best and second-best results are shown in bold and <u>underlined</u>, respectively.

MethodOutdoorIndoorConcurrent IndoorAverage
BalconyCarKitchen 1Flat 1Flat 2Flat 3ChairM Kitchen 1M Kitchen 2M Kitchen 3
Static method
FullCircle (3DGS)25.7026.8629.3229.2628.4628.9229.8830.1630.1330.1028.88
FullCircle (3DGUT)25.7726.7729.3529.2728.6229.1430.0630.3830.2330.3528.99
Dynamic method
DeGauss25.8627.3129.6929.3630.1128.9134.1032.7232.2932.6830.30
4DGS26.3629.1430.8330.7830.1729.5430.9531.8432.7533.1430.55
State-Conditioned method
SwitchSplat (3DGS)<u>26.89</u>28.92<u>32.32</u><u>32.87</u><u>32.58</u><u>33.33</u><u>35.55</u><u>33.20</u><u>33.97</u><u>34.44</u><u>32.41</u>
SwitchSplat (3DGUT)26.97<u>29.03</u>32.5533.1732.7833.6336.0433.5734.1734.5632.65

Table 1.

Qualitative Results

Figures 5 to 7 illustrate qualitative comparisons between SwitchSplat and baselines. While our method consistently yields high-fidelity renderings, dynamic baselines frequently exhibit pervasive floating artifacts during object interactions. For example, in the “Car” scene (Figure 6), both 4DGS and DeGauss generate significant floaters around the hood region. Similarly, in the “Kitchen” environment, these baselines produce substantial noise when the refrigerator is in its on state. Comparable degradations, manifesting as unanchored geometry, are observed during cabinet and wardrobe interactions in the “Flat” scenes. In contrast, SwitchSplat maintains significantly cleaner reconstructions by effectively disentangling static environment components from discrete state-specific changes.

Golden set comparisons

Beyond the standard held-out evaluation on the original trajectories, we evaluate SwitchSplat on two additional generalization settings: novel camera frames and novel state-combination frames. For two representative scenes, we acquire dedicated test trajectories containing previously unseen camera viewpoints and object-state combinations, and compare against 4DGS and DeGauss. Because 4DGS and DeGauss parameterize scene variation by continuous time rather than a discrete state code, we evaluate both dynamic baselines on the novel-camera and novel state-combination splits using a per-image

oracle over time. For each target image, we select the rendering with the highest PSNR for both methods; these results therefore constitute upper bounds. As shown in Figure 5, SwitchSplat consistently outperforms the dynamic baselines on novel camera frames and produces sharper reconstructions with fewer structural artifacts. We further evaluate novel state-combination frames, where multiple independently controlled interactions are composed into configurations not observed during training. Quantitative results in Table 2 demonstrate that SwitchSplat maintains high reconstruction quality under these unseen configurations, highlighting the compositional generalization enabled by our state-conditioned representation.

Table 2. Generalization to novel cameras and state combinations. † Dynamic baselines use a per-image oracle over time, selecting the highest-PSNR rendering from a set of sampled time; these results are upper bounds.

MethodKitchen 2Oslo
PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
Novel camera frames
DeGauss†32.380.9650.17429.400.9600.207
4DGS†32.050.9670.16829.930.9660.194
SwitchSplat (3DGS)33.900.9740.15531.820.9720.186
SwitchSplat (3DGUT)34.260.9750.15432.550.9730.185
Novel state-combination frames
DeGauss†30.850.9580.18027.510.9400.214
4DGS†29.630.9560.18226.530.9400.211
SwitchSplat (3DGS)31.290.9620.16429.290.9590.186
SwitchSplat (3DGUT)32.580.9670.16129.100.9580.186

Table 2.

Qualitative comparison across three evaluation settings

Figure 5. Qualitative comparison across three evaluation settings: standard held-out views, novel-camera views and novel state combinations. Columns show FullCircle, DeGauss, 4DGS, SwitchSplat, and the corresponding ground truth.

Ablation study

We perform a comprehensive ablation study to evaluate the contribution of individual components within our framework. These experiments are conducted on the “Kitchen 2” scene, which features the golden test set. We report PSNR averaged over three evaluation settings: the standard 8th-frame test set, novel camera frames, and novel state-combination frames, with each configuration further averaged over five independent runs.

Slot representation. We first evaluate the importance of the slot representation. When interaction slots are removed, the model reduces to a scene representation without any of our slot-specific components, including slot optimization and the associated regularization terms. The substantial performance drop in Table 3 demonstrates that explicitly introducing interaction slots is essential for modeling state-dependent scene changes.

SlotsSlots opt.Lcompactness\mathcal{L}_{\mathrm{compactness}}Lsparsity\mathcal{L}_{\mathrm{sparsity}}PSNR ↑
✗–––31.98
✓✗✓✓34.25
✓✓✗✓32.88
✓✓✓✗34.29
✓✓✓✓34.31

Table 3. Ablation Study. We evaluate the impact of various components in our method by systematically excluding them and analyzing the reconstruction quality (PSNR) on the “Kitchen 2” scene. We report average PSNR across the 3 evaluation splits and 5 independent runs.

Slot optimization. We next assess the importance of optimizing the initialized slot parameters. For this ablation, slots are initialized normally via VLM outputs but their parameters are kept fixed throughout training. The observed deterioration in reconstruction quality (Table 3) shows that differentiable slot optimization is important for refining the initial estimates according to photometric evidence.

Compactness regularization. The compactness loss encourages state-conditioned Gaussians associated with the same interaction to remain spatially localized, preventing them from dispersing across unrelated regions of the scene. This spatial prior therefore plays an important role in learning coherent state-dependent representations.

Sparsity regularization. Removing the sparsity regularization has a modest effect on average reconstruction quality. The regularizer nevertheless discourages unnecessary assignment of otherwise static Gaussians to interactive states, promoting a more parsimonious state decomposition and reducing spurious state-dependent changes.

Conclusion

We presented SwitchSplat, a 3D Gaussian Splatting framework that learns state-aware scene representations directly from casual video captures. By decomposing the scene into a shared static base and a set of independently controllable slots, each modeling the binary “on”/“off” states of one interactable object, SwitchSplat decouples scene state from time – enabling a capability that time-conditioned dynamic methods fundamentally cannot provide: free composition of object configurations at inference, including combinations never observed during capture. The slot table is initialized via a VLM and jointly refined end-to-end through differentiable rendering, requiring no manual annotation. Experiments on our new casual-capture dataset show that SwitchSplat outperforms static and dynamic baselines in novel-view synthesis quality, and uniquely generalizes to out-of-distribution camera paths and unseen state combinations.

Limitations and future work. The current model assumes binary object states with the constraints of single-operator capture with one free hand. Extending to continuous or multi-valued states by multiple operators – would broaden the applicability of the framework. The slot count and approximate interaction windows are currently seeded by a VLM; a fully unsupervised discovery of both the number and timing of interactions from raw video remains an open problem. Finally, SwitchSplat focuses on novel-view synthesis; combining our state-aware decomposition with physics-based priors or generative models could enable downstream applications such as scene editing and simulation.

Acknowledgments

We thank the anonymous reviewers for their helpful comments and suggestions. We also thank Zhihan Zhang, Egor Fokin, and Mingrui Zhao for contributing scenes and objects used in our dataset captures. We thank Lily Goli for her helpful advice and suggestions. This work was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant [2023-05617], the Canada Foundation for Innovation (CFI) John R. Evans Leaders Fund (JELF), the SFU Visual Computing Research Chair, Google Research, and the Digital Research Alliance of Canada.

Qualitative comparison of FullCircle, DeGauss, 4DGS, SwitchSplat, and Ground Truth across six scenes

Figure 6. Qualitative comparison. We compare FullCircle, DeGauss, 4DGS, and SwitchSplat against ground-truth images from standard held-out test views. Scenes are shown from top to bottom: Balcony, Balcony, Car, Flat 2, Flat 2, and Flat 3.

Qualitative comparison showing columns labeled FullCircle, DeGauss, 4DGS, SwitchSplat, and Ground Truth

Figure 7. Qualitative comparison. We compare FullCircle, DeGauss, 4DGS, and SwitchSplat against ground-truth images from standard held-out test views. Scenes are shown from top to bottom: Kitchen 1, Kitchen 1, Oslo, Flat 1, Flat 1, and Flat 3.

Supplementary Material

The supplementary materials are organized as follows. We first provide additional qualitative video results in the accompanying HTML supplement. To view them locally, open the provided index.html file in a modern web browser. Additional interactive results are available on our project page: https://switchsplat.github.io.

Dataset Capture and Preprocessing

We captured twelve real-world scenes using an Insta360 X4 dual-fisheye 360 camera, comprising ten standard scenes and two golden scenes. Video frames were extracted at 4 FPS. We use the raw fisheye frames for reconstruction to avoid propagating stitching artifacts into the scene representation. Camera-holder masks are automatically obtained using the FullCircle masking pipeline. These masks, together with the fisheye frames, are passed to COLMAP for camera pose estimation and subsequent image undistortion. Input images are downsampled by a factor of two. An overview of the object-interaction intervals and temporal structure of all twelve scenes is provided in Figure 18.

Figure 18. SwitchSplat dataset interaction events across 12 scenes featuring indoor, outdoor, rigid and non-rigid scenarios. Scenes contain 3–9 object-interaction episodes with various temporal structure: interactions may occur sequentially or overlap in time. Semantic labels shown are obtained through VLM predictions.

Benchmark Construction and Evaluation Splits

We evaluate reconstruction and generalization using four evaluation splits.

Base-state test frames. This split contains held-out frames in which all objects remain in their initial base state. It evaluates reconstruction of the canonical scene configuration and provides a natural comparison point for static reconstruction methods.

All standard test frames. For each scene, we hold out every eight frame from the original capture sequence. Frames occurring during object transitions are excluded, such that evaluation is performed only on stable object configurations. The remaining frames constitute our standard test set and cover the object states observed throughout the original capture.

For the two golden scenes, Kitchen 2 and Oslo, we additionally perform controlled recaptures under prescribed binary object states and new camera trajectories, as illustrated in Figure 8. These recaptures define two additional evaluation splits.

Figure 8. Golden-set benchmark construction. (a) The training sequence contains one interaction at a time, repeatedly returning to the base state. (b) We additionally recapture the scene under controlled object states using new camera trajectories. States observed during training evaluate novel camera trajectories, while previously unseen joint states evaluate novel state combinations.

Novel camera frames. We recapture object-state configurations observed during the original training sequence using new camera trajectories. This split evaluates generalization to novel viewpoints while keeping the scene configuration within the set observed during training.

Novel state-combination frames. We additionally recapture joint object-state configurations that never occurred during the training sequence, again using new camera trajectories. This split evaluates whether the learned representation can compose individually observed object states into previously unseen scene configurations.

Dynamic-baseline evaluation. DeGauss and 4DGS represent scene variation using a continuous timestamp and cannot directly condition on the prescribed state code of a Golden-set test image. For the novel-camera and novel state-combination splits, we therefore evaluate both methods using the same per-image oracle-over-time protocol. We first evaluate 30 uniformly spaced timestamps over

[0,1][0,1], then evaluate 8 additional timestamps within a ±0.03\pm0.03 neighborhood of the best coarse prediction. The rendering with the highest PSNR across both stages is retained. PSNR, SSIM, and LPIPS are computed from this same selected rendering; SSIM and LPIPS are not independently optimized. Since the ground-truth image is used to select the timestamp, these results should be interpreted as oracle upper bounds.

Additional Training Details

For fair comparison, all methods use corresponding undistorted pinhole images for both training and rendering, with the camera-holder exclusion masks mapped into the respective undistorted image representation.

We initialize our Gaussians from the COLMAP SfM point cloud and use the same densification and pruning schedule as 3DGUT. Gaussian parameters and state logits are optimized with fused Adam, while the slot-window endpoints and temperatures use a separate Adam optimizer; both optimizers are stepped jointly. SwitchSplat, FullCircle, and 4DGS are trained for 30,000 iterations with batch size 4. DeGauss uses batch size 4 with a 1,000-iteration coarse stage followed by a 30,000-iteration fine stage. The learning rates for opacity and state logits are both 5×10−25 \times10^{-2}.

The state-sparsity loss has weight 10−410^{-4}, starts at iteration 18,000, and ramps linearly over 1,000 iterations. The compactness loss has weight 5×10−45 \times10^{-4}, starts at iteration 21,000, and also ramps linearly over 1,000 iterations. For each slot, both states are measured relative to the opacity- and assignment-weighted centroid of the active state. After iterations 18,000 and 21,000, we mean-center the state logits and scale them by 0.3, reducing assignment confidence as the corresponding regularizer becomes active.

Full Quantitative Evaluation

For completeness, we report the full per-scene evaluation in terms of PSNR, SSIM, and LPIPS. Table 4 reports results on the 10 standard benchmark scenes for both the base-state subset and all standard test frames, while Table 6 reports results on the two Golden scenes across the base-state, standard-test, novel-camera, and novel state-combination splits. Across these evaluations, SwitchSplat achieves the strongest overall performance, with particularly consistent gains on the more challenging novel-camera and novel state-combination settings. Results with both 3DGS and 3DGUT further show that these improvements are not specific to a particular rendering backend. We provide further qualitative comparisons in Figure 16 and Figure 17.

MethodOutdoor: BalconyOutdoor: CarIndoor: Kitchen 1Indoor: Flat 1Indoor: Flat 2Indoor: Flat 3Indoor: ChairConcurrent indoor: M Kitchen 1Concurrent indoor: M Kitchen 2Concurrent indoor: M Kitchen 3Avg.
(a) Base-state test frames — PSNR ↑
FullCircle (3DGS)26.8727.8831.6829.3030.4030.7230.5729.7628.3227.3029.28
FullCircle (3DGUT)26.8927.7331.6230.3330.7431.5231.1530.5928.5428.1429.72
DeGauss26.9527.8631.9530.5431.1230.8234.5133.3931.3131.8731.03
4DGS26.7029.1931.3731.5830.6630.4832.6432.6531.8131.6930.88
SwitchSplat (3DGS)27.1928.8932.8233.8032.8934.1535.8933.7734.4134.1932.80
SwitchSplat (3DGUT)27.2828.9533.0134.2833.0934.4136.5234.1734.8034.5233.10
(a) Base-state test frames — SSIM ↑
FullCircle (3DGS)0.8560.8930.9670.9510.9610.9630.9770.9620.9500.9440.943
FullCircle (3DGUT)0.8560.8920.9680.9540.9620.9650.9780.9650.9500.9490.944
DeGauss0.8490.8560.9650.9520.9570.9590.9800.9710.9620.9650.942
4DGS0.8510.8890.9680.9600.9620.9620.9810.9750.9700.9700.949
SwitchSplat (3DGS)0.8580.8960.9720.9650.9670.9710.9860.9750.9750.9750.954
SwitchSplat (3DGUT)0.8590.8970.9720.9650.9670.9720.9860.9760.9750.9760.955
(a) Base-state test frames — LPIPS ↓
FullCircle (3DGS)0.2790.2310.2400.2770.2530.2630.1900.2430.2600.2670.250
FullCircle (3DGUT)0.2780.2320.2410.2750.2530.2610.1900.2420.2630.2630.250
DeGauss0.3060.2760.2510.2880.2660.2770.1890.2370.2500.2480.259
4DGS0.2840.2320.2450.2700.2560.2690.1880.2230.2290.2340.243
SwitchSplat (3DGS)0.2790.2310.2380.2620.2470.2530.1760.2300.2270.2290.237
SwitchSplat (3DGUT)0.2790.2310.2380.2610.2470.2530.1770.2290.2270.2290.237
(b) All standard test frames — PSNR ↑
FullCircle (3DGS)25.7026.8629.3229.2628.4628.9229.8830.1630.1330.1028.88
FullCircle (3DGUT)25.7726.7729.3529.2728.6229.1430.0630.3830.2330.3528.99
DeGauss25.8627.3129.6929.3630.1128.9134.1032.7232.2932.6830.30
4DGS26.3629.1430.8330.7830.1729.5430.9531.8432.7533.1430.55
SwitchSplat (3DGS)26.8928.9232.3232.8732.5833.3335.5533.2033.9734.4432.41
SwitchSplat (3DGUT)26.9729.0332.5533.1732.7833.6336.0433.5734.1734.5632.65
(b) All standard test frames — SSIM ↑
FullCircle (3DGS)0.8410.8810.9540.9470.9500.9540.9720.9590.9580.9570.937
FullCircle (3DGUT)0.8420.8810.9550.9470.9500.9540.9720.9590.9590.9590.938
DeGauss0.8330.8510.9510.9410.9490.9490.9780.9640.9640.9650.934
4DGS0.8460.8910.9620.9520.9560.9550.9730.9690.9700.9710.944
SwitchSplat (3DGS)0.8510.8940.9670.9590.9640.9670.9840.9690.9710.9730.950
SwitchSplat (3DGUT)0.8520.8950.9670.9590.9640.9680.9840.9700.9720.9740.951
(b) All standard test frames — LPIPS ↓
FullCircle (3DGS)0.2930.2470.2630.2900.2700.2790.2060.2510.2540.2670.262
FullCircle (3DGUT)0.2920.2480.2630.2940.2700.2800.2070.2510.2550.2660.263
DeGauss0.3200.2850.2700.3070.2740.2920.1990.2460.2470.2580.270
4DGS0.2910.2360.2520.2830.2590.2780.2050.2320.2320.2440.251
SwitchSplat (3DGS)0.2860.2370.2450.2730.2490.2590.1860.2370.2360.2450.245
SwitchSplat (3DGUT)0.2860.2370.2450.2730.2480.2590.1860.2360.2360.2450.245

Table 4. Full per-scene quantitative results on the standard benchmark. We report PSNR, SSIM, and LPIPS on (a) base-state test frames outside interaction intervals and (b) all held-out test frames. SwitchSplat with 3DGUT is our canonical configuration; the 3DGS variant evaluates effect of the rendering backend.

MethodKitchen 2 PSNR ↑Kitchen 2 SSIM ↑Kitchen 2 LPIPS ↓Oslo PSNR ↑Oslo SSIM ↑Oslo LPIPS ↓
Base-state test frames
FullCircle (3DGS)33.820.9720.17533.200.9760.190
FullCircle (3DGUT)33.880.9720.17532.840.9750.189
DeGauss34.370.9710.18434.970.9800.184
4DGS33.790.9750.17134.680.9810.178
SwitchSplat (3DGS)35.950.9790.16736.520.9840.178
SwitchSplat (3DGUT)36.170.9790.16736.730.9840.177
All standard test frames
FullCircle (3DGS)33.210.9690.18531.120.9670.205
FullCircle (3DGUT)33.310.9700.18631.300.9680.204
DeGauss34.210.9700.18732.870.9690.198
4DGS34.080.9750.17433.650.9770.183
SwitchSplat (3DGS)35.840.9780.17334.970.9790.184
SwitchSplat (3DGUT)35.980.9780.17335.180.9800.184
Novel camera frames
FullCircle (3DGS)32.250.9670.16628.330.9560.211
FullCircle (3DGUT)32.550.9670.16628.730.9570.210
DeGauss†32.380.9650.17429.400.9600.207
4DGS†32.050.9670.16829.930.9660.194
SwitchSplat (3DGS)33.900.9740.15531.820.9720.186
SwitchSplat (3DGUT)34.260.9750.15432.550.9730.185
Novel state-combination frames
FullCircle (3DGS)29.910.9550.17826.140.9320.216
FullCircle (3DGUT)30.100.9550.17726.440.9340.215
DeGauss†30.850.9580.18027.510.9400.214
4DGS†29.630.9560.18226.530.9400.211
SwitchSplat (3DGS)31.290.9620.16429.290.9590.186
SwitchSplat (3DGUT)32.580.9670.16129.100.9580.186

Table 6. Full quantitative evaluation on the Golden scenes. We report PSNR, SSIM, and LPIPS on Kitchen 2 and Oslo across base-state, standard-test, novel-camera, and novel state-combination frames. For the two OOD splits, DeGauss† and 4DGS† are evaluated using the oracle-over-time upper bound described in Section A.2, while SwitchSplat is evaluated directly from the requested state code. Best results are shown in bold.

Qualitative comparison of FullCircle, DeGauss, 4DGS, SwitchSplat, and ground-truth images

Figure 16. Qualitative comparison. We compare FullCircle, DeGauss, 4DGS, and SwitchSplat against ground-truth images from standard held-out test views. Scenes are shown from top to bottom: Car, Chair, Flat2, Kitchen 1, Kitchen 1, Flat 3.

Qualitative comparison of FullCircle, DeGauss, 4DGS, SwitchSplat, and ground-truth images

Figure 17. Qualitative comparison. We compare FullCircle, DeGauss, 4DGS, and SwitchSplat against ground-truth images from standard held-out test views. Scenes are shown from top to bottom: Flat2, Flat2, Kitchen 1, Kitchen 1, Kitchen 1, Flat 3.

VLM Performance

We use GPT-5.6 Sol to automatically initialize both the number of interaction slots and their approximate temporal extents. We uniformly sample the camera stream every 8 frames, resize each frame to 480 × 480, and render its frame index directly onto the image. The sampled frames are provided as separate, chronologically ordered

inputs to the VLM, which is prompted to identify distinct object-state episodes and return approximate on/off transition timestamps together with a confidence score. We discard predicted intervals shorter than 20 frames and apply confidence-ranked temporal non-maximum suppression with an IoU threshold of 0.8.

As shown in Figure 10, prediction quality generally improves with increased reasoning effort. At the maximum reasoning setting, the VLM recovers the complete event set perfectly in 6 out of 10 independent runs, i.e., achieving 100% event-set Jaccard. Moreover, when the event set is recovered correctly, the predicted transition times are highly accurate, with a mean absolute endpoint error within approximately two frames.

Figure 10. VLM performance across reasoning-effort levels. We evaluate 10 independent predictions at each reasoning-effort level. Top: event-set Jaccard measures whether the VLM recovers the correct set of interaction slots; bottom: endpoint MAE measures temporal localization error. Dots denote individual runs and diamonds denote the mean at each effort level. Increasing reasoning effort generally improves both event recovery and endpoint localization. At maximum reasoning effort, 6 of 10 runs recover the event set perfectly (100% Jaccard), with predicted transition endpoints localized to within only a few frames.

We further characterize the remaining prediction errors in Figure 11. Across approximately 3,000 predictions from five reasoning-effort levels, failures are sparse and are dominated by two modes: missing slots, where an interaction episode is not detected, and over-segmentation, where a single episode is split into multiple predictions. Incorrect merging, where two distinct interaction episodes are combined into a single predicted slot, occurs less frequently, while false-positive interactions are rare. At maximum reasoning effort, only a small number of such errors remain. Together, these results indicate that modern VLMs can provide a remarkably accurate and repeatable initialization for the discrete interaction structure required by SwitchSplat.

Figure 11. VLM slot-prediction failure modes. We aggregate approximately 3,000 predictions across five reasoning-effort levels, with 10 independent runs per level. Each bar reports the total number of occurrences of a failure mode, with colored segments indicating contributions from different reasoning-effort levels. Most errors arise from missing slots or over-segmentation, while incorrect merging is less common and false positives are rare. At higher reasoning effort, only a small number of failures remain, indicating that the VLM provides a highly reliable initialization in practice.

Prompt and output format. We use the same prompt template for all scenes. The complete prompt is reproduced verbatim in Figure 9. Scene-dependent quantities, such as the valid frame range, are inserted at runtime. The VLM is constrained to return structured JSON containing the predicted object, on/off transition frames, confidence score, uncertainty description, and supporting evidence for each detected event.

VLM prompt used for interaction-slot initialization

Figure 9. VLM prompt used for interaction-slot initialization. The prompt template is reproduced verbatim; only line wrapping is introduced for typesetting. Scene-dependent quantities, such as the valid frame range, are inserted at runtime.

Ablations on Slot Initialization

We examine how errors from slot initialization affect the subsequent optimization with two factors: the number of interaction slots and the temporal endpoints of each slot.

Number of slots. Table 5 varies the number of slots on Kitchen 2, for which the correct value is K=3K = 3. Removing slots causes a consistent degradation in reconstruction quality. By contrast, overestimating the slot count has little effects since extra slots can remain inactive during optimization.

Missing slotsExact slotsExtra slots
Number of slots (KK)0123456
PSNR ↑\uparrow33.3134.0535.1135.9835.9235.9835.96
SSIM ↑\uparrow0.970.970.980.980.980.980.98
LPIPS ↓\downarrow0.190.180.180.170.170.170.17

Table 5. Sensitivity to VLM-predicted slot count on the Kitchen 2 scene. The VLM predicts three interaction slots (K=3K=3). Performance degrades when slots are removed, while adding up to three extra slots has little effect.

Slot endpoints. We examine how accurately each interaction must be localized in time. Figure 12 perturbs the initialization of the middle slot in Kitchen 2 by independently varying its center and duration. Let cc denote the interval center and ww its duration. We parameterize slot center displacement by

Figure 12. Sensitivity of slot initialization on Kitchen 2. The heatmaps report final temporal IoU and test PSNR across endpoint shifts and scales. The bottom rows compare the ground truth, initialization, and final learned intervals for six marked settings. Cases A–C recover from substantial perturbations, while D–F show failures under more extreme perturbations.

δ=cinit−cGTwGT.\delta= \frac{c_{\mathrm{init}} - c_{\mathrm{GT}}}{w_{\mathrm{GT}}}.

and duration scales by

s=log⁡2winitwGT.s = \log_{2}\frac{w_{\mathrm{init}}}{w_{\mathrm{GT}}}.

All runs use 8×8\times image downsampling.

The resulting heatmaps reveal a broad region in which substantially different initial intervals converge to nearly the same learned solution. Within this region, both the final temporal IoU and test PSNR remain high despite large errors in the initial center or duration; cases A–C illustrate several such trajectories. Only for sufficiently extreme perturbations does optimization converge to an incorrect temporal interval, as illustrated by cases D–F.

Effect of Dynamic Moving Backgrounds

Our outdoor scenes contain substantial temporal variation unrelated to the discrete object-state changes targeted by SwitchSplat, including wind-driven vegetation, changing sky and clouds, and distant street activity. Such variation can disproportionately affect full-frame image metrics and partially obscure differences in the reconstruction quality of the interactive scene content. We therefore additionally evaluate all methods after excluding these moving-background regions, while otherwise retaining the same evaluation protocol. As shown in Table 7, under the original evaluation, SwitchSplat (3DGUT) achieves 29.03 dB compared with 29.14 dB for 4DGS, despite producing visibly cleaner reconstructions. After excluding the moving background, SwitchSplat improves to 32.87 dB, compared with 31.87 dB for 4DGS, while also achieving the best SSIM and LPIPS. A similar trend is observed on Balcony, where SwitchSplat remains the strongest method. These results suggest that the apparent discrepancy between quantitative and qualitative performance on the outdoor scenes is largely attributable to unrelated background dynamics.

MethodBalcony PSNR ↑Balcony SSIM ↑Balcony LPIPS ↓Car PSNR ↑Car SSIM ↑Car LPIPS ↓
Full evaluation
DeGauss25.860.8330.32027.310.8510.285
4DGS26.360.8460.29129.140.8910.236
SwitchSplat (3DGS)26.890.8510.28628.920.8940.237
SwitchSplat (3DGUT)26.970.8520.28629.030.8950.237
Moving background excluded
DeGauss26.340.8430.30330.580.9030.148
4DGS26.830.8520.27931.870.9240.119
SwitchSplat (3DGS)27.440.8600.27432.840.9320.111
SwitchSplat (3DGUT)27.500.8600.27432.870.9320.111

Table 7. Evaluation on outdoor scenes excluding moving backgrounds. We compare the original full-frame evaluation against an evaluation that excludes temporally varying background regions, including foliage, sky, clouds, and street activity. Results are reported on the full test set.

Comparison with Articulation baselines

SwitchSplat targets general object-state changes in full scenes, whereas ArticulatedGS [5] and SplArt [14] are designed specifically for part-level articulation of isolated objects. Their articulation-specific constraints prevent them from training

on our multi-object dataset. However, SwitchSplat can be directly applied to PARIS [15], the benchmark used by both ArticulatedGS and SplArt. As shown in Table 8, SwitchSplat achieves 47.62 dB, substantially outperforming ArticulatedGS (39.82 dB) and SplArt (43.70 dB). To rule out an evaluation discrepancy, we trained independent 3DGUT models on each articulation state separately, obtaining 47.21 dB; this sanity check confirms that the high PSNR achieved by SwitchSplat is indeed attainable. We attribute the gap to the additional constraints imposed by explicit articulation modeling, which create a harder optimization landscape at the expense of photometric reconstruction quality.

SceneArticulatedGSSplArtIndependent 3DGUTSwitchSplat (3DGUT)
USB41.6845.7049.1849.72
Blade41.3446.1550.8550.06
Folding chair42.2643.9850.2250.42
Fridge37.2641.4043.1044.18
Laptop36.2340.9045.5545.84
Oven36.6641.5943.0344.62
Scissors41.0646.1549.8149.73
Stapler43.1044.8150.5949.71
Storage38.3742.6243.2844.16
Washer40.3143.7546.5147.78
Average39.8243.7047.2147.62

Table 8. Quantitative comparison on the PARIS [15] dataset. We report PSNR (dB) across all ten articulated objects. For ArticulatedGS [5], we average the published start- and end-state PSNR for each object; SplArt [14] reports the corresponding per-object results directly. For 3DGUT [32] we report the average of models trained on separate states. Best and second-best results are shown in bold and underline, respectively.

Training on Native Fisheye Frames

This section and Section A.10 describe two extensions of the pipeline presented in the main paper. The main paper trains all methods on undistorted pinhole images. This choice serves only a fair comparison, since existing baselines do not support our fisheye setup. SwitchSplat itself can train directly on the raw fisheye frames, which cover a wider field of view.

To evaluate whether this is beneficial, we train one model on the pinhole frames of Kitchen 2 and one on its fisheye frames. We then evaluate both models on both camera types (Table 9). The pinhole frames are the COLMAP-undistorted fisheye frames used in the main paper, so the two datasets share camera poses, splits, and object timings.

Trained onTested on pinholeTested on fisheye
PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
Base-state test frames
Pinhole36.170.9790.16734.880.9790.090
Fisheye35.660.9780.16737.090.9830.084
All standard test frames
Pinhole35.980.9780.17334.120.9760.094
Fisheye35.490.9770.17436.590.9820.088
Novel camera frames
Pinhole34.260.9750.15434.630.9790.080
Fisheye34.630.9760.15235.740.9820.076
Novel state-combination frames
Pinhole32.580.9670.16133.560.9760.083
Fisheye33.120.9720.15634.690.9800.079

Table 9. Pinhole vs. fisheye training on Kitchen 2. Rows give the training camera, and columns give the test camera. Compare numbers within a column only, because the two columns score different images. Best result per column in bold. On fisheye views, both models sort Gaussians by distance from the camera. Each row is a single training run.

On the base-state and standard test frames, each model performs best on its own camera type, which produces a diagonal pattern in Table 9. These frames are close to the training views, so each model can fit its own image type closely. The two gaps differ in size, however. The fisheye-trained model is only about 0.5 dB behind on pinhole views, whereas the pinhole-trained model is more than 2 dB behind on fisheye views.

On novel cameras and novel state combinations, the settings that matter most for our task, the fisheye-trained model performs best in both columns. Part of its advantage on fisheye views comes from the outer rim of the lens, which a pinhole camera never observes. Figure 13 shows a qualitative example. We therefore recommend training on fisheye frames whenever they are available.

Qualitative comparison of ground truth, pinhole-trained, and fisheye-trained images tested on pinhole and fisheye cameras

Figure 13. Pinhole vs. fisheye training, qualitative example. A novel state-combination view of Kitchen 2, with all three objects switched at once. The top row shows the ground truth, the other rows give the training camera, and the columns give the test camera. The pinhole-trained model blurs the fridge door and leaves floaters in front of the oven and the floor, most visibly on the fisheye view. The fisheye-trained model is cleaner overall and closer to the ground truth.

Automatic Mask-Guided Slot Refinement

During an interaction, the video also records changes away from the object itself. Some changes are unrelated to the object, such as the reflection of the person who holds the camera on the steel fridge in Figure 14. Others are caused by the object but appear elsewhere, because a change of its state can alter the lighting of the surrounding scene, especially when the object is close to a light source. The Gaussians of the corresponding slot can learn to explain both kinds of change. When objects are switched later, these Gaussians appear as haze and floaters away from the object. The effect is strongest when many objects are switched at once (Figure 15).

Automatic object mask panels showing photo, render, SSIM difference, pseudo-labels, MLP mask, and final mask

Figure 14. Automatic object mask. The open drawer of Kitchen 2 in one training frame. (a) Photo. (b) Render with the drawer switched back to its closed state. (c) 1−SSIM1 - \mathrm{SSIM} between (a) and (b). (d) Pseudo-labels from thresholding (c): object (red) and background (blue). (e) Mask predicted by the MLP. (f) Final mask after SAM 2.1 propagation. Reflections on the steel fridge, including one of the person who holds the camera, also appear as change and produce wrong object pseudo-labels. The MLP inherits these errors and spills onto the fridge and the floor, whereas the propagated mask stays on the drawer.

Kitchen 1 novel view without and with mask fine-tuning

Figure 15. Mask fine-tuning removes haze and floaters. Kitchen 1 with all nine objects switched at once, seen from a new viewpoint with no ground truth. (Top) Without mask fine-tuning, haze and floaters appear in the scene. (Bottom) With mask fine-tuning, the scene is cleaner.

To address this problem, we compute an accurate mask for each object. We then briefly fine-tune the model, so that each slot keeps its Gaussians inside the object mask and discards those outside it.

Object masks. The mask pipeline is fully automatic and requires no human annotation. For each training frame during an interaction, we render the same view with the object switched back to its base state. We then compare this render with the photo. Thresholding this difference yields pseudo-labels: pixels with a large difference are labeled as object, and pixels with almost no difference as background. Following RoMo [4], we train a small MLP on these pseudo-labels, with DINOv3 [23] features and pixel positions as input. The MLP then predicts a mask for the entire frame.

Finally, SAM 2.1 [21] refines the masks with video propagation. It propagates the MLP masks of the odd frames to the even frames, and the masks of the even frames to the odd frames. Each final mask thus draws on neighboring frames rather than on a single prediction, which reduces errors that are specific to one frame. Each mask covers the whole region that changes with the object, including the space that the object reveals when it opens. Figure 14 illustrates these stages for one frame.

Mask fine-tuning. We fine-tune the fisheye model of Section A.9 for 3,000 iterations with one additional loss. This loss renders each object state in isolation and penalizes its opacity outside the object mask.

Results. Table 10 shows the expected trade-off. On the base-state and standard test frames, the scores decrease slightly, by less than 0.3 dB. The removed Gaussians also explained lighting effects in these held-out frames. On novel state combinations, where objects are switched independently, PSNR improves by 0.54 dB. Novel-camera scores remain unchanged. Figure 15 shows a novel view without and with mask fine-tuning.

SplitWithout mask fine-tuningWith mask fine-tuning
PSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
Base state37.090.9830.08436.920.9830.084
Standard test36.590.9820.08836.340.9810.089
Novel camera35.740.9820.07635.730.9820.076
Novel combination34.690.9800.07935.230.9810.078

Table 10. Mask fine-tuning improves novel state combinations. Kitchen 2, fisheye test views. The model without mask fine-tuning is the fisheye-trained model of Table 9. Best result per split in bold. Each column group is a single training run.

References

  1. [1]Volker Blanz and Thomas Vetter. 2023. A morphable model for the synthesis of 3D faces. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. 157–164.
  2. [2]Yunxiang Duan, Fangyin Wei, Qiu Dai, Yuhang He, Wenzheng Chen, and Baoquan Chen. 2024. 4d-rotor gaussian splatting: towards efficient novel view synthesis for dynamic scenes. In ACM SIGGRAPH 2024 Conference Papers. 1–11.arxiv.org/abs/2402.03307
  3. [3]Yalda Foroutan, Ipek Oztas, Daniel Rebain, Aysegul Dundar, Kwang Moo Yi, Lily Goli, and Andrea Tagliasacchi. 2026. FullCircle: Effortless 3D Reconstruction from Casual 360° Captures. arXiv preprint arXiv:2603.22572 (2026).arxiv.org/abs/2603.22572
  4. [4]Lily Goli, Sara Sabour, Mark Matthews, Marcus Brubaker, Dmitry Lagun, Alec Jacobson, David J. Fleet, Saurabh Saxena, and Andrea Tagliasacchi. 2025. RoMo: Robust Motion Segmentation Improves Structure from Motion. In Int. Conf. Comput. Vis.arxiv.org/abs/2411.18650
  5. [5]Junfu Guo, Yu Xin, Gaoyi Liu, Kai Xu, Lìang Liu, and Ruizhen Hu. 2025. ArticulatedS: Self-supervised digital twin modeling of articulated objects using 3d gaussian splatting. In IEEE Conf. Comput. Vis. Pattern Recog. 27144–27153.arxiv.org/abs/2503.08135
  6. [6]Wenhao Hu, Zesheng Li, Haonan Zhou, Liu Liu, Xuexiang Wen, Zhizhong Su, Xi Li, and Gaoang Wang. 2026. IGFuse: Interactive 3D Gaussian Scene Reconstruction via Multi-Scans Fusion. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 4932–4940.
  7. [7]Wenhao Hu, Haoran Zhou, Zesheng Li, Liu Liu, Jiacheng Dong, Zhizhong Su, and Gaoang Wang. 2025. RecurGS: Interactive Scene Modeling via Discrete-State Recurrent Gaussian Fusion. arXiv preprint arXiv:2512.18386 (2025).
  8. [8]Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojian Qi. 2024. Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. In IEEE Conf. Comput. Vis. Pattern Recog. 4220–4230.arxiv.org/abs/2312.14937
  9. [9]Umangi Jain, Ashkan Mirzaei, and Igor Gilitschenski. 2024. GaussianInt: Interactive segmentation via graph cut for 3d gaussian splatting. Adv. Neural Inform. Process. Syst. 37 (2024), 89184–89212.arxiv.org/abs/2411.07555
  10. [10]Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis, et al. 2023. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42, 4 (2023), 139–1.arxiv.org/abs/2308.04079
  11. [11]Deqi Li, Shi-Sheng Huang, Zhiyuan Lu, Xinran Duan, and Hua Huang. 2024b. St-4dgs: Spatial-temporally consistent 4d gaussian splatting for efficient dynamic scene rendering. In ACM SIGGRAPH 2024 Conference Papers. 1–11.DOI
  12. [12]Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. 2024a. Spatiotemporal gaussian feature splatting for real-time dynamic view synthesis. In IEEE Conf. Comput. Vis. Pattern Recog. 8508–8520.arxiv.org/abs/2312.16812
  13. [13]Yiqing Liang, Numair Khan, Zhengqin Li, Thu H Nguyen-Phuoc, Douglas Lanman, James Tompkin, and Lei Xiao. 2025. Gaufre: Gaussian deformation fields for real-time dynamic novel view synthesis. In Int. Conf. on App. Comput. Vis. 2642–2652.arxiv.org/abs/2312.11458
  14. [14]Shengjie Lin, Jiadang Fang, Muhammad Zubair Irshad, Vitor Campagnolo Guizilini, Rares Andrei Ambrus, Greg Shakhnarovich, and Matthew R. Walter. 2025. SplArt: Articulation Estimation and Part-Level Reconstruction with 3D Gaussian Splatting. arXiv preprint arXiv:2506.03594 (2025).arxiv.org/abs/2506.03594
  15. [15]Jiayi Liu, Ali Mahdavi-Amiri, and Manolis Savva. 2023. Paris: Part-level reconstruction and motion analysis for articulated objects. In Int. Conf. Comput. Vis. 352–363.arxiv.org/abs/2308.07391
  16. [16]Qingming Liu, Yuan Liu, Jiepeng Wang, Xianqiang Lyu, Peng Wang, Wenping Wang, and Junhui Hou. 2025b. Dmgs: Dynamic gaussian splatting from casually-captured monocular videos with depth priors. In Int. Conf. Learn. Represent., Vol. 2025. 97048–97074.arxiv.org/abs/2406.00434
  17. [17]Yu Liu, Baoxiong Jia, Ruijie Lu, Junfeng Ni, Song-Chun Zhu, and Siyuan Huang. 2025a. Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting. In Int. Conf. Learn. Represent.arxiv.org/abs/2502.19459
  18. [19]Nicolas Moenne-Loccoz, Ashkan Mirzaei, Or Perel, Riccardo De Iulio, Janick Martinez Estrou, Gavriel State, Sanja Fidler, Nicholas Sharp, and Zan Gojcic. 2024. 3d gaussian ray tracing: Fast tracing of particle scenes. ACM Trans. Graph. 43, 6 (2024), 1–19.
  19. [20]Atsuhiro Noguchi, Umar Iqbal, Jonathan Tremblay, Tatsuya Harada, and Orazio Gallo. 2022. What if supervised discovery of 3d joints for re-posing of articulated objects. In IEEE Conf. Comput. Vis. Pattern Recog. 3677–3687.
  20. [21]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. 2025. Sam 2: Segment anything in images and videos. In Int. Conf. Learn. Represent., Vol. 2025. 28085–28128.arxiv.org/abs/2408.00714
  21. [22]Sara Sabour, Lily Goli, Georgios Kopanas, Mark Matthews, Dmitry Lagun, Leonidas Guibas, Alec Jacobson, David J. Fleet, and Andrea Tagliasacchi. 2025. SpotlessSplats: Ignoring Distractors in 3D Gaussian Splatting. ACM Trans. Graph. (2025).arxiv.org/abs/2406.20055
  22. [23]Oriane Siméoni et al. 2025. DINOv3. arXiv preprint arXiv:2508.10104 (2025).arxiv.org/abs/2508.10104
  23. [24]Dai Sun, Huahao Guan, Kun Zhang, Xike Xie, and S Kevin Zhou. 2025. SDD-4DGS: Static-Dynamic Aware Decoupling in Gaussian Splatting for 4D Scene Reconstruction. arXiv preprint arXiv:2503.09332 (2025).arxiv.org/abs/2503.09332
  24. [25]Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 2024. 3dgstream: On-the-fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoint videos. In IEEE Conf. Comput. Vis. Pattern Recog. 20675–20685.arxiv.org/abs/2403.01444
  25. [26]Weiwei Sun, Andrea Tagliasacchi, Boyang Deng, Sara Sabour, Soroosh Yazdani, Geoffrey Hinton, and Kwang Moo Yi. 2021. Canonical Capsules: Self-supervised Capsules in Canonical Pose.
  26. [27]Andrea Tagliasacchi and Ben Mildenhall. 2022. Volume Rendering Digest (for NeRF). Technical Report.arxiv.org/abs/2209.02417
  27. [28]Diwen Wan, Yuxiang Wang, Ruijie Lu, and Gang Zeng. 2024. Template-free articulated gaussian splatting for real-time reposable dynamic view synthesis. Adv. Neural Inform. Process. Syst. 37 (2024), 62000–62023.arxiv.org/abs/2412.05570
  28. [29]Miaowei Wang, Yibo Zhang, Weiwei Xu, Rui Ma, Changqing Zou, and Daniel Morris. 2025. Decoupledgaussian: Object-scene decoupling for physics-based interaction. In IEEE Conf. Comput. Vis. Pattern Recog. 11361–11372.
  29. [30]Rui Wang, Quentin Lohmeyer, Mirko Meboldt, and Siyu Tang. 2025a. Degauss: Dynamic-static decomposition with gaussian splatting for distractor-free 3d reconstruction. In Int. Conf. Comput. Vis. 6294–6303.
  30. [31]Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 2024. 4d gaussian splatting for real-time dynamic scene rendering. In IEEE Conf. Comput. Vis. Pattern Recog. 20310–20320.arxiv.org/abs/2310.08528
  31. [32]Jiahao Wu, Rui Peng, Zhiyan Wang, Lu Xiao, Luyang Tang, Jinbo Yan, Kaiqiang Xiong, and Ronggang Wang. 2025b. Swift4d: Adaptive divide-and-conquer gaussian splatting for compact and efficient reconstruction of dynamic scene. arXiv preprint arXiv:2503.12307 (2025).arxiv.org/abs/2503.12307
  32. [35]Zeyu Yang, Hongye Yang, Zijie Pan, and Lei Zhang. 2024b. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. In ICLR, Vol. 2024. 9142–9159.arxiv.org/abs/2310.10642
  33. [36]Yuxin Yao, Zhi Deng, and Junhui Hou. 2025. Riggs: Rigging of 3d gaussians for modeling articulated objects in videos. In IEEE Conf. Comput. Vis. Pattern Recog. 5592–5601.arxiv.org/abs/2503.16822

Paper details

vlnc:W_rbdrk3u6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents