betaFeedback

WorksComputer Science

MOMASPLAT: Dynamic Novel-View Demonstration Augmentation for Mobile Manipulation

Abstract

Visuomotor policies trained through imitation learning are highly sensitive to changes in camera viewpoint. This is particularly problematic for mobile manipulation (MoMa), where robots rarely execute manipulation from the exact same base pose used during demonstration collection due to navigation uncertainty, localization drift, and environmental constraints. Recent approaches use novel-view synthesis (NVS) to augment demonstrations with unseen viewpoints, but they assume static scenes and therefore fail to re-render the manipulated object during the post-contact interaction phase, leaving the most critical parts of the demonstration unaugmented. We present MOMASPLAT, a novel-view demonstration augmentation framework for mobile manipulation that synthesizes full manipulation trajectories from unseen robot base poses, including dynamic object interactions. Our key insight is that standard MoMa data collection naturally provides the multiview observations needed to reconstruct both the static scene and the manipulated object in 3D. MOMASPLAT combines 3D Gaussian Splatting, feed-forward 3D foundation models, and kinematics-based action retargeting to separately reconstruct the scene and manipulated object, render them from novel viewpoints, and generate geometrically consistent synthetic demonstrations paired with transformed actions. We evaluate MOMASPLAT on five real-world mobile manipulation tasks involving both rigid and articulated objects across three scenes, and validate the fidelity of its synthesized demonstrations against ground truth in simulation. MOMASPLAT substantially improves robustness to unseen base poses, more than doubling the average success rate of the strongest baseline and maintaining non-trivial success at large base displacements where all baselines fail, while reducing the need for largescale real-world data collection.

INTRODUCTION

Imitation learning has emerged as a powerful paradigm for training visuomotor policies that perform complex manipulation tasks directly from image observations [1], [2]. However, these policies are notoriously sensitive to changes in camera viewpoint [3]–[5]. Even small deviations in camera pose can significantly degrade performance, since the policy learns a narrow visual distribution tied to the demonstrations used during training.

This sensitivity is particularly problematic for mobile manipulation (MoMa). Unlike tabletop manipulation setups, where demonstrations are often collected from a fixed robot camera pose, mobile manipulators rarely execute tasks from the exact same location at deployment time. Navigation

uncertainty, localization drift, obstacle avoidance, and imperfect planning routinely place the robot at different base poses relative to the workspace. As a result, the robot observes the manipulation scene from viewpoints that may differ substantially from those seen during demonstration collection. Robust deployment therefore requires visuomotor policies that generalize across a distribution of robot base poses rather than a single viewpoint.

A straightforward solution would be to collect demonstrations from many different robot locations. However, large-scale real-world teleoperation is expensive and difficult to scale [2], [6], [7]. Recent work has therefore explored synthesizing additional demonstrations through novel-view synthesis (NVS), generating virtual observations from unseen viewpoints using real-world captures [8]–[10]. While promising, existing approaches assume a static scene and therefore fail during the most important part of manipulation: object interaction. Since the manipulated object moves throughout the demonstration, prior NVS methods cannot consistently re-render the object from novel viewpoints during the post-contact interaction phase, restricting augmentation primarily to pre-contact motions or introducing geometrically inconsistent generations.

In this work, we present MoMASPLAT, a demonstra-

tion augmentation framework for mobile manipulation that synthesizes complete manipulation trajectories from unseen robot base poses, including dynamic object interactions (Figure 1). Our key insight is that standard mobile-manipulation data collection naturally provides the multi-view observations needed to reconstruct both the static scene and the manipulated object in 3D. During navigation, the robot observes the scene from diverse viewpoints that enable scene reconstruction, while the manipulation demonstrations themselves provide object-centric observations that enable reconstructing the manipulated object and its motion.

MOMASPLAT demonstration augmentation

Figure 1. MOMASPLAT. Starting from mobile-manipulation data collected at the original base pose (white and red robot), MoMASPLAT synthesizes demonstrations from sampled novel base poses (yellow-outlined gray robot). The top-right sequence shows the source manipulation demonstration, while the bottom-right shows the corresponding synthesized trajectory, including the dynamic object as it is moved by the robot. Augmenting the source manipulation data with these demonstrations enables execution from unseen deployment-time base poses (blue-outlined gray robot).

Building on this observation, MoMaSplat separately reconstructs the static scene and manipulated objects using 3D Gaussian Splatting [11] and feed-forward 3D foundation models [12]. Given a novel robot base pose, MoMaSplat re-renders the scene, manipulated object, and robot from the new viewpoint and retargets the original actions through kinematics-based transformations to generate geometrically consistent synthetic demonstrations.

We evaluate MoMaSplat on five real-world mobile manipulation tasks involving both rigid and articulated objects across three scenes, and validate the fidelity of its synthesized demonstrations against ground truth in simulation. Our experiments show that MoMaSplat substantially improves robustness to unseen deployment-time base poses, outperforming prior augmentation approaches across 12 unseen base poses (3 rollouts per pose) per task as the robot moves farther away from the source demonstration locations.

Our contributions are threefold:

  • A novel-view demonstration augmentation framework for mobile manipulation that synthesizes complete trajectories, including dynamic object interactions.

  • A geometry-consistent generation pipeline that separately reconstructs scenes and manipulated objects and retargets actions to unseen base poses.

  • Substantial improvements in robustness to deployment-time viewpoint variation across multiple real-world mobile manipulation tasks.

Our method proposes a solution to train imitation learning visuomotor policies for mobile manipulators via data augmentation. Here, we review the most relevant related work in those areas.

Demonstration Synthesis in Simulation. A prominent line of work synthesizes new demonstrations by replaying source actions in randomized simulated scenes. The X-Gen family [7, 13, 14, 15, 16] procedurally generates data by transforming and sequencing object-centric trajectory segments, with extensions such as MoMaGen [15] addressing mobile-manipulation constraints like base reachability and visibility. However, because the generated plans are purely kinematic, they typically rely on open-loop on-robot rollouts to verify feasibility and capture visual observations, at a cost that rivals manual data collection [8, 17]. A parallel line of work uses simulators to generate data from multiple viewpoints [18, 19, 20], but requires modeling the full environment and task dynamics in simulation, suffering from the visual and physical sim-to-real gap. Our work avoids on-robot rollouts and full-scene simulation by reconstructing the real scene and manipulated object via novel-view synthesis, using simulation only to render the known robot embodiment.

Demonstration Augmentation from Real-World Captures. Beyond 2D image augmentation [21, 22, 23], recent methods synthesize observations from real-world captures. Neural simulators [24, 25, 26] use generative video models but can suffer from hallucinations and geometric inconsistencies. Alternatively, novel-view synthesis (NVS) enables viewpoint augmentation [9, 10, 17, 27, 28, 29, 30, 31]. VISTA [27] uses ZeroNVS [32], while Yang et al. [9] employ 3D Gaussian Splatting [11]; however, both focus on third-person tabletop settings requiring multi-view pre-scans. Closer to our setting, One- Thousand Trajectories [8] supports moving eye-in-hand cameras, while EgoDemoGen [33] generates paired egocentric observations and actions using a generative model. However, these methods do not explicitly reconstruct dynamic objects and restrict augmentation to pre-contact motion. In contrast, MoMaSplat jointly models the static scene and manipulated objects, enabling kinematically consistent augmentation across both pre-contact and post-contact interaction phases for mobile robots with egocentric cameras.

Mobile Manipulation. Mobile manipulation remains a key challenge in robot learning [34, 35, 36, 37, 38], particularly in selecting robot base poses. Traditional methods use geometric heuristics such as Inverse Reachability Maps [39, 40, 41], while learning-based navigation [42, 43, 44, 45] often overlooks downstream policy sensitivity to viewpoint changes. Mobiπ [3] selects base poses matching the training distribution but requires pre-scanning and test-time optimization. In contrast, we train policies robust to diverse base poses, reducing the need for precise base placement at deployment.

MoMaSplat

We aim to augment a mobile-manipulation dataset with synthetic demonstrations from unseen robot base poses. From source demonstrations at a few robot locations, MoMaSplat synthesizes observations and actions for nearby deployment-time poses never physically demonstrated.

To achieve this, MoMaSplat decomposes the environment into three components: i) a static scene model, ii) a dynamic object model for the manipulated object, and iii) the robot embodiment itself. The scene and object models are reconstructed independently and aligned into a shared metric frame. A key observation is that the navigation trajectories naturally provide the bridge between both reconstructions: during navigation, the manipulated objects remain static, allowing the object-centric and scene-centric trajectories to be consistently registered.

For augmentation, given a sampled robot base pose, MoMaSplat renders the scene, manipulated object, and robot from the corresponding viewpoint and retargets the demonstrated actions through kinematic transformations. The resulting observations and transformed actions form synthetic demonstrations that augment the dataset for policy training. Figure 2 provides an overview of the full process.

Figure 2. Overview of MOMASPLAT. 3D Reconstruction: We build 3D models of the environment from the navigation and manipulation data. We estimate camera poses via structure-from-motion and fit a 3D Gaussian Splatting model of the static scene, Gscene\mathcal{G}_{\mathrm{scene}}. For the dynamic objects, we segment each manipulated object and estimate its per-frame pose trajectory with a feed-forward 3D foundation model; since the scene and object poses lie in separate reference frames, we align them with a Sim(3) transformation and fit a separate object model Gobj\mathcal{G}_{\mathrm{obj}}. Synthesizing Demonstrations from Unseen Base Poses: Given a robot base pose, T~\widetilde{T}, sampled around the original demonstration poses (left), we render the scene and object from the new viewpoint, transform the source end-effector trajectory into the new base frame to render the robot from a simulator, and composite the three streams into a photorealistic novel observation. Paired with the transformed actions a~1:Lj\widetilde{a}_{1:L_j}, this forms a synthetic demonstration; repeating across sampled poses yields the generated dataset. Imitation Learning: the policy is trained on the union of source and generated demonstrations.

Problem Formulation. We consider an imitation learning setting on a mobile manipulator operating in a scene. The robot alternates between two modes of operation: navigation, which changes the robot base pose, and manipulation, which executes arm actions from a fixed base location. We represent the robot base pose as a transform wbT∈SE(2){}^{b}_{w}T \in SE(2) with respect to a stationary world frame ww, parameterized by a planar position (xb,yb)(x_b, y_b) and heading θb\theta_b. The robot observes the scene through an egocentric RGB camera mounted on the robot body (e.g., head or torso). The camera pose relative to the robot base is determined by the robot configuration through forward kinematics, such that both changes in base pose and articulation of the robot body induce changes in viewpoint.

During data collection, the robot navigates to a predefined set of NN target base poses:

T={wbT1,…,wbTN},\mathcal{T} = \left\{{}^{b}_{w}T_1,\ldots,{}^{b}_{w}T_N\right\},

from which manipulation demonstrations are collected. The robot executes NnavN_{\mathrm{nav}} navigation trajectories from different starting locations toward the target base poses, either teleoperated or autonomously, and performs MM manipulation demonstrations from each fixed base pose without further base motion.

This process produces two complementary datasets. First, we collect a navigation dataset Dnav={dnav1,…,dnavNnav}\mathcal{D}_{\mathrm{nav}} = \left\{d_{\mathrm{nav}}^1,\ldots,d_{\mathrm{nav}}^{N_{\mathrm{nav}}}\right\}, where NnavN_{\mathrm{nav}} denotes the number of navigation trajectories and each trajectory dnavk=((I1,q1),…,(ILk,qLk))d_{\mathrm{nav}}^k = \left((I_1,q_1),\ldots,(I_{L_k},q_{L_k})\right) contains RGB observations and robot states recorded while navigating toward one of the target base poses. These trajectories provide diverse multi-view observations of the environment and manipulated objects before interaction. Second, we collect a source manipulation dataset Dsrc={dsrc1,…,dsrcNsrc}\mathcal{D}_{\mathrm{src}} = \left\{d_{\mathrm{src}}^1,\ldots,d_{\mathrm{src}}^{N_{\mathrm{src}}}\right\} of Nsrc=N×MN_{\mathrm{src}} = N \times M demonstrations, where each trajectory dsrcj=((I1,q1,a1),…,(ILj,qLj,aLj))d_{\mathrm{src}}^j = \left((I_1,q_1,a_1),\ldots,(I_{L_j},q_{L_j},a_{L_j})\right) contains manipulation observations and actions recorded during task execution from one of the NN base locations. We represent actions as 7-D commands at=(pt,ot,gt)a_t = (p_t,o_t,g_t), consisting of desired end-effector translation and rotation deltas pt∈R3p_t \in\mathbb{R}^3 and ot∈SO(3)o_t \in SO(3) applied in the robot base frame, together with the gripper command gt∈Rg_t \in\mathbb{R}.

Our goal is to augment the source manipulation dataset Dsrc\mathcal{D}_{\mathrm{src}} with a synthetic dataset Dgen\mathcal{D}_{\mathrm{gen}} of demonstrations generated at unseen base poses to obtain a base-pose robust policy trained on D=Dsrc∪Dgen\mathcal{D} = \mathcal{D}_{\mathrm{src}} \cup\mathcal{D}_{\mathrm{gen}}. To achieve this, MoMaSplat first reconstructs the scene and manipulated objects in 3D and aligns them into a shared metric frame (Sec. III-A). It then synthesizes demonstrations from sampled base poses by rendering the scene, object, and robot from the corresponding viewpoint and retargeting the actions and proprioception to remain geometrically consistent (Sec. III-B). The synthesized demonstrations are combined with the source dataset for policy training (Sec. III-C).

3D Reconstruction

To synthesize demonstrations from unseen robot base poses, MoMaSplat requires a 3D representation that can be rendered from arbitrary viewpoints. A central challenge is that manipulation involves dynamic objects whose pose changes throughout the demonstration. Unlike prior novel-view synthesis approaches that model the environment as a single static scene, MoMaSplat reconstructs the static scene and manipulated objects separately, leveraging observations naturally produced during mobile-manipulation data collection. Navigation trajectories cover the environment while the objects are static, and manipulation trajectories capture the objects in motion.

3D Scene Model Reconstruction. As the robot moves through the workspace, its camera naturally captures diverse viewpoints, providing the multi-view coverage needed for 3D reconstruction without external cameras or scanning. We therefore use Dnav\mathcal{D}_{\mathrm{nav}} to build a photorealistic 3D scene model: we recover per-image camera poses with structure-from-motion via COLMAP [46], which also produces a sparse point cloud that initializes a 3D Gaussian Splatting (3DGS) reconstruction [11].

We optimize the Gaussians by minimizing a photometric loss between rendered and observed images. Since the scene is not perfectly static across navigation trajectories, transient content may appear between demonstrations; we adopt the uncertainty-aware formulation of WildGaussians [47], which down-weights inconsistent regions during optimization to yield a robust model Gscene\mathcal{G}_{\mathrm{scene}}. We additionally mask out pixels belonging to the manipulated object and the robot during scene reconstruction, preventing the object from being baked into Gscene\mathcal{G}_{\mathrm{scene}} and avoiding a static “ghost” when the independently reconstructed dynamic object is later composited into the scene.

Structure-from-motion reconstructs the scene only up to an unknown global scale, while our framework transforms metric actions and therefore requires a metric reconstruction. We recover the scale using the robot’s onboard depth sensor: for the frames in Dnav\mathcal{D}_{\mathrm{nav}}, we compare the measured depth maps to depth maps rendered from Gscene\mathcal{G}_{\mathrm{scene}} and take the median per-pixel ratio of measured to rendered depth as the scale factor.

3D Object Model Reconstruction. Since the object of interest moves independently from the scene, we build a separate 3DGS model for it. Conceptually, the steps mirror those for the scene: we estimate the object’s pose in each frame and use these poses to optimize a 3D Gaussian reconstruction. However, the challenges here are different. Correspondence-based methods [46] are poorly suited to small, weakly-textured objects, while instance-level pose trackers [48] require an existing 3D model of the object, which is not available to us. Instead, we find that our setting is amenable to recent feed-forward 3D foundation models, like VGGT [12, 49], which estimate camera poses directly from an unposed RGB sequence.

Specifically, for each trajectory dnavk∈Dnavd_{\mathrm{nav}}^k \in\mathcal{D}_{\mathrm{nav}} and source demonstration dsrcj∈Dsrcd_{\mathrm{src}}^j \in\mathcal{D}_{\mathrm{src}}, we extract the RGB frames

{Itk}t=1Lk\{I^k_t\}_{t=1}^{L_k} and {Itj}t=1Lj\{I^j_t\}_{t=1}^{L_j}, respectively, and obtain per-frame segmentation masks Mtk\mathcal{M}^k_t and Mtj\mathcal{M}^j_t for the object using an open-world segmentation pipeline [50, 51], which masks out the background and the robot.

Then, we apply FastVGGT [49] to the object-masked frames {Itk⊙Mtk}\{I^k_t \odot\mathcal{M}^k_t\} and {Itj⊙Mtj}\{I^j_t \odot\mathcal{M}^j_t\}, respectively. With everything else masked out, the model predicts a camera pose at each frame that is expressed relative to the object itself. These object-relative camera poses T^o,t∈SE(3)\hat{T}_{o,t} \in SE(3) act as the initial pose estimates in the model’s internal coordinate frame vv.

Frame Alignment. The estimates T^o,t\hat{T}_{o,t} lie in vv, with arbitrary scale, orientation, and origin relative to the metric world frame ww established by the Scene Reconstruction. To align the two frames, we use a set of frames from the navigation demonstrations Dnav\mathcal{D}_{\mathrm{nav}} as a bridge. During navigation the object remains static, so the only motion comes from the robot itself, i.e., the pose trajectory recovered by the object model is purely due to camera motion. This is not true for the manipulation frames: the object moves, so the recovered trajectory conflates camera and object motion. On the selected navigation frames, the same cameras are posed twice, by the feed-forward model in vv and by structure-from-motion in ww. The two trajectories must describe the same motion up to an unknown Sim(3) transformation. We recover this transformation S:v→wS:v\rightarrow w in closed form via

the Umeyama algorithm [52]. Applying SS to the estimated poses yields the aligned pose trajectory in the world frame, To,t=S⋅T^o,tT_{o,t}=S\cdot\hat{T}_{o,t}.

Object 3DGS Optimization. Finally, we jointly optimize these aligned poses To,tT_{o,t} alongside the Gaussian parameters of Gobj\mathcal{G}_{\mathrm{obj}}, backpropagating the photometric loss through both. This yields a pose-disentangled representation Gobj\mathcal{G}_{\mathrm{obj}} that can be rendered at arbitrary 6-DoF poses for compositing into novel-pose observations. For articulated objects, we model the moving rigid link separately from the static remainder of the object. For example, a laptop lid or the basket of an air fryer is represented by Gobj\mathcal{G}_{\mathrm{obj}}, while the keyboard base or the air fryer’s surrounding body remains part of the static scene. The component represented by Gobj\mathcal{G}_{\mathrm{obj}} therefore undergoes a rigid 6-DoF motion over the demonstrated trajectory, and we do not estimate articulation parameters or extrapolate beyond the demonstrated motion.

Synthesizing Demonstrations from Unseen Base Poses

Given the scene and object models, our framework synthesizes new demonstrations at unseen base poses in three stages. For each original robot base pose, wTiwT_i, we first sample a set of target base poses that cover possible deployment-time locations while remaining kinematically feasible (e.g., reachable for the arm). We then synthesize each source demonstration from each sampled pose by composing three separately rendered streams: the static scene (from the 3D scene model), the manipulated objects (from the 3D object model), and the robot body as seen from the new base location. Finally, we retarget the source action to be defined with respect to the virtual sampled base location to obtain a new pair of virtual image and action label.

Sampling Base Poses. To generate synthetic demonstrations, we sample a set of target base poses Tisampled={T~1,…,T~K}T_i^{\mathrm{sampled}} = \{\widetilde{T}_1, \ldots, \widetilde{T}_K\} (here, we omit the superindex bb and subindex ww from bwT~{}_b^w\widetilde{T} for simplicity) that the policy should learn to handle around each of the original base locations, TiT_i. We sample the planar position (x~j,y~j)(\widetilde{x}_j, \widetilde{y}_j) uniformly within a region surrounding the original base pose, chosen to cover the range of deployment-time poses the robot may reach. For each sampled position, we set the base heading θ~j\widetilde{\theta}_j so that the object of interest is centered in the egocentric camera’s field of view, then perturb it with uniform noise to expose the policy to a range of object-in-frame configurations. We then apply the transformation ΔjiT=(T~j)−1⋅Ti\Delta_j^iT = (\widetilde{T}_j)^{-1} \cdot T_i that transforms the original base frame, TiT_i, to the sampled base frame, T~j\widetilde{T}_j (where we lift ΔjiT\Delta_j^iT from SE(2)SE(2) into SE(3)SE(3) with zero vertical translation and zero roll/pitch when transforming 3D poses and vectors). We retain a sampled base pose, T~j\widetilde{T}_j, only if the source arm trajectory, where the end-effector remains at the same pose with respect to the manipulated object, admits a valid inverse-kinematics solution at several probe timesteps spanning the demonstration; infeasible poses are rejected and resampled. To obtain the end-effector pose in the sampled base frame, we apply ΔjiT\Delta_j^iT to the source end-effector pose beeTi{}_b^{ee}T_i, obtained from qq through forward kinematics. The resulting pose, beeT~j=ΔjiT⋅beeTi{}_b^{ee}\widetilde{T}_j = \Delta_j^iT \cdot{}_b^{ee}T_i, preserves the end-effector pose in the world frame.

Novel View Synthesis. We virtually place the egocentric camera at its mounted offset relative to T~j\widetilde{T}_j using forward kinematics and the robot’s recorded configuration, qq, and matching the physical camera’s intrinsics. We then render three elements from the new view: i) the scene, ii) the manipulated object, and iii) the robot. From the scene model, Gscene\mathcal{G}_{\mathrm{scene}}, we obtain a virtual RGB image I~scene\widetilde{I}_{\mathrm{scene}}, the depth map D~scene\widetilde{D}_{\mathrm{scene}} and the accumulated opacity α~scene\widetilde{\alpha}_{\mathrm{scene}}. For the manipulated object, we transform Gobj\mathcal{G}_{\mathrm{obj}} to its source-demonstration pose To,tT_{o,t} and rasterize it from the same camera, yielding per-timestep renders I~obj\widetilde{I}_{\mathrm{obj}}, depth maps D~obj\widetilde{D}_{\mathrm{obj}} and opacity image α~obj\widetilde{\alpha}_{\mathrm{obj}}. For the robot, we use inverse kinematics on the end-effector pose for the sampled base location, beeT~j{}_b^{ee}\widetilde{T}_j, to recover a joint configuration, q~tr\widetilde{q}_t^r, that would place the end-effector at the same location with respect to the object. Then, we apply the joint configuration to the robot and render it using OmniGibson [53] to obtain a robot-only RGB image I~robot\widetilde{I}_{\mathrm{robot}}, depth map D~robot\widetilde{D}_{\mathrm{robot}} and opacity mask α~robot\widetilde{\alpha}_{\mathrm{robot}}.

To get a single RGB observation from the three virtual images, we need to combine them together, while respecting the depths and the opacities at each pixel. To achieve this, we use per-pixel alpha compositing. Specifically, at each pixel, we sort the three layers by depth, letting (ci,αi)(c_i,\alpha_i) denote the color and opacity of the ii-th layer, with i=1i = 1 being the nearest. Then, we combine them in a front-to-back manner:

c=∑i=13ciαi∏j=1i−1(1−αj),c = \sum_{i=1}^{3} c_i \alpha_i \prod_{j=1}^{i-1}(1-\alpha_j),

so that nearer surfaces occlude farther ones. This generates virtual RGB images as would be seen from a physical camera at sampled location T~j\widetilde{T}_j.

Robot Action and Proprioception Retargeting. A change in base pose moves the coordinate frame in which both the robot’s end-effector proprioception and actions are expressed, so both must be retargeted to the sampled base frame bwT~j{}_b^w\widetilde{T}_j. Because the policy conditions on end-effector proprioception in the robot base frame (consisting of the end-effector pose beeTi{}_b^{ee}T_i and gripper state), we transform the end-effector pose into the sampled base frame via beeT~j=ΔjiT⋅beeTi{}_b^{ee}\widetilde{T}_j = \Delta_j^iT \cdot{}_b^{ee}T_i (as above) while keeping the gripper state unchanged. Robot actions at=(pt,ot,gt)a_t = (p_t,o_t,g_t) consist of desired end-effector translation deltas pt∈R3p_t \in\mathbb{R}^3, rotation deltas ot∈SO(3)o_t \in SO(3), and gripper commands gt∈Rg_t \in\mathbb{R}, where both translation and rotation deltas are defined and applied in the robot base frame (i.e., beeRt+1=R(ot)beeRt{}_b^{ee}R_{t+1} = R(o_t){}_b^{ee}R_t). To retarget the desired end-effector translation delta, we apply the rotation component of ΔjiT\Delta_j^iT so that p~t=RΔjiT⋅pt\widetilde{p}_t = R_{\Delta_j^iT} \cdot p_t expresses the same world-frame displacement in the sampled base frame. Because rotation deltas are applied in the base frame, conjugating R(ot)R(o_t) by RΔjiTR_{\Delta_j^iT} rotates its axis of rotation by RΔjiTR_{\Delta_j^iT} while preserving the rotation angle; in axis-angle format ot=(u^t,ϕt)o_t = (\hat{u}_t,\phi_t), the retargeted rotation delta is therefore o~t=(RΔjiT⋅u^t,ϕt)\widetilde{o}_t = (R_{\Delta_j^iT} \cdot\hat{u}_t,\phi_t). The gripper command remains unchanged (g~t=gt\widetilde{g}_t = g_t), yielding the retargeted action a~t=(p~t,o~t,gt)\widetilde{a}_t = (\widetilde{p}_t,\widetilde{o}_t,g_t). Pairing the virtual RGB images with the retargeted proprioception and actions a~tj\widetilde{a}_t^j across the trajectory provides a synthetic demonstration at bwT~j{}_b^w\widetilde{T}_j.

Constructing the Augmented Dataset. For each of the Nsrc=N×MN_{\mathrm{src}} = N \times M source manipulation demonstrations in Dsrc\mathcal{D}_{\mathrm{src}}, we generate KK synthetic demonstrations from KK sampled novel base poses, producing a synthetic dataset Dgen\mathcal{D}_{\mathrm{gen}} of K⋅NsrcK \cdot N_{\mathrm{src}} demonstrations that we combine with Dsrc\mathcal{D}_{\mathrm{src}} to form D=Dsrc∪Dgen\mathcal{D} = \mathcal{D}_{\mathrm{src}} \cup\mathcal{D}_{\mathrm{gen}} for policy training (Sec. III-C).

Imitation Learning

We train a visuomotor policy on the union of source demonstrations Dsrc\mathcal{D}_{\mathrm{src}} and synthetic demonstrations Dgen\mathcal{D}_{\mathrm{gen}} using Diffusion Policy [1], which models the conditional action distribution as a denoising diffusion process and is trained by minimizing the standard noise prediction loss:

L=Ek,ϵ,(o,a)∼D[∥ϵ−ϵθ(o,ak,k)∥2].(1)\mathcal{L} = \mathbb{E}_{k,\epsilon,(o,a)\sim\mathcal{D}}\left[\left\|\epsilon-\epsilon_\theta(o,a^k,k)\right\|^2\right]. \tag*{(1)}

where aka^k is the ground-truth action sequence noised at denoising step kk with Gaussian noise ϵ\epsilon. The policy conditions on the RGB observation and end-effector proprioception, and outputs an action sequence of end-effector pose deltas. To balance the two datasets during training, we co-train with a sampling ratio of 0.25 for Dsrc\mathcal{D}_{\mathrm{src}} and 0.75 for Dgen\mathcal{D}_{\mathrm{gen}}, anchoring the policy to the source data while leveraging the broader pose coverage of the synthetic data.

Experimental Evaluation

We evaluate MoMaSplat on five real-world mobile manipulation tasks across three scenes, comparing against four baselines: no augmentation and three augmentation methods. Our experiments focus on a) whether MoMASPLAT improves policy success at unseen base poses, b) how performance varies with the distance between the deployment-time base pose and the closest demonstration pose, and c) how policy success changes with the number of source demonstrations available for augmentation. Additionally, we evaluate the quality of the generated data itself. This requires ground truth observations at poses where no demonstration was collected, which is impractical in the real world. A human teleoperator produces a different motion on every attempt, so even a re-collected trajectory cannot serve as ground-truth for the generated demonstration. Therefore, we develop one task in simulation, where camera and object poses are known and source trajectories can be replayed from arbitrary base poses.

Experimental Setup

Tasks. The robot performs (i) PICK-PLACE BOTTLE and (ii) OPEN DRAWER in the desk scene; (iii) PICK-PLACE REMOTE and (iv) CLOSE LAPTOP in the living room; and (v) OPEN AIR-FRYER in the kitchen (Figure 3). We additionally build (vi) SIM PICK-PLACE BOTTLE in OmniGibson [53], a simulated scene modeled after the desk environment that follows the same task structure as (i) without replicating its exact geometry. The tasks span rigid pick-place and single-DoF articulated manipulation. We collect 75 manipulation demonstrations for CLOSE LAPTOP and 90 for each other task across three base poses, forming DsrcD_{\mathrm{src}}. We additionally collect nine navigation trajectories DnavD_{\mathrm{nav}} from different starting poses to the target base poses.

Scenes and tasks used in the experiments

Figure 3. Tasks and scenes used in our experiments. We evaluate MoMASPLAT on five real-world manipulation tasks across three scenes (top: scenes; bottom: per-task close-ups), covering manipulation of both rigid and articulated objects. The simulated scene and its task, SIM PICK-PLACE BOTTLE (right of the dashed lines), are used to measure the fidelity of MoMASPLAT’s generated demonstrations against ground truth.

Evaluation Protocol. For each task, we deploy the trained policy at 12 unseen base poses across the scene, with heading centered on the object, and perform 3 rollouts per pose.

Baselines. We compare against four baselines. Source Only (labeled “Original data only” in Figure 4) trains Diffusion Policy [1] without augmentation. Action Augmentation [54] adds noise to proprioceptive states and actions while preserving observations. VISTA [27] generates novel-view images without transforming actions. One-Thousand Trajectories [8] reconstructs the static scene using 3DGS; we additionally provide our action transformation to account for changes in the robot base frame. All methods train on the same source demonstrations. For PICK-PLACE BOTTLE, we restrict synthetic generation for One-Thousand Trajectories to demonstrations whose initial object pose matches the reconstructed scene, since its static reconstruction cannot represent variation in this pose.

Figure 4. Policy success across base poses. For each task, we evaluate 12 unseen base locations distributed across categorical distance bins that vary per task, with 3 trials per pose, and report success rate by distance to the nearest demonstration pose. The shaded gray region denotes the original base poses; locations 5–50 cm away are unseen deployment poses. MoMaSplat maintains substantially higher success as the base pose shifts away from the source demonstrations (where Source Only is labeled “Original data only”).

Data Augmentation Setup. Each method generates 10 synthetic demonstrations per source demonstration, yielding 750–900 synthetic demonstrations per task except One-Thousand Trajectories on PICK-PLACE BOTTLE, which generates 400. For MoMASPLAT and One-Thousand Trajectories, we sample base poses within a 1.2×1.21.2 \times1.2 m region around each source pose (Sec. III-B). Frame alignment uses 50 frames from one navigation demonstration.

Results

Performance at Unseen Base Poses. Our main experimental results for policy success using MoMASPLAT are summarized in Figure 4, where success rates are binned by their minimum distance to the nearest demonstration pose. We additionally evaluate the 3 original demonstration base poses with 15 rollouts per pose. At the original base poses, all methods perform comparably. The picture changes sharply at unseen poses. When we average the results over all five tasks, MoMASPLAT succeeds on 37% of unseen evaluations, more than twice the rate of the strongest baseline (One-Thousand Trajectories at 17%) and four to six times that of the remaining baselines (Source Only 6%, VISTA 8%, Action Augmentation 6%). Critically, Source Only and Action Augmentation collapse to near-zero success beyond 10 cm on most tasks, confirming that perturbing actions without changing the observation gives little robustness to base-pose shifts. VISTA performs poorly throughout; since it is designed for static-base settings, it does not transform actions to match the synthesized viewpoints. Moreover, its lack of temporal consistency produces jittery camera trajectories that further degrade the augmented data. One-Thousand Trajectories degrades more gracefully but still falls off at large distances, since it augments only the free-space approach motion. MoMASPLAT achieves the highest overall unseen-pose success rate on every task, outperforming or matching all baselines across every unseen distance bin while sustaining substantial performance at the largest evaluated distances. For example, MoMASPLAT achieves 33% on PICK-PLACE BOTTLE at 30 cm and 50% on CLOSE LAPTOP at 40 cm, where every baseline has dropped to zero success rates. Finally, to better contextualize the geometry of our tasks and scenes, Figure 5 shows the spatial layout of these evaluation locations across each workspace.

Spatial distribution of policy evaluation poses

Figure 5. Spatial distribution of policy evaluation poses. Top-down view of each task showing the table and target-object locations, the original demonstration base poses, and the test base poses used for policy evaluation. Original demonstration poses are marked with circles, while evaluation poses are marked with squares; arrows indicate the robot heading at each pose. Evaluation poses are colored by MoMaSplat policy success rate, providing spatial context for the distance-based results in Figure 4.

Effect of Number of Source Demonstrations. To evaluate how the number of source demonstrations affects MoMASPLAT, we use the PICK-PLACE BOTTLE task and train policies using 10, 50, and 90 source demonstrations. For each source demonstration, we generate 10 synthetic demonstrations. Across four test locations at a distance of 10–20 cm from the demonstration poses, we observe that success degrades gracefully with fewer source demonstrations.

Specifically, we achieve success rates of 33%, 42%, and 50% for 10, 50, and 90 source demonstrations, respectively. For reference, training on the full set of 90 source demonstrations without augmentation yields only an 8% success rate at the same test poses. These results show that MoMaSplat substantially improves base-pose robustness even with a limited number of source demonstrations.

Generation Quality. We first evaluate generation quality in simulation, where the ground-truth counterpart of a generated demonstration is obtained by executing its transformed actions in the simulator and rendering the on-board camera at every timestep. We use the SIM PICK-PLACE BOTTLE task and evaluate 75 generated demonstrations, taking every tenth frame for a total of 2,486 frames. Unlike the policy evaluations above, which measure distance between base poses, here we measure how far each generated view is from the views that were used to build the reconstruction. For the 75 demonstrations, we find this distance to be between 0.4 and 27.3 cm. Since the static reconstruction does not change over a demonstration, we score the scene separately by rendering the scene splat and the empty simulated scene from the same camera pose once per demonstration, rather than considering every frame. We report PSNR, SSIM, and LPIPS over the full image, the scene, and over the robot and object regions using the simulator’s segmentation masks. Table I reports the detailed results. The reconstructed scene achieves the highest fidelity at 26.40 dB, followed by the manipulated object at 18.18 dB and the robot at 17.43 dB. For reference, the original 3DGS work [11] reports roughly 23 dB PSNR on Tanks & Temples [55]. This places our scene reconstruction in the range of standard novel-view synthesis, while the lower object PSNR reflects the harder setting of reconstructing a moving object with limited viewpoint coverage. The robot error is mainly caused by lighting and shadow differences introduced during compositing. We also evaluate object-pose accuracy to measure how well our estimated object trajectory matches the true object motion of the observations. On 30 source demonstrations, including poses directly registered by VGGT and interpolated poses, the predicted object pose has a mean translation error of 1.46 ± 0.71 cm and rotation error of 4.01° ± 4.66° relative to the simulator ground truth.

On real data, we evaluate scene reconstruction by masking out the robot and the manipulated object, and we obtain 24.00 dB PSNR on 110 held-out frames. For the manipulated object, we use its bounding-box crop and we mask out the background. At poses directly registered by VGGT, we obtain 23.63 dB, while at interpolated poses we obtain 22.59 dB. We do not report full-image view-synthesis metrics on real data because the simulated robot differs in appearance from the real robot and this appearance mismatch dominates the pixel-wise error that the view synthesis metrics evaluate.

RegionPSNR (dB) ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
Full image21.27±1.1221.27 \pm1.120.902±0.0190.902 \pm0.0190.253±0.0300.253 \pm0.030
Scene26.40±2.1826.40 \pm2.180.936±0.0210.936 \pm0.0210.248±0.0370.248 \pm0.037
Object18.18±2.1518.18 \pm2.150.688±0.0810.688 \pm0.0810.341±0.1060.341 \pm0.106
Robot17.43±1.3017.43 \pm1.300.812±0.0570.812 \pm0.0570.287±0.0480.287 \pm0.048

Table 1. Generated-observation fidelity against simulator ground truth. Mean ±\pm std over 75 generated demonstrations spanning viewpoints with distances of 0.4–27.3 cm from the reconstruction views. Full image, object, and robot use every tenth frame (2486 total frames), with object and robot restricted to their masks; scene PSNR evaluation uses one frame per demonstration (75 in total).

Overall, the simulation results show that MOMASPLAT produces observations that closely match the ground truth, while the real-world results show high-fidelity scene reconstruction and consistent object reconstruction across both directly estimated and interpolated poses.

Conclusion and Limitations

Conclusion. We propose MOMASPLAT, a framework for novel-view augmentation of mobile manipulation demonstrations that reconstructs the static scene and manipulated object in 3D, then re-renders full demonstrations, including the post-contact interaction phase, from unseen base poses. Unlike prior NVS augmentation methods that model only the static scene, MOMASPLAT produces geometrically consistent observations across the entire trajectory. Across five real-world tasks in three scenes, this improves policy robustness, outperforming prior augmentation methods and maintaining performance beyond the distances where baselines fail. We will publicly release our code to support further research.

Limitations. MOMASPLAT assumes mostly static scenes and rigid, visible objects, making deformable objects, heavy occlusions, and texture-poor surfaces challenging. Furthermore, because GsceneG_{\mathrm{scene}} is reconstructed from pre-interaction navigation views with object masks excluded, regions occluded by the object in its initial state (such as the surface beneath a lifted bottle or the interior of an opened drawer or air-fryer basket) are never observed during scene reconstruction and remain unmodeled when disoccluded. It also depends on consistent open-world segmentation and produces non-physical demonstrations by replaying source object trajectories without modeling contact forces or dynamics. Finally, view quality depends on navigation-data coverage. Addressing dynamic backgrounds, deformable objects, physical contact, and cluttered scenes remains future work.

Acknowledgment

This work was conducted in the Robot Interactive Intelligence (RobIn) Lab at The University of Texas at Austin and was supported in part by the National Science Foundation (CAREER Award 2543745), the NVIDIA Academic Grant Program, Amazon, and DARPA TIAMAT program (HR0011-24-9-0428). G. Pavlakos acknowledges support by NSF-2504906 and 2544200, and the NVIDIA Academic Grant Program. S. Kumar is supported in part by the Amazon AI PhD Fellowship program. The opinions, findings, and conclusions expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.

Appendix

Baseline Details

In this section, we provide more details about the baselines we are using in our evaluation.

Source Only. This baseline trains the same Diffusion Policy [1] as we do, but using only the source demonstrations DsrcD_{\mathrm{src}}, without any augmentation. This allows us to quantify how well a policy trained only on the original base poses generalizes to unseen poses.

Action Augmentation. Following [54], we augment each demonstration by perturbing the proprioceptive state and the action labels with zero-mean Gaussian noise while leaving the image observations unchanged. This exposes the policy to small action variations, but because the images remain the same, this augmentation provides no information about how the scene looks from a different base pose. We use this data to train the same Diffusion Policy and use the same training schedule as we do with all other baselines.

VISTA. VISTA [27] augments a demonstration by re-rendering its observations from perturbed camera viewpoints with ZeroNVS [32], a zero-shot single-image novel-view-synthesis diffusion model, while leaving the actions unchanged. We run the publicly released ZeroNVS model and checkpoint.1 For each source demonstration, we sample a small set of viewpoint perturbations: a planar camera offset (dx,dy)(dx,dy) drawn uniformly within ±10\pm10 cm on each axis (the first sample is the unperturbed view), together with a single per-demonstration yaw that re-aims the camera at the object centroid, clipped to ±0.5\pm0.5 rad; beyond this perturbation range, we observe large artifacts in image generation and worse imitation learning results. We then process each frame independently: conditioning on the original egocentric RGB image (center-cropped and resized to 256×256256 \times256) and the relative perturbation, ZeroNVS synthesizes the frame from the novel viewpoint. We sample with DDIM (50 steps, η=0\eta= 0), a classifier-free guidance scale of 7.5, and a 70° field of view matching our camera. Because ZeroNVS is a single-image model applied frame by frame, the synthesized views are not temporally consistent across a trajectory. The generated frames are paired with the original, unmodified actions and are used to train the same Diffusion Policy as all other methods.

One-Thousand Trajectories. One-Thousand Trajectories [8] reconstructs the static scene as a 3D Gaussian Splatting model and re-renders observations from novel viewpoints. In our implementation, we reconstruct the static scene using the same pipeline and navigation data DnavD_{\mathrm{nav}} as GsceneG_{\mathrm{scene}} in MoMaSplat, but without masking out the manipulated object, so that the static scene model contains the manipulated object at its initial pose.

We use its novel-view augmentation component without the additional trajectory augmentation proposed in [8]. To account for changes in the robot base frame, we apply the same delta-based action retargeting used by MoMaSplat (Sec. III-B). This baseline relies on a static scene reconstruction rather than explicitly reconstructing the dynamic manipulated object. For Pick-Place Bottle, we restrict synthetic generation to the 40 demonstrations whose initial object pose matches the reconstructed scene, producing 400 synthetic demonstrations. All 90 source demonstrations remain in the policy-training set. For the other tasks, the initial object pose is fixed, so all source demonstrations are used for synthetic generation.

Method Details

3D Scene Model Reconstruction

Structure from motion and Gaussian Splatting. We reconstruct the static scene from the navigation frames DnavD_{\mathrm{nav}}. We run structure-from-motion using COLMAP [46] with a sequential matcher over the navigation frames, together with a small set of context frames sampled just before each manipulation. This step returns per-image camera poses and a sparse point cloud. The sparse points are used to initialize a 3D Gaussian Splatting reconstruction that we train with WildGaussians [47].2 We use the nerfbaselines framework and train for 30,000 iterations. WildGaussians augments 3DGS with per-image appearance and exposure embeddings and a DINOv2-based uncertainty head that down-weights transient or inconsistent regions, such as the robot or moving objects, during optimization. We use a DSSIM weight of 0.2 in the photometric loss and hold out every eighth frame for evaluation.

Metric scaling. Structure-from-motion recovers the scene only up to an unknown global scale. However, our action retargeting is metric, so we rescale the reconstruction using the robot’s onboard depth sensor measurements. For each navigation frame we render a depth map from the scene model and compare it to the measured depth from the head-mounted RGB-D camera (stored in millimeters and converted to meters). We consider only pixels with valid measured depth (in (0,9](0,9] m, at least 1000 per frame) and for each frame, we take the median of the per-pixel ratios of measured to rendered depth. Eventually, the scale factor is set as the median of these per-frame median values, making it robust to outliers. This factor allows us to convert rendered depth and Gaussian coordinates into meters.

Scene pose interpolation. For the navigation frames that have not been registered by COLMAP, we estimate their scene camera poses using the same strategy as we do with objects (see Sec. B-2). This includes SLERP-based interpolation, smoothness filtering, and cyclic-outlier rejection, applied per demonstration.

3D Object Model Reconstruction

Segmentation. To obtain the per-frame object masks Mtj\mathcal{M}_{t}^{j}, we use Grounded-SAM [50], prompting an open-world detector, GroundingDINO [56], with a per-object text label to localize the manipulated object, and propagating the detected box through the clip using SAM 2 [51].3 We use a forward and a reverse pass from a per-object seed frame. When multiple objects are tracked, pixels claimed by more than one object are removed from all of them (exclusive masks). Object reconstruction uses the per-object masks to exclude the background and robot. For scene reconstruction, we additionally obtain a robot mask by rendering the robot in the simulator at the recorded configuration and camera pose. We exclude the union of the object masks and this robot mask when training Gscene\mathcal{G}_{\mathrm{scene}}.

Frame Selection Strategy. A source manipulation demonstration dsrc∈Dsrcd_{\mathrm{src}} \in\mathcal{D}_{\mathrm{src}} contains several hundred RGB frames recorded at video rate, far more than the feed-forward model can process. Simultaneously, near-duplicate frames add little to the pose estimate while inflating compute. We therefore subsample each demonstration to a compact, motion-diverse set of roughly 10–20 frames before passing it to the model, using the per-frame object masks MtjM^j_t.

We first discard frames whose object mask is small or poorly segmented, removing views where the object is distant or occluded. From the remaining frames we drop near-duplicate, stationary views: scanning in temporal order, we keep a frame only when its mask bounding-box IoU with the previously kept frame falls below 0.8 (the same threshold for all tasks), or when more than 20 frames have elapsed since the last kept frame. The first and last frame of every demonstration are always retained to anchor the reconstruction to the endpoints of the clip. Additionally, 50 frames from one navigation demonstration are included for frame alignment, and the combined set is capped at the feed-forward model’s 400-frame input budget.

Bundle Adjustment. For our feed-forward model, we use VGGT [12]4, as implemented by the training-free FastVGGT [49] acceleration5. For each selected frame, it predicts an object-relative camera pose T^o,t\hat{T}_{o,t} together with a per-frame intrinsics matrix and a dense depth map. These predictions are accurate but not multi-view consistent, so we refine them with a bundle-adjustment step. We first unproject the predicted depth into a set of 3D points and predict dense 2D feature tracks across the selected frames. Initializing a reconstruction with the predicted poses, intrinsics, and points, we filter tracks by reprojection error, using an adaptive threshold that grows until a sufficient fraction of frames retain enough inliers, and by track visibility, requiring every 3D point to be observed in at least two frames. We then run bundle adjustment, jointly optimizing the camera poses and 3D points (and, when intrinsics are not supplied, the focal length and principal point) to minimize reprojection error. The refined, multi-view-consistent poses replace T^o,t\hat{T}_{o,t} as the input to the Sim(3) alignment.

Frame Alignment. The poses estimated above lie in the object model’s internal frame vv with arbitrary scale, orientation, and origin, whereas the scene lives in the world frame ww. We align them using the navigation frames, where the object is static so its recovered trajectory is explained purely by the camera motion. Effectively, the same cameras are posed twice: by the object model in vv and by structure-from-motion in ww. Matching the navigation cameras by frame, we fit the similarity transform S:v→wS:v \to w from their camera centers with the Umeyama algorithm [52], solving for scale, rotation, and translation in closed form. To be robust to mis-registered frames, we wrap Umeyama in RANSAC: we repeatedly fit SS from minimal 5-camera subsets, reject hypotheses whose scale leaves a plausible range, score each by the number of camera centers whose residual is below 0.2 m, keep the highest-consensus hypothesis, and refit on its inliers followed by a Huber-robust nonlinear refinement restricted to those inliers. We report the alignment quality as the camera-center residual (mean and median) and the RMS of the aligned sparse-point residuals. Applying SS maps the object poses into the world frame, To,t=S⋅T~o,tT_{o,t}=S\cdot\tilde{T}_{o,t}.

Object Gaussian Optimization. Given the registered frames and their aligned poses To,tT_{o,t}, we fit a 3D Gaussian Splatting model [11] Gobj\mathcal{G}_{\mathrm{obj}} of the object with the gsplat [57] trainer6 for 60,000 steps, supervising only the segmented object through its RGBA images, where the alpha channel is the object mask composited over a white background. We minimize an L1L_1 plus SSIM photometric loss (SSIM weight 0.2) together with an L1L_1 term on the rendered opacity against the mask. The per-frame poses are optimized jointly with the Gaussian parameters at a small learning rate of 10−510^{-5}, so the photometric loss is backpropagated through both appearance and pose, yielding the pose-disentangled model Gobj\mathcal{G}_{\mathrm{obj}} that can be rendered at arbitrary 6-DoF poses.

Pose Interpolation. Because of frame selection, the feed-forward model only returns camera poses at the selected keyframes. In contrast to that, synthesizing a demonstration requires an object pose To,tT_{o,t} at every timestep of the source trajectory. We therefore densify the keyframe poses by interpolation: rotations are interpolated with spherical linear interpolation (SLERP) and translations are interpolated linearly, between consecutive keyframes. To suppress residual estimation noise we add three refinements before interpolating. (i)(i) A smoothness filter drops keyframes whose translation and rotation relative to the previous keyframe both fall below small thresholds, removing jitter from near-stationary stretches. (ii)(ii) A cyclic-outlier filter removes back-tracking A→B→AA \to B \to A keyframes, where an excursion to BB is large enough to be meaningful yet the trajectory returns close to AA, a signature of transient pose-estimation errors. (iii)(iii) An ease-in parameterization places interpolated samples so that pose changes are concentrated toward the end of each keyframe interval, better matching the observed motion profile than uniform piecewise-linear filling. The densified trajectory {To,t}\{T_{o,t}\} is what we rasterize Gobj\mathcal{G}_{\mathrm{obj}} along when compositing novel-view observations.

Base Pose Sampling Strategy

Here, we expand on the procedure used to draw the sampled base poses Tisampled={T~1,…,T~K}\mathcal{T}^{\mathrm{sampled}}_i=\{\tilde{T}_1,\ldots,\tilde{T}_K\} around each original base pose TiT_i (Sec. III-B). Sampling operates per source demonstration about an anchor pose recovered for that demonstration, i.e., the original base position and heading expressed in the world frame, as it is established by the scene reconstruction.

Planar position. We draw the planar offset (x~j,y~j)(\tilde{x}_j,\tilde{y}_j) from a distribution centered at the anchor. By default this is uniform over a square region around the original base pose (the region size used in our experiments is reported in Sec. IV-A); we also support an isotropic Gaussian with magnitude bounds for annulus-shaped coverage.

Heading. For each sampled position we set the base heading θ~j\tilde{\theta}_j so that the egocentric camera points at the object’s 3D centroid (accounting for the camera’s fixed offset from the base), then perturb it with uniform noise (default ±20∘\pm20^\circ) so the policy sees the object at a range of positions in the camera view. The robot head is fixed, so the object is kept in view through the base pose alone; we discard samples that would place the object near the image edge or out of view before running the costlier IK check.

IK feasibility. Following Sec. III-B, a sampled base pose is retained only if the source arm trajectory, with the end effector held at the same pose relative to the manipulated object, admits a valid inverse-kinematics solution. We verify reachability at several probe timesteps spanning the demonstration and accept the sample only if all of them are reachable, querying the curobo inverse-kinematics solver for the TIAGo robot inside the OmniGibson simulator [53]. Infeasible candidates are rejected and resampled, up to a fixed draw budget per demonstration, and the first KK accepted samples are kept.

Imitation Learning

We provide implementation details for the policy trained with Diffusion Policy [1] on the union of source and generated demonstrations (Sec. III-C).

Architecture. The policy is a conditional 1D U-Net denoiser. The egocentric head-camera RGB image is encoded by a ResNet-18 trained from scratch followed by a spatial-softmax pooling layer (32 keypoints), and the resulting visual features are concatenated with the low-dimensional proprioception to form the conditioning vector.

Observations and actions. At each step the policy conditions on the egocentric head-camera RGB image (resized to 224 ×\times 224) and the end-effector proprioception, stacked over the most recent nobs=2n_{\mathrm{obs}} = 2 steps. It predicts a chunk of nact=8n_{\mathrm{act}} = 8 future actions, each a 7-dimensional command consisting of an end-effector translation delta, a rotation delta, and a gripper command expressed in the robot base frame, normalized to zero mean and unit variance. During training we apply random crop and translation together with color-jitter augmentation to the RGB input.

Diffusion process. We use a DDIM noise scheduler with 50 training timesteps, a cosine-based β\beta schedule, and ϵ\epsilon (noise) prediction, optimizing the mean-squared denoising loss of (1). At inference, actions are produced by running the reverse process for the same number of denoising steps.

Optimization. We optimize with AdamW (learning rate 3×10−43 \times10^{-4}, weight decay 10−410^{-4}) under a cosine schedule with 1000 warm-up steps and a batch size of 32 for 100,000 gradient steps.

Source and synthetic co-training. We train on the union of the source demonstrations Dsrc\mathcal{D}_{\mathrm{src}} and the generated demonstrations Dgen\mathcal{D}_{\mathrm{gen}} using a weighted sampler that draws each training sequence from Dsrc\mathcal{D}_{\mathrm{src}} with probability 0.25 and from Dgen\mathcal{D}_{\mathrm{gen}} with probability 0.75. This anchors the policy to the source data while exploiting the broader base-pose coverage of the synthetic data.

Hardware and data. We use a TIAGo mobile manipulator with a head-mounted RGB-D camera (70° field of view). We collect 75 source demonstrations for CLOSE LAPTOP and 90 for each other task across three base poses, and generate 10 synthetic demonstrations per source demonstration within a 120 ×\times 120 cm region around each source base pose. We evaluate each policy at 12 unseen base poses with 3 rollouts per pose. Table II summarizes the main hyperparameters across the pipeline.

StageParameterValue
Scene SfMmatchersequential (COLMAP)
Scene 3DGSmethod / iterationsWildGaussians / 30,000
Scene 3DGSDSSIM weight0.2
Metric scalevalid-depth range(0, 9] m, ≥1000\ge1000 px/frame
Segmentationdetector / trackerGroundingDINO / SAM2
Frame selectionIoU thresh. / max gap0.8 / 20 frames
Frame selectionVGGT input budget400 frames per object
Object BAvisibility per 3D point≥2\ge2 views
Sim(3) RANSACsubset / trials / inlier5 / 8000 / 0.2 m
Object 3DGSsteps / SSIM weight60,000 / 0.2
Object 3DGSpose-opt learning rate10−510^{-5}
Base samplingregion / yaw noise120 ×\times 120 cm / ±20∘\pm20^\circ
Generationsynthetic per source demonstration10
Policyobs / action horizon2 / 8
PolicyDDIM train timesteps50
Policyoptimizer / learning rateAdamW / 3×10−43 \times10^{-4}
Policybatch size / training steps32 / 100,000
Policysource : generated ratio0.25 : 0.75

Table II. Key hyperparameters across the MoMaSPLAT pipeline.

References

  1. [1]C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion Policy: Visuomotor policy learning via action diffusion,” in RSS, 2023.
  2. [2]T. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” in RSS, 2023.arxiv.org/abs/2304.13705
  3. [3]J. Yang, I. Huang, B. Vu, M. Bajracharya, R. Antonova, and J. Bohg, “Mobi-π: Mobilizing your robot learning policy,” in CoRL, 2025.
  4. [5]S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei et al., “LIBERO-Plus: In-depth robustness analysis of vision-language-action models,” arXiv preprint arXiv:2510.13626, 2025.arxiv.org/abs/2510.13626
  5. [6]Z. Fu, T. Z. Zhao, and C. Finn, “Mobile ALOHA: Learning bimanual mobile manipulation with low-cost whole-body teleoperation,” in CoRL, 2024.arxiv.org/abs/2401.02117
  6. [7]A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox, “MimicGen: A data generation system for scalable robot learning using human demonstrations,” in CoRL, 2023.arxiv.org/abs/2310.17596
  7. [8]C. Pan, L. Liang, D. Bauer, E. Cousineau, B. Burchfiel, S. Feng, and S. Song, “One demo is worth a thousand trajectories: Action-view augmentation for visuomotor policies,” in CoRL, 2025.arxiv.org/abs/2606.19586
  8. [9]S. Yang, W. Yu, J. Zeng, J. Lv, K. Ren, C. Lu, D. Lin, and J. Pang, “Novel demonstration generation with Gaussian splatting enables robust one-shot manipulation,” arXiv preprint arXiv:2504.13175, 2025.arxiv.org/abs/2504.13175
  9. [10]A. Zhou, M. J. Kim, L. Wang, P. Florence, and C. Finn, “NeRF in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis,” in CVPR, 2023, pp. 17 907–17 917.arxiv.org/abs/2301.08556
  10. [11]B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3D Gaussian splatting for real-time radiance field rendering,” ACM Trans. Graph., vol. 42, no. 4, pp. 139:1–139:14, 2023.
  11. [12]J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny, “VGGT: Visual geometry grounded transformer,” in CVPR, 2025, pp. 5294–5306.arxiv.org/abs/2503.11651
  12. [13]Z. Jiang, Y. Xie, K. Lin, Z. Xu, W. Wan, A. Mandlekar, L. J. Fan, and Y. Zhu, “DexMimicGen: Automated data generation for bimanual dexterous manipulation via imitation learning,” in ICRA, 2025, pp. 16 923–16 930.arxiv.org/abs/2410.24185
  13. [14]C. Garrett, A. Mandlekar, B. Wen, and D. Fox, “SkillMimicGen: Automated demonstration generation for efficient skill learning and deployment,” in CoRL, 2024.arxiv.org/abs/2410.18907
  14. [15]C. Li, M. Xu, A. Bathey, H. Yin, Y. Jiang, H. Huang, J. Wong, S. Garlanka, C. Gokmen, R. Zhang et al., “MoMaGen: Generating demonstrations under soft and hard constraints for multi-step bimanual mobile manipulation,” in ICLR, 2026.arxiv.org/abs/2510.18316
  15. [16]K. Lin, V. Ragunath, A. McAlinden, A. Prasad, J. Wu, Y. Zhu, and J. Bohg, “Constraint-preserving data generation for visuomotor policy learning,” in CoRL, 2025.arxiv.org/abs/2508.03944
  16. [17]Z. Xue, S. Deng, Z. Chen, Y. Wang, Z. Yuan, and H. Xu, “DemoGen: Synthetic demonstration generation for data-efficient visuomotor policy learning,” in RSS, 2025.arxiv.org/abs/2502.16932
  17. [18]F. Sadeghi, A. Teshn, E. Jang, and S. Levine, “Sim2Real viewpoint invariant visual servoing by recurrent control,” in CVPR, 2018, pp. 4691–4699.DOI
  18. [19]N. Hirose, D. Shah, A. Sridhar, and S. Levine, “ExAug: Robot-conditioned navigation via geometric experience augmentation,” arXiv preprint arXiv:2210.07450, 2022.arxiv.org/abs/2210.07450
  19. [20]Y. Seo, J. Kim, S. James, K. Lee, J. Shin, and P. Abbeel, “Multi-view masked world models for visual robotic manipulation,” in ICML, 2023, pp. 30 613–30 632.arxiv.org/abs/2302.02408
  20. [21]M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas, “Reinforcement learning with augmented data,” in NeurIPS, vol. 33, 2020, pp. 19 884–19 895.
  21. [22]D. Yarats, I. Kostrikov, and R. Fergus, “Image augmentation is all you need: Regularizing deep reinforcement learning from pixels,” in ICLR, 2021.
  22. [23]N. Hansen, R. Jangir, Y. Sun, G. Alenyà, P. Abbeel, A. A. Efros, L. Pinto, and X. Wang, “Self-supervised policy adaptation during deployment,” arXiv preprint arXiv:2007.04309, 2020.arxiv.org/abs/2007.04309
  23. [24]J. Jang, S. Ye, Z. Lin, J. Xiang, J. Bjorck, Y. Fang, F. Hu, S. Huang, K. Kundalia, Y.-C. Lin et al., “DreamGen: Unlocking generalization in robot learning through video world models,” arXiv preprint arXiv:2505.12705, 2025.arxiv.org/abs/2505.12705
  24. [25]J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mayankalnar, R. Steigerwald, C. Apps et al., “Genie: Generative interactive environments,” in ICML, 2024.arxiv.org/abs/2402.15391
  25. [26]N. Agarwal, A. Ali, M. Bala, Y. Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y. Chen, Y. Cui, Y. Ding et al., “Cosmos world foundation model platform for physical AI,” arXiv preprint arXiv:2501.03575, 2025.arxiv.org/abs/2501.03575
  26. [27]S. Tian, B. Wulfe, K. Sargent, K. Liu, S. Zakharov, V. Guizilini, and J. Wu, “View-invariant policy learning via zero-shot novel view synthesis,” in CoRL, 2024.arxiv.org/abs/2409.03685
  27. [28]A. Tagliabue and J. P. How, “Tube-NeRF: Efficient imitation learning of visuomotor policies from MPC using tube-guided data augmentation and NeRFs,” arXiv preprint arXiv:2311.14153, 2023.arxiv.org/abs/2311.14153
  28. [29]L. Y. Chen, C. Xu, K. Dharmarajan, M. Z. Irshad, R. Cheng, K. Keutzer, M. Tomizuka, Q. Vuong, and K. Goldberg, “RoVi-Aug: Robot and viewpoint augmentation for cross-embodiment robot learning,” in CoRL, 2024.arxiv.org/abs/2409.03403
  29. [30]S. Wang, H. Dong, J. Tian, J. Li, Z. Yang, T. Cao, A. Chen, S. Wu, L. Wang, and S. Zhou, “Efficient camera pose augmentation for view generalization in robotic policy learning,” arXiv preprint arXiv:2603.29192, 2026.arxiv.org/abs/2603.29192
  30. [31]X. Zhang, M. Chang, P. Kumar, and S. Gupta, “Diffusion meets DAgger: Supercharging eye-in-hand imitation learning,” in RSS, 2024.arxiv.org/abs/2402.17768
  31. [32]K. Sargent, Z. Li, T. Shah, C. Herrmann, H.-X. Yu, Y. Zhang, E. R. Chan, D. Lagun, L. Fei-Fei, D. Sun et al., “Zero-shot 360-degree view synthesis from a single image,” in CVPR, 2024, pp. 9420–9429.arxiv.org/abs/2310.17994
  32. [33]Y. Xu, J. Yang, X. Wang, Y. Chen, Z. Zhu, B. Fang, G. Huang, X. Chen, Y. Ye, Q. Zhang et al., “EgoDemoGen: Novel egocentric demonstration generation enables viewpoint-robust manipulation,” arXiv preprint arXiv:2509.22578, 2025.
  33. [34]A. Bahety, A. Balaji, B. Abbate mat teo, and R. Martín-Martín, “SafeMimic: Towards safe and autonomous human-to-robot imitation for mobile manipulation,” in RSS, 2025.arxiv.org/abs/2506.15847
  34. [35]J. Wu, R. U. An, K. Lan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, “TidyBot: Personalized robot assistance with large language models,” Autonomous Robots, 2023.DOI
  35. [36]P. Liu, Y. Orru, C. Paxton, N. M. M. Shafiullah, and L. Pinto, “OK-Robot: What really matters in integrating open-knowledge models for robotics,” in RSS, 2024.DOI
  36. [37]Y. Jiang, R. Zhang, J. Wong, C. Wang, Y. Ze, H. Yin, C. Gokmen, S. Song, J. Wu, and L. Fei-Fei, “BEHAVIOR robot suite: Streamlining real-world whole-body manipulation for everyday household activities,” arXiv preprint arXiv:2503.05652, 2025.arxiv.org/abs/2503.05652
  37. [38]S. Yenamandra, A. Ramachandran, K. Yadav, A. Wang, M. Khanna, T. Gervet, T.-Y. Yang, V. Jain, A. W. Clegg, J. Turner et al., “HomeRobot: Open-vocabulary mobile manipulation,” in CoRL, 2023.arxiv.org/abs/2306.11565
  38. [39]F. Stulp, A. Fedrizzi, L. Mösenlechner, and M. Beetz, “Learning and reasoning with action-related places for robust mobile manipulation,” Journal of Artificial Intelligence Research, vol. 43, pp. 1–42, 2012.DOI
  39. [40]S. Jauhri, J. Peters, and G. Chaltzaki, “Robot learning of mobile manipulation with reachability behavior priors,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 8399–8406, 2022.DOI
  40. [41]E. Rosen, S. James, S. Orozco, V. Gupta, M. Merlin, S. Tellex, and G. Konidaris, “Synthesizing navigation abstractions for planning with portable manipulation skills,” in CoRL, 2023, pp. 2278–2287.
  41. [42]N. Hirose, C. Glossop, A. Sridhar, D. Shah, O. Mees, and S. Levine, “LeLaN: Learning a language-conditioned navigation policy from in-the-wild videos,” arXiv preprint arXiv:2410.03603, 2024.arxiv.org/abs/2410.03603
  42. [43]N. Yokoyama, S. Ha, D. Batra, J. Wang, and B. Bucher, “VLFM: Vision-language frontier maps for zero-shot semantic navigation,” in ICRA, 2024, pp. 42–48.arxiv.org/abs/2312.03275
  43. [44]N. Yokoyama, A. Clegg, J. Truong, E. Undersander, T.-Y. Yang, S. Arnaud, S. Ha, D. Batra, and A. Rai, “ASC: Adaptive skill coordination for robotic mobile manipulation,” IEEE Robotics and Automation Letters, vol. 9, no. 1, pp. 779–786, 2023.arxiv.org/abs/2304.00410
  44. [45]N. M. M. Shafiullah, A. Rai, H. Etukuru, Y. Liu, I. Misra, S. Chintala, and L. Pinto, “On bringing robots home,” arXiv preprint arXiv:2311.16098, 2023.arxiv.org/abs/2311.16098
  45. [46]J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in CVPR, 2016, pp. 4104–4113.DOI
  46. [47]J. Kulhanek, S. Peng, Z. Kukelova, M. Pollefeys, and T. Sattler, “WildGaussians: 3D Gaussian splatting in the wild,” in NeurIPS, 2024.arxiv.org/abs/2407.08447
  47. [48]B. Wen, W. Yang, J. Kautz, and S. Birchfield, “FoundationPose: Unified 6D pose estimation and tracking of novel objects,” in CVPR, 2024.arxiv.org/abs/2312.08344
  48. [49]Y. Shen, Z. Zhang, Y. Qu, and L. Cao, “FastVGGT: Training-free acceleration of visual geometry transformer,” arXiv preprint arXiv:2509.02560, 2025.arxiv.org/abs/2509.02560
  49. [50]T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F. Yan et al., “Grounded SAM: Assembling open-world models for diverse visual tasks,” arXiv preprint arXiv:2401.14159, 2024.arxiv.org/abs/2401.14159
  50. [51]N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C.-Y. Wu, R. Girshick, P. Dollár, and C. Feichtenhofer, “SAM 2: Segment anything in images and videos,” in ICLR, 2025.arxiv.org/abs/2408.00714
  51. [52]S. Umeyama, “Least-squares estimation of transformation parameters between two point patterns,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 4, pp. 376–380, 1991.DOI
  52. [53]C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, M. Lingelbach, J. Sun et al., “BEHAVIOR-1K: A benchmark for embodied AI with 1,000 everyday activities and realistic simulation,” in CoRL, 2023, pp. 80–93.arxiv.org/abs/2403.09227
  53. [54]P. Florence, L. Manuelli, and R. Tedrake, “Self-supervised correspondence in visuomotor policy learning,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 492–499, 2019.arxiv.org/abs/1909.06933
  54. [55]A. Knappitsch, J. Park, Q.-Y. Zhou, and V. Koltun, “Tanks and Temples: Benchmarking large-scale scene reconstruction,” ACM Trans. Graph., vol. 36, no. 4, pp. 78:1–78:13, 2017.
  55. [56]S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang, “Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection,” in ECCV, 2024.arxiv.org/abs/2303.05499
  56. [57]V. Ye, R. Li, J. Kerr, M. Turkulainen, B. Yi, Z. Pan, O. Seiskari, J. Ye, J. Hu, M. Tancik, and A. Kanazawa, “gsplat: An open-source library for Gaussian splatting,” Journal of Machine Learning Research, vol. 26, no. 34, pp. 1–17, 2025.arxiv.org/abs/2409.06765

Paper details

vlnc:W_7z5snaeu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 15

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents