betaFeedback

WorksComputer Science

Gaussian Episodic Memory for Long-Term Spatiotemporal Reasoning

Abstract

Real-world environments undergo continual semi-static changes, requiring a memory representation that captures both the spatial and temporal evolution of a scene. Existing spatio-temporal memories for robots typically rely either on structured representations, which are limited to the entities and attributes captured during construction, or on image-based memories, which are constrained to previously observed viewpoints. We introduce GEMS, a framework that combines a persistent spatio-temporal 3D Gaussian Splatting (3DGS) map with an agentic reasoning pipeline. The 3DGS map is reconstructed once and subsequently updated only in regions where the scene changes, storing only these modifications while preserving visual detail across time. The agentic pipeline leverages this map to render novel views on demand, iteratively accumulating evidence to answer spatio-temporal questions and ground objects from natural-language queries. To advance future research, we introduce GEMS-Bench, a spatiotemporal question-answering benchmark focused on semi-static environments. It comprises 3 real-world and 12 synthetic environments, spanning 473 question-answer pairs. We evaluate GEMS on novel-view synthesis quality across scene updates, 3D object grounding, spatiotemporal question answering, and storage efficiency. GEMS outperforms recent baselines in 3D object grounding and spatiotemporal question answering, while achieving rendering quality on par with dedicated scene-update methods. Code and data: https://abwerby.github.io/GEMS/

Figure 1. Overview of GEMS. A robot revisits a semi-static scene, yielding a posed RGB-D sequence Ot\mathcal{O}^{t} per visit (left). GEMS reconstructs the first visit into a Gaussian map G0\mathcal{G}^{0} and records each later visit as a difference ΔGt\Delta\mathcal{G}^{t} capturing only the regions that changed, so the scene at any past visit is recovered by rendering G0\mathcal{G}^{0} together with the differences up to that visit (middle). Given a question (right), the agent explores the scene using its toolbox and answers based on the collected evidence.

Introduction

Robots operating for long periods in human-centered environments must model not only the geometry and semantics of their surroundings but also how those surroundings evolve over time. Real-world environments, such as homes and offices, experience frequent semi-static changes in which objects are moved, furniture is rearranged, and equipment may appear, disappear, or change state. Modeling these changes and building episodic memory that captures when and where they occurred enables robots to reason about the environment’s history and answer spatial, semantic, and temporal questions, such as where an object was previously located, when a change occurred, or how the current state of a scene differs from an earlier one.

3D scene graphs (3DSGs) [1, 2, 3] have emerged as compact, structured representations that organize spatial-semantic concepts. However, typical 3DSGs represent a single snapshot of the environment. More recent 3DSG approaches incorporate temporal information into scene representations to model semi-static changes [4, 5, 6, 7]. However, they either preserve only object-level states and textual descriptions, precommitting to objects segmented at construction time, or preserve observations only from fixed viewpoints, limiting access to the underlying visual evidence. Recently, 3D Gaussian Splatting (3DGS) [8] has emerged as a high-fidelity representation that preserves fine-grained visual details, supports novel-view synthesis, and enables localized updates [9], [10], making it well suited for long-term embodied reasoning. Existing 3DGS-based mapping and reasoning methods, however, either address static scenes [11], [12] or maintain only the latest scene state [13].

To this end, we present GEMS, a persistent spatiotemporal episodic memory based on 3D Gaussian Splatting (3DGS), coupled with an agentic reasoning pipeline. GEMS construct an initial 3DGS map, then detect changes from observations collected during each subsequent scene revisit and locally update the Gaussian primitives in affected regions. Unchanged Gaussians are shared across temporal states, while additions, removals, and modifications are stored as temporal deltas. As a result, storage grows with environmental changes rather than the number of revisits. Given a query, the agentic pipeline can select informative viewpoints for reconstructed regions rather than being restricted to a fixed set of viewpoints. The 3DGS-based memory enables the agent to render the map from a consistent virtual camera pose across different timesteps. This controlled alignment supports cross-temporal visual comparison and reasoning about object motion and state changes. Moreover, photorealistic novel-view rendering enables the agent to perform on-demand segmentation of target objects from informative, query-dependent viewpoints.

To foster future research, we introduce GEMS-Bench, a multi-revisit spatiotemporal embodied question-answering benchmark. It comprises 12 synthetic household scenes and 3 real-world scenes, each with at least 3 revisits. GEMS-Bench spans 473 question-answer pairs categorized into 5 types, as well as posed RGB-D for each revisit.

In summary, we make the following contributions:

  • We introduce GEMS, a 3DGS-based episodic memory framework that maintains long-term scene history via efficient, localized updates, enabling high-fidelity novel-view rendering over time.

  • We develop an agentic EQA pipeline that utilizes spatiotemporal novel-view synthesis for query-dependent viewpoint selection, visual alignment across time, and on-demand object grounding.

  • We introduce a new multi-visit spatiotemporal question-answering benchmark, GEMS-Bench, comprising semi-static environmental changes across 15 scenes and 473 question-answer pairs.

  • We evaluate GEMS across four dimensions: novel-view synthesis quality under scene updates, 3D object grounding from natural-language queries, spatiotemporal question answering, and storage requirements.

We provide code and data at: https://awberby.github.io/GEMS/

Spatial-Semantic and Spatiotemporal Mapping

Early works on robotics mapping [14] focused only on geometric modeling of the environment. Subsequent works [15], [16] augmented such 3D geometric maps with semantic and open-vocabulary language features, enabling object retrieval and language-guided navigation. Structured representations, such as 3D Scene Graphs (3DSGs) [2], [3], [17]–[20], have emerged as more efficient representations that encode both geometric and semantic information in a topological graph; however, they only maintain a single snapshot of the scene state.

To extend robotics maps to also capture the temporal aspect of the environment, Khronos [4] performs spatiotemporal metric-semantic SLAM that jointly captures short-term dynamics and long-term changes and can be queried for the estimated scene state at any past timestamp. However, its metric-semantic map does not preserve photorealistic appearance, so visual evidence beyond that abstraction cannot be re-examined. Other methods maintain a persistent object- or point-level representation and revise it as the scene changes [21], [22], whereas [5], [7] additionally retain temporal traces in the form of object trajectories and timestamped descriptions, respectively. The former keep only the current state and therefore cannot answer questions about earlier configurations, while all of them precommit to the objects and attributes instantiated at construction time and discard the underlying visual details.

3DGS-based Maps

3D Gaussian Splatting (3DGS) [8] is an explicit representation for 3D scene reconstruction that enables photorealistic rendering while preserving fine-grained visual details. Because it stores explicit primitives, 3DGS also supports cheap local updates, unlike implicit representations such as NeRF [23]. Recent extensions [24] further enrich the Gaussian primitives with high-dimensional semantic attributes for semantic tasks. For long-term scene updates, O-SCD [9] localizes changes by directly comparing new observations against views rendered from the previous map, and others [10], [25] further show the gain in time and storage from updating only the changed area. A third line of work uses 3DGS as the map representation for embodied reasoning. SplatReasoner [12] synthesizes query-conditioned viewpoints that reveal evidence missing from fixed-view memories, and GSMem [11] renders informative viewpoints on demand for exploration; both reason over a single static capture. GS-LTS [13] instead keeps the map current across revisits, yet stores only the latest state and discards the scene’s history. GEMS renders query-conditioned views over a memory spanning visits, so the agent can hold a viewpoint fixed while varying the time.

Embodied Question Answering and Episodic Memory

Introduced by OpenEQA [26], Episodic Memory EQA (EM-EQA) evaluates an agent’s capability to answer questions based on past multimodal experiences. In this task, a key requirement for the agent is to correctly retrieve historical contexts during querying. REMemBR [27] retrieves by embedding similarity over textual descriptions of video segments, and so discards the visual and geometric evidence entirely; DAAAM [7] retrieves by matching per-node descriptions, GraphEQA [6] by traversing a semantic scene graph, and STaR [28] by task-conditioned retrieval over saved keyframes, textual descriptions, and object point clouds. In all cases the information granularity is fixed when the memory is built. 3D-Mem [29] and Enter the Mind Palace [30] retain raw images, are constrained to pre-recorded views, and segment objects in advance, which makes queries about off-list objects unanswerable. Our method runs object segmentation only upon receiving a query, which makes it easy to adjust object granularity without being fixed to predefined lists, utilizing the photorealistic novel view synthesis capabilities of 3DGS.

Beyond memory representations, existing works also diverge in their evaluation benchmarks. EM-EQA benchmarks [26] predominantly assume single-session explorations of static scenes, overlooking temporal evolution entirely. Other temporal benchmarks, such as NaVQA [27], assess reasoning over continuous navigation videos, focusing on when and where an event was observed along a continuous trajectory rather than how the environment altered between visits. While recent efforts begin to investigate dynamic environments, Situat3DChange [31] is confined to change understanding between paired static 3D point scans, whereas Enter the Mind Palace [30] focuses on multi-episode navigation in simulated settings with limited scene scale and question diversity.

Method

The goal of this work is to construct a photorealistic spatiotemporal memory of a semi-static scene observed over multiple revisits. The memory supports spatial and temporal reasoning and question answering over the scene’s history. As illustrated in Figure 2, GEMS consists of two main components. First, the mapping component initializes a Gaussian splatting map and incrementally maintains its history by detecting and updating changes. This map represents a photorealistic episodic memory capable of on-demand rendering of query-related views and object segmentation across time and space (Sec. III-A). Second, the agentic reasoning component analyzes a query and iteratively gathers grounded evidence from the memory using retrieval, rendering, and segmentation tools (Sec. III-B). It then synthesizes the final answer from the collected evidence.

Figure 2. Overview of GEMS. Left: mapping. On each revisit, keyframes rendered from the previous map are compared against the incoming observations to obtain a change mask, components of which are labeled as Add, Remove, or Refine based on the depth error. The affected Gaussian primitives are re-optimized. Each Gaussian carries a lifespan [b,d)[b,d), so the state of scene at any past time can be retrieved as a temporal slice. Right: reasoning. An LLM planner answers a query through memory tools, and an answer synthesizer concludes from the gathered evidence.

Spatiotemporal Gaussian Map

We consider a robot that observes a semi-static environment over T+1T+1 revisits. For each revisit tt, the input is a posed RGB-D sequence:

Ot={(Iit,Dit,Kit,Pit)}i=1nt,t∈{0,…,T}.(1)\mathcal{O}^{t}=\left\{\left(\mathbf{I}_{i}^{t},\mathbf{D}_{i}^{t},\mathbf{K}_{i}^{t},\mathbf{P}_{i}^{t}\right)\right\}_{i=1}^{n_t},\quad t\in\{0,\ldots,T\}. \tag*{(1)}

Where Iit\mathbf{I}_{i}^{t} and Dit\mathbf{D}_{i}^{t} denote the RGB image and depth map, Kit\mathbf{K}_{i}^{t} denotes the camera intrinsics, Pit∈SE(3)\mathbf{P}_{i}^{t}\in SE(3) denotes the camera pose in a common world coordinate frame, ntn_t denotes the number of frames in revisit tt.

From the first visit O0\mathcal{O}^{0}, we construct the initial 3D Gaussian map G0\mathcal{G}^{0} using gsplat [32]. We follow the loss formulation of GaME [10], including its photometric and geometric terms, but replace its isotropic regularizer with a normalized, thresholded penalty on the per-axis scales,

Lscale=1∣Ω∣+ϵ∑(i,k)∈Ωmax⁡(0,∣si,k−sˉi∣sˉi+ϵ−τscale),(2)L_{\mathrm{scale}}=\frac{1}{|\Omega|+\epsilon}\sum_{(i,k)\in\Omega}\max\left(0,\frac{|s_{i,k}-\bar{s}_{i}|}{\bar{s}_{i}+\epsilon}-\tau_{\mathrm{scale}}\right), \tag*{(2)}

which penalizes extreme anisotropic elongation of the Gaussians.

Cross-Visit Change Detection

  1. Cross-Visit Change Detection: Given the next revisit Ot\mathcal{O}^{t}, we first select a spatially diverse subset of keyframes. A frame ii is considered keyframe if its pose distance d(Pit,Pjt)d(\mathbf{P}_{i}^{t},\mathbf{P}_{j}^{t}) from every previously selected keyframe jj exceeds τpose\tau_{\mathrm{pose}}. We denote the set of the selected keyframes by Kt\mathcal{K}^{t}. For each i∈Kti\in\mathcal{K}^{t}, we render a color image Iˉit−1\bar{\mathbf{I}}_{i}^{t-1} and a depth image Dˉit−1\bar{\mathbf{D}}_{i}^{t-1} from the previous Gaussian map Gt−1\mathcal{G}^{t-1} at the pose of the current observation. Because the new observation and the rendering share the same camera pose, their differences primarily capture changes in the scene rather than in the viewpoint. Then we score both visual and semantic changes. The visual term is the SSIM similarity map between the observed and the rendered images, and the semantic term is the cosine similarity between their DINOv3 [33] feature maps fit\mathbf{f}_{i}^{t} and fˉit−1\bar{\mathbf{f}}_{i}^{t-1}, bilinearly upsampled from patch to pixel resolution. Since both terms quantify similarity, the change score cit\mathbf{c}_{i}^{t} is defined as their complement:

cit=1−[w SSIM⁡(Iˉit−1,Iit)+(1−w)cos⁡(fˉit−1,fit)],(3)\mathbf{c}_{i}^{t}=1-\left[w\,\operatorname{SSIM}\left(\bar{\mathbf{I}}_{i}^{t-1},\mathbf{I}_{i}^{t}\right)+(1-w)\cos\left(\bar{\mathbf{f}}_{i}^{t-1},\mathbf{f}_{i}^{t}\right)\right], \tag*{(3)}

with w=0.3w=0.3. Both similarity terms are clamped to [0,1][0,1], so that cit∈[0,1]H×W\mathbf{c}_{i}^{t}\in[0,1]^{H\times W}. Thresholding it yields the binary change mask Mit=I[cit>τs]\mathbf{M}_{i}^{t}=\mathbb{I}[\mathbf{c}_{i}^{t}>\tau_{s}].

Using the observed and the rendered depth, we classify each connected component S\mathcal{S} of Mit\mathbf{M}_{i}^{t} into three types of general object movement: Add, Remove, and Refine. We compute the relative depth error δDit=(Dˉit−1−Dit)/Dit\delta\mathbf{D}_{i}^{t}=(\bar{\mathbf{D}}_{i}^{t-1}-\mathbf{D}_{i}^{t})/\mathbf{D}_{i}^{t}. A positive error indicates newly observed geometry in front of the preceding map, whereas a negative error indicates that previously observed foreground geometry is no longer present. We therefore compute the fractions

radd(S),rremove(S)=1∣S∣∑uI[±δDit>τd],(4)r_{\mathrm{add}}(\mathcal{S}),r_{\mathrm{remove}}(\mathcal{S})=\frac{1}{|\mathcal{S}|}\sum_{\mathbf{u}}\mathbb{I}\left[\pm\delta\mathbf{D}_{i}^{t}>\tau_{d}\right], \tag*{(4)}

where I[⋅]\mathbb{I}[\cdot] is the indicator function and the positive and negative signs give raddr_{\mathrm{add}} and rremover_{\mathrm{remove}}, respectively.

The Add operation is triggered if radd(S)>τmr_{\mathrm{add}}(\mathcal{S})>\tau_{m} and the Remove operation is triggered if rremove(S)>τmr_{\mathrm{remove}}(\mathcal{S})>\tau_{m}. Both operations may be triggered for the same component, as commonly occurs when an object is relocated. If neither condition is satisfied, the component is assigned a Refine operation to account for appearance changes or local reconstruction errors, thereby maintaining high-fidelity rendering over a long period of time.

Local Map Updates and Temporal State Storage

  1. Local Map Updates and Temporal State Storage: We adopt distinct selection and initialization strategies tailored to the three mask types.

Refine Mask: We project Gaussian primitives onto the camera plane and set the per-pixel depth threshold to τrefine(u)=max⁡(Dˉit−1(u),Dit(u))\tau_{\mathrm{refine}}(\mathbf{u})=\max(\bar{\mathbf{D}}_{i}^{t-1}(\mathbf{u}),\mathbf{D}_{i}^{t}(\mathbf{u})). Gaussian primitives that fall within the mask and lie closer to the camera than τrefine\tau_{\mathrm{refine}} are selected and denoted as Grefinet−1\mathcal{G}_{\mathrm{refine}}^{t-1}.

Add Mask: We back-project the pixels within the mask into 3D space using Dti\mathbf{D}_{t}^{i}, and follow gsplat [32] to initialize the parameters for these new Gaussian primitives. These newly created primitives are added to the set Gnew\mathcal{G}_{\mathrm{new}}.

Remove Mask: We apply the same selection strategy as in the Refine Mask, but set the depth threshold to τremove(u)=Dti(u)\tau_{\mathrm{remove}}(\mathbf{u}) = D_{t}^{i}(\mathbf{u}). The selected primitives are directly deleted from the map. Subsequently, we sample new Gaussians within these masked regions using the same sampling method to fill potential disocclusion holes caused by the removal of foreground objects. These newly sampled Gaussians are also added to Gnew\mathcal{G}_{\mathrm{new}}.

The remaining unselected primitives are designated as static Gaussians, denoted by Gstatict−1\mathcal{G}_{\mathrm{static}}^{t-1}. To accelerate the optimization procedure, we only update the active primitives in Gnewt∪Grefinet−1\mathcal{G}_{\mathrm{new}}^{t}\cup\mathcal{G}_{\mathrm{refine}}^{t-1}, while keeping Gstatict−1\mathcal{G}_{\mathrm{static}}^{t-1} frozen, however, all primitives still participate in rendering. The active Gaussians are optimized using the same loss formulation as during initial map construction (Sec. III-A), but excluding the scale regularization term LscaleL_{\mathrm{scale}}. Because each keyframe’s Gaussians are optimized for only 100 iterations, the horizon is too short for significant shape distortion to accumulate, and enforcing the regularizer over so few steps merely adds computational overhead while prematurely constraining reconstruction quality.

To avoid saving one map Gt\mathcal{G}^{t} per revisit, we keep a single set G\mathcal{G} of all Gaussian primitives ever created and attach to each primitive gg a lifespan [bg,dg)[b_g,d_g), where bgb_g is the revisit at which gg was added and dgd_g the revisit at which it was removed, with dg=∞d_g=\infty while gg is still present. An update writes only these two fields: primitives in Gnewt\mathcal{G}_{\mathrm{new}}^{t} are appended with bg=tb_g=t, those selected by Remove are closed with dg=td_g=t, and those in Grefinet−1\mathcal{G}_{\mathrm{refine}}^{t-1} are closed with dg=td_g=t while their re-optimized copies are appended with bg=tb_g=t. The map at any revisit is then recovered by a temporal slice.

Gt={g∈G∣bg≤t<dg}.(5)\mathcal{G}^{t}=\{g\in\mathcal{G}\mid b_g\leq t<d_g\}. \tag*{(5)}

Thus, the representation grows with accumulated changes rather than repeated visits. We iterate these steps for every subsequent revisit t∈{1,…,T}t\in\{1,\ldots,T\}. The resulting spatiotemporal memory retains only the Gaussian primitives G\mathcal{G} and the keyframe poses of each revisit:

M=(G,{Pit∣t∈{0,…,T},i∈Kt}).(6)\mathcal{M}=(\mathcal{G},\{\mathbf{P}_{i}^{t}\mid t\in\{0,\ldots,T\},i\in\mathcal{K}^{t}\}). \tag*{(6)}

Agentic Spatiotemporal Question Answering

Given a query qq and our episodic memory M\mathcal{M}, we instantiate an agentic reasoning pipeline consisting of an LLM [34] planner π\pi that iteratively calls a set of memory-interaction tools to collect query-related evidence. At each iteration kk, the planner selects a tool aka_k and its arguments zkz_k based on the query and the evidence collected so far:

(ak,zk)=π(q,E1:k−1),ek=Tak(M,zk),(7)(a_k,z_k)=\pi(q,\mathcal{E}_{1:k-1}),\qquad e_k=\mathcal{T}_{a_k}(\mathcal{M},z_k), \tag*{(7)}

where Tak\mathcal{T}_{a_k} is the selected tool, eke_k its returned evidence, and E1:k−1={e1,…,ek−1}\mathcal{E}_{1:k-1}=\{e_1,\ldots,e_{k-1}\} the evidence gathered so far. The available tools are restricted by an execution state machine (Fig. 3). The process continues until the planner terminates the search or reaches a predefined tool-call budget.

To speed up the agent’s search in episodic memory M\mathcal{M}, we render an RGB frame for every keyframe in K=⋃tKt\mathcal{K} = \bigcup_t \mathcal{K}^t from G\mathcal{G}, and cache its OWLv2 [35] feature. Moreover, we group the keyframe poses of each revisit into spatial regions Ct\mathcal{C}^t by hierarchical agglomerative clustering [36]. This enables the agent to search for concepts at two levels of granularity in the memory.

Agentic tools interact with M\mathcal{M} by rendering query-related color and depth frames, R(Gt,P)↦(I^,D^)\mathcal{R}(\mathcal{G}^t,\mathcal{P}) \mapsto(\hat{I},\hat{D}), from query-selected poses, they fall into two groups:

Retrieval tools (Aretrieve\mathcal{A}_{\mathrm{retrieve}}) locate relevant views: sweep gives a global scene overview from Ct\mathcal{C}^t; search retrieves query-relevant keyframe poses by OWLv2 [35] similarity to the cached features and renders one candidate per relevant viewpoint cluster; covisible prunes candidates by volumetric covisibility with a target frame; and recall re-renders the cited visual evidence in the report.

Reasoning tools (Areason\mathcal{A}_{\mathrm{reason}}) inspect a located region in detail: time_view renders one camera pose across selected revisits for viewpoint-aligned comparison; relook refines a pose to clarify an ambiguous region; locate segments a queried entity with SAM3 [37] and lifts its mask to metric 3D; obj_view renders further views of a localized entity; compare compares two localized instances; and track associates instances across revisits.

To structure decision-making and prevent early termination, the agentic pipeline is designed as a finite-state machine, in which each state restricts which tools are available (Fig. 3). The agent starts in SinitS_{\mathrm{init}} with global retrieval (sweep, search), then alternates between detailed reasoning (SreasonS_{\mathrm{reason}}) and retrieval (SretrieveS_{\mathrm{retrieve}}); one pass through both is a retrieval-reasoning round. conclude ends a round and purges raw visual context from the context history to limit the number of tokens used and speed up the process; exit ends the search; and report compiles the evidence report.

The report, together with all cited visual and 3D evidence E\mathcal{E}, is passed to the answer synthesizer α\alpha, which returns the answer A=α(q,E)A = \alpha(q,\mathcal{E}), a rationale, and a confidence score.

Dataset

We present GEMS-Bench, a spatiotemporal question-answering benchmark focused on reasoning across discrete revisits to the same semi-static environment. It comprises multi-visit RGB-D trajectories from real-world and synthetic environments, camera poses, object-level change annotations, and question-answer pairs. The real-world split comprises three indoor environments, each with four visits, captured using an Intel RealSense D455f camera, estimates camera poses using COLMAP [38], and contains a total of 244 question-answer (QA) pairs. The synthetic split comprises 12 HSSD scenes [39] simulated using Habitat-Sim [40], with at least three visits per scene. It includes 97 annotated object-level changes and 229 QA pairs. Across both splits, all scene changes and questions are manually verified by a human expert to ensure correct ground-truth answers and unambiguous visual references.

We categorize the benchmark questions into five distinct classes: (1) Counting, covering object quantities and event frequencies; (2) Movement and Presence, addressing spatial relocation and object appearance, disappearance, or persistence over time; (3) State Change, referring to non-spatial property changes such as articulation, orientation, color, or configuration; (4) Static, comprising questions that can be answered from a single visit without temporal context; and (5) Negative, comprising questions that test robustness to false premises, incorrectly presumed changes, or events that did not occur. Across all 473 questions, the distribution is 30.7% Negative, 22.8% Movement and Presence, 20.5% State Change, 16.9% Static, and 9.1% Counting.

Experiments

Reconstruction Quality

We assess whether GEMS preserves rendering quality while updating the map incrementally, holding out every 10th input image as the test set, following the same evaluation protocol as GaME [10]. We compare against two dedicated scene-update methods: GaME [10], which uses depth to prune obsolete Gaussians and initialize primitives for newly introduced objects, and O-SCD [9], which updates the map from RGB-only change detection and therefore neither consumes nor predicts depth. We evaluate on two real-world benchmarks with one temporal update per scene: GaME-Aria [10], with two indoor scenes and dense posed RGB-D sequences, and CL-Splats Real-World [25], with five scenes, sparser views, and no ground-truth depth. Since GEMS and GaME both require depth, we synthesize depth for CL-Splats using Depth Anything 3 [41]. We report PSNR, SSIM [44], and LPIPS [45] for photometric quality, depth L1 error for geometric accuracy, and the wall-clock time of a single map update.

Table I shows that GEMS preserves rendering quality on par with other methods while updating the map far more cheaply: an update takes about one minute on both benchmarks,

MethodPSNR [dB] ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowDepth L1 [cm] ↓\downarrowTime [min] ↓\downarrow
(a) Evaluation on GaME-Aria [10] Aria Dataset
O-SCD [9]32.320.9270.121—2.79
GaME [10]31.120.9160.1221.1529.40
GEMS (ours)29.230.9040.1282.281.12
(b) Evaluation on CL-Splats Real-World [25] w/ DA3 [41] Depth
O-SCD [9]30.040.9430.056—2.03
GaME [10]24.250.8870.13410.7929.81
GEMS (ours)27.870.9010.0899.381.67
MethodOverall IoU≥0.10w Spatial Lang. IoU≥0.10w/o Spatial Lang. IoU≥0.10w Color Lang. IoU≥0.10w/o Color Lang. IoU≥0.10w Shape Lang. IoU≥0.10w/o Shape Lang. IoU≥0.10w Target Mention IoU≥0.10w/o Target Mention IoU≥0.10
OpenFusion [43]10.78.922.311.810.59.810.911.34.9
ConceptGraphs [19]16.015.022.317.615.710.816.916.96.6
BBQ [20]28.328.129.825.229.034.327.329.614.8
KeySG [3]49.849.257.048.350.750.050.352.625.4
GEMS (ours)58.957.071.163.257.958.758.962.223.4

Table I. Reconstruction quality and update cost of scene-update methods. Time is the wall-clock cost of a single map update.

Execution state machine

Figure 3. Execution state machine. Each state exposes only its tool group (Areason\mathcal{A}_{\mathrm{reason}} or Aretrieve\mathcal{A}_{\mathrm{retrieve}}) plus the control tool on each outgoing edge.

between 18× and 26× faster than GaME. GEMS achieves the lowest depth error on CL-Splats and the second lowest on GaME-Aria, and stays within a few dB PSNR of the best photometric scores, which O-SCD achieves without producing any depth at all.

Object Grounding from Natural-Language Queries

To evaluate GEMS on grounding objects from natural-language queries in an indoor scene, we use the NR3D dataset [42], which contains single-visit indoor scenes paired with a diverse set of natural-language queries grouped by query type. We adopt the same evaluation protocol and scene identifiers as BBQ [20]. Tab. II compares GEMS against recent methods that build structured representations, reporting grounding accuracy at IoU ≥0.1\ge0.1.

GEMS achieves the best overall accuracy of 58.9% and outperforms all baselines in every query category except queries without a target mention. Both GEMS and KeySG [3] are substantially more accurate than the remaining methods, since both exploit (raw or rendered) visual evidence rather than abstracted object nodes. While KeySG retains a slight edge on queries that omit target mention, GEMS achieves superior performance across all other spatial, color, and shape categories.

Spatiotemporal Question Answering

We evaluate GEMS on spatiotemporal question answering using our GEMS-Bench benchmark. For each scene, we first build the episodic memory M\mathcal{M}, then run the agentic pipeline to explore it and answer every question in that scene. Following OpenEQA [26], we score each predicted answer with an LLM matcher on a scale from 1 to 5, where 1 is completely wrong and 5 is a perfect answer, and report the mean score rescaled to [0, 1]. We use GPT-5.6-Luna as the judge.

We compare against a range of memory representations. DAAAM [7] and Enter the Mind Palace [30] build 3D scene graphs, ReMEmbR [27] stores textual descriptions alone, and STaR [28] combines saved keyframes with textual descriptions and object point clouds. We additionally compare to GEMS-all-images, a variant that retains all posed RGB-D frames across revisits as its memory and runs the same agentic pipeline, except that it cannot use the NVS tools; it acts as an upper bound on the visual information available to the agent. We also compare against a blind LLM, Qwen3.5 [34], where the model is purely guessing at the answers to the questions; it serves as a lower bound. For fairness, every method, including GEMS, uses Qwen3.5 [34] as its underlying LLM. Table III breaks the results down by query type. GEMS attains the best overall score on both splits, 60.9% on the real-world and 71.0% on the synthetic split, against 44.8% and 35.5% for the strongest baseline, STaR. On each question class, GEMS outperforms all the other baselines with around 10%.

MethodCount.Move./Pres.Neg.State Ch.StaticOverall
(a) Real-world benchmark
Blind-Qwen3.5 [27]26.721.228.318.733.125.0
ReMEMbR [27]26.725.538.626.217.628.3
DAAAM [7]10.822.840.125.424.327.1
Mind Palace [30]34.228.830.224.645.331.3
STaR [28]39.246.245.242.950.044.8
GEMS-all-images63.359.265.463.962.863.2
GEMS (ours)65.053.360.360.768.260.9
(b) Synthetic benchmark
Blind-Qwen3.5 [27]38.539.135.416.924.431.8
ReMEMbR [27]11.542.335.430.935.535.3
DAAAM [7]23.137.148.722.836.638.0
Mind Palace [30]15.425.031.834.253.734.2
STaR [28]23.154.426.037.527.335.5
GEMS-all-images48.164.965.659.669.663.8
GEMS (ours)63.571.474.758.176.271.0
(c) NVS tool ablation on questions invoking NVS (n=245n = 245)
number of questions1753876523245
GEMS-no-NVS52.959.965.253.148.958.5
GEMS (ours)66.274.565.556.964.165.1

Table III. BLOCKS (A) AND (B): SPATIOTEMPORAL EQA ACCURACY FROM AN LLM JUDGE (0–100%) BY QUESTION CATEGORY. BLOCK (C): NVS TOOL ABLATION ON THE QUESTIONS WHERE THE FULL AGENT INVOKED NVS.

On the real-world split, GEMS slightly underperforms GEMS-all-images, yet it achieves higher performance on the synthetic split despite storing no images. The per-class breakdown and overall metrics reveal that while GEMS-all-images exhibits consistent performance across both splits, GEMS shows a noticeable performance gap. Specifically, GEMS achieves nearly a 10% improvement on the synthetic split for movement and presence, negative, and static questions, while yielding nearly identical results on both splits for state change and counting questions. On the real-world split, sensor noise and pose estimation errors result in imperfect Gaussian maps. Although the synthesized views remain high quality when close to the training trajectories (§ 5-A), our novel view synthesis (NVS) tools often query views far from the training poses, occasionally introducing visual artifacts. These degraded images lower the performance of GEMS on the real-world split. Conversely, perfect depth maps and poses in the synthetic split eliminate these artifacts, yielding significant gains. Overall, the performance benefits provided by NVS tools outweigh their potential drawbacks. A detailed isolation study on the impact of NVS is presented in (§ 5-E).

Storage Efficiency

Finally, we compare the growth of GEMS’s episodic memory against two alternatives that retain images directly: all posed RGB-D frames, and the posed keyframes only. Figure 5 reports all of them on one synthetic and one real-world scene of GEMS-Bench. Since GEMS re-optimizes and stores Gaussian primitives only where a change is detected, its memory grows in proportion to the change observed at each revisit rather than paying a fixed cost per visit. After four revisits of the real-world scene, GEMS occupies 1.95 GiB at single precision and 0.98 GiB at half precision, compared with 2.73 GiB for the keyframes alone and 3.94 GiB for the full-frame sequence. The absolute gap widens with each revisit, so GEMS is more efficient whenever changes are localized, the common case in semi-static environments.

Figure 5. Storage growth. Memory footprint against the number of revisits, for one synthetic and one real-world scene of GEMS-Bench. Retaining all posed RGB-D frames, or only the keyframes, costs a fixed amount per revisit, whereas GEMS stores only the Gaussian primitives affected by a change. We report GEMS at single (fp32) and half (fp16) precision.

Ablation Studies

To assess the contribution of NVS, we select the 245 questions for which the full agent invoked time_view or relook, and rerun the complete pipeline on them with both tools disabled, using the same Gaussian memory and inference budget. Restricting to these questions separates the agent’s decision to invoke the tools from the tools’ own contribution. Access to the two tools improves the mean QA score by 6.6 percentage points, with gains across all five categories (Table III). This is the benefit of NVS, where the agent elects to use it, not across the whole benchmark. Figure 4 illustrates how viewpoint-aligned rendering supports temporal comparison: restricted to recorded viewpoints, the agent may fail to re-identify an object across perspective changes (right) or mistake a viewpoint difference for a physical state change (middle). Because both tools are removed jointly, the ablation does not separate spatial view refinement from temporal alignment.

Figure 4. Qualitative comparison of GEMS against GEMS-all-images. Each column shows a question, the evidence each agent collected, and the LLM-judge score. Left: a synthesized closer view resolves a spatial ambiguity from an oblique camera angle. Middle and right: time_view lets GEMS compare both visits from one pose, where the baseline is restricted to two different stored viewpoints.

Conclusion

We presented GEMS, a persistent spatiotemporal memory built on 3D Gaussian Splatting that stores environmental change as compact temporal deltas and answers episodic-memory questions by rendering novel views across space and time. By updating and storing only the Gaussians in changed areas, GEMS achieves superior storage efficiency and is still able to re-render the required images on demand from arbitrary camera poses at any time. Experiments on our long-term spatiotemporal benchmark show that combining spatial viewpoint selection with cross-temporal rendering improves question-answering accuracy and consistently outperforms established baselines.

Acknowledgements

This work was supported by the German Federal Ministry of Research, Technology, and Space (BMFTR) through the Robotics Institute Germany (RIG), and in part by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC 2120/2 – 390831618 – Cluster of Excellence Integrative Computational Design and Construction for Architecture (IntCDC).

References

References

  1. [1]D. Rotondi, F. Argenziano, S. Koch, N. Hughes, M. Büchner, J. Wald, L. Rosenberger Schmid, D. Nardi, A. Valada, L. Paull, F. Tombari, L. Carlone, and K. O. Arras, “3d scene graphs: Open challenges and future directions,” arXiv 2606.19383, 2026.arxiv.org/abs/2606.19383
  2. [2]A. Werby, C. Huang, M. Büchner, A. Valada, and W. Burgard, “Hierarchical open-vocabulary 3D scene graphs for language-grounded robot navigation,” in Robotics: Science and Systems (RSS), 2024.DOI
  3. [3]A. Werby, D. Rotondi, F. Scaparro, and K. O. Arras, “KeySG: Hierarchical keyframe-based 3D scene graphs,” in IEEE Int. Conf. Robot. Autom. (ICRA), 2026.arxiv.org/abs/2510.01049
  4. [4]L. Schmid, M. Abate, Y. Chang, and L. Carlone, “Khronos: A unified approach for spatio-temporal metric-semantic SLAM in dynamic environments,” in Robotics: Science and Systems (RSS), 2024.arxiv.org/abs/2402.13817
  5. [5]T. Behrens, R. Zurbrügg, M. Pollefeys, Z. Bauer, and H. Blum, “Lost & found: Tracking changes from egocentric observations in 3d dynamic scene graphs,” IEEE Robot. Autom. Lett., 2025.DOI
  6. [6]S. Saxena, B. Buchanan, C. Paxton, P. Liu, B. Chen, N. Vaskevicius, L. Palmieri, J. Francis, and O. Kroemer, “Grapheqa: Using 3d semantic scene graphs for real-time embodied question answering,” in Conf. Robot Learn. (CoRL), 2025.arxiv.org/abs/2412.14480
  7. [7]N. Gorlo, L. Schmid, and L. Carlone, “Describe anything anywhere at any moment,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026.arxiv.org/abs/2512.00565
  8. [8]B. Kerbl, G. Kopanas, T. Leimkühler, G. Drettakis, et al., “3d gaussian splatting for real-time radiance field rendering.” ACM Trans. Graph., 2023.
  9. [9]C. J. Galappaththige, J. Lai, L. Windrim, D. Dansereau, N. Suenderhauf, and D. Miller, “Changes in real time: Online scene change detection with multi-view fusion,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026.arxiv.org/abs/2511.12370
  10. [10]V. Yugay, T. Kersten, L. Carlone, T. Gevers, M. R. Oswald, and L. Schmid, “Gaussian mapping for evolving scenes,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026.arxiv.org/abs/2506.06909
  11. [11]Y. Lu, Y. Du, D. Liu, Y. Zhou, C. Wang, and Y. Yin, “Gsmm: 3d gaussian splatting as persistent spatial memory for zero-shot embodied exploration and reasoning,” arXiv preprint arXiv:2603.19137, 2026.arxiv.org/abs/2603.19137
  12. [12]Y.-J. Kim, D. Lee, J.-S. Kim, H.-W. Nam, G. Kim, Y. Kwon, Y.-C. F. Wang, J. Choe, and T.-H. Oh, “Splatreason: Enhancing embodied reasoning and grounding by novel view synthesis,” in Eur. Conf. Comput. Vis. (ECCV), 2026.arxiv.org/abs/2601.13132
  13. [13]B. Fu, J. Li, B. Zhang, R. Wang, and X. Chen, “Gs-lts: 3d gaussian splatting-based adaptive modeling for long-term service robots,” arXiv preprint arXiv:2503.17733, 2025.arxiv.org/abs/2503.17733
  14. [14]R. Mur-Artal and J. D. Tardós, “ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras,” IEEE Trans. Robot., 2017.
  15. [15]J. McCormac, A. Handa, A. Davison, and S. Leutenegger, “SemanticFusion: Dense 3D semantic mapping with convolutional neural networks,” in IEEE Int. Conf. Robot. Autom. (ICRA), 2017.arxiv.org/abs/1609.05130
  16. [16]C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual language maps for robot navigation,” in IEEE Int. Conf. Robot. Autom. (ICRA), 2023.arxiv.org/abs/2210.05714
  17. [17]I. Armeni, Z.-Y. He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and S. Savarese, “3d scene graph: A structure for unified semantics, 3d space, and camera,” in IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2019.arxiv.org/abs/1910.02527
  18. [19]Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, et al., “Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning,” in IEEE Int. Conf. Robot. Autom. (ICRA), 2024.arxiv.org/abs/2309.16650
  19. [20]S. Linok, T. Zemskova, S. Ladanova, R. Titkov, D. Yudin, M. Monastyrny, and A. Valenkov, “Beyond bare queries: Open-vocabulary object grounding with 3d scene graph,” in IEEE Int. Conf. Robot. Autom. (ICRA), 2025.arxiv.org/abs/2406.07113
  20. [21]Z. Yan, S. Li, Z. Wang, L. Wu, H. Wang, J. Zhu, L. Chen, and J. Liu, “Dynamic open-vocabulary 3d scene graphs for long-term language-guided mobile manipulation,” IEEE Robot. Autom. Lett., 2025.arxiv.org/abs/2410.11989
  21. [22]P. Liu, Z. Guo, M. Warke, S. Chintala, C. Paxton, N. M. M. Shafullah, and L. Pinto, “Dynamem: Online dynamic spatio-semantic memory for open world mobile manipulation,” in IEEE Int. Conf. Robot. Autom. (ICRA), 2025.arxiv.org/abs/2411.04999
  22. [23]B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” in Eur. Conf. Comput. Vis. (ECCV), 2020.arxiv.org/abs/2003.08934
  23. [24]J. Yu, K. Hari, K. Srinivas, et al., “Language-embedded gaussian splats (legs): Incrementally building room-scale representations with a mobile robot,” in IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2024.arxiv.org/abs/2409.18108
  24. [25]J. Ackermann, J. Kulhanek, S. Cai, H. Xu, M. Pollefeys, G. Wetzstein, L. J. Guibas, and S. Peng, “Gs-splats: Continual learning of gaussian splatting with local optimization,” in IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2025.arxiv.org/abs/2506.21117
  25. [26]A. Majumdar, A. Ajay, X. Zhang, P. Putta, Y. Senamandri, M. Henaff, S. Silwal, P. Mcvay, O. Maksymets, S. Arnaud, et al., “Openeqa: Embodied question answering in the era of foundation models,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024.DOI
  26. [27]A. Anwar, J. Welsh, J. Biswas, S. Pouya, and Y. Chang, “Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation,” in IEEE Int. Conf. Robot. Autom. (ICRA), 2025.arxiv.org/abs/2409.13682
  27. [28]M. Yuan, H. Zhang, M. Mohammad, R. Li, J. Shan, and S. L. Waslander, “Star: Scalable task-conditioned retrieval for long-horizon multi-modal robot memory,” IEEE Robot. Autom. Lett., 2026.DOI
  28. [29]Y. Yang, H. Yang, J. Zhou, P. Chen, H. Zhang, Y. Du, and C. Gan, “3d-mem: 3d scene memory for embodied exploration and reasoning,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025.arxiv.org/abs/2411.17735
  29. [30]M. F. Ginting, D.-K. Kim, X. Meng, et al., “Enter the mind palace: Reasoning and planning for long-term active embodied question answering,” in Conf. Robot Learn. (CoRL), 2025.arxiv.org/abs/2507.12846
  30. [31]R. Liu, J. Zheng, Y. Chen, Z. Wang, K. Peng, K. Yang, J. Zhang, M. Pollefeys, and R. Stiefelhagen, “Situat3dchange: Situated 3d change understanding dataset for multimodal large language model,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2026.arxiv.org/abs/2510.11509
  31. [32]V. Ye, R. Li, J. Kerr, M. Turkulainen, B. Yi, Z. Pan, O. Seiskari, J. Ye, J. Hu, M. Tancik, et al., “gsplat: An open-source library for gaussian splatting,” J. Mach. Learn. Res., 2025.arxiv.org/abs/2409.06765
  32. [33]O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al., “DinoV3,” arXiv preprint arXiv:2508.10104, 2025.
  33. [34]Q. Team, “OWen3S-Omni technical report,” arXiv preprint arXiv:2604.15804, 2026.
  34. [35]M. Minderer, A. Gritsenko, and N. Houlsby, “Scaling open-vocabulary object detection,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2023.arxiv.org/abs/2306.09683
  35. [36]D. Müllner, “Modern hierarchical, agglomerative clustering algorithms,” arXiv preprint arXiv:1109.2378, 2011.arxiv.org/abs/1109.2378
  36. [37]N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, R. Hu, D. Suris Coll-Vinent, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, et al., “Sam 3: Segment anything with concepts,” in Int. Conf. Learn. Represent. (ICLR), 2026.arxiv.org/abs/2511.16719
  37. [38]J. L. Schönberger and J.-M. Frahm, “Structure-from-motion revisited,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2016.DOI
  38. [39]M. Khanna, Y. Mao, H. Jiang, et al., “Habitat synthetic scenes dataset (HSSD-200): An analysis of 3D scene scale and realism tradeoffs for ObjectGoal navigation,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024.arxiv.org/abs/2306.11290
  39. [40]M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra, “Habitat: A platform for embodied AI research,” in IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2019.
  40. [41]H. Lin, S. Chen, J. Liew, D. Y. Chen, Z. Li, G. Shi, J. Feng, and B. Kang, “Depth anything 3: Recovering the visual space from any views,” arXiv preprint arXiv:2511.10647, 2025.arxiv.org/abs/2511.10647
  41. [42]P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. Guibas, “Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes,” in Eur. Conf. Comput. Vis. (ECCV), 2020.DOI
  42. [43]K. Yamazaki, T. Hanyu, K. Vo, T. Pham, M. Tran, G. Doretto, A. Nguyen, and N. Le, “Open-fusion: Real-time open-vocabulary 3d mapping and queryable scene representation,” in IEEE Int. Conf. Robot. Autom. (ICRA), 2024.
  43. [44]Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process., 2004.pubmed.ncbi.nlm.nih.gov/15376593
  44. [45]R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018.arxiv.org/abs/1801.03924

Paper details

vlnc:W_4f2e7u75
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents