betaFeedback

WorksComputer Science

Real-VAS: a Real-World Video Amodal Segmentation Dataset

Abstract

Amodal video object segmentation is fundamentally limited by the absence of datasets that combine real-world complexity with precise ground-truth annotations. To address this, we present Real Video Amodal Segmentation (Real-VAS), a new large-scale, zero-shot evaluation dataset built from composited real-image video. We introduce a novel data generation pipeline that enables the creation of programmatically consistent amodal ground truth for real-world footage, in which the object mask, occlusion boundary, and composited image are all derived from a single unoccluded source, without relying on human estimation or expensive 3D reconstruction. Our dataset is structured into two challenging scenarios: dynamic Occlusion, created by compositing two clips of moving objects, and a unique Container category featuring complex, physically constrained interactions. These container scenarios–On Surface Containment, Articulated Containment, and Mobile Containment–allow us to generate precise ground truth by simulating an object's motion based on its container's tracked transformation. As a result, Real-VAS provides a diverse and challenging benchmark for evaluating amodal segmentation models on realistic video with the precision of synthetic data. The dataset and generation code are publicly available at https://finlay-hudson.github.io/real_vas

Introduction

Video Object Segmentation (VOS) is the problem of densely tracking the visible pixels of an object. Methods such as SAM2 [37] and Mask2Former [8] have made remarkable progress on this problem in recent years, offering very high accuracy and robustness. However, in humans our visual system has a strong notion of object permanence [1] – the idea that objects maintain their identity and continuity over time, regardless of visibility – allowing us to predict where an object exists and how it may have deformed, even when occluded. This concept is called amodal completion [3, 34] and is the focus of this paper. Robust amodal completion is critical for many real-world applications, from autonomous driving, where vehicles must track occluded pedestrians, to robotic manipulation, where a robot needs to infer an object’s full shape to grasp it in a cluttered environment.

Progress on machine Video Amodal Segmentation (VAS), i.e. densely tracking object segmentations through occlusions, has received relatively less attention and progress has not been as rapid [24, 58]. One of the reasons for this limited progress is the lack of sufficiently challenging datasets with ground truth segmentations behind occlusions. Existing datasets suffer from several important weaknesses. On the one hand, synthetic datasets provide exact ground truth but are limited in realism and diversity, particularly for dynamic objects. On the other hand, obtaining ground truth segmentations for objects behind occlusions in real-world video is an extremely difficult and resource intensive task [45]. Possible solutions include multi-camera or 3D setups to record ground truth, which is not scalable, or human best-guess annotations behind occlusions, which is not accurate. The real-world benchmarks that are available do not provide accurate (or any) ground truth segmentation behind occlusions. Instead, they either provide ground truth only on unoccluded frames (in which case the benchmark is really testing reidentification [4, 54, 55, 57] rather than amodal segmentation) or rely on hand-drawn, approximate masks. Finally, existing datasets do not contain sufficiently challenging scenarios such as occlusion by a container (which requires reasoning about the motion of the container) or objects never re-appearing after occlusion (which requires extrapolation without relying on reidentification).

In this paper we propose scalable methods for creating a large-scale VAS dataset using real-world videos and without specialist acquisition setups. While synthetic datasets naturally offer a greater sheer volume of scenes, Real-VAS bridges a critical sim2real gap by focusing on the density and severity of real-world occlusions. Specifically, Real-VAS provides over 10,600 occluded frames, more than three times the number of occluded frames found in the next largest real-world competitor (see Table 1). With an occlusion prevalence of 72.53% and a mean occlusion severity of 67.33%, it offers a substantially harder and more comprehensive benchmark than existing datasets. We propose two pipelines (see Figure 1). The first allows arbitrary rigid or non-rigid, possibly dynamic occluders to be photorealistically composited over arbitrary occludee objects. The second allows complex occludee-container relationships to be accurately tracked. We use these methods to create the most challenging VAS dataset to date, Real-VAS, which is publicly available along with code for dataset generation.1

Figure 1. An overview of our Real-VAS data generation pipeline. Occlusion pipeline: from static camera videos, we automatically find compatible pairs of clips that can be composited to form realistic occlusions. Occluder objects and their effect (e.g. shadows) are composited over occludee objects for which accurate ground truth is computed from the unoccluded sequence. Containment pipeline: we identify three scenarios in which tracking the container is sufficient to provide ground truth tracking of the contained object. For both pipelines, additional camera motion can be subsequently emulated by cropping to a narrower moving field of view.

MetricReal-VAS OursCOCOA [59]COCOA-cls [14]D2S [13]MOVi-MC-AC [33]SAIL-VOS 3D [19]SAIL-VOS [20]DYCE [11]
Image or VideoVideoImageImageImageVideoVideoVideoImage
Accurate Annotations✓✗✗✗✓✓✓
Scene Images14,6775,0733,4995,600293,904237,611111,6545,500
Unique Objects160-80601,03317816279
Occlusion Prevalence72.53%60.7%49.0%56.9%69.3%-87.2%82.3%
Mean Occlusion Severity67.33%18.8%10.7%15.0%45.2%-56.3%27.7%

Table 1. Comparison of amodal segmentation datasets. Occlusion Prevalence measures how often each instance is occluded. Mean Occlusion Severity measures what proportion of an object is occluded on average when they are occluded.

Related Work

Amodal Segmentation Methods. Current amodal segmentation methods can be categorized by architectural paradigm and temporal context usage. Architecturally, feedforward models like TCOW [45] directly predict amodal states, while generative approaches treat completion as conditional in-painting using diffusion priors. Image-based diffusion methods [35, 51, 52] process frames independently, whereas video-based approaches [7, 21, 25] operate offline on sequences for temporal consistency. These methods rely on various contextual cues, often requiring additional inputs like modal masks [7, 21, 25, 35, 51] or bounding boxes [51], and employ techniques like depth estimation [7, 21, 35] or target region estimation [21].

Taking an alternative approach, [50] introduced a self-supervised framework (SaVos) for amodal video object segmentation. By exploiting the observation that occluded parts of an object in one frame are often visible in other frames, SaVos leverages motion cues and optical flow to learn amodal completions without requiring manually annotated amodal ground truth, demonstrating strong results on both synthetic and real-world benchmarks.

The field has recently seen significant advances, particularly in adapting foundation models. SAMBA [29] introduces the first SAM-based amodal segmentation foundation model for zero-shot inference, while SAMEO [42] adapts SAM as a versatile amodal mask decoder, also introducing a large-scale synthetic training dataset. For video amodal completion, TACO [30] repurposes video diffusion models for in-the-wild scenarios, and A2VIS [44] incorporates amodal representation

tations into video instance segmentation. Other work has extended amodal reasoning to language-guided settings [2, 28] and shown that amodal segmentation emerges when scaling diffusion models for perceptual tasks [38].

Among the state-of-the-art methods evaluated in our benchmarks, Diffusion VAS [7] and TABE [21] formulate amodal completion as a conditional video generation task. Diffusion VAS leverages pre-trained diffusion priors combined with depth estimation to hallucinate occluded regions while maintaining temporal consistency. Similarly, TABE employs an offline video diffusion architecture, leveraging global temporal context to track object states through severe and complex occlusions.

Amodal Segmentation Datasets. Existing datasets (Tab. 1) face a trade-off between annotation accuracy and real-world fidelity. Synthetic datasets achieve perfect ground truth via physics modeling [12, 20, 32, 39, 45], motion tools [15, 16, 40, 43, 50], and image occlusion [11, 13], but suffer from sim2real gaps [17, 47]. Real-world approaches face annotation challenges since occluded regions require human estimation rather than ground truth measurement—a process that is time-intensive, inaccurate, and subjective [14, 36, 59]. Large-scale tracking benchmarks like MOTChallenge [9] and TAO-Amodal [18] rely on human estimates of bounding boxes rather than dense, pixel-level amodal masks. 3D acquisition methods [27, 51] provide precise masks but lack scalability. A compositing approach has also been adopted for the related task of amodal depth estimation [26], though not for pixel-level amodal segmentation in video.

Compositing approaches, successful in other vision tasks [5, 10, 23, 46, 56], often rely on simplistic overlays in amodal segmentation [25, 35], creating realism gaps. Additionally, existing datasets focus on simple “pass-by” occlusions, neglecting challenging containment scenarios where objects physically interact [45]. TCOW’s real-world evaluation only annotates unoccluded frames, reducing the task to re-identification rather than true amodal segmentation (see Table 2).

RubricRubricRubric
OfficeCup GamesDAV/YTB
TCOW69.438.352.8
SAM272.755.868.9

Table 2. Comparison of TCOW [45] and SAM2 [37] mean IoU on amodal segmentation datasets. Despite SAM2 providing only modal segmentation, it outperforms TCOW’s amodal result.

To address these significant gaps in existing work, we introduce Real-VAS, a novel dataset and data generation pipeline for zero-shot video amodal segmentation. Crucially, Real-VAS operates on a scale previously unseen in real-world amodal benchmarks, containing over 3 times more occluded frames than its closest competitor. Our approach is designed to synthesise the strengths of prior methods while mitigating their respective drawbacks. Our Occlusion methodology uses a compositing pipeline on real-world footage, achieving the precise ground truth of synthetic data without the sim2real gap or the imprecision and high cost of manual annotation. To overcome the realism gap of prior compositing work, our Occlusion subset features dynamic interactions between moving objects and includes emulated camera motion. Furthermore, to move beyond simple pass-by events and benchmark physical reasoning, we introduce the novel Containment category with three physically-grounded scenarios—On Surface Containment, Articulated Containment, and Mobile Containment.

Task

The zero-shot video amodal segmentation (ZS-VAS) task is defined as follows. We are given as input a TT-frame video x∈RT×H×W×3\mathbf{x} \in\mathbb{R}^{T \times H \times W \times3} along with a binary query mask mq∈{0,1}H×W\mathbf{m}_{q} \in\{0,1\}^{H \times W} which segments the object of interest in some reference frame (assumed to be frame 1). It is assumed that the object is completely unoccluded in the reference frame. The query mask may itself have been computed from a prompt of some form (e.g. text or point clicks). The goal is to learn a function ff that estimates amodal segmentation masks for the tracked object in every frame, mo=f(x,mq)\mathbf{m}_{o} = f(\mathbf{x}, \mathbf{m}_{q}), where mo∈{0,1}T×H×W\mathbf{m}_{o} \in\{0,1\}^{T \times H \times W} is the binary object mask in every frame. This must delineate the object even behind occluders and even when none of the object is visible based on information from surrounding frames and the context of the object itself. This initialisation differs from datasets like TAO-Amodal (amodal object tracking) and SAIL-VOS (synthetic amodal object segmentation), where the object of interest may be occluded from the start, often requiring a model to rely on pre-existing class knowledge for identification. The zero-shot paradigm is therefore class-agnostic; it evaluates the more general and realistic ability of a model to segment any novel object based purely on the provided query, rather than its familiarity with specific object categories.

We created this dataset to be an evaluation task, providing a wide-ranging benchmark of real-world scenarios, dynamic object classes, and physically complex interactions designed to test the limits of zero-shot amodal segmentation and aid in the development of the field.

Evaluation Metrics

The standard metrics used for evaluating amodal segmentation are mean Intersection-over-Union (mIoU) and mIoU computed over only occluded pixels, referred to as mIoUocc_{\mathrm{occ}} [7, 12, 15]. However, pooled mIoU is misleading in this regime: high scores on unoccluded frames obscure poor performance on heavily occluded ones, allowing a purely modal tracker like SAM2 to score deceptively well (0.621 in Table 3) while completely failing under total occlusion (mIoUocc_{\mathrm{occ}} = 0.066). For this reason we report a hierarchy of

SAM2TABEDiffusionVAS
mIoU0.6210.5420.655
mIoUocc_{\mathrm{occ}}0.0660.2070.262

Table 3. Standard mIoU is a misleading metric for amodal segmentation. A purely modal method (SAM2) achieves a high overall score while failing on occluded regions (mIoUocc_{\mathrm{occ}}).

Qualitative examples from the Real-VAS dataset

Figure 2. Qualitative examples from the Real-VAS dataset. The yellow dashed line indicates the initial query mask, followed by the ground-truth amodal segmentation frames. The rows showcase our scenarios: On Surface Containment, Articulated Containment, Mobile Containment, and two Dynamic Occlusion examples, one captured in-house (row 4) and one sourced from the web (row 5).

three metrics, each targeting a different failure mode. mIoUfo\mathrm{mIoU}_{\mathrm{fo}} is our primary metric: it measures overall tracking stability over all frames with any level of occlusion, including partial occlusion. mIoUffo\mathrm{mIoU}_{\mathrm{ffo}} is a stricter variant restricted to fully occluded frames, isolating the model’s ability to hallucinate the object during total invisibility. mIoUocc\mathrm{mIoU}_{\mathrm{occ}} measures per-pixel precision on the occluded region only, and so is not inflated by correctly predicting visible pixels:

mIoUfo=∑i∈Focc∣A^i∩Ai∣∑i∈Focc∣A^i∪Ai∣,mIoUffo=∑i∈Ffull∣A^i∩Ai∣∑i∈Ffull∣A^i∪Ai∣\mathrm{mIoU}_{\mathrm{fo}} = \frac{\sum_{i \in F_{\mathrm{occ}}} \lvert\hat{A}_i \cap A_i \rvert}{\sum_{i \in F_{\mathrm{occ}}} \lvert\hat{A}_i \cup A_i \rvert}, \quad\mathrm{mIoU}_{\mathrm{ffo}} = \frac{\sum_{i \in F_{\mathrm{full}}} \lvert\hat{A}_i \cap A_i \rvert}{\sum_{i \in F_{\mathrm{full}}} \lvert\hat{A}_i \cup A_i \rvert}
mIoUocc=∑i∈Focc∣(A^i−Mi)∩(Ai−Mi)∣∑i∈Focc∣(A^i−Mi)∪(Ai−Mi)∣\mathrm{mIoU}_{\mathrm{occ}} = \frac{\sum_{i \in F_{\mathrm{occ}}} \lvert(\hat{A}_i - M_i) \cap(A_i - M_i) \rvert}{\sum_{i \in F_{\mathrm{occ}}} \lvert(\hat{A}_i - M_i) \cup(A_i - M_i) \rvert}

where MiM_i is the ground-truth modal mask, A^i\hat{A}_i the predicted amodal mask, and AiA_i the ground-truth amodal mask for frame ii. We define FoccF_{\mathrm{occ}} as the set of occluded frames and Ffull⊆FoccF_{\mathrm{full}} \subseteq F_{\mathrm{occ}} as the set of fully occluded frames (for which Mi=∅M_i = \varnothing).

Real-VAS Dataset

We introduce Real-VAS, a novel dataset with precise ground-truth annotations for the evaluation of video amodal segmentation. As shown in Figure 1, Real-VAS is constructed using two distinct pipelines, ‘occlusion’ and ‘container’, to test performance across a diverse range of amodal scenarios, while remaining automatic and scalable to enable the creation of large-scale datasets. To ensure the final quality of the dataset, we include two manual quality control checks (Appendix A.5).

We use two sources of raw data: controlled scenes captured in-house and scraping diverse videos from public video repositories. This hybrid approach allows us to combine staged, highly controlled interaction scenarios with the vast diversity of real-world footage. Our in-house data was captured using a tripod-mounted iPhone 15, recording at a high frame rate (slow-motion) with locked focus and exposure to ensure consistent framing and minimal motion blur for dynamic objects. For the externally sourced data, we utilised footage from both existing large-scale video datasets [6, 41] and in-the-wild videos from public platforms such as YouTube, filtered to provide videos from static cameras.

Real-VAS Occlusion

We use a compositing strategy to obtain precise masks for occluded objects. For scenes we capture ourselves, we record two clips with a static camera: one showing the object to be occluded (the “occludee”) without obstruction, and a second clip showing the “occluder”. For scraped video data, we propose an automated pipeline for finding suitable pairs of sub-clips that contain a compatible occluder and occludee. We begin by feeding text prompts from a predefined list of dynamic object classes (see Appendix § 7.§ 1) into Language Segment-Anything [31]. This open-vocabulary model segments the corresponding objects by combining a grounding detector [53] with SAM2. The resulting masks are then tracked across all frames with SAM2.

To ensure the dataset is both high-quality and challenging, a dynamic object checker filters for sequences where both the occluder and occludee are dynamic. To determine if an object is dynamic, we calculate a mean motion score SS based on the trajectories of its points over time, gathered using CoTracker3 [22]. Let the set of trajectories for a single object be represented by T∈RT×P×2\mathbf{T} \in\mathbb{R}^{T \times P \times2}, where TT is the total number of frames (time steps), PP is the number of points tracked on the object, pt,i∈R2\mathbf{p}_{t,i} \in\mathbb{R}^{2} denotes the (x,y)(x,y) coordinates of the ii-th point at frame tt. We define the mean motion score, SS, as the average displacement (Euclidean distance) of all points over all frames:

S=1PT∑i=1P∑t=1T−1∥pt+1,i−pt,i∥2.S = \frac{1}{PT} \sum_{i=1}^{P} \sum_{t=1}^{T-1} \left\lVert\mathbf{p}_{t+1,i} - \mathbf{p}_{t,i} \right\rVert_{2}.

An object is classified as dynamic if its mean motion score exceeds a predefined minimum motion threshold, S>τS > \tau. This formulation effectively captures the object’s overall movement by averaging the speed of its constituent points across the entire video sequence.

Now that we have segmentation masks for dynamic objects in the video we can find where composited occlusion can be created. The core purpose of this method is to discover and validate high-quality object occlusion events by finding the best way to temporally align two separate video sequences of object masks (M1M_1 and M2M_2). The process is structured into two main stages: Candidate Generation and Rigorous Filtering.

Candidate Generation via Cross-Correlation. The algorithm first identifies the most promising temporal alignments by calculating an overlap score for every possible time shift (Δt\Delta t) between the two valid mask sequences, i.e. a cross-correlation. The score S(Δt)S(\Delta t) for a given shift is the sum of the spatial intersection (the overlapping pixel area) over the duration of the sequence:

S(Δt)=∑t∣M1[t]∩M2[t−Δt]∣,S(\Delta t) = \sum_t \lvert M_1[t] \cap M_2[t - \Delta t] \rvert,

where ∣⋅∣\lvert\cdot\rvert denotes the number of pixels in the intersection. We compute this score for all possible shifts and sort them, creating a ranked list of the best potential occlusion events.

Rigorous Filtering Pipeline. Next, the algorithm iterates through the top-ranked candidates and subjects each one to a comprehensive set of validation checks to ensure the resulting composite scene is physically plausible, interesting, and non-redundant. For each candidate, the objects are first assigned foreground (FG) and background (BG) roles based on their relative depth values. The sequence is then trimmed or rejected if it fails any of the following key tests: Physical plausibility: We perform several automated checks using monocular depth estimates from Depth Anything V2 [49]. First, we prevent Reverse Occlusion by analysing the pixel region where the foreground (FG) and background (BG) masks overlap. The sequence is rejected if any FG pixel in this region has a greater depth than its corresponding BG pixel. Second, our “On Top” Detection filters out scenarios where objects appear to improperly occupy the same 3D space. Finally, we check for External Occlusion by detecting if an object’s size unexpectedly reduces. If the depth indicates a closer surface at the point of shrinkage, we infer an external occlusion and discard the candidate.

Quality & Interest Checks: First, every sequence must have a clean start, meaning the background object is fully visible in the initial frame. Next, the interaction must result in sufficient occlusion, where the occluded area exceeds a minimum threshold.

Redundancy Checks: A similarity test discards candidate sequences that are too visually similar to previously accepted ones, while a frame overlap check prevents the same source frames from being used for both the foreground and background objects.

Only candidates that successfully pass this entire pipeline are considered valid and passed forward to the next stage of the pipeline.

High Resolution Sequence Generation. To ensure computational efficiency, the preceding steps operated on video clips temporally downsampled to 1–3 fps. This frame rate provided sufficient temporal resolution to find candidate overlaps while minimising processing time. To convert the coarse, low-framerate mask sequences (1–3 fps) into high-quality masks at the video’s native framerate, we employ a propagation-based approach once again using SAM2. This process involves an initialisation step followed by a focused, fine-grained segmentation pass. The process begins by identifying the first valid mask in the low-framerate sequence to serve as an initial prompt. However, temporal downsampling can miss the true first appearance of an object. To correct this, we perform a preliminary backwards process: the segmentation model is run on a reversed video at high framerate, immediately preceding this initial mask. This refinement step pinpoints the object’s true starting frame and provides a more accurate initial mask for the main forward process. To generate fine-grained masks with high efficiency and precision, we first create a dense sequence of guiding bounding boxes. This is done by linearly interpolating the sparse set of boxes from the low-framerate sequence, one for every high-framerate frame, and adding a motion-compensation padding. The segmentation model is then applied exclusively within these tight, content-focused crops of the high-framerate video. This approach significantly boosts both processing speed and the resulting boundary detail. Finally, these high-detail mask segments are re-projected onto the full-frame canvas at their original coordinates, and the backwards process is run again to refine the initial frames of the new, high-resolution sequence.

Compositing. With our high-resolution sequences, we use a multi-stage compositing pipeline to create the final dataset by realistically overlaying the foreground object, IfgI_{\mathrm{fg}}, onto the background scene, IbgI_{\mathrm{bg}}. We do so using a standard alpha blend with a soft alpha matte, αfg∈[0,1]\alpha_{\mathrm{fg}} \in[0, 1], to allow for seamless, semi-transparent boundaries. The composite image, IcompI_{\mathrm{comp}}, is given by:

Icomp=(1−αfg)⋅Ibg+αfg⋅Ifg.I_{\mathrm{comp}} = (1 - \alpha_{\mathrm{fg}}) \cdot I_{\mathrm{bg}} + \alpha_{\mathrm{fg}} \cdot I_{\mathrm{fg}}.

In practice, the alpha mattes are provided by Generative Omnimatte [25]. This model captures both the foreground object and its secondary effects on the scene, such as cast shadows or reflections, and represents them in the soft alpha matte. Including these effects significantly increases the realism of the final composite, as shown in Figure 3. It is important to note that the soft alpha matte is used strictly for seamless compositing of boundaries. The interior of the occluder remains fully opaque (αfg=1\alpha_{\mathrm{fg}} = 1), ensuring the background object is completely occluded in the final video data.

Comparison of data compositing with and without Generative Omnimatte

Figure 3. Comparison of data compositing with and without Generative Omnimatte. The bottom row, using our method, successfully captures secondary effects such as shadows and reflections. This results in a more realistic composite compared to the baseline method shown in the top row.

Real-VAS Containment

Our Container category is composed of three physically distinct scenarios (see Figure 1). While the initial interactions are unique, all three are built upon a unified principle for generating precise ground truth: once an object is fully contained, its motion is dictated by the motion of its container. To enable this, we enforce a physical constraint by using form-fitting object-container pairs. Computationally, this process relies on two key models. First, SAM2, is used to track an object while it is visible, establishing its last known mask before occlusion. Second, once the object is hidden, a point tracker, CoTracker3, captures the container’s precise transformation. This transformation is then directly applied to the object’s last known mask, treating them as a single rigid unit. A simpler alternative would be be to propagate the object’s mask by the container’s frame-to-frame centroid displacement. This is insufficient in our setting: a centroid recovers only a translation vector, whereas a snugly-contained object inherits the container’s full rigid or affine motion, including scale changes as the container moves toward or away from the camera, and rotation. CoTracker3 tracks multiple surface points on the container, from which we can estimate the full transform at sub-pixel precision.

Formally, let OtO_t and CtC_t denote the target-object and container masks at frame tt, and let tentryt_{\mathrm{entry}} be the frame at which the object becomes fully contained. During the visible phase (t<tentryt < t_{\mathrm{entry}}), OtO_t is tracked directly with CoTracker3. At tentryt_{\mathrm{entry}} the object’s rigid boundary becomes tightly bounded by the inner topology of CC, and the motion model transitions to

TOt=TCt+ϵ,∀ t≥tentry,\mathbf{T}_{O_t} = \mathbf{T}_{C_t} + \epsilon, \qquad\forall\, t \geq t_{\mathrm{entry}},

where T\mathbf{T} denotes the estimated rigid/affine transform and ϵ\epsilon is a bounded local-deformation term whose magnitude we validate empirically in Figure 9 of the supplementary (mean IoU 97.5%). When an external actuator (e.g. a human hand) manipulates OO during insertion, we substitute the actuator’s tracked translation for TCt\mathbf{T}_{C_t} until the object settles, at which point tracking transfers to CC (see Figure 11 in the supplementary). The difference between scenarios lies in how these models are applied during the initial interaction:

Validation of containment logic across three key scenarios

Figure 9. Validation of our containment logic across three key scenarios: On-Surface, Articulated, and Mobile Containment. For each example, the rows display (from top to bottom): the original frame, the synthetically applied container occlusion, the ground truth mask, and the final mask predicted by our logic. Our approach is highly accurate, achieving a 97.5% Intersection over Union (IoU) with the ground truth masks.

A visualisation of the actuator tracking mechanism used during the Mobile Containment phase

Figure 11. A visualisation of the actuator tracking mechanism used during the Mobile Containment phase. The red point indicates the tracked location on the actuator (the human hand), while the green mask represents the object being placed into the container. By tracking the hand’s motion, we accurately simulate the object’s settling trajectory as it becomes fully occluded.

On Surface Containment: inspired by the shell game, the object is static. The CoTracker3 logic is applied to the container from the moment of full occlusion.

Articulated Containment: we first use SAM2 to track the object’s visible movement as it enters a fixture (e.g. a drawer). Once occluded, we switch to using CoTracker3 on the container.

Mobile Containment: requires a multi-stage entry where we first reconstruct the partially occluded mask. To simulate the object settling, we infer the object’s movement by tracking the visible actuator (e.g. a human hand or tool) using CoTracker3 during the insertion phase. By assuming a fixed spatial relationship between the actuator and the object while it is being placed, we apply the tracked motion of the actuator directly to the object’s last-known mask to determine its trajectory, speed, and final resting position within the container (see Supplementary Material). Only after the actuator releases the object and is removed do we engage the CoTracker3 logic on the container itself.

Emulated Camera Motion

To ensure our dataset is as realistic and varied as possible, we emulate camera motion for scenes where the original data was collected with a static camera (Figure 4). This is mainly applied for our Real-VAS Occlusion set, which requires static camera data as input, whereas our container logic allows for camera motion. Our approach applies a diverse set of predefined camera motion profiles (e.g. pans, zooms, and translations) to the static source videos.

Example of emulated camera motion showing original frames, camera areas, and transformed frames

Figure 4. An example of emulated camera motion. This figure demonstrates a dynamic zoom and pan by showing the original frame, the specific sub-region that constitutes the new camera view, and the resulting transformed frame.

To ensure that the emulated motion does not corrupt the data by pushing key objects out of frame, we employ an automated validation and selection pipeline. Each potential motion profile is used to generate a candidate sequence, which is then evaluated against two critical criteria. First, we ensure the preservation of visibility, verifying that the emulated motion does not push the primary objects (particularly the occludee) out of the frame. Second, we check for the preservation of occlusion quality, ensuring that the interaction is maintained and the degree of occlusion is not substantially reduced from the original static version. A suitable motion profile is then randomly selected from the pool of candidates that successfully pass these checks, introducing valuable diversity into our dataset. If no emulated motion satisfies our quality constraints for a given scene, the camera remains static, thereby preserving integrity.

Dataset Statistics

Table 1 presents a comparison between Real-VAS and other notable amodal segmentation datasets. Our contribution is significant in several key areas. First, Real-VAS provides precise annotations, removing the imprecision of human estimation found in other real-world datasets. Second, in terms of scale, it contains over 2.5 times as many frames as the next largest real-world benchmark. This scale is matched by its diversity, offering a substantially larger number of unique object classes. Finally, it is specifically designed for the amodal task, featuring both a higher occlusion prevalence and a much greater mean occlusion severity, making it a more challenging and suitable benchmark for evaluating amodal models. When compared to synthetic datasets, Real-VAS naturally trades sheer scale for real-world fidelity. While synthetic benchmarks like MOVi-MC-AC [33] offer a greater volume of scenes and classes, Real-VAS is intentionally curated to feature more challenging and realistic occlusions. This is evidenced by its mean occlusion severity, which is higher than that of most synthetic datasets. This focus on realism and the quality and difficulty of occlusions, rather than just the quantity, makes Real-VAS a suitable evaluation dataset for amodal segmentation. Further dataset metrics are presented in Appendix § 8.§ 5. We also present a scaling analysis (Figure 5) in § 4.5 that validates the sufficiency of this scale for stable benchmarking.

Scaling analysis plots for model performance on the Real-VAS dataset

Figure 5. Scaling analysis of model performance on the Real-VAS dataset. The plots show the mean score for three different mIoU metrics as the percentage of the evaluation data is increased from 10% to 100%. The general convergence of the curves validates the dataset’s scale as a stable and comprehensive benchmark.

While our comparison in Table 1 is comprehensive, we have deliberately excluded some related datasets due to fundamental differences in their annotation formats. For instance, TAO-Amodal [18], despite being a large-scale real-world benchmark, is omitted as it only provides amodal annotations in the form of bounding boxes, which are not comparable to the pixel-level segmentation masks that our work focuses on. Similarly, the ‘Rubric’ real-world evaluation set from TCOW [45] is excluded because it exclusively contains modal annotations (visible parts) of objects preceding and following an occlusion, and therefore lacks the necessary amodal ground truth to evaluate a model’s performance during an occlusion event itself.

Scaling Analysis

To validate Real-VAS as a robust evaluation benchmark, we performed a scaling analysis. We measured the performance of the top performing baseline methods on progressively larger random subsets of our test data, from 10% to 100%. The results (Figure 5) demonstrate two key properties. First, the changing scores in the initial subsets (e.g. from 10% to 70%) highlight the dataset’s diversity, showing that different parts of the dataset contain unique challenges that test various model capabilities. More importantly, as the evaluation size increases towards its full scale (from 80% to 100%), the performance scores for all models begin to stabilise and the curves flatten. This convergence is critical: it indicates that while an evaluation on a small subset might be sensitive to the specific scenes chosen, an evaluation on the full dataset provides a stable and reliable measure of a model’s true performance. The fact that the scores stabilise demonstrates that Real-VAS is sufficiently large and comprehensive to serve as a conclusive benchmark.

Evaluation

To benchmark performance on Real-VAS, we evaluated a diverse set of state-of-the-art (SOTA) models, as shown in Table 4. These benchmarks include image-based models (pix2gestalt [35], SDAmodal [51], PCNet [52]), video diffusion approaches (TABE [21], Diffusion VAS [7], Generative Omnimatte [25]), and video transformers (TCOW [45]). We excluded methods that output 3D amodal shapes from our benchmarks, such as Amodal3R [48], as the lack of ground-truth camera intrinsics in our dataset prevents a fair evaluation of their projected 2D segmentation masks. Several of these frameworks require additional inputs beyond the initial query mask. For consistency, where a Modal Seg. input is required, we provide visible-part segmentation masks generated by SAM2. BBox indicates that the method requires a ground-truth bounding box to define the area for amodal completion. Despite this significant advantage, neither SDAmodal nor PCNet surpasses many methods lacking this extra information. We also distinguish between models by their Run Type. Online methods process video sequentially and are suitable for real-time application, whereas offline methods process the entire clip at once, granting them far more context at the cost of higher latency. A significant performance trade-off exists between modelling paradigms. Video Transformer methods like TCOW offer substantially lower latency, often by orders of magnitude, compared to iterative Video Diffusion approaches such as TABE and Diffusion VAS.

MethodInputsRun TypemIoUfo\mathrm{mIoU}_{\mathrm{fo}}mIoUffo\mathrm{mIoU}_{\mathrm{ffo}}mIoUocc\mathrm{mIoU}_{\mathrm{occ}}
pix2gestalt [35]Modal Seg.Online0.411 ±\pm0.21670.060 ±\pm0.10740.136 ±\pm0.1088
TCOW [45]/Offline0.426 ±\pm0.23040.179 ±\pm0.19070.165 ±\pm0.1324
SDAmodal [51]Modal Seg. & GT BBoxOnline0.526 ±\pm0.24330.059 ±\pm0.11760.146 ±\pm0.1293
Generative Omnimatte [25]Modal Seg.Offline0.416 ±\pm0.21940.045 ±\pm0.12840.062 ±\pm0.1002
TABE [21]Modal Seg.Offline0.501 ±\pm0.27680.209 ±\pm0.25720.207 ±\pm0.1760
PCNet [52]Modal Seg. & GT BBoxOnline0.452 ±\pm0.24210.030 ±\pm0.11140.011 ±\pm0.0435
Diffusion VAS [7]Modal Seg.Offline0.604 ±\pm0.25800.267 ±\pm0.29440.262 ±\pm0.2087

Table 4. Quantitative results on complete Real-VAS dataset.

Quantitative results are shown in Table 4. A clear trend emerges where video-based methods (e.g. TABE [21], Diffusion VAS [7], TCOW [45]) generally out-perform image-based approaches (pix2gestalt [35], SDAmodal [51], PCNet [52]). This is expected, as video models can leverage temporal context, which is particularly useful for interpolating across a full occlusion. A notable exception is the lower performance of Generative Omnimatte [25]; this result should be contextualised, as our evaluation protocol intentionally withholds the foreground (occluder) masks that the model is designed to use, thereby highlighting its dependency on that specific input. To further analyse model performance, Table 5 presents results split by our Occlusion and Containment scenarios. We observe a consistent drop in performance for all methods on the Containment tasks when compared to the simpler Occlusion events. This underscores the difficulty of these interactive scenarios and confirms the importance of including them in a comprehensive amodal segmentation benchmark. It is noteworthy, however, that TCOW outperforms all other methods on Containment mIoUffo\mathrm{mIoU}_{\mathrm{ffo}} and mIoUocc\mathrm{mIoU}_{\mathrm{occ}}, this is likely attributable to its training on synthetic containment-style videos, which demonstrates that performance on these complex physical reasoning tasks is heavily influenced by exposure to similar training data. To further investigate cross-dataset generalisation and the sim2real gap, we evaluated the transferability of models trained on synthetic data to our real-world benchmark. Using Diffusion VAS as a case study, we observed a substantial performance degradation. When evaluated on the large-scale synthetic SAIL-VOS dataset [20], Diffusion VAS achieved a reported mIoUocc\mathrm{mIoU}_{\mathrm{occ}} of 0.55 [7]. However, when this same model is evaluated on the Real-VAS Occlusion split, performance drops significantly to 0.31 mIoUocc\mathrm{mIoU}_{\mathrm{occ}}. The degradation is even more severe on the Real-VAS Containment split, where the model achieves just 0.13 mIoUocc\mathrm{mIoU}_{\mathrm{occ}}. This substantial performance gap highlights that Real-VAS captures distinct, real-world challenges and complex physical interactions that are fundamentally absent from existing synthetic benchmarks, confirming the necessity of a real-world evaluation dataset.

MethodReal-VAS ContainmentReal-VAS Occlusion
mIoUfo\mathrm{mIoU}_{\mathrm{fo}}mIoUffo\mathrm{mIoU}_{\mathrm{ffo}}mIoUocc\mathrm{mIoU}_{\mathrm{occ}}mIoUfo\mathrm{mIoU}_{\mathrm{fo}}mIoUffo\mathrm{mIoU}_{\mathrm{ffo}}mIoUocc\mathrm{mIoU}_{\mathrm{occ}}
pix2gestalt [35]0.203 ±\pm0.110.016 ±\pm0.030.053 ±\pm0.040.478 ±\pm0.200.096 ±\pm0.130.163 ±\pm0.11
TCOW [45]0.325 ±\pm0.140.108 ±\pm0.140.128 ±\pm0.110.458 ±\pm0.240.236 ±\pm0.210.177 ±\pm0.14
SDAmodal [51]0.335 ±\pm0.190.022 ±\pm0.040.076 ±\pm0.050.587 ±\pm0.230.088 ±\pm0.150.168 ±\pm0.14
Generative Omnimatte [25]0.250 ±\pm0.150.000 ±\pm0.000.003 ±\pm0.010.470 ±\pm0.210.082 ±\pm0.160.081 ±\pm0.11
TABE [21]0.225 ±\pm0.140.044 ±\pm0.060.118 ±\pm0.090.590 ±\pm0.250.340 ±\pm0.280.236 ±\pm0.19
PCNet [52]0.353 ±\pm0.2270.013 ±\pm0.020.014 ±\pm0.020.533 ±\pm0.190.070 ±\pm0.190.009 ±\pm0.05
Diffusion VAS [7]0.368 ±\pm0.200.066 ±\pm0.140.125 ±\pm0.130.681 ±\pm0.230.428 ±\pm0.290.306 ±\pm0.21

Table 5. Quantitative evaluation on Real-VAS, separated by Containment and Occlusion scenarios.

Beyond evaluation, Real-VAS also provides useful training signal: fine-tuning Diffusion VAS on a random 300-sequence subset improves both in-domain performance and cross-domain generalisation to TAO-Amodal (see Sec. B.3).

Discussion

In this work, we addressed the critical challenge of creating a large-scale, high-fidelity dataset for video amodal segmentation. We introduced Real-VAS, a novel dataset generated through a sophisticated pipeline that leverages real-world footage to produce programmatically consistent ground-truth annotations, avoiding the independent annotation error found in manually-labelled real-world benchmarks. This approach successfully bridges the sim2real gap found in purely synthetic data while bypassing the inaccuracies and high costs associated with manual annotation or 3D reconstruction. A key contribution of our work is the introduction of the Container scenarios, which push beyond the simple pass-by occlusions prevalent in existing datasets. By providing structured, physically-grounded interactions such as On Surface Containment, Articulated Containment, and Mobile Containment, Real-VAS establishes a new and challenging benchmark for evaluating a model’s capacity for physical reasoning and object permanence. The inclusion of both Occlusion and Container scenarios is deliberate, as we hypothesise they require different model capabilities. For instance, the structured logic of Containment may be solvable by models that excel at physical or relational reasoning, yet these same models might fail at the visual completion needed for unstructured Occlusion. Conversely, powerful generative models that can hallucinate appearance for Occlusion events may lack the explicit physical constraints required to succeed at Containment. Our dataset is designed to spur the development of unified solutions that can solve both. Our pipeline further enhances realism by emulating camera motion and incorporating secondary effects like cast shadows.

Scope and Limitations. Real-VAS is a composited real-image benchmark: the source footage is real, and camera noise, texture and sensor artefacts are preserved, but occlusion events are synthesised by compositing rather than physically recorded. This design choice lets us scale programmatically consistent amodal ground truth without 3D-capture rigs or subjective human annotation of hidden regions, at the cost of occasional lighting inconsistencies or contact-point artefacts. Our Container scenarios further rely on a bounded “snug-fit” proxy constraint, which does not cover loose containment, such as a small object moving freely inside a large box. Future work could extend the generation pipeline to include controlled real-world camera motion, fluid dynamics, and more complex self-occlusion.

Further Dataset Generation Details

Segmentation Class List

To identify objects that are inherently dynamic, we define a comprehensive set of ‘moving object’ classes. This set is broadly categorized to cover a wide range of potential motion.

It includes biological entities such as person, animal, bird, dog, cat, fish, horse, and insect. It also features a wide array of vehicles, from terrestrial (car, bus, motorcycle, train) and marine (boat, ship) to aerial (airplane, helicopter, drone). Furthermore, we include mechanical systems and devices known for their movement, like robot, conveyor belt, fan, and clock hands. Finally, the list is rounded out by common recreational and everyday objects whose behavior is often dynamic, including ball, kite, swing, and shopping cart.

Use of CoTracker3 Within Containment Logic

For our ground-truth generation in containment scenarios, we found that running CoTracker3 in online mode was significantly more robust than in offline mode. This is because the offline tracker, which processes the entire video at once, was prone to identity switching. In cases where multiple identical containers crossed paths, the offline model would incorrectly assign the track to a different container that later occupied the same space. The online, frame-by-frame approach mitigates this issue by maintaining a continuous track without being influenced by future events.

Combining Monocular Depth with Segmentation

In cases where segmentation (SAM2) struggles with fine details, we leverage monocular depth (Depth Anything V2) to refine compositing. By calculating the average depth within the mask, we can exclude regions with greater depth values, effectively isolating the background. This technique substantially improves the realism of our composites. As shown in Figure 6, it allows for visibility through intricate structures, like the gaps between the leaves of a plant, correctly revealing the parts of the person.

Five-panel illustration showing segmentation refinement using monocular depth

Figure 6. Refining a segmentation mask using monocular depth. From left to right: (a) The initial composite showing the person occluded by the plant (missing upper torso and upper leg). (b) The coarse mask from SAM2. (c) The corresponding depth map. (d) The refined mask after using depth to exclude background pixels. (e) The final, improved composite, where details of the person are correctly revealed through the gaps in the plant.

Dataset Generation Speed and Computation Requirements

Our data generation pipeline was executed on an NVIDIA A40 GPU. While it is not strictly a requirement, a GPU with a large VRAM is beneficial for processing long video sequences, where the computational demands of the SAM2 mask propagation step can be significant.

Quantifying the precise generation speed is challenging due to the highly parallelised and automated nature of the pipeline. In particular, the clip alignment stage, where our system searches for plausible occlusion events, was designed to run overnight on large batches of videos to maximise the number of high-quality candidates discovered.

Manual Quality Control Verification Stages

While our data generation pipeline is fully automated, we incorporate two quality control checks on the videos processed in the Real-VAS dataset to ensure the final dataset meets a high standard of quality and accuracy required for an evaluation dataset. We have built tools to ensure these QC checks are highly efficient.

First, following the automated mask generation, we perform a rapid curation step. Using a lightweight, purpose-built GUI (Figure 8), human curators can quickly cycle through masks to correct obvious segmentation errors or add any that were missed. This interface is designed for speed, turning a potentially laborious task into a quick quality check that focuses only on refining clear inaccuracies.

The GUI for the mask verification step

Figure 8. The GUI for the mask verification step. This tool allows users to easily correct inaccuracies by adding, modifying, or removing masks.

Second, after a full sequence has been composited, it undergoes a final acceptance review. Curators use a simple viewing tool to watch the short clip and give a quick pass/fail judgement based on its overall realism and interest, as shown in Figure 7. This final check is extremely fast and ensures that only the most compelling and accurate sequences are included in the dataset. By designing these lightweight interfaces, our human-in-the-loop stages serve as an efficient quality control mechanism rather than a time-consuming annotation bottleneck.

Results Checker GUI showing a segmented video frame and verification controls

Figure 7. Our simple GUI for the final verification step, where generated clips are quickly accepted or rejected based on their overall quality.

To quantify the fidelity of the resulting masks, we additionally re-annotated a random sample of 500 frames drawn from 50 sequences in CVAT and compared the manual masks against those produced by our pipeline. The mean per-sequence IoU is 0.966, with a minimum of 0.947, confirming high fidelity on the source segmentation that underlies the composited ground truth.

Ablation Studies

Precision of Containment Logic

A key challenge for our Containment scenarios is that once an object is hidden, its true position cannot be visually verified. To quantitatively validate the precision of our ground-truth generation logic, we therefore conducted a specialised experiment using videos of transparent containers.

This validation process involves two steps. First, we establish a verifiable ground truth by running a standard segmentation model on the original transparent videos. Since the object is always visible through the container, this provides its true mask for every frame. Second, we digitally render the transparent containers opaque in these same videos and then run our proposed containment logic. This logic takes the object’s last visible mask before it becomes hidden and uses a point tracker on the (now opaque) container to predict the object’s subsequent amodal masks.

Across all our container scenarios, comparing our logic’s predicted masks against the verifiable ground truth yields a mean IoU of 97.5%. While this score is very high, we note that IoU is highly sensitive to minor pixel shifts, and the segmentation masks used as a reference are not perfectly noise-free. Nevertheless, this result demonstrates that our containment logic is highly precise. We showcase the experimental setup for each scenario in Figure 9.

Pipeline Component Ablation

To quantify how each stage of our generation pipeline contributes to benchmark difficulty, we ablate two components, emulated camera motion (Figure 4) and Generative Omnimatte compositing (Figure 3), and re-evaluate two representative models. Because the pipeline produces evaluation data rather than model outputs, we treat zero-shot performance as a proxy for realism: components that increase realism should make the task harder.

Table 6 confirms this. Removing camera motion produces the largest single gain for TCOW, as the model no longer needs to disentangle ego-motion from true object displacement. Removing Generative Omnimatte accounts for a further, independent gain, because the resulting composites have harsher artificial boundaries and lack the cast shadows and reflections that would otherwise prevent models from exploiting simple texture discontinuities. Removing both components is easiest of all for both models, confirming that they contribute independently to difficulty.

VariantTCOW mIoUfo_{\mathrm{fo}}Diffusion VAS mIoUfo_{\mathrm{fo}}
Real-VAS (full)0.4260.604
w/o Generative Omnimatte0.5240.619
w/o camera motion0.5470.613
w/o both0.5600.628

Table 6. Pipeline component ablation. Both emulated camera motion and Generative Omnimatte compositing independently contribute to benchmark difficulty.

Training Utility

While Real-VAS is designed primarily as an evaluation benchmark, we verify that it also provides useful training signal. We fine-tune Diffusion VAS [7] on a random 300-sequence subset of Real-VAS and evaluate on (i) the held-out 100-sequence test split, and (ii) TAO-Amodal [18] zero-shot. Results are shown in

Tab. 7. Fine-tuning yields improvements on both in-domain and cross-domain

evaluations. In particular the AP75_{75} gain on TAO-Amodal (0.720 →\to 0.742), a strict metric requiring high-overlap masks, indicates that the precise composited masks in Real-VAS teach the model tighter output estimates than those learned from the much larger synthetic SAIL-VOS [20] corpus used during pre-training. Achieving these gains from only 300 sequences suggests Real-VAS transfers substantially more efficiently than existing synthetic training data, and that scaling the pipeline offers a viable path to a large-scale real-world training set.

TAO-Amodal (Top-1)Real-VAS
AP25_{25}AP50_{50}AP75_{75}mIoUfo_{\mathrm{fo}}mIoUocc_{\mathrm{occ}}
Baseline0.9730.8930.7200.6280.288
Fine-tuned0.9740.8990.7420.6520.293

Table 7. Fine-tuning Diffusion VAS on 300 randomly-selected Real-VAS sequences improves both in-domain performance (100-sequence held-out split) and cross-domain generalisation to TAO-Amodal.

Performance vs. Occlusion Duration

To verify that Real-VAS stresses long-horizon object permanence rather than short pass-by occlusions, we bin every evaluation clip by occlusion duration into Short (1–14 frames), Medium (15–41), and Long (42–132) and re-evaluate all baselines. Figure 10 shows the resulting mIoUfo_{\mathrm{fo}} scores within each bin. Every method’s mIoUfo_{\mathrm{fo}} drops from Medium to Long occlusions, confirming that the difficulty of Real-VAS is driven by sustained occlusion rather than an imbalance in short pass-by events.

Performance vs. Occlusion Duration on Real-VAS bar charts

Figure 10. Performance vs. occlusion duration across SOTA models on Real-VAS. Clips are binned by duration into Short (1–14 frames), Medium (15–41), and Long (42–132). Every method’s mIoUfo\mathrm{mIoU}_{\mathrm{fo}} drops from Medium to Long occlusions, validating that Real-VAS tests object permanence over extended periods rather than a bias towards short pass-by events.

Further Dataset Statistics

In Tab. 8 we showcase some additional statistics of our Real-VAS dataset supplementing the comparisons in Tab. 1. These metrics highlight the dataset’s diversity, showing a wide variation in clip lengths, frame rates and resolutions.

Furthermore, Table 9 demonstrates good coverage across our Occlusion and Container scenarios.

Num. Occlusion ClipsNum. Containment Clips
TotalOn SurfaceArticulatedMobile
300100383329

Table 9. The distribution of clips between our Occlusion and Container categories. The number of container scenarios is smaller by design, reflecting the inherent difficulty of sourcing the specific, high-quality physical interactions required for this category. In contrast, simple occlusion events are far more common in in-the-wild footage, allowing for a greater volume and variety.

Use of CoTracker3 On Actuators Within Containment Logic

As discussed in § 4.2, determining the precise resting position of an object during Mobile Containment requires tracking the actuator that places it. Figure 11 illustrates this multi-stage settling process. We initialise a point track (visualised as a red dot) on a consistently visible portion of the actuator, in this case, the human hand, using CoTracker3. By assuming a rigid spatial relationship between the actuator and the object during the insertion phase, the frame-to-frame displacement of this tracked point is applied directly to the object’s last known segmentation mask (shown in green). This methodology ensures that the object’s inferred trajectory, speed, and final settling position inside the container are physically grounded in visual evidence rather than relying on arbitrary physics simulations.

Clip Length (frames)Frames Per SecondResolution
MinMaxAvgMinMaxAvgMinMaxMode
313237106016720 × 12802160 × 38401080 × 1920

Table 8. Further statistics of our Real-VAS Dataset.

A selection of example outputs from our camera emulation module

Figure 12. A selection of example outputs from our camera emulation module. The figure displays four profiles from our larger library of predefined motions, which are applied to static videos to enhance realism and introduce challenging, dynamic viewpoints to the dataset.

Qualitative outputs from each method

Figure 13. Qualitative outputs from each method – Part 1.

Qualitative outputs from each method – Part 2

Figure 14. Qualitative outputs from each method – Part 2.

Qualitative outputs from each method – Part 3.

Figure 15. Qualitative outputs from each method – Part 3.

Qualitative outputs from each method – Part 4.

Figure 16. Qualitative outputs from each method – Part 4.

Qualitative outputs from each method – Part 5.

Figure 17. Qualitative outputs from each method – Part 5.

Further examples from the Real-VAS dataset

Figure 18. Further examples from the Real-VAS dataset. The yellow dashed line indicates the initial query mask, followed by the ground-truth amodal segmentation frames.

References

  1. [1]An, M., Marcinowski, E.C., Hsu, L.Y., Stankus, J., Jancart, K.L., Lobo, M.A., Dusing, S.C., McCoy, S.W., Bovaird, J.A., Willett, S., et al.: Object permanence and the relationship to sitting development in infants with motor delays. Pediatric Physical Therapy 34(3), 309–316 (2022)DOI
  2. [2]Ao, J., Jiang, Y., Ke, Q., Ehlinger, K.A.: Open-world amodal appearance completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)arxiv.org/abs/2411.13019
  3. [3]Ao, J., Ke, Q., Ehlinger, K.A.: Image amodal completion: A survey. Computer Vision and Image Understanding 229, 103661 (2023)DOI
  4. [4]Bergmann, P., Meinhardt, T., Leal-Taixé, L.: Tracking without bells and whistles. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 941–951 (2019)DOI
  5. [5]Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020)arxiv.org/abs/2004.10934
  6. [6]Chavdarova, T., Baqué, P., Bouquet, S., Maksai, A., Jose, C., Lettry, L., Fua, P., Van Gool, L., Fleuret, F.: The wildtrack multi-camera person dataset. arXiv preprint arXiv:1707.09299 (2017)arxiv.org/abs/1707.09299
  7. [7]Chen, K., Ramanan, D., Khurana, T.: Using diffusion priors for video amodal segmentation. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 22890–22900 (2025)arxiv.org/abs/2412.04623
  8. [8]Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299 (2022)arxiv.org/abs/2112.01527
  9. [9]Dendorfer, P., Rezatofighi, H., Milan, A., Shi, J., Cremers, D., Reid, I., Roth, S., Schindler, K., Leal-Taixé, L.: MOTChallenge: A benchmark for single-camera multiple target tracking. International Journal of Computer Vision 129(4), 845–881 (2021)arxiv.org/abs/2010.07548
  10. [10]DeVries, T.: Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017)arxiv.org/abs/1708.04552
  11. [11]Ehsani, K., Mottaghi, R., Farhadi, A.: Segan: Segmenting and generating the invisible. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6144–6153 (2018)arxiv.org/abs/1703.10239
  12. [12]Fan, K., Lei, J., Qian, X., Yu, M., Xiao, T., He, T., Zhang, Z., Fu, Y.: Rethinking amodal video segmentation from learning supervised signals with object-centric representation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1272–1281 (2023)arxiv.org/abs/2309.13248
  13. [13]Follmann, P., Bottger, T., Hartinger, P., Konig, R., Ulrich, M.: Mvtec d2s: densely segmented supermarket dataset. In: Proceedings of the European conference on computer vision (ECCV). pp. 569–585 (2018)arxiv.org/abs/1804.08292
  14. [14]Follmann, P., König, R., Härtinger, P., Klostermann, M., Böttger, T.: Learning to see the invisible: End-to-end trainable amodal instance segmentation. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 1328–1336. IEEE (2019)arxiv.org/abs/1804.08864
  15. [15]Gao, J., Qian, X., Wang, Y., Xiao, T., He, T., Zhang, Z., Fu, Y.: Coarse-to-fine amodal segmentation with shape prior. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1262–1271 (2023)arxiv.org/abs/2308.16825
  16. [16]Girdhar, R., Ramanan, D.: Cater: A diagnostic dataset for compositional actions and temporal reasoning. arXiv preprint arXiv:1910.04744 (2019)arxiv.org/abs/1910.04744
  17. [17]Höfer, S., Bekris, K., K., Andana, A., Gamboa, J.C., Mozifian, M., Golemo, F., Atkeson, C., Fox, D., Goldberg, K., Leonard, J., et al.: Sim2real in robotics and automation: Applications and challenges. IEEE transactions on automation science and engineering 18(2), 398–400 (2021)DOI
  18. [18]Hsieh, C.Y., Chen, K., Dave, A., Khurana, T., Ramanan, D.: Tao-amodal: A benchmark for tracking any object amodally. arXiv preprint arXiv:2312.12433 (2023)arxiv.org/abs/2312.12433
  19. [19]Hu, Y.T., Wang, J., Yeh, R.A., Schwing, A.G.: SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction from Video Data. In: Proc. CVPR (2021)
  20. [20]Hu, Y.T., Chen, H.S., Hui, K., Huang, J.B., Schwing, A.G.: Sail-vos: Semantic amodal instance level video object segmentation-a synthetic dataset and baselines. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3105–3115 (2019)DOI
  21. [21]Hudson, F.G., Smith, W.A.: Track anything behind everything: Zero-shot amodal video object segmentation. arXiv preprint arXiv:2411.19210 (2024)arxiv.org/abs/2411.19210
  22. [22]Karaev, N., Makarov, I., Wang, J., Neverova, N., Vedaldi, A., Rupprecht, C.: Co-tracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.11831 (2024)arxiv.org/abs/2410.11831
  23. [23]Kini, J., Khan, F.S., Khan, S., Shah, M.: Ct-vos: Cutout prediction and tagging for self-supervised video object segmentation. Computer Vision and Image Understanding 238, 103860 (2024)
  24. [24]Kortylewski, A., He, J., Liu, Q., Yuille, A.L.: Compositional convolutional neural networks: A deep architecture with innate robustness to partial occlusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8940–8949 (2020)arxiv.org/abs/2003.04490
  25. [25]Lee, Y.C., Lu, E., Rumbley, S., Geyer, M., Huang, J.B., Dekel, T., Cole, F.: Generative omnimatte: Learning to decompose video into layers. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 12522–12532 (2025)arxiv.org/abs/2411.16683
  26. [26]Li, Z., et al.: Amodal depth anything: Amodal depth estimation in the wild. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2025)arxiv.org/abs/2412.02336
  27. [27]Li, Z., Ye, W., Terven, J., Bennett, Z., Zheng, Y., Jiang, T., Huang, T.: Muva: A new large-scale benchmark for multi-view amodal instance segmentation in the shopping scenario. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 23504–23513 (2023)DOI
  28. [28]Li, Z., Yoon, H., Lee, S., Lin, W.: Unveiling the invisible: Reasoning complex occlusions amodally with AURA. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2025)arxiv.org/abs/2503.10225
  29. [29]Liu, J., Yang, S., Fang, P., Zhang, Y., Wang, L.: Towards efficient foundation model for zero-shot amodal segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)DOI
  30. [30]Lu, R., Chen, Y., Liu, Y., Tang, J., Ni, J., Wan, D., Zeng, G., Huang, S.: TACO: Taming diffusion for in-the-wild video amodal completion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2025)
  31. [31]Medeiros, L.: Language segment-anything. https://github.com/luca-medeiros/ lang-segment-anything (2025)
  32. [32]Mohan, R., Valada, A.: Amodal panoptic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21023– 21032 (2022)arxiv.org/abs/2202.11542
  33. [33]Moore, A., Saini, A., Cancilla, K., Poland, D., Carrano, C.: Training for x-ray vision: Amodal segmentation, amodal content completion, and view-invariant object representation from multi-camera video. arXiv preprint arXiv:2507.00339 (2025)arxiv.org/abs/2507.00339
  34. [34]Nanay, B.: The importance of amodal completion in everyday perception. i-Perception 9(4), 2041669518788887 (2018)DOI
  35. [35]Ozguroglu, E., Liu, R., Surís, D., Chen, D., Dave, A., Tokmakov, P., Vondrick, C.: pix2gestalt: Amodal segmentation by synthesizing wholes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)arxiv.org/abs/2401.14398
  36. [36]Qi, L., Jiang, L., Liu, S., Shen, X., Jia, J.: Amodal instance segmentation with kins dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3014–3023 (2019)DOI
  37. [37]Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rä́dle, R., Rolland, C., Gustafson, L., et al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)arxiv.org/abs/2408.00714
  38. [38]Ravishankar, R., Patel, Z., Rajasegaran, J., Malik, J.: Scaling properties of diffusion models for perceptual tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)arxiv.org/abs/2411.08034
  39. [39]Reddy, N.D., Tamburo, R., Narasimhan, S.G.: Walt: Watch and learn 2d amodal representation from time-lapse imagery. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9356–9366 (2022)DOI
  40. [40]Shamsian, A., Kleinfeld, O., Globerson, A., Chechik, G.: Learning object permanence from video. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16. pp. 35–50. Springer (2020)arxiv.org/abs/2003.10469
  41. [41]Shuai, B., Bergamo, A., Buechler, U., Berneshawi, A., Boden, A., Tighe, J.: Large scale real-world multi person tracking. In: European Conference on Computer Vision. Springer (2022)arxiv.org/abs/2211.02175
  42. [42]Tai, W., Shih, Y.L., Sun, C., Wang, Y.C.F., Chen, H.T.: SAMEO: Segment anything, even occluded. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)
  43. [43]Tangemann, M., Schneider, S., Von Kügelgen, J., Locatello, F., Gehler, P., Brox, T., Kümmerer, M., Bethge, M., Schölkopf, B.: Unsupervised object learning via common fate. arXiv preprint arXiv:2110.06562 (2021)arxiv.org/abs/2110.06562
  44. [44]Tran, M., Pham, T., Bounsavy, W., Nguyen, T., Le, N.: A2VIS: Amodal-aware approach to video instance segmentation. Image and Vision Computing (2025)
  45. [45]Van Hoorick, B., Tokmakov, P., Stent, S., Li, J., Vondrick, C.: Tracking through containers and occluders in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)arxiv.org/abs/2305.03052
  46. [46]Walawalkar, D., Shen, Z., Liu, Z., Savvides, M.: Attentive cutmix: An enhanced data augmentation approach for deep learning based image classification. arXiv preprint arXiv:2003.13048 (2020)
  47. [47]Wechsler, I., Wolf, A., Shanbhag, J., Leyendecker, S., Eskofier, B.M., Koelewijn, A.D., Wartzack, S., Miehling, J.: Bridging the sim2real gap. investigating deviations between experimental motion measurements and musculoskeletal simulation results—a systematic review. Frontiers in Bioengineering and Biotechnology 12, 1386874 (2024)
  48. [48]Wu, T., Zheng, C., Guan, F., Vedaldi, A., Cham, T.J.: Amodal3r: Amodal 3d reconstruction from occluded 2d images. arXiv preprint arXiv:2503.13439 (2025)arxiv.org/abs/2503.13439
  49. [49]Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., Zhao, H.: Depth anything v2. Advances in Neural Information Processing Systems 37, 21875–21911 (2024)arxiv.org/abs/2406.09414
  50. [50]Yao, J., Hong, Y., Wang, C., Xiao, T., He, T., Locatello, F., Wipf, D.P., Fu, Y., Zhang, Z.: Self-supervised amodal video object segmentation. Advances in Neural Information Processing Systems 35, 6278–6291 (2022)arxiv.org/abs/2210.12733
  51. [51]Zhan, G., Zheng, C., Xie, W., Zisserman, A.: Amodal ground truth and completion in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 28003–28013 (2024)arxiv.org/abs/2312.17247
  52. [52]Zhan, X., Pan, X., Dai, B., Liu, Z., Lin, D., Loy, C.C.: Self-supervised scene de-occlusion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3784–3792 (2020)arxiv.org/abs/2004.02788
  53. [53]Zhang, S., Liu, X., Wang, X., Wu, Y., Wang, L., Wang, Y.: Grounding dino: marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2 (2023)arxiv.org/abs/2303.05499
  54. [54]Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., Wang, X.: Bytetrack: Multi-object tracking by associating every detection box. In: European conference on computer vision. pp. 1–21. Springer (2022)arxiv.org/abs/2110.06864
  55. [55]Zhang, Y., Wang, C., Wang, X., Zeng, W., Liu, W.: Fairmot: On the fairness of detection and re-identification in multiple object tracking. International journal of computer vision 129, 3069–3087 (2021)DOI
  56. [56]Zhao, S., Liu, Z., Lin, J., Zhu, J.Y., Han, S.: Differentiable augmentation for data-efficient gan training. Advances in neural information processing systems 33, 7559–7570 (2020)arxiv.org/abs/2006.10738
  57. [57]Zheng, L., Bie, Z., Sun, Y., Wang, J., Su, C., Wang, S., Tian, Q.: Mars: A video benchmark for large-scale person re-identification. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14. pp. 868–884. Springer (2016)DOI
  58. [58]Zhu, H., Tang, P., Park, J., Park, S., Yuille, A.: Robustness of object recognition under extreme occlusion in humans and computational models. arXiv preprint arXiv:1905.04598 (2019)arxiv.org/abs/1905.04598
  59. [59]Zhu, Y., Tian, Y., Metaxas, D., Dollár, P.: Semantic amodal segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1464–1472 (2017)arxiv.org/abs/1509.01329

Paper details

vlnc:W_ku7fb5p3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents