WorksComputer Science
TAP-Out: Tracking Any Point in 360° via Outpainting
Abstract
Tracking points that leave the camera's field of view (FoV) requires reasoning about unobserved scene dynamics, a fundamentally different problem from tracking visible correspondences. We present TAP-Out, an offline method that makes this out-of-view (OOV) reasoning explicit by leveraging generative video outpainting as a world-model prior. Given a narrow-FoV input video, TAP-Out outpaints a complete 360° equirectangular video, locates target objects within the panoramic scene via video object segmentation, and synthesises object-centred perspective crops on which standard in-frame trackers operate. This decoupled design composes off-the-shelf foundation models without requiring task-specific architectural modifications. On the TAPVid360 benchmark, TAP-Out achieves state-of-the-art performance, improving OOV tracking accuracy by approximately 3x over the best prior method. We demonstrate its flexibility on out-of-frame object segmentation and detection and we provide detailed diagnostic analyses across motion regimes, static versus dynamic content, and hallucination quality, characterising when and why the approach succeeds or fails. Code is publicly available at https://github.com/finlay-hudson/TAP-Out
Introduction
Humans and animals possess a remarkable ability to track dynamic objects even after those objects leave their field of view. This capability has been extensively studied in cognitive psychology under the framework of spatial updating [15, 23]. When moving through dynamic scenes, observers continuously update the egocentric locations of objects relative to their body, even when those objects are no longer visible [15, 23]. Crucially, these maintained representations take the form of spatial images, externalised, three-dimensional representations in working memory [29, 30] that persist after sensory input ceases and can represent locations in any direction around the observer [15]. We note that spatial updating is a motivating analogy for our computational approach, not a claim of biological equivalence. The cognitive mechanisms underlying this ability remain debated: some evidence favours egocentric reference frames updated via proprioceptive feedback [22, 44], while other work points to allocentric (world-centred) representations [7, 32], 3D scene graphs, or hybrid approaches. Our method draws conceptual inspiration from the idea of maintaining a complete spatial representation, regardless of the precise cognitive implementation.
In computational vision, the Track Any Point (TAP) task densely tracks point correspondences over long ranges and through occlusions, with downstream applications in robotics [6, 41], action recognition [26], video editing [35, 46], and medicine [1, 38]. Current 2D formulations [10, 21, 50] cannot, by definition, represent or track points that leave the camera frustum, yet this is the default behaviour for a tracker operating in a 3D world. A logical extension is 3D point tracks [25, 49], but obtaining dense long-term 3D ground truth for dynamic OOV objects remains very challenging. Recently, the TAPVid-360 benchmark [18] was introduced, which requires tracking the egocentric 3D direction to a point even when it leaves the FoV, leveraging video data to create ground truth OOV tracks. The accompanying baseline, CoTracker360 [18], modifies CoTracker3 [20] to regress unit-sphere direction vectors. However, despite this specialisation, it performs poorly on OOV points: the model must simultaneously solve high-frequency local correspondence (for in-view points) and high-level world-model inference (for OOV points), capabilities that are very different and difficult to learn jointly from a tracking loss alone.
In this work, we advocate for decoupling these two problems. We propose TAP-Out, a modular pipeline that uses a standard off-the-shelf 2D tracker across two operational states. When objects are in view, the tracker consumes high-fidelity original frames. When objects move out of view, we reframe tracking as a scene completion problem: a powerful pre-trained generative video model acts as a world-model prior, outpainting the full scene so that the target is always visible to the tracker. This design offers three practical advantages. First, it requires no custom 360-specific architectures, relying instead on modular composition of existing models. Second, it inherits progress from generative modelling: as video generation improves, our OOV tracking improves without retraining. Third, despite this simplicity, it substantially outperforms the specialised CoTracker360.
A key insight is that all OOV tracking methods must extrapolate beyond observed data, a form of implicit hallucination. CoTracker360, for instance, must predict plausible directions for points it has never observed, but does so without any explicit visual justification. TAP-Out makes this extrapolation explicit: the generative model produces a complete visual scene consistent with the observed video, providing a human-interpretable explanation of why each point is believed to lie in a particular direction. Even if the generated content is imperfect, this explicit visual hallucination outperforms implicit direction prediction, as our results demonstrate.
We note that TAP-Out is an offline method, processing the complete video before producing tracks. This is suited to applications where track quality supersedes speed, such as visual effects (VFX), video editing, forensic video analysis, and sports analytics. Runtime is approximately 470 seconds per video compared to 0.15 seconds for CoTracker360; we view speed improvements via distillation as promising future work.
Our contributions are as follows:
A new paradigm for point tracking that formally decouples in-view correspondence from OOV generation.
TAP-Out, a modular system that composes off-the-shelf foundation models with a pre-trained generative video world-model prior, providing the first empirical evidence that generative video models can serve as effective priors for OOV tracking.
State-of-the-art results on TAPVid-360, achieving an approximate threefold improvement on dynamic OOV points compared to existing methods, alongside comprehensive diagnostic analyses across motion regimes, static versus dynamic content, and hallucination quality.
Related Work
Amodal Completion. Amodal completion estimates the full shape or appearance of objects including unseen parts. Image-based methods use spatial context from a single frame to complete occluded regions [34, 48], while temporal methods leverage cues across sequential frames [8, 19, 40]. The most challenging form is out-of-view completion, where an object has entirely left the sensor’s FoV, directly related to human spatial updating [12, 24]. CoTracker360 [18] addresses this with promising but limited results, as joint learning of camera motion and spatial understanding constrains generalisation.
Video Generation and World Models. Large-scale generative video models such as Sora [33] and VEO [16] have demonstrated remarkable zero-shot capabilities [45], suggesting the acquisition of implicit world models that capture object dynamics and physical plausibility. These capabilities are being applied to tasks including amodal completion [8, 19], robotic manipulation [36], self-supervised learning [5], and autonomous driving [4]. The Argus model [31] is particularly relevant: it trains a video diffusion model to produce temporally consistent equirectangular video from perspective input, explicitly estimating scene dynamics beyond the camera’s FoV. Our work provides the first empirical demonstration that such generative models can serve as effective world-model priors for OOV point tracking, a capability not previously explored in the 2D, 3D, or tracking literature.
Camera Estimation. Accurate camera parameters are essential for mapping perspective frames onto the spherical plane. Recent feed-forward methods such as VGGT [43], MASt3R [27], and MegaSaM [28] directly regress 3D scene attributes with strong performance. We evaluate our pipeline with both ground-truth and estimated cameras to characterise the impact of estimation errors on downstream tracking.
Track Any Point and 3D Tracking. The TAP task [17] extends classical optical flow [2, 3, 13, 39] to long-term, occlusion-robust tracking, with benchmarks [9] and models including TAPIR [10], CoTracker [20, 21], and TAPNext [50]. These 2D methods cannot represent OOV points by construction.
3D tracking methods [25, 42, 47, 49] estimate trajectories in 3D space but typically focus on points within the visible frustum. Extending them to OOV is fundamentally ill-posed: estimating the 3D position (including depth) of an unobserved point requires strong scene priors that current 3D trackers lack. The depth component is especially ambiguous outside the frustum, where no visual evidence constrains the estimate. In contrast, the 360° formulation [18] requires only direction (on ), sidestepping the depth ambiguity. This is well-matched to generative video models that operate in pixel space and can leverage massive 2D video pretraining to answer “where is the object?” (direction) without solving “how far away is it?” (depth). The TAPVid-360-10k benchmark [18] provides large-scale evaluation data by sampling perspective camera trajectories from 360° video and modifying CoTracker3 [20] to regress unit-sphere direction vectors as the CoTracker360 baseline.
Method
Given an input perspective video consisting of RGB frames and a set of query points specified in the first frame, our goal is to track these points throughout the video, even when they move outside the camera’s FoV. Unlike traditional TAP methods that predict 2D pixel coordinates , we adopt the TAPVid-360 formulation [18] and predict 3D direction vectors with in the camera coordinate system for each query point at each frame.
Our method transforms this challenging OOV tracking problem into a standard in-frame tracking task through a series of geometric transformations and generative modelling. Figure Figure 1 provides an overview of the pipeline.

Figure 1. TAP-Out pipeline overview. From a narrow field-of-view input video, we estimate camera poses and reproject frames onto an equirectangular canvas. A generative video outpainting model completes the full scene, including unobserved regions. Given query points, we locate the target object in the panoramic video via video object segmentation, then synthesise object-centred perspective crops. Standard in-frame point tracking on these crops yields accurate point tracks, even for points that have left the original field of view. This decoupled design enables any component to be independently upgraded as the field advances.
Coordinate Systems and Transformations
We work with two primary coordinate representations: perspective images and equirectangular panoramas. We define a shared world coordinate frame aligned with the first camera frame, i.e. , where is the identity matrix. Placing all frames in a single consistent coordinate frame yields a consistent scene arrangement that simplifies the outpainting model’s task. Each frame is associated with a camera rotation relative to the first frame and intrinsic parameters . Since we reason only about direction and not 3D position, translation vectors are discarded.
Perspective to Equirectangular Projection. To map a pixel coordinate from the perspective image to the equirectangular representation, we first unproject it to a 3D ray in camera coordinates via normalised device coordinates:
Given horizontal and vertical FoVs and , the 3D direction vector in camera coordinates is:
Transforming to the shared world frame using the inverse camera rotation:
We then convert to spherical coordinates :
and map to equirectangular pixel coordinates in an image of dimensions :
TAP-Out Pipeline
Input Preprocessing. Given the input video , we first estimate or obtain the camera parameters relative to the first frame, where . When ground-truth parameters are unavailable, we employ VGGT [43]. Using the projection described above, we map each perspective frame onto a shared equirectangular canvas, producing a masked equirectangular video where . Regions outside the original FoV are masked. The initial query points are similarly transformed:
where denotes the perspective-to-equirectangular projection operator.
360° Scene Completion. We leverage Argus [31], a diffusion-based video-to-360 model, to complete the masked equirectangular video:
This model predicts both spatial layout and object dynamics in unobserved regions, providing a temporally consistent panoramic representation. To preserve high-frequency input details, we create a composite video:
where denotes a frame from , is the binary mask indicating observed regions, and denotes element-wise multiplication.
Object Localisation in Equirectangular Space. Given the complete equirectangular video and the initial query , we employ SAM2 [37] to track the query object:
where is a binary segmentation mask. Although equirectangular videos are out-of-domain for SAM2, it generalises sufficiently to provide coarse masks for approximate object localisation. From each mask, we compute the object centroid:
Object-Centred Perspective Generation. For each frame, we generate a new perspective view centred on the tracked object. We convert the centroid to a 3D direction:
and define a virtual camera with rotation aligning its optical axis with and specified FoV :
This produces object-centred perspective frames where the query object is consistently visible and centred.
360° Point Tracking. We apply a standard tracker such as CoTracker3 [20] to obtain refined tracks:
where is the query point in centred-view coordinates. The tracked points are then transformed back to the original camera coordinate system: (1) convert to equirectangular coordinates , (2) convert to a 3D direction in the shared world frame , and (3) transform to the original camera frame .
The pipeline is modular by design: any component (camera estimation, generative outpainting, segmentation, or point tracker) can be independently upgraded as the field advances. This “future-proofing” means that improvements in video generation directly translate to better OOV tracking without requiring retraining or architectural changes.
Extension to Other Tasks
Once we have the object-centred video and corresponding query points, we can solve several related tracking tasks. For video object segmentation, the coarse equirectangular masks can be refined by re-running SAM2 on the object-centred narrow-FoV frames. For object detection, bounding boxes can be computed in the centred view and transformed back. In both cases, outputs are reprojected to the original camera coordinates via:
where denotes perspective division.
Evaluation
Experimental Setup
Data. We evaluate TAP-Out on the TAPVid360-10k benchmark [18], which provides perspective frames with ground-truth OOV information including world-to-camera rotations and focal lengths. Since Argus [31] was trained on data partially overlapping with TAPVid360-10k, we isolated a non-overlapping subset of 8,887 videos, termed TAPVid360-TAPOut.
Point Categorisation
Point Categorisation. To isolate the contribution of the generative world model from simple geometric reprojection, we categorise points based on their intrinsic motion. Since the TAPVid360-10k benchmark provides ground-truth camera rotations, tracking static points is trivial: such points can be precisely located by reprojecting their first-frame positions into subsequent views using only the known camera motion. To move beyond this baseline and evaluate the generative model’s capacity to predict actual object dynamics, we define dynamic points as those with cumulative world-frame motion exceeding over the video sequence. This threshold ensures that points exhibit significant intrinsic motion beyond typical camera estimation jitter or sensor noise, specifically capturing objects moving relative to the background. Points below this threshold are considered static or near-static.
Metrics. We adopt the TAPVid360-10k metrics [18]. : the fraction of predicted points within a threshold angular distance of the ground truth (higher is better). : the mean angular distance in degrees between predicted and ground-truth points (lower is better). Both metrics are computed separately for points that are in-frame (if), out-of-frame (oof), and for all points combined (all).
Implementation Details. TAP-Out uses four off-the-shelf foundation models: VGGT [43] for camera estimation (when ground truth is unavailable), Argus [31] for 360° outpainting, SAM2 [37] for object segmentation, and CoTracker3 [20] (offline mode) for point tracking. All experiments use a single NVIDIA A100 GPU. Total inference time is approximately 470 seconds per video for TAP-Out, compared to 0.15 seconds for CoTracker360. This 3,000× latency difference reflects our design choice: TAP-Out is an offline, high-precision tool for applications such as VFX, video editing, and forensic analysis, where track quality takes priority over speed.
Main Results
Table 1 presents the main results. TAP-Out with ground-truth (GT) cameras achieves state-of-the-art performance across all metrics. The most pronounced advantage is on OOV points: , approximately 3.1× higher than CoTracker360’s 0.106. This validates that explicit scene completion produces more accurate OOV tracks than implicit direction prediction.
| Method | ||||||
| TAPNext [50] | 0.0075 ±0.0077 | 0.0178 ±0.0180 | 0.0004 ±0.0010 | 52.6330 ±18.4256 | 37.2676 ±19.6733 | 62.6947 ±19.8463 |
| TAPIR [10] | 0.0096 ±0.0096 | 0.0229 ±0.0225 | 0.0003 ±0.0006 | 50.7070 ±18.4413 | 35.0867 ±21.8100 | 60.9705 ±17.8270 |
| BootsTAPIR [11] | 0.0106 ±0.0106 | 0.0254 ±0.0251 | 0.0005 ±0.0011 | 49.1382 ±18.5384 | 34.4510 ±21.4859 | 58.6180 ±18.2366 |
| TAPIP3D [49] | 0.2514 ±0.1606 | 0.4773 ±0.2429 | 0.0897 ±0.1301 | 35.9088 ±26.7414 | 22.9776 ±23.0838 | 44.8629 ±32.1248 |
| CoTracker3 (offline) [21] | 0.2375 ±0.0899 | 0.5516 ±0.1588 | 0.0146 ±0.0208 | 38.2831 ±21.9041 | 18.4400 ±22.2117 | 51.6813 ±24.4290 |
| CoTracker360 [18] | 0.2259 ±0.1059 | 0.3925 ±0.1326 | 0.1062 ±0.0984 | 8.6054 ±7.6706 | 4.1213 ±4.8057 | 11.3619 ±9.8862 |
| Static Reprojection (Gt Camera) | 0.2497 ±0.1989 | 0.3192 ±0.2177 | 0.2032 ±0.2030 | 8.5689 ±10.9995 | 5.8791 ±7.4396 | 10.2048 ±13.5424 |
| TAP-Out (GT Camera) | 0.4418 ±0.1769 | 0.6038 ±0.2103 | 0.3318 ±0.1794 | 4.0788 ±7.2187 | 2.5947 ±6.8070 | 5.0180 ±7.9054 |
| TAP-Out (VGGT Est Camera) | 0.2960 ±0.1499 | 0.5296 ±0.2126 | 0.1354 ±0.1314 | 17.4218 ±17.3905 | 8.0179 ±11.9628 | 23.2742 ±23.5585 |
Table 1. Tracking performance on TAPVid360-TAPOut. Subscripts denote in-frame (if), out-of-frame (oof), and all points (all).
While these results are strong, they represent the performance across the entire benchmark. As established in Section 4, tracking static points is trivial given GT camera rotations.
Therefore, the more rigorous test of our generative world model lies in its performance on dynamic objects, where geometric reprojection provides no solution.
We also report results with estimated cameras (VGGT). With estimated cameras, TAP-Out achieves , exceeding CoTracker360 while substantially outperforming it on in-frame and overall metrics. The higher indicates that while camera estimation is frequently accurate, failures can be catastrophic, skewing mean error. This highlights camera estimation as a key bottleneck for deployment.
Static Reprojection Baseline
A natural question is whether generative outpainting is necessary at all: if we have camera poses, can we simply reproject points geometrically? To answer this, we implemented a static reprojection baseline that takes the first visible position of each query point, assumes the point is static in the world, and reprojects it into subsequent frames using ground truth camera rotations.
As shown in Table 1, the static baseline achieves an out-of-view accuracy of , which is a significant improvement over 2D trackers but still lags far behind TAP-Out’s 0.332. While reprojection is effective for stationary background points, it is fundamentally unable to account for the dynamic object motion that characterizes the majority of the TAPVid360 benchmark. This confirms that while camera pose estimation provides a necessary geometric foundation, a generative world model is essential for reasoning about object dynamics and permanence during out-of-view tracking.
Motion Regime Analysis
To understand how performance depends on camera motion, we partition the data into three regimes based on total camera rotation (Table 2). TAP-Out achieves at low motion, where the scene remains similar to the observed input. Performance remains strong at 0.45 for medium motion (the dominant regime at 59.6% of points). At high motion (), accuracy drops to 0.21 as the generative model must hallucinate increasingly large unobserved regions. Future improvements in video generation fidelity should directly benefit this regime.
| Regime | Low (0°–5°) | Medium (5°–30°) | High (30°+) |
| % Points in Dataset | 0.8% | 59.6% | 39.6% |
| TAP-Out | 0.90 | 0.45 | 0.21 |
Table 2. TAP-Out OOV tracking accuracy () across camera motion magnitudes, measured as total rotation over the video clip.
Hallucination Analysis
Since TAP-Out depends on the quality of the generated panorama, we examine when outpainting produces useful content. Figure 2 compares the outpainted panorama to ground truth over time. Argus maintains global scene structure (room layout, horizon) with good fidelity, providing spatial anchoring for segmentation. For typical occlusion durations, it captures approximate object trajectories sufficient for coarse localisation. The primary failure mode is identity drift over long occlusions, where generated content gradually deviates from the true scene. Even imperfect hallucination helps: CoTracker360 must extrapolate

Figure 2. Hallucination quality over time. Each row shows the original perspective view, the ground-truth equirectangular panorama, and the predicted (outpainted) equirectangular panorama at different timesteps.
OOV directions without visual explanation, while TAP-Out’s explicit scene generation constrains the problem more effectively. The substantial improvement in demonstrates that approximate scene completion substantially outperforms implicit extrapolation.
Qualitative Results
Figure 3 and Figure 4 present qualitative results on TAPVid360. Standard 2D trackers (TAPIP3D) lose track when points leave the FoV. CoTracker360 produces noisy OOV estimates that often diverge from ground truth. TAP-Out maintains coherent tracks through extended OOV periods by operating on the outpainted panoramic scene.

Figure 3. Qualitative tracking results on TAPVid360. Each column shows tracked point positions on the equirectangular view across timesteps.

Figure 4. Further qualitative comparison on TAPVid360 with significant camera motion.
Additional Tasks: Out-of-Frame Object Tracking
We also demonstrate TAP-Out on the LaSOT benchmark [14], which contains clips where objects leave and re-enter the scene, creating a challenging re-identification (Re-ID) problem. Figure 5 highlights a critical case: a white ball in a snooker clip vanishes, the camera moves, and when the ball reappears, SAM2 fails to re-identify it. TAP-Out circumvents Re-ID entirely by continuously tracking in the equirectangular domain. Further qualitative LaSOT results are provided in Figure 6 and showcase the generalised nature of the TAP-Out pipeline.

Figure 5. Eliminating re-identification failures on LaSOT [14]. A standard tracker (SAM2, left) tracks a white ball when visible (blue circles) but fails to re-identify it upon return (red circle) after extended absence. TAP-Out (right) maintains continuous localisation in the equirectangular domain.

Figure 6. TAP-Out applied to out-of-frame object tracking on the LaSOT benchmark [14].
Discussion
Why Direction Over Explicit 3D. A natural alternative would be to extend 3D tracking beyond the visible frustum. However, this is fundamentally ill-posed: depth is ambiguous outside the frustum where no visual evidence constrains the estimate. Methods such as SpatialTracker [47] require observed depth and cannot extrapolate beyond it. The formulation sidesteps this by predicting only direction (), which is naturally aligned with generative video models that operate in pixel space.
Explicit vs. Implicit Extrapolation. All OOV tracking methods must extrapolate beyond observed data. CoTracker360 [18] directly predicts direction vectors, implicit hallucination without visual explanation. TAP-Out makes extrapolation explicit: the generative model produces a complete visual scene, providing a human-interpretable explanation of why each point is tracked to a particular direction. This is useful for debugging and for applications requiring transparency, and constrains the tracking problem more effectively than direct coordinate regression.
Failure Modes. TAP-Out degrades in identifiable regimes: (1) high camera motion (), where the generative model must hallucinate large unobserved regions; (2) long occlusions, where identity drift causes segmentation failures; (3) camera estimation errors, which propagate through the pipeline; and (4) out-of-distribution inputs, as Argus expects world-up-aligned views.
Conclusions
We have presented TAP-Out, a modular pipeline that enables continuous tracking of points and objects regardless of whether they are within the camera’s field of view. By decoupling in-view correspondence from OOV scene completion, and leveraging a generative video model as an explicit world-model prior, TAP-Out achieves state-of-the-art performance on the TAPVid360 benchmark with approximately improvement on OOV points over the best prior method.
Limitations. TAP-Out is restricted to offline use (approximately 470s vs. 0.15s for CoTracker360). Camera estimation errors propagate through the pipeline (Table 1). Argus expects world-up-aligned views, and arbitrary orientations degrade outpainting. Prolonged occlusions cause identity drift. Static points could be handled more efficiently via geometric reprojection alone.
Future Work. Promising directions include distillation into a faster feed-forward model, sampling multiple outpainting hypotheses to model uncertainty, and developing online variants. The modular design ensures that improvements in any component directly benefit the overall system.
Supplementary Material
Justification of the TAPVid360-TAPOut Split
To evaluate our method on the TAPVid360 benchmark, we utilised a subset of 8,887 clips from the original 10,000 to avoid data contamination from Argus training data. As detailed in Table A1, the results obtained on our subset (main values) exhibit negligible deviation from those of the full dataset (values in parentheses). Note that this was run on all points, not just dynamic ones. The performance metrics remain statistically consistent across all evaluated methods, and crucially, the relative ranking of the trackers is unchanged. This strongly suggests that the TAPVid360-TAPOut split preserves the data distribution and difficulty of the original benchmark, serving as a valid and representative proxy for evaluation.
| Method | ||||||
| TAPNext [50] | 0.0082 (0.0082) | 0.0190 (0.0191) | 0.0004 (0.0004) | 51.98 (51.98) | 36.58 (36.59) | 62.45 (62.46) |
| TAPIR [10] | 0.0106 (0.0106) | 0.0250 (0.0251) | 0.0003 (0.0003) | 49.83 (49.81) | 33.87 (33.88) | 60.53 (60.52) |
| BootsTAPIR [11] | 0.0125 (0.0126) | 0.0291 (0.0293) | 0.0005 (0.0005) | 48.37 (48.36) | 33.30 (33.32) | 58.38 (58.38) |
| TAPIP3D [49] | 0.2449 (0.2476) | 0.4685 (0.4698) | 0.0817 (0.0850) | 36.44 (36.44) | 23.08 (23.32) | 45.90 (45.80) |
| SpatialTracker [47] | 0.2235 (0.2239) | 0.4889 (0.4893) | 0.0300 (0.0303) | 38.90 (38.88) | 22.16 (22.16) | 50.45 (50.44) |
| CoTracker3 (offline) [21] | 0.2444 (0.2435) | 0.5601 (0.5588) | 0.0158 (0.0158) | 37.40 (37.43) | 17.60 (17.64) | 50.94 (50.98) |
| CoTracker360 [18] | 0.2389 (0.2386) | 0.4064 (0.4060) | 0.1164 (0.1160) | 8.24 (8.27) | 3.92 (3.95) | 10.93 (10.98) |
Table A1. Comparison of tracking performance on the TAPVid360-TAPOut split versus the original full TAPVid360-10k dataset (shown in parentheses).
Rationale for Dynamic Point Threshold
To categorize tracking challenges in the TAPVid360 benchmark, we define dynamic points as those exhibiting an independent world-frame motion magnitude of at least . This threshold is primarily motivated by the error floor of a non-dynamic tracking assumption. As shown in Table 1 of the main manuscript, the Static Reprojection (Gt Camera) baseline, which accounts for generated camera rotation but assumes objects are stationary in the world, results in a mean angular error of . Consequently, point motion below is frequently indistinguishable from the baseline noise of a static-world tracker.
2D Tracker Modularity
To validate that our framework is modular and not coupled to a specific tracking backbone, we keep the upstream camera estimation (VGGT) and outpainting modules (Argus) frozen, and swap our default CoTracker3 backbone for BootsTAPIR without any retraining or fine-tuning. As shown in Table A2, whilst performance drops slightly when moving away from CoTracker3 (OOV accuracy decreases from 0.339 to 0.298), our pipeline still maintains solid out-of-frame tracking capability, confirming its plug-and-play modularity.
| Generative Prior | 2D Tracker Backbone | (OOV Acc.) | (Mean Error) |
| Argus | CoTracker3 (Ours, Default) | 0.339 | 5.08° |
| Argus | BootsTAPIR | 0.298 | 5.85° |
Table A2. Table A2: Plug-and-Play 2D Tracker Ablation on a random sample of 500 videos.
Additional Qualitative Results
Figure A1 and Figure A2 provide additional qualitative comparisons on TAPVid360 across different scene types and motion patterns. Figure A3 shows an extended visualisation of the TAP-Out pipeline output, illustrating how the method maintains tracking consistency between the perspective and equirectangular domains.

Figure A1. Additional qualitative comparison on TAPVid360 showing tracking results across methods.

Figure A2. Further qualitative comparison on TAPVid360 showing another example sequence.

Figure A3. Extended visualisation of TAP-Out results.
References
- [1]Md Abulkalam Azad, John Nyberg, Havard Dalen, Bjørnar Grenne, Lasse Lovstakken, and Andreas Østvik. Taming modern point tracking for speckle tracking echocardiography via impartial motion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1104–1113, 2025.arxiv.org/abs/2507.10127
- [2]Simon Baker and Iain Matthews. Lucas-kanade 20 years on: A unifying framework. International journal of computer vision, 56:221–255, 2004.
- [3]J. L. Barron, D. J. Fleet, and S. S. Beauchemin. Performance of optical flow techniques. International Journal of Computer Vision, 12(1):43–77, Feb 1994. ISSN 1573-1405. doi: 10.1007/BF01420984. URL https://doi.org/10.1007/BF01420984.
- [4]Florent Bartoccioni, Elias Ramzi, Victor Besnier, Shashanka Venkataramanan, Tuan-Hung Vu, Yihong Xu, Loick Chambon, Spyros Gidaris, Serkan Odabas, David Hurych, et al. Vavim and vavam: Autonomous driving through video generative modeling. arXiv preprint arXiv:2502.15672, 2025.arxiv.org/abs/2502.15672
- [5]Varun Belagali, Srikar Yellapragada, Alexandros Graikos, Saarthak Kapse, Zilinghan Li, Tarak Nath Nandi, Ravi K Madduri, Prateek Prasanna, Joel Saltz, and Dimitris Samaras. Gen-sis: Generative self-augmentation improves self-supervised learning. arXiv preprint arXiv:2412.01672, 2024.arxiv.org/abs/2412.01672
- [6]Homanga Bharadwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tuliani. Track2Act: Predicting point tracks from internet videos enables generalizable robot manipulation. In European Conference on Computer Vision, pages 306–324. Springer, 2024.
- [7]Neil Burgess. Spatial memory: how egocentric and allocentric combine. Trends in Cognitive Sciences, 10(12):551–557, 2006.pubmed.ncbi.nlm.nih.gov/17071127
- [8]Kaihua Chen, Deva Ramanan, and Tarasha Khurana. Using diffusion priors for video amodal segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22890–22900, 2025.arxiv.org/abs/2412.04623
- [9]Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-vid: A benchmark for tracking any point in a video. Advances in Neural Information Processing Systems, 35:13610–13626, 2022.arxiv.org/abs/2211.03726
- [10]Carl Doersch, Yi Yang, Mel Vecerik, and Dilara Gokay. TAPIR: Tracking any point with per-frame initialization and temporal refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10061–10072, 2023.arxiv.org/abs/2306.08637
- [11]Carl Doersch, Pauline Luc, Yi Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ignacio Rocco, Ross Goroshin, João Carreira, and Andrew Zisserman. BootsTAP: Bootstrapped training for tracking-any-point. Asian Conference on Computer Vision, 2024.arxiv.org/abs/2402.00847
- [12]Milos Dordevic, Sonja Hoelzer, Augusta Russo, José C García Alanis, and Notger G Müller. The role of the precuneus in human spatial updating in a real environment setting—a ctbs study. Life, 12(8):1239, 2022.DOI
- [13]Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758–2766, 2015.arxiv.org/abs/1504.06852
- [14]Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5374–5383, 2019.
- [15]Nicholas A Giudice, Michael R Betty, and Jack M Loomis. Functional equivalence of spatial images from touch and vision: Evidence from spatial updating in blind and sighted individuals. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(3):621–634, 2011.
- [16]Google DeepMind. Veo: The next generation of generative video models. https://deepmind.google/models/veo/, 2024. Accessed: 14 November 2025.
- [17]Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vision, pages 59–75. Springer, 2022.arxiv.org/abs/2204.04153
- [18]Finlay Hudson, James A D Gardner, and William A P Smith. TAPVid-360: Tracking any point in 360 from narrow field of view video. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL https://openreview.net/forum?id=S3LwZ0k6ia.arxiv.org/abs/2511.21946
- [19]Finlay GC Hudson and William AP Smith. Track anything behind everything: Zero-shot amodal video object segmentation. arXiv preprint arXiv:2411.19210, 2024.arxiv.org/abs/2411.19210
- [20]Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos, 2024. URL https://arxiv.org/abs/2410.11831.
- [21]Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In Proc. ECCV, 2024.arxiv.org/abs/2307.07635
- [22]Roberta L Klatzky. Allocentric and egocentric spatial representations: Definitions, distinctions, and interconnections. In Spatial Cognition: An Interdisciplinary Approach to Representing and Processing Spatial Knowledge, pages 1–17. Springer, 1998.DOI
- [23]Roberta L Klatzky, Yvonne Lippa, Jack M Loomis, and Reginald G Golledge. Encoding, learning, and spatial updating of multiple object locations specified by 3-D sound, spatial language, and vision. Experimental Brain Research, 149:48–61, 2003.pubmed.ncbi.nlm.nih.gov/12592503
- [24]Eliana M Klier and Dora E Angelaki. Spatial updating and the maintenance of visual constancy. Neuroscience, 156(4):801–818, 2008.DOI
- [25]Skanda Koppula, Ignacio Rocco, Yi Yang, Joe Heyward, João Carreira, Andrew Zisserman, Gabriel Brostow, and Carl Doersch. Tapvid-3d: A benchmark for tracking any point in 3d. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 82149–82165. Curran Associates, Inc., 2024.arxiv.org/abs/2407.05921
- [26]Pulkit Kumar, Namitha Padmanabhan, Luke Luo, Sai Saketh Rambhatla, and Abhinav Shrivastava. Trajectory-aligned space-time tokens for few-shot action recognition. In European Conference on Computer Vision, pages 474–493. Springer, 2024.arxiv.org/abs/2407.18249
- [27]Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision, pages 71–91. Springer, 2024.arxiv.org/abs/2406.09756
- [28]Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast and robust structure and motion from casual dynamic videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10486–10496, 2025.arxiv.org/abs/2412.04463
- [29]Jack M Loomis, Roberta L Klatzky, Brendan McHugh, and Nicholas A Giudice. Spatial working memory for locations specified by vision and audition: Testing the amodality hypothesis. Attention, Perception, & Psychophysics, 74(6):1260–1267, 2012.DOI
- [30]Jack M Loomis, Roberta L Klatzky, and Nicholas A Giudice. Representing 3D space in working memory: Spatial images from vision, hearing, touch, and language. In Multisensory Imagery, pages 147–174. Springer, 2013.DOI
- [31]Rundong Luo, Matthew Wallingford, Ali Farhadi, Noah Snavely, and Wei-Chiu Ma. Beyond the frame: Generating 360 panoramic videos from perspective videos, 2025. URL https://arxiv.org/abs/2504.07940.
- [32]Weimin Mou, Timothy P McNamara, Christine M Valiquette, and Bryan Rump. Allocentric and egocentric updating of spatial memories. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30(1):142–157, 2004.pubmed.ncbi.nlm.nih.gov/14736303
- [33]OpenAI. Video generation models as world simulators, 2024.
- [34]Ege Ozguroglu, Ruoshi Liu, Dídac Surís, Dian Chen, Achal Dave, Pavel Tokmakov, and Carl Vondrick. pix2gestalt: Amodal segmentation by synthesizing wholes. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3931–3940. IEEE Computer Society, 2024.arxiv.org/abs/2401.14398
- [35]Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image manifold. In ACM SIGGRAPH 2023 conference proceedings, pages 1–11, 2023.DOI
- [36]Shivansh Patel, Shraddhaa Mohan, Hanlin Mai, Unnat Jain, Svetlana Lazebnik, and Yunzhu Li. Robotic manipulation by imitating generated videos without physical demonstrations. arXiv preprint arXiv:2507.00990, 2025.arxiv.org/abs/2507.00990
- [37]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024.arxiv.org/abs/2408.00714
- [38]Adam Schmidt, Omid Mohareri, Simon P DiMaio, and Septimiu E Salcudean. Surgical tattoos in infrared: A dataset for quantifying tissue tracking and mapping. IEEE Transactions on Medical Imaging, 43(7):2634–2645, 2024.DOI
- [39]Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020.arxiv.org/abs/2003.12039
- [40]Basile Van Hoorick, Pavel Tokmakov, Simon Stent, Jie Li, and Carl Vondrick. Tracking through containers and occluders in the wild. 2023.arxiv.org/abs/2305.03052
- [41]Mel Vecerik, Carl Doersch, Yi Yang, Todor Davchev, Yusuf Aytar, Guangyao Zhou, Raia Hadsell, Lourdes Agapito, and Jon Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5397–5403. IEEE, 2024.arxiv.org/abs/2308.15975
- [42]Bo Wang, Jian Li, Yang Yu, Li Liu, Zhenping Sun, and Dewen Hu. Scenetracker: Long-term scene flow estimation network. arXiv preprint arXiv:2403.19924, 2024.DOI
- [43]Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.arxiv.org/abs/2503.11651
- [44]Ranxiao Frances Wang. Building a cognitive map by assembling multiple path integration systems. Psychonomic Bulletin & Review, 23(3):692–702, 2016.DOI
- [45]Thaddäus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, and Robert Geirhos. Video models are zero-shot learners and reasoners. arXiv preprint arXiv:2509.20328, 2025.arxiv.org/abs/2509.20328
- [46]Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. In European Conference on Computer Vision, pages 331–348. Springer, 2024.arxiv.org/abs/2403.07420
- [47]Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20406–20417, 2024.arxiv.org/abs/2404.04319
- [48]Guangqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman. Amodal ground truth and completion in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28003–28013, 2024.arxiv.org/abs/2312.17247
- [49]Bowei Zhang, Lei Ke, Adam W Harley, and Katerina Fragkiadaki. Tapip3d: Tracking any point in persistent 3d geometry. arXiv preprint arXiv:2504.14717, 2025.arxiv.org/abs/2504.14717
- [50]Artem Zholus, Carl Doersch, Yi Yang, Skanda Koppula, Viorica Patraucean, Xu Owen He, Ignacio Rocco, Mehdi S. M. Sajjadi, Sarath Chandar, and Ross Goroshin. Tapnext: Tracking any point (tap) as next token prediction. arXiv preprint arXiv:2504.05579, 2025.arxiv.org/abs/2504.05579