WorksComputer Science
Detection Is Not the Finish Line: Multiscale Decoding and Constraint-Aware Global Association for Multi-Camera 3D Tracking
Abstract
A query-based multi-view 3D detector can transfer well to a new calibrated camera network and still produce badly fragmented tracks: on the AI City Challenge 2026 Track 1 validation set, the frozen Sparse4D detector behind the official challenge winner splits each Person identity into a median of eight fragments, and half of the gaps between fragments exceed the horizon of a causal tracker. We show that most of the remaining headroom of such a system lies after detection. Keeping the public detector frozen, we add (i) class-wise source selection between two input scales, which recovers small and distant people from a 0.75x pass while leaving large objects at the scale the detector was trained for; (ii) a geometry-to-appearance bridge that projects every 3D box into all calibrated views, matches it to 2D person detections, and aggregates robust track-level re-identification descriptors; and (iii) constraint-aware consolidation, which combines a causal short-gap stitch with offline centroid-linkage clustering under two hard constraints, temporal exclusion and metric 3D reachability, plus class-aware long-gap stitching for rigid objects. Every association stage changes object identities only. On the challenge evaluation server the system scores 62.3947 3D HOTA (DetA 57.4501, AssA 56.8131, LocA 80.1272), first on the general leaderboard snapshot of 30 October 2026 (second at the time of writing, after a competitor's detector update) and 5.63 points above our reproduction of the official winner; the constrained association stage alone contributes +3.73 HOTA through a +5.49 AssA gain at unchanged DetA. Controlled fixed-box validation studies, a constraint ablation, sensitivity curves, and a catalogue of negative results support the design.
Introduction
Multi-camera 3D perception asks for a scene-level account of every object: a position in a shared world frame and one identity for as long as the object is present, however many cameras it crosses. Track 1 of the 2026 AI City Challenge [20, 25] poses this problem in warehouses with up to 20 calibrated cameras, seven object classes, synthetic and real scenes, and no depth at inference time. The metric is a 3D adaptation of HOTA [15], whose factors separate detection quality (DetA), association quality (AssA), and localization (LocA).
The official winner of the track, GaugeAlign [27], runs a frozen NVIDIA TAO Sparse4D checkpoint [12] after a closed-form canonicalization of the camera network’s coordinate gauge, and won on detection alone: its authors report a 10-point DetA margin over the runner-up and a 7-point AssA deficit. Our reproduction of that system scores 56.7690 3D HOTA on the evaluation server with 55.7189 DetA but only 49.6755 AssA. Figure 1 quantifies the reason. Matching the detector’s raw tracks to validation ground truth, each Person identity is covered by a median of eight predicted identities, the median gap between consecutive fragments of one person is 274 frames, and 52% of those gaps are longer than the 240-frame horizon of a causal stitcher; 18% exceed 900 frames (30 s). The tracklets themselves are about 95% pure. The detector, in other words, is not the bottleneck; the topology of its tracks is.
Figure 1. Why a causal tracker cannot repair the frozen detector’s tracks. Validation Person predictions (half-resolution production boxes) are matched to ground truth per frame with a 1 m centre gate; a fragment is a predicted identity matched to one ground-truth person on at least ten frames. (a) Gaps between consecutive fragments of the same person: with raw Sparse4D identities (grey) the median gap is 274 frames and 52% of gaps exceed the 240-frame causal horizon (dashed); the causal stitch (orange) removes mostly short gaps, leaving 57% of the remainder beyond the horizon. Constrained global clustering (blue) removes two thirds of all gaps. (b) Fragments per ground-truth identity: median 8 (interquartile range 6–10) raw, 7 after the causal stitch, 3 after global clustering.
This observation motivates a detector-preserving strategy. Rather than train another 3D detector, we ask two questions. First, can a different input scale recover the small, distant people the detector misses, without harming the classes it already handles well? Second, can fragmented world-space tracks be consolidated over arbitrarily long gaps using appearance, while the physical constraints available in a calibrated 3D scene prevent the over-merges that make unconstrained re-identification (ReID) clustering unsafe? Both answers are affirmative, and together they move the server score to 62.3947 3D HOTA, first on the general leaderboard snapshot of 30 October 2026 (Table 2).
| Ranking | System | HOTA | DetA | AssA | LocA |
| official (online) | GaugeAlign (EVA) [27] | 56.5447 | 55.6444 | 49.3929 | 79.5558 |
| Syn2RealTrack (SKKU-AL-T1) [22] | 52.0118 | 45.3056 | 56.5047 | 76.2410 | |
| SparseQID (Playbox) [18] | 38.0105 | 40.2592 | 31.0978 | 75.1778 | |
| general leaderboard (30 Oct 2026) | ParkLabs (ours) | 62.3947 | 57.4501 | 56.8131 | 80.1272 |
| ELMO | 61.8129 | 60.1596 | 54.1241 | 80.8056 | |
| Metropolis | 60.7019 | – | – | – | |
| (3 Nov 2026) | ELMO, updated entry | 62.6375 | 60.1570 | 55.3186 | 80.8063 |
| (6 Nov 2026) | ELMO, updated entry | 65.8422 | 63.8761 | 57.1033 | 81.9793 |
Table 2. Hidden-test results from the official evaluation server (percent). Top: the official Track 1 online ranking at the close of the challenge, as reported by the respective teams [18, 22, 27]. Bottom: the general leaderboard snapshot of 30 October 2026 after our final submission, and the updated ELMO entries of 3 and 6 November 2026. Our entry is offline and post-deadline; it is a server rank, not a claim to the challenge award or the online bonus.
Our contributions are:
Class-wise multiscale source selection for a frozen Sparse4D model (§ 3.2). Person boxes come from a 0.75× pass and every other class from the 0.50× training scale; the output of each class is replaced, never unioned, so no duplicates arise. Full-validation Person HOTA after association rises from 35.115 to 40.108, and the server score by 0.9243.
A geometry-to-appearance bridge (§ 3.3) from world-space 3D boxes to multi-view Person descriptors: projection, one-to-one matching, confidence- and overlap-weighted cross-view fusion, and trimmed track-level prototypes. It supplies descriptors for 98.6% of the Person rows in the final test submission.
Constraint-aware consolidation (§ 3.4–§ 3.6): a causal short-gap stitch, an offline centroid-linkage clustering of whole tracks under temporal-exclusion and 3D-reachability constraints, and class-aware long-gap stitching for six rigid classes. On the server this stage adds 3.7321 HOTA with DetA unchanged to four decimals. An ablation shows that temporal exclusion, which refuses hundreds of above-threshold pairs of look-alike people seen simultaneously, is what makes a fixed moderate threshold safe, while the reachability gate is a cost-free safety margin.
Evidence. A seven-step server progression, fixed-box validation studies in which only identities change, a constraint ablation, sensitivity curves over the clustering hyperparameters, and a catalogue of nineteen negative results, all with an evaluator verified to reproduce the official TrackEval numbers exactly.
Related Work
Multi-view 3D detection. Early multi-view detectors project per-camera features onto a ground plane and reason in bird’s-eye view [8, 21]; the automotive literature replaced dense BEV grids with learned queries that sample image features through calibration [10, 24]. Sparse4D [11, 12] keeps a recurrent bank of sparse instance queries whose temporal propagation yields detection and tracking in one model. GaugeAlign [27] showed that the learned world-coordinate anchors of NVIDIA’s warehouse checkpoint of this family [23] are not translation equivariant across sites, and repaired the failure with a label-free canonicalization of the calibration. We keep the frozen checkpoint and that correction, and address the two error sources GaugeAlign left open: scale-dependent misses and fragmented identities.
Query-based and online tracking. MOTR [28] and MUTR3D [29] carry identity implicitly in persistent queries; MOTIP [5] instead predicts identity labels from a trajectory memory. SparseQID [18] applies the latter to Sparse4D queries for this challenge and ranked third in the official online track. The SORT family [1, 2, 3, 4, 26, 30] associates detections frame by frame with motion and appearance. All of these are causal; the stage that produces most of our gain clusters whole tracks with future observations, so we do not claim the challenge’s online bonus.
Multi-camera association with appearance. Hierarchical clustering of tracklets under ReID affinity and spatio-temporal constraints is the classical multi-camera design [31], refined at city scale by CityTrack [13] and, in the 2025 edition of this challenge, by an offline global-tracking stage that the winning team found decisive [19, 33]. Our setting differs in that the primitives are already fused 3D tracks from a query-based detector, not single-camera tracklets. We therefore project the 3D hypotheses back into every camera to collect appearance, and we impose reachability in metric world coordinates rather than in image space.
Track 1 systems. Syn2RealTrack [22] (second in the official track) lifts 2D detections through calibration and adapts to the synthetic-to-real gap online; our own earlier system [17] followed a similar geometry-first route with estimated depth. The present work abandons that route in favour of the frozen 3D detector and concentrates on what happens after it.
Method
Figure 2 gives the pipeline. Sections 3.1 and 3.2 produce boxes; Sections 3.3–3.6 change only identities.

Figure 2. System overview. The 3D detector is a frozen public checkpoint run at two input scales after gauge canonicalization; class-wise source selection is the only stage that changes boxes. All later stages are asserted to change object identities only. Blue stages are shared with GaugeAlign [27]; the 0.75× pass, the source selection, and every orange stage are ours.
Frozen Detector and Gauge Canonicalization
We use the public NVIDIA TAO Sparse4D warehouse checkpoint (release 2.2, ResNet-50 backbone, 900 anchors) and never fine-tune it. Let calibrated camera have extrinsics and world-frame centre . Following GaugeAlign [27], the network is centred before inference,
and every predicted centre is mapped back by ; dimensions, yaw, class, and instance identity are untouched. We also keep GaugeAlign’s class-taxonomy mapping, its temporal confidence decay of 0.95 for the instance bank, and its floor-region filter. Within each pass the detector’s instance identities are kept as raw tracklets.
Class-Wise Multiscale Source Selection
The checkpoint was trained on 540 × 960 inputs, half the native 1920 × 1080 resolution, and the production pipeline runs it there. Matching misses to ground truth on validation scene 20 shows where recall is lost: 46% of the missed Person frames are below 40 pixels tall in every camera and another 45% are 40–80 pixels; 54% are visible in only one camera. Running the same checkpoint on 810 × 1440 inputs (0.75×) raises raw Person HOTA on a validation segment from 39.39 to 45.32 but collapses Forklift (57.78 to 36.32) and PalletTruck (75.42 to 41.51), whose large extents fall outside the anchor statistics the model learned (Figure 4a). A union of both passes creates duplicate boxes for every object both passes see.
Figure 4. (a) Raw per-class HOTA on validation scene 20, frames 0–1,799, as a function of the Sparse4D input scale (exact evaluation; only the input dimension changes). Person improves up to and then saturates; both large classes collapse away from the training scale. (b) Full-validation Person HOTA, DetA, and AssA after the complete association chain for the source at five score thresholds; dotted lines are the source with the same chain.
We therefore select the source per class. For input scale , scene , and class let
where is detector confidence. The final box set for is exactly one : Person uses with (chosen on validation, § 4.5); Forklift, NovaCarter, Transporter, FourierGR1T2, AgilityDigit, and PalletTruck use with the production thresholds (0.7 for Forklift, 0.6 for PalletTruck, 0.2 otherwise). Because each class comes from one pass, raw instance identities remain consistent within a class and no deduplication is needed. Source selection precedes every association stage, so appearance descriptors are built for the selected rows. A variant that also takes the three humanoid and robot classes from 0.75× was built but not scored on the server (§ 4.8).
Geometry-to-Appearance Bridge
Sparse4D’s instance identities do not survive long occlusions, so we attach image appearance to its 3D tracks through the calibration.
ReID encoder. The encoder is an ImageNet-pretrained ResNet-50 [6] with a 512-dimensional linear–batch-norm head [16] and normalization. Training crops are cut from the 20 labelled training scenes at a 15-frame stride: 290,592 Person crops of 621 identities, resized to 256 × 128. Batches hold 32 identities with four images each; the loss is
with the batch-hard triplet loss [7], AdamW (learning rate , weight decay ), and random erasing [32] with probability 0.4. An epoch is one identity-balanced pass over the 621 identities (20 batches of 128 crops); training runs 60 such epochs, 1,200 iterations in about 15 minutes on one H100. The recipe was chosen among four by nearest-neighbour rank-1 on a held-out 10,000-crop subset; the final checkpoint reaches 90.03% (peak 90.34%). Degraded training twins (§ 4.1) are part of the crop set.
Projection and matching. For every Person box we project its eight 3D corners into every camera that sees it and take the enclosing rectangle. Within each scene–frame–camera tuple, projected boxes are matched one-to-one by the Hungarian algorithm [9] to the cached 2D person detections of the ReID extractor with cost
where is the centre distance normalized by the detection diagonal. A pair is a candidate if or , and is accepted when . On the test set this yields 5,450,393 camera projections and at least one accepted match for 820,383 of the 832,033 Person rows (98.60%).
Robust aggregation. For aggregation a match must additionally have detection confidence , , and cost ; its descriptor is weighted by
Simultaneous camera observations are first fused into one descriptor per 3D row. Per raw tracklet, a provisional weighted mean is formed, the 10% of rows least similar to it are dropped (never below a similarity of 0.20), and the mean is recomputed to give a global prototype; start and end prototypes use the first and last 90 frames. The test submission has 977 tracklet prototypes.
Short-Gap Causal Stitching
For each tracklet we fit a ground-plane velocity to its last 15 observations and clip its magnitude to 2 m/s. A predecessor that ended frames before successor started is a candidate if
with , an absolute residual of at most 2 m, a mean log-size change of at most 0.35, and, when both prototypes exist, an end-to-start ReID cosine similarity of at least 0.625. Motion, gap, size, and appearance terms are normalized and summed, and the assignment at each successor start frame is solved by the Hungarian algorithm. Only tracklets that have already ended are considered, so this stage is causal. It makes 133 joins in the test submission and is retained because it is cheap and raises the quality of the prototypes that feed the next stage.
Constraint-Aware Global Track Consolidation
Figure 1 shows that most same-identity gaps are longer than any reasonable causal horizon. We therefore consolidate the whole Person tracks of a scene offline. Each track is a node with the row-weighted centroid of its raw tracklets’ global prototypes,
where counts the rows tracklet contributes; tracks with fewer than 30 rows are not eligible. We repeatedly take the live pair with the highest cosine similarity and merge it if the similarity is at least and both hard constraints hold:
Temporal exclusion: the two tracks share no frame.
3D reachability: in the merged, time-ordered sequence, every switch from one track to the other spanning frames covers a ground-plane distance of at most metres.
After a merge the centroid is updated and the search continues; a pair that fails a constraint is excluded for good. There is no gap limit, so the procedure uses future observations and is offline. It makes 387 merges in the test submission. The constraints matter because appearance is weakest precisely where it is most needed: warehouse workers wear similar clothing, and in the restyled scenes distinct people routinely exceed . Two such people who are visible at the same time are certainly different, and temporal exclusion turns that into a free, exact veto; on validation it refuses more than 600 above-threshold pairs. The reachability gate bounds the remaining risk of joining people who are never seen together. § 4.6 measures both; Figure 3 shows one consolidated identity.
Figure 3. A fixed-box validation example (scene 20). Before global clustering, ten predicted identities cover one ground-truth person (dashed); the constrained global stage merges them into one scene-level identity. The right panel shows that only labels change: the temporal support of every fragment is preserved, and no two fragments overlap in time.
Class-Aware Long-Gap Stitching for Rigid Classes
For the six non-Person classes, a Person-trained appearance model is not informative, but these objects are rigid, their boxes are consistent, and their motion is slow. We reuse the causal stitcher of § 3.4 with a 1,800-frame gap and class-specific gates in place of Eq. (6): Forklift and Transporter use a 3 m base gate and 3 m/s; PalletTruck, NovaCarter, FourierGRI2, and AgilityDigit use 1 m and 2 m/s, and PalletTruck additionally caps the absolute residual at 4 m. The gates were selected on validation scene 22 and on training scenes 10 and 15, which contain the rare classes (Sec. 4.7). In the test submission these stages make 152 joins (53 Forklift, 4 PalletTruck, 1 NovaCarter, 10 Transporter, 26 FourierGR1T2, 58 AgilityDigit). Every association stage asserts numerically that frames, classes, positions, and dimensions are identical to its input and that no identity occupies a frame twice.
Experiments
Data, Metric, and Protocol
Table 1 summarizes the data [25]. Each scene provides synchronized video at 30 fps, calibration, and a map; training and validation scenes also provide annotations. Depth is not used. The seven classes are Person,
Forklift, NovaCarter, Transporter, FourierGR1T2, AgilityDigit, and PalletTruck. Validation has 144 Person identities but only 19 Forklift, 20 PalletTruck, two NovaCarter, and two Transporter identities, and no humanoid robots, so rare-class decisions were cross-checked on training scenes 10 and 15 with the production detector.
Hidden-test scores come from the official evaluation server. Local scores use the challenge’s TrackEval adaptation [14] with its 3D IoU matching; to iterate on identity-only changes we cache the per-frame 3D IoU matrices once and re-run TrackEval’s HOTA aggregation for every relabelling. This cached evaluator reproduces the official script digit for digit on every scene and class we checked. Local figures are combined over classes by detection count unless noted; the server’s class and scene weighting is not published, so validation is used for controlled comparisons and the server only for end-to-end submissions.
Implementation and Cost
Both detector passes use the same checkpoint, anchors, confidence decay, and floor filter; the pass changes only the input dimension. On one H100 with six decoding workers, 1,800 frames of 16 cameras take 442, 467, 565, and 647 s at , , , and . All post-detection stages for the whole test set (1.26 million rows) run in about 20 minutes on 16 CPU cores, of which global clustering takes well under a minute; the association scripts are pure NumPy. Association hyperparameters were selected on validation and held fixed on test.
Evaluation-Server Results
Table 2 places the result. Our reproduction of the official winner (56.7690) is slightly above the authors’ own 56.5447, which we attribute to threshold details of the release. On the 30 October snapshot our system led ELMO by 0.5818 HOTA. ELMO’s subsequent entries of 3 and 6 November reached 62.6375 and 65.8422; the second step added 3.7 DetA and 1.2 LocA points at once, which indicates a detector change. In every snapshot ELMO holds the detection advantage (2.7, 2.7, and 6.4 DetA points) while ours was association (2.69 and 1.49 AssA points, now −0.29). This is the paper’s premise in one table: once the detector is strong, association decides the order, and once both detectors are strong, detection decides it again. The frozen-detector route of this paper has reached the point where the detector itself is the limit.
Table 3 separates the effects. Rows 2–5 are the causal-stitcher refinements of the production system; together they are worth 0.97 HOTA, and the last of them only 0.07, which is the diminishing return that motivated an offline stage. Row 6, constrained global clustering plus rigid-class stitching, adds 3.7321 HOTA and 5.49 AssA while DetA is unchanged to four decimals, so the whole gain is identity topology. Row 7, the 0.75× Person source, adds 0.9187 DetA, 0.7577 AssA, and 0.2629 LocA; its association gain is indirect, since more complete Person observations yield longer tracklets and better prototypes. Validation showed a larger detection-weighted gain for the same change (+3.2 det-averaged HOTA over the five annotated classes), which is consistent with a server weighting in which Person is one class among seven and with test scenes 25–27 containing few people.
| Configuration | HOTA | DetA | AssA | LocA | HOTA |
| 1 GaugeAlign reproduction (frozen Sparse4D, 0.50×) | 56.7690 | 55.7189 | 49.6755 | 79.5895 | – |
| 2 + ReID gate in the causal stitcher | 56.8737 | 55.7188 | 49.8028 | 79.5895 | +0.1047 |
| 3 + class-wise score thresholds, motion gate | 57.3536 | 56.5312 | 50.0756 | 79.8645 | +0.4799 |
| 4 + causal ReID stitch, 180-frame gap | 57.6727 | 56.5311 | 50.4811 | 79.8646 | +0.3191 |
| 5 + 240-frame gap (previous champion) | 57.7383 | 56.5314 | 50.5654 | 79.8645 | +0.0656 |
| 6 + constrained global clustering, rigid-class stitching | 61.4704 | 56.5314 | 56.0554 | 79.8643 | +3.7321 |
| 7 + Person from 0.75×, (final) | 62.3947 | 57.4501 | 56.8131 | 80.1272 | +0.9243 |
Table 3. Our evaluation-server progression. Rows are submitted systems in order, not independent single-factor ablations. Rows 2–6 change identities only (DetA constant to the third decimal); row 7 changes the Person box source as well.
Controlled Validation: Stages, Fixed Boxes
Table 4 isolates association on the final boxes. The causal stitch recovers 1.8 Person HOTA; constrained global clustering recovers a further 9.1, reducing 1,041 identities to 516 for 144 people, and raising AssA from 23.5 to 39.2. Per scene, Person HOTA reaches 46.24 on scene 20, 31.58 on its degraded twin, and 44.58 on scene 22. The remaining excess of predicted over true identities is deliberate: with the constraints in place, lowering further merges fewer true pairs than it damages (Sec. 4.6). The same long-gap principle gives rigid classes +3.0 (Forklift), +2.5 (PalletTruck), +11.1 (NovaCarter), and +8.3 (Transporter) HOTA from motion and box consistency alone. On the half-resolution boxes of the previous champion the same Person stage moved HOTA from 26.849 to 35.115 (1,046 to 511 identities), which is the validation counterpart of row 6 in Table 3.
| Class | GT IDs | HOTA | AssA | IDs | HOTA | IDs | HOTA | AssA | IDs |
| Person | 144 | 29.189 | 20.887 | 1,210 | 31.009 | 1,041 | 40.108 | 39.219 | 516 |
| Forklift | 19 | 46.627 | 50.553 | 124 | – | – | 49.655 | 56.901 | 25 |
| PalletTruck | 20 | 58.398 | 67.709 | 33 | – | – | 60.936 | 73.752 | 22 |
| NovaCarter | 2 | 16.978 | 8.381 | 10 | – | – | 28.038 | 22.814 | 3 |
| Transporter | 2 | 16.979 | 11.472 | 14 | – | – | 25.251 | 23.943 | 4 |
Table 4. Stage-wise validation results on the final box set (Person from at , other classes from ). Within each class only object identities change between columns, so DetA and LocA are constant. “IDs” is the number of predicted identities; the number of ground-truth identities is in parentheses. Person passes through the causal and the global stage; rigid classes pass through the long-gap stitch.
Input Scale and Threshold
Figure 4a sweeps five input scales on a validation segment. Person HOTA rises from 39.39 at the training scale to 43.48, 45.32, 44.79, and 44.89 at , , , and native resolution, while Forklift and PalletTruck lose 20–35 points at every scale other than . Native resolution is thus not automatically best: the middle scale balances small-object evidence against the image statistics the checkpoint learned, and it costs 28% less than . Figure 4b then fixes the source and varies the Person threshold through the whole association chain on the full validation set: HOTA is 39.118, 39.234, 39.601, 40.108, and 40.044 at thresholds 0.50 to 0.70 in steps of 0.05, against 35.115 for the half-resolution source, while LocA rises monotonically from 78.41 to 78.94. The optimum is broad: every threshold beats the half-resolution source by at least 4.0 HOTA, and 0.65 is best by 0.06 over 0.70, with the same ordering on each of the three scenes. The threshold interacts with association: a higher threshold raises raw precision but removes the low-confidence rows that bridge fragments, which is why the raw optimum on the segment (0.70) is not the chain optimum.
Constraint Ablation and Sensitivity
Table 5 varies the constraints on the final boxes. Temporal exclusion is the decisive one. At the production setting it refuses 606 candidate pairs whose similarity exceeds , twelve times as many as reachability refuses (52), and 459 of them are in the degraded scene 21, where restyled textures make distinct people look alike. Such pairs are present in the same frame, so merging them is never right; the counts measure how often appearance alone would have done so. The reachability gate, by contrast, is a safety margin rather than a source of gain on this validation set: at removing it changes HOTA by +0.03, loosening it to 5 m and 5 m/s by +0.02, and tightening it to 0.5 m and 1 m/s costs 0.82, because genuine hand-offs across long occlusions are refused (182 rejections). Without the gate, can even be lowered to 0.55 with almost no loss (40.084), at the price of a visibly lower AssPr (60.9) and fewer identities than ground truth would warrant in the degraded scene. We keep the production gate because it bounds the damage of a wrong merge at no measurable cost, and because the test scenes (20 cameras) are larger than any validation scene. Prototype eligibility is immaterial (40.097 with every track eligible), and average linkage loses 0.50: its mean-pairwise criterion is more conservative, leaving 558 identities after 483 merges.
| Variant | HOTA | AssA | AssRe | AssPr | IDs | merges | refused | |
| none (after causal stitch) | – | 31.009 | 23.523 | 25.086 | 72.281 | 1,041 | 0 | – |
| temporal exclusion only | 0.55 | 40.084 | 39.169 | 44.641 | 60.871 | 414 | 627 | 1,077 / – |
| temporal exclusion only | 0.65 | 40.134 | 39.269 | 44.316 | 63.103 | 498 | 543 | 618 / – |
| temporal exclusion only | 0.75 | 39.421 | 37.905 | 42.292 | 66.238 | 628 | 413 | 234 / – |
| temporal exclusion only | 0.85 | 37.559 | 34.455 | 38.059 | 68.672 | 786 | 255 | 86 / – |
| + reachability, tight (0.5 m, 1 m/s) | 0.65 | 39.292 | 37.663 | 41.815 | 65.586 | 565 | 476 | 617 / 182 |
| + reachability, production (2 m, 2 m/s) | 0.65 | 40.108 | 39.219 | 44.047 | 63.849 | 516 | 525 | 606 / 52 |
| + reachability, loose (5 m, 5 m/s) | 0.65 | 40.128 | 39.257 | 44.165 | 63.472 | 505 | 536 | 635 / 26 |
| production, every track eligible | 0.65 | 40.097 | 39.199 | 44.023 | 63.834 | 492 | 549 | 621 / 55 |
| production, average linkage | 0.65 | 39.606 | 38.258 | 42.676 | 64.519 | 558 | 483 | 291 / 21 |
Table 5. Constraint ablation for global Person clustering on the final validation boxes (identities only change; exact evaluation). Every variant keeps temporal exclusion, without which the output would assign one identity to two boxes of the same frame. “Merges” and “refused” (candidate pairs above that a constraint rejected: temporal exclusion / reachability) are summed over the three scenes.
Figure 5 shows the operating region. Across the 36 settings on the final boxes with from 0.50 to 0.75, gates of 1, 2, and 3 m, and a minimum of 15 or 30 rows, Person HOTA stays within 39.36–40.22; on the half-resolution boxes, 24 settings of , gate, and speed stay within 34.89–35.15. The production setting (, 2 m, 2 m/s, 30 rows) was chosen on the half-resolution boxes and transferred to the 0.75 boxes without retuning; retuning there would have gained 0.11. The plateau towards lower is what temporal exclusion and the trimmed prototypes buy: the harmful merges that a lower threshold admits are overwhelmingly between people who are visible at the same time, and the constraint removes them for free.
Figure 5. Sensitivity of constrained global clustering. (a) Full-validation Person HOTA against the similarity threshold for three reachability base gates, on the final 0.75 boxes (circles) and on the previous half-resolution boxes (squares). (b) Predicted Person identities against at the production gate; the dotted lines mark the input identity counts and the dashed line the 144 ground-truth identities.
Rare Classes
If the server averages HOTA over classes, the four rare classes that appear only in test scenes 25–27 weigh as much as Person, and they are heavily fragmented there (our raw detector output has 74 AgilityDigit identities in scene 25 for at most seven simultaneous objects). Table 6 reports the evidence behind the class-aware gates: every class gains 2–14 HOTA from motion-only stitching on held-out and training scenes, the Transporter gate was widened to 3 m because scene 10 showed it beneficial and scene 22 was indifferent, and a per-instance class majority vote across the humanoid classes was tested and found neutral. On the test submission the rare-class stage reduces, for example, scene-25 FourierGR1T2 identities from 40 to 14 and AgilityDigit from 74 to 17.
| Class | Gate (base, speed) | Validation: HOTA | Validation: IDs (GT) | Train 010+015: HOTA |
| NovaCarter | 1 m, 2 m/s | 16.978 → 28.038 | 10 → 3 (2) | 54.059 → 68.506 |
| Transporter | 3 m, 3 m/s | 16.979 → 25.251 | 14 → 4 (2) | 48.210 → 55.859 |
| FourierGR1T2 | 1 m, 2 m/s | no annotations | – | 52.657 → 58.812 |
| AgilityDigit | 1 m, 2 m/s | no annotations | – | 60.829 → 63.115 |
| Forklift | 3 m, 3 m/s | 46.627 → 49.655 | 124 → 25 (19) | – |
| PalletTruck | 1 m, 2 m/s, 4 m cap | 58.398 → 60.936 | 33 → 22 (20) | – |
Table 6. Long-gap stitching for rigid classes. Validation columns use all validation scenes (NovaCarter and Transporter occur only in scene 22). Training scenes 10 and 15 were run with the production detector, which was trained on the training split, so their absolute values are optimistic; the gates were fixed from this evidence and applied unchanged to the test submission. Values are before → after stitching; ground-truth identity counts in parentheses.
Negative Results
Table 7 lists what was tried and rejected, with the exact validation outcome. Three lessons shaped the final design. Refinements that try to extract more from appearance, such as adaptive thresholds, average linkage, and junction vetoes, all lose, whereas the one constraint that matters is geometric, exact, and free. Detection cannot be improved by adding rows: every form of union, fusion, or low-confidence recovery lost to replacing the source for one class, and the union of both scales is worse than either alone. And the detector itself resists improvement by fine-tuning: the last two rows of Table 7 train the released checkpoint with the TAO training stack at a batch-matched learning rate (the released rate collapses the model, as GaugeAlign reports [27]), and Person is lower at every checkpoint of every recipe, on held-out scenes and even on the frames it was trained on, while the large classes adapt readily to the scale. The released checkpoint scores 69 Person HOTA on a training scene against 39–45 on validation, so the training split is in its training data and re-training on it adds no information; the small, distant people that account for most misses are not recovered by this model at this capacity.
| Alternative | Outcome |
| Association | |
| Average linkage instead of centroid linkage | 34.41–34.58 vs 34.79–35.15 across the same thresholds |
| Per-scene adaptive from “certain-negative” pairs (simultaneous tracks m apart) | 31.45–33.88 vs 34.80; scene 22 holds look-alike people |
| End-to-start ReID veto at every internal hand-off (0.2–0.6) | monotone loss, 35.11 to 32.77; endpoint crops are the most occluded |
| Retuning , gate, and minimum rows on the 0.75 boxes | best 40.222 vs 40.108 transferred |
| Linear gap interpolation inside merged tracks | only 8–11% of missed frames lie in gaps frames |
| Track-median box size or ground-plane smoothing | neutral or worse; Person length varies with pose |
| Causal-stitcher gap beyond 240 frames, reverse-motion gates | no gain; reverse gates lose 1–2 HOTA (server rows 4–5) |
| Detection | |
| Union of 0.50 and 0.75 outputs (scaled rows added where no 0.50 row lies within 0.5 m) | Person 38.89 vs 39.39 (0.50) and 45.32 (0.75); Forklift 51.11 vs 57.78; PalletTruck 64.52 vs 75.42 (segment) |
| Track-level fusion of 0.50 Person tracks not covered by 0.75 | 45.02–45.28 vs 45.32 (segment), 9 variants |
| Input scales 0.625, 0.875, 1.0 for Person | 43.48, 44.79, 44.89 vs 45.32 (segment) |
| Instance-bank confidence decay 0.90 / 0.98 at 0.75 | 45.39 / 44.71 vs 45.39 (segment, =0.65) |
| Lower Forklift / PalletTruck thresholds (0.6) | +0.14 / 0.00 HOTA; production kept |
| Transporter, FourierGR1T2, AgilityDigit from 0.75 (variant D-prime) | val Transporter +0.95 but the 0.75 pass loses all robot detections in test scene 26; built, not scored |
| Per-instance class majority vote among Person and humanoids | ±0.2 HOTA |
| 2D confirmation of low-score 3D candidates (0.05–0.6) | ≈ 50% false positives after confirmation; covers 14% of misses |
| Broad recovery of low-confidence 3D rows | lower HOTA (production system) |
| ResNet-101 Sparse4D variant | 41.43 HOTA on the server |
| Fine-tuning the checkpoint on five training scenes at 0.75 (LR 5e-6 / 2e-5, with and without the background-query loss, with and without a frozen backbone; 1,000 iterations) | Person 40.0–43.8 vs 45.4 at every checkpoint (segment, raw); LocA −1 to −3.5; only Forklift / PalletTruck adapt to 0.75 |
| Fine-tuning on validation scenes 20–21, tested on scene 22 | Person 44.6–45.3 vs 46.9 on scene 22; 43.4–44.4 vs 45.4 in-sample on scene 20 |
Table 7. Alternatives that did not improve exact validation HOTA (Person unless stated). “Segment” is validation scene 20, frames 0–1,799; “raw” means detector output before association.
Limitations and Reproducibility
Global clustering uses future frames, so the system is offline and ineligible for the online bonus; a causal variant would need to forgo the long-gap merges that account for most of the gain. The leaderboard result is a hidden-test measurement with no per-class or per-scene breakdown, and local validation has three sequences, two of which are the same world. Hyperparameters are fixed class-specific gates; they sit on broad plateaus (Figure 5) but a learned merge probability with the same hard constraints would transfer more gracefully. The largest remaining headroom is detection: the leading entry’s DetA is now 6.4 points higher than ours, and our own Person recall on validation is 45.5% at 69.1% precision. Centroid linkage can propagate a wrong merge; the constraints bound but do not eliminate this, and we observed AssPr fall as AssRe rises (Table 4). Finally, the paired clean and restyled scenes (Table 1) share a world; we treat every test scene independently, and did not attempt cross-scene fusion.
| Split | Scene | Cameras | Frames | Person | Forklift | PalletTruck | NovaCarter | Transporter |
| train | 000–019 | 4–19 | 9,000 | ReID training (621 identities); gates (010, 015) | ||||
| val | 020 | 16 | 9,000 | 60 | 9 | 10 | – | – |
| val | 021 (degraded twin) | 16 | 9,000 | 60 | 9 | 10 | – | – |
| val | 022 | 4 | 9,270 | 24 | 1 | – | 2 | 2 |
| test | 023 / 024 (twins) | 20 / 20 | 9,000 | hidden | ||||
| test | 025 | 10 | 9,000 | hidden; all rare classes present | ||||
| test | 026 / 027 | 4 / 7 | 1,800 | hidden |
Table 1. Scenes used in this paper. Ground-truth identity counts are given for the validation classes that have annotations; FourierGR1T2 and AgilityDigit occur only in the test split (and in training scenes). Validation scene 021 is a generatively restyled rendering of scene 020 with identical ground truth; test scenes 023/024 form the same kind of pair.
Reproducibility
scored submission: agent2_globalreid_D_fd075p065_C2/track1.zip
SHA-256 92581abca47eb8108c60fec52452885d441e10bff2bc43acc36aac4bf3b2e7d4
The archive contains one file, track1.txt, with 1,264,782 rows (832,033 Person, 143,932 Forklift, 20,132 NovaCarter, 46,827 Transporter, 24,890 FourierGR1T2, 41,520 AgilityDigit, 155,448 PalletTruck), no duplicate (scene, class, identity, frame) key, and no row dropped by the strict packager. Its non-Person rows are identical as a multiset to the previous champion’s, and its non-Person identity partition is identical to that of row 6 in Table 3; these properties were asserted at build time. Every stage is deterministic given its inputs; we keep the input hashes, configuration scripts, merge logs, per-stage outputs, and the cached-evaluator JSON files for every number in this paper. The detector is the public NVIDIA checkpoint; the ReID encoder is trained locally in 15 minutes from the released annotations. Public repository details and a license table will accompany the final release.
Conclusion
We presented a detector-preserving system for calibrated multi-camera 3D tracking that reaches 62.3947 3D HOTA on the AI City Challenge 2026 Track 1 evaluation server, first on the general leaderboard snapshot of 30 October 2026; a competitor’s later detector update has since moved the lead to 65.84, which marks the limit of the frozen-detector route: fine-tuning the released checkpoint did not raise Person detection under any recipe we tried, so the next step is a different detector, not a better use of this one. The frozen Sparse4D detector was not replaced or fine-tuned; the gain came from choosing the input scale per class and from repairing the topology of the detector’s tracks with appearance under hard physical constraints. Two results seem general. Replacing a class’s source at a second scale is safer than any form of multi-scale union for a query-based detector with learned size priors. And offline ReID clustering of 3D tracks becomes reliable once simultaneous presence is enforced as a hard veto, because it supplies exactly the negative evidence that appearance lacks among similarly dressed people; a metric reachability gate then bounds the remaining risk at no cost. For calibrated 3D tracking, repairing a strong detector’s trajectories can be as consequential as replacing the detector.
References
- [1]Aharon, N., Orfaig, R., Bobrovsky, B.Z.: BoT-SORT: Robust associations multi-pedestrian tracking. arXiv preprint (2022), https://arxiv.org/abs/2206.14651
- [2]Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online and realtime tracking. In: IEEE International Conference on Image Processing (ICIP). pp. 3464–3468 (2016)DOI
- [3]Cao, J., Pang, J., Weng, X., Khairdoar, R., Kitani, K.: Observation-centric SORT: Rethinking SORT for robust multi-object tracking. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)arxiv.org/abs/2203.14360
- [4]Du, Y., Zhao, Z., Song, Y., Zhao, Y., Su, F., Gong, T., Meng, H.: StrongSORT: Make DeepSORT great again. IEEE Transactions on Multimedia 25, 8725–8737 (2023)arxiv.org/abs/2202.13514
- [5]Gao, R., Qi, J., Wang, L.: Multiple object tracking as ID prediction. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)arxiv.org/abs/2403.16848
- [6]He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016)arxiv.org/abs/1512.03385
- [7]Hermans, A., Beyer, L., Leibe, B.: In defense of the triplet loss for person re-identification. arXiv preprint (2017), https://arxiv.org/abs/1703.07737
- [8]Hou, Y., Zheng, L., Gould, S.: Multiview detection with feature perspective transformation. In: European Conference on Computer Vision (ECCV) (2020)arxiv.org/abs/2007.07247
- [9]Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Research Logistics Quarterly 2(1–2), 83–97 (1955)
- [10]Li, Z., Wang, W., Li, H., Xie, E., Sima, C., Lu, T., Qiao, Y., Dai, J.: BEVFormer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In: European Conference on Computer Vision (ECCV) (2022)arxiv.org/abs/2203.17270
- [11]Lin, X., Lin, T., Pei, Z., Huang, L., Su, Z.: Sparse4D v2: Recurrent temporal fusion with sparse model. arXiv preprint (2023), https://arxiv.org/abs/2305.14018
- [12]Lin, X., Pei, Z., Lin, T., Huang, L., Su, Z.: Sparse4D v3: Advancing end-to-end 3D detection and tracking. arXiv preprint (2023), https://arxiv.org/abs/2311.11722
- [13]Lu, J., Yang, X., Ye, J., Zhang, Y., Zou, Z., Zhang, W., Tan, X.: CityTrack: Improving city-scale multi-camera multi-target tracking by location-aware tracking and box-grained matching. arXiv preprint (2023), https://arxiv.org/abs/2307.02753
- [14]Luiten, J., Hoffhues, A.: TrackEval. https://github.com/JonathonLuiten/TrackEval (2020)
- [15]Luiten, J., Osep, A., Dendorfer, P., Torr, P., Geiger, A., Leal-Taixé, L., Leibe, B.: HOTA: A higher order metric for evaluating multi-object tracking. International Journal of Computer Vision 129, 548–578 (2021)DOI
- [16]Luo, H., Gu, Y., Liao, X., Lai, S., Jiang, W.: Bag of tricks and a strong baseline for deep person re-identification. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)arxiv.org/abs/1903.07071
- [17]Naeem, A., Katwal, A., Dey, A., Khan, N., Hoque, M.T.: Geometry beats estimated depth: RGB-only multi-camera 3D tracking under Sim2Real. arXiv preprint (2026), https://arxiv.org/abs/2608.07579
- [18]Shrestha, P., Nakayama, H., Scott, A.: Online multi-camera 3D tracking via ID prediction over recurrent sparse queries. arXiv preprint (2026), https://arxiv.org/abs/2609.18363
- [19]Tang, Z., Wang, S., Anastasiu, D.C., Chang, M.C., Sharma, A., et al.: The 9th AI City Challenge. arXiv preprint (2025), https://arxiv.org/abs/2508.13564
- [20]Tang, Z., Wang, S., Anastasiu, D.C., Chang, M.C., Sharma, A., et al.: The 10th AI City Challenge. arXiv preprint (2026), https://arxiv.org/abs/2608.17044
- [21]Teepe, T., Wolters, P., Gilg, J., Herzog, F., Rigoll, G.: EarlyBird: Early-fusion for multi-view tracking in the bird’s eye view. In: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops (2024)arxiv.org/abs/2310.13350
- [22]Tran, D.N.N., Huynh, N.D.M., Le, C.Q., Nguyen, H.K., Pham, L.H., et al.: Syn2RealTrack: Bridging the gap between synthetic and real-world datasets for online multi-view multi-target tracking. arXiv preprint (2026), https://arxiv.org/abs/2608.24130, accepted at the AI City Challenge Workshop, ECCV 2026
- [23]Wang, Y., Pusegaonkar, S., Wang, Y., Li, A., Kumar, V., Sethi, C., et al.: A unified 3D object perception framework for real-time outside-in multi-camera systems. arXiv preprint (2026), https://arxiv.org/abs/2601.10819
- [24]Wang, Y., Guizilini, V., Zhang, T., Wang, Y., Zhao, H., Solomon, J.: DETR3D: 3D object detection from multi-view images via 3D-to-2D queries. In: Conference on Robot Learning (CoRL) (2021)arxiv.org/abs/2110.06922
- [25]Wang, Y., Wang, Y., Li, A., Wang, S., Pusegaonkar, S.S., Liang, H., Li, J., Jiang, S., Yuan, J., Li, S., Dai, T., Chen, Z., Anastasiu, D., Tang, Z., Biswas, S., Wu, X.: Physical AI smart spaces: A large-scale benchmark for multi-camera 3D perception in smart spaces. In: Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track (2026), openReview id SdCUsRcgQIarxiv.org/abs/2610.02580
- [26]Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric. In: IEEE International Conference on Image Processing (ICIP). pp. 3645–3649 (2017)arxiv.org/abs/1703.07402
- [27]Wu, X., Fang, H., Xi, X., Tang, Y., Xi, D., Bai, X., Liang, D.: Aligning learned spatial priors: Gauge canonicalization for multi-camera 3D perception under sim-to-real transfer. In: European Conference on Computer Vision (ECCV) Workshops (2026), code: https://github.com/HyperbolicCurve/GaugeAlign-AICity2026
- [28]Zeng, F., Dong, B., Zhang, Y., Wang, T., Zhang, X., Wei, Y.: MOTR: End-to-end multiple-object tracking with transformer. In: European Conference on Computer Vision (ECCV) (2022)arxiv.org/abs/2105.03247
- [29]Zhang, T., Chen, X., Wang, Y., Wang, Y., Zhao, H.: MUTR3D: A multi-camera tracking framework via 3D-to-2D queries. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2022)arxiv.org/abs/2205.00613
- [30]Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., Wang, X.: ByteTrack: Multi-object tracking by associating every detection box. In: European Conference on Computer Vision (ECCV) (2022)arxiv.org/abs/2110.06864
- [31]Zhang, Z., Wu, J., Zhang, X., Zhang, C.: Multi-target, multi-camera tracking by hierarchical clustering: Recent progress on DukeMTMC project. arXiv preprint (2017), https://arxiv.org/abs/1712.09531
- [32]Zhong, Z., Zheng, L., Kang, G., Li, S., Yang, Y.: Random erasing data augmentation. In: AAAI Conference on Artificial Intelligence (2020)arxiv.org/abs/1708.04896
- [33]ZIOVISION: Multi-camera 3D object tracking via V-DETR and point cloud re-identification: First-place solution to the 2025 AI City Challenge track 1. https://github.com/ZIOVISION/AIC2025_Track1_ZV (2025)



