betaFeedback

WorksComputer Science

Detection Is Not the Finish Line: Multiscale Decoding and Constraint-Aware Global Association for Multi-Camera 3D Tracking

Abstract

A query-based multi-view 3D detector can transfer well to a new calibrated camera network and still produce badly fragmented tracks: on the AI City Challenge 2026 Track 1 validation set, the frozen Sparse4D detector behind the official challenge winner splits each Person identity into a median of eight fragments, and half of the gaps between fragments exceed the horizon of a causal tracker. We show that most of the remaining headroom of such a system lies after detection. Keeping the public detector frozen, we add (i) class-wise source selection between two input scales, which recovers small and distant people from a 0.75x pass while leaving large objects at the scale the detector was trained for; (ii) a geometry-to-appearance bridge that projects every 3D box into all calibrated views, matches it to 2D person detections, and aggregates robust track-level re-identification descriptors; and (iii) constraint-aware consolidation, which combines a causal short-gap stitch with offline centroid-linkage clustering under two hard constraints, temporal exclusion and metric 3D reachability, plus class-aware long-gap stitching for rigid objects. Every association stage changes object identities only. On the challenge evaluation server the system scores 62.3947 3D HOTA (DetA 57.4501, AssA 56.8131, LocA 80.1272), first on the general leaderboard snapshot of 30 October 2026 (second at the time of writing, after a competitor's detector update) and 5.63 points above our reproduction of the official winner; the constrained association stage alone contributes +3.73 HOTA through a +5.49 AssA gain at unchanged DetA. Controlled fixed-box validation studies, a constraint ablation, sensitivity curves, and a catalogue of negative results support the design.

Introduction

Multi-camera 3D perception asks for a scene-level account of every object: a position in a shared world frame and one identity for as long as the object is present, however many cameras it crosses. Track 1 of the 2026 AI City Challenge [20, 25] poses this problem in warehouses with up to 20 calibrated cameras, seven object classes, synthetic and real scenes, and no depth at inference time. The metric is a 3D adaptation of HOTA [15], whose factors separate detection quality (DetA), association quality (AssA), and localization (LocA).

The official winner of the track, GaugeAlign [27], runs a frozen NVIDIA TAO Sparse4D checkpoint [12] after a closed-form canonicalization of the camera network’s coordinate gauge, and won on detection alone: its authors report a 10-point DetA margin over the runner-up and a 7-point AssA deficit. Our reproduction of that system scores 56.7690 3D HOTA on the evaluation server with 55.7189 DetA but only 49.6755 AssA. Figure 1 quantifies the reason. Matching the detector’s raw tracks to validation ground truth, each Person identity is covered by a median of eight predicted identities, the median gap between consecutive fragments of one person is 274 frames, and 52% of those gaps are longer than the 240-frame horizon of a causal stitcher; 18% exceed 900 frames (30 s). The tracklets themselves are about 95% pure. The detector, in other words, is not the bottleneck; the topology of its tracks is.

Figure 1. Why a causal tracker cannot repair the frozen detector’s tracks. Validation Person predictions (half-resolution production boxes) are matched to ground truth per frame with a 1 m centre gate; a fragment is a predicted identity matched to one ground-truth person on at least ten frames. (a) Gaps between consecutive fragments of the same person: with raw Sparse4D identities (grey) the median gap is 274 frames and 52% of gaps exceed the 240-frame causal horizon (dashed); the causal stitch (orange) removes mostly short gaps, leaving 57% of the remainder beyond the horizon. Constrained global clustering (blue) removes two thirds of all gaps. (b) Fragments per ground-truth identity: median 8 (interquartile range 6–10) raw, 7 after the causal stitch, 3 after global clustering.

This observation motivates a detector-preserving strategy. Rather than train another 3D detector, we ask two questions. First, can a different input scale recover the small, distant people the detector misses, without harming the classes it already handles well? Second, can fragmented world-space tracks be consolidated over arbitrarily long gaps using appearance, while the physical constraints available in a calibrated 3D scene prevent the over-merges that make unconstrained re-identification (ReID) clustering unsafe? Both answers are affirmative, and together they move the server score to 62.3947 3D HOTA, first on the general leaderboard snapshot of 30 October 2026 (Table 2).

RankingSystemHOTADetAAssALocA
official (online)GaugeAlign (EVA) [27]56.544755.644449.392979.5558
Syn2RealTrack (SKKU-AL-T1) [22]52.011845.305656.504776.2410
SparseQID (Playbox) [18]38.010540.259231.097875.1778
general leaderboard (30 Oct 2026)ParkLabs (ours)62.394757.450156.813180.1272
ELMO61.812960.159654.124180.8056
Metropolis60.7019–––
(3 Nov 2026)ELMO, updated entry62.637560.157055.318680.8063
(6 Nov 2026)ELMO, updated entry65.842263.876157.103381.9793

Table 2. Hidden-test results from the official evaluation server (percent). Top: the official Track 1 online ranking at the close of the challenge, as reported by the respective teams [18, 22, 27]. Bottom: the general leaderboard snapshot of 30 October 2026 after our final submission, and the updated ELMO entries of 3 and 6 November 2026. Our entry is offline and post-deadline; it is a server rank, not a claim to the challenge award or the online bonus.

Our contributions are:

  • Class-wise multiscale source selection for a frozen Sparse4D model (§ 3.2). Person boxes come from a 0.75× pass and every other class from the 0.50× training scale; the output of each class is replaced, never unioned, so no duplicates arise. Full-validation Person HOTA after association rises from 35.115 to 40.108, and the server score by 0.9243.

  • A geometry-to-appearance bridge (§ 3.3) from world-space 3D boxes to multi-view Person descriptors: projection, one-to-one matching, confidence- and overlap-weighted cross-view fusion, and trimmed track-level prototypes. It supplies descriptors for 98.6% of the Person rows in the final test submission.

  • Constraint-aware consolidation (§ 3.4–§ 3.6): a causal short-gap stitch, an offline centroid-linkage clustering of whole tracks under temporal-exclusion and 3D-reachability constraints, and class-aware long-gap stitching for six rigid classes. On the server this stage adds 3.7321 HOTA with DetA unchanged to four decimals. An ablation shows that temporal exclusion, which refuses hundreds of above-threshold pairs of look-alike people seen simultaneously, is what makes a fixed moderate threshold safe, while the reachability gate is a cost-free safety margin.

  • Evidence. A seven-step server progression, fixed-box validation studies in which only identities change, a constraint ablation, sensitivity curves over the clustering hyperparameters, and a catalogue of nineteen negative results, all with an evaluator verified to reproduce the official TrackEval numbers exactly.

Related Work

Multi-view 3D detection. Early multi-view detectors project per-camera features onto a ground plane and reason in bird’s-eye view [8, 21]; the automotive literature replaced dense BEV grids with learned queries that sample image features through calibration [10, 24]. Sparse4D [11, 12] keeps a recurrent bank of sparse instance queries whose temporal propagation yields detection and tracking in one model. GaugeAlign [27] showed that the learned world-coordinate anchors of NVIDIA’s warehouse checkpoint of this family [23] are not translation equivariant across sites, and repaired the failure with a label-free canonicalization of the calibration. We keep the frozen checkpoint and that correction, and address the two error sources GaugeAlign left open: scale-dependent misses and fragmented identities.

Query-based and online tracking. MOTR [28] and MUTR3D [29] carry identity implicitly in persistent queries; MOTIP [5] instead predicts identity labels from a trajectory memory. SparseQID [18] applies the latter to Sparse4D queries for this challenge and ranked third in the official online track. The SORT family [1, 2, 3, 4, 26, 30] associates detections frame by frame with motion and appearance. All of these are causal; the stage that produces most of our gain clusters whole tracks with future observations, so we do not claim the challenge’s online bonus.

Multi-camera association with appearance. Hierarchical clustering of tracklets under ReID affinity and spatio-temporal constraints is the classical multi-camera design [31], refined at city scale by CityTrack [13] and, in the 2025 edition of this challenge, by an offline global-tracking stage that the winning team found decisive [19, 33]. Our setting differs in that the primitives are already fused 3D tracks from a query-based detector, not single-camera tracklets. We therefore project the 3D hypotheses back into every camera to collect appearance, and we impose reachability in metric world coordinates rather than in image space.

Track 1 systems. Syn2RealTrack [22] (second in the official track) lifts 2D detections through calibration and adapts to the synthetic-to-real gap online; our own earlier system [17] followed a similar geometry-first route with estimated depth. The present work abandons that route in favour of the frozen 3D detector and concentrates on what happens after it.

Method

Figure 2 gives the pipeline. Sections 3.1 and 3.2 produce boxes; Sections 3.3–3.6 change only identities.

System overview diagram showing gauge canonicalization, frozen Sparse4D at two scales, class-wise source selection, and identity-association stages

Figure 2. System overview. The 3D detector is a frozen public checkpoint run at two input scales after gauge canonicalization; class-wise source selection is the only stage that changes boxes. All later stages are asserted to change object identities only. Blue stages are shared with GaugeAlign [27]; the 0.75× pass, the source selection, and every orange stage are ours.

Frozen Detector and Gauge Canonicalization

We use the public NVIDIA TAO Sparse4D warehouse checkpoint (release 2.2, ResNet-50 backbone, 900 anchors) and never fine-tune it. Let calibrated camera ii have extrinsics [Ri∣ti][R_i \mid t_i] and world-frame centre Ci=−RiTtiC_i = -R_i^{\mathsf{T}}t_i. Following GaugeAlign [27], the network is centred before inference,

Cˉ=1N∑i=1NCi,ti′=ti+RiCˉ,(1)\bar{C} = \frac{1}{N}\sum_{i=1}^{N} C_i,\qquad t'_i = t_i + R_i\bar{C}, \tag*{(1)}

and every predicted centre p′p' is mapped back by p=p′+Cˉp = p' + \bar{C}; dimensions, yaw, class, and instance identity are untouched. We also keep GaugeAlign’s class-taxonomy mapping, its temporal confidence decay of 0.95 for the instance bank, and its floor-region filter. Within each pass the detector’s instance identities are kept as raw tracklets.

Class-Wise Multiscale Source Selection

The checkpoint was trained on 540 × 960 inputs, half the native 1920 × 1080 resolution, and the production pipeline runs it there. Matching misses to ground truth on validation scene 20 shows where recall is lost: 46% of the missed Person frames are below 40 pixels tall in every camera and another 45% are 40–80 pixels; 54% are visible in only one camera. Running the same checkpoint on 810 × 1440 inputs (0.75×) raises raw Person HOTA on a validation segment from 39.39 to 45.32 but collapses Forklift (57.78 to 36.32) and PalletTruck (75.42 to 41.51), whose large extents fall outside the anchor statistics the model learned (Figure 4a). A union of both passes creates duplicate boxes for every object both passes see.

Figure 4. (a) Raw per-class HOTA on validation scene 20, frames 0–1,799, as a function of the Sparse4D input scale (exact evaluation; only the input dimension changes). Person improves up to 0.75×0.75\times and then saturates; both large classes collapse away from the 0.50×0.50\times training scale. (b) Full-validation Person HOTA, DetA, and AssA after the complete association chain for the 0.75×0.75\times source at five score thresholds; dotted lines are the 0.50×0.50\times source with the same chain.

We therefore select the source per class. For input scale ss, scene ww, and class cc let

Bw,cs(τc)={b∈Bws:y(b)=c, q(b)≥τc},(2)\mathcal{B}^{s}_{w,c}(\tau_c) = \{b \in\mathcal{B}^{s}_{w} : y(b) = c,\ q(b) \ge\tau_c\}, \tag*{(2)}

where qq is detector confidence. The final box set for (w,c)(w,c) is exactly one Bw,cs(τc)\mathcal{B}^{s}_{w,c}(\tau_c): Person uses s=0.75s = 0.75 with τ=0.65\tau= 0.65 (chosen on validation, § 4.5); Forklift, NovaCarter, Transporter, FourierGR1T2, AgilityDigit, and PalletTruck use s=0.50s = 0.50 with the production thresholds (0.7 for Forklift, 0.6 for PalletTruck, 0.2 otherwise). Because each class comes from one pass, raw instance identities remain consistent within a class and no deduplication is needed. Source selection precedes every association stage, so appearance descriptors are built for the selected rows. A variant that also takes the three humanoid and robot classes from 0.75× was built but not scored on the server (§ 4.8).

Geometry-to-Appearance Bridge

Sparse4D’s instance identities do not survive long occlusions, so we attach image appearance to its 3D tracks through the calibration.

ReID encoder. The encoder is an ImageNet-pretrained ResNet-50 [6] with a 512-dimensional linear–batch-norm head [16] and ℓ2\ell_2 normalization. Training crops are cut from the 20 labelled training scenes at a 15-frame stride: 290,592 Person crops of 621 identities, resized to 256 × 128. Batches hold 32 identities with four images each; the loss is

L=LCE+2LBH(m=0.4),(3)\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + 2\mathcal{L}_{\mathrm{BH}}(m=0.4), \tag*{(3)}

with LBH\mathcal{L}_{\mathrm{BH}} the batch-hard triplet loss [7], AdamW (learning rate 3×10−43 \times10^{-4}, weight decay 10−410^{-4}), and random erasing [32] with probability 0.4. An epoch is one identity-balanced pass over the 621 identities (20 batches of 128 crops); training runs 60 such epochs, 1,200 iterations in about 15 minutes on one H100. The recipe was chosen among four by nearest-neighbour rank-1 on a held-out 10,000-crop subset; the final checkpoint reaches 90.03% (peak 90.34%). Degraded training twins (§ 4.1) are part of the crop set.

Projection and matching. For every Person box we project its eight 3D corners into every camera that sees it and take the enclosing rectangle. Within each scene–frame–camera tuple, projected boxes are matched one-to-one by the Hungarian algorithm [9] to the cached 2D person detections of the ReID extractor with cost

c(p,d)=0.7(1−IoU⁡(p,d))+0.3min⁡(δctr(p,d),2)2,(4)c(p,d) = 0.7\left(1-\operatorname{IoU}(p,d)\right) + 0.3\frac{\min(\delta_{\mathrm{ctr}}(p,d),2)}{2}, \tag*{(4)}

where δctr\delta_{\mathrm{ctr}} is the centre distance normalized by the detection diagonal. A pair is a candidate if IoU⁡≥0.02\operatorname{IoU} \ge0.02 or δctr≤1\delta_{\mathrm{ctr}} \le1, and is accepted when c≤0.82c \le0.82. On the test set this yields 5,450,393 camera projections and at least one accepted match for 820,383 of the 832,033 Person rows (98.60%).

Robust aggregation. For aggregation a match must additionally have detection confidence ≥0.15\ge0.15, IoU⁡≥0.10\operatorname{IoU} \ge0.10, and cost ≤0.72\le0.72; its descriptor is weighted by

ω=q(0.25+0.75IoU⁡)exp⁡(−δctr).(5)\omega= \sqrt{q}\left(0.25 + 0.75\operatorname{IoU}\right)\exp(-\delta_{\mathrm{ctr}}). \tag*{(5)}

Simultaneous camera observations are first fused into one descriptor per 3D row. Per raw tracklet, a provisional weighted mean is formed, the 10% of rows least similar to it are dropped (never below a similarity of 0.20), and the mean is recomputed to give a global prototype; start and end prototypes use the first and last 90 frames. The test submission has 977 tracklet prototypes.

Short-Gap Causal Stitching

For each tracklet we fit a ground-plane velocity to its last 15 observations and clip its magnitude to 2 m/s. A predecessor aa that ended Δf\Delta f frames before successor bb started is a candidate if

∥xbstart−(xaend+vaΔf)∥2≤0.35+2Δf30 m,(6)\left\lVert\boldsymbol{x}_{b}^{\mathrm{start}}-\left(\boldsymbol{x}_{a}^{\mathrm{end}}+\boldsymbol{v}_{a}\Delta f\right)\right\rVert_2 \le0.35 + 2\frac{\Delta f}{30}\ \mathrm{m}, \tag*{(6)}

with Δf≤240\Delta f \le240, an absolute residual of at most 2 m, a mean log-size change of at most 0.35, and, when both prototypes exist, an end-to-start ReID cosine similarity of at least 0.625. Motion, gap, size, and appearance terms are normalized and summed, and the assignment at each successor start frame is solved by the Hungarian algorithm. Only tracklets that have already ended are considered, so this stage is causal. It makes 133 joins in the test submission and is retained because it is cheap and raises the quality of the prototypes that feed the next stage.

Constraint-Aware Global Track Consolidation

Figure 1 shows that most same-identity gaps are longer than any reasonable causal horizon. We therefore consolidate the whole Person tracks of a scene offline. Each track TT is a node with the row-weighted centroid of its raw tracklets’ global prototypes,

eT=∑k∈Tnkek∥∑k∈Tnkek∥2,(7)e_T = \frac{\sum_{k\in T} n_k e_k}{\left\lVert\sum_{k\in T} n_k e_k \right\rVert_2}, \tag*{(7)}

where nkn_k counts the rows tracklet kk contributes; tracks with fewer than 30 rows are not eligible. We repeatedly take the live pair with the highest cosine similarity and merge it if the similarity is at least τ=0.65\tau= 0.65 and both hard constraints hold:

  1. Temporal exclusion: the two tracks share no frame.

  2. 3D reachability: in the merged, time-ordered sequence, every switch from one track to the other spanning Δf\Delta f frames covers a ground-plane distance of at most 2.0+2.0 Δf/302.0 + 2.0\,\Delta f/30 metres.

After a merge the centroid is updated and the search continues; a pair that fails a constraint is excluded for good. There is no gap limit, so the procedure uses future observations and is offline. It makes 387 merges in the test submission. The constraints matter because appearance is weakest precisely where it is most needed: warehouse workers wear similar clothing, and in the restyled scenes distinct people routinely exceed τ\tau. Two such people who are visible at the same time are certainly different, and temporal exclusion turns that into a free, exact veto; on validation it refuses more than 600 above-threshold pairs. The reachability gate bounds the remaining risk of joining people who are never seen together. § 4.6 measures both; Figure 3 shows one consolidated identity.

Figure 3. A fixed-box validation example (scene 20). Before global clustering, ten predicted identities cover one ground-truth person (dashed); the constrained global stage merges them into one scene-level identity. The right panel shows that only labels change: the temporal support of every fragment is preserved, and no two fragments overlap in time.

Class-Aware Long-Gap Stitching for Rigid Classes

For the six non-Person classes, a Person-trained appearance model is not informative, but these objects are rigid, their boxes are consistent, and their motion is slow. We reuse the causal stitcher of § 3.4 with a 1,800-frame gap and class-specific gates in place of Eq. (6): Forklift and Transporter use a 3 m base gate and 3 m/s; PalletTruck, NovaCarter, FourierGRI2, and AgilityDigit use 1 m and 2 m/s, and PalletTruck additionally caps the absolute residual at 4 m. The gates were selected on validation scene 22 and on training scenes 10 and 15, which contain the rare classes (Sec. 4.7). In the test submission these stages make 152 joins (53 Forklift, 4 PalletTruck, 1 NovaCarter, 10 Transporter, 26 FourierGR1T2, 58 AgilityDigit). Every association stage asserts numerically that frames, classes, positions, and dimensions are identical to its input and that no identity occupies a frame twice.

Experiments

Data, Metric, and Protocol

Table 1 summarizes the data [25]. Each scene provides synchronized 1920×10801920 \times1080 video at 30 fps, calibration, and a map; training and validation scenes also provide 2D/3D2\mathrm{D}/3\mathrm{D} annotations. Depth is not used. The seven classes are Person,

Forklift, NovaCarter, Transporter, FourierGR1T2, AgilityDigit, and PalletTruck. Validation has 144 Person identities but only 19 Forklift, 20 PalletTruck, two NovaCarter, and two Transporter identities, and no humanoid robots, so rare-class decisions were cross-checked on training scenes 10 and 15 with the production detector.

Hidden-test scores come from the official evaluation server. Local scores use the challenge’s TrackEval adaptation [14] with its 3D IoU matching; to iterate on identity-only changes we cache the per-frame 3D IoU matrices once and re-run TrackEval’s HOTA aggregation for every relabelling. This cached evaluator reproduces the official script digit for digit on every scene and class we checked. Local figures are combined over classes by detection count unless noted; the server’s class and scene weighting is not published, so validation is used for controlled comparisons and the server only for end-to-end submissions.

Implementation and Cost

Both detector passes use the same checkpoint, anchors, confidence decay, and floor filter; the 0.75×0.75\times pass changes only the input dimension. On one H100 with six decoding workers, 1,800 frames of 16 cameras take 442, 467, 565, and 647 s at 0.625×0.625\times, 0.75×0.75\times, 0.875×0.875\times, and 1.0×1.0\times. All post-detection stages for the whole test set (1.26 million rows) run in about 20 minutes on 16 CPU cores, of which global clustering takes well under a minute; the association scripts are pure NumPy. Association hyperparameters were selected on validation and held fixed on test.

Evaluation-Server Results

Table 2 places the result. Our reproduction of the official winner (56.7690) is slightly above the authors’ own 56.5447, which we attribute to threshold details of the release. On the 30 October snapshot our system led ELMO by 0.5818 HOTA. ELMO’s subsequent entries of 3 and 6 November reached 62.6375 and 65.8422; the second step added 3.7 DetA and 1.2 LocA points at once, which indicates a detector change. In every snapshot ELMO holds the detection advantage (2.7, 2.7, and 6.4 DetA points) while ours was association (2.69 and 1.49 AssA points, now −0.29). This is the paper’s premise in one table: once the detector is strong, association decides the order, and once both detectors are strong, detection decides it again. The frozen-detector route of this paper has reached the point where the detector itself is the limit.

Table 3 separates the effects. Rows 2–5 are the causal-stitcher refinements of the production system; together they are worth 0.97 HOTA, and the last of them only 0.07, which is the diminishing return that motivated an offline stage. Row 6, constrained global clustering plus rigid-class stitching, adds 3.7321 HOTA and 5.49 AssA while DetA is unchanged to four decimals, so the whole gain is identity topology. Row 7, the 0.75× Person source, adds 0.9187 DetA, 0.7577 AssA, and 0.2629 LocA; its association gain is indirect, since more complete Person observations yield longer tracklets and better prototypes. Validation showed a larger detection-weighted gain for the same change (+3.2 det-averaged HOTA over the five annotated classes), which is consistent with a server weighting in which Person is one class among seven and with test scenes 25–27 containing few people.

ConfigurationHOTADetAAssALocAΔ\DeltaHOTA
1 GaugeAlign reproduction (frozen Sparse4D, 0.50×)56.769055.718949.675579.5895–
2 + ReID gate in the causal stitcher56.873755.718849.802879.5895+0.1047
3 + class-wise score thresholds, motion gate57.353656.531250.075679.8645+0.4799
4 + causal ReID stitch, 180-frame gap57.672756.531150.481179.8646+0.3191
5 + 240-frame gap (previous champion)57.738356.531450.565479.8645+0.0656
6 + constrained global clustering, rigid-class stitching61.470456.531456.055479.8643+3.7321
7 + Person from 0.75×, τ=0.65\tau=0.65 (final)62.394757.450156.813180.1272+0.9243

Table 3. Our evaluation-server progression. Rows are submitted systems in order, not independent single-factor ablations. Rows 2–6 change identities only (DetA constant to the third decimal); row 7 changes the Person box source as well.

Controlled Validation: Stages, Fixed Boxes

Table 4 isolates association on the final boxes. The causal stitch recovers 1.8 Person HOTA; constrained global clustering recovers a further 9.1, reducing 1,041 identities to 516 for 144 people, and raising AssA from 23.5 to 39.2. Per scene, Person HOTA reaches 46.24 on scene 20, 31.58 on its degraded twin, and 44.58 on scene 22. The remaining excess of predicted over true identities is deliberate: with the constraints in place, lowering τ\tau further merges fewer true pairs than it damages (Sec. 4.6). The same long-gap principle gives rigid classes +3.0 (Forklift), +2.5 (PalletTruck), +11.1 (NovaCarter), and +8.3 (Transporter) HOTA from motion and box consistency alone. On the half-resolution boxes of the previous champion the same Person stage moved HOTA from 26.849 to 35.115 (1,046 to 511 identities), which is the validation counterpart of row 6 in Table 3.

ClassGT IDsHOTAAssAIDsHOTAIDsHOTAAssAIDs
Person14429.18920.8871,21031.0091,04140.10839.219516
Forklift1946.62750.553124––49.65556.90125
PalletTruck2058.39867.70933––60.93673.75222
NovaCarter216.9788.38110––28.03822.8143
Transporter216.97911.47214––25.25123.9434

Table 4. Stage-wise validation results on the final box set (Person from 0.75×0.75\times at τ=0.65\tau=0.65, other classes from 0.50×0.50\times). Within each class only object identities change between columns, so DetA and LocA are constant. “IDs” is the number of predicted identities; the number of ground-truth identities is in parentheses. Person passes through the causal and the global stage; rigid classes pass through the long-gap stitch.

Input Scale and Threshold

Figure 4a sweeps five input scales on a validation segment. Person HOTA rises from 39.39 at the training scale to 43.48, 45.32, 44.79, and 44.89 at 0.625×0.625\times, 0.75×0.75\times, 0.875×0.875\times, and native resolution, while Forklift and PalletTruck lose 20–35 points at every scale other than 0.50×0.50\times. Native resolution is thus not automatically best: the middle scale balances small-object evidence against the image statistics the checkpoint learned, and it costs 28% less than 1.0×1.0\times. Figure 4b then fixes the source and varies the Person threshold through the whole association chain on the full validation set: HOTA is 39.118, 39.234, 39.601, 40.108, and 40.044 at thresholds 0.50 to 0.70 in steps of 0.05, against 35.115 for the half-resolution source, while LocA rises monotonically from 78.41 to 78.94. The optimum is broad: every threshold beats the half-resolution source by at least 4.0 HOTA, and 0.65 is best by 0.06 over 0.70, with the same ordering on each of the three scenes. The threshold interacts with association: a higher threshold raises raw precision but removes the low-confidence rows that bridge fragments, which is why the raw optimum on the segment (0.70) is not the chain optimum.

Constraint Ablation and Sensitivity

Table 5 varies the constraints on the final boxes. Temporal exclusion is the decisive one. At the production setting it refuses 606 candidate pairs whose similarity exceeds τ\tau, twelve times as many as reachability refuses (52), and 459 of them are in the degraded scene 21, where restyled textures make distinct people look alike. Such pairs are present in the same frame, so merging them is never right; the counts measure how often appearance alone would have done so. The reachability gate, by contrast, is a safety margin rather than a source of gain on this validation set: at τ=0.65\tau= 0.65 removing it changes HOTA by +0.03, loosening it to 5 m and 5 m/s by +0.02, and tightening it to 0.5 m and 1 m/s costs 0.82, because genuine hand-offs across long occlusions are refused (182 rejections). Without the gate, τ\tau can even be lowered to 0.55 with almost no loss (40.084), at the price of a visibly lower AssPr (60.9) and fewer identities than ground truth would warrant in the degraded scene. We keep the production gate because it bounds the damage of a wrong merge at no measurable cost, and because the test scenes (20 cameras) are larger than any validation scene. Prototype eligibility is immaterial (40.097 with every track eligible), and average linkage loses 0.50: its mean-pairwise criterion is more conservative, leaving 558 identities after 483 merges.

Variantτ\tauHOTAAssAAssReAssPrIDsmergesrefused
none (after causal stitch)–31.00923.52325.08672.2811,0410–
temporal exclusion only0.5540.08439.16944.64160.8714146271,077 / –
temporal exclusion only0.6540.13439.26944.31663.103498543618 / –
temporal exclusion only0.7539.42137.90542.29266.238628413234 / –
temporal exclusion only0.8537.55934.45538.05968.67278625586 / –
+ reachability, tight (0.5 m, 1 m/s)0.6539.29237.66341.81565.586565476617 / 182
+ reachability, production (2 m, 2 m/s)0.6540.10839.21944.04763.849516525606 / 52
+ reachability, loose (5 m, 5 m/s)0.6540.12839.25744.16563.472505536635 / 26
production, every track eligible0.6540.09739.19944.02363.834492549621 / 55
production, average linkage0.6539.60638.25842.67664.519558483291 / 21

Table 5. Constraint ablation for global Person clustering on the final validation boxes (identities only change; exact evaluation). Every variant keeps temporal exclusion, without which the output would assign one identity to two boxes of the same frame. “Merges” and “refused” (candidate pairs above τ\tau that a constraint rejected: temporal exclusion / reachability) are summed over the three scenes.

Figure 5 shows the operating region. Across the 36 settings on the final boxes with τ\tau from 0.50 to 0.75, gates of 1, 2, and 3 m, and a minimum of 15 or 30 rows, Person HOTA stays within 39.36–40.22; on the half-resolution boxes, 24 settings of τ\tau, gate, and speed stay within 34.89–35.15. The production setting (τ=0.65\tau= 0.65, 2 m, 2 m/s, 30 rows) was chosen on the half-resolution boxes and transferred to the 0.75×\times boxes without retuning; retuning there would have gained 0.11. The plateau towards lower τ\tau is what temporal exclusion and the trimmed prototypes buy: the harmful merges that a lower threshold admits are overwhelmingly between people who are visible at the same time, and the constraint removes them for free.

Figure 5. Sensitivity of constrained global clustering. (a) Full-validation Person HOTA against the similarity threshold τ\tau for three reachability base gates, on the final 0.75×\times boxes (circles) and on the previous half-resolution boxes (squares). (b) Predicted Person identities against τ\tau at the production gate; the dotted lines mark the input identity counts and the dashed line the 144 ground-truth identities.

Rare Classes

If the server averages HOTA over classes, the four rare classes that appear only in test scenes 25–27 weigh as much as Person, and they are heavily fragmented there (our raw detector output has 74 AgilityDigit identities in scene 25 for at most seven simultaneous objects). Table 6 reports the evidence behind the class-aware gates: every class gains 2–14 HOTA from motion-only stitching on held-out and training scenes, the Transporter gate was widened to 3 m because scene 10 showed it beneficial and scene 22 was indifferent, and a per-instance class majority vote across the humanoid classes was tested and found neutral. On the test submission the rare-class stage reduces, for example, scene-25 FourierGR1T2 identities from 40 to 14 and AgilityDigit from 74 to 17.

ClassGate (base, speed)Validation: HOTAValidation: IDs (GT)Train 010+015: HOTA
NovaCarter1 m, 2 m/s16.978 → 28.03810 → 3 (2)54.059 → 68.506
Transporter3 m, 3 m/s16.979 → 25.25114 → 4 (2)48.210 → 55.859
FourierGR1T21 m, 2 m/sno annotations–52.657 → 58.812
AgilityDigit1 m, 2 m/sno annotations–60.829 → 63.115
Forklift3 m, 3 m/s46.627 → 49.655124 → 25 (19)–
PalletTruck1 m, 2 m/s, 4 m cap58.398 → 60.93633 → 22 (20)–

Table 6. Long-gap stitching for rigid classes. Validation columns use all validation scenes (NovaCarter and Transporter occur only in scene 22). Training scenes 10 and 15 were run with the production detector, which was trained on the training split, so their absolute values are optimistic; the gates were fixed from this evidence and applied unchanged to the test submission. Values are before → after stitching; ground-truth identity counts in parentheses.

Negative Results

Table 7 lists what was tried and rejected, with the exact validation outcome. Three lessons shaped the final design. Refinements that try to extract more from appearance, such as adaptive thresholds, average linkage, and junction vetoes, all lose, whereas the one constraint that matters is geometric, exact, and free. Detection cannot be improved by adding rows: every form of union, fusion, or low-confidence recovery lost to replacing the source for one class, and the union of both scales is worse than either alone. And the detector itself resists improvement by fine-tuning: the last two rows of Table 7 train the released checkpoint with the TAO training stack at a batch-matched learning rate (the released rate collapses the model, as GaugeAlign reports [27]), and Person is lower at every checkpoint of every recipe, on held-out scenes and even on the frames it was trained on, while the large classes adapt readily to the 0.75×0.75 \times scale. The released checkpoint scores 69 Person HOTA on a training scene against 39–45 on validation, so the training split is in its training data and re-training on it adds no information; the small, distant people that account for most misses are not recovered by this model at this capacity.

AlternativeOutcome
Association
Average linkage instead of centroid linkage34.41–34.58 vs 34.79–35.15 across the same thresholds
Per-scene adaptive τ\tau from “certain-negative” pairs (simultaneous tracks ≥2\ge2 m apart)31.45–33.88 vs 34.80; scene 22 holds look-alike people
End-to-start ReID veto at every internal hand-off (0.2–0.6)monotone loss, 35.11 to 32.77; endpoint crops are the most occluded
Retuning τ\tau, gate, and minimum rows on the 0.75×\times boxesbest 40.222 vs 40.108 transferred
Linear gap interpolation inside merged tracksonly 8–11% of missed frames lie in gaps ≤120\le120 frames
Track-median box size or ground-plane smoothingneutral or worse; Person length varies with pose
Causal-stitcher gap beyond 240 frames, reverse-motion gatesno gain; reverse gates lose 1–2 HOTA (server rows 4–5)
Detection
Union of 0.50×\times and 0.75×\times outputs (scaled rows added where no 0.50×\times row lies within 0.5 m)Person 38.89 vs 39.39 (0.50×\times) and 45.32 (0.75×\times); Forklift 51.11 vs 57.78; PalletTruck 64.52 vs 75.42 (segment)
Track-level fusion of 0.50×\times Person tracks not covered by 0.75×\times45.02–45.28 vs 45.32 (segment), 9 variants
Input scales 0.625×\times, 0.875×\times, 1.0×\times for Person43.48, 44.79, 44.89 vs 45.32 (segment)
Instance-bank confidence decay 0.90 / 0.98 at 0.75×\times45.39 / 44.71 vs 45.39 (segment, τ\tau=0.65)
Lower Forklift / PalletTruck thresholds (0.6)+0.14 / 0.00 HOTA; production kept
Transporter, FourierGR1T2, AgilityDigit from 0.75×\times (variant D-prime)val Transporter +0.95 but the 0.75×\times pass loses all robot detections in test scene 26; built, not scored
Per-instance class majority vote among Person and humanoids±0.2 HOTA
2D confirmation of low-score 3D candidates (0.05–0.6)≈ 50% false positives after confirmation; covers 14% of misses
Broad recovery of low-confidence 3D rowslower HOTA (production system)
ResNet-101 Sparse4D variant41.43 HOTA on the server
Fine-tuning the checkpoint on five training scenes at 0.75×\times (LR 5e-6 / 2e-5, with and without the background-query loss, with and without a frozen backbone; 1,000 iterations)Person 40.0–43.8 vs 45.4 at every checkpoint (segment, raw); LocA −1 to −3.5; only Forklift / PalletTruck adapt to 0.75×\times
Fine-tuning on validation scenes 20–21, tested on scene 22Person 44.6–45.3 vs 46.9 on scene 22; 43.4–44.4 vs 45.4 in-sample on scene 20

Table 7. Alternatives that did not improve exact validation HOTA (Person unless stated). “Segment” is validation scene 20, frames 0–1,799; “raw” means detector output before association.

Limitations and Reproducibility

Global clustering uses future frames, so the system is offline and ineligible for the online bonus; a causal variant would need to forgo the long-gap merges that account for most of the gain. The leaderboard result is a hidden-test measurement with no per-class or per-scene breakdown, and local validation has three sequences, two of which are the same world. Hyperparameters are fixed class-specific gates; they sit on broad plateaus (Figure 5) but a learned merge probability with the same hard constraints would transfer more gracefully. The largest remaining headroom is detection: the leading entry’s DetA is now 6.4 points higher than ours, and our own Person recall on validation is 45.5% at 69.1% precision. Centroid linkage can propagate a wrong merge; the constraints bound but do not eliminate this, and we observed AssPr fall as AssRe rises (Table 4). Finally, the paired clean and restyled scenes (Table 1) share a world; we treat every test scene independently, and did not attempt cross-scene fusion.

SplitSceneCamerasFramesPersonForkliftPalletTruckNovaCarterTransporter
train000–0194–199,000ReID training (621 identities); gates (010, 015)
val020169,00060910––
val021 (degraded twin)169,00060910––
val02249,270241–22
test023 / 024 (twins)20 / 209,000hidden
test025109,000hidden; all rare classes present
test026 / 0274 / 71,800hidden

Table 1. Scenes used in this paper. Ground-truth identity counts are given for the validation classes that have annotations; FourierGR1T2 and AgilityDigit occur only in the test split (and in training scenes). Validation scene 021 is a generatively restyled rendering of scene 020 with identical ground truth; test scenes 023/024 form the same kind of pair.

Reproducibility

scored submission: agent2_globalreid_D_fd075p065_C2/track1.zip

SHA-256 92581abca47eb8108c60fec52452885d441e10bff2bc43acc36aac4bf3b2e7d4

The archive contains one file, track1.txt, with 1,264,782 rows (832,033 Person, 143,932 Forklift, 20,132 NovaCarter, 46,827 Transporter, 24,890 FourierGR1T2, 41,520 AgilityDigit, 155,448 PalletTruck), no duplicate (scene, class, identity, frame) key, and no row dropped by the strict packager. Its non-Person rows are identical as a multiset to the previous champion’s, and its non-Person identity partition is identical to that of row 6 in Table 3; these properties were asserted at build time. Every stage is deterministic given its inputs; we keep the input hashes, configuration scripts, merge logs, per-stage outputs, and the cached-evaluator JSON files for every number in this paper. The detector is the public NVIDIA checkpoint; the ReID encoder is trained locally in 15 minutes from the released annotations. Public repository details and a license table will accompany the final release.

Conclusion

We presented a detector-preserving system for calibrated multi-camera 3D tracking that reaches 62.3947 3D HOTA on the AI City Challenge 2026 Track 1 evaluation server, first on the general leaderboard snapshot of 30 October 2026; a competitor’s later detector update has since moved the lead to 65.84, which marks the limit of the frozen-detector route: fine-tuning the released checkpoint did not raise Person detection under any recipe we tried, so the next step is a different detector, not a better use of this one. The frozen Sparse4D detector was not replaced or fine-tuned; the gain came from choosing the input scale per class and from repairing the topology of the detector’s tracks with appearance under hard physical constraints. Two results seem general. Replacing a class’s source at a second scale is safer than any form of multi-scale union for a query-based detector with learned size priors. And offline ReID clustering of 3D tracks becomes reliable once simultaneous presence is enforced as a hard veto, because it supplies exactly the negative evidence that appearance lacks among similarly dressed people; a metric reachability gate then bounds the remaining risk at no cost. For calibrated 3D tracking, repairing a strong detector’s trajectories can be as consequential as replacing the detector.

References

  1. [1]Aharon, N., Orfaig, R., Bobrovsky, B.Z.: BoT-SORT: Robust associations multi-pedestrian tracking. arXiv preprint (2022), https://arxiv.org/abs/2206.14651
  2. [2]Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online and realtime tracking. In: IEEE International Conference on Image Processing (ICIP). pp. 3464–3468 (2016)DOI
  3. [3]Cao, J., Pang, J., Weng, X., Khairdoar, R., Kitani, K.: Observation-centric SORT: Rethinking SORT for robust multi-object tracking. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)arxiv.org/abs/2203.14360
  4. [4]Du, Y., Zhao, Z., Song, Y., Zhao, Y., Su, F., Gong, T., Meng, H.: StrongSORT: Make DeepSORT great again. IEEE Transactions on Multimedia 25, 8725–8737 (2023)arxiv.org/abs/2202.13514
  5. [5]Gao, R., Qi, J., Wang, L.: Multiple object tracking as ID prediction. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)arxiv.org/abs/2403.16848
  6. [6]He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016)arxiv.org/abs/1512.03385
  7. [7]Hermans, A., Beyer, L., Leibe, B.: In defense of the triplet loss for person re-identification. arXiv preprint (2017), https://arxiv.org/abs/1703.07737
  8. [8]Hou, Y., Zheng, L., Gould, S.: Multiview detection with feature perspective transformation. In: European Conference on Computer Vision (ECCV) (2020)arxiv.org/abs/2007.07247
  9. [9]Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Research Logistics Quarterly 2(1–2), 83–97 (1955)
  10. [10]Li, Z., Wang, W., Li, H., Xie, E., Sima, C., Lu, T., Qiao, Y., Dai, J.: BEVFormer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In: European Conference on Computer Vision (ECCV) (2022)arxiv.org/abs/2203.17270
  11. [11]Lin, X., Lin, T., Pei, Z., Huang, L., Su, Z.: Sparse4D v2: Recurrent temporal fusion with sparse model. arXiv preprint (2023), https://arxiv.org/abs/2305.14018
  12. [12]Lin, X., Pei, Z., Lin, T., Huang, L., Su, Z.: Sparse4D v3: Advancing end-to-end 3D detection and tracking. arXiv preprint (2023), https://arxiv.org/abs/2311.11722
  13. [13]Lu, J., Yang, X., Ye, J., Zhang, Y., Zou, Z., Zhang, W., Tan, X.: CityTrack: Improving city-scale multi-camera multi-target tracking by location-aware tracking and box-grained matching. arXiv preprint (2023), https://arxiv.org/abs/2307.02753
  14. [14]Luiten, J., Hoffhues, A.: TrackEval. https://github.com/JonathonLuiten/TrackEval (2020)
  15. [15]Luiten, J., Osep, A., Dendorfer, P., Torr, P., Geiger, A., Leal-Taixé, L., Leibe, B.: HOTA: A higher order metric for evaluating multi-object tracking. International Journal of Computer Vision 129, 548–578 (2021)DOI
  16. [16]Luo, H., Gu, Y., Liao, X., Lai, S., Jiang, W.: Bag of tricks and a strong baseline for deep person re-identification. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)arxiv.org/abs/1903.07071
  17. [17]Naeem, A., Katwal, A., Dey, A., Khan, N., Hoque, M.T.: Geometry beats estimated depth: RGB-only multi-camera 3D tracking under Sim2Real. arXiv preprint (2026), https://arxiv.org/abs/2608.07579
  18. [18]Shrestha, P., Nakayama, H., Scott, A.: Online multi-camera 3D tracking via ID prediction over recurrent sparse queries. arXiv preprint (2026), https://arxiv.org/abs/2609.18363
  19. [19]Tang, Z., Wang, S., Anastasiu, D.C., Chang, M.C., Sharma, A., et al.: The 9th AI City Challenge. arXiv preprint (2025), https://arxiv.org/abs/2508.13564
  20. [20]Tang, Z., Wang, S., Anastasiu, D.C., Chang, M.C., Sharma, A., et al.: The 10th AI City Challenge. arXiv preprint (2026), https://arxiv.org/abs/2608.17044
  21. [21]Teepe, T., Wolters, P., Gilg, J., Herzog, F., Rigoll, G.: EarlyBird: Early-fusion for multi-view tracking in the bird’s eye view. In: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops (2024)arxiv.org/abs/2310.13350
  22. [22]Tran, D.N.N., Huynh, N.D.M., Le, C.Q., Nguyen, H.K., Pham, L.H., et al.: Syn2RealTrack: Bridging the gap between synthetic and real-world datasets for online multi-view multi-target tracking. arXiv preprint (2026), https://arxiv.org/abs/2608.24130, accepted at the AI City Challenge Workshop, ECCV 2026
  23. [23]Wang, Y., Pusegaonkar, S., Wang, Y., Li, A., Kumar, V., Sethi, C., et al.: A unified 3D object perception framework for real-time outside-in multi-camera systems. arXiv preprint (2026), https://arxiv.org/abs/2601.10819
  24. [24]Wang, Y., Guizilini, V., Zhang, T., Wang, Y., Zhao, H., Solomon, J.: DETR3D: 3D object detection from multi-view images via 3D-to-2D queries. In: Conference on Robot Learning (CoRL) (2021)arxiv.org/abs/2110.06922
  25. [25]Wang, Y., Wang, Y., Li, A., Wang, S., Pusegaonkar, S.S., Liang, H., Li, J., Jiang, S., Yuan, J., Li, S., Dai, T., Chen, Z., Anastasiu, D., Tang, Z., Biswas, S., Wu, X.: Physical AI smart spaces: A large-scale benchmark for multi-camera 3D perception in smart spaces. In: Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track (2026), openReview id SdCUsRcgQIarxiv.org/abs/2610.02580
  26. [26]Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric. In: IEEE International Conference on Image Processing (ICIP). pp. 3645–3649 (2017)arxiv.org/abs/1703.07402
  27. [27]Wu, X., Fang, H., Xi, X., Tang, Y., Xi, D., Bai, X., Liang, D.: Aligning learned spatial priors: Gauge canonicalization for multi-camera 3D perception under sim-to-real transfer. In: European Conference on Computer Vision (ECCV) Workshops (2026), code: https://github.com/HyperbolicCurve/GaugeAlign-AICity2026
  28. [28]Zeng, F., Dong, B., Zhang, Y., Wang, T., Zhang, X., Wei, Y.: MOTR: End-to-end multiple-object tracking with transformer. In: European Conference on Computer Vision (ECCV) (2022)arxiv.org/abs/2105.03247
  29. [29]Zhang, T., Chen, X., Wang, Y., Wang, Y., Zhao, H.: MUTR3D: A multi-camera tracking framework via 3D-to-2D queries. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2022)arxiv.org/abs/2205.00613
  30. [30]Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., Wang, X.: ByteTrack: Multi-object tracking by associating every detection box. In: European Conference on Computer Vision (ECCV) (2022)arxiv.org/abs/2110.06864
  31. [31]Zhang, Z., Wu, J., Zhang, X., Zhang, C.: Multi-target, multi-camera tracking by hierarchical clustering: Recent progress on DukeMTMC project. arXiv preprint (2017), https://arxiv.org/abs/1712.09531
  32. [32]Zhong, Z., Zheng, L., Kang, G., Li, S., Yang, Y.: Random erasing data augmentation. In: AAAI Conference on Artificial Intelligence (2020)arxiv.org/abs/1708.04896
  33. [33]ZIOVISION: Multi-camera 3D object tracking via V-DETR and point cloud re-identification: First-place solution to the 2025 AI City Challenge track 1. https://github.com/ZIOVISION/AIC2025_Track1_ZV (2025)

Paper details

vlnc:W_64sm5tvw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents