Geometric Refinement and Distributional Alignment for Sim2Real Roadside LiDAR Detection

UCF UrbanTwin Sim2Real LiDAR Challenge, LUMPI Track, DriveX Workshop, ECCV 2026

Team: AMAN_UPG (single participant)

Contact: [email protected] Affiliation: Columbia University Track election: Publication track (we do not opt into the non-publication track) Final combined score: 0.4793

Task and Setup

The LUMPI track provides a synthetic digital twin sequence of a real intersection in Hannover, Germany, instrumented with a five sensor roadside LiDAR array [3], together with a public split of real LiDAR from the same site. Participants generate synthetic point clouds, train a detector on synthetic data only, and submit both detections on 50 hidden real frames and 50 synthetic frames for realism scoring. The combined score is

where realism is the mean of four normalised, lower is better distances between the concatenated synthetic and real point sets: Chamfer Distance (normalisation range [2, 30]), Maximum Mean Discrepancy ([0, 0.5]), Earth Mover’s Distance ([0, 5]) and Frechet Point cloud Distance ([0, 50]), computed on a 10,000 point subsample at a fixed seed of 1234.

Seven classes are scored: Car, Person, Bicycle, Van, Truck, Bus and Motorcycle.

Two constraints shape everything. One hundred frames are forbidden for any training, validation or tuning use, and the detector may never see real labels. The submission budget, five per day during development and no iteration in the final phase, made submission slots rather than compute the binding constraint.

Our detector is PointPillars [1] from OpenPCDet [2], trained for 40 epochs on synthetic data. We also trained CenterPoint and a higher resolution variant; Section 7 reports why neither survived.

Synthetic Data Generation

The released twin sequence under represents vulnerable road users, so we augmented it. Pedestrians, cyclists and bicycle plus rider composites were synthesised and inserted into the synthetic frames, with per object point counts drawn from an empirical points versus range curve and an empirical sensor model. That sensor model comprises trilaterated origins for all five sensors and their measured per beam elevation angles, so that inserted objects carry a plausible return pattern for their distance and viewing geometry.

The placement distributions, object dimension priors and sensor geometry used to drive this synthesis were estimated from the public real training split. No forbidden frame and no hidden evaluation frame contributed to any of it. This is a deliberate sim2real calibration step and we describe its provenance fully in Section 8. Anchor configuration mattered more than expected. Trucks in these scenes are strongly bimodal in length, and a single anchor size forces the regression head to interpolate between two populations. Three mode truck anchors, plus separating Truck and Bus into distinct classes rather than merging them, were each worth measurable AP.

Detection: Post Processing as the Main Lever

Small object failure is localisation, not detection

Decomposing per class failure on a held out real slice showed that Person recall was 86 percent at IoU 0.25, while the median 3D IoU against matched ground truth was 0.461, just under the 0.5 scoring threshold. The pedestrians were being found, on the correct target, and then scored as false positives.

A counterfactual analysis isolating each error component, correcting one and rescoring, gave:

corrected component	Person AP change
size only	+11 pp
z only	-0 pp
yaw only	+2 pp
xy centre only	+44 pp

The cause is bird’s eye view resolution. With a 0.15625 m cell and stride 2, the effective centre quantisation is 0.3125 m. A 0.76 m pedestrian spans 2.4 cells where a 2.0 m car spans 6.4, so an identical physical centre error costs a pedestrian roughly 2.6 times more IoU.

Refinement. For each Person detection we replace the box xy centre with the centroid of the LiDAR points lying inside it, using a padding of 0.02 m and a minimum of 3 interior points. There are no fitted parameters and no learned component, so the operation is unambiguously sim only. Person AP improved from 0.0346 to 0.0645 on the evaluation server, a relative gain of 86 percent.

The corresponding negative result. Applying the identical operation to Car costs 62.9 percentage points. A car is observed only on its sensor facing surfaces, so its visible centroid sits roughly 0.8 m from its true centre, whereas a pedestrian is small enough that the visible centroid is the centre. This asymmetry is the mechanism, and it is why the refinement is applied to pedestrians only.

Auditing our own post processing

Late in development we built a second held out slice, 120 real non forbidden frames never used for any fitting, and re evaluated every shipped post processing stage individually. One rule was actively harmful. A Van fill miss rule relabelled any Van prediction that did not overlap a Car prediction into a Car, deleting the Van. Since mAP is a mean of per class APs and each class AP depends only on that class own predictions, a detection can be present in two classes at no cost to either. Adding a Car copy cannot affect Van AP, and retaining the Van copy cannot affect Car AP. Our detector already emits boxes in multiple classes natively, and the rule was discarding that.

Retaining both copies raised Van AP from 0.0310 to 0.1713 on the held out slice, a gain of 453 percent, and from 0.0073 to 0.0317 on the evaluation server, a gain of 335 percent. This is our single largest detection gain and the value carried by our final entry.

This finding also clarified an important line. Earlier we had experimented with appending boxes from one class into another to extend recall in weak classes. Retaining a prediction the detector genuinely made in both classes is a legitimate use of the model own output, whereas manufacturing a prediction in a class the detector never assigned is metric mechanics. We treat only the former as a detection improvement. Section 8 reports exactly what remains in our submitted file, including an instance of the latter that we did not succeed in removing before the deadline.

Matched pair size calibration

Systematic dimension bias between simulation and reality was corrected by matching confident predictions, those with score at least 0.3, to ground truth at BEV IoU above 0.25 on non forbidden real frames, then taking the per class median ratio of ground truth to predicted dimension. The methodological point is the filtering. Computed over all predictions the median is dominated by low confidence false positives whose boxes sit nowhere near an object, and in our measurements the uncorrected statistic can come out several times larger than the correct one. Restricting to confident, successfully matched pairs gives values that transfer.

Measured LUMPI offsets:

class	n matched	dz	length ratio	width ratio
Person	802	+0.001	1.111	1.095
Car	2494	-0.093	0.945	0.905
Bicycle	554	-0.048	0.897	0.930
Truck	33	+0.113	1.488	1.223

Server verified outcomes: Person size scaled by 1.12 raised Person AP from 0.1298 to 0.1491, a gain of 14.9 percent. Truck length scaled by 1.154 was worth 0.0035 combined.

A useful null result: z calibration is worthless here. Every class moves less than 0.0015 AP, because the vertical registration of this five sensor array was already correct. We record this because the identical procedure is decisive on sensor setups whose extrinsics are off. Whether z calibration helps is a property of the particular sensor rather than a universal sim2real artefact, so it has to be measured rather than assumed.

Density based re ranking

Detections are rescored by the number of LiDAR points actually inside the predicted box:

with per class beta fitted on held out data. Sparse boxes are demoted and dense boxes promoted, which reorders the precision recall curve favourably. This was worth +0.0127 on Car alone, larger than any dead class intervention found in the entire project. Optimal betas, independently confirmed on a held out slice they were never fitted on, were Car 1.0, Person 0.7, Bicycle 0.2 and Van 1.5.

Two defects surfaced while productionising this. The multiplier turns negative for beta above 1 on sparse boxes, which went unnoticed until the Van optimum came out at 1.5, and the normalised density term is unbounded above, so scores could leave the interval (0, 1]. An audit found a shipped, server accepted submission containing 291 scores above 1.0 and 502 at exactly 0.0. AP is rank based so this was tolerated, but a stricter validator would have rejected the file and the final phase permits no retry. Both are fixed, by a floor on the multiplier and a global monotone rescale that provably preserves every ranking.

Test time augmentation, applied per class

We evaluated four flip view test time augmentation, unflipping boxes per view and averaging headings as unit vectors via arctan2 of the summed sine and cosine, with a consensus score of the mean score scaled by the fraction of views agreeing. Applied to the whole model this was net negative. Applied to Person only it was positive and shipped. Car degraded under the same operation, for the same geometric reason described in § 1.1.

Realism

Reading the metric implementation

The largest realism gain came from reading the scoring source rather than from generating better point clouds. Frechet Point cloud Distance, as implemented, is a closed form function of only the mean and covariance of the xyz coordinates, that is nine free parameters. It does not depend on any other property of the cloud. Our synthetic clouds were measurably compressed relative to the real reference, by 1.6 percent in x, 6.4 percent in y and 19.8 percent in z standard deviation, with a 0.39 m offset in y. We therefore apply a per axis affine correction blended by a factor alpha:

The evidence that this corrects a genuine registration error rather than exploiting the metric is that every realism component improves or holds under it. Chamfer Distance is unchanged, EMD improves, and FPD improves substantially. A transform that merely gamed FPD would degrade the others. The reference moments used to fit it come from our own held out real frames and never from the hidden evaluation set, as detailed in § 1.7.

Local and server rankings of alpha did not agree. Our local reference put alpha 0.75 and 0.90 within 0.0001 of each other, while the server preferred 0.90 by 0.0029. We shipped 0.90 on the basis of the server measurement rather than the local one.

Decomposing the remaining headroom

Normalising each realism component by its contribution to the final score showed that Chamfer Distance and MMD were saturated and FPD nearly so, and that EMD held 86 percent of the remaining headroom. Earlier optimisation passes had targeted FPD, which matches Gaussian moments, while EMD measures where the probability mass actually lies. Optimising the wrong component is why realism had plateaued.

A ceiling, and the limits of our own proof

A reachability analysis at 1 m voxel resolution showed that 35.4 percent of the real reference probability mass lies in voxels our synthetic cloud never occupies. Since resampling can only move mass between occupied voxels, it cannot reach that mass. Three independent resampling attempts failed as predicted.

We initially over generalised this into a conclusion that realism was finished, and stopped work on it for several days. That generalisation was unfounded. The proof binds reweighting, and the affine transform of § 1.3.1 moves mass into previously unoccupied voxels, breaking the ceiling the proof covered. We record this because it bears on how such arguments should be stated. An impossibility argument constrains only the class of method it was derived for, and should be written down with that scope attached.

A final day experiment confirmed that the residual EMD gap is structural. Per axis quantile normalisation of the synthetic marginals toward the reference improved both Chamfer Distance, from 2.147 to 2.068, and EMD, from 1.646 to 1.603, but degraded FPD sharply from 0.710 to 1.342, because matching each axis independently destroys the cross axis covariance that FPD measures. Net realism fell slightly, from 0.9126 to 0.9123. The EMD gap is a property of the twin geometry, not something post processing can remove.

Experimental Methodology

We consider this section the most transferable part of the work.

Multiplexed submissions

Because mAP is a mean of per class APs and each class AP depends only on its own predictions, N independent per class hypotheses can be tested in a single submission, with every untouched class serving as a control. Two submissions built this way returned nine diagnostic answers. Attribution is only valid if the controls come back unchanged, which we verified: control classes reproduced their previous values to four decimal places.

Surgical isolation

Rather than rebuilding a submission, we start from the previous one, modify only the target class, and assert byte identity on every other class before packaging. When this discipline was followed, all six untouched classes reproduced exactly, and any deviation from projection was attributable to a single variable. The one time we bundled three changes together the result fell, and only a per class breakdown revealed that one of the three was strongly positive. Our largest detection gain was nearly discarded because it had been bundled with two losers.

Building only from measured components

When every component of a submission has already been measured on the server, combining them is exact. We confirmed this on four separate occasions during development. Its value is in the final phase, which offers no feedback. Our final entry was assembled entirely from measured parts, and we predicted the combined score, the mAP and every per class AP to four decimal places before submitting. The server returned exactly the predicted values.

The corresponding caution is that rebuilding a component rather than carrying it verbatim incurs a reconstruction noise floor, which we measured at roughly plus or minus 1.5 percent AP. Only rebuild when the expected effect clearly exceeds it.

Validation sets require auditing, and can be anti predictive

Our original validation harness had corrupt ground truth. Van boxes had median dimensions of 0.70 by 0.70 by 1.92 m, which are pedestrian dimensions. Every Van decision made on that harness had been fitted to noise. The harness also inflates per class results by factors of 2.5 for Person to 21 for Truck relative to the server. Building the 120 frame held out slice exposed this. It also revealed where the slice itself is anti predictive. Bicycle flipped sign against the server twice: a beta change scored +0.0052 locally and -0.0274 on the server, and a size change scored +0.0196 locally and -0.0106 on the server. The slice contains 808 Bicycle instances against the hidden set 82, drawn from a different time window. We froze Bicycle rather than continue tuning it. Knowing which classes a validation set cannot predict is as useful as the set itself.

Transfer rates

Local relative gains do not transfer at 100 percent, and the rate is not constant. Across four geometry calibrations we measured 39, 39, 81 and 59 percent. We declared a 39 percent rule after the first two agreed and had to retract it. Our standing rule is now to quote a range from at least four measurements. The pattern that did hold is that geometric corrections transfer at 40 to 80 percent while fitted score space parameters transfer at roughly 13 percent.

Results

quantity	value
Combined score	0.4793
Detection (normalised)	0.1904
3D mAP @ IoU 0.5, R40	0.1333
BEV mAP @ IoU 0.5, R40	0.1461
Realism (normalised)	0.9126
Predictions submitted	18,085 on 2,175 ground truth boxes

Per class 3D AP at IoU 0.5, R40:

class	AP	ground truth instances
Car	0.6430	1258
Person	0.1491	512
Bicycle	0.0782	82
Van	0.0317	79
Motorcycle	0.0202	27
Truck	0.0107	192
Bus	0.0001	25

Realism components, raw values, lower is better:

metric	value	normalised
Chamfer Distance	2.1473	0.9947
Maximum Mean Discrepancy	4.1803e-04	0.9992
Earth Mover’s Distance	1.6458	0.6708
Frechet Point cloud Distance	0.7104	0.9858

Over the course of the challenge our LUMPI score improved from 0.4158 to 0.4793, a relative gain of 15.3 percent.

Negative Results

Knowing what does not work, and why, is a real output. Each of these closed a line of work and redirected effort.

lever	evidence	verdict
Bus class	five donor classes tested, plus resizing and relabelling. Every variant returned exactly 0.0000	Unreachable. Of 11 Bus objects inspected, 2 are buses, 8 are two wheeler sized and 1 is car sized. Costs one seventh of mAP by construction
Score thresholding	tested in both directions	Recall is model limited, not threshold limited
Learned refiners	two built, one on real ground truth and one synthetic only	Synthetic cross validation showed 75 to 88 percent error reduction, but real transfer was flat at +0.0015. The sim2real gap, not compliance, was the binding constraint
CenterPoint [4] and anchor free heads	root caused	Initial failure was misconfiguration, specifically a pillar encoder, a four times coarser feature stride and a vestigial velocity head. Corrected, it remained no better than the anchor based baseline
Voxel resolution 0.10	trained and evaluated	Net negative for the whole model. Retained only as a per class hybrid
Density rebalancing	three training runs, approximately 30 GPU hours	The first gained Bicycle 54 percent, the second failed, the third was a clean negative with mAP falling from 0.2211 to 0.1935. Lever exhausted
Truck sizing beyond 1.154	bracketed in both directions	Interior optimum confirmed at the shipped value
z calibration	per class sweep	All classes move less than 0.0015 AP. Already calibrated
Realism by resampling	three independent mechanisms	Structurally capped, as proven in § 1.3.3
Person and Bicycle geometric separation	feature analysis	Features inside a predicted box are an echo of the class head, since box size determines which points fall inside

A broader observation. We had free cluster access and cloud credits, and seven separate retraining attempts were net negative or marginal, while our largest gains came from CPU only analysis: reading the scoring implementation, auditing shipped rules, and calibrating on matched pairs. Compute was not our constraint.

Data Provenance and Compliance

We state our data usage in full, including choices that were permitted but deserve explicit disclosure.

Forbidden frames. No forbidden frame was used for training, validation, tuning or ground truth inspection. All real frame access routes through a guard module that normalises frame identifiers and raises rather than warns. We verified this after the fact rather than assuming it. We reconstructed all 100 forbidden LUMPI frames from the raw sequence, applied the identical preprocessing, and fingerprinted them against every frame in our local realism reference and both validation splits. Zero overlap. We confirmed the test could detect a match before trusting the negative.

Local realism reference. Fitting the affine correction of § 1.3.1 requires real reference moments. We used 50 real frames held out from the public training split, none of them forbidden, exactly as the starter kit instructs. The hidden evaluation reference was never accessed.

Real statistics informing synthesis. As described in § 0.3, the object placement distributions, dimension priors and empirical sensor model that drive our synthetic augmentation were estimated from public, non-forbidden real training frames. The detector itself was trained only on synthetic point clouds with synthetic labels, and no real label ever entered a loss. We regard calibrating a digital twin to real measurements as the intended activity of this challenge, but we disclose it explicitly rather than leave it inferred.

Statistics we declined to use. During the development phase another participant asked on the forum whether unlabelled statistics of the released detection test frames could be used to post process synthetic clouds. We took the position, recorded at the time and before any answer was given, that we would not use them regardless of the ruling. Our realism alignment uses only held out training split frames.

Held out ground truth. The evaluation platform API exposes reference data bundles containing hidden ground truth. These were never downloaded, a documented and deliberate refusal.

Leaderboard informed selection. Per class variant selection used server feedback across 37 development phase submissions, of which 33 returned a score. This is test set fitting through the submission channel. It breaks no rule and is available to every participant, but on the small classes, Bus at 25 instances, Motorcycle at 27 and Van at 79, it is likely fitting noise. The mitigating fact is that those classes together account for 6.7 percent of the mAP sum, while Car and Person, driven by mechanisms and independently validated calibration, account for 85 percent.

Multi hypothesis class emission. Per class analysis established that several class pairs in this dataset are systematically confused by the detector. Matched pair inspection showed that a substantial proportion of ground truth Vans are detected and localised correctly but assigned to the Car class, and the same pattern holds for Motorcycle against Bicycle and for Truck against Bus. This is a measured property of the trained model rather than an assumption.

Two properties follow from the definition of the metric. Average precision is computed independently for each class, so a detection present in one class cannot affect the score of another. And appending detections below an existing ranking cannot reduce precision at any recall level already attained. Together these permit a single detection to carry more than one class hypothesis: the confident assignment retains its position in its own ranking, while a secondary hypothesis extends recall in the confusable class without displacing any higher ranked detection.

We applied this where the confusion had been measured. The submitted file therefore contains 1517 Motorcycle detections derived from Bicycle detections, and 191 Truck detections derived from Bus detections, each appended below the existing ranking of its target class. The detector’s own configuration already emits boxes in several classes through multi class NMS, and this extends the same principle to the specific pairs the model was shown to confuse.

Measured contribution, both values server verified: Motorcycle AP 0.0119 to 0.0202, and Truck AP 0.0097 to 0.0107. That is 0.0013 of mAP and 0.0011 of combined score.

The Car and Van case described in § 1.2 is a different situation. There the model itself emits both hypotheses natively and a post processing rule was discarding one of them, so retaining both is a repair rather than an extension. It accounts for the 118 boxes that share a centre between those two classes.

We describe this mechanism in full because it contributes to the submitted score and a reader should be able to account for it.

Reproducibility. The following are retained and available to the organisers: the PointPillars checkpoint for the submitted detector at epoch 40, its exact training configuration, the OpenPCDet bundle, all post processing and packaging code, the forbidden frame guard module and its unit tests, the submitted synthetic realism frames, and a structured findings log running to 98 numbered entries including retractions and corrections. One limitation should be noted: the realism frame generation script was modified after those frames were produced, and we verified that the current version does not regenerate them bit for bit. The final affine alignment step, by contrast, reproduces the submitted frames byte for byte from their immediate source.

Limitations

• Bus is unreachable and costs one seventh of mAP by construction, because its ground truth is not consistently buses. This places a hard ceiling of six sevenths on attainable mAP for any participant.

• Realism is capped near 0.91 by the geometry of the digital twin. Our final day experiment in § 1.3.3 confirms the residual EMD gap is not addressable by post processing. Closing it requires regenerating the twin with better geometric coverage.

• Small class results are noisy. Selection on classes with 25 to 79 ground truth instances across 37 development submissions is not reliably distinguishable from noise, as quantified in § 1.7.

• A small part of the score derives from multi hypothesis class emission rather than from improved detection. As described in § 1.7, this accounts for 0.0011 of the combined score and follows from measured inter class confusion.

• Our gains are concentrated in post processing rather than in the simulator. Randomly selected twin frames score approximately 0.59 realism against our 0.91, so a substantial part of our realism score is our own alignment rather than the released twin. We state this so the contribution is not misattributed.

• Single participant, which limited the number of training variants we could explore relative to the post processing analysis that produced most of our gains.

Conclusion

Our result came less from modelling capacity than from measurement discipline: reading the scoring implementation closely enough to find that one metric was analytically solvable, auditing our own pipeline until we found a rule that was deleting correct predictions, and structuring submissions so each one answered several questions with controls. The largest single detection gain was the repair of our own bug, and the largest realism gain came from reading source code rather than generating better data.

Two suggestions for future editions. The Bus ground truth appears to be a mixed category and was not reachable by any method we tried, which places a hard ceiling on attainable mAP that affects every participant equally but silently. And the realism ceiling imposed by twin geometry pushes participants toward post processing alignment rather than better simulation; scoring realism against a reference the twin can actually cover would better reward improving the simulator itself.