WorksPhysics and Astronomy
An Updated Meta-Analysis of LHC Exotic Physics Results
Abstract
The replication crisis has raised concern across many fields of science about whether published results are as reliable as their stated uncertainties imply. Searches for new physics at the Large Hadron Collider offer a direct test of this for high-energy physics: those that find no signal report exclusion limits together with the limits expected in the absence of a signal and their and bands, which readers take to contain the observed limits with frequencies of 68.3% and 95.4%. Following a preliminary study in 2019, we test this reading for the ATLAS and CMS searches published between 2020 and 2025, 55 publications with nearly 400 exclusion plots, and reanalyze the publications of the earlier study. The limits are read directly from the vector graphics of the figures, and the statistical uncertainties are estimated with a bootstrap over publications. Averaging over the plots with equal weights, the CMS results agree with that reading, with and of the observed points within the and bands. For ATLAS, of the points lie within the band, compatible with it, while lie within the band, more often than it implies. Pseudo-experiments show that this can arise from conservative uncertainties, from the few events of many searches, or from expected limits computed after a fit to the data, the default of common statistical tools, which the plots do not distinguish. We recommend that publications state how their expected limits are computed.