betaFeedback

WorksMathematics

The random analyst

Abstract

Modern data-analysis pipelines increasingly involve searching among candidate analyses before reporting an estimate. We investigate the replicability of the resulting conclusions by studying independent repetitions of the entire search-and-report procedure on new samples. Can the reported estimates become increasingly similar across studies while remaining systematically displaced from the scientific target? We first show that this can happen in a simple Gaussian model. Although every prespecified candidate estimator is unbiased and consistent, searching over exponentially many candidates within each study yields a selected estimator converging to a limit different from the target, a selection-induced pseudotruth. Across independent repetitions, the reported estimates thus become replicable but systematically inaccurate, while the selected candidates differ with probability tending to one. We then study dependent Gaussian candidates, including marginal regression coefficients, and show that Gaussian width governs the displacement, which persists when squared width is of the same order as sample size. Metric entropy and generic chaining reveal that the effective complexity of the search is determined by the geometry of statistically distinguishable analyses rather than by their nominal number. Large-deviation arguments further extend the phenomenon beyond Gaussian models. Numerical agreement between independent studies therefore need not indicate either a stable analytical path or proximity to the scientific target.

Paper details

vlnc:W_7skhwp5b
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
arXiv nonexclusive license
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…