betaFeedback

WorksMathematics

Pro-Active Inference

Abstract

Estimating a population mean is fundamental across scientific applications, but observing every outcome can be prohibitively expensive. Often, covariates and machine learning (ML) predictions are available for the full population, even when only a small subset of outcomes can be observed. How can we leverage this information to proactively select observations to label, update ML predictions as labels arrive, and construct statistically valid confidence intervals? We develop a general, without-replacement formulation of Pro-Active Inference (PAI), extending our earlier with-replacement approach and building on Active Statistical Inference. Rather than making fixed-order label-skip decisions, PAI proactively selects which observation to label next from the remaining covariate pool while accommodating online ML updates. We establish PAI's finite-sample unbiasedness and asymptotic validity, develop an improved finite-sample variance estimator, and extend PAI to superpopulation inference without Bonferroni corrections or αα-splitting. We characterize PAI's potential efficiency gains over Batch Active Inference in a quasi-oracle setting, highlighting the role of prediction-residual imbalance. Across eight diverse datasets, PAI achieves greater sampling efficiency than Sequential Active Inference in the vast majority of tested settings with near-nominal statistical coverage, using the same ML predictor and uncertainty score. However, comparisons with simpler baselines such as Prediction-Powered Inference and uniform sampling, as well as the closely related Prediction-Powered Active Testing, reveal when additional adaptivity helps or hurts statistical efficiency. Through interpretable, theory-motivated diagnostics and frozen-ML ablations, we investigate when and why PAI gains or loses efficiency, highlighting the initial residual-estimation problem and subsequent online learning dynamics.

Introduction

Estimating a population mean—such as disease prevalence, the proportion of households below the poverty line, or the fraction of files containing malware—is fundamental across scientific and policy applications. However, collecting ground-truth outcomes can be prohibitively expensive, often requiring human expert annotation, specialized testing, or additional surveys. In many modern settings, however, covariates and machine learning (ML) predictions are readily available for the entire population, even when ground-truth outcomes can only be obtained for a small subset. How can we fully leverage this information to decide which outcomes to collect and efficiently estimate a population mean, with valid confidence intervals?

As a concrete example, consider a policymaker seeking to estimate the proportion of households living below the poverty line following a major economic disruption. Suppose demographic information is available for many households, but determining their current economic circumstances requires costly surveys. An ML model trained on historical data could predict each household’s probability of poverty, although its predictions may be imperfect, particularly under changing economic conditions. Given a limited survey budget, the policymaker could survey households uniformly at random, obtaining a statistically valid sample mean but potentially leaving efficiency gains from ML predictions untapped. Alternatively, they could preferentially survey households whose poverty status the model is most uncertain about, update the model as new responses arrive, and adapt subsequent sampling accordingly. However, simply averaging the adaptively collected responses can introduce selection bias, while relying on ML predictions alone risks bias from prediction errors.

The challenge, therefore, is to leverage the full covariate pool and progressively acquired responses to construct efficient estimators and confidence intervals under adaptive, nonuniform sampling.

Problem setup Formally, suppose we observe covariates x1,…,xNx_1,\ldots,x_N with corresponding unobserved responses y1,…,yNy_1,\ldots,y_N, each of which may be collected at a (potentially substantial) cost. Given a labeling budget n≪Nn \ll N, we seek to sequentially label data points, so as to construct an estimator and confidence interval, by leveraging, at each timestep, the full covariate pool, previously observed responses, and a black-box ML model that can be updated as new responses arrive. Specifically, we aim to use ML predictions both to adaptively select which responses to collect and to improve the resulting estimator through regression adjustment. Our primary goal is to construct an efficient estimator and statistically valid confidence interval for the finite-population mean

θ:=1N∑i=1Nyi,(1)\theta:= \frac{1}{N}\sum_{i=1}^{N} y_i, \tag*{(1)}

treating the population as fixed and the sampling process alone as random. We also consider inference for the superpopulation mean E[Y]\mathbb{E}[Y] when the (xi,yi)(x_i,y_i) are assumed to have been drawn i.i.d. from an underlying distribution.

Positioning A natural starting point for the above problem is Active Statistical Inference (ASI) (Zrnic & Candès, 2024), which leverages ML predictions to guide label acquisition and construct statistically valid estimators and confidence intervals. In its sequential formulation (SAI), observations are processed in a fixed, predetermined order, with the ML model potentially updated as labels are acquired. Crucially, at round tt, SAI can only make a Bernoulli decision to either acquire or skip the response yty_t associated with the current observation xtx_t. Once an observation is skipped, its response cannot be acquired later. When the full covariate pool is available in advance, however, why restrict each round to a label-skip decision about the current observation, rather than proactively selecting which observation to label next from the entire remaining pool?

Motivated by this question, our earlier work [27] introduced Pro-Active Inference (PAI) in the context of LLM evaluation, combining proactive acquisition with online-updated ML predictions and regression adjustment. However, this initial formulation sampled with replacement, potentially revisiting previously labeled observations and wasting sampling opportunities. Extending PAI to sampling without replacement, however, requires more than modifying the acquisition policy: the estimator and its underlying martingale-based analysis must also be reformulated, as naive modifications can violate conditional unbiasedness and lead to severe confidence-interval miscoverage.

Our contributions In the present work, we therefore develop a general, without-replacement formulation of PAI, together with a reformulated estimator and corresponding martingale-based inferential guarantees. We establish its finite-sample unbiasedness and asymptotic validity, develop an improved finite-sample variance estimator, and extend PAI to superpopulation inference without requiring Bonferroni corrections or α\alpha-splitting. Theoretically, we characterize PAI’s potential efficiency gains over Batch Active Inference (BAI) in a quasi-oracle setting, highlighting the role of prediction-residual imbalance. Empirically, through experiments across eight diverse testbeds, we investigate PAI’s statistical efficiency and confidence-interval coverage relative to existing methods. Beyond comparing performance, we develop interpretable diagnostics of residual imbalance, the relative difficulty of direct versus prediction-adjusted estimation, and the quality of online ML updates, together with targeted ablations, to understand when and why PAI gains or loses efficiency, and, more broadly, when ML-assisted statistical inference is beneficial.1

Towards the completion of this manuscript, we also became aware of the closely related works of Ashouritaklimi et al. [4]; Hamilton et al. [15], which develop alternative approaches to adaptive sampling without replacement. While these approaches share important methodological features with PAI, they differ in their estimator constructions and uses of ML predictions. We discuss these and other related approaches in § 2 and empirically compare PAI with the former in § 6.

Related work

Prediction-powered inference A natural starting point for estimating a population mean is uniform sampling, which yields statistically valid inference without involving an ML model, but may leave substantial efficiency gains from available ML predictions untapped. Prediction-Powered Inference (PPI; Angelopoulos et al. [1]) leverages ML predictions to impute unobserved outcomes and corrects the resulting estimation error using a smaller sample of observed labels. Subsequent extensions include PPI++ (Angelopoulos et al., [3]), which adapts the strength of the prediction-based adjustment to improve statistical efficiency, and cross-PPI (Zrnic & Candès, 2024), which allows the available labeled data to be used for both training ML predictors and constructing inference through cross-fitting. While these approaches primarily consider nonadaptively acquired labels, PAI additionally uses ML predictions to guide which outcomes to acquire, potentially updating both its regression adjustment and acquisition policy as new labels are observed.

Active statistical inference Our work builds on the Active Statistical Inference (ASI) framework of Zrnic & Candès (2024), which leverages ML predictions to guide label acquisition while constructing statistically valid estimators and confidence intervals for statistical estimands. In its sequential formulation (Sequential Active Inference; SAI), observations arrive in a stream, with covariates modeled as i.i.d. draws from a superpopulation, and the method makes Bernoulli label-skip decisions according to a potentially updated ML-based acquisition policy, subject to a prespecified labeling budget. Once an observation is skipped, however, its label cannot be acquired later. When the full covariate pool is available in advance, PAI instead proactively selects which observation to label next from the remaining unlabeled pool, using the full covariate information and previously acquired labels to update its ML-based regression adjustment and acquisition policy. Related developments include optimized and robust acquisition policies for ASI (Ma & Candès, [20]; Li et al., [19]), as well as applications to human annotation and evaluation of LLM outputs (Gligorić et al., [13]; Angelopoulos et al., [2]).

In our earlier work ([27]), we introduced a with-replacement formulation of PAI for LLM evaluation, using sampling policies and regression adjustments based on an online-updated working Bayesian factor model. However, this formulation allowed previously labeled observations to be sampled again, potentially wasting part of the labeling budget on repeated queries. The present work instead samples exclusively from the remaining unlabeled observations, ensuring that each query acquires a new label and that the full budget is spent on distinct observations, while developing the corresponding estimator and inferential guarantees. Finally, Tolochinsky et al. (2026) also consider sampling without replacement in the context of best-model identification for LLMs, but use uniform sampling rather than an adaptive, ML-guided acquisition policy.

Active testing and measurement PAI is also closely connected to a growing literature on efficient statistical inference in the presence of missing outcome data, particularly when labels are acquired adaptively. Most closely related is a line of work studying importance-weighted estimation under adaptive label acquisition [11, 18, 15, 4]. With the exception of Hamilton et al. [15], these works focus primarily on estimating the risk of a fixed ML model, but their core methods readily apply to our population mean estimation task outlined in § 1. In particular, the active-testing literature [11, 18] develops unbiased estimators under adaptive sampling without replacement.

Building on this literature, one of the methods most closely related to PAI is Active Measurement [15], which studies the estimation of finite-population quantities, such as bird counts, rather than the prediction risk of a fixed ML model. AM uses online-updated ML predictions to guide acquisition and develops alternative weighting schemes and variance estimators for combining importance-weighted estimates. In particular, AM generalizes the estimation framework underlying Farquhar et al. [11]’s LURE estimator by considering alternative schemes for aggregating conditionally unbiased per-round estimates. Indeed, at the level of the per-round estimator, AM coincides with PAI when our prediction-based regression adjustment is omitted. PAI instead incorporates online-updated ML predictions into both acquisition and regression adjustment, potentially improving sampling efficiency while preserving finite-sample unbiasedness.

Another closely related method is Prediction-Powered Active Testing [4], which similarly builds on the active-testing literature, but additionally incorporates ML predictions directly into the estimator through a prediction-powered control variate. In all of their experiments, PPAT uses a fixed ML proxy for regression adjustment and a potentially updated surrogate model to guide acquisition, with a tunable control-variate coefficient. The authors further note that the control-variate proxy could also be updated as new labels are acquired, although they do not investigate this extension experimentally. In contrast to their experimental implementation, PAI directly incorporates online-updated ML predictions into its imputation and residual correction.

Like PAI, AM and PPAT construct their estimators by aggregating per-round importance-weighted estimates, which are conditionally unbiased under fixed tuning parameters. Under suitable regularity conditions, both approaches establish asymptotic Gaussian limiting distributions through martingale arguments, enabling the construction of asymptotically valid confidence intervals using corresponding variance estimators. However, PPAT aggregates its per-round estimates using nonuniform LURE-based weights [11], whereas PAI weights them equally.

Among these closely related approaches, we focus our empirical comparisons on PPAT, since its use of ML predictions in both acquisition and regression-adjusted estimation makes it a particularly direct methodological comparator to PAI.

Pro-Active Inference methodology

We first describe PAI’s adaptive sampling procedure, then develop estimators and confidence intervals for finite-population means, before extending our methodology to superpopulation inference.

Sampling setup and notation

Let I=(1,…,N)\mathcal{I} = (1,\ldots,N) index the finite population, and let Ot−1⊆IO_{t-1} \subseteq\mathcal{I} denote the indices with observed responses before round tt, with O0=∅O_0 = \varnothing. Given a labeling budget n≪Nn \ll N, at each round t=1,…,nt = 1,\ldots,n, PAI constructs an acquisition distribution (or sampling policy qtq_t) over the remaining unlabeled indices Rt−1:=I∖Ot−1R_{t-1}:=\mathcal{I}\setminus O_{t-1}, using the full covariate pool and all previously acquired responses. It then samples an index It∼qtI_t\sim q_t, observes its response yIty_{I_t}, and updates Ot:=Ot−1∪{It}O_t:=O_{t-1}\cup\{I_t\}. We denote by f^(t−1)\hat{f}^{(t-1)} the ML predictor available before round tt, initialized at f^(0)\hat{f}^{(0)} and potentially updated after each newly acquired response. Formally, let Ft−1:=σ({Is,yIs}s=1t−1)\mathcal{F}_{t-1}:=\sigma(\{I_s,y_{I_s}\}_{s=1}^{t-1}). We require f^(t−1)\hat{f}^{(t-1)} and qtq_t to be Ft−1\mathcal{F}_{t-1}-measurable: depending only on past queries/labels.

The sampling policy qtq_t may depend on both the full covariate pool and previously acquired labels, but not on the unobserved responses. A natural strategy is to assign higher sampling probabilities to observations where the ML predictor is most uncertain. For binary outcomes, writing pt(x):=f^(t−1)(x)p_t(x):=\hat{f}^{(t-1)}(x), common uncertainty scores include ut(x)=2min⁡(pt(x),1−pt(x))u_t(x)=2\min(p_t(x),1-p_t(x)) (Zrnic & Candès, 2024) and ut(x)=pt(x)(1−pt(x))u_t(x)=\sqrt{p_t(x)(1-p_t(x))}, the latter motivated by Neyman allocation. Our experiments use the former, with uniform mixing to ensure positive sampling probabilities (§ 5), although our theoretical framework supports general adaptive sampling policies satisfying the regularity conditions below.

Pro-Active Inference estimator for finite-population estimands

After adaptively collecting nn labels without replacement, we propose the following Pro-Active Inference (PAI) estimator of the finite-population mean:

θ^n:=1n∑t=1nϕt,ϕt:=1N(∑i∈Ot−1yi+∑i∉Ot−1f^(t−1)(xi))+1NyIt−f^(t−1)(xIt)qt(It).(2)\hat{\theta}_n:=\frac{1}{n}\sum_{t=1}^{n}\phi_t,\qquad \phi_t:=\frac{1}{N}\left(\sum_{i\in O_{t-1}}y_i+\sum_{i\notin O_{t-1}}\hat{f}^{(t-1)}(x_i)\right)+\frac{1}{N}\frac{y_{I_t}-\hat{f}^{(t-1)}(x_{I_t})}{q_t(I_t)}. \tag*{(2)}

Intuitively, the first term in ϕt\phi_t represents our current plug-in estimate of θ\theta—using observed labels where available and imputing ML predictions for the remaining unobserved labels—while the second term provides an importance-weighted bias correction for prediction errors. In Theorem 1, we show that θ^n\hat{\theta}_n is unbiased for θ\theta at every finite nn and derive its asymptotic distribution using a martingale central limit theorem under mild regularity conditions.

Theorem 1. Consider a triangular array of problems (indexed by mm) such that nm↑∞n_m\uparrow\infty, Nm↑∞N_m\uparrow\infty, and nm/Nm→ρ∈[0,1−ϵ]n_m/N_m\to\rho\in[0,1-\epsilon] for some ϵ>0\epsilon>0.2 For each tt, suppose that (a) f^(t−1)(⋅)\hat{f}^{(t-1)}(\cdot) and qt(⋅)q_t(\cdot) are Ft−1\mathcal{F}_{t-1}-measurable, and that (b) conditional on Ft−1\mathcal{F}_{t-1}, qt(⋅)q_t(\cdot) is a probability distribution on Rt−1:=I∖Ot−1R_{t-1}:=\mathcal{I}\setminus O_{t-1} with qt(i)≥τ/∣Rt−1∣=τ/(N−t+1)q_t(i)\ge\tau/|R_{t-1}|=\tau/(N-t+1) for all i∈Rt−1i\in R_{t-1}, for some fixed τ∈(0,1]\tau\in(0,1].

Then, θ^n\hat{\theta}_n is unbiased for θ\theta at every finite nn. Furthermore, under mild regularity conditions (variance stabilization, Lindeberg, and uniform fourth-moment control for weighted residuals; Assumptions 4–6 in Appendix § 9.§ 1), θ^n\hat{\theta}_n obeys the following limiting distribution:

n(θ^n−θ)/σ^n→dN(0,1).(3)\sqrt{n}\left(\hat{\theta}_n-\theta\right)/\hat{\sigma}_n\xrightarrow{d}\mathcal{N}(0,1). \tag*{(3)}

And, the variance estimator is given by the following, where θ^t−1:=1t−1∑s=1t−1ϕs\hat{\theta}_{t-1}:=\frac{1}{t-1}\sum_{s=1}^{t-1}\phi_s for t≥2t\ge2:

σ^n2:=A^n−B^n(4)\hat{\sigma}_n^2:=\hat{A}_n-\hat{B}_n \tag*{(4)}

References

  1. [1]Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I Jordan, and Tijana Zrnic. Prediction-Powered Inference. Science, 382(6671):669–674, 2023.
  2. [2]Anastasios N Angelopoulos, Jacob Eisenstein, Jonathan Berant, Alekh Agarwal, and Adam Fisch. Cost-optimal Active AI Model Evaluation. arXiv preprint arXiv:2506.07949, 2025.arxiv.org/abs/2506.07949
  3. [3]Anastasios N Angelopoulos, John C Duchi, and Tijana Zrnic. PPI++: Efficient Prediction-Powered Inference. The Annals of Applied Statistics, 20(3):2235–2252, 2026.arxiv.org/abs/2311.01453
  4. [4]Kianoosh Ashouritaklimi, Valentin Kilian, Daolang Huang, Tom Rainforth, and François Caron. Prediction-Powered Active Testing. arXiv preprint arXiv:2607.08347, 2026.arxiv.org/abs/2607.08347
  5. [5]Bernard Bercu, Antoine Godichon, and Bruno Portier. An Efficient Stochastic Newton Algorithm for Parameter Estimation in Logistic Regressions. SIAM Journal on Control and Optimization, 58(1):348–367, 2020.arxiv.org/abs/1904.07908
  6. [6]Isabell Bludau, Sander Willems, Wen-Feng Zeng, Maximilian T Strauss, Fynn M Hansen, Maria C Tanzer, Ozge Karayel, Brenda A Schulman, and Matthias Mann. The Structural Context of Posttranslational Modifications at a Proteome-Wide Scale. PLoS biology, 20(5):e3001636, 2022.DOI
  7. [7]Andrea Dal Pozzolo, Olivier Caelen, Reid A Johnson, and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In SSCL, pp. 159–166, 2015.
  8. [8]Thomas Diciccio and Bradley Efron. More Accurate Confidence Intervals in Exponential Families. Biometrika, 79(2):231–245, 1992.
  9. [9]Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring Adult: New Datasets for Fair Machine Learning. Advances in Neural Information Processing Systems, 34, 2021.arxiv.org/abs/2108.04884
  10. [10]Bradley Efron. Better Bootstrap Confidence Intervals. Journal of the American Statistical Association, 82(397):171–185, 1987.
  11. [11]Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On Statistical Bias in Active Learning: How and When to Fix It. arXiv preprint arXiv:2101.11665, 2021.arxiv.org/abs/2101.11665
  12. [12]Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open LLM Leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard, 2024.
  13. [13]Kristina Gligorić, Tijana Zrnic, Cinoo Lee, Emmanuel Candes, and Dan Jurafsky. Can Unconfident LLM Annotations be Used for Confident Conclusions? In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3514–3533, 2025.arxiv.org/abs/2408.15204
  14. [14]Peter Hall and Christopher C Heyde. Martingale Limit Theory and its Application. Academic Press, 2014.
  15. [15]Max Hamilton, Jinlin Lai, Wenlong Zhao, Subransu Maji, and Daniel Sheldon. Active Measurement: Efficient Estimation at Scale. Advances in Neural Information Processing Systems, 38:145769–145807, 2026.arxiv.org/abs/2507.01372
  16. [16]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. Advances in Neural Information Processing Systems, 2021.
  17. [17]Craig Kapfer, Kurt Stine, Balasubramanian Narasimhan, Christopher Mentzel, and Emmanuel Candès. Marlowe: Stanford’s GPU-based Computational Instrument, 2025. URL https://doi.org/10.5281/zenodo.14751899.
  18. [18]Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active Testing: Sample-Efficient Model Evaluation. In International Conference on Machine Learning, pp. 5753–5763. PMLR, 2021.
  19. [19]Puheng Li, Tijana Zrnic, and Emmanuel Candes. Robust Sampling for Active Statistical Inference. Advances in Neural Information Processing Systems, 38:68727–68756, 2026.arxiv.org/abs/2511.08991
  20. [20]Virginia L Ma and Emmanuel J Candès. Optimized Labeling Resource Allocation for Prediction-Assisted Inference via OPAL. arXiv preprint arXiv:2606.03211, 2026.arxiv.org/abs/2606.03211
  21. [21]Aidar Myrzakhan, Sondos Mahmoud Bsharat, and Zhiqiang Shen. Open-LLM-Leaderboard: From Multi-Choice to Open-Style Questions for LLMs Evaluation, Benchmark, and Arena. arXiv preprint arXiv:2406.07545, 2024.arxiv.org/abs/2406.07545
  22. [22]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conference on Language Modeling, 2024.arxiv.org/abs/2311.12022
  23. [23]Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2310.16049
  24. [26]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024.arxiv.org/abs/2406.01574
  25. [27]Skyler Wu, Yash Nair, and Emmanuel J Candés. Efficient Evaluation of LLM Performance with Statistical Guarantees. arXiv preprint arXiv:2601.20251, 2026.arxiv.org/abs/2601.20251
  26. [28]Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: A Benchmark for Molecular Machine Learning. Chemical Science, 9(2):513–530, 2018.arxiv.org/abs/1703.00564
  27. [29]Limin Yang, Arridhana Ciptadi, Ihar Laziuk, Ali Ahmadzadeh, and Gang Wang. BODMAS: An Open Dataset for Learning based Temporal Analysis of PE Malware. In 4th Deep Learning and Security Workshop, 2021.DOI
  28. [30]Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-Following Evaluation for Large Language Models. CoRR, 2023.arxiv.org/abs/2311.07911

Paper details

vlnc:W_jzndpr9k
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents