betaFeedback

WorksMathematics

Not All Mislabels Are Equal: Efficient Task-Aware Auditing with Confidence Sequences

Abstract

We study efficient, adaptive auditing of categorical labels in a fixed dataset, motivated by biodiversity credit programs in which expert verification of species records is costly and downstream scores (e.g., payment) are more sensitive to errors on certain labels than on others. Our proposed ATLAS framework couples anytime-valid inference with a task-aware adaptive auditing policy. Representing label corrections through a confusion matrix, we prioritize which data points to inspect next based on both label uncertainty and the influence of potential label corrections on the downstream score function. In our main case study on Kenyan biodiversity data, task-aware auditing reduces the expert review needed to estimate both the mislabel rate and the nonlinear biodiversity score while maintaining anytime-valid guarantees. We also evaluate ATLAS on public datasets for image classification and named-entity recognition. Across our experiments with linear and nonlinear scores, ATLAS generally reaches the same precision as competing anytime-valid methods with substantially fewer audits. When distribution-free guarantees are prohibitively costly, we develop a model-based Bayesian alternative that can further reduce the number of audits by relying on modeling assumptions. Our biodiversity case study highlights a gap between statistical validity and practical feasibility: generic methods such as Serfling's bound and existing confidence-sequence procedures are rigorous but can require prohibitively many expert reviews in practice. Together, these results show task-aware auditing can significantly reduce the number of expert audits needed while providing finite-sample, anytime-valid guarantees

Introduction

Large labeled datasets are routinely used in consequential inference tasks, from model evaluation to scientific analysis. However, the labels in these datasets are not always correct, with annotation errors documented in settings ranging from healthcare ([38] (Rädsch et al. 2023)) to remote sensing ([27]). Even widely used machine-learning benchmarks contain nontrivial label noise, with one study estimating an average error rate of 3.3% ([26]). Especially in consequential settings, practitioners may want rigorous uncertainty quantification for downstream quantities computed from the true labels, but doing so requires verifying at least some of the labels. When expert verification of an entire dataset is impractical, two natural questions arise: How many data points must be audited to certify a downstream quantity with a desired level of confidence? Which records should be audited to reduce that uncertainty as efficiently as possible?

When audits are costly, confidence sequences are useful as they provide confidence intervals that are simultaneously valid over all sample sizes (i.e., anytime-valid), allowing the auditor to inspect data sequentially and stop adaptively without invalidating the coverage guarantee. Recent work develops such guarantees for sampling without replacement ([43, 32]) to certify finite-population quantities, e.g., population mean, but these methods were not designed for the precision and affordability that audits need in practice: on our data of 33,403 labels, the prior–posterior-ratio confidence sequence of [43] needs a median of 4,021 audits to certify the mislabel rate to ±2\pm2 percentage points at 95.5% confidence, against 603 for our method, over 100 paired paths. Adaptive sampling without replacement, importance weighting, and control variates already support efficient weighted-average audits [37]. Our question is how to make these tools effective for categorical corrections and their downstream scores. We use Bayesian predictions of label corrections to decide where expert review would tell us the most about the downstream score. This rule approximates the variance-minimizing audit allocation, and our confidence sequence remains valid whenever the auditor stops, even if the predictions are inaccurate. We also extend the audit to structured nonlinear scores. For example, we use a linear-envelope representation of entropy to certify the Hill–Shannon biodiversity score. Empirically, our method typically reached the target precision with substantially fewer audits than competing methods (Table 1 and Figure 3).

MethodSampling strategyGuaranteeKenyaCIFAR-10NReImageNetCleanCoNLL
ATLAS-PAS (ours)Estimated variance-minimizingAnytime-valid603 [555, 636]790 [725, 881]1,394 [1,161, 1,700]281 [242, 334]
ATLAS-SRS (ours)UniformAnytime-valid814 [718, 866]980 [914, 1,056]1,539 [1,287, 1,764]293 [225, 368]
Prop-MS ((Shekhar et al.))Proportional to predicted mislabel probabilityAnytime-valid>5,000 [> 5k, > 5k]>5,000 [4,530, > 5k]1,671 [1,417, 1,994]1,575 [1,151, 1,966]
Prop-M + CV ((Shekhar et al.))UniformAnytime-valid2,016 [1,552, 2,233]2,404 [2,107, 2,886]2,631 [2,038, 3,495]233 [100, > 5k]
PPR ((Waudby-Smith & Ramdas))UniformAnytime-valid4,021 [3,572, 4,237]4,035 [3,541, 4,310]2,216 [2,021, 2,338]435 [340, 529]
Wald1UniformAsymptotic, fixed nn1,4811,434748119
SerflingUniformFinite-sample, fixed nn4,1544,3324,1124,670
Chebyshev2UniformFinite-sample, fixed nn9,81110,8709,58313,278

Table 1. Median audits to reach an interval half-width of 2 percentage points at 95.5% nominal confidence over 100 paired paths. Counts are rounded to the nearest audit. A budget of 5,000 audits was used for the anytime-valid methods. # audits to ±2\pm2 percentage points. Median [25th, 75th percentiles]. Bold: smallest median among anytime-valid methods. † Oracle population variance. ‡ Analytic fixed-nn count using variance bound 1/41/4.

Figure 3. Anytime-valid certification of nonlinear scores on four datasets: (A) pooled diversity P2P_2 on the Kenya biodiversity dataset, (B) worst-three precision on CIFAR-10N, and (C,D) F1 score on ReImageNet and CleanCoNLL. Curves show median interval half-widths over 100 paired audit paths; shaded regions show interquartile ranges. Panels (A,B) express half-width as a percentage of the interval midpoint; panels (C,D) use percentage points. We compare ATLAS-PAS, ATLAS-SRS, PPR with projection, and adapted Manole–Ramdas stitching. Dashed horizontal lines indicate the precision targets. Full results, including for an uncertainty-only policy ablation, are in section D.5.

Motivation: Auditing for biodiversity credits. Biodiversity credits are purchased by companies and investors to fund nature conservation and restoration worldwide, compensating individual landowners for measurable improvements in ecosystem health [47]. The market is estimated to grow from US$8M annually to $69B by 2050 [46], but credits are burdened by high overhead costs and concerns about trust and verification [1]. Each project may generate tens or hundreds of thousands of sensor records documenting wildlife presence and abundance, making full expert audit impractical. Automated species classifiers reduce labeling cost but can have substantial error: one widely used classifier had an 18.2% error rate on an out-of-sample dataset [39]. Because credit payments depend on these metrics, label errors have direct financial consequences, creating a need for efficient expert auditing. In our biodiversity credit data from Kenya, the submitted labels imply a pooled Hill–Shannon diversity of 8.16, compared with only 4.94 after expert validation—a 65% inflation in the downstream score. An aggregate mislabel rate (e.g., “13% of labels are incorrect”) is insufficient because the direction of an error determines its downstream consequence. Incorrectly labeling cattle as lion, for example, may falsely introduce a rare species and substantially inflate estimated richness and taxonomic diversity (and therefore payment), whereas confusing two common, closely related species may have little effect. An effective audit should therefore account not only for where errors are likely, but also for how potential corrections affect the downstream scoring function.

The same structure arises in myriad settings. In model evaluation, for example, noisy benchmark labels impact downstream quantities such as F1F_1 score, balanced accuracy, or model rankings [26], each of which may depend nonlinearly on the corrected labels. In epidemiology, label corrections may change prevalence estimates [12], while in content moderation they may alter policy-compliance rates.

To address these challenges, we make the following contributions:

  • A general framework for efficient label auditing (fig. 1). We introduce an anytime-valid task-aware label auditing and sampling framework (ATLAS) for fixed categorical datasets. It uses Bayesian predictions of label corrections to guide expert review and e-processes to certify the downstream score. In our experiments, ATLAS typically reached the target precision with substantially fewer audits than competing anytime-valid methods.

  • Efficient anytime-valid inference for structured nonlinear scores. For nonlinear scores such as Hill–Shannon diversity, we exploit linear-envelope representations to build score-specific confidence sequences from our linear auditing framework. The resulting confidence sequences retain anytime-valid guarantees while supporting efficient adaptive auditing.

  • A model-based Bayesian alternative using the same task-aware sampling strategy. When distribution-free guarantees require an impractically large number of audits, we retain the same task-aware strategy but use a Bayesian model of the confusion matrix for uncertainty quantification, trading assumption-free validity for greater statistical efficiency.

Problem Formulation

A provided dataset contains NN records. Record i∈[N]i \in[N] has a submitted label yiy_i (known) and verified label viv_i (unknown until verification). Let L:={1,…,L}\mathcal{L} := \{1,\ldots,L\} be the full normalized label vocabulary, and let R:={yi:i∈[N]}⊆L\mathcal{R} := \{y_i : i \in[N]\} \subseteq\mathcal{L} be the set of submitted labels with R:=∣R∣R := |\mathcal{R}| elements. Thus yi∈Ry_i \in\mathcal{R} and vi∈Lv_i \in\mathcal{L}.

Confusion matrix. A scalar mislabel rate records how often yi≠viy_i \ne v_i across the entire dataset; it does not capture error rates across different classes. We therefore track a full confusion matrix, which counts records for each pair of submitted and verified labels (Figure 2). The diagonal represents how many records are correctly labeled. The unknown finite-population confusion matrix is C⋆∈Z≥0R×LC^\star\in\mathbb{Z}_{\ge0}^{R \times L}, where each entry Crs⋆:=#{i:yi=r,vi=s}C^\star_{rs} := \#\{i : y_i = r, v_i = s\} indicates the number of data points submitted with row rr and verified with column ss. Row rr contains records submitted as label rr, while column ss contains records verified as label ss. The row sums Nr:=∑s∈LCrs⋆=#{i:yi=r}N_r := \sum_{s \in\mathcal{L}} C^\star_{rs} = \#\{i : y_i = r\} are known before auditing.

Figure 2. Audits on the Kenya biodiversity dataset. (Left) Submitted-to-verified confusion matrix (input data). Diagonal entries are correct labels, off-diagonal entries are mislabels. Color indicates counts on a log scale. (Right) 95.5% confidence sequence half-width as the number of audits increases. Curves show medians over 100 audit paths with the 25th–75th percentiles. Annotated dots mark the audits to reach a two percentage point precision. Corresponding results for CIFAR-10N (section D.2.2), ReImageNet (section D.3.2), and CleanCoNLL (section D.4.2) are reported in table 1.

Score functions. We use score function to denote any downstream quantity computed from the corrected labels. Formally, a score function maps the submitted-to-verified confusion matrix C\mathbf{C} to a scalar outcome of interest, such as an error rate, model-evaluation metric, or biodiversity score. Importantly, we are interested in finite-population inference. When auditing labels we are only interested in the score for the particular submitted dataset, unlike in conventional superpopulation inference where we seek to estimate a parameter of an underlying distribution from which the data are assumed to be i.i.d. draws. For the biodiversity credits setting, the relevant score function is challenging as it is nonlinear and nonsmooth; details in appendix § 13.

Strata: splitting rows by metadata. Records often carry metadata, such as the source, location, device, or time period, that may be associated with different correction patterns. Splitting each row by these metadata creates smaller groups of records which we call strata. We write g=(r,a)g = (r, a) for the stratum containing records with submitted label rr and metadata value aa. In our biodiversity application, aa refers to a camera station, where corrections are strongly clustered. Then r(g)r(g) is the submitted label shared by records in stratum gg. The size of each stratum is known before any audit, and each row is the union of its strata. When no metadata is used, each row is its own stratum. Stratification is useful when correction behavior varies across metadata-defined subgroups, because it allows the audit to focus more precisely on where uncertainty remains.

Confidence Sequences for Auditing Linear Score Functions

We first explore how to turn expert audits into a confidence sequence, illustrated in Figure 1b. Confidence sequences provide uncertainty intervals that are valid after every audit, allowing the auditor to monitor uncertainty and stop as soon as desired precision is reached. In this section, we instantiate the auditing workflow in Figure 1 for scores that are linear in the confusion matrix.

Figure 1. Our ATLAS framework for Anytime-valid Task-aware Label Auditing and Sampling. (A) Each step uses past audits to predict the unverified labels and the downstream score. We select a stratum GtG_t with probability πGt,t\pi_{G_t,t}, then an expert audit VtV_t corrects the predicted score through inverse-probability weighting to produce an unbiased score observation ZtZ_t. (B) E-processes accumulate evidence against each candidate score value mm, reject values with sufficient evidence, and use remaining values to form a confidence sequence ItI_t, stopping once the width falls below a tolerance τ\tau. (C) Our priority-aware sampling strategy directs reviews toward large strata with high predicted score uncertainty. We compare against random sampling as a baseline, but note that different sampling strategies only change the efficacy, not the validity of the confidence sequences.

Let D∈RR×L\mathbf{D} \in\mathbb{R}^{R \times L} be a deterministic contribution matrix, where the (r,s)(r,s) element Drs∈[0,1]D_{rs} \in[0,1] is the contribution to the score from a data point submitted as label rr and verified as label ss and is known in advance. For example, when the score function is the mislabel rate, D\mathbf{D} is a binary matrix with Dr,s=1{r≠s}D_{r,s} = \mathbb{1}\{r \ne s\}. We denote the rr-th row of this matrix as drT\mathbf{d}_r^{\mathsf{T}}.

A score is linear if it can be expressed as θ⋆=1N∑i=1NDyi,vi=1N∑r∈R∑s∈LCrsDrs\theta^\star= \frac{1}{N}\sum_{i=1}^{N}D_{y_i,v_i} = \frac{1}{N}\sum_{r\in\mathcal{R}}\sum_{s\in\mathcal{L}}C_{rs}D_{rs}. With linear scores, each record contributes a fixed, known amount determined by its submitted and verified labels. If all verified labels were known, the true score θ⋆\theta^\star could be computed exactly. During the audit, however, only a subset of the data have been reviewed.

A starting point: Serfling’s inequality. For a fixed number of labels sampled uniformly without replacement, Serfling’s inequality Serfling, [35] sharpens the standard Hoeffding bound and gives a finite-sample, distribution-free confidence interval for the mislabel rate. Even so, on the Kenyan dataset, certifying the mislabel rate to ±2\pm2 percentage points at 95.5% confidence requires 4,154 reviews, about one in eight records (Table 1 and Figure 2). A Wald interval reaches the same precision in 1,481 reviews, but relies on an asymptotic approximation and, in this comparison, the true population variance. We therefore seek greater efficiency while retaining rigorous finite-sample validity.

Constructing an unbiased audit observation. We first construct a quantity ZtZ_t whose conditional mean, given everything observed before audit tt, is the downstream score θ⋆\theta^\star:

E[Zt∣Ft−1]=θ⋆for every labeling(1)\mathbb{E}[Z_t \mid\mathcal{F}_{t-1}] = \theta^\star\qquad\text{for every labeling} \tag*{(1)}

where Ft−1\mathcal{F}_{t-1} denotes the audit history available before audit tt. Let At−1A_{t-1} denote the records audited so far. For each stratum gg, let Mg,t=∣(g∖At−1)∣M_{g,t} = |(g \setminus A_{t-1})| be the number of unaudited records remaining. Their score contributions have mean μg,t=1Mg,t∑i∈g∖At−1Dyi,vi\mu_{g,t} = \frac{1}{M_{g,t}}\sum_{i\in g\setminus A_{t-1}}D_{y_i,v_i} and variance σg,t2=1Mg,t∑i∈g∖At−1(Dyi,vi−μg,t)2\sigma^2_{g,t} = \frac{1}{M_{g,t}}\sum_{i\in g\setminus A_{t-1}}(D_{y_i,v_i}-\mu_{g,t})^2. The linear score can be split into:

Nθ⋆=∑i∈At−1Dyi,vi+∑gMg,tμg,t.(2)N\theta^\star= \sum_{i\in A_{t-1}}D_{y_i,v_i} + \sum_g M_{g,t}\mu_{g,t}. \tag*{(2)}

After t−1t-1 audits, the first sum is known and the second must be learned (or estimated) by the auditor. Before audit tt, using only Ft−1\mathcal{F}_{t-1}, the auditor fixes sampling probabilities {πg,t}g\{\pi_{g,t}\}_g (where ∑gπg,t=1\sum_g\pi_{g,t}=1) and a predicted verified-label distribution qg,t∈ΔL−1\mathbf{q}_{g,t}\in\Delta^{L-1} for each nonempty stratum gg. Using these predictions, we predict the downstream score θ⋆\theta^\star via the estimator:

θ^t−1=1N(∑i∈At−1Dyi,vi+∑gMg,t dr(g)Tqg,t).(3)\widehat{\theta}_{t-1} = \frac{1}{N}\left(\sum_{i\in A_{t-1}}D_{y_i,v_i}+\sum_g M_{g,t}\,\mathbf{d}_{r(g)}^{\mathsf{T}}\mathbf{q}_{g,t}\right). \tag*{(3)}

The first term uses the verified contributions of audited records, while the second predicts the contributions of the remaining records. The auditor then draws a stratum GtG_t according to Pr⁡(Gt=g∣Ft−1)=πg,t\Pr(G_t=g\mid\mathcal{F}_{t-1})=\pi_{g,t}, selects a record JtJ_t uniformly from Gt∖At−1G_t\setminus A_{t-1}, and reveals its verified label Vt=vJtV_t=v_{J_t}. Thus an unaudited record in stratum gg is selected with probability πg,t/Mg,t\pi_{g,t}/M_{g,t}.

Because strata may be sampled nonuniformly, simply replacing the predicted contribution with the newly verified one would generally introduce bias. To update this prediction with the next audit, we therefore use an inverse-probability-weighted residual correction Horvitz & Thompson, [14]:

Zt=θ^t−1+MGt,tNπGt,t(Dr(Gt),Vt−dr(Gt)TqGt,t).(4)Z_t = \widehat{\theta}_{t-1} + \frac{M_{G_t,t}}{N\pi_{G_t,t}}\left(D_{r(G_t),V_t}-\mathbf{d}_{r(G_t)}^{\mathsf{T}}\mathbf{q}_{G_t,t}\right). \tag*{(4)}

The residual compares the newly verified contribution Dr(Gt),VtD_{r(G_t),V_t} with its prediction dr(Gt)TqGt,t\mathbf{d}_{r(G_t)}^{\mathsf{T}}\mathbf{q}_{G_t,t}, while the inverse-probability weight MGt,tNπGt,t\frac{M_{G_t,t}}{N\pi_{G_t,t}} corrects for nonuniform sampling. Consequently, ZtZ_t is an unbiased estimate for θ⋆\theta^\star for any predictable predictions and sampling probabilities, as formalized in theorem 1. The prediction term acts as a control variate, analogous to Särndal et al. [33], and improves efficiency while preserving validity.

References

  1. [1]T. Mitchell Aide. The biodiversity credit market needs rigorous baseline, monitoring, and validation practices. npj Biodiversity, 3(1):30, 2024.DOI
  2. [2]Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. Prediction-powered inference. Science, 382(6671):669–674, 2023.arxiv.org/abs/2301.09633
  3. [5]Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, Cambridge, 2004.
  4. [6]Thomas M. Cover, Joy A. Thomas, and John Kieffer. Elements of Information Theory. Wiley, New York, 1991.
  5. [7]D. Csillag, C. J. Struchiner, and G. T. Goedert. Prediction-powered e-values. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 11493–11514, 2025.arxiv.org/abs/2502.04294
  6. [8]Donald A. Darling and Herbert Robbins. Confidence sequences for mean, variance, and median. Proceedings of the National Academy of Sciences, 58(1):66–68, 1967.pubmed.ncbi.nlm.nih.gov/16578652
  7. [9]Christiane Fellbaum (ed.). WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA, 1998.
  8. [10]Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian Data Analysis. Chapman & Hall, London, 1995.
  9. [11]Andrew Gonzalez, Tom August, Sallie Bailey, Kyle Bobiwash, Philipp H. Boersch-Supan, Neil D. Burgess, Barnabas H. Daru, Chris S. Elphick, Robert P. Freckleton, Winifred F. Frick, et al. From data to decisions: Toward a biodiversity monitoring standards framework. Proceedings of the National Academy of Sciences, 123(10):e2519347123, 2026.
  10. [12]Sarah R. Haile and David Kronthaler. Potential for bias in prevalence estimates when not accounting for test sensitivity and specificity: A systematic review of COVID-19 seroprevalence studies. International Journal of Public Health, 70:1608343, 2025.
  11. [13]Timm Haucke, Lauren Harrell, Justin Kay, Mary Clapp, and Sara Beery. Targeted review for ai-assisted biodiversity surveys: Active continuous-score occupancy modeling. arXiv preprint arXiv:2609.25657, 2026.arxiv.org/abs/2609.25657
  12. [14]Daniel G. Horvitz and Donovan J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260):663–685, 1952.DOI
  13. [15]Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametric confidence sequences. The Annals of Statistics, 49(2):1055–1080, 2021.DOI
  14. [16]Hiroaki Kaido, Francesca Molinari, and Jörg Stoye. Confidence intervals for projections of partially identified parameters. Econometrica, 87(4):1397–1432, 2019.
  15. [17]Justin Kay, Grant Van Horn, Subhransu Maji, Daniel Sheldon, and Sara Beery. Consensus-driven active model selection. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4594–4604. IEEE, 2025.arxiv.org/abs/2507.23771
  16. [18]Valentin Kilian, Stefano Cortinovis, and François Caron. Anytime-valid, bayes-assisted, prediction-powered inference. Advances in Neural Information Processing Systems, 38:900–918, 2026.arxiv.org/abs/2505.18000
  17. [19]Hyun-Chul Kim and Zoubin Ghahramani. Bayesian classifier combination. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pp. 619–627, 2012.
  18. [20]Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5753–5763, 2021.arxiv.org/abs/2103.05331
  19. [21]Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active surrogate estimators: An active learning approach to label-efficient model evaluation. In Advances in Neural Information Processing Systems, volume 35, pp. 24557–24570, 2022.arxiv.org/abs/2202.06881
  20. [22]Puheng Li, Tijana Zrnic, and Emmanuel J. Candès. Robust sampling for active statistical inference. In Advances in Neural Information Processing Systems, volume 38, pp. 68727–68756, 2025.arxiv.org/abs/2511.08991
  21. [23]Tudor Manole and Aaditya Ramdas. Martingale methods for sequential estimation of convex functionals and divergences. IEEE Transactions on Information Theory, 69(7):4641–4658, 2023.DOI
  22. [24]Daksh Mittal, Yuanzhe Ma, Shalmali Joshi, and Hongseok Namkoong. Adaptive labeling for efficient out-of-distribution model evaluation. In Advances in Neural Information Processing Systems, volume 37, pp. 70981–71003, 2024.DOI
  23. [25]Jerzy Neyman. On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society, 97(4):558–606, 1934.
  24. [26]Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.arxiv.org/abs/2103.14749
  25. [27]Charlotte Pelletier, Silvia Valero, Jordi Inglada, Nicolas Champion, Claire Marais Sicre, and Gérard Dedieu. Effect of training class label noise on classification performances for land cover mapping with satellite image time series. Remote Sensing, 9(2):173, 2017.DOI
  26. [28]Plan Vivo Foundation. PV Nature methodology and data protocol, April 2026. URL https://www.planvivo.org/resource/pv-nature-methodology-data-protocol. Version 1.2. Accessed July 29, 2026.
  27. [30]Aaditya Ramdas, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4):576–601, 2023.arxiv.org/abs/2210.01948
  28. [31]Susanna Rücker and Alan Akbik. CleanCoNLL: A nearly noise-free named entity recognition dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8628–8645. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.533. URL https://aclanthology.org/2023.emnlp-main.533/.DOI
  29. [32]J. Jon Ryu and Gregory W. Wornell. Gambling-based confidence sequences for bounded random vectors. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 42856–42869, 2024.arxiv.org/abs/2402.03683
  30. [33]Carl-Erik Särndal, Bengt Swensson, and Jan Wretman. Model Assisted Survey Sampling. Springer Series in Statistics. Springer, New York, 2003.
  31. [34]Christoph Sawade, Niels Landwehr, and Tobias Scheffer. Active comparison of prediction models. In Advances in Neural Information Processing Systems, volume 25, pp. 1763–1771, 2012.
  32. [35]Robert J. Serfling. Probability inequalities for the sum in sampling without replacement. The Annals of Statistics, 2(1):39–48, 1974.DOI
  33. [36]Shubhanshu Shekhar and Aaditya Ramdas. Tighter confidence intervals under without replacement sampling via empirical rate functions. arXiv preprint arXiv:2603.14423, 2026.arxiv.org/abs/2603.14423
  34. [37]Shubhanshu Shekhar, Ziyu Xu, Zachary Lipton, Pierre Liang, and Aaditya Ramdas. Risk-limiting financial audits via weighted sampling without replacement. In Proceedings of the 39th Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pp. 1932–1941, 2023.arxiv.org/abs/2305.06884
  35. [38]Jialin Shi, Kailai Zhang, Chenyu Guo, Youquan Yang, Yali Xu, and Ji Wu. A survey of label-noise deep learning for medical image analysis. Medical Image Analysis, 95:103166, 2024.DOI
  36. [39]Michael A. Tabak, Mohammad S. Norouzzadeh, David W. Wolfson, Steven J. Sweeney, Kurt C. VerCauteren, Nathan P. Snow, Joseph M. Halseth, Paul A. Di Salvo, Jesse S. Lewis, Michael D. White, et al. Machine learning to classify animal species in camera trap images: Applications in ecology. Methods in Ecology and Evolution, 10(4):585–590, 2019.DOI
  37. [40]Jean Ville. Étude critique de la notion de collectif. Number 3 in Monographies des Probabilités. Gauthier-Villars, Paris, 1939.
  38. [41]Ilia Volkov, Nikita Kisel, Tetiana Mishkina, Klara Janouskova, and Jiri Matas. Doomed to re-annotate, forever: The ImageNet story. arXiv preprint arXiv:2608.13783, 2026. URL https://arxiv.org/abs/2608.13783.
  39. [42]Siruo Wang, Tyler H McCormick, and Jeffrey T Leek. Post-prediction inference. BioRxiv, pp. 2020–01, 2020.DOI
  40. [43]Ian Waudby-Smith and Aaditya Ramdas. Confidence sequences for sampling without replacement. In Advances in Neural Information Processing Systems, volume 33, pp. 20204–20214, 2020.
  41. [44]Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(1):1–27, 2024.
  42. [45]Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022.arxiv.org/abs/2110.12088
  43. [46]World Economic Forum. Biodiversity credits: Demand drivers and guidance on early use, December 2023. URL https://www.weforum.org/publications/biodiversity-credits-demand-drivers-and-guidance-on-early-use/. Accessed July 29, 2026.
  44. [47]Sven Wunder, Cecilia Fraccaroli, Joseph W Bull, Trishna Dutta, Alison Eyres, Megan C Evans, Bo Jellesmark Thorsen, Julia PG Jones, Martine Maron, Bart Muys, et al. Biodiversity credits: An overview of the current state, future opportunities, and potential pitfalls. Business Strategy and the Environment, 34(7):8470–8499, 2025.DOI
  45. [48]Ziyu Xu, Nikos Karampatziakis, and Paul Mineiro. Active, anytime-valid risk controlling prediction sets. In Advances in Neural Information Processing Systems, volume 37, pp. 60110–60132, 2024.arxiv.org/abs/2406.10490
  46. [49]Tijana Zrnic and Emmanuel J. Candès. Active statistical inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 62993–63010, 2024.arxiv.org/abs/2403.03208

Paper details

vlnc:W_h2p7nh2v
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Generative AI tools

View statement
Looking for related papers…
Contents