betaFeedback

WorksMathematics

Testing for Outliers with Conformal p-values

Abstract

This paper studies the construction of p-values for nonparametric outlier detection, taking a multiple-testing perspective. The goal is to test whether new independent samples belong to the same distribution as a reference data set or are outliers. We propose a solution based on conformal inference, a broadly applicable framework which yields p-values that are marginally valid but mutually dependent for different test points. We prove these p-values are positively dependent and enable exact false discovery rate control, although in a relatively weak marginal sense. We then introduce a new method to compute p-values that are both valid conditionally on the training data and independent of each other for different test points; this paves the way to stronger type-I error guarantees. Our results depart from classical conformal inference as we leverage concentration inequalities rather than combinatorial arguments to establish our finite-sample guarantees. Furthermore, our techniques also yield a uniform confidence bound for the false positive rate of any outlier detection algorithm, as a function of the threshold applied to its raw statistics. Finally, the relevance of our results is demonstrated by numerical experiments on real and simulated data.

Introduction

Problem statement and motivation

We consider an outlier detection problem in which one observes a data set D={Xi}i=12n\mathcal{D} = \{X_i\}_{i=1}^{2n} containing 2n2n independent and identically distributed points Xi∈RdX_i \in\mathbb{R}^d drawn from an unknown distribution PXP_X (which may be continuous, discrete, or mixed). The goal is to test which among a new set of ntest≥1n_{\mathrm{test}} \ge1 independent observations Dtest={X2n+i}i=1ntest\mathcal{D}^{\mathrm{test}} = \{X_{2n+i}\}_{i=1}^{n_{\mathrm{test}}} are outliers, in the sense that they were not drawn from the same distribution PXP_X. By contrast, we refer to points drawn from PXP_X as inliers. This problem has applications in many domains, including medical diagnostics [1], spotting frauds or intrusions [2], forensic analysis [3], monitoring engineering systems for failures [4], and out-of-distribution detection in machine learning [5, 6, 7, 8]. A variety of machine-learning tools have been developed to address this classification task, which is sometimes referred to as one-class classification [9, 10] because the data in D\mathcal{D} do not contain any outliers. However, such algorithms are often complex and their outputs are not directly covered by any precise statistical guarantees. Fortunately, conformal inference [11, 12] allows one to practically convert the output of any one-class classifier (if it is invariant to the ordering of the training observations) into a provably valid p-value for the null hypothesis H0,i:Xi∼PX\mathcal{H}_{0,i}: X_i \sim P_X, for any Xi∈DtestX_i \in\mathcal{D}^{\mathrm{test}}.

In many applications, the number of outlier tests, ntestn_{\mathrm{test}}, is large and, therefore, it may be necessary to account for multiple comparisons to avoid making an excessive number of false discoveries. A meaningful error rate in this setting is the false discovery rate (FDR) [13]: the expected proportion of true inliers among the test points reported as outliers. For example, if a particular financial transaction is labeled by an automated system as likely to be fraudulent (i.e., unusual, or out-of-distribution compared to a data set of normal transactions), someone may then need to review it manually, and possibly contact the involved customer. Since these follow-up procedures have a cost, controlling the FDR may be a sensible solution to ensure resources are allocated efficiently. From a statistical perspective, multiple testing in this setting requires some care because classical conformal p-values corresponding to different values of i>2ni > 2n are independent of each other only conditional on D\mathcal{D}, although they are valid only marginally over D\mathcal{D}. This situation is delicate because FDR control typically requires p-values that either are mutually independent or follow certain patterns of dependence [14, 15]. Similarly, global testing (i.e., aggregating evidence from multiple observations to test weaker batch-level hypotheses) may also require independent p-values. This paper addresses the above issues by carefully studying the theoretical properties of some standard multiple testing procedures applied to conformal p-values, and by developing new methods to compute p-values with stronger validity properties.

The conformal inference methods studied in this paper are statistical wrappers for one-class classifiers. The latter are algorithms trained on data clean of any outliers to compute a score function s^:Rd→R\hat{s}: \mathbb{R}^{d} \to\mathbb{R} assigning a scalar value to any future data point, so that smaller (for example) values of s^(X)\hat{s}(X) provide evidence that XX may be an outlier. By design, the classifier attempts to construct scores that separate outliers from inliers effectively, by learning from the data what inliers typically look like, and it may be based on sophisticated black-box models to maximize power. While often effective in practice, these machine-learning algorithms have the drawback of not offering any clear guarantees about the quality of their output. For example, they do not directly provide a null distribution for the classification scores s^\hat{s} evaluated on true inliers, or any particular threshold to limit the rate of false positives. This is where conformal inference comes to help. After training s^\hat{s} on a subset of the observations in D\mathcal{D}, namely those in Dtrain={X1,…,Xn}\mathcal{D}^{\mathrm{train}} = \{X_1,\ldots,X_n\}, the scores are evaluated on the remaining nn hold-out samples in Dcal={Xn+1,…,X2n}\mathcal{D}^{\mathrm{cal}} = \{X_{n+1},\ldots,X_{2n}\}. (Note that Dtrain\mathcal{D}^{\mathrm{train}} and Dcal\mathcal{D}^{\mathrm{cal}} do not need to contain the same number of observations, although the current choice simplifies the notation without loss of generality.) Let us assume, for simplicity, that s^(X)\hat{s}(X) has a continuous distribution if X∼PXX \sim P_X is independent of the data used to train s^\hat{s}, although this assumption could be relaxed at the cost of some additional technical details. Then, define FF as the cumulative distribution function (CDF) of s^(X)\hat{s}(X). If we knew FF, we could utilize F(s^(Xi))F(\hat{s}(X_i)) as an exact p-value for the null hypothesis H0,i:Xi∼PX\mathcal{H}_{0,i}: X_i \sim P_X, for any Xi∈DtestX_i \in\mathcal{D}^{\mathrm{test}}, in the sense that F(s^(Xi))F(\hat{s}(X_i)) would be uniformly distributed if H0,i\mathcal{H}_{0,i} is true. In practice, however, we do not have direct access to FF because PXP_X is unknown and the machine-learning algorithm upon which s^\hat{s} depends is assumed to be a black-box. Instead, we can evaluate the empirical CDF of s^(Xi)\hat{s}(X_i) for all Xi∈DcalX_i \in\mathcal{D}^{\mathrm{cal}}, which we denote as F^n\hat{F}_n. In the following, we will discuss how to construct provably valid conformal p-values for a future observation X2n+1X_{2n+1} by evaluating

u^(X2n+1)=(g∘F^n∘s^)(X2n+1),(1)\hat{u}(X_{2n+1}) = \left(g \circ\hat{F}_n \circ\hat{s}\right)(X_{2n+1}), \tag*{(1)}

where gg is a suitable adjustment function, and the symbol ∘\circ denotes a composition; i.e., (f∘g)(x)=f(g(x))(f \circ g)(x) = f(g(x)). Note that, hereafter, we will treat the observations in Dtrain\mathcal{D}^{\mathrm{train}} as fixed and focus on the randomness in the calibration (Dcal\mathcal{D}^{\mathrm{cal}}) and test (Dtest\mathcal{D}^{\mathrm{test}}) data, upon which conformal inferences are generally based.

Preview of contributions

In § 2, we will focus on the classical conformal inference methods, which produce marginally super-uniform (conservative) p-values u^(marg)(X2n+1)\hat{u}^{(\mathrm{marg})}(X_{2n+1}) satisfying

P[u^(marg)(X2n+1)≤t]≤t,(2)\mathbb{P}\left[\hat{u}^{(\mathrm{marg})}(X_{2n+1}) \le t\right] \le t, \tag*{(2)}

for any t∈(0,1)t \in(0,1), whenever X2n+1X_{2n+1} is an inlier. We say these p-values are marginally valid because they depend on the calibration data in Dcal\mathcal{D}^{\mathrm{cal}}, and both Dcal\mathcal{D}^{\mathrm{cal}} and X2n+1X_{2n+1} are random in (2). In particular, the classical u^(marg)\hat{u}^{(\mathrm{marg})} is computed by applying the adjustment function g(marg)(x)=(nx+1)/(n+1)g^{(\mathrm{marg})}(x)=(nx+1)/(n+1) to (1), i.e.,

u^(marg)(x)=1+∣{i∈Dcal:s^(Xi)≤s^(x)}∣n+1.(3)\hat{u}^{(\mathrm{marg})}(x)=\frac{1+\left|\left\{i\in\mathcal{D}^{\mathrm{cal}}:\hat{s}(X_i)\le\hat{s}(x)\right\}\right|}{n+1}. \tag*{(3)}

Note that (2) is implied by (3) because when s^(X)\hat{s}(X) follows a continuous distribution, u^(marg)(X)\hat{u}^{(\mathrm{marg})}(X) is uniformly distributed on {1/(n+1),2/(n+1),…,1}\{1/(n+1),2/(n+1),\ldots,1\} if X∼PXX\sim P_X independently of the data in Dtrain\mathcal{D}^{\mathrm{train}} [11, 12]. (If s^(X)\hat{s}(X) is not continuous, one can still verify that u^(marg)(X)\hat{u}^{(\mathrm{marg})}(X) is super-uniform in distribution.) However, this is not necessarily true if one conditions on D=Dtrain∪Dcal\mathcal{D}=\mathcal{D}^{\mathrm{train}}\cup\mathcal{D}^{\mathrm{cal}}, in which case u^(marg)(X)\hat{u}^{(\mathrm{marg})}(X) may become anti-conservative due to random fluctuations in the distribution of scores within Dcal\mathcal{D}^{\mathrm{cal}}. Intuitively, this means the marginal p-values in (3) are only valid on average if data in Dcal\mathcal{D}^{\mathrm{cal}} are treated as random. Unfortunately, this guarantee may be too weak to be satisfactory for a practitioner who wants to compute p-values for a large number of test points but is constrained to working with a single calibration data set. Indeed, the numerical experiments presented in § 5.2 will show that inferences based on marginal conformal p-values may be systematically invalid for a large fraction of practitioners working with “unlucky” calibration data sets.

Furthermore, marginal p-values corresponding to different test points, {u^(marg)(X)}X∈Dtest\{\hat{u}^{(\mathrm{marg})}(X)\}_{X\in\mathcal{D}^{\mathrm{test}}}, are not mutually independent because they are all affected by Dcal\mathcal{D}^{\mathrm{cal}}; see Figure 1 for a visualization of this dependence. This should be taken into account when adjusting for multiplicity in outlier detection applications because some common testing procedures are not generally valid for dependent p-values. For example, we will prove in § 2 that the dependence among marginal p-values invalidates Fisher’s combination test [16] for the global null that there are no outliers in Dtest\mathcal{D}^{\mathrm{test}}, even if the calibration data in Dcal\mathcal{D}^{\mathrm{cal}} are treated as random, although this can be easily fixed by suitably adjusting the critical value. By contrast, we can prove the dependence between conformal p-values does not break the average FDR control of the Benjamini-Hochberg (BH) procedure [13], even if the latter is applied with Storey’s correction [17]. The behaviours of additional multiple testing procedures, such as the harmonic mean method [18], Simes method [19], and Stouffer’s method [20], applied to conformal p-values will be investigated empirically in § 5.

Figure 1. Visualization of the joint distribution of the conformal p-values. The distribution of s^(x)\hat{s}(x) is the same for calibration and inlier test points. The conformal p-value for each test point is the number of calibration points to its left, divided by the total number of calibration points plus one, as in (3).

In any case, regardless of whether the mutual dependence among marginal p-values theoretically invalidates the average inferences of a particular multiple-testing procedure, one may sometimes be interested in obtaining stronger guarantees conditional on the calibration data. Consider for instance the following prototypical scenario. A researcher, or a company, acquires an expensive data set D\mathcal{D} containing clean examples of some variable XX of interest, and wishes to leverage that information to construct a system to detect outliers in future test points, while avoiding an excess of false positives. Assuming the stakes in this application are sufficiently high, the researcher may need clear statistical guarantees about the output of such procedure (as opposed to blindly trusting a black-box model), and thus decides to employ conformal inference. Unfortunately, the marginal validity property in (2) tells us very little about how this outlier detection system may perform in the future for this particular researcher relying on this particular data set DD. Instead, marginal validity suggests the system will work on average for different researchers starting from different data sets; of course, that may not feel fully satisfactory for any one of them.

Therefore, we will construct in § 3 conformal p-values satisfying a stronger property, which we call calibration-conditional validity (CCV). Formally, the novel p-values u^(ccv)(x)\hat{u}^{(\mathrm{ccv})}(x) will satisfy

P[P[u^(ccv)(X2n+1)≤t∣D]≤t for all t∈(0,1)]≥1−δ,(4)\mathbb{P}\left[\mathbb{P}\left[\hat{u}^{(\mathrm{ccv})}(X_{2n+1}) \le t \mid D\right] \le t\text{ for all }t\in(0,1)\right] \ge1-\delta, \tag*{(4)}

if X2n+1∼PXX_{2n+1}\sim P_X, for any value of δ∈(0,1)\delta\in(0,1) pre-specified by the user. The crucial difference between (4) and (2) is that the latter intuitively guarantees the p-values are valid for at least a fraction 1−δ1-\delta of researchers; this can give a precise measure of confidence to each one of them. Furthermore, calibration-conditional p-values have the advantage of making multiple testing straightforward. In fact, these p-values are still trivially independent of one another conditional on the calibration data, so their high-probability guarantee of validity will immediately extend to the output of any downstream multiple-testing procedure that assumes independence.

While most of this paper focuses on the validity of conformal p-values from a multiple-testing perspective, we will see in § 4 that our high-probability results can also be utilized to construct a uniform upper confidence bound for the false positive rate of any machine-learning algorithm for outlier detection, as a function of the threshold applied to its raw output scores. This may help practitioners interpret the output of black-box methods directly, without necessarily operating in terms of p-values. (However, as statisticians, we prefer the p-value approach because it is more versatile.) Furthermore, our results can be easily leveraged to obtain predictive sets with stronger coverage guarantees compared to existing conformal methods.

Finally, in § 5, we will compare the performance of marginal and calibration-conditional conformal p-values on simulated as well as real data, in combination with different multiple testing procedures. These numerical experiments will provide an empirical confirmation of our theoretical results, and also highlight how stronger guarantees sometimes come at the cost of lower power.

Related work

References

  1. [1]L. Tarassenko, P. Hayton, N. Cerneaz, and M. Brady. “Novelty detection for the identification of masses in mammograms”. In: 1995 Fourth International Conference on Artificial Neural Networks. IET. 1995, pp. 442–447.DOI
  2. [2]A. Patcha and J.-M. Park. “An overview of anomaly detection techniques: Existing solutions and latest technological trends”. In: Computer networks 51.12 (2007), pp. 3448–3470.DOI
  3. [3]F. Fortunato, L. Anderluci, and A. Montanari. “One-class classification with application to forensic analysis”. In: Journal of the Royal Statistical Society: Series C (Applied Statistics) 69.5 (2020), pp. 1227–1249.arxiv.org/abs/1905.02406
  4. [4]L. Tarassenko, D. A. Clifton, P. R. Bannister, S. King, and D. King. “Novelty Detection”. In: Encyclopedia of Structural Health Monitoring. American Cancer Society, 2009.DOI
  5. [5]D. Hendrycks and K. Gimpel. “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks”. In: Proceedings of International Conference on Learning Representations (2017).
  6. [6]S. Liang, Y. Li, and R. Srikant. “Enhancing the reliability of out-of-distribution image detection in neural networks”. In: arXiv preprint arXiv:1706.02690 (2017).arxiv.org/abs/1706.02690
  7. [7]K. Lee, K. Lee, H. Lee, and J. Shin. “A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks”. In: NeurIPS. 2018.arxiv.org/abs/1807.03888
  8. [8]K. Lee, H. Lee, K. Lee, and J. Shin. “Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples”. In: International Conference on Learning Representations. 2018.arxiv.org/abs/1711.09325
  9. [9]M. M. Moya, M. W. Koch, and L. D. Hostetler. “One-class classifier networks for target recognition applications”. In: NASA STI/Recon Technical Report N 93 (1993), p. 24043.
  10. [10]M. A. Pimentel, D. A. Clifton, L. Clifton, and L. Tarassenko. “A review of novelty detection”. In: Signal Processing 99 (2014), pp. 215–249.DOI
  11. [11]V. Vovk, A. Gammerman, and C. Saunders. “Machine-learning applications of algorithmic randomness”. In: International Conference on Machine Learning. 1999, pp. 444–453.
  12. [12]V. Vovk, A. Gammerman, and G. Shafer. Algorithmic learning in a random world. Springer, 2005.
  13. [13]Y. Benjamini and Y. Hochberg. “Controlling the false discovery rate: a practical and powerful approach to multiple testing”. In: Journal of the Royal statistical society: series B (Methodological) 57.1 (1995), pp. 289–300.DOI
  14. [14]Y. Benjamini and D. Yekutieli. “The control of the false discovery rate in multiple testing under dependency”. In: Annals of Statistics (2001), pp. 1165–1188.DOI
  15. [15]S. Clarke and P. Hall. “Robustness of multiple testing procedures against dependence”. In: Annals of Statistics 37.1 (2009), pp. 332–358.DOI
  16. [16]R. Fisher. Statistical methods for research workers. Oliver & Boyd (Edinburgh), 1925.
  17. [17]J. D. Storey, J. E. Taylor, and D. Siegmund. “Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach”. In: Journal of the Royal Statistical Society: Series B (Statistical Methodology) 66.1 (2004), pp. 187–205.DOI
  18. [18]D. J. Wilson. “The harmonic mean p-value for combining dependent tests”. In: Proceedings of the National Academy of Sciences 116.4 (2019), pp. 1195–1200.DOI
  19. [19]R. J. Simes. “An improved Bonferroni procedure for multiple tests of significance”. In: Biometrika 73.3 (1986), pp. 751–754.DOI
  20. [20]S. A. Stouffer, E. A. Suchman, L. C. DeVinney, S. A. Star, and R. M. Williams Jr. “The american soldier: Adjustment during army life.(studies in social psychology in world war II), vol. 1”. In: (1949).
  21. [21]S. S. Wilks. “Multivariate statistical outliers”. In: Sankhyā: The Indian Journal of Statistics, Series A (1963), pp. 407–426.
  22. [22]D. M. Hawkins. Identification of outliers. Vol. 11. Springer, 1980.
  23. [23]M. Riani, A. C. Atkinson, and A. Cerioli. “Finding an unknown number of multivariate outliers”. In: Journal of the Royal Statistical Society: series B (statistical methodology) 71.2 (2009), pp. 447–466.
  24. [24]A. Cerioli. “Multivariate outlier detection with high-breakdown estimators”. In: Journal of the American Statistical Association 105.489 (2010), pp. 147–156.DOI
  25. [25]S. S. Khan and M. G. Madden. “One-class classification: taxonomy of study and review of techniques”. In: The Knowledge Engineering Review 29.3 (2014), pp. 345–374.DOI
  26. [26]S. Agrawal and J. Agrawal. “Survey on anomaly detection using data mining techniques”. In: Procedia Computer Science 60 (2015), pp. 708–713.DOI
  27. [27]C. C. Aggarwal. “Outlier analysis”. In: Data mining. Springer. 2015, pp. 237–263.
  28. [28]M. Sabokrou, M. Khalooei, M. Fathy, and E. Adeli. “Adversarially learned one-class classifier for novelty detection”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, pp. 3379–3388.arxiv.org/abs/1802.09088
  29. [29]R. Chalapathy and S. Chawla. “Deep learning for anomaly detection: A survey”. In: preprint at arXiv:1901.03407 (2019).
  30. [30]R. Laxhammar and G. Falkman. “Inductive conformal anomaly detection for sequential detection of anomalous sub-trajectories”. In: Annals of Mathematics and Artificial Intelligence 74.1-2 (2015), pp. 67–94.
  31. [31]J. Smith, I. Nouretdinov, R. Craddock, C. Offer, and A. Gammerman. “Conformal anomaly detection of trajectories with a multi-class hierarchy”. In: International symposium on statistical learning and data sciences. Springer. 2015, pp. 281–290.
  32. [32]V. Ishimtsev, A. Bernstein, E. Burnaev, and I. Nazarov. “Conformal k-NN Anomaly Detector for Univariate Data Streams”. In: Conformal and Probabilistic Prediction and Applications. PMLR. 2017, pp. 213–227.arxiv.org/abs/1706.03412
  33. [33]L. Guan and R. Tibshirani. “Prediction and outlier detection in classification problems”. In: arXiv preprint arXiv:1905.04396 (2019).arxiv.org/abs/1905.04396
  34. [34]F. Cai and X. Koutsoukos. “Real-time Out-of-distribution Detection in Learning-Enabled Cyber-Physical Systems”. In: 2020 ACM/IEEE 11th International Conference on Cyber-Physical Systems (ICCPS). IEEE. 2020, pp. 174–183.arxiv.org/abs/2001.10494
  35. [35]M. Haroush, T. Frostig, R. Heller, and D. Soudry. “Statistical Testing for Efficient Out of Distribution Detection in Deep Neural Networks”. In: arXiv preprint arXiv:2102.12967 (2021).arxiv.org/abs/2102.12967
  36. [36]V. Vovk, I. Nouretdinov, and A. Gammerman. “Testing Exchangeability On-Line”. In: Jan. 2003, pp. 768–775.
  37. [37]V. Fedorova, A. Gammerman, I. Nouretdinov, and V. Vovk. “Plug-in Martingales for Testing Exchangeability on-Line”. In: Proceedings of the 29th International Conference on International Conference on Machine Learning. ICML’12. Edinburgh, Scotland: Omnipress, 2012, pp. 923–930.arxiv.org/abs/1204.3251
  38. [38]V. Vovk. “Testing Randomness Online”. In: Statistical Science 36.4 (2021), pp. 595–611.DOI
  39. [39]V. Vovk. “Testing for concept shift online”. In: arXiv preprint arXiv:2012.14246 (2020).arxiv.org/abs/2012.14246
  40. [40]V. Vovk, I. Petej, I. Nouretdinov, E. Ahlberg, L. Carlsson, and A. Gammerman. “Retrain or not retrain: Conformal test martingales for change-point detection”. In: Conformal and Probabilistic Prediction and Applications. PMLR. 2021, pp. 191–210.arxiv.org/abs/2102.10439
  41. [41]V. Vovk. “Conditional Validity of Inductive Conformal Predictors”. In: Proceedings of the Asian Conference on Machine Learning. Vol. 25. 2012, pp. 475–490.arxiv.org/abs/1209.2673
  42. [42]R. Foygel Barber, E. J. Candès, A. Ramdas, and R. J. Tibshirani. “The limits of distribution-free conditional predictive inference”. In: Information and Inference: A Journal of the IMA 10.2 (2021), pp. 455–482.arxiv.org/abs/1903.04684
  43. [43]Y. Hechtlinger, B. Póczos, and L. Wasserman. Cautious Deep Learning. arXiv:1805.09460. 2018.arxiv.org/abs/1805.09460
  44. [44]Y. Romano, M. Sesia, and E. J. Candès. “Classification with Valid and Adaptive Coverage”. In: Advances in Neural Information Processing Systems 33 (2020).arxiv.org/abs/2006.02544
  45. [45]M. Cauchois, S. Gupta, and J. C. Duchi. “Knowing what You Know: valid and validated confidence sets in multiclass and multilabel prediction”. In: Journal of Machine Learning Research 22.81 (2021), pp. 1–42.
  46. [46]A. N. Angelopoulos, S. Bates, J. Malik, and M. I. Jordan. “Uncertainty Sets for Image Classifiers using Conformal Prediction”. In: preprint at arXiv:2009.14193 (2020).arxiv.org/abs/2009.14193
  47. [47]Y. Romano, E. Patterson, and E. Candès. “Conformalized Quantile Regression”. In: Advances in Neural Information Processing Systems 32. 2019, pp. 3543–3553.arxiv.org/abs/1905.03222
  48. [48]R. Izbicki, G. Shimizu, and R. Stern. “Flexible distribution-free conditional predictive bands using density estimators”. In: International Conference on Artificial Intelligence and Statistics. PMLR. 2020, pp. 3068–3077.arxiv.org/abs/1910.05575
  49. [49]V. Chernozhukov, K. Wüthrich, and Y. Zhu. “Distributional conformal prediction”. In: Proceedings of the National Academy of Sciences 118.48 (2021).DOI
  50. [50]D. Kivaranovic, K. D. Johnson, and H. Leeb. “Adaptive, Distribution-Free Prediction Intervals for Deep Networks”. In: International Conference on Artificial Intelligence and Statistics. PMLR. 2020, pp. 4346–4356.arxiv.org/abs/1905.10634
  51. [51]C. Gupta, A. K. Kuchibhotla, and A. K. Ramdas. “Nested conformal prediction and quantile out-of-bag ensemble methods”. In: Pattern Recognition (2021), p. 108496.DOI
  52. [52]V. Vovk. “Cross-conformal predictors”. In: Annals of Mathematics and Artificial Intelligence 74.1-2 (2015), pp. 9–28.
  53. [53]R. F. Barber, E. J. Candès, A. Ramdas, R. J. Tibshirani, et al. “Predictive inference with the jackknife+”. In: Annals of Statistics 49.1 (2021), pp. 486–507.arxiv.org/abs/1905.02928
  54. [54]B. Kim, C. Xu, and R. Foygel Barber. “Predictive inference is free with the jackknife+-after-bootstrap”. In: Advances in Neural Information Processing Systems 33 (2020).arxiv.org/abs/2002.09025
  55. [55]H. Papadopoulos, K. Proedrou, V. Vovk, and A. Gammerman. “Inductive Confidence Machines for Regression”. In: Machine Learning: European Conference on Machine Learning ECML 2002. 2002, pp. 345–356.
  56. [56]J. Lei, A. Rinaldo, and L. Wasserman. “A Conformal Prediction Approach to Explore Functional Data”. In: Annals of Mathematics and Artificial Intelligence 74 (Feb. 2013).arxiv.org/abs/1302.6452
  57. [57]F. Wilcoxon. “Individual comparisons by ranking methods”. In: Breakthroughs in statistics. Springer, 1992, pp. 196–202.
  58. [58]J. Friedman. On multivariate goodness-of-fit and two-sample testing. Tech. rep. No. SLAC-PUB-10325. Stanford Linear Accelerator Center, Menlo Park, CA (US), 2004.DOI
  59. [59]D. Lopez-Paz and M. Oquab. “Revisiting classifier two-sample tests”. In: International Conference on Learning Representations. 2017.
  60. [60]A. K. Kuchibhotla. “Exchangeability, Conformal Prediction, and Rank Tests”. In: arXiv preprint arXiv:2005.06095 (2020).arxiv.org/abs/2005.06095
  61. [61]X. Hu and J. Lei. “A Distribution-Free Test of Covariate Shift Using Conformal Prediction”. In: arXiv preprint arXiv:2010.07147 (2020).
  62. [62]I. Kim, A. Ramdas, A. Singh, L. Wasserman, et al. “Classification accuracy as a proxy for two-sample testing”. In: Annals of Statistics 49.1 (2021), pp. 411–434.arxiv.org/abs/1602.02210
  63. [63]S. S. Wilks. “Determination of Sample Sizes for Setting Tolerance Limits”. In: Ann. Math. Statist. 12.1 (Mar. 1941), pp. 91–96.DOI
  64. [64]S. S. Wilks. “Statistical Prediction with Special Reference to the Problem of Tolerance Limits”. In: Ann. Math. Statist. 13.4 (Dec. 1942), pp. 400–409.DOI
  65. [65]A. Wald. “An Extension of Wilks’ Method for Setting Tolerance Limits”. In: Ann. Math. Statist. 14.1 (Mar. 1943), pp. 45–55.DOI
  66. [66]J. W. Tukey. “Non-Parametric Estimation II. Statistically Equivalent Blocks and Tolerance Regions–The Continuous Case”. In: Ann. Math. Statist. 18.4 (Dec. 1947), pp. 529–539.DOI
  67. [67]K. Krishnamoorthy and T. Mathew. Statistical Tolerance Regions: Theory, Applications, and Computation. Wiley Series in Probability and Statistics. Wiley, 2009.DOI
  68. [68]S. Park, O. Bastani, N. Matni, and I. Lee. “PAC Confidence Sets for Deep Neural Networks via Calibrated Prediction”. In: International Conference on Learning Representations. 2020.arxiv.org/abs/2001.00106
  69. [69]S. Bates, A. Angelopoulos, L. Lei, J. Malik, and M. Jordan. “Distribution-free, risk-controlling prediction sets”. In: Journal of the ACM (JACM) 68.6 (2021), pp. 1–34.arxiv.org/abs/2101.02703
  70. [70]Y. Zhang and D. N. Politis. “Bootstrap prediction intervals with asymptotic conditional validity and unconditional guarantees”. In: arXiv preprint arXiv:2005.09145 (2020).arxiv.org/abs/2005.09145
  71. [71]M. B. Brown. “400: A method for combining non-independent, one-sided tests of significance”. In: Biometrices (1975), pp. 987–992.DOI
  72. [72]J. T. Kost and M. P. McDermott. “Combining dependent P-values”. In: Statistics & Probability Letters 60.2 (2002), pp. 183–190.DOI
  73. [73]D. Mary and E. Roquain. “Semi-supervised multiple testing”. In: arXiv preprint arXiv:2106.13501 (2021).DOI
  74. [74]A. Weinstein, R. Barber, and E. Candes. “A power and prediction analysis for knockoffs with lasso statistics”. In: arXiv preprint arXiv:1712.06465 (2017).arxiv.org/abs/1712.06465
  75. [75]C.-Y. Yang, L. Lei, N. Ho, and W. Fithian. “BONuS: Multiple multivariate testing with a data-adaptive test statistic”. In: arXiv preprint arXiv:2106.15743 (2021).arxiv.org/abs/2106.15743
  76. [76]B. Rava, W. Sun, G. M. James, and X. Tong. “A Burden Shared is a Burden Halved: A Fairness-Adjusted Approach to Classification”. In: arXiv preprint arXiv:2110.05720 (2021).DOI
  77. [77]J. D. Storey. “A direct approach to false discovery rates”. In: Journal of the Royal Statistical Society: Series B (Statistical Methodology) 64.3 (2002), pp. 479–498.DOI
  78. [78]Y. Benjamini, A. M. Krieger, and D. Yekutieli. “Adaptive linear step-up procedures that control the false discovery rate”. In: Biometrika 93.3 (2006), pp. 491–507.DOI
  79. [79]M. Sesia and E. J. Candès. “A comparison of some conformal quantile regression methods”. In: Stat 9.1 (2020).DOI
  80. [80]A. Dvoretzky, J. Kiefer, and J. Wolfowitz. “Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator”. In: Ann. Math. Stat. (1956), pp. 642–669.
  81. [81]P. Massart. “The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality”. In: Annals of Probability (1990), pp. 1269–1283.DOI
  82. [82]S. K. Sarkar. “Generalizing Simes’ test and Hochberg’s stepup procedure”. In: Annals of Statistics 36.1 (2008), pp. 337–363.DOI
  83. [83]A. Dempster. “Generalized D⁺ₙ Statistics”. In: Ann. Math. Stat. 30.2 (1959), pp. 593–597.DOI
  84. [84]A. Kolmogorov. “Sulla determinazione empirica di una legge di distribuzione”. In: Inst. Ital. Attuari, Giorn. 4 (1933), pp. 83–91.
  85. [85]F. Eicker. “The asymptotic distribution of the suprema of the standardized empirical processes”. In: Annals of Statistics (1979), pp. 116–138.DOI
  86. [86]F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. “Scikit-learn: Machine Learning in Python”. In: Journal of Machine Learning Research 12 (2011), pp. 2825–2830.arxiv.org/abs/1201.0490
  87. [87]S. K. Sarkar and C.-K. Chang. “The Simes method for multiple hypothesis testing with positively dependent test statistics”. In: Journal of the American Statistical Association 92.440 (1997), pp. 1601–1608.
  88. [88]M. J. Strube. “Combining and comparing significance levels from nonindependent hypothesis tests.” In: Psychological bulletin 97.2 (1985), p. 334.DOI
  89. [89]G. O. Campos, A. Zimek, J. Sander, R. J. Campello, B. Micenková, E. Schubert, I. Assent, and M. E. Houle. “On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study”. In: Data mining and knowledge discovery 30.4 (2016), pp. 891–927.
  90. [90]Amsterdam Library of Object Images (ALOI) Data Set. https://www.dbs.ifi.lmu.de/research/outlier-evaluation/DAMI/literature/ALOI. Not normalized, without duplicates. Accessed: January, 2021.
  91. [91]Covertype Data Set. http://odds.cs.stonybrook.edu/forestcovercovertype-dataset. Accessed: January, 2021.
  92. [92]Credit Card Fraud Detection Data Set. https://www.kaggle.com/mlg-ulb/creditcardfraud. Accessed: January, 2021.
  93. [93]KDD Cup 1999 Data Set. https://www.kaggle.com/mlg-ulb/creditcardfraud. Not normalized, without duplicates, categorical attributes removed. Accessed: January, 2021.
  94. [94]Mammography Data Set. http://odds.cs.stonybrook.edu/mammography-dataset/. Accessed: January, 2021.
  95. [95]Pen-Based Recognition of Handwritten Digits Data Set. http://odds.cs.stonybrook.edu/pendigits-dataset. Accessed: January, 2021.
  96. [96]Statlog (Shuttle) Data Set. http://odds.cs.stonybrook.edu/shuttle-dataset. Accessed: January, 2021.
  97. [97]F. T. Liu, K. M. Ting, and Z.-H. Zhou. “Isolation forest”. In: 2008 eighth ieee international conference on data mining. IEEE. 2008, pp. 413–422.
  98. [98]A. Li and R. F. Barber. “Multiple testing with the structure-adaptive Benjamini–Hochberg algorithm”. In: Journal of the Royal Statistical Society: Series B (Statistical Methodology) 81.1 (2019), pp. 45–74.arxiv.org/abs/1606.07926
  99. [99]V. Vovk, I. Nouretdinov, V. Manokhin, and A. Gammerman. “Cross-conformal predictive distributions”. In: Conformal and Probabilistic Prediction and Applications. PMLR. 2018, pp. 37–51.
  100. [100]W. Wisniewski, D. Lindsay, and S. Lindsay. “Application of conformal prediction interval estimations to market makers’ net positions”. In: Conformal and Probabilistic Prediction and Applications. PMLR. 2020, pp. 285–301.
  101. [101]M. J. Holland. “Making learning more transparent using conformalized performance prediction”. In: arXiv preprint arXiv:2007.04486 (2020).arxiv.org/abs/2007.04486
  102. [102]P. Bayle, A. Bayle, L. Mackey, and L. Janson. “Cross-validation confidence intervals for test error”. In: Advances in Neural Information Processing Systems 33 (2020).arxiv.org/abs/2007.12671
  103. [103]T. Lipták. “On the combination of independent tests”. In: Magyar Tud Akad Mat Kutato Int Kozl 3 (1958), pp. 171–197.
  104. [104]W. Van Zwet and J. Oosterhoff. “On the combination of independent test statistics”. In: Ann. Math. Stat. 38.3 (1967), pp. 659–680.DOI
  105. [105]V. Vovk and R. Wang. “Combining p-values via averaging”. In: Biometrika 107.4 (2020), pp. 791–808.
  106. [107]P. Moran. “The random division of an interval”. In: Supplement to the Journal of the Royal Statistical Society 9.1 (1947), pp. 92–98.DOI
  107. [108]B. C. Arnold, N. Balakrishnan, and H. N. Nagaraja. A first course in order statistics. SIAM, 2008.
  108. [109]M. J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint. Vol. 48. Cambridge University Press, 2019.
  109. [110]N. Ross. “Fundamentals of Stein’s method”. In: Probability Surveys 8 (2011), pp. 210–293.arxiv.org/abs/1109.1880
  110. [111]S. Boucheron, G. Lugosi, and P. Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
  111. [112]J. Durbin. Distribution Theory for Tests Based on Sample Distribution Function. Vol. 9. SIAM, 1973.
  112. [113]V. Kotel’Nikova and E. Chmala[d]ze. “On computing the probability of an empirical process not crossing a curvilinear boundary”. In: Theory of probability & its applications 27.3 (1983), pp. 640–648.DOI
  113. [114]D. Siegmund. “Boundary crossing probabilities and statistical applications”. In: Annals of Statistics (1986), pp. 361–404.DOI

Paper details

vlnc:W_9p55em9h
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents