Decision making or scientific discovery pipelines such as job hiring and drug discovery often involve multiple stages: before any resource-intensive step, there is often an initial screening that uses predictions from a machine learning model to shortlist a few candidates from a large pool. We study screening procedures that aim to select candidates whose unobserved outcomes exceed user-specified values. We develop a method that wraps around any prediction model to produce a subset of candidates while controlling the proportion of falsely selected units. Building upon the conformal inference framework, our method first constructs p-values that quantify the statistical evidence for large outcomes; it then determines the shortlist by comparing the p-values to a threshold introduced in the multiple testing literature. In many cases, the procedure selects candidates whose predictions are above a data-dependent threshold. Our theoretical guarantee holds under mild exchangeability conditions on the samples, generalizing existing results on multiple conformal p-values. We demonstrate the empirical performance of our method via simulations, and apply it to job hiring and drug discovery datasets.
Introduction
Decision making and scientific discovery are resource intensive tasks: human evaluation is needed before high-stakes decisions such as job hiring [52] and disease diagnosis [19]; several rounds of expensive clinical trials are required before a drug can receive FDA approval [22]. Early on, we often hope to identify viable candidates from a very large pool—consider hundreds of applicants to a position or hundreds of thousands of potential compounds that may bind to the target. In such problems, machine learning prediction is useful for an initial screening step to shortlist a few candidates; in later, more costly stages, only these shortlisted candidates are carefully investigated to confirm the interesting cases.
This paper concerns scenarios where outcomes taking on higher values are of interest. Formally, suppose we have access to a set of training data {(Xi,Yi)}i=1n and a set of test samples {Xn+j}j=1m whose outcomes {Yn+j}j=1m are unobserved, all (X,Y)∈X×Y pairs being i.i.d. from some arbitrary and unknown distribution.1 Given some thresholds {cj}j=1m, our goal is to find as many test units with Yn+j>cj as possible, while ensuring the false discovery rate (FDR), the expected proportion of errors (Yn+j≤cj) among all shortlisted candidates, is controlled. To be specific, letting R⊆{1,…,m} be the selection set, we define FDR as the expectation of the false discovery proportion (FDP), so that
FDR=E[FDP],FDP=1∨∣R∣∑j=1m1{j∈R,Yn+j≤cj},(1)
where we denote a∨b=max{a,b} for any a,b∈R, and the expectation is over the randomness of all training data and all test samples. The FDR is a natural measure of type-I error for binary classification [24]. For regression problems with a continuous response, counting the error is reasonable if each selected candidate incurs a similar cost. We discuss below potential applications with binary or quantitative outcomes.
Candidate screening. Companies are turning to machine learning to support recruitment decisions [20, 50]. Predictions using automatic resume screening [3, 21], semantic matching [40] or AI-assisted interviews are used to screen and select candidates from a large pool. Related tasks include talent sourcing, e.g., finding people who are likely to search for new opportunities, and candidate screening, i.e., selecting qualified applicants before further human evaluation [25]. One may be interested in controlling the FDR (1) for resource efficiency: each shortlisted candidate incurs similar costs such as communication for talent sourcing and interviews before the hiring decisions. In candidate screening, controlling FDR ensures most of the costs are devoted to evaluating and ranking qualified candidates. Job recruitment also has fairness concerns; an alternative goal is to ensure that qualified candidates do not get screened out before human evaluation. To this end, one can flip the sign of the outcomes, and (1) represents the proportion of qualified candidates in the filtered out ones.
Drug discovery. Machine learning is playing a similar role in accelerating the drug discovery pipeline. Early stages of drug discovery aim at finding molecules or compounds—from a diverse library [57] developed by institutions across the world [32]—with strong effects such as high binding affinity to a specific target. The activity of drug candidates can be evaluated by high-throughput screening (HTS) [38]. However, the capacity of this approach is quite limited in practice, and it is generally infeasible to screen the whole library of readily synthesized compounds. Instead, virtual screening [28] by machine learning models has enabled the automatic search of promising drugs. Often, a representative (ideally diverse) subset of the whole library is evaluated by HTS; machine learning models are then trained on these data to predict other candidates’ activity based on their physical, geometric and chemical features [11, 33, 59, 15] and select promising ones for further HTS and/or clinical trials. Given the cost of subsequent investigation, false positives in this process are a major concern [53]. Ensuring that a sufficiently large proportion of resources is devoted to promising drugs is thus important for the efficiency of the whole pipeline.
In these two examples, the FDR quantifies a trade-off between the resources devoted to shortlisted candidates (the selection set) and the benefits from finding interesting candidates (the true positives). This interpretation is similar to the justification of FDR in multiple testing [5, 6, 7]: when evaluating a large number of hypotheses, the FDR measures the proportion of “false leads” for follow-up confirmatory studies. However, in our prediction problem, the affinity of a new drug is inferred not from the observations, but from other similar compounds, i.e., other drugs in the training data. This perspective also blurs the distinction between statistical inference and prediction; we will draw more connections between these sub-fields later.
The FDR may not necessarily be interpreted as a resource-efficiency measure. In the next two examples, controlling the FDR, which limits the error in inferring the direction of outcomes, is relevant to monitoring risk in healthcare and counterfactual inference.
Healthcare. With increasingly available patient data, machine learning is widely adopted to assist human decisions in healthcare. For example, many works use machine learning prediction for large-scale early disease diagnosis [52, 43] and patient risk prediction [41, 13, 30]. Calibrating black-box models is important in such high-stakes problems. When it is more desired to limit false negatives than false positives, machine learning prediction might be used to filter out low-risk cases, leaving other cases for careful human evaluation. It is sensible to control the proportion of high-risk cases among all filtered out samples.
Counterfactual inference. In randomized experiments that run over a period of time, inferring whether the patients have benefited from the treatment option compared to an alternative might inform decisions such as early stopping of the trial for some patients. More generally, inferring the benefit of certain patients also provides evidence on treatment effect heterogeneity. This is a counterfactual inference problem [36, 31] in which one could predict the counterfactuals, i.e., what would happen should one takes an alternative option, by learning from the outcomes of patients under that option, and then compare the prediction to the realized outcomes. In this case, the set of those declared as having benefited from the treatment is informative if the FDR is controlled.
The generic task underlying these applications is to find a subset of candidates whose not-yet-observed outcomes are of interest (e.g., qualification or high binding affinity to the target) from a potentially enormous pool of test samples. This is often achieved by thresholding their test scores—the model prediction on the test samples—from models built on a set of training data that are assumed to be from the same distribution. However, controlling the error in the selected set is a nontrivial task.
Why calibrated predictive inference is insufficient
We consider a binary example to fix ideas, so that Y={0,1}. Our goal is to find test samples with Yn+j=1. A natural starting point is to train a machine learning model that predicts (classifies) Y given X, with the hope that test samples with higher predicted values are more promising. To achieve valid prediction, one could calibrate the model [60] to output a prediction set C1−α(X) taking the form ∅, {0}, {1} or {0,1} with the prescription that C1−α(X) must cover the outcome Y with probability at least 1−α for some user-specified α∈(0,1). The probability is averaged over the randomness in the test sample and the training process.
However, a prediction set with marginal coverage guarantees is insufficient for selection. For instance, one might consider selecting all test samples j with C1−α(Xn+j)={1}. The FDR of the selected set would then be below α if C1−α(Xn+j) covers 1 with probability at least 1−α for the selected units, that is, conditional on selection. However, this is clearly a false statement because predictive inference only ensures (1−α) coverage averaged over all test samples. In fact, no matter how large we set the coverage (1−α), such a naive approach might still return a selection set that contains too many uninteresting candidates.
It might be best to preview our results on a real-world drug discovery dataset properly introduced and studied later in the paper (§ 4.2.1). In short, the goal is to find promising drug candidates, among thousands of molecules, that are active (Y=1) for the HIV target. This dataset is highly imbalanced in the sense that only 3% of the drugs are active, as is often the case in studies on drug discovery. Our main purpose is here to rapidly demonstrate that a straightforward application of conformal prediction methods, selecting those leads with C1−α(Xn+j)={1}, results in over-confident predictions in a sense described below.
We use a deep learning model (this is introduced in § 4.2.1) to construct conformity scores, and ultimately, conformal prediction sets that are one-sided in the sense that they only take on three possible values: ∅, {1} and {0,1}; (see Appendix § 9.§ 1 for details). The left panel of Figure 1 shows the FDR of the naive approach as a function of the confidence level 1−α∈{0.99,0.98,…,0.70}, along with the marginal miscoverage of conformal prediction sets and the proportion of cases in the test set for which C1−α(X)={1}.
Figure 1. Left: FDR (set-conditional miscoverage) of the naive approach and marginal miscoverage as a function of the parameter α; the light blue bars are the proportion of cases among all test samples for which C1−α(X)={1}. Right: FDR (curve) and power (bar) of our selective inference approach and of Bonferroni’s method as a function of the nominal FDR target q. The FDR (resp. power) is computed by averaging the FDP (resp. proportion of true positives) in N=100 independent splits of training, calibration, and test data.
While conformal prediction always achieves nearly exact marginal validity (brown), it is overconfident for seemingly promising candidates, as the error rate among the selected (FDR)—those with C1−α(X)={1}—is very high (orange). When 1−α=0.90, we witness an error rate of about 80%, meaning that 4 out of 5 ‘discoveries’ are false. Even in the extremely conservative case where 1−α=0.99, the FDR exceeds 35%. Note that this phenomenon is independent of the target FDR level. We can thus see that the selection issue would be especially pressing if, say, we aim for a small FDR level. In fact, conformal prediction outputs a large proportion of uninformative sets: as seen from the light bars, about 1−α of the prediction sets are C1−α(X)={0,1} (we observe no empty prediction sets for this data). Thus, it ensures valid marginal coverage even though those C1−α(X)={1} seldom cover the true label.
To make sure the FDR falls below a user specified tolerance q∈{0,1}, one might want to employ a Bonferroni correction. To do this we would pick test cases for which C1−q/m(Xn+j)={1}, where m is the number of test samples. That is, we apply a Bonferroni correction to the marginal coverage, and this ensures that the probability of making a single false selection—which upper bounds the FDR—is below q. In the right panel of Figure 1, we compare the FDR and power of our approach and Bonferroni’s method applied to a range of nominal FDR levels q∈{0.11,0.15,…,0.3}. 2 Our approach yields almost exact FDR control and much higher power than Bonferroni’s.
To ensure calibration on the selected, we will bridge conformal inference and selective inference and devise cfBH, an algorithm that turns any prediction model into a screening mechanism. In a nutshell, instead of calibrating to a fixed confidence level α, we will use tools from conformal inference to quantify the model confidence in outcomes with larger values, and then employ multiple testing ideas to construct a shortlist of candidates with statistical guarantees.
Returning to the drug discovery application, we acknowledge a substantial literature using conformal inference for uncertainty quantification in compound activity prediction, see Lampa et al. [ [34], Eklund et al. [ [18], Svensson et al. [ [56, 55], Lindh et al. [ [37], and Cortés-Ciriano and Bender [ [14] for a recent review. Whether explicitly stated or not, the goal is eventually to select or prioritize compounds that progress to later stages of drug discovery Ahlberg et al. [ [1] after constructing valid prediction intervals. That said, current tools for selection are all heuristic, e.g., picking cases with a high predicted value and a relatively short prediction interval. As already mentioned, a marginally valid prediction set does not necessarily imply reliable selection. The method from this paper fills this gap, and can wrap around the predictions from the literature to produce reliable selection rules for drug discovery.
Hypothesis testing and conformal p-values
One may view our problem as testing the random hypotheses
Hj:Yn+j≤cj,j=1,…,m.(2)
From now on, we denote H0={j:Yn+j≤cj} as the set of null hypotheses. That is, we define a hypothesis Hj for each test sample j, and we say Hj is non-null if Yn+j exceeds the threshold cj. This is perhaps non-classical since the hypothesis Hj is random: it concerns a random variable rather than a model parameter. However, we show that we can still use p-values and rely on multiple hypothesis testing ideas to construct the “rejection” set R.
We start by introducing the tool we use to quantify model confidence: conformal p-values; as its name suggests, these p-values build upon the conformal inference framework Vovk et al. [ [60, 61]. Suppose we are given any prediction model from a training process that is independent of the calibration and test samples. We condition on the training process and view the prediction model as given. We first define a nonconformity score V:X×Y→R based on the prediction model. Intuitively, V(x,y) measures how well a value yconforms to the prediction of the model at x. For example, given a prediction μ^:X→R, one could use V(x,y)=∣y−μ^(x)∣; other popular choices in the literature include ideas based on quantile regression Romano et al., 2019] and conditional density estimation [12]. Should Yn+j be observed, one could compute the nonconformity scores Vi=V(Xi,Yi) for i=1,…,n and Vn+j=V(Xn+j,Yn+j). The corresponding conformal p-value [60, 61, 4] is defined as
where Uj∼Unif[0,1] are i.i.d. random variables to break ties. If the test sample (Xn+j,Yn+j) follows the same distribution as the training data, then pj∗∼Unif[0,1]. However, the mutual dependence among {pj∗} is complicated as they all depend on the same calibration data. A recent paper [4] used conformal p-values for outlier detection; in their setting, observations {(Xn+j,Yn+j)}j=1m are available, and the null set {j:Hj is true} is deterministic since the null hypothesis Hj posits that (Xn+j,Yn+j) follows the same distribution as the training samples. In our setting, the response Yn+j is not observed. This leads us to introduce a different set of conformal p-values. Our analysis also generalizes Bates et al. [ [4] to exchangeable data.
Related work
This work concerns calibrating prediction models to obtain correct directional conclusions on the outcomes. In situations where one cares more about the mistakes on the selected subset, our error notion, the FDR, might be more relevant than average prediction errors. That said, several works have studied FDR control in prediction problems, especially in binary classification. Among them, Dickhaus [ [17] connects classification to multiple testing, showing that controlling type-I error (FDR) at certain levels by thresholding an oracle classifier asymptotically achieves the optimal (Bayes) classification risk; Scott et al. [ [48] provides high-probability bounds for estimating the FDR achieved by classification rules, rather than adaptively controlling it at a specific level.
Our problem setup is close to several recent works on calibrated screening or thresholding [62, 47] in classification or regression problems. These works however focus on different targets; Wang et al. [ [62] focuses on selecting a subset with a prescribed expected number of qualified candidates; Sahoo et al. [ [47] focuses on the calibration of the predicted score itself to achieve a similar notion of error control as ours, but at varying levels for all thresholds. The difference is that our method rigorously controls FDR in finite samples, while it might be difficult to obtain such guarantees for the targets in [62, 47].
Our methods build upon the conformal inference framework [60, 61]. Although conformal-inference-based methods have been developed for reliable uncertainty quantification in various problems [36, 10, 31, 58], the theoretical guarantee usually concerns a single test point. However, in many applications, one might be interested in a batch of individuals and desire uncertainty quantification for multiple test samples simultaneously; in such situations, these methods are insufficient due to the complex dependence structure of test scores and p-values as well as multiplicity issues.
This work is closely related to Bates et al. [ [4], in which the authors use conformal p-values (3) to test for multiple outliers. Our conformal p-values differ from theirs as the outcomes are not observed. A few works [39, 45] are parallel to Bates et al. [ [4], studying multiple testing in a setting where all null hypotheses specify an identical null distribution; they are further generalized by Rava et al. [ [42] to achieve subgroup FDR control in classification. Our method is similar to this line of work in constructing a threshold for certain “scores” and selecting candidates with scores above that threshold. However, we work with random hypotheses and propose distinct procedures, whereas in their works, the hypotheses are deterministic (or conditioned on). We will discuss these distinctions in more detail as we present our results.
References
[1]Ernst Ahlberg, Oscar Hammar, Claus Bendtsen, and Lars Carlsson. Current application of conformal prediction in drug discovery. Annals of Mathematics and Artificial Intelligence, 81(1):145–154, 2017a.DOI
[3]Soumaya Amdouni and Wahiba Ben abdesselam Karaa. Web-based recruiting. In ACS/IEEE International Conference on Computer Systems and Applications-AICCSA 2010, pages 1–7. IEEE, 2010.DOI
[4]Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. arXiv preprint arXiv:2104.08279, 2021.DOI
[5]Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289–300, 1995.DOI
[6]Yoav Benjamini and Yosef Hochberg. Multiple hypotheses testing with weights. Scandinavian Journal of Statistics, 24(3):407–418, 1997.DOI
[7]Yoav Benjamini and Daniel Yekutieli. The control of the false discovery rate in multiple testing under dependency. Annals of statistics, pages 1165–1188, 2001.DOI
[8]Robert Bohrer. Multiple three-decision rules for parametric signs. Journal of the American Statistical Association, 74(366a):432–437, 1979.DOI
[9]Robert Bohrer and Mark J Schervish. An optimal multiple decision rule for signs of parameters. Proceedings of the National Academy of Sciences, 77(1):52–56, 1980.pubmed.ncbi.nlm.nih.gov/16592763
[11]Paula Carracedo-Reboredo, Jose Liñares-Blanco, Nereida Rodríguez-Fernández, Francisco Cedrón, Francisco J Novoa, Adrian Carballal, Victor Maojo, Alejandro Pazos, and Carlos Fernandez-Lozano. A review on machine learning approaches and trends in drug discovery. Computational and Structural Biotechnology Journal, 19:4538–4558, 2021.
[12]Victor Chernozhukov, Kaspar Wüthrich, and Yinchu Zhu. Distributional conformal prediction. Proceedings of the National Academy of Sciences, 118(48), 2021.DOI
[13]Kristin M Corey, Sehi Kashyap, Elizabeth Lorenzi, Sandhya A Lagoo-Deenadayalan, Katherine Heller, Krista Whalen, Suresh Balu, Mitchell T Heflin, Shelley R McDonald, Madhav Swaminathan, et al. Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (pythia): A retrospective, single-site study. PLoS medicine, 15(11):e1002701, 2018.
[14]Isidro Cortés-Ciriano and Andreas Bender. Concepts and applications of conformal prediction in computational drug discovery. arXiv preprint arXiv:1908.03569, 2019.
[15]Suresh Dara, Swetha Dhamercherla, Surender Singh Jadav, CH Babu, and Mohamed Jawed Ahsan. Machine learning in drug discovery: A review. Artificial Intelligence Review, pages 1–53, 2021.
[16]Mindy I Davis, Jeremy P Hunt, Sanna Herrgard, Pietro Ciceri, Lisa M Wodicka, Gabriel Pallares, Michael Hocker, Daniel K Treiber, and Patrick P Zarrinkar. Comprehensive analysis of kinase inhibitor selectivity. Nature biotechnology, 29(11):1046–1051, 2011.DOI
[17]Thorsten Dickhaus. Multiple testing and binary classification. In Simultaneous Statistical Inference, pages 91–101. Springer, 2014.DOI
[18]Martin Eklund, Ulf Norinder, Scott Boyer, and Lars Carlsson. The application of conformal prediction to the drug discovery process. Annals of Mathematics and Artificial Intelligence, 74(1):117–132, 2015.
[19]Ruth Etzioni, Nicole Urban, Scott Ramsey, Martin McIntosh, Stephen Schwartz, Brian Reid, Jerald Radich, Garnet Anderson, and Leland Hartwell. The case for early detection. Nature reviews cancer, 3(4):243–252, 2003.pubmed.ncbi.nlm.nih.gov/12671663
[20]Evanthia Faliagka, Kostas Ramantas, Athanasios Tsakalidis, and Giannis Tzimas. Application of machine learning algorithms to an online recruitment system. In Proc. International Conference on Internet and Web Applications and Services, pages 215–220. Citeseer, 2012.
[21]Evanthia Faliagka, Lazaros Iliadis, Ioannis Karydis, Maria Rigou, Spyros Sioutas, Athanasios Tsakalidis, and Giannis Tzimas. On-line consistent ranking on e-recruitment: seeking the truth behind a well-formed cv. Artificial Intelligence Review, 42(3):515–528, 2014.DOI
[22]U.S. FDA. The drug development process, 2018. URL https://www.fda.gov/patients/learn-about-drug-and-device-approvals/drug-development-process.
[23]Wenge Guo, Sanat K Sarkar, and Shyamal D Peddada. Controlling false discoveries in multidimensional directional decisions, with applications to gene expression data on ordered categories. Biometrics, 66(2):485–492, 2010.DOI
[24]Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
[25]Emily Heaslip. Ai tools for talent acquisition to help you hire, 2022. URL https://vervoe.com/ai-tools-for-talent-acquisition/.
[26]Yosef Hochberg. Multiple classification rules for signs of parameters. Journal of statistical planning and inference, 15:177–188, 1986.DOI
[27]Kexin Huang, Tianfan Fu, Lucas M Glass, Marinka Zitnik, Cao Xiao, and Jimeng Sun. Deeppurpose: A deep learning library for drug-target interaction prediction. Bioinformatics, 2020.DOI
[28]Ziwei Huang. Drug discovery research: new frontiers in the post-genomic era. John Wiley & Sons, 2007.DOI
[29]Guido W. Imbens and Donald B. Rubin. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press, 2015. doi: 10.1017/CBO9781139025751.DOI
[30]Ali Jalali, Hannah Lonsdale, Nhue Do, Jacquelin Peck, Monesha Gupta, Shelby Kutty, Sharon R Ghazarian, Jeffrey P Jacobs, Mohamed Rehman, and Luis M Ahumada. Deep learning for improved risk prediction in surgical outcomes. Scientific reports, 10(1):1–13, 2020.DOI
[31]Ying Jin, Zhimei Ren, and Emmanuel J Candès. Sensitivity analysis of individual treatment effects: A robust conformal inference approach. Proceedings of the National Academy of Sciences, 120(6):e2214889120, 2023.DOI
[32]Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, et al. PubChem in 2021: new data content and improved web interfaces. Nucleic acids research, 49(D1):D1388–D1395, 2021.DOI
[33]Alexios Koutsoukas, Keith J Monaghan, Xiaoli Li, and Jun Huan. Deep-learning: investigating deep neural networks hyper-parameters and comparison of performance to shallow methods for modeling bioactivity data. Journal of cheminformatics, 9(1):1–13, 2017.
[34]Samuel Lampa, Jonathan Alvarsson, Staffan Arvidsson Mc Shane, Arvid Berg, Ernst Ahlberg, and Ola Spjuth. Predicting off-target binding profiles with confidence using conformal prediction. Frontiers in pharmacology, 9:1256, 2018.DOI
[35]Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111, 2018.arxiv.org/abs/1604.04173
[36]Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2021.arxiv.org/abs/2006.06138
[37]Martin Lindh, Anders Karlén, and Ulf Norinder. Predicting the rate of skin penetration using an aggregated conformal prediction framework. Molecular Pharmaceutics, 14(5):1571–1576, 2017.DOI
[38]Ricardo Macarron, Martyn N Banks, Dejan Bojanic, David J Burns, Dragan A Cirovic, Tina Garyantes, Darren VS Green, Robert P Hertzberg, William P Janzen, Jeff W Paslay, et al. Impact of high-throughput screening in biomedical research. Nature reviews Drug discovery, 10(3):188–195, 2011.DOI
[39]David Mary and Etienne Roquain. Semi-supervised multiple testing. arXiv preprint arXiv:2106.13501, 2021.DOI
[40]Malgorzata Mochol, Holger Wache, and Lyndon Nixon. Improving the accuracy of job search with semantic techniques. In International Conference on Business Information Systems, pages 301–313. Springer, 2007.DOI
[41]Kazem Rahimi, Derrick Bennett, Nathalie Conrad, Timothy M Williams, Joyee Basu, Jeremy Dwight, Mark Woodward, Anushka Patel, John McMurray, and Stephen MacMahon. Risk prediction in patients with heart failure: a systematic review and analysis. JACC: Heart Failure, 2(5):440–446, 2014.DOI
[42]Bradley Rava, Wenguang Sun, Gareth M James, and Xin Tong. A burden shared is a burden halved: A fairness-adjusted approach to classification. arXiv preprint arXiv:2110.05720, 2021.DOI
[43]Jonathan G Richens, Ciarán M Lee, and Saurabh Johri. Improving the accuracy of medical diagnosis with causal machine learning. Nature communications, 11(1):1–9, 2020.DOI
[45]Etienne Roquain and Nicolas Verzelen. False discovery rate control with unknown null distribution: Is it possible to mimic the oracle? The Annals of Statistics, 50(2):1095–1123, 2022.arxiv.org/abs/1912.03109
[47]Roshni Sahoo, Shengjia Zhao, Alyssa Chen, and Stefano Ermon. Reliable decisions with threshold calibration. Advances in Neural Information Processing Systems, 34, 2021.
[48]Clayton Scott, Gowtham Bellala, and Rebecca Willett. The false discovery rate for statistical pattern recognition. Electronic Journal of Statistics, 3:651–677, 2009.arxiv.org/abs/0901.4184
[49]Xiaojian Shao, Yingjie Tian, Lingyun Wu, Yong Wang, Ling Jing, and Naiyang Deng. Predicting dna-and rna-binding proteins from sequences with kernel methods. Journal of Theoretical Biology, 258(2):289–293, 2009.DOI
[50]Muhammad Ahmad Shehu and Faisal Saeed. An adaptive personnel selection model for recruitment using domain-driven data mining. Journal of Theoretical and Applied Information Technology, 91(1):117, 2016.
[51]Juwen Shen, Jian Zhang, Xiaomin Luo, Weiliang Zhu, Kunqian Yu, Kaixian Chen, Yixue Li, and Hualiang Jiang. Predicting protein–protein interactions based only on sequences information. Proceedings of the National Academy of Sciences, 104(11):4337–4341, 2007.pubmed.ncbi.nlm.nih.gov/17360525
[52]Li Shen, Laurie R Margolies, Joseph H Rothstein, Eugene Fluder, Russell McBride, and Weiva Sieh. Deep learning to improve breast cancer detection on screening mammography. Scientific reports, 9(1):1–12, 2019.DOI
[53]Roman Sink, Stanislav Gobec, S Pecar, and Anamarija Zega. False positives in the early stages of drug discovery. Current medicinal chemistry, 17(34):4231–4255, 2010.pubmed.ncbi.nlm.nih.gov/20939815
[54]John D Storey, Jonathan E Taylor, and David Siegmund. Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 66(1):187–205, 2004.DOI
[55]Fredrik Svensson, Ulf Norinder, and Andreas Bender. Improving screening efficiency through iterative screening using docking and conformal prediction. Journal of chemical information and modeling, 57(3):439–444, 2017.DOI
[56]Fredrik Svensson, Natalia Aniceto, Ulf Norinder, Isidro Cortes-Ciriano, Ola Spjuth, Lars Carlsson, and Andreas Bender. Conformal regression for quantitative structure–activity relationship modeling—quantifying prediction uncertainty. Journal of Chemical Information and Modeling, 58(5):1132–1140, 2018.DOI
[57]Paweł Szymański, Magdalena Markowicz, and Elżbieta Mikiciuk-Olasik. Adaptation of high-throughput screening in drug discovery—toxicological screening tests. International journal of molecular sciences, 13(1):427–452, 2011.DOI
[58]Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1904.06019
[59]Jessica Yamathevan, Dominic Clark, Paul Czodrowski, Ian Dunham, Edgardo Ferran, George Lee, Bin Li, Anant Madabhushi, Parantu Shah, Michaela Spitzer, et al. Applications of machine learning in drug discovery and development. Nature reviews Drug discovery, 18(6):463–477, 2019.
[60]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer Science & Business Media, 2005.
[61]Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. 1999.
[62]Lequn Wang, Thorsten Joachims, and Manuel Gomez Rodriguez. Improving screening processes via calibrated subset selection. arXiv preprint arXiv:2202.01147, 2022.arxiv.org/abs/2202.01147
[63]Asaf Weinstein and Aaditya Ramdas. Online control of the false coverage rate and false sign rate. In International Conference on Machine Learning, pages 10193–10202. PMLR, 2020.arxiv.org/abs/1905.01059