WorksComputer Science
Calibrated Uncertainty Quantification on TDC ADMET: What Can Be Resolved Across 22 Datasets?
Abstract
Absorption, distribution, metabolism, excretion, and toxicity (ADMET) models often rely on small, heterogeneous datasets, yet their predictions may determine which compounds proceed to experimental testing. Useful evaluations must therefore assess both predictive performance and the quality of reported probabilities or intervals. We ask which common uncertainty-quantification methods perform best on the 22 Therapeutics Data Commons (TDC) ADMET datasets and what this benchmark can resolve. We compare eleven calibrated pipelines using nested five-fold cross-validation across ten chemical partitions and five model seeds. Ensemble Monte Carlo (MC) dropout has the best average rank for TDC performance, Brier score, and 90% interval score. For TDC performance, however, it is not separated from the other ensemble pipelines; for Brier and interval score, LightGBM is also not ruled out as best. LightGBM wins more individual datasets than any other method for TDC performance and Brier score, but performs poorly on several endpoints. Smaller evaluations can identify a different leader and often fail to recover the broader comparison. These results make LightGBM and an ensemble neural model complementary baselines. Comparative claims should be checked across multiple chemical partitions using statistical analyses matched to the question being asked.
Introduction
ADMET models help prioritize compounds for synthesis and experimental testing. A useful model must therefore do more than make accurate predictions: it should also indicate when those predictions are uncertain. Reliable uncertainty estimates can help decide which compounds to test, where more data are needed, and when a prediction should not drive an experimental decision [18].
ADMET datasets are often small, and a more complicated model is not necessarily better. Fingerprint and descriptor models remain competitive across many small molecular datasets [30, 33], while predictability differs substantially across assays [2]. Uncertainty quantification (UQ) methods may likewise perform differently from one endpoint to another.
The comparison itself also varies. Chemical partitions, model initialization, and the data used for tuning can change the observed differences between both point predictors and UQ methods [5, 3]. This matters for the TDC ADMET benchmark, which contains 22 public prediction tasks with endpoint-specific performance measures and a fixed scaffold test split [20]. After our structure curation, the datasets contain 475 to 13,094 compounds. An audit of leading submissions found reproducibility failures, data leakage, and model
Preprint. choices influenced by the public test sets [23]. We therefore ask which methods perform consistently across datasets, which work best on particular endpoints, and whether those conclusions survive changes in the evaluation design.
We study eleven calibrated pipelines based on LightGBM and single or ensemble neural models on all 22 datasets, crossing ten chemical partitions, five model seeds, and five outer folds. This grid supports comparisons across datasets and on each dataset’s original score scale, followed by smaller-design stability analyses.
Our contributions are:
We compare eleven calibrated UQ pipelines on all 22 TDC ADMET datasets using nested cross-validation across multiple chemical partitions and model seeds.
We show that ensemble MC dropout is the most consistent method across datasets, while LightGBM wins many individual endpoints and performs poorly on others.
We examine how reduced split and tuning designs affect both the apparent leader and the differences supported by the statistical analysis.
Related work
Public ADMET benchmarks and data regimes. TDC brought established ADMET endpoints into a widely used collection with fixed scaffold splits [20]. A later audit of leading submissions found reproducibility, leakage, and test-set reuse problems [23]. Since TDC was introduced, newer resources have extended ADMET evaluation through stronger curation, blinded or prospective tests, and multitask structure. PharmaBench adds curation and experimental-condition extraction [29], while the ASAP–Polaris–OpenADMET challenge used blinded, temporally split measurements from an active lead-optimization programme [27]. The Biogen and ExpansionRx releases provide industry assay collections [10, 31]. These and public assay data also support multitask studies: Adrian et al. [2025] fine-tune models on Biogen and public assay data, while Xue et al. [2026] evaluate probabilistic pretraining on Biogen, ExpansionRx, and ChEMBL-MT.
Simple molecular representations as baselines. Recent molecular benchmarks increasingly include graph neural networks and pretrained molecular representations [26, 33]. Their gains over simpler representations are not uniform. ADMET predictability can vary more by assay than by algorithm or data volume [2], and MapLight obtained strong TDC results by combining fingerprints and physicochemical descriptors with boosted trees [30]. In a separate study across 25 classification datasets, most frozen pretrained molecular embeddings offered little or no improvement over an extended-connectivity fingerprint baseline [33]. A carefully tuned fingerprint model is therefore an important reference for small molecular datasets.
What molecular UQ evaluations measure. The relevant UQ evaluation depends on how the prediction will be used. Ranking-based evaluation is appropriate when the aim is to prioritize predictions or reject the least reliable ones. Decisions based on a numerical probability or interval instead require uncertainty on a meaningful scale. UNIQE makes this distinction explicit by separating ranking-based, calibration-based, and proper-score evaluations [25]. Proper scores reward calibration together with sharpness; Brier score and interval score are examples for probabilities and intervals [15]. Post-hoc transformations can also change the ordering of models, making the choice of calibration procedure part of the comparison design [34]. MUBen broadened molecular UQ comparisons across representations and both classification and regression datasets [26]. Other work has tested whether error-ranking performance persists under chemical and temporal shifts [32]. Results vary across tasks and measures [18], while ensembles have provided strong results in several molecular UQ benchmarks [36, 26].
Chemical splits and method comparison. Random splits tend to preserve close chemical analogues across training and test sets. Scaffold or similarity-based splits reduce structural overlap, while temporal splits can approximate prospective use when reliable dates are available [37] (Yang et al. 2019). Split, optimization, and other stochastic choices all contribute to benchmark variation [5, 3]. Ash et al. [ [3] recommend shared resamples and paired comparisons, generally using five-by-five repeated cross-validation (CV) for 500–100,000 molecules and five-by-two CV below 500. They allow a single split for very large datasets when repeated CV is computationally impractical. Their lower-cost hyperparameter protocol uses one training–validation split inside each repeated-CV training set rather than full inner CV.
The statistical procedure should follow the comparison being made. Across datasets, the Friedman test asks whether method ranks differ overall [8]. Nemenyi critical-difference diagrams provide a conventional all-pairs summary, while Holm-adjusted pairwise tests can give more power and adjusted -values for the chosen comparison family [14]. Within one dataset, Tukey’s honestly significant difference procedure is useful when every pair of method means is of interest [3]. Hsu’s multiple-comparisons-with-the-best (MCB) procedure focuses instead on which methods can be ruled out as best [19]. Cross-dataset ranks address consistency across tasks; comparisons on the original score scale address what can be distinguished on one task.
Benchmark and evaluation
Datasets and chemical partitions. We use all 22 TDC ADMET datasets as separate tasks: 13 binary and nine regression. Molecules are standardized to parent structures with the ChEMBL pipeline [4] and keyed by parent Standard InChIKey. We keep repeated measurements but assign all records for one compound to the same fold. Inputs concatenate 2048-dimensional radius-2 Morgan count fingerprints [35] and 200 CDF-normalized RDKit2D descriptors computed with DescriptaStorus 2.8.0 [22]. Every pipeline is fitted on the log target for four endpoints: vdss_lombardo, half_life_obach, clearance_microsome_az, and clearance_hepatocyte_az. Predictions are transformed back before scoring.
Five outer folds are formed from Taylor–Butina clusters at Tanimoto distance cutoff 0.5 [6]. For each outer fold, we recompute the clusters within the outer-training compounds and divide them into a dedicated calibration set and a model-development set. Complete clusters are added to the calibration set until it contains approximately 20% of the observations. Only the model-development set enters five-fold inner cross-validation. We average each configuration’s five TDC-score trajectories step by step, choose its best step on that mean curve, and select the configuration with the best resulting score. We refit it on the full model-development set and fit the post-hoc calibrator on the untouched calibration set.
Prediction pipelines. We compare LightGBM [21] with five neural pipelines: a deterministic multilayer perceptron (MLP), MC dropout [13], checkpoint-resumed stochastic weight averaging Gaussian (SWAG) [28], last-layer Laplace [7], and a post-hoc variational Bayesian last layer (VBLL) [17]. Their ten-member counterparts are deep ensemble [24], ensemble MC dropout [11], MultiSWAG [38], mixture of Laplace approximations (MoLA) [9], and ensemble post-hoc VBLL, giving eleven pipelines. The ten neural pipelines use the same MLP architecture, while LightGBM provides the alternative model family; all eleven use the same fingerprint-and-descriptor input.
LightGBM and the MLP backbones use binary cross-entropy for the 13 classification tasks, loss for the five MAE-scored regression tasks, and loss for the four Spearman-scored regression tasks. The post-hoc VBLL layer is fitted by minimizing the negative ELBO.
LightGBM and the deterministic MLP are selected from 32 presampled Sobol configurations using the mean inner-fold TDC metric. All neural UQ pipelines inherit the MLP architecture, hyperparameters, and training epoch selected for deterministic point prediction. MC dropout, SWAG, and Laplace use the selected checkpoint; post-hoc VBLL fits a variational last layer over its frozen features. For each feature map, we select among 81 last layers spanning nine prior and nine Wishart scales by the final ELBO on the model-development rows; ensemble post-hoc VBLL repeats this for every member. Ensembles combine ten independently initialized refits of the same selection. This holds backbone selection fixed across neural UQ constructions. Fits run 2,000 LightGBM rounds or 300 MLP epochs, with the deployed step selected from the inner validation trajectory.
Deterministic MLP and MC dropout use the same selected network and training-time dropout. At inference, deterministic MLP disables dropout and uses one forward pass, whereas MC dropout keeps dropout active and averages 30 stochastic passes. Starting from the selected checkpoint, SWAG runs ten epochs of stochastic gradient descent with momentum and collects up to four parameter snapshots per epoch from the first epoch. The resulting Gaussian approximation has diagonal-plus-rank-20 covariance and is evaluated with 30 posterior weight draws. MultiSWAG uses ten independently refitted SWAG members with equal weight, each evaluated with 30 posterior weight draws. For regression, its combined variance includes both the average uncertainty within members and the variance between member means. For binary prediction, we average the ten member probabilities after averaging each member’s 30 draws.
Calibration and evaluation measures. We estimate post-hoc calibration parameters from the dedicated calibration set. For binary tasks, we apply temperature scaling to the logit of each pipeline’s final marginal probability after averaging ensemble members [16]. We evaluate the resulting probabilities with Brier score, a proper score that reflects both calibration and refinement.
Regression requires different calibration procedures because the pipelines supply different forms of uncertainty. For LightGBM and the deterministic MLP, we estimate one constant residual standard deviation. For MC dropout, SWAG, deep ensemble, ensemble MC dropout, and MultiSWAG, we estimate an additional observation-noise variance and add it to the supplied predictive variance. For Laplace, post-hoc VBLL, MoLA, and ensemble post-hoc VBLL, we multiply the predictive standard deviation by a fitted non-negative factor. These adjusted distributions define central 90% intervals, evaluated by interval score. For the four log-trained endpoints, we estimate calibration parameters and form intervals on the log target, then exponentiate the interval limits and score them in the endpoint’s original units.
Our main measures are the endpoint-specific TDC metric, Brier score for binary tasks, and interval score for regression tasks. Repeated labels and predictions are averaged within compound for the TDC metrics; Brier and interval scores use individual observations. The appendix also reports AUROC, average precision, NLL, and ECE for binary tasks, and MAE, RMSE, Spearman correlation, Gaussian NLL, and coverage error for regression.
Full grid and comparison across datasets. For each dataset and method under each evaluation measure, we average exactly outer-fold scores: ten separately generated chemical partitions, five model seeds, and five outer folds. These 250 values estimate one score. We rank the eleven methods within each dataset, with rank 1 best and average ranks for exact score ties. Each dataset contributes one score and one rank to the cross-dataset comparison. The 1,826,000 from-scratch model fits are itemized in Appendix Table 4.
For each evaluation measure, the tie-corrected Friedman statistic with the Iman–Davenport approximation tests whether method ranks differ overall across datasets [8]. We then compare all 55 method pairs using two-sided normal tests of their mean-rank differences and apply Holm’s procedure to this fixed all-pairs family, controlling the family-wise error rate at [14]. We report the comparisons involving the method with the best average rank. A non-significant comparison means that the data do not rule out the alternative as best; it does not establish equal performance.
Per-dataset comparisons. Average ranks do not show what can be resolved on one endpoint, so we also compare methods on each dataset’s original score scale. We average the five model seeds and five outer folds within each chemical partition, giving 10 paired complete-partition scores per method. A repeated-measures analysis of variance first tests whether any method differs; if so, Hsu’s MCB procedure at the level assesses which methods can be ruled out as best [19]. We call the observed mean in the favorable direction the empirical best. Appendix § 10 gives the decision threshold and its coverage interpretation.
The ten partitions are generated separately but reuse the same finite set of compounds; their variation therefore measures sensitivity to the chemical partition rather than population-sampling uncertainty for new compounds. The analysis also assumes normally distributed errors and one residual variance across methods. Its -values should therefore be read as conditional on these assumptions, the observed datasets, and the completed evaluation grid.
Stability under smaller evaluation designs. We use finite-grid subsampling to ask how much of the completed evaluation is needed to recover its conclusions. We compare one held-out fold with one or more complete five-fold CVs, using either one model seed or the average of all five. Partition and fold levels are selected without replacement and shared across methods and datasets. Every reported design enumerates all distinct paired subsets. For each, we recalculate the empirical winners, mean ranks, Holm decisions, and, when at least two blocks are available, the per-dataset MCB conclusions. The resulting rates measure agreement with the completed grid.
Results
Across datasets, ensemble MC dropout has the best average rank for the TDC metric, calibrated Brier score, and calibrated interval score (Figure 1). The Friedman tests find clear differences among the methods for all three measures. The follow-up comparisons qualify the ranking: the group not separated from ensemble MC dropout contains all five ensemble pipelines for TDC performance. For Brier score, only the deterministic MLP and Laplace are separated from the leader. For interval score, the group not separated from the leader includes LightGBM, MC dropout, and all five ensemble pipelines.
Figure 1. Mean ranks across datasets, with rank 1 best and the empirical leader outlined. Marker shape distinguishes LightGBM, single MLP, and MLP ensemble pipelines. “Not significantly worse” marks methods that the Holm-adjusted comparison does not distinguish from the leader; “Significantly worse” marks methods that it does. Adjustment is performed separately for each evaluation measure across all 55 method pairs.
Appendix Table 3 gives every mean rank and adjusted -value.
LightGBM has the most empirical wins for both the TDC metric and Brier score, yet its performance is strongly endpoint dependent. Its TDC ranks range from first to last, producing a substantially worse average rank than ensemble MC dropout. For interval score, LightGBM and ensemble post-hoc
VBLL each win three datasets, while ensemble MC dropout ranks between first and fourth on every regression dataset.
What individual datasets can distinguish. Average ranks summarize consistency across the suite; comparisons on the original score scale show which methods the repeated evaluations can distinguish on each endpoint. Figure 2 makes the contrast clear. LightGBM is often the sole method not ruled out as best, but it is also ruled out on many other endpoints. Ensemble MC dropout wins fewer datasets but is not ruled out as best across a broader range of them. The evaluations find some method difference for all 44 dataset-score combinations.
Figure 2. Per-dataset classifications from Hsu’s MCB procedure after averaging the five model seeds and five outer folds within each of ten chemical partitions, giving ten paired complete-partition blocks. “Statistically best” means that the empirical winner is the only method not ruled out by MCB. “Empirical best” marks other observed winners, and “Not ruled out as best” marks additional methods not excluded by the test. The initial test finds a method difference for all 44 dataset-score combinations; each is assessed separately at the 5% level, with the coverage interpretation given in Appendix D.
Appendix § 10 shows the corresponding scores and decisions for every dataset.
Evaluation-design stability. With one model seed, a single held-out fold selects the same leader as the completed grid in only 37.2% of TDC-metric, 20.4% of Brier, and 39.2% of interval-score subsets (Figure 3). The apparent leader from one fold may therefore differ from the leader obtained after averaging the full evaluation. At the equal cost of five scores, one complete cross-validation reproduces the full-grid rank ordering more closely than averaging five model seeds on one held-out fold. Repeated complete cross-validations narrow the rank variation, with Brier score settling most slowly.
Figure 3. Suite-wide mean ranks under smaller evaluation designs. Open circles and bars show the median, interquartile range, and 5th–95th percentile range over paired finite-grid subsets; diamonds show the completed grid. Solid branches add folds and partitions, while dashed branches average model seeds. Percentages give the subsets that recover the sole full-grid leader. These empirical ranges are not confidence intervals.
Tuning with a single validation split. To test whether a single train–validation split is sufficient for tuning, we compare its selected configuration with the choice from all five inner folds. Median agreement is 19.2% for LightGBM and 17.8% for the MLP across datasets, and is still incomplete when four folds are used. The one-fold choices also score worse when evaluated across all five validation folds (Appendix Table 6).
Discussion
Baseline choice. Ensemble MC dropout provides a consistently strong neural reference, but its best average rank does not establish an advantage over the other ensemble pipelines. LightGBM’s many individual wins and poor results elsewhere show why consistency across a benchmark and performance on a particular assay can differ. The per-dataset comparisons identify where one method has a clear advantage and where several alternatives cannot be ruled out as best. A comparison using only one baseline family can therefore give an incomplete picture of a new method’s performance. Both LightGBM and an ensemble neural baseline should be included when evaluating new molecular predictors, including pretrained models.
Variation across evaluations. Evaluation introduces uncertainty of its own. Changing which compounds enter training and testing can change the apparent leader, and averaging model seeds on one fold leaves variation between chemical partitions unexplored. At the same number of fold-score evaluations, completing cross-validation gives a more stable full rank ordering than repeating seeds on one fold. A compact evaluation can help screen methods; a claim of improvement should be checked across multiple chemical partitions. Paired comparisons account for the variation in method differences, while multiplicity adjustment limits spurious claims of a difference when several methods are compared.
Shared training compounds weaken the independence assumption behind the per-dataset tests. Some endpoint families also share compounds, and only nine datasets contribute to the cross-dataset interval-score comparison. Its unresolved differences deserve caution: the absence of a detected difference gives little reason to prefer one method over another on statistical grounds alone.
Tuning and computational cost. Validation data affect the comparison before test scores are calculated. A single validation split often selects a different configuration from five-fold selection, and those choices perform worse across the five validation folds. Authors proposing a new model should give simple baselines a fair chance through careful hyperparameter and stopping-point selection. Their competitiveness here makes that tuning part of establishing an improvement, including for expensive pretrained architectures.
Ensembles require additional model fits, while other UQ methods add posterior sampling or last-layer optimization. We made the full benchmark feasible by sharing MLP hyperparameter and checkpoint selection across UQ methods. These choices optimize point prediction. Selecting the final ensemble or using a calibrated validation score could favor different hyperparameters or stopping epochs [34, 12]. The rankings consequently describe the complete pipelines evaluated here, including their selection procedures and additional computation.
Using the uncertainty estimates. The relevant score depends on what the predictions will be used for. Brier score assesses probabilities through both calibration and refinement without separating epistemic and aleatoric uncertainty. The regression results depend on the calibration maps and Gaussian assumptions used to construct the intervals. An application that uses uncertainty mainly to rank candidates would also need to assess that ranking.
Claims of improvement should be supported by comparisons with both carefully tuned baselines. Reporting the size of the gain and its variation across chemical partitions helps readers judge its practical value.
Code and data availability. Code and data accompanying this study will be released before the camera-ready submission.
Acknowledgements. The authors acknowledge support from the Novo Nordisk Foundation under grant no. NNF22OC0076658 (Bayesian neural networks for molecular discovery). We acknowledge the Danish e-infrastructure Consortium (DeiC) for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium.
Disclosure. Large language model assistance was used for prose refinement and code writing. The authors reviewed the resulting material.
Calibration procedures and results
Every calibrator is fitted on the dedicated calibration holdout inside the corresponding outer training split. Table 1 summarizes how each form of uncertainty is calibrated. Figure 10, Figure 14, and Figure 15 show Brier score, binary NLL, and ECE by dataset. Figure 11, Figure 19, and Figure 20 show interval score, Gaussian NLL, and coverage error for regression.
| Pipelines | Supplied uncertainty | Calibration applied |
| All eleven binary pipelines | Marginal event probability | One scalar temperature |
| LightGBM; deterministic MLP | Point prediction only | Constant Gaussian residual scale |
| MC dropout; SWAG; deep ensemble; ensemble MC dropout; MultiSWAG | Stochastic-pass, posterior-draw, or ensemble-member spread | Additive observation-noise variance |
| Laplace; post-hoc VBLL; MoLA; ensemble post-hoc VBLL | Total predictive standard deviation | Standard-deviation multiplier |
Table 1. Calibration applied to each pipeline. A supplied scale is a standard deviation on the target used to train the model.

Figure 10. Full-grid temperature-calibrated Brier scores for all 13 binary ADMET datasets. Lower is better. Marks and the decision region are defined as in Figure 9. A common scalar temperature is fitted on each dedicated calibration holdout.

Figure 9. Full-grid TDC leaderboard metric values for all 22 ADMET datasets. The arrow below each heading gives the metric direction. Filled circles mark the empirical best, open circles mark other methods not ruled out as best, and crosses mark methods classified as significantly worse. The pale region shows the MCB decision threshold described above.
Figure 11. Full-grid method-specifically calibrated 90% interval scores for all nine regression ADMET datasets. Lower is better. Marks and the decision region are defined as in Figure 9. The horizontal axes differ because interval score is reported in the original units of each endpoint.
Figure 14. Temperature-calibrated binary negative log-likelihood on all 13 binary datasets. Lower is better. Each marker is the mean of 250 outer-fold scores. Outlined markers identify empirical bests.
Figure 15. Temperature-calibrated 10-bin expected calibration error on all 13 binary datasets. Lower is better. Each marker is the mean of 250 outer-fold scores. Outlined markers identify empirical bests.
Figure 19. Gaussian negative log-likelihood on all nine regression datasets under the fixed method-specific calibration map. Lower is better. Each marker is the mean of 250 outer-fold scores. Outlined markers identify empirical bests.
Figure 20. Absolute 90% coverage error, , where PICP is the prediction interval coverage probability, on all nine regression datasets under the fixed method-specific calibration map. Lower is better. Each marker is the mean of 250 outer-fold scores. Outlined markers identify empirical bests.
For binary predictions, we choose the temperature that minimizes NLL on the calibration set and divide the probability logits by this value. If the calibration labels cannot support a finite temperature estimate, we set the temperature to one and keep the original probabilities.
For regression pipelines with only a point prediction, the fitted constant scale is the root-mean-square calibration residual. For pipelines with a predictive standard deviation, the scale multiplier is the root-mean-square residual after division by that standard deviation. For pipelines supplying epistemic spread, we choose a non-negative observation-noise variance by minimizing Gaussian NLL on the calibration set and add it to the supplied variance. Central 90% intervals extend 1.645 adjusted standard deviations on either side of the mean.
The four log-trained endpoints are calibrated on the log target. Exponentiating the interval limits produces asymmetric intervals in the original units used for scoring. The appendix NLL values for these endpoints use the corresponding log-normal density in the original units.
Figure 4 and Figure 5 show the full-grid mean score for each method and endpoint before and after calibration. Each point averages 250 outer-fold evaluations. LightGBM and the deterministic MLP supply only point predictions for regression, so their interval-score panels begin after the constant residual scale is fitted.
Figure 4. Full-grid mean Brier scores before and after temperature scaling for all 13 binary datasets and eleven pipelines. Open markers show uncalibrated scores and filled markers show calibrated scores. Lower is better.
Figure 5. Full-grid mean 90% interval scores before and after method-specific calibration for all nine regression datasets. Open markers show native intervals and filled markers show calibrated intervals. LightGBM and deterministic MLP supply no native interval, so only their calibrated scores are shown. Axes are endpoint-specific; lower is better.
Global rank comparisons
The Friedman test finds differences among methods for all three evaluation measures (Table 2).
| Evaluation measure | Datasets | -value | |
| TDC leaderboard | 22 | 107.13 | |
| Calibrated Brier | 13 | 45.75 | |
| Calibrated interval score | 9 | 54.40 |
Table 2. Cross-dataset Friedman tests. is the tie-corrected Friedman statistic.
Table 3 gives every mean rank and the Holm-adjusted comparison with ensemble MC dropout. The adjustment includes all 55 pairwise comparisons within each evaluation measure.
| Method | TDC leaderboard Rank | TDC leaderboard | Brier Rank | Brier | Interval score Rank | Interval score |
| LightGBM | 6.36 | 0.0097 | 4.23 | 1.0000 | 3.56 | 1.0000 |
| Deterministic MLP | 8.70 | < 0.0001 | 8.38 | 0.0101 | 10.33 | < 0.0001 |
| MC dropout | 7.91 | < 0.0001 | 7.69 | 0.0690 | 5.67 | 1.0000 |
| SWAG | 7.32 | 0.0002 | 7.38 | 0.1431 | 7.78 | 0.0486 |
| Laplace | 8.16 | < 0.0001 | 9.23 | 0.0007 | 9.22 | 0.0014 |
| Post-hoc VBLL | 8.82 | < 0.0001 | 6.46 | 0.9364 | 8.33 | 0.0142 |
| Deep ensemble | 4.20 | 1.0000 | 4.54 | 1.0000 | 6.00 | 1.0000 |
| Ensemble MC dropout | 2.73 | – | 3.54 | – | 2.67 | – |
| MultiSWAG | 3.86 | 1.0000 | 5.46 | 1.0000 | 5.00 | 1.0000 |
| MoLA | 3.20 | 1.0000 | 5.46 | 1.0000 | 3.44 | 1.0000 |
| Ensemble post-hoc VBLL | 4.73 | 1.0000 | 3.62 | 1.0000 | 4.00 | 1.0000 |
Table 3. Mean ranks and Holm-adjusted comparisons with ensemble MC dropout. Bold adjusted -values indicate a significant difference at the 5% level. The adjustment accounts for all 55 pairwise method comparisons within each evaluation measure.
Computational scale
Let denote the number of dataset–partition–model–seed–outer-fold units. Table 4 counts shared neural sweeps and ensemble backbones once, irrespective of how many UQ readouts they produce.
| Computation | Count |
| LightGBM inner-CV fits | 880,000 |
| LightGBM outer refits | 5,500 |
| Shared MLP inner-CV fits | 880,000 |
| Single-model MLP outer refits | 5,500 |
| Ensemble-member MLP refits | 55,000 |
| From-scratch model fits | 1,826,000 |
| Checkpoint-resumed SWAG continuations | 60,500 |
| Post-hoc VBLL last-layer grid-point fits | 4,900,500 |
| Calibration applications | 60,500 |
| Outer-fold method evaluations | 60,500 |
| Score evaluations across the three main measures | 121,000 |
Table 4. Computation underlying the eleven-pipeline comparison.
The 55,000 ensemble-member entries are full MLP refits. The larger post-hoc VBLL count consists of 81 last-layer optimizations for each of 60,500 frozen feature maps, comprising 5,500 single models
and 55,000 ensemble members. Each optimization fits one prior-scale and Wishart-scale pair without retraining the MLP backbone.
Additional evaluation measures
Figure 6–Figure 8 show mean ranks for the additional measures listed in § 3. Each score averages the same 250 outer-fold values as the main analysis. The corresponding values on each dataset’s original scale appear in Figure 12–Figure 20.
Figure 6. Mean ranks for five binary evaluation measures after temperature scaling. Each measure contains the same 13 datasets and eleven methods. The outlined marker identifies the lowest observed mean rank; coincident points have equal mean ranks.
Figure 8. Mean ranks for interval score, Gaussian NLL, and absolute coverage error after method-specific regression calibration. Each measure contains the same nine datasets and eleven methods. The outlined marker identifies the lowest observed mean rank; coincident points have equal mean ranks.
Figure 12. Temperature-calibrated AUROC on all 13 binary datasets. Higher is better. Each marker is the mean of 250 outer-fold scores. Outlined markers identify empirical bests.
Figure 7. Mean ranks for regression point-prediction measures. Each measure contains the same nine datasets and eleven methods. The outlined marker identifies the lowest observed mean rank; coincident points have equal mean ranks.
Hyperparameter optimization procedure
Search allocation. LightGBM and the deterministic MLP are selected from 32 configurations generated by a scrambled Sobol sequence with seed 0. Each configuration is evaluated on all five inner folds, and selection uses the mean inner-fold TDC metric and the fold-averaged validation curve. MC dropout, SWAG, and Laplace inherit the deterministic MLP selection. For post-hoc VBLL, the nine prior scales are logarithmically spaced from to and the nine Wishart scales from to 1. Each of the 81 pairs is fitted for 1,000 full-batch Adam steps on the model-development rows, and the pair with the largest final ELBO is used. Each ensemble inherits the selection of its corresponding single-model pipeline.
Table 5: Hyperparameter search spaces for LightGBM and the deterministic MLP. Uniform and log-uniform dimensions are mapped from the Sobol points; discrete levels receive equal intervals of the unit range.
| Model | Hyperparameter | Sampling | Range or levels |
| LightGBM | learning_rate | Log-uniform | |
| LightGBM | num_leaves | Log-uniform integer | |
| LightGBM | min_child_samples | Uniform integer | |
| LightGBM | feature_fraction | Uniform | |
| LightGBM | bagging_fraction | Uniform | |
| LightGBM | lambda_l1 | Log-uniform | |
| LightGBM | lambda_l2 | Log-uniform | |
| MLP | n_layers | Discrete | |
| MLP | hidden_dim | Powers of two | |
| MLP | dropout | Uniform | |
| MLP | learning_rate | Log-uniform | |
| MLP | weight_decay | Log-uniform | |
| MLP | batch_size | Powers of two |
Table 5.
Stability of inner-fold selection. Using the stored inner-validation trajectories, we recalculated model selection for every LightGBM and deterministic-MLP outer-fold fit in the grid. We compared the hyperparameters and training iteration selected from one or four inner folds with those selected from all five. Table 6 reports equal-dataset medians over all 22 datasets. A single fold selects the same configuration as all five folds in 19.2% of LightGBM cases and 17.8% of MLP cases. Using four folds raises these rates to 67.2% and 59.3%. Across the five individual folds, the median outer evaluation contains four distinct configurations and five distinct configuration–iteration pairs for both models. Raw differences between stopping iterations are not reported because most folds also choose a different configuration and therefore a different training trajectory.
| Model | One inner fold: Config. agree | One inner fold: Joint agree | One inner fold: Relative loss | Four inner folds: Config. agree | Four inner folds: Joint agree | Four inner folds: Relative loss |
| LightGBM | 19.2% | 1.0% | 0.911 | 67.2% | 24.3% | 0.027 |
| MLP | 17.8% | 2.6% | 1.281 | 59.3% | 36.0% | 0.056 |
Table 6.
Table 6: Stability of inner-CV selection relative to the choice made from all five folds. Values are medians of the 22 dataset-level summaries. Joint agreement requires both the configuration and stopping iteration to match. Relative loss divides the five-fold validation-objective loss by the gap between the best and median configuration within the same sweep.
Subsequent refitting used the configuration selected by all five folds. The table therefore measures how often reduced inner-CV designs reproduce that selection. The stored results do not compare the outer-test performance of their alternative choices.
Scores by dataset
The figures show every dataset and all eleven methods on the original metric scale. Each score marker is the mean of 250 outer-fold scores. For MCB, we average the five model seeds and five outer folds within each chemical partition, giving 10 paired complete-partition scores. For residual mean square (MSE), the one-step Hsu decision threshold is , where is the corresponding 95% MCB critical value for eleven methods with 90 residual degrees of freedom. The pale region starts at the empirical winner and extends by toward worse scores. A method inside the region cannot be ruled out as best; a method beyond its boundary is classified as significantly worse by Hsu’s test.
Under the assumed block model, Hsu’s selection rule includes at least one method with the best population mean score with probability at least 95% [19 p. 1141]. Excluding any particular method is a 5%-level test of whether it performs worse than the best alternative. If several methods share the optimal population mean, the 95% guarantee concerns including a best method, not simultaneously including all of them.
Additional metrics on their original scales
The following figures complement the mean-rank rails with the underlying full-grid metric values. They use the same fixed method order and pairing as the three formal score sheets. Outlined markers identify empirical bests. Horizontal axes are fitted separately for each dataset because several regression metrics have different endpoint units and scales.
Figure 13. Temperature-calibrated average precision on all 13 binary datasets. Higher is better. Each marker is the mean of 250 outer-fold scores. Outlined markers identify empirical bests.
Figure 16. Mean absolute error on all nine regression datasets under the fixed method-specific calibration pipelines. Lower is better. Each marker is the mean of 250 outer-fold scores. The outlined markers identify empirical bests.
Figure 17. Root mean squared error on all nine regression datasets under the fixed method-specific calibration pipelines. Lower is better. Each marker is the mean of 250 outer-fold scores. Outlined markers identify empirical bests.
Figure 18. Spearman correlation on all nine regression datasets under the fixed method-specific calibration pipelines. Higher is better. Each marker is the mean of 250 outer-fold scores. The outlined markers identify empirical bests.
References
- [2]Stevan Aleksić, Daniel Seeliger, and J. B. Brown. ADMET predictability at Boehringer Ingelheim: State-of-the-art, and do bigger datasets or algorithms make a difference? Molecular Informatics, 41(2):e2100113, 2022. doi: 10.1002/minf.202100113. URL https://doi.org/10.1002/minf.202100113.
- [3]Jeremy R. Ash, Cas Wognum, Raquel Rodríguez-Pérez, Matteo Aldeghi, Alan C. Cheng, Djork-Arné Clever, Ola Engkvist, Cheng Fang, Daniel J. Price, Jacqueline M. Hughes-Oliver, and W. Patrick Walters. Practically significant method comparison protocols for machine learning in small molecule drug discovery. Journal of Chemical Information and Modeling, 65(18):9398–9411, 2025. doi: 10.1021/acs.jcim.5c01609. URL https://doi.org/10.1021/acs.jcim.5c01609.
- [4]A. Patrícia Bento, Anne Hersey, Eloy Félix, Greg Landrum, Anna Gaulton, Francis Atkinson, Louisa J. Bellis, Marleen De Veij, and Andrew R. Leach. An open source chemical structure curation pipeline using RDKit. Journal of Cheminformatics, 12:51, 2020. doi: 10.1186/s13321-020-00456-1. URL https://doi.org/10.1186/s13321-020-00456-1.
- [5]Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Gaël Varoquaux, and Pascal Vincent. Accounting for variance in machine learning benchmarks. In Proceedings of Machine Learning and Systems, volume 3, pages 747–769, 2021. URL https://proceedings.mlsys.org/paper_files/paper/2021/file/0184b0cd3cfb185989f858a1d9f5c1eb-Paper.pdf.arxiv.org/abs/2103.03098
- [6]Darko Butina. Unsupervised data base clustering based on Daylight’s fingerprint and Tanimoto similarity: A fast and automated way to cluster small and large data sets. Journal of Chemical Information and Computer Sciences, 39(4):747–750, 1999. doi: 10.1021/ci9803381. URL https://doi.org/10.1021/ci9803381.
- [7]Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace Redux – effortless Bayesian deep learning. In Advances in Neural Information Processing Systems, volume 34, pages 20089–20103. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/hash/a7c9585703d275249f30a088cebba0ad-Abstract.html.arxiv.org/abs/2106.14806
- [8]Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(1):1–30, 2006. URL https://www.jmlr.org/papers/v7/demsar06a.html.
- [9]Runa Eschenhagen, Erik Daxberger, Philipp Hennig, and Agustinus Kristiadi. Mixtures of Laplace approximations for improved post-hoc uncertainty in deep learning. In Bayesian Deep Learning Workshop at NeurIPS, 2021. URL https://bayesiandeeplearning.org/2021/papers/34.pdf.arxiv.org/abs/2111.03577
- [10]Cheng Fang, Ye Wang, Richard Grater, Sudarshan Kapadnis, Cheryl Black, Patrick Trapa, and Simone Sciabolоa. Prospective validation of machine learning algorithms for absorption, distribution, metabolism, and excretion prediction: An industrial perspective. Journal of Chemical Information and Modeling, 63(11):3263–3274, 2023. doi: 10.1021/acs.jcim.3c00160. URL https://doi.org/10.1021/acs.jcim.3c00160.
- [11]Angelos Filos, Sebastian Farquhar, Aidan N. Gomez, Tim G. J. Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud de Kroon, and Yarin Gal. A systematic comparison of Bayesian deep learning robustness in diabetic retinopathy tasks. In Bayesian Deep Learning Workshop at NeurIPS, 2019. URL https://bayesiandeeplearning.org/2019/papers/12.pdf.arxiv.org/abs/1912.10481
- [12]Laurits Fredsgaard and Mikkel N. Schmidt. On joint regularization and calibration in deep ensembles. Transactions on Machine Learning Research, 2025. URL https://openreview.net/forum?id=6xqV7DP3Ep.arxiv.org/abs/2511.04160
- [13]Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050–1059, 2016. URL https://proceedings.mlr.press/v48/gal16.html.arxiv.org/abs/1506.02142
- [14]Salvador García and Francisco Herrera. An extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons. Journal of Machine Learning Research, 9(89):2677–2694, 2008. URL https://jmlr.org/papers/v9/garcia08a.html.
- [15]Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. doi: 10.1198/016214507000001437.
- [16]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330, 2017. URL https://proceedings.mlr.press/v70/guo17a.html.arxiv.org/abs/1706.04599
- [17]James Harrison, John Willes, and Jasper Snoek. Variational Bayesian last layers. In International Conference on Learning Representations, pages 54251–54280, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/ee56aa4fe26a189782f507d843fd5272-Abstract-Conference.html.arxiv.org/abs/2404.11599
- [18]Lior Hirschfeld, Kyle Swanson, Kevin Yang, Regina Barzilay, and Connor W. Coley. Uncertainty quantification using neural networks for molecular property prediction. Journal of Chemical Information and Modeling, 60(8):3770–3780, 2020. doi: 10.1021/acs.jcim.0c00502.DOI
- [19]Jason C. Hsu. Constrained simultaneous confidence intervals for multiple comparisons with the best. The Annals of Statistics, 12(3):1136–1144, 1984. doi: 10.1214/aos/1176346732. URL https://doi.org/10.1214/aos/1176346732.
- [20]Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W. Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics Data Commons: Machine learning datasets and tasks for drug discovery and development. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/hash/4c56ff4ce4aaf9573aa5dff913df997a-Abstract-round1.html.arxiv.org/abs/2102.09548
- [21]Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/hash/6449f44a102fde848669badd9ebb67b6fa-Abstract.html.
- [22]Brian Kelley. DescriptaStorus. Software, version 2.8.0, 2024. URL https://pypi.org/project/descriptastorus/2.8.0/.
- [23]Ihor Koleiev, Roman Stratiiсhuk, Nazar Shevchuk, Mykola Melnychenko, Alex Nyporko, Daniil Todoryshyn, Vladyslav Husak, Sergii Starosyla, Semen Yesylevskyy, and Alan Nafiev. An end-user audit of reproducibility, data leakage, and overfitting of the top-ranked ADMET prediction models in TDC leaderboards. Journal of Chemical Information and Modeling, 66(14):8045–8058, 2026. doi: 10.1021/acs.jcim.6c00819. URL https://doi.org/10.1021/acs.jcim.6c00819.
- [24]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://papers.nips.cc/paper_files/paper/2017/hash/9ef2ed4b7fd2c810847ffa5fa85bce38-Abstract.html.arxiv.org/abs/1612.01474
- [25]Jessica Lanini, Minh Tam Davide Huynh, Gaetano Scebbа, Nadine Schneider, and Raquel Rodríguez-Pérez. UNIQUE: A framework for uncertainty quantification benchmarking. Journal of Chemical Information and Modeling, 64(22):8379–8386, 2024. doi: 10.1021/acs.jcim.4c01578. URL https://doi.org/10.1021/acs.jcim.4c01578.
- [26]Yinghao Li, Lingkai Kong, Yuanqi Du, Yue Yu, Yuchen Zhuang, Wenhao Mu, and Chao Zhang. MUBen: Benchmarking the uncertainty of molecular representation models. Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=qYceEhmG4.arxiv.org/abs/2306.10060
- [27]Hugo MacDermott-Opeskin, Jenke Scheen, Cas Wognum, et al. A computational community blind challenge on pan-coronavirus drug discovery data. Journal of Chemical Information and Modeling, 66(6):3129–3149, 2026. doi: 10.1021/acs.jcim.5c02106. URL https://doi.org/10.1021/acs.jcim.5c02106.
- [28]Wesley J. Maddox, Pavel Izmailov, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://papers.nips.cc/paper_files/paper/2019/hash/118921efba23fc329e6560b27861f0c2-Abstract.html.arxiv.org/abs/1902.02476
- [29]Zhangming Niu, Xianglu Xiao, Wenfan Wu, Qiwei Cai, Yinghui Jiang, Wangzhen Jin, Minhao Wang, Guojian Yang, Lingkang Kong, Xurui Jin, Guang Yang, and Hongming Chen. PharmaBench: Enhancing ADMET benchmarks with large language models. Scientific Data, 11:985, 2024. doi: 10.1038/s41597-024-03793-0. URL https://doi.org/10.1038/s41597-024-03793-0.
- [30]James H. Notwell and Michael W. Wood. ADMET property prediction through combinations of molecular fingerprints. arXiv preprint arXiv:2310.00174, 2023. doi: 10.48550/arXiv.2310.00174. URL https://arxiv.org/abs/2310.00174.
- [31]Open ADMET Consortium. openadmet-expansionrx-challenge-data. Hugging Face dataset, version 6b898cc, 2026. URL https://huggingface.co/datasets/openadmet/openadmet-expansionrx-challenge-data.
- [32]Raquel Parrondo-Pizarro, Jessica Lanini, and Raquel Rodríguez-Pérez. Uncertainty quantification in molecular machine learning for property predictions under data shifts. Journal of Chemical Information and Modeling, 66(6):923–935, 2026. doi: 10.1021/acs.jcim.5c02381. URL https://doi.org/10.1021/acs.jcim.5c02381.
- [33]Mateusz Praski, Jakub Adamczyk, and Wojciech Czech. Benchmarking pretrained molecular embedding models for molecular representation learning. arXiv preprint arXiv:2508.06199, 2025. doi: 10.48550/arXiv.2508.06199. URL https://arxiv.org/abs/2508.06199.
- [34]Rishabh Ranjan, Saurabh Garg, Mrigank Raman, Carlos Guestrin, and Zachary Lipton. Post-hoc reversal: Are we selecting models prematurely? In Advances in Neural Information Processing Systems, volume 37, pages 91460–91491, 2024. doi: 10.52202/1090717.2903. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/a6805b5546bd8d813a81c4b5a97e5ca6-Abstract-Conference.html.arxiv.org/abs/2404.07815
- [35]David Rogers and Mathew Hahn. Extended-connectivity fingerprints. Journal of Chemical Information and Modeling, 50(5):742–754, 2010. doi: 10.1021/ci100050t. URL https://doi.org/10.1021/ci100050t.
- [36]Gabriele Scalia, Colin A. Grambow, Barbara Pernici, Yi-Pei Li, and William H. Green. Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction. Journal of Chemical Information and Modeling, 60(6):2697–2717, 2020. doi: 10.1021/acs.jcim.9b00975. URL https://doi.org/10.1021/acs.jcim.9b00975.
- [37]Robert P. Sheridan. Time-split cross-validation as a method for estimating the goodness of prospective prediction. Journal of Chemical Information and Modeling, 53(4):783–790, 2013. doi: 10.1021/ci400084k. URL https://doi.org/10.1021/ci400084k.
- [38]Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. In Advances in Neural Information Processing Systems, volume 33, pages 4697–4708. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/hash/322f62469c5e3c7dc3e58f5a4d1ea399-Abstract.html.arxiv.org/abs/2002.08791

















