WorksComputer Science
Do LLMs Know When They Are Wrong About Molecules? In-Context Property Prediction with Self-Reported Uncertainty
Abstract
Large language models (LLMs) are used in many domains, but molecular property prediction with them is still little explored. In this work we evaluate the capability of 7 frontier LLMs to predict molecular properties together with a self-reported uncertainty from in-context examples, and compare them with classical baselines trained on the same molecules. We use 4 datasets and screen each for contamination of the training data of the LLMs: three are clean, and the fourth, ESOL, is memorised by most of the models and serves as a comparison. On the clean datasets every LLM improves with the context, at 30 examples the most accurate one beats every baseline. The requested reasoning effort changes accuracy little. The self-reported uncertainty ranks the prediction errors weakly, with Spearman rank correlations of up to 0.46, about the level reached by the uncertainties of the baselines. Combining uncertainty and prediction, on lipophilicity the most certain half of the best LLM is the most accurate at small context sizes; on BACE and antiviral, and on lipophilicity at large context sizes, the best LLMs are on par with the best baseline. On the contaminated ESOL dataset the LLMs show no improvement with the context size, as expected for a memorised dataset, and their stated uncertainty is smaller on the molecules whose value they recall.
*Corresponding authors: [email protected], [email protected]
Introduction
Large language models (LLMs) predict molecular properties in context from a Simplified Molecular Input Line Entry System (SMILES) string, and in several studies they reach the accuracy of baseline models trained on the same small datasets [6, 19, 26, 39, 46]. Outside chemistry, Vacareanu et al. [57] found that LLMs given in-context examples solve synthetic regression tasks about as well as random forests and gradient boosting, and Requeima et al. [50] elicited numerical predictive distributions from them. An in-context regressor needs no training, reads a molecule in the notation chemists use, and can be asked in the same prompt how uncertain its prediction is.
Uncertainty matters specifically in two scenarios. The first is active learning, where an uncertainty selects the samples to label. Ideally, the samples with the highest prediction error should be taken, but without label, the prediction error is unknown. The second is selective prediction [14], the use of predictions without measuring them: the most certain predictions are kept, and the error of the kept fraction measures accuracy and ranking together. For both purposes, the uncertainty must rank the actual prediction errors well; whether the magnitude of the stated uncertainty also matches the observed errors, its calibration [32], is a separate property that this study does not assess. For classical molecular models ranking and calibration have been compared for ensembles, sampling and evidential methods [24, 51], and stated confidences of language models have been found over-confident on text and on chemistry questions [41, 60]. An uncertainty benchmark needs data the models cannot recall, because a recalled value is predicted with almost no error and nothing requires the stated uncertainty to register that it was recalled rather than predicted [7, 8].
In-context regression has a basis outside chemistry: transformers trained on function classes recover linear and tree-like functions from the examples in their prompt [2, 13], and pre-trained LLMs forecast time series [18] and classify tabular rows [20] from text alone, with an accuracy that grows with the number of examples [1]. In chemistry, LLMs have served as few-shot predictors of molecular properties [19, 26, 39], as generators of rules that feed a classical model [61], and as surrogates for Bayesian optimisation over molecules and catalysts [42, 47]; fine-tuned LLM regressors, however, trail dedicated models on materials data [27], and LLM surrogates help Bayesian optimisation mainly after adaptation to the domain [30]. These studies compare accuracy, mostly at one training-set size; the learning curve over the number of examples and the overlap of the test molecules with the pre-training data are rarely reported.
Work on the uncertainty of LLMs has concentrated on classification and question answering [15]: models can be taught to state their confidence in words [37], the stated confidence is uninformative [55] but often too high [60, 62], prompting alone yields a weaker uncertainty than a short fine-tuning on graded answers [29], and the disagreement between repeated samples is an alternative to self-report [31, 40]. For numerical prediction the uncertainty has so far been read from token likelihoods or sampled outputs [18, 47, 50], and the one molecular study we are aware of uses sampling on classification tasks [9]. The uncertainties of classical molecular models, in contrast, have been benchmarked across methods [36], with the recommendation to judge them together with the accuracy of the model [21, 56], and they are used for selective prediction [28] and for choosing the next experiment [23, 49]. A like-for-like comparison of the self-reported of an in-context regressor with classical uncertainties on the same molecules, judged by these two uses over the number of examples and on data the models have not seen, has not been made; it is the comparison of this study.
We evaluate 7 frontier language models on 4 molecular regression datasets: lipophilicity, BACE-1 inhibition, inhibition of the SARS-CoV-2 main protease from a challenge of 2025, and ESOL. The test molecules of the first three occur in none of seven open pre-training corpora, and a zero-shot screen finds only insignificant retrieval of their values by any panel model, whereas ESOL is retrieved by most and serves as the control. Each model predicts 150 test molecules per dataset with a stated standard deviation from nested contexts of 30 to 1,000 training molecules, in three calls per molecule and, at the small context sizes, in three independent context draws; eight classical methods are trained on exactly the same molecules, and six of the models are also run along a ladder of reasoning efforts. We analyze how accurate the models are over context sizes and reasoning levels, which uncertainty ranks the prediction errors best, as needed for active learning, how accurate the most certain predictions are, as needed for selective prediction, and, with the contaminated ESOL dataset as the comparison, how a memorised dataset behaves on the same measures.
Methods
Datasets and contamination control
We use four molecular regression datasets (Table 1a): lipophilicity, 4,200 octanol/water distribution coefficients at pH 7.4 [59]; BACE, 1,513 inhibitory potencies () against BACE-1 [53]; antiviral, 1,006 inhibitory potencies () against the SARS-CoV-2 main protease released with the ASAP Discovery–Polaris–OpenADMET challenge of 2025 [3]; and ESOL, 1,128 aqueous solubilities [10]. Each dataset is split once into a test set of 150 molecules, drawn from a seeded permutation, and a training pool. On the three clean datasets a test molecule is eligible only if none of four string forms, the verbatim SMILES, the RDKit canonical isomeric and non-isomeric SMILES [34] and the InChIKey, occurs in any of seven open pre-training corpora searched with infini-gram [38]: the training data of OLMo-2 [54], Dolma [52], DCLM-baseline [35], The Pile [12], RedPajama [58] and C4 [45]; the ESOL test set is drawn without this filter. Every pool molecule whose InChIKey connectivity block equals that of a test molecule is removed, so that no stereoisomer, tautomer or renamed duplicate of a test molecule can enter a context (Appendix § 11).
| dataset | property | test | pool | largest | in corpus | flagged |
| Lipophilicity [59] | logD at pH 7.4 | 150 | 4,042 | 1,000 | 0.0 % | 0 of 7 |
| BACE [53] | BACE-1 pIC50 | 150 | 1,355 | 1,000 | 0.0 % | 0 of 7 |
| Antiviral [3] | SARS-CoV-2 Mpro pIC50 | 150 | 844 | 844 | 0.0 % | 0 of 7 |
| ESOL [10] | logS (log₁₀ mol/L) | 150 | 976 | 976 | 32.4 % | 6 of 7 |
| model | effort none | at low | along the ladder | hit₃ (%) | outcome | |
| Haiku 5.5 | accepted | 0 | from medium on | 10.6 | flagged, heavy | |
| Sonnet 5.5 | rejected | 0 | none at any effort | 37.2 | flagged, heavy | |
| GPT-6.1 Sol | rejected | 74–286 | rise with the effort | 19.5 | flagged, heavy | |
| GPT-6 Luna | accepted | 392–581 | rise with the effort | 8.8 | flagged, partial | |
| GPT-6 Astra | rejected | 74–298 | no ladder | 18.6 | flagged, heavy | |
| Gemini 3.8 Flash | rejected | 0–1,532 | rise with the effort | 12.4 | flagged, heavy | |
| GLM 5.3 Flash | rejected | 51–144 | rise only at high | 2.7 | clean |
Table 1. Table 1: Datasets and model panel. (a) The four datasets: test-set size, deduplicated training pool, largest context size , share of the test molecules found in any of the seven searched pre-training corpora as a SMILES string (in corpus; maximum over the corpora, Figure 5b), and the number of panel models the zero-shot screen flags (flagged). (b) The seven language models: whether the endpoint accepts the reasoning effort none (every endpoint accepts minimal to high), the median emitted reasoning tokens per call at the effort low (range over the 24 main-run cells) and how they change along the reasoning ladder (Figure 2), and on the ESOL test set the share of the molecules whose published value the model reproduced to three significant figures (hit₃; answers given in mol/L converted to logS, § 2.1) with the outcome of the zero-shot screen and its regime. Endpoints, service tiers and prices: Table A1. Endpoints as of 2026-10-08, zero-shot outcomes as of 2026-10-09 (GPT-6 Astra 2026-10-13).
Contamination is measured with the corpus search as well as with the zero-shot screen of Busch et al. [7]. In that study, each panel model is asked, with the dataset named, for the published value of each test molecule to three significant figures, and a model–dataset cell is flagged when the retention of matching digits exceeds a floor computed from the labels alone after Benjamini–Hochberg correction [4]. This neither proves contamination nor excludes it, but is an indicator that correlates with the presence of molecules in open pre-training corpora [7].
In-context protocol and model panel
The prediction prompt consists of three messages, given verbatim in Appendix § 10: a system message that assigns the role of a chemist and names the property with its unit, a training block with one record per molecule, its SMILES string as written in the source file and its label with two decimals, marked for prompt caching, and a task message with the test SMILES that asks for a prediction and a 1-sigma uncertainty, defined as the standard deviation of the predictive distribution so that the measured value falls within prediction uncertainty about 68% of the time, in a fenced JSON block. A reply is parsed from its last readable JSON block, and stated standard deviations below 0.01 are set to 0.01 for scoring. The panel consists of seven models served through OpenRouter [43]: Claude Haiku 5.5 and Claude Sonnet 5.5 (Anthropic), GPT-6.1 Sol, GPT-6 Luna and GPT-6 Astra (OpenAI), Gemini 3.8 Flash (Google) and GLM 5.3 Flash (Z.ai) (Table 1b, Appendix § 11). Each model is pinned to one provider endpoint with fallback disabled, the flex tier is used where the provider offers one, every request uses the reasoning effort low and a completion cap of 32,000 tokens, and temperature and top-p are not sent, so the provider defaults apply; Claude Opus 5.5 is not in the panel because its endpoint returned a content-filter stop without an answer in every call with 250 or more training molecules.
Context sizes, repeats and context draws
The context sizes are , 60, 120, 250, 500 and the largest available, 1,000 for lipophilicity and BACE, 844 for antiviral and 976 for ESOL; the in-context set of size is the first molecules of a seeded permutation of the pool, so the sets are nested and identical for every test molecule. A cell is one combination of dataset, model and context size. Each test molecule is predicted in calls per cell, and the prediction and the stated standard deviation of a molecule are the means over its parsed repeats. Two further context draws use independent permutations of the same pool at with the same test set and settings. Every value in this paper is context draw 1 at every ; at the range over the three draws is given in brackets, and the draws are never averaged into a learning curve. Every cell of the main run has an answer for all 150 test molecules and no call that reached the completion cap.
Classical baselines
Eight classical methods are trained on exactly the in-context molecules of each cell and predict the same 150 test molecules (Appendix § 12). TabPFN v3.5 [17, 25], a tabular foundation model trained on synthetic tasks only, receives RDKit two-dimensional descriptors; its prediction is the mean of its predictive distribution and its is half the distance between the 16th and the 84th percentile. An exact Gaussian process (GP) on Morgan fingerprints uses a Tanimoto kernel [16] with amplitude and noise chosen by maximising the marginal likelihood [48]; random forests [5] on the descriptors and on the fingerprints use the spread over the trees as ; deep ensembles [33] of five Chemprop networks [22] and of five multilayer perceptrons use the member spread; a -nearest-neighbour model and a trivial model that predicts the training mean complete the set. Every fit receives the labels of its training rows only. TabPFN, both random forests, the GP and the nearest-neighbour model were run with three seeds in draw 1, and values are averaged over the seeds within a draw; the paired bootstrap uses seed 0.
Metrics and statistics
For a cell with test molecules (), is the error and the stated standard deviation of a language model or the of a baseline. The stated is defined operationally, by the prompt of § 2.2, as the standard deviation of a distribution for the measured value; the of a baseline is the raw spread indicator of its method (§ 2.4). Between models, is therefore compared only through ranks and through the error of the molecules each model ranks as most certain, not through its values. Accuracy is the root-mean-square error (RMSE). Ranking for measurement selection is the Spearman rank correlation , where a large should mark a large error; ties in are determined after rounding to 10 decimals. Selective prediction keeps the molecules with the smallest . Stated standard deviations are coarse, with a median of 30 distinct values among the 150 molecules of a cell, so several molecules often share the value at the cut. Let be the -th smallest , , with and ; if the places are filled by a uniformly random subset of , the expected mean squared error of the kept molecules is
and ; the relative error reduction is . depends on accuracy and ranking together, while isolates the ranking part. Two sources of uncertainty affect every value beyond the three calls per molecule, which are averaged: the sampling of the 150 test molecules and the choice of the context molecules. The first is covered by a paired bootstrap over test molecules with 1,000 resamples [11], one index matrix per dataset shared by every model and baseline, for the difference between a language model and TabPFN or the GP; the second by the three context draws of § 2.3, and at the bootstrap difference is computed in each draw on the same resampled molecules and averaged, so that the interval reflects the sampling of test molecules with the variation between draws averaged out. Intervals are not corrected for multiplicity.
Reasoning ladder
The reasoning ladder is a sub-experiment with six of the seven models: the efforts none, minimal, low, medium and high at and at the largest , with , the main-run cells serving as the step low. GPT-6 Astra joined the panel after the ladder, and its ladder would have cost at least as much as the main run of all seven models. Because a requested effort does not correspond to the same amount of reasoning across providers, the results are plotted against the median number of emitted reasoning tokens per call. The effort none was rejected by four of the six models, and at the effort high the completion cap truncated 15 to 35 % of Gemini 3.8 Flash’s calls on the clean datasets, so those cells rest on the molecules with a parsed answer and are censored (Appendix § 16).
Results and Discussion
Prediction Accuracy
Figure 1 shows the RMSE of the seven language models and of four classical baselines against the number of in-context molecules on the four datasets. On the three clean datasets every model improves with the context size, the language models by between 8 % and 37 % per model and dataset from to the largest , whereas on ESOL only the baselines improve and the language models keep an almost constant RMSE. The curves fall mostly monotonically; the clearest exception is antiviral, where every language model has a higher RMSE at than at in context draw 1 and none has in draws 2 and 3, so a single step of this size is a property of the context draw (Appendix § 15).
Figure 1. Test error against the number of in-context molecules. Root-mean-square error on the 150 test molecules of each dataset for the seven language models at the effort low and four of the eight classical baselines trained on the same molecules. Lines follow context draw 1; at bars span the range over the three context draws, baselines averaged over their seeds within a draw. ESOL is the dataset the zero-shot screen flags; its axis is cut, and baselines above the cut are marked by arrows.
At the most accurate language model, Sonnet 5.5, has a lower RMSE than every classical baseline on every dataset in each of the three context draws; on lipophilicity its RMSE is about half that of TabPFN (0.60 against 1.17, Figure 1). The margins differ between the datasets and shrink with increasing . On ESOL, the dataset the language models have memorised, their learning curves are flat at a level the baselines do not reach: Sonnet 5.5 has an RMSE of 0.30 at and of 0.29 at , while TabPFN falls from 0.97 to 0.51 over the same context sizes (§ 3.4).
How far the memorised knowledge of aqueous solubility carries over to molecules outside the memorised set is unclear. On lipophilicity the language models improve less with than the strongest baselines, by 18 to 37 % from to against 46 to 48 % for TabPFN and Chemprop, which indicates that prior knowledge contributes to their predictions here as well: the models know the property, but not the values of this dataset (§ 3.4). How far this accuracy generalises to other molecules then depends on how well the models know them. On BACE and antiviral the language models improve with about as much as the baselines, apart from a flat start up to for Sonnet 5.5 and GPT-6 Astra on BACE, which again points to prior knowledge at a lower level.
Figure 2 shows the RMSE of the six ladder models against the median number of reasoning tokens emitted per call at each requested effort level. There is no consistent relation between the RMSE and the number of reasoning tokens: over the 36 clean cells of the ladder the RMSE at the effort high differs from that at low by between and , with no direction shared by the six models, and the effort none, which only Haiku 5.5 and GPT-6 Luna accept, raises the error of GPT-6 Luna on ESOL from 0.56 to 0.81.
Figure 2. Accuracy along the reasoning ladder. Root-mean-square error against the median number of reasoning tokens a model emitted per call at each requested effort level, for the six ladder models at 30 in-context molecules (top) and at the largest context size (bottom); models that emit no reasoning tokens are drawn left of the axis break. At the effort high, Gemini 3.8 Flash reached the completion cap in up to 35 % of its calls on the clean datasets, so its points there rest on the remaining molecules.
The number of emitted reasoning tokens varies strongly between models and requested effort levels: Sonnet 5.5 emits no reasoning tokens at any level, Haiku 5.5 emits them from the effort medium on, and Gemini 3.8 Flash emits a median of up to 30,717 per call at high. The cells of Gemini 3.8 Flash at high rest on the 126 to 150 molecules the model finished, a subset biased towards the molecules it found easy.
Uncertainty Quantification
Figure 3 shows the Spearman rank correlation between the stated standard deviation and the absolute error against the context size, for the language models and for the classical baselines with their own . The stated standard deviation ranks the errors similarly for every model, mostly between 0.2 and 0.4. On contaminated ESOL the language models and TabPFN reach higher values of about 0.4 to 0.6, while the other baselines stay mostly below 0.2. The rank correlation stays at about the same level across the context sizes; it varies, but with no clear trend of improvement or deterioration.
Figure 3. Ranking of the errors by the uncertainty, the measure for measurement selection. Spearman rank correlation between the stated standard deviation and the absolute error on the 150 test molecules against the number of in-context molecules, for the seven language models at the effort low and four classical baselines with their own ; rounded to 10 decimals before ranking. Lines and bars as in Figure 1.
For active learning the aim is a model that predicts well after each round, and which molecule to measure next depends on that model: different models learn differently from the same molecules, and some bring prior knowledge, as transfer-learned models and language models do. A stated uncertainty is therefore worth something only together with the accuracy of the model that states it. With few labels on clean data, up to about 120 molecules, the strongest language models, Sonnet 5.5, GPT-6 Astra, GPT-6.1 Sol and Haiku 5.5, are probably the better basis for active learning: they predict more accurately than every classical baseline, and their stated ranks the errors about as well as a classical on the same molecules, better on lipophilicity and about the same on BACE and antiviral (§ 3.1 and Figure 3). An actual active-learning loop was not run here, and it would be needed to confirm this (§ 3.5).
Selective Prediction
Selective prediction replaces measurements by predictions: only the most certain predictions are used, so it needs a model with a usable uncertainty estimate. combines the accuracy and the uncertainty estimate into one number and thus assesses how useful a model is for this purpose. Figure 4 shows the RMSE of the most certain half of the predictions, , against the context size. On lipophilicity and on the contaminated ESOL the language models are ahead of every baseline at most context sizes. As for the RMSE in § 3.1, the margins are largest at and shrink with increasing ; at TabPFN reaches the level of the best language models on lipophilicity. On antiviral the best language models are ahead of TabPFN at but within 0.06 of the GP, at the same level as TabPFN from on, and slightly behind it at the largest , except GPT-6 Astra. On BACE Sonnet 5.5 and GPT-6 Astra lie below TabPFN at every , at by 0.01 in draw 1, and the difference excludes zero only with the three context draws combined at , not in a single draw (Appendix § 15). The certain half of the best language model is thus clearly the most accurate only on lipophilicity; on BACE and antiviral a classical baseline is on par from 30 examples on.
Figure 4. Error of the most certain half, the measure for selective prediction. Root-mean-square error of the 75 test molecules with the smallest stated standard deviation against the number of in-context molecules, with molecules tied at the cut selected at random (square root of the expected mean squared error, Eq. (1)), for the seven language models at the effort low and four classical baselines. Lines and bars as in Figure 1; the ESOL axis is cut, baselines above it are marked by arrows.
The lead of the language models is smaller in than in the RMSE of all molecules on BACE and antiviral, although their ranks the errors about as well (§ 3.2): on antiviral the of TabPFN lies between 0.33 and 0.51 from on, against language-model medians of 0.16 to 0.32, whereas on lipophilicity the language models gain at least as much as the baselines at . The best model for selective prediction therefore depends on the setting. On lipophilicity the language models remain the best choice, but the RMSE of Sonnet 5.5 and Haiku 5.5 hardly improves with , so their most certain half rests mostly on prior knowledge, which may not generalise to other molecules. On BACE and antiviral the best language models are on par with the best baseline or slightly ahead, and the baselines reach this level at a fraction of the cost (Appendix H), so TabPFN or the GP can be the better option there.
Zero-shot correlation, corpus presence and retention
Figure 5 shows the three contamination measurements per model and dataset: the correlation between the zero-shot predictions and the labels, the share of test molecules found in a pre-training corpus, and the retention of matching digits against its floor. The three clean datasets show no significant retrieval and none of their test molecules occurs in a searched corpus, whereas ESOL is flagged for six of the seven models (§ 2.1). Without any training molecule the models already rank the clean test molecules, with Spearman correlations between their zero-shot predictions and the labels of 0.09 to 0.83, while no clean cell retains matching digits significantly above the floor computed from the labels alone; the language models thus know the property lipophilicity, not the values of the dataset. On ESOL all seven models reach correlations of 0.91 to 0.98.
Figure 5. Contamination screen of the four test sets. (a) Spearman rank correlation between the labels and the zero-shot predictions of each language model, asked with the dataset named for the published value of each test molecule (ESOL: positive answers to molecules with a negative label read as mol/L and converted to logS, § 2.1). (b) Share of the test molecules found in the seven searched pre-training corpora as a SMILES string (verbatim or RDKit canonical) and as InChIKey; bars give the maximum over the corpora, dots the single corpora (ESOL: the 136 test molecules with a searchable string). (c) Retention , the share of the molecules whose zero-shot value matches the label to one significant figure that also match it to two, against the floor expected from the labels alone (black line). The screen tests and , the retention from two to three matching figures, against their floors with Benjamini–Hochberg correction; hit3, the share of the molecules whose published value is reproduced to three significant figures, is given for ESOL in Table 1b. Filled markers: cell flagged by the screen; open: clean. Scored are the labels with at least three significant figures, 81 (Lipo., lipophilicity), 143 (BACE), 136 (Antivir., antiviral) and 113 (ESOL); , hit3, the floors and the -values of every cell are in the repository.
The 63 ESOL test molecules whose value at least one panel model reproduced in the zero-shot screen are easier for every model than the 50 that no model reproduced, but by different factors: at the RMSE on the never-recalled molecules is 1.9 to 2.4 times that on the recalled ones for the four models with the highest hit rate, against 1.3 for TabPFN, which has no access to published values, and for GLM 5.3 Flash, the one model the screen does not flag. The stated follows the recall only in part: it is smaller on the recalled molecules, but by less than the error, so that for Sonnet 5.5 the root-mean-square error in units of the stated is 0.42 on the recalled and 1.27 on the never-recalled molecules (Appendix § 13).
ESOL shows what an uncertainty benchmark measures on a dataset the models have memorised. Accuracy is constant in because the prior dominates (Figure 1), which also holds for GLM 5.3 Flash, the model the screen does not flag, so a flat learning curve is not by itself evidence of recall; GLM 5.3 Flash, however, has more than twice the RMSE of Sonnet 5.5 on ESOL. Recall lowers the error on the recalled molecules more than it lowers the stated , and the spread over the three repeated calls ranks the errors better than the stated in 24 of the 42 ESOL cells, where repeats agree on a recalled value, against 8 of the 126 clean cells (Appendix § 14). An uncertainty benchmark on a memorised dataset therefore measures in part how confidently a model looks a value up. The protection used here is a property of the datasets, not of the prompt: blinding reduces retrieval but does not remove it in every case [7, 8], and the prompts of this study show real property names and values.
Limitations
Each dataset has one test set of 150 molecules, rank correlations over 150 molecules have intervals of about , and the intervals are not corrected for the number of comparisons. The clean datasets are three, and the context draws cover only ; at larger every value rests on one nested context chain. The protocol is one prompt with real property names and values, three calls per molecule and the provider default temperature. The recall flags of ESOL come from one zero-shot call per model, the corpus search covers open corpora and is a proxy for the closed training data of the panel models, lipophilicity and BACE originate from ChEMBL and may be present in other forms, and approximate knowledge of a value cannot be separated from chemical knowledge, so how far the accuracy generalises to compounds the language models know less about is unclear. The cells of Gemini 3.8 Flash at the effort high are censored, GPT-6 Astra has no ladder, and the endpoints change without notice, so every result describes them on the dates given in § 2.1 and Appendix § 17. The uncertainty was assessed only through its rank correlation with the error and the error of the most certain fraction on fixed test sets; an experiment with an actual active-learning loop, in which the selected molecules are added to the context and the model is queried again, is still outstanding, so the usefulness of the stated uncertainty for active learning is inferred from these two quantities and not measured. The test sets are random splits, so a test molecule can share its scaffold with context molecules; a scaffold split would make the task harder for every model, and probably more so for the classical baselines, which know nothing about a chemical series absent from their training set, than for the language models, which bring prior knowledge of it; on the random splits used here, the error of every model depends only weakly on the similarity of a test molecule to its context (Appendix E).
Conclusion
In this study, we evaluated seven frontier LLMs as in-context regressors of molecular properties with a self-reported uncertainty, on three datasets that a corpus search and a zero-shot screen find uncontaminated and on the memorised ESOL dataset as a comparison, against eight classical baselines trained on the same molecules, over context sizes from 30 to 1,000 molecules, three context draws and a ladder of reasoning efforts.
On the clean datasets every LLM learns from its context, and with few labels the strongest models predict more accurately than every classical baseline; the margin shrinks with the context size. The requested reasoning effort changes accuracy little and does not set the amount of reasoning.
The stated uncertainty ranks the errors about as well as a classical uncertainty on the same molecules. Because an uncertainty is worth something only together with the accuracy of the model that states it, the strongest LLMs are the better basis for active learning when labels are few, and the distinction fades with more labels. Whether this holds in an actual active-learning loop remains to be tested.
For selective prediction the most certain half of the best LLM is the most accurate with small contexts only on lipophilicity, where its predictions are; where an LLM has to learn the property from the context, as on BACE and antiviral, the best LLMs are only on par with the classical baselines and the baselines reach this level at a fraction of the cost.
On the memorised ESOL dataset the LLMs do not learn with increasing context sizes, as expected when the models know the dataset, and their uncertainty correlates better with their error than on the clean datasets, as expected when they know which values they recall and which they have to predict.
Acknowledgements
M.B., S.V.L., M.L.Z., C.J.C., R.C.A. and C.F. acknowledge funding by the Deutsche Forschungsgemeinschaft (DFG; project 535656357). We gratefully acknowledge the support of the Helmholtz-Gemeinschaft Deutscher Forschungszentren (HGF).
Funding
This publication was funded by the Deutsche Forschungsgemeinschaft (DFG; project 535656357): http://gepris.dfg.de/gepris/projekt/535656357.
Author contribution: CRediT
Matthias Busch: Conceptualization, Methodology, Software, Formal analysis, Investigation, Data curation, Writing – Original Draft, Writing – Review & Editing, Visualization. Marius Tacke: Writing – Review & Editing. Tamadur Albaraghthеh: Writing – Review & Editing. Sviatlana V. Lamaka: Writing – Review & Editing. Mikhail L. Zheludkevich: Writing – Review & Editing. Roland C. Aydin: Supervision, Writing – Review & Editing. Christian Feiler: Supervision, Writing – Review & Editing. Christian J. Cyron: Writing – Review & Editing.
Declaration of competing interests
The authors declare no competing interests.
Data and code availability
All code, the prompts, the raw per-call model replies, the splits, the per-cell and per-molecule tables of every run, the baseline predictions, the contamination-screen replies and flags, the full bootstrap tables, the per-cell LaTeX tables, and the scripts that regenerate every figure and table are publicly available at https://github.com/MatthiasHBusch/LLMsUncertaintyMolecules. The TabPFN v3.5 weights are licence-gated and are not redistributed; the library version (tabpfn 9.1.0) and the checkpoint (tabpfn-v3.5-20260909) are given instead. Endpoint identifiers and prices refer to OpenRouter as of 2026-10-08. An AI assistant was used during manuscript preparation for writing and correcting text and for writing the code that produced the tables and figures; all authors have read the manuscript and take full responsibility for its content.
This appendix holds the processed material behind the main text: the prompts, the panel and splits, the baseline settings, the contamination screen, the repeat spread and the context similarity, the selective-prediction detail, the reasoning ladder and the operational record. Every value per cell, that is per dataset, model, context size, context draw and reasoning effort, including the per-molecule predictions, the full paired-bootstrap tables and the per-cell LaTeX tables, is in the repository named in the data-availability statement and is not reproduced here.
Prompts
The three messages for lipophilicity at , with the first two of the 30 training records and the first test molecule (CHEMBL1951022), generated by src/icl_common.py::build_messages. The markers ** are part of the prompt text. System message:
**Your Role**
You are a professional chemist with expert knowledge in physical chemistry
→ and lipophilicity prediction.
You are tasked with predicting the lipophilicity (octanol/water
→ distribution coefficient logD at pH 7.4) of organic molecules based
→ on their SMILES strings.
**Problem Description**
You will be provided with:
1. A training dataset of molecules with their SMILES and known logD (pH
→ 7.4) values.
2. A test molecule (SMILES) for which you must predict the logD (pH 7.4).
Use your chemical knowledge and the relationships between molecular
→ structure and the logD (pH 7.4) in the training molecules to make an
→ accurate prediction.
Training block (second message, marked cache_control: ephemeral):
**Training Data:**
30 molecules from the same source as the test molecule, with their SMILES
→ and known logD (pH 7.4) values:
SMILES: Oc1ccc(cc1)C2CNCCc3c(Cl)c(O)c(O)cc23
logD (pH 7.4): 0.87
SMILES: COc1ccc2cc(ccc2c1)[C@H](C)C(=O)O
logD (pH 7.4): 0.13
[... 28 further records in the same format ...]
Task message:
**Prediction Task:**
Predict the octanol/water distribution coefficient logD at pH 7.4 for the
→ following molecule:
- SMILES: CC(C)OC(=O)N1CCC(CC1)Oc2ncnc(Nc3ccc(nc3C)S(=O)(=O)C)c2C
Also report your 1-sigma uncertainty for this prediction, in the same
→ units as the logD (pH 7.4): the standard deviation of your
↪ predictive distribution, so that the measured value should fall ↪ within prediction +/- uncertainty about 68 % of the time.
Output (JSON)
Output the result as a single fenced JSON code block with EXACTLY this structure:
{
"prediction": ...,
"uncertainty": ...
}
Hard requirements:
predictionis your predicted logD (pH 7.4).uncertaintyis that 1-sigma value (a non-negative number).The JSON block MUST be the LAST thing in your response (no text after the closing ```).
The other datasets substitute the expertise (“medicinal chemistry and BACE-1 inhibitor potency prediction”, “medicinal chemistry and SARS-CoV-2 main protease (Mpro) inhibitor potency prediction”, “physical chemistry and solubility prediction”), the task (“BACE-1 inhibitory potency (pIC50)”, “SARS-CoV-2 main protease (Mpro) inhibitory potency (pIC50)”, “measured aqueous solubility logS (log10 of the solubility in mol/L)”) and the label (“BACE-1 pIC50”, “SARS-CoV-2 Mpro pIC50”, “measured logS (log10 mol/L)”); everything else is identical. The parser takes the last fenced JSON block of a reply that can be read, or the outermost brace-delimited text if there is none, and a reply without both numbers is excluded from the molecule’s repeat means; the parse rate is at least 0.995 in every cell of the main run and of the context draws. The zero-shot screen uses the prompt of Busch et al. [7] unchanged, with the dataset named; for ESOL it asks an “expert chemist [who knows] the Delaney (ESOL) dataset very well” for the measured aqueous solubility “as reported in that dataset” to three significant figures, and the other datasets substitute the dataset name, property and unit.
Panel, endpoints and splits
Table A1 lists the seven endpoints with their service tiers and prices; tbl:1b gives the reasoning efforts they accept and the reasoning tokens they emit. Every request names one provider endpoint and disables OpenRouter’s fallback to other hosts; the flex tier, billed at half the standard price, is used for the OpenAI and Google models. Claude Opus 5.5 answered both calls of an in-context probe with 30 training molecules and returned a content-filter stop without an answer in both calls with 250 and in all four calls with 1,000 training molecules. Further candidates were left out because their reasoning was either switched off or ran to the completion cap, because the account’s privacy setting blocked their only provider, or because of cost; the probe records are in the repository.
| model | vendor | provider endpoint | tier | input | cached | output |
| Haiku 5.5 | Anthropic | anthropic | standard | 0.1 | 0.01 | 0.5 |
| Sonnet 5.5 | Anthropic | anthropic | standard | 2 | 0.1 | 10 |
| GPT-6.1 Sol | OpenAI | openai/flex | flex | 1 | 0.05 | 5 |
| GPT-6 Luna | OpenAI | openai/flex | flex | 0.05 | 0.005 | 0.25 |
| GPT-6 Astra | OpenAI | openai/flex | flex | 5 | 0.5 | 25 |
| Gemini 3.8 Flash | google-ai-studio/flex | flex | 0.375 | 0.0375 | 1.875 | |
| GLM 5.3 Flash | Z.ai | z-ai/fp8 | standard | 0.15 | 0.03 | 0.5 |
Table A1. Endpoints and prices. The provider endpoint on OpenRouter to which the requests of each language model are pinned, with fallback to other hosts disabled, its service tier (flex: the provider’s flex tier, billed at half the standard price), and the price per million input, cached-input and output tokens at that endpoint. The reasoning efforts an endpoint accepts and the emitted reasoning tokens are in Table 1b. Prices fetched from OpenRouter on 2026-10-08 (src/registry/models_v2.json, which also gives the OpenRouter ids). USD per million tokens.
Table A2 gives the splits. Every dataset is permuted once with the seed 20260907; on the clean datasets the test set is the first 150 molecules of the permutation that pass the corpus filter, which rejected 20 of the first 170 lipophilicity candidates and no BACE or antiviral candidate. After the test set was fixed, every pool molecule with the InChIKey connectivity block of a test molecule was removed: enantiomers or stereo-unspecified forms on lipophilicity, stereoisomers and protonation or tautomer forms on BACE, stereoisomers on antiviral, and on ESOL two compounds that appear twice under different names with the identical label. Draws 2 and 3 permute the deduplicated pool of draw 1 with seeds derived from the main seed by SHA-256 and keep the test set; their first 120 molecules share at most 20 molecules with those of another draw, close to the expectation for independent permutations. Draw 1 is the split fixed before any model was run; draws 2 and 3 were generated after the main run.
| dataset | molecules | corpus filter | test | dedup. | pool | draw overlap (exp.) |
| Lipophilicity | 4,200 | 20 of 170 | 150 | 8 | 4,042 | 5 (3.6) |
| BACE | 1,513 | 0 of 150 | 150 | 8 | 1,355 | 10 (10.6) |
| Antiviral | 1,006 | 0 of 150 | 150 | 12 | 844 | 20 (17.1) |
| ESOL | 1,128 | 0 of 150 | 150 | 2 | 976 | 19 (14.8) |
Table A2. Datasets and splits. Number of molecules; test candidates rejected by the corpus filter (present in a searched corpus) among the positions checked; test-set size; pool molecules removed because their InChIKey connectivity block equals that of a test molecule (dedup.); training pool after removal; and the largest number of molecules shared by the first 120 molecules of two context draws, with the expectation for independent permutations in parentheses. Largest context size, corpus presence and screen outcome: Table 1a and Fig. 5 (data/splits/*_v2*.json, results/meta/context_draw_overlap.csv).
Classical baselines
Table A3 gives the configuration of the eight methods. TabPFN v3.5 (library tabpfn 9.1.0, checkpoint of 2026-09-09) receives the 217 two-dimensional descriptors of RDKit; per cell, columns that are constant or more than half missing on the training rows are dropped and missing values are imputed with the training median. One fit takes 9 to 17 s on a CPU at and at most 98 s at . The GP maximises the log marginal likelihood of amplitude and noise variance from 14 starts within the bounds and . At the optimum lies on a bound in all 12 cells, four datasets in three draws: in 8 on the lower noise bound, which interpolates the training labels, and in 4 on the lower amplitude bound, which reduces the GP to the trivial predictor, so at the GP is either the training standard deviation or close to the posterior variance alone. The ensembles’ member spread measures disagreement between members only and is far narrower than the errors: the root-mean-square error in units of the member spread lies between 3.8 and 26.8 for the Chemprop ensemble and between 8.9 and 31.8 for the MLP ensemble over the cells of draw 1. Every fit receives a label vector in which all labels outside the cell’s training rows are missing; a self-check replaces every label outside the training rows of one cell by arbitrary values and confirms that the predictions and of every method are bit-identical, and confirms for all stored cells that the training molecules are the first of the cell’s draw.
| method | features | settings | seeds | |
| TabPFN v3.5 | RDKit 2D descriptors (159–201 after the filter) | v3.5, tabpfn 9.1.0, default estimators, CPU | of the predictive distribution | 0,1,2 / 0 |
| GP (Tanimoto) | Morgan r = 2, 2,048 bits | , ML-II from 14 starts | posterior predictive SD incl. noise | 0,1,2 / 0,1,2 |
| RF (descriptors) | RDKit 2D descriptors (as TabPFN) | 200 trees, scikit-learn defaults | SD over the trees | 0,1,2 / 0 |
| RF (Morgan) | Morgan r = 2, 2,048 bits | 200 trees, scikit-learn defaults | SD over the trees | 0,1,2 / 0 |
| Chemprop ensemble | molecular graph (chemprop 2 featurizer) | 5 members, chemprop 2.0.0 defaults, 50 epochs | SD over the members | 0 / 0 |
| MLP ensemble | Morgan r = 2, 2,048 bits | 5 members, ReLU, dropout 0.1, Adam 0.001 | SD over the members | 0 / 0 |
| kNN (Tanimoto) | Morgan r = 2, 2,048 bits | , Tanimoto-weighted mean | SD of the 5 neighbour labels (floor 0.01) | 0,1,2 / 0,1,2 |
| trivial | – | training mean | training SD | 0 / 0 |
Table A3. Table A3: Classical baselines. Input features, fixed settings, the definition of the uncertainty (SD: standard deviation) and the seeds run in context draw 1 / in draws 2 and 3 (values are averaged over the seeds of a draw; the paired bootstrap uses seed 0). Every method is trained on exactly the in-context molecules of a cell and predicts the same 150 test molecules (results/baselines/*.meta.json, summary.csv).
Contamination
Figure 5 gives the corpus presence and the zero-shot outcome per dataset and model; the per-cell values of the screen, the hit rates, retentions, floors and -values, are in the repository (results/contamination/zeroshot/cells.csv). The flagged ESOL cells fall in the regime of heavy retrieval for Sonnet 5.5, GPT-6.1 Sol, GPT-6 Astra, Gemini 3.8 Flash and Haiku 5.5 and of partial retrieval for GPT-6 Luna [7]. The corpus search queried all molecules of the four datasets in seven infini-gram indexes in four string forms, the ESOL molecules also by name; strings shorter than eight characters were not queried, which leaves 7,753 of the 7,847 molecules searchable. Only the training data of OLMo-2 32B Instruct is the actual training set of a model; the other indexes measure how widely a string is redistributed on the open web, and absence there does not prove absence from a private crawl. A third of the ESOL test molecules occur as a SMILES string in Dolma. In the zero-shot screen all replies were parsed and none was truncated. The unit rule for ESOL (§ 2.1) is a plausibility rule on the sign of the label and an addition to the protocol of Busch et al. [7], which scores the parsed number as given: without it Sonnet 5.5 reaches a hit₃ of 31.0 % instead of 37.2 % and Haiku 5.5 8.8 % instead of 10.6 %, which moves Haiku 5.5 from partial to heavy retrieval, and the zero-shot Spearman correlations of the two models are 0.27 and 0.51 instead of 0.94 and 0.94.
Figure A1 and Table A4 split every ESOL cell into the 63 recalled and the 50 never-recalled test molecules; the subsets are the same for every model, because a molecule counts as recalled when any panel model reproduced it, and the 37 molecules whose label has fewer than three significant figures belong to neither. Every model is more accurate on the recalled molecules in every cell, and in every one of the 42 language-model cells the stated is smaller on the recalled than on the never-recalled molecules, but by less than the error: the root-mean-square error in units of the stated , , is lower on the recalled subset in all 42 cells. The flat ESOL curves are not explained by recall alone: GLM 5.3 Flash, which the screen does not flag, has an RMSE of 0.74 at and 0.73 at .
Figure A1. Error and error in units of the stated on recalled and never-recalled ESOL molecules. Root-mean-square error and root mean square of for the 63 recalled (filled) and the 50 never-recalled (open) test molecules, at 30 (faint) and 976 (solid) in-context molecules; ticks mark all 150 test molecules, and the vertical line marks a root mean square of of 1. TabPFN, which has no access to published values, is shown for reference, averaged over its seeds.
Table A4: ESOL by recall at 976 in-context molecules. Test molecules whose value at least one panel model reproduced to three significant figures in the zero-shot screen (rec., 63 molecules) against those no model reproduced (never, 50); the remaining 37 labels have fewer than three significant figures and are in neither subset. Root-mean-square error with the ratio never / recalled, and root mean square of the error in units of the stated , with floored at 0.01, on all 150 molecules and on the two subsets; language models at effort low, TabPFN v3.5 seed 0, context draw 1 (results/icl/calibration_cells.csv, columns RMSE and s).
| model | hit % | RMSE all | RMSE rec. | RMSE never | ratio | RMS all | RMS rec. | RMS never |
| Haiku 5.5 | 10.6 | 0.44 | 0.43 | 0.54 | 1.2 | 0.95 | 0.87 | 1.11 |
| Sonnet 5.5 | 37.2 | 0.29 | 0.20 | 0.40 | 2.0 | 0.87 | 0.42 | 1.27 |
| GPT-6.1 Sol | 19.5 | 0.40 | 0.29 | 0.58 | 2.0 | 0.76 | 0.52 | 1.06 |
| GPT-6 Luna | 8.8 | 0.56 | 0.56 | 0.71 | 1.3 | 1.02 | 1.04 | 1.22 |
| GPT-6 Astra | 18.6 | 0.37 | 0.28 | 0.53 | 1.9 | 0.72 | 0.52 | 1.00 |
| Gemini 3.8 Flash | 12.4 | 0.34 | 0.20 | 0.48 | 2.4 | 0.88 | 0.51 | 1.22 |
| GLM 5.3 Flash | 2.7 | 0.73 | 0.72 | 0.92 | 1.3 | 1.00 | 1.03 | 1.16 |
| TabPFN v3.5 | – | 0.51 | 0.50 | 0.63 | 1.3 | 1.05 | 0.92 | 1.36 |
Table A4.
Repeat spread and similarity to the context
The spread over the three repeated calls of a molecule, the standard deviation of its three predictions, is a second ranking signal that the repeats provide. It is exactly zero for between 0.7 and 33 % of the molecules of a cell. It ranks the errors better than the stated in 8 of the 126 clean cells and in 24 of the 42 ESOL cells; for Sonnet 5.5 on ESOL at its rank correlation with the absolute error is 0.65 against 0.40 for the stated , and Sonnet 5.5 returns three identical predictions for 38 % of the recalled and 18 % of the never-recalled molecules there. Combining both signals in the variance of the mixture of the three repeats, with the prediction and the stated of repeat and their mean prediction , changes the rank correlation with the error by a median of 0.00 over the 126 clean cells, upwards in 68 of them and downwards in the rest, and raises it in all 42 ESOL cells, by a median of 0.03 (results/icl/mixture_sigma.csv).
Figure A2 relates the error and the stated to the similarity of a test molecule to its context, measured as the largest Tanimoto similarity of its Morgan fingerprint to any of the in-context molecules. On the clean datasets a close context neighbour predicts a smaller error only weakly: the rank correlation between this similarity and the absolute error has a median of −0.21 over the language models at and of −0.12 at the largest , with a range from −0.31 to 0.01 over both sizes, and the four baselines lie between −0.37 and 0.15. The stated of the language models follows the similarity more closely than their error does, with a median rank correlation of −0.56 at and of −0.49 at the largest , more closely than the of TabPFN at the largest , −0.54 and −0.28, and less closely than that of the GP, −0.97 and −0.89, whose is built from the kernel similarities to the context molecules.
Figure A2. Similarity of the test molecules to their context against the error and the stated uncertainty. Spearman rank correlation between the largest Tanimoto similarity of a test molecule (Morgan fingerprint, radius 2, 2,048 bits) to any of the in-context molecules and its absolute error (top) and its stated standard deviation (bottom), against the number of in-context molecules, for the seven language models at the effort low and four classical baselines; a negative value means that molecules with a close context neighbour are predicted more accurately (top) or with a smaller (bottom). Lines and bars as in Figure 1.
Selective prediction and context draws
Figure A3 extends Figure 4 to the strict cut and the loose cut . At the ordering of the models equals that of the RMSE on all molecules; at TabPFN keeps the more accurate fifth on antiviral from on, while on lipophilicity and BACE the fifth kept by the best language model is as accurate as TabPFN’s or better at every (§ 3.3). Figure A4 shows every cut at , and Figure A5 places accuracy and ranking side by side at : the most accurate language models stand to the left of every baseline on all three clean datasets, but above the baselines only on lipophilicity. Because the stated standard deviations take few distinct values, the cut of splits a tied group in 286 of the 336 language-model cells of the main run and the context draws; the exact range of over all tie orders has a median width of 0.03 and is at most 0.23 in a draw-1 cell, so the tie rule of Eq. (1) is needed for a deterministic value.

Figure A3. Error of the most certain fifth and of the most certain four fifths. Root-mean-square error of the 20 % (top) and the 80 % (bottom) of the test molecules with the smallest stated standard deviation against the number of in-context molecules, with the tie rule of Eq. (1), for the seven language models at the effort low and four classical baselines. Lines and bars as in Fig. 1; the ESOL axes are cut, baselines above them are marked by arrows.
Figure A4. Error of the most certain molecules as a function of how many are kept. For each dataset at (context draw 1), the root-mean-square error of the fraction of test molecules with the smallest stated (tie rule of Eq. (1)), for Sonnet 5.5, Haiku 5.5, GPT-6 Astra, the pointwise median of the seven language models (thick grey), TabPFN and the GP (mean over seeds). Dotted references for Sonnet 5.5 and TabPFN in their colours: the oracle that keeps the molecules with the smallest absolute error, and random selection. The ESOL axis is cut at 1.12; the GP lies at 1.74 to 1.86 for every (arrows).
Figure A5. Accuracy and selection at 30 in-context molecules. Root-mean-square error of all test molecules against the relative error reduction obtained by keeping the half with the smallest stated standard deviation, for the language models (filled) and the classical baselines (open; further baselines in light grey); markers show context draw 1, bars the range over the three context draws. Dashed lines connect points of equal error on the selected half.
Figure A6 shows the learning curves of the three context draws at . The median standard deviation of the RMSE over the three draws, taken over the models of a dataset and context size, is lower for the language models than for the baselines in 10 of the 12 combinations of dataset and , which fits predictions that lean on a prior more than on the particular training molecules. Table A5 gives the paired-bootstrap differences to TabPFN at that the main text quotes; the bootstrap interval of
| model | Lipophilicity | BACE | Antiviral |
| Haiku 5.5 | [, ]† | [, ] | [, ] |
| Sonnet 5.5 | [, ]† | [, ]† | [, ]† |
| GPT-6.1 Sol | [, ]† | [, ]† | [, ] |
| GPT-6 Luna | [, ] | [, ] | [, ] |
| GPT-6 Astra | [, ]† | [, ]† | [, ] |
| Gemini 3.8 Flash | [, ]† | [, ] | [, ] |
| GLM 5.3 Flash | [, ]† | [, ]‡ | [, ] |
| Haiku 5.5 | [, ] | [, ] | [, ]‡ |
| Sonnet 5.5 | [, ]† | [, ] | [, ]‡ |
| GPT-6.1 Sol | [, ] | [, ] | [, ]‡ |
| GPT-6 Luna | [, ] | [, ] | [, ] |
| GPT-6 Astra | [, ]† | [, ] | [, ]‡ |
| Gemini 3.8 Flash | [, ] | [, ] | [, ] |
| GLM 5.3 Flash | [, ] | [, ] | [, ] |
| Haiku 5.5 | [, ]† | [, ] | [, ] |
| Sonnet 5.5 | [, ]† | [, ]† | [, ] |
| GPT-6.1 Sol | [, ]† | [, ] | [, ] |
| GPT-6 Luna | [, ]† | [, ] | [, ] |
| GPT-6 Astra | [, ]† | [, ]† | [, ] |
| Gemini 3.8 Flash | [, ]† | [, ] | [, ] |
| GLM 5.3 Flash | [, ]† | [, ]‡ | [, ] |
| Haiku 5.5 | [, ]† | [, ] | [, ] |
| Sonnet 5.5 | [, ]† | [, ] | [, ] |
| GPT-6.1 Sol | [, ] | [, ] | [, ] |
| GPT-6 Luna | [, ] | [, ] | [, ] |
| GPT-6 Astra | [, ] | [, ] | [, ] |
| Gemini 3.8 Flash | [, ]† | [, ] | [, ] |
| GLM 5.3 Flash | [, ] | [, ] | [, ] |
Table A5. Table A5: Paired bootstrap at 30 in-context molecules, language model minus TabPFN v3.5. Difference with the percentile 95 % interval from 1,000 resamples of the 150 test molecules, computed in each of the three context draws on the same resampled molecules and averaged (TabPFN seed 0). † / ‡: the interval excludes 0 in favour of the language model / of TabPFN. The intervals at , 120 and the largest , and against the GP, are in the repository (results/icl/bootstrap_cells.csv).
Figure A6. Learning curves of the three context draws. Test RMSE at , 60 and 120 for the seven language models and TabPFN (averaged over its seeds within a draw), one line per context draw: draw 1, the nested chain of all other figures, bold, and draws 2 and 3 thin; models slightly offset in .
a single rank correlation over 150 molecules has a median half-width of 0.15, and the of one language model at differs by up to 0.22 between context draws, so a selection difference of that size from a single draw is not interpretable.
Reasoning ladder
Figure 2 shows the emitted reasoning tokens and the accuracy per model, effort and context size; the values of every ladder cell, including the change of the rank correlation from the effort low to high, are in the repository (results/icl/ladder_cells.csv). Five of the seven panel endpoints reject the effort none with HTTP status 400, so the ladder has four steps for Sonnet 5.5, GPT-6.1 Sol, Gemini 3.8 Flash and GLM 5.3 Flash and five for Haiku 5.5 and GPT-6 Luna, 160 cells beyond the main run. Outside Gemini 3.8 Flash at high, at most two of the 450 calls of a cell were truncated. Zero reasoning tokens do not mean that a model does not reason: the Anthropic models write visible reasoning while reporting no reasoning tokens, and Gemini 3.8 Flash reports no reasoning tokens in 18 % of its main-run calls and writes its reasoning into the visible answer instead. The ladder steps other than low cost USD 3.01 (none), 71.62 (minimal), 78.65 (medium) and 241.68 (high), of which Gemini 3.8 Flash at high accounts for USD 171.70. For GPT-6 Luna and GPT-6.1 Sol the steps other than low cost 4.8 and 6.0 times their step low; applied to the step-low cells of GPT-6 Astra, a ladder would have cost USD 315 to 393, at least as much as the whole main run of seven models (USD 311.02).
Operational findings
The calls, the billed cost and the truncated and declined calls of every cell are in the repository (results/icl/cells.csv, results/contamination/zeroshot/cells.csv). Endpoints decline requests in ways that shape a panel: besides the Claude Opus 5.5 refusals (Appendix § 11), the OpenAI endpoints declined single prompts with a reference to their content policy, mostly antiviral prompts; the missing calls were requested again with the identical prompt, and the cells with fewer than 450 stored calls after these gap-fill runs are all OpenAI cells, with at least 445 calls and every test molecule answered. The prompt cache serves most of the context: in the main-run cells with the cached share of the prompt tokens is between 0.91 and 1.00. The 450 replies of a main-run cell state between 7 and 37 distinct standard deviations, which is why the selective metrics need a rule for ties. The context draws contain 1,613 duplicated records, written when two processes worked on the same cells; they were billed and are excluded from the analysis. Endpoint lists and prices were fetched from OpenRouter on 2026-10-08, the zero-shot screen ran on 2026-10-09 (GPT-6 Astra on 2026-10-13), the main run between 2026-10-09 and 2026-10-13, the reasoning ladder on 2026-10-12 and 2026-10-13 and the context draws on 2026-10-13; hosted endpoints change without notice, so every measurement describes the endpoint as served on its date. Requests were sent by a Julia runner that stores every reply with its token counts, cached tokens, finish reason, serving provider and billed cost; the analysis uses Python with numpy, scipy, scikit-learn [44] and RDKit, and the baselines tabpfn 9.1.0, chemprop 2.0.0 and PyTorch. Figure A7 sets the cost of a cell against its error.
Figure A7. Figure A7: Cost and error of each model at the smallest and the largest context. Top: OpenRouter cost in USD of one cell (150 test molecules, calls, effort low, context draw 1; log scale) against the test RMSE, open marker , filled marker largest . Bottom: four classical baselines, CPU wall time per cell in seconds (mean over seeds; log scale) against the RMSE; seconds and dollars are not on one axis.
References
- [1]Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie C. Y. Chan, Biao Zhang, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2404.11018.arxiv.org/abs/2404.11018
- [2]Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? Investigations with linear models. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2211.15661
- [3]ASAP Discovery Consortium, Polaris, and OpenADMET. ASAP discovery–OpenADMET antiviral potency challenge 2025 (unblinded dataset). Polaris Hub, March 2025. URL https://polarishub.io/datasets/asap-discovery/antiviral-potency-2025-unblinded. Competition announced 3 December 2024; training data released 13 January 2025; unblinded dataset published 28 March 2025.
- [4]Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. R. Stat. Soc. B, 57(1):289–300, 1995. doi: 10.1111/j.2517-6161.1995.tb02031.x.DOI
- [5]Leo Breiman. Random forests. Machine Learning, 45:5–32, 2001. doi: 10.1023/A:1010933404324.DOI
- [6]Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Kevin Linka, Christian J. Cyron, Christian Feiler, and Roland C. Aydin. Large language models predicting the corrosion inhibition efficiency of magnesium dissolution modulators. Corros. Sci., 255:113080, 2025. doi: 10.1016/j.corsci.2025.113080.
- [7]Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Christian J. Cyron, Roland C. Aydin, and Christian Feiler. Molecular déjà vu: Digit-level retrieval of molecular properties in frontier language models, 2026. arXiv:2609.05381.arxiv.org/abs/2609.05381
- [8]Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Christian J. Cyron, Christian Feiler, and Roland C. Aydin. In-context molecular property prediction with LLMs: A blinding study on memorization and knowledge conflicts. J. Chem. Inf. Model., 2026. doi: 10.1021/acs.jcim.6c00961.DOI
- [9]Zizhang Chen, Pengyu Hong, and Sandeep Madireddy. Question rephrasing for quantifying uncertainty in large language models: Applications in molecular chemistry tasks. arXiv:2408.03732, 2024.arxiv.org/abs/2408.03732
- [10]John S. Delaney. ESOL: Estimating aqueous solubility directly from molecular structure. J. Chem. Inf. Comput. Sci., 44(3):1000–1005, 2004. doi: 10.1021/ci034243x.DOI
- [11]Bradley Efron and Robert J. Tibshirani. An Introduction to the Bootstrap, volume 57 of Monographs on Statistics and Applied Probability. Chapman & Hall, New York, 1993.
- [12]Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling, 2020. arXiv:2101.00027.arxiv.org/abs/2101.00027
- [13]Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? A case study of simple function classes. In Advances in Neural Information Processing Systems, volume 35, pages 30583–30598, 2022.arxiv.org/abs/2208.01066
- [14]Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems 30 (NIPS 2017), pages 4878–4887, 2017. arXiv:1705.08500.arxiv.org/abs/1705.08500
- [15]Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6577–6595, 2024. doi: 10.18653/v1/2024.naacl-long.366.DOI
- [16]Ryan-Rhys Griffiths, Leo Klarner, Henry B. Moss, Aditya Ravuri, Sang Truong, Yuanqi Du, Samuel Stanton, Gary Tom, Bojana Rankovic, Arian Jamasb, Aryan Deshwal, Julius Schwartz, Austin Tripp, Gregory Kell, Simon Frieder, Anthony Bourached, Alex Chan, Jacob Moss, Chengzhi Guo, Johannes P. Dürholt, Saudamini Chaurasia, Ji Won Park, Felix Strieth-Kalthoff, Alpha A. Lee, Bingqing Cheng, Alán Aspuru-Guzik, Philippe Schwaller, and Jian Tang. GAUCHE: A library for Gaussian processes in chemistry. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023. arXiv:2212.04450.arxiv.org/abs/2212.04450
- [17]Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Mihir Maniam, Shi Bin Hoo, et al. TabPFN-3: Technical report, 2026. arXiv:2605.13986.arxiv.org/abs/2605.13986
- [18]Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. In Advances in Neural Information Processing Systems, volume 36, pages 19622–19635, 2023.arxiv.org/abs/2310.07820
- [19]Taicheng Guo, Kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. In Adv. Neural Inf. Process. Syst., Datasets and Benchmarks Track, 2023. arXiv:2305.18365.arxiv.org/abs/2305.18365
- [20]Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. TabLLM: Few-shot classification of tabular data with large language models. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 206 of Proceedings of Machine Learning Research, pages 5549–5581, 2023.arxiv.org/abs/2210.10723
- [21]Esther Heid, Charles J. McGill, Florence H. Vermeire, and William H. Green. Characterizing uncertainty in machine learning for chemistry. Journal of Chemical Information and Modeling, 63 (13):4012–4029, 2023. doi: 10.1021/acs.jcim.3c00373.DOI
- [22]Esther Heid, Kevin P. Greenman, Yunsie Chung, Shih-Cheng Li, David E. Graff, Florence H. Vermeire, Haoyang Wu, William H. Green, and Charles J. McGill. Chemprop: A machine learning package for chemical property prediction. J. Chem. Inf. Model., 64(1):9–17, 2024. doi: 10.1021/acs.jcim.3c01250.DOI
- [23]Brian Hie, Bryan D. Bryson, and Bonnie Berger. Leveraging uncertainty in machine learning accelerates biological discovery and design. Cell Systems, 11(5):461–477.e9, 2020. doi: 10.1016/j.cels.2020.09.007.DOI
- [24]Lior Hirschfeld, Kyle Swanson, Kevin Yang, Regina Barzilay, and Connor W. Coley. Uncertainty quantification using neural networks for molecular property prediction. J. Chem. Inf. Model., 60(8):3770–3780, 2020. doi: 10.1021/acs.jcim.0c00502.DOI
- [25]Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637:319–326, 2025. doi: 10.1038/s41586-024-08328-6.DOI
- [26]Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerrero, and Berend Smit. Leveraging large language models for predictive chemistry. Nat. Mach. Intell., 6(2):161–169, 2024. doi: 10.1038/s42256-023-00788-1.DOI
- [27]Ryan Jacobs, Maciej P. Polak, Lane E. Schultz, Hamed Mahdavi, Vasant Honavar, and Dane Morgan. Regression with large language models for materials and molecular property prediction. arXiv:2409.06080, 2024.arxiv.org/abs/2409.06080
- [28]Jon Paul Janet, Chenru Duan, Tzuhsiung Yang, Aditya Nandy, and Heather J. Kulik. A quantitative uncertainty metric controls error in neural network-driven chemical discovery. Chemical Science, 10 (34):7913–7922, 2019. doi: 10.1039/C9SC02298H.DOI
- [29]Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. Large language models must be taught to know what they don’t know. In Advances in Neural Information Processing Systems, volume 37, pages 85932–85972, 2024. doi: 10.52202/079017-2729.DOI
- [30]Agustinus Kristiadi, Felix Strieth-Kalthoff, Marta Skreta, Pascal Poupart, Alán Aspuru-Guzik, and Geoff Pleiss. A sober look at LLMs for material discovery: Are they actually good for Bayesian optimization over molecules? In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 25603–25622, 2024.arxiv.org/abs/2402.05015
- [31]Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In ICLR, 2023. arXiv:2302.09664.arxiv.org/abs/2302.09664
- [32]Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pages 2796–2804, 2018.arxiv.org/abs/1807.00263
- [33]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30 (NIPS 2017), pages 6402–6413, 2017. arXiv:1612.01474.arxiv.org/abs/1612.01474
- [34]Greg Landrum. RDKit: Open-source cheminformatics, 2026. URL https://www.rdkit.org. Version 2026.03.6; release DOIs under 10.5281/zenodo.591637.
- [35]Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, et al. DataComp-LM: In search of the next generation of training sets for language models. In Adv. Neural Inf. Process. Syst., Datasets and Benchmarks Track, 2024. arXiv:2406.11794.arxiv.org/abs/2406.11794
- [36]Yinghao Li, Lingkai Kong, Yuanqi Du, Yue Yu, Yuchen Zhuang, Wenhao Mu, and Chao Zhang. MUBeN: Benchmarking the uncertainty of molecular representation models. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2306.10060
- [37]Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Trans. Mach. Learn. Res., 2022. arXiv:2205.14334.arxiv.org/abs/2205.14334
- [38]Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. In Conference on Language Modeling (COLM), 2024. arXiv:2401.17377.arxiv.org/abs/2401.17377
- [39]Yuyan Liu, Sirui Ding, Sheng Zhou, Wenqi Fan, and Qiaoyu Tan. MolecularGPT: Open large language model for few-shot molecular property prediction, 2024. arXiv:2406.12950.arxiv.org/abs/2406.12950
- [40]Potsawее Manakul, Adian Liусie, and Mark J. F. Gales. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In EMNLP, 2023. arXiv:2303.08896.arxiv.org/abs/2303.08896
- [41]Adrian Mirza, Nawaf Alampара, Sreekanth Kunchапu, Martino Ríos-García, Benedict Emoekabu, Aswanth Krishnan, Tanya Gupta, Mara Schilling-Wilhelmi, et al. A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists. Nat. Chem., 17(7):1027–1034, 2025. doi: 10.1038/s41557-025-01815-x.DOI
- [42]Tung Nguyen and Aditya Grover. LICO: Large language models for in-context molecular optimization. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2406.18851
- [43]OpenRouter. OpenRouter API. https://openrouter.ai, 2026. Accessed 2026-10-08.
- [44]F. Pedregosa et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.arxiv.org/abs/1201.0490
- [45]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.arxiv.org/abs/1910.10683
- [46]Mayk Caldas Ramos, Christopher J. Collison, and Andrew D. White. A review of large language models and autonomous agents in chemistry. Chem. Sci., 16(6):2514–2572, 2025. doi: 10.1039/D4SC03921A.DOI
- [47]Mayk Caldas Ramos, Shane S. Michtavy, Andrew D. White, and Marc D. Porosoff. Bayesian optimization of catalysis with in-context learning. ACS Central Science, 12(5):599–615, 2026. doi: 10.1021/acscentsci.5c02418. Preprint: arXiv:2304.05341.
- [48]Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA, 2006.DOI
- [49]Daniel Reker and Gisbert Schneider. Active-learning strategies in computer-assisted drug discovery. Drug Discovery Today, 20(4):458–465, 2015. doi: 10.1016/j.drudis.2014.12.004.DOI
- [50]James Requima, John Bronskill, Dami Choi, Richard E. Turner, and David Duvenaud. LLM processes: Numerical predictive distributions conditioned on natural language. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. arXiv:2405.12856.
- [51]Gabriele Scalia, Colin A. Grambow, Barbara Pernici, Yi-Pei Li, and William H. Green. Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction. J. Chem. Inf. Model., 60(6):2697–2717, 2020. doi: 10.1021/acs.jcim.9b00975.DOI
- [52]Luca Soldaini, Rodney Kinney, Akshita Bhabia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 15725–15788, 2024. arXiv:2402.00159.arxiv.org/abs/2402.00159
- [53]Govindan Subramanian, Bharath Ramsundar, Vijay Pande, and Rajiah Aldrin Denny. Computational modeling of β-secretase 1 (BACE-1) inhibitors using ligand based approaches. J. Chem. Inf. Model., 56(10):1936–1949, 2016. doi: 10.1021/acs.jcim.6b00290.DOI
- [54]Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhabia, Yuling Gu, et al. 2 OLMo 2 furious. In Conference on Language Modeling (COLM), 2025. arXiv:2501.00656.arxiv.org/abs/2501.00656
- [55]Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, et al. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In EMNLP, 2023. arXiv:2305.14975.arxiv.org/abs/2305.14975
- [56]Kevin Tran, Willie Neiswanger, Junwoong Yoon, Qingyang Zhang, Eric Xing, and Zachary W. Ulissi. Methods for comparing uncertainty quantifications for material property predictions. Machine Learning: Science and Technology, 1(2):025006, 2020. doi: 10.1088/2632-2153/ab7e1a.
- [57]Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, and Mihai Surdeanu. From words to numbers: Your large language model is secretly a capable regressor when given in-context examples. In Proceedings of the First Conference on Language Modeling (COLM), 2024. arXiv:2404.07544.arxiv.org/abs/2404.07544
- [58]Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. RedPajama: an open dataset for training large language models. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track, 2024. arXiv:2411.12372.arxiv.org/abs/2411.12372
- [59]Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: a benchmark for molecular machine learning. Chem. Sci., 9(2):513–530, 2018. doi: 10.1039/C7SC02664A.DOI
- [60]Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In ICLR, 2024. arXiv:2306.13063.arxiv.org/abs/2306.13063
- [61]Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh T. N. Nguyen, Lauren T. May, Geoffrey I. Webb, and Shirui Pan. Large language models for scientific discovery in molecular property prediction. Nature Machine Intelligence, 7(3):437–447, 2025. doi: 10.1038/s42256-025-00994-z.DOI
- [62]Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5506–5524, 2023. doi: 10.18653/v1/2023.emnlp-main.335.DOI










