betaFeedback

WorksPhysics and Astronomy

A Systematic Comparison of In-Context Learning Strategies for Peptide Property Prediction in Low-Data Regimes

Abstract

Peptide sequence-property relationships are often highly non-linear, but can be learned with machine learning models and exploited for the design of new materials and bioactive compounds. Various methods based on transformer architectures have become available for molecular property predictions. These foundation models promise straightforward inference and prediction of properties of new materials through in-context learning. Benchmarks of individual model architectures exist, but a systematic comparison of various foundation models against classical baselines on similar datasets is lacking. Here, we provide a systematic evaluation of the predictive performance of various in-context learning strategies. We compare several large language models, tabular foundation models, and typed decision models against an XGBoost baseline. We evaluate predictive performance on binary and multiclass classification tasks on pH-dependent surface activ-ity, IR spectral signatures, self-assembly, and anti-cancer activities. Our results show that ICL is competitive, yet does not significantly outperform classical baselines, and performance varies between models and tasks. Our findings further suggest that language models use both in context examples and knowledge acquired during pretraining to infer properties of unseen samples. This finding indicates that pretrained models can leverage the existing literature for predictions, but may also bias their predictions towards established sequence-property relationships.

ICL_paper.pdf

Abstract

Introduction

The properties of peptides are distinctly dependent on their sequence. Myriad non-linear interactions between residues drive the emergence of secondary structure elements and can cause counterintuitive effects also in short peptides.[1, 2, 3] These non-linear interactions make straightforward, linear interpretation of sequence-property relationships often challenging, and create a necessity to use more advanced statistical methods to understand these relationships and predict properties of new peptides.[4] [5, 6, 7, 8] Such non-linear mapping of sequence to property can ideally be done with machine learning (ML) models.

The choice for a specific machine learning model architecture typically depends on specific details of the data on which the model will be trained. Many areas of peptide science are constrained by synthetic throughput, and typically are in a low data regime, with 10110^{1}–10310^{3} labeled peptide sequences available.9 Whereas models to predict properties that can readily be measured with high-throughput screens, such as antimicrobial or other biological activities, are often trained on thousands of datapoints,10,11 datasets on many other properties, such as surface activity,6 IR spectral features,7, aggregation,[4] [8], and peptide catalysis12 are considerably smaller in size. In these cases, deep neural networks, which often achieve superior performance in high-data regimes, are less suited.13 Other supervised ML techniques are needed to learn sequence-property relationships and predict properties of uncharacterized peptides.

Most common classical supervised machine learning models rely on various inductive biases in model design and data representation. For instance, the choice for molecular embedding, kernel functions, and model hyperparameters can all drastically affect model performance, and typically require task-specific optimization. Recently, large pretrained foundation models have emerged as a promising alternative for molecular property prediction.14,15 Rather than relying on specific inductive biases, these models leverage their large-scale pretraining to identify correlations from the data and enable prediction of unseen samples. These models keep their weights fixed, so there are no training hyperparameters to tune, and effort shifts from model engineering to the data.

Whereas earlier examples focused on fine-tuning large language models on hand-curated datasets,14,16,17 modern frontier models often give good performance without fine-tuning, and can infer correlations with some, or even no examples provided in the language model prompt.18 Trends and correlations from the large-scale pretraining enable this few-shot or zero-shot prediction through what is known as in-context learning.19–21 The in-context learning that large language models enable provides a straightforward and hassle-free approach to predict properties of new molecules and materials,21,22 but running large language models comes at considerable computational cost compared to classical ML models. Admittedly, this comparatively large computational costs of LLMs make them unattractive for widespread deployment.

The large computational overhead that burdens large language models is mitigated by more specialized foundation models that are specifically designed for numerical and Boolean prediction. Tabular foundation models (TFMs) are gaining increased interest for such tasks.23,24 These transformer-based models are trained on a large amount of tabular data. Large-scale pretraining on real-world and synthetic data enables these models to relate patterns in the in-context examples to previously learned associations, allowing them to infer properties of unseen samples. Analogous to large language models, TFMs require little to no hyperparameter optimization or inductive biases, are robust to shifts in the input distribution, and already function well in the low data regime.

Very recently, typed decision models (TDMs),25 such as Jev,26 have emerged as interesting alternative approaches to combine pretraining on large bodies of natural language, including scientific texts, with fast and inexpensive inference. The output of typed decision models is not generated through autoregressive token generation, but is simply a probability distribution of predefined class labels. The combination of fast and inexpensive inference, a lack of required hyperparameter finetuning, and freeform data representation make TDMs interesting as potential platforms for the prediction of molecular properties.

Hence, the increasingly lower entry barriers for using pretrained foundation models appear to make in-context learning an attractive and simple strategy to predict properties of molecules and materials. However, the distinct design principles underlying these models raise various questions on the capacity of these models to predict new material properties for diverse molecular classes, including peptides. While various studies have reported benchmarks and comparisons of specific foundation models against classical baselines,22,27–29 a systematic comparison of various ICL approaches on the same datasets is required to reliably assess relative performance of these new approaches to machine learning for chemistry and materials.

Here, we systematically investigate the use of in-context learning for peptide property prediction in low data regimes. We compare general-purpose LLMs, TFMs, and TDMs with conventional supervised machine learning approaches across a diverse collection of peptide classification tasks covering self-assembly, surface activity, spectroscopy-derived structural descriptors, and biological activity. We investigate classification on a range of different properties and perform a series of control experiments to assess the limitations of ICL for peptide property prediction. Through this benchmarking study, we aim to establish whether and how in-context learning can serve as a practical alternative to classical ML methods for peptide materials research.

Classical machine learning and in-context learning for peptide property prediction

Figure 1. Classical machine learning and in-context learning for peptide property prediction. (a) Non-linear sequence–property relationships depend on interactions such as hydrogen bonds, hydrophobic contacts, π\pi-stacking, and electrostatics. (b) The classical route uses a task-specific encoding and a fitted ML model. (c) In-context learning gives labelled reference examples from the full or class-balanced training set (many-shot) and a target sample to a frozen LLM, TDM, or TFM. (d) Scope: eight models, five tasks, and a comparison of classical ML against LLMs, TDMs, and TFMs in low-data regimes. Controls and modes cover the endpoint models only.

Methods

We established predictive performance for various in-context learning models on various datasets. To determine whether a model uses the supplied context or its pretraining prior, we give every model the same five tasks, frozen splits, and reference sets. Only the model, its input representation, and the prompt change. We compare three LLMs, two typed decision models, two tabular foundation models, and an XGBoost baseline. In the following sections, we describe the selected low-data regime tasks, the used models, the input representations, and the evaluation.

Peptide datasets

We selected five peptide classification tasks for evaluating ICL in the low-data regime. The tasks cover both physicochemical peptide properties and biological activity, and include binary and multiclass classification problems. They also differ in the information available to the model: four tasks use peptide sequence as the sole input, whereas the surface activity task additionally includes pH. This diversity allows us to assess whether differences between model classes are consistent across peptide properties, label structures, and input representations. Table 1 summarizes the datasets and the resulting sample counts used in the study.

TaskSurface activity6Beta-sheet formation7Self-assembly8Breast-cancer activity30Lung-cancer activity30
Number of samples409268368199151
Classes22233
InputsSequence, pHSequenceSequenceSequenceSequence
Full train samples328214294159120
Test samples8154744031
Class-balanced training samples2421461904557

Table 1. Peptide classification tasks used in this study.

For the benchmark on the surface activity dataset, we selected the minimum reported surface tension for each peptide amphiphile headgroup sequence and pH, and followed reported surface activity: samples with a minimum surface tension below 50 mN/m were classified as active. For β\beta-sheet formation and self-assembly, we used the experimentally assigned binary labels provided in the source datasets. The two cancer datasets were treated as separate three-class tasks, using the three original labels: experimentally inactive, moderately active, and very active. All tasks used the peptide sequence as an input, and the surface activity task additionally used pH and tail length.

We fixed the train–test partitions before model evaluation. Four datasets used a stratified 80:20 split. For surface activity, we used a stratified group split by peptide sequence to prevent observations for the same sequence at different pHs from appearing in both partitions. We evaluated both the full training sets and balanced variants obtained by random undersampling to the size of the smallest training class. These test sets retained their original class distributions.

§ 3.1.1 and Table S9 provide the data preparation procedures, label mappings, sample counts, split implementation, and class-balancing rules.

TaskTotalClassesTrainTestBalanced trainFull per-class countsTest per-class counts
Surface activity409232881242False 258; True 151False 51; True 30
β\beta-sheet formation268221454146False 177; True 91False 36; True 18
Self-assembly368229474190False 119; True 249False 24; True 50
Breast-cancer activity19931594045Inactive 83; Moderately active 98; Very active 18Inactive 17; Moderately active 20; Very active 3
Lung-cancer activity15131203157Inactive 52; Moderately active 75; Very active 24Inactive 11; Moderately active 15; Very active 5

Table S9. Dataset sizes, class counts, and split sizes. Binary classes are shown by their source labels, false and true. Full counts are the cleaned datasets. Train and test counts define the fixed splits; balanced train counts give the class-balanced training subsets. Per-class counts are listed in class order.

Models and input representations

We compared eight models across four model families: three LLMs at various sizes, two typed decision models, two tabular foundation models, and an XGBoost baseline. Table 2 summarizes the models and their families.

ModelTypeModel sizeCost per 1M in/out tokens
GPT-6.1 SolLLMN/A$2.00$ / $10.00$
DeepSeek-V4.1 FlashLLM552B$0.15$ / $0.60$
Gemma-4 31BLLM31B$0.10$ / $0.34$
Jev 1.14TDMN/A$0.042$ per 1M input tokens
GPT-6 Luna DecisionsTDMN/A$0.10$ per 1M input tokens
TabPFN v2TFM7.24MLocal
TabFMTFM1.64BLocal
XGBoostBaselineN/ALocal

Table 2. Models compared in this study.

N/A marks a model for which a parameter count is not published or not applicable.

All models, except XGBoost, were used without fine-tuning. For each prediction, the models received the training examples as context together with the test sample. The local TFMs were similarly provided with the training data as in-context examples and did not update their pretrained weights. In contrast, XGBoost was fitted to the training data for each replicate and therefore represents a conventional supervised learning approach.

The models use different input representations. The LLMs and TDMs received peptide sequences as text. For the surface activity task, pH and tail length were provided as additional input variables. TabPFN23 and TabFM24 received numerically encoded peptide sequences, with each residue represented by a positional token. XGBoost used a one-hot representation of the same positional sequence encoding. The additional surface activity variables were provided as numerical features to the local models.

The sequence encoder preserves the original sequence representation, including residue case. Sequences were padded to the maximum sequence length observed for each task. The model identifiers, provider information, API settings, computational details, complete encoding procedure and model-specific input formats are described in § 3.1.2.

In-context learning protocol

For each task, models received a labeled reference set drawn from the training data and were evaluated on the same fixed test set. We used either the full or the class-balanced training set as the reference set. In standard in-context learning, the reference set contained the correct training labels. In the zero-shot condition, no labeled reference examples were provided. The same zero-shot predictions were used for comparison with both reference data variants.

For the LLMs, each request contained task instructions, a description of the prediction task, the allowed class labels, the labeled reference examples, the target samples, and a prescribed JSON output schema. The models returned class predictions and a brief reasoning summary for each batch. These summaries were mainly enforced to provide additional context for the LLM-based decisions, but may not be fully accurate31,32 and were not used for evaluation. The TDMs received the task instructions and labeled reference examples in a structured state, followed by a separate classification question for each target sample.

To determine if the models perform better when predicting in batches or individually, we evaluated two inference modes for the LLMs and TDMs. In one-by-one mode, each request contained one target sample and the complete reference set. In all-at-once mode, a single request contained the complete test set and the reference examples. Before the main evaluation, we compared the two modes on the surface activity and self-assembly tasks, using both the balanced and full training sets and all remote, API-accessed endpoint models. Based on the results of this ablation, we decided to request only one outcome per request for the main evaluation.

To test whether predictions depended on the supplied labels, we also evaluated a corrupted-label condition for the endpoint models. In this condition, every reference label was changed while retaining all declared classes: binary labels were inverted, and the three-class cancer labels were cyclically shifted. Test labels remained unchanged. The corrupted reference set was generated once for each condition. The detailed relabeling procedure is described in more detail in Section 1.3.

Model predictions can vary between replicates. No seed was sent to the endpoint models, so they were evaluated using five independent API calls. Local models were evaluated using 100 replicates, with a different fixed seed for each replicate. Unless otherwise noted, reported means and standard deviations are calculated across these replicates. This design accounts for differences in reproducibility between API-based inference and local model fitting.

The complete evaluation campaign comprised 3,725 replicate evaluations. § 3.1.3 provides the prompt templates, output schema, replicate design, and seed handling. § 3.3.3 describes the statistical tests.

Evaluation and statistics

Macro-F1 was the primary evaluation metric. It was calculated as the unweighted mean of the per-class F1 scores for each task. For binary classification tasks, we also report positive-class F1, corresponding to the true class. Secondary metrics included per-class precision, recall, F1, and support, together with confusion matrices. Token use, inference cost, and latency are reported in Table S8.

Table S8. Endpoint cost, token use, request count, and mean run latency. Costs, token counts, and request counts are complete provider-log totals. Mean latency is for full-training one-by-one runs (25 runs per model); the total row gives the mean over all 125 runs.

ModelCost (USD)TokensRequestsMean latency (s)
GPT-6.1 Sol27.6214,922,8037,660178.97
DeepSeek-V4.1 Flash27.7452,516,0297,227364.85
Gemma-4 31B11.8042,978,8607,6269.74
Jev 1.140.9723,427,5837,0263.52
GPT-6 Luna Decisions1.3113,102,9847,0233.57
Total69.43146,948,25936,562112.13

Table S8.

For each task, reference data variant, model, and applicable inference mode, we report the mean and standard deviation of each metric across replicates. These statistics describe variability across repeated evaluations on the same fixed train–test split. Furthermore, to compare one-by-one and all-at-once inference, we calculated the paired difference in performance (one-by-one minus all-at-once) for each task, model, and reference data variant. The comparison comprised 20 pairs of matched subgroups: two tasks, five endpoint models, and two reference data variants. We report the mean paired difference, its 95% tt confidence interval, and two-sided paired tt-test and Wilcoxon signed-rank test results. Section S1.4 provides the metric definitions and statistical procedures. Sections S2 and S3 contain the complete result tables and supplementary figures.

Results & Discussion

Predictive performance on full training sets

The main comparison in this study is the systematic comparison of various ICL methods against a classical baseline on diverse classification tasks. Figure 2 and Table S1 summarize the performance of all eight models on the five tasks using the full training sets. All endpoint models used one-by-one inference. Here, “highest-performing” denotes the model with the highest observed mean macro-F1 in a given task. Figure S1 shows the corresponding performance with the class-balanced training sets.

Mean macro-F1 of the eight models on the five tasks with full training sets

Figure 2. Mean macro-F1 of the eight models on the five tasks with full training sets. Colour gives the model family: LLM, decision model, tabular foundation model, or classical baseline. The legend gives the number nn of replicates per model family: five for the endpoint models and 100 for the local models. Endpoint models use one-by-one inference. Error bars give the standard deviation across replicates. The bold number above the best bar in each task is its mean macro-F1.

ModelFamilySurface activityβ\beta-sheet formationSelf-assemblyBreast-cancer activityLung-cancer activity
GPT-6.1 SolLLM0.762±0.0100.762 \pm0.0100.781±0.0180.781 \pm0.0180.857±0.0110.857 \pm0.0110.538±0.0140.538 \pm0.0140.389±0.0290.389 \pm0.029
DeepSeek-V4.1 FlashLLM0.680±0.0360.680 \pm0.0360.791±0.0220.791 \pm0.0220.816±0.0240.816 \pm0.0240.453±0.0660.453 \pm0.0660.400±0.0430.400 \pm0.043
Gemma-4 31BLLM0.676±0.0050.676 \pm0.0050.785±0.0160.785 \pm0.0160.826±0.0250.826 \pm0.0250.455±0.0200.455 \pm0.0200.389±0.0230.389 \pm0.023
Jev 1.14TDM0.696±0.0170.696 \pm0.0170.753±0.0100.753 \pm0.0100.836±0.0060.836 \pm0.0060.504±0.0120.504 \pm0.0120.493±0.0120.493 \pm0.012
GPT-6 Luna DecisionsTDM0.597±0.0000.597 \pm0.0000.784±0.0000.784 \pm0.0000.748±0.0000.748 \pm0.0000.437±0.0000.437 \pm0.0000.351±0.0000.351 \pm0.000
TabPFN v2Tabular foundation model0.682±0.0280.682 \pm0.0280.653±0.0170.653 \pm0.0170.750±0.0190.750 \pm0.0190.469±0.0180.469 \pm0.0180.479±0.0220.479 \pm0.022
TabFMTabular foundation model0.700±0.0070.700 \pm0.0070.733±0.0000.733 \pm0.0000.808±0.0060.808 \pm0.0060.440±0.0080.440 \pm0.0080.540±0.0120.540 \pm0.012
XGBoostClassical baseline0.714±0.0190.714 \pm0.0190.800±0.0120.800 \pm0.0120.750±0.0140.750 \pm0.0140.377±0.0210.377 \pm0.0210.368±0.0260.368 \pm0.026

Table S1.

Balanced-training performance chart

Figure S1. Figure S1. Balanced-training performance. Macro-F1 (mean ±\pm SD) for the eight models on the five tasks after class-balanced training. Each task group contains eight bars, one per model. Bar colour denotes the model family. Error bars show the standard deviation (n=5n = 5 independent repeats for endpoint models; n=100n = 100 seeded repeats for local models). The bold number above the best bar in each group gives its macro-F1.

Performance varied substantially across tasks, and no model dominated across the benchmark. GPT-6.1 Sol achieved the highest mean macro-F1 on three tasks, whereas XGBoost and TabFM each achieved the highest mean on one task. The spread between the best and second-best mean score ranged from 0.008 to 0.048 macro-F1 on the binary tasks.

All models had lower macro-F1 on the two three-class cancer activity tasks than on the three binary tasks. On the breast cancer activity dataset, mean scores ranged from 0.377 to 0.538, with GPT-6.1 Sol performing best. On the lung cancer activity dataset, mean scores ranged between 0.351 and 0.540, with TabFM achieving the highest score. Per-class inspection showed that several models never predicted the rare very active class. On lung-cancer activity, GPT-6.1 Sol, Gemma-4 31B, GPT-6 Luna Decisions, and TabPFN had zero recall for this class, whereas on breast-cancer activity, neither TabPFN nor TabFM predicted it. These failures are only partly reflected in the aggregate macro-F1 scores and highlight the difficulty of minority-class prediction in the multiclass tasks. Table S3 provides the per-class results, and Table S4 reports precision, recall, and F1 for the very-active class. Because the breast- and lung-cancer activity tasks differ from the other, binary tasks in both class structure and class distribution, so the lower scores cannot be attributed to either factor alone.

ClassGPT-6.1 SolDeepSeek-V4.1 FlashGemma-4 31BJev 1.14GPT-6 Luna DecisionsTabPFN v2TabFMXGBoost
Surface activity, false0.8000.7330.6870.7910.6870.8240.7940.769
Surface activity, true0.7240.6270.6650.6020.5080.5390.6060.659
β\beta-sheet formation, false0.8510.8490.8250.8190.8360.7930.8170.869
β\beta-sheet formation, true0.7100.7340.7450.6870.7320.5130.6490.730
Self-assembly, false0.8020.7390.7630.7690.6520.6640.7310.647
Self-assembly, true0.9120.8930.8890.9020.8430.8350.8860.853
Breast-cancer activity, inactive0.6180.4850.5300.5920.3570.7020.6370.592
Breast-cancer activity, moderately active0.6480.6170.6130.6160.5530.7050.6830.531
Breast-cancer activity, very active0.3470.2580.2220.3050.4000.0000.0000.007
Lung-cancer activity, inactive0.5350.4150.5520.5220.3530.7640.6150.474
Lung-cancer activity, moderately active0.6320.6830.6160.6790.7000.6720.6710.627
Lung-cancer activity, very active0.0000.1000.0000.2790.0000.0000.3330.003

Table S3. Table S3. Per-class F1 for full training. Binary classes are shown by their source labels, false and true. Values are means over n=5n = 5 independent repeats for endpoint models and n=100n = 100 seeded repeats for local models. Test support for each class is given in Table S9. Precision and recall are given in the deposited per-class metrics, and Table S4 gives precision, recall, and F1 for the very active class under full and class-balanced training.

TaskModelFull precisionFull recallFull F1Balanced precisionBalanced recallBalanced F1
Breast-cancer activityGPT-6.1 Sol0.3670.3330.3470.1750.6670.276
Breast-cancer activityDeepSeek-V4.1 Flash0.1900.4000.2580.1140.6670.194
Breast-cancer activityGemma-4 31B0.1670.3330.2220.1390.6670.229
Breast-cancer activityJev 1.140.2830.3330.3050.1930.6670.299
Breast-cancer activityGPT-6 Luna Decisions0.5000.3330.4000.1670.3330.222
Breast-cancer activityTabPFN v20.0000.0000.0000.1500.6670.244
Breast-cancer activityTabFM0.0000.0000.0000.3010.6670.414
Breast-cancer activityXGBoost0.0070.0070.0070.1920.4600.268
Lung-cancer activityGPT-6.1 Sol0.0000.0000.0000.5330.3600.424
Lung-cancer activityDeepSeek-V4.1 Flash0.1330.0800.1000.2250.4800.304
Lung-cancer activityGemma-4 31B0.0000.0000.0000.3510.4800.404
Lung-cancer activityJev 1.140.4670.2000.2790.3130.6000.411
Lung-cancer activityGPT-6 Luna Decisions0.0000.0000.0000.0000.0000.000
Lung-cancer activityTabPFN v20.0000.0000.0000.5070.4000.444
Lung-cancer activityTabFM1.0000.2000.3330.1670.2000.182
Lung-cancer activityXGBoost0.0050.0020.0030.1430.2220.173

Table S4. Table S4. Precision, recall, and F1 for the very active class under full and class-balanced training on the two cancer tasks. Values are means over n=5n = 5 (endpoint) or n=100n = 100 (local) repeats.

Overall, model rankings differed substantially between tasks. XGBoost had the highest mean on beta-sheet formation but the lowest on breast-cancer activity, while TabFM had the highest mean on lung-cancer activity but the second-lowest on beta-sheet formation. To test whether these rank differences are resolvable at the present sample sizes, we bootstrapped the test partition (2,000 resamples, with peptides resampled as clusters for surface activity) and recomputed the macro-F1 difference between the highest- and second-highest-ranked model on each task (Table S10). In all five tasks this difference was not significant. Thus, the results do not support a single model ranking best across these tasks.

TaskHighestSecondΔ\Delta95% CI
Surface activityGPT-6.1 SolXGBoost+0.048−0.033 to +0.130
β\beta-sheet formationXGBoostDeepSeek-V4.1 Flash+0.008−0.126 to +0.134
Self-assemblyGPT-6.1 SolJev 1.14+0.022−0.048 to +0.095
Breast-cancer activityGPT-6.1 SolJev 1.14+0.033−0.105 to +0.161
Lung-cancer activityTabFMJev 1.14+0.047−0.041 to +0.141

Table S10. Statistical significance of the macro-F1 difference between the highest- and second-highest-ranked model on each task, using the full reference set. Δ\Delta is the observed difference in mean macro-F1 between the two models, where the mean is taken over replicates (the values in Table S1); positive values indicate that the top-ranked model scored higher than the runner-up. Confidence intervals were obtained by bootstrapping the test partition (2,000 resamples; peptides resampled as clusters for surface activity). On each resample, macro-F1 was recomputed for every replicate of both models, averaged over replicates (100 for XGBoost, TabPFN and TabFM; five for the endpoint models), and the difference between the two averages taken; the interval is the 2.5th–97.5th percentile of these differences. Because every replicate of a model shares one fixed test partition, a test across replicates would measure seed-to-seed stability rather than performance on unseen peptides; the intervals are therefore conditional on that split. Percentile intervals need not be symmetric about Δ\Delta, and the bootstrap mean differs from Δ\Delta by at most 0.007. The p-value is two-sided and uncorrected for multiple comparisons. No 95% confidence interval excludes zero.

Replicate-to-replicate variability also differed between models and tasks. Some endpoint models had larger standard deviations than the local models, but this pattern was not consistent across all comparisons. GPT-6 Luna Decisions had no variation in macro-F1 at the displayed precision on any task, as did TabFM on beta-sheet formation. Low replicate variability for GPT-6 Luna Decisions indicates that the model is deterministic for the same input, although this was not disclosed by the model provider.

Having established a baseline performance for various foundation models against the XGBoost baseline, we next inspected how and to what extent different inference strategies affect predictive performance.

Inferring one-by-one versus all-at-once

Whereas deterministic classical ML baselines operate by inferring individual examples after training, the transformer-based architecture of the foundation models we tested process the full training and inference data simultaneously. As a result, inference can be done either by one-by-one prediction of individual test samples, or all-at-once inference of the full test set. We compared these two approaches to evaluate the impact of inference mode on the predictive performance. For this, we included two tasks, surface activity and self-assembly, five endpoint models, and two reference sets, full and class-balanced. This gave 20 combinations of task, model, and reference set, with both inference modes evaluated for each combination. Figure 3 shows the paired differences for each task, model, and reference set. Table S2 reports the group-level paired differences in macro-F1, and Table S5 gives the per-stratum results. Positive differences therefore indicate better performance with one-by-one inference.

Improvement in macro-F1 of one-by-one prompting relative to all-at-once prompting for full and class-balanced reference sets

Figure 3. One-by-one versus all-at-once inference with the (a) full and (b) class-balanced reference sets. Each point is the mean paired difference in macro-F1 (one-by-one minus all-at-once) for one model on surface activity or self-assembly, across five paired replicates. Horizontal bars give the 95% t confidence interval for each point. Dark points mark intervals that exclude zero, and light grey points mark intervals that include zero. The diamond gives the mean difference over the ten task and model combinations in a panel, and its bar gives the 95% t confidence interval. Positive values indicate better performance with one-by-one inference.

GroupnnMean difference95% t intervalPaired t ppWilcoxon pp
All endpoint models20+0.0086−0.0056 to +0.02280.22070.2146
LLMs12+0.0140−0.0106 to +0.03860.23520.2661
TDMs8+0.0005−0.0040 to +0.00490.80940.8750
Surface activity10−0.0085−0.0237 to +0.00680.23990.4609
Self-assembly10+0.0257+0.0048 to +0.04650.02120.0234

Table S2.

Table S5. Mode comparison by task, reference set, and model. Values are mean macro-F1; the difference is one-by-one minus all-at-once; intervals are 95% tt intervals from the five replicate-level paired differences. Differences and intervals were calculated from unrounded values.

TaskReference setModelOne-by-oneAll-at-onceDifference95% t interval
Surface activityFullGPT-6.1 Sol0.7620.769−0.007−0.0324 to +0.0188
DeepSeek-V4.1 Flash0.6800.733−0.053−0.1405 to +0.0340
Gemma-4 31B0.6760.684−0.008−0.0456 to +0.0296
Jev 1.140.6960.686+0.010−0.0059 to +0.0257
GPT-6 Luna Decisions0.5970.5970.000+0.0000 to +0.0000
BalancedGPT-6.1 Sol0.7270.723+0.004−0.0118 to +0.0202
DeepSeek-V4.1 Flash0.6720.659+0.013−0.0537 to +0.0796
Gemma-4 31B0.6030.642−0.039−0.0546 to −0.0238
Jev 1.140.7140.718−0.004−0.0292 to +0.0204
GPT-6 Luna Decisions0.6340.6340.000+0.0000 to +0.0000
Self-assemblyFullGPT-6.1 Sol0.8570.813+0.044+0.0288 to +0.0587
DeepSeek-V4.1 Flash0.8160.807+0.009−0.0582 to +0.0767
Gemma-4 31B0.8260.788+0.038+0.0218 to +0.0546
Jev 1.140.8360.830+0.006−0.0141 to +0.0251
GPT-6 Luna Decisions0.7480.7480.000+0.0000 to +0.0000
BalancedGPT-6.1 Sol0.8650.830+0.036+0.0012 to +0.0700
DeepSeek-V4.1 Flash0.8360.790+0.046−0.0202 to +0.1126
Gemma-4 31B0.8510.766+0.085+0.0734 to +0.0973
Jev 1.140.8090.817−0.007−0.0386 to +0.0241
GPT-6 Luna Decisions0.6590.6590.000+0.0000 to +0.0000

Table S5.

The pilot gives 20 task, model, and reference-set combinations in total. Local models are not included. Across all 20 combinations, the overall mean difference was +0.0086 macro-F1 (95% confidence interval: −0.0056 to +0.0228). Neither the paired t-test (p = 0.2207) nor the Wilcoxon signed-rank test (p = 0.2146) showed a statistically significant overall difference between the modes. No significant difference was detected within either model family. Task-specific results differed. For self-assembly, one-by-one inference improved mean macro-F1 by 0.0257 (95% confidence interval: 0.0048 to 0.0465; p = 0.0212). For surface activity, the mean difference was −0.0085 (95% confidence interval: −0.0237 to +0.0068; p = 0.2399), with no statistically significant difference between modes. Furthermore, the comparison also showed no statistically significant difference within either model family. For the three LLMs, the mean difference was +0.0140 (95% confidence interval: −0.0106 to +0.0386; p = 0.2352). For the two TDMs, the mean difference was +0.0005 (p = 0.8094). GPT-6 Luna Decisions returned identical predictions in both modes for all four combinations of task and reference set, across five replicates per combination.

Together, these results show no consistent significant effect of inference mode. The self-assembly subset improved with one-by-one inference, whereas the overall and model-family comparisons did not. Although one-by-one inference requires more API requests, this approach is more robust against failed requests. In the case of one-by-one inference, failures only affect a single target sample rather than a complete batch.

Zero-shot and corrupted-label conditions

The performance comparison identifies the highest-performing models, but it does not establish whether those predictions are driven by the supplied reference examples or by information acquired during pretraining. In fact, the identical predictions returned for both inference modes for GPT-6 Luna Decisions appears to imply some models may predict from memory rather than inference. To assess the dependence of endpoint models on the provided context, we compared standard in-context learning with two conditions: a zero-shot condition, in which no labeled reference examples were supplied, and a corrupted-label condition, in which the reference labels were flipped.

Figure 4 summarizes these comparisons, where all results use the full training sets and one-by-one inference. Corresponding results for the balanced training sets are provided in Figure S2 and Table S6. Local models were not included because analogous label-corruption ablations are not applicable to conventionally trained models.

Label ablations for the five endpoint models on the five tasks

Figure 4. Label ablations for the five endpoint models on the five tasks. Within each model, bars give mean macro-F1 for standard in-context learning (dark blue), zero-shot prediction (light blue), and corrupted reference labels (hatched). Models are grouped by family, with LLMs before TDMs. Error bars give the standard deviation across five replicates. Dashed horizontal lines give the macro-F1 of a majority-class predictor. This predictor always selects the most frequent test label. All results use full training and one-by-one inference. Local models are not included because label ablations do not apply to them.

Table S6. Label-control macro-F1 (mean (SD)) for the five endpoint models on the five tasks. Standard full and standard balanced use the corresponding reference set. Zero-shot uses no reference rows and is shared by the full and balanced comparisons. Corrupted-full and corrupted-balanced labels changed every reference label; test labels were unchanged. n = 5 independent repeats.

TaskModelStandard fullZero-shotCorrupted fullStandard balancedCorrupted balanced
Surface activityGPT-6.1 Sol0.762 (0.010)0.291 (0.003)0.223 (0.010)0.727 (0.020)0.251 (0.012)
DeepSeek-V4.1 Flash0.680 (0.036)0.283 (0.021)0.261 (0.011)0.672 (0.024)0.265 (0.041)
Gemma-4 31B0.676 (0.005)0.270 (0.000)0.233 (0.016)0.603 (0.009)0.270 (0.017)
Jev 1.140.696 (0.017)0.397 (0.020)0.284 (0.031)0.714 (0.015)0.223 (0.009)
GPT-6 Luna Decisions0.597 (0.000)0.375 (0.000)0.284 (0.000)0.634 (0.000)0.361 (0.000)
β\beta-sheet formationGPT-6.1 Sol0.781 (0.018)0.546 (0.009)0.176 (0.011)0.746 (0.013)0.225 (0.030)
DeepSeek-V4.1 Flash0.791 (0.022)0.647 (0.043)0.200 (0.011)0.774 (0.023)0.220 (0.027)
Gemma-4 31B0.785 (0.016)0.610 (0.024)0.215 (0.024)0.763 (0.010)0.246 (0.009)
Jev 1.140.753 (0.010)0.722 (0.030)0.218 (0.020)0.784 (0.010)0.194 (0.009)
GPT-6 Luna Decisions0.784 (0.000)0.629 (0.000)0.203 (0.000)0.766 (0.000)0.257 (0.000)
Self-assemblyGPT-6.1 Sol0.857 (0.011)0.696 (0.011)0.164 (0.025)0.865 (0.014)0.189 (0.024)
DeepSeek-V4.1 Flash0.816 (0.024)0.688 (0.048)0.216 (0.019)0.836 (0.036)0.254 (0.022)
Gemma-4 31B0.826 (0.025)0.685 (0.018)0.184 (0.011)0.851 (0.014)0.223 (0.014)
Jev 1.140.836 (0.006)0.343 (0.009)0.202 (0.000)0.809 (0.021)0.235 (0.015)
GPT-6 Luna Decisions0.748 (0.000)0.665 (0.000)0.174 (0.000)0.659 (0.000)0.256 (0.000)
Breast-cancer activityGPT-6.1 Sol0.538 (0.014)0.235 (0.010)0.131 (0.001)0.518 (0.017)0.177 (0.021)
DeepSeek-V4.1 Flash0.453 (0.066)0.203 (0.032)0.141 (0.016)0.380 (0.017)0.175 (0.045)
Gemma-4 31B0.455 (0.020)0.198 (0.037)0.140 (0.010)0.504 (0.026)0.223 (0.000)
Jev 1.140.504 (0.012)0.297 (0.040)0.156 (0.016)0.506 (0.030)0.188 (0.000)
GPT-6 Luna Decisions0.437 (0.000)0.159 (0.000)0.164 (0.000)0.505 (0.000)0.183 (0.000)
Lung-cancer activityGPT-6.1 Sol0.389 (0.029)0.306 (0.028)0.220 (0.013)0.526 (0.030)0.233 (0.000)
DeepSeek-V4.1 Flash0.400 (0.043)0.331 (0.058)0.192 (0.013)0.500 (0.054)0.196 (0.039)
Gemma-4 31B0.389 (0.023)0.323 (0.051)0.203 (0.031)0.552 (0.022)0.266 (0.029)
Jev 1.140.493 (0.012)0.254 (0.000)0.260 (0.001)0.480 (0.025)0.179 (0.002)
GPT-6 Luna Decisions0.351 (0.000)0.078 (0.000)0.154 (0.000)0.426 (0.000)0.203 (0.000)

Table S6.

Balanced label controls chart

Figure S2. Figure S2. Balanced label controls. Macro-F1 for the five endpoint models on the five tasks with standard in-context learning (dark blue), zero-shot prediction (light blue), and corrupted class-balanced labels (hatched). Panels correspond to tasks, and bars are grouped by model. Dashed horizontal lines give the macro-F1 of a majority-class predictor. This predictor always selects the most frequent test label. Standard in-context learning uses the class-balanced reference set. The vertical axis is raw macro-F1, and error bars show the standard deviation across n=5n = 5 independent repeats.

Both conditions reduced the performance of every endpoint model on every task. Relative to standard in-context learning, the decrease in macro-F1 ranged from 0.030 to 0.492 in the zero-shot condition and from 0.169 to 0.693 in the corrupted-label condition. The largest zero-shot reduction was observed for the Jev TDM on self-assembly (Δ=0.492\Delta= 0.492), whereas the largest corrupted-label reduction was observed for GPT-6.1 Sol on self-assembly (Δ=0.693\Delta= 0.693). This is the largest corrupted-label reduction in Figure 4.

Corrupted labels were more damaging than missing labels in 22 of the 25 model–task combinations. The mean loss across the five tasks was 0.223 in the zero-shot condition and 0.432 in the corrupted-label condition. Because corrupted-label scores were nearly constant across models and tasks (mean 0.200, range 0.131–0.284), the magnitude of the corrupted-label loss largely follows the standard-ICL baseline. The comparison between corrupted and zero-shot losses is therefore the more informative test of label dependence. The corrupted-label loss was larger on all five tasks for GPT-6.1 Sol, DeepSeek-V4.1 Flash, and Gemma-4 31B, on four tasks for Jev 1.14, and on three tasks for GPT-6 Luna Decisions. The three exceptions were GPT-6 Luna Decisions on breast-cancer activity (loss 0.273 under corrupted labels versus 0.277 under zero-shot prediction) and on lung-cancer activity (0.197 versus 0.273), and Jev 1.14 on lung-cancer activity (0.233 versus 0.239). Table S7 gives the cell values, and Appendix F.5 gives the comparison.

Table S7. Label-control losses (standard ICL minus condition) for the cells in Table S6. Zero-shot loss uses the standard full score; corrupted-full and corrupted-balanced losses use the matching standard score. The final column compares the corrupted-full loss with the zero-shot loss: Yes or No.

TaskModelZero-shot lossCorrupted-full lossCorrupted-balanced lossCorrupted-full loss > zero-shot loss
Surface activityGPT-6.1 Sol0.4710.5390.476Yes
Surface activityDeepSeek-V4.1 Flash0.3970.4190.407Yes
Surface activityGemma-4 31B0.4060.4430.332Yes
Surface activityJev 1.140.2990.4130.490Yes
Surface activityGPT-6 Luna Decisions0.2220.3130.273Yes
β\beta-sheet formationGPT-6.1 Sol0.2350.6050.521Yes
β\beta-sheet formationDeepSeek-V4.1 Flash0.1440.5910.554Yes
β\beta-sheet formationGemma-4 31B0.1750.5700.517Yes
β\beta-sheet formationJev 1.140.0300.5350.590Yes
β\beta-sheet formationGPT-6 Luna Decisions0.1550.5800.509Yes
Self-assemblyGPT-6.1 Sol0.1610.6930.677Yes
Self-assemblyDeepSeek-V4.1 Flash0.1280.6000.582Yes
Self-assemblyGemma-4 31B0.1410.6410.628Yes
Self-assemblyJev 1.140.4920.6340.574Yes
Self-assemblyGPT-6 Luna Decisions0.0830.5740.404Yes
Breast-cancer activityGPT-6.1 Sol0.3030.4070.341Yes
Breast-cancer activityDeepSeek-V4.1 Flash0.2500.3120.205Yes
Breast-cancer activityGemma-4 31B0.2570.3150.281Yes
Breast-cancer activityJev 1.140.2070.3480.319Yes
Breast-cancer activityGPT-6 Luna Decisions0.2770.2730.322No
Lung-cancer activityGPT-6.1 Sol0.0830.1690.292Yes
Lung-cancer activityDeepSeek-V4.1 Flash0.0690.2070.304Yes
Lung-cancer activityGemma-4 31B0.0670.1860.286Yes
Lung-cancer activityJev 1.140.2390.2330.301No
Lung-cancer activityGPT-6 Luna Decisions0.2730.1970.223No

Table S7.

The larger loss in performance for inference with corrupting labels than with missing labels indicates that the models generally used the supplied labels and not only the similarity between the query and the reference sequences. A model that relied only on sequence similarity would not be expected to lose more performance when the labels were corrupted than when they were removed.

The three LLMs and the two decision models all showed substantial reductions in performance when labels were corrupted or missing. The overall stronger decrease when labels were corrupted is consistent with the endpoint models using the in-context reference labels when making predictions. However, the conditions do not demonstrate that the models rely exclusively on the provided context. A model may simultaneously use the supplied examples and information acquired during pretraining, including knowledge of peptide sequences, peptide properties, or related published datasets.

The interpretation is particularly informative for the surface-activity dataset, as it was collected from a recent study that was published after the training cutoffs of the evaluated endpoint models.6 Hence, the exact dataset is unlikely to have been present during pretraining. A comparatively strong loss in performance is observed upon scrambling labels. Conversely, the loss in performance scrambled labels of sequences from datasets that were published before training cutoffs is relatively modest. This observation suggests that the LLMs may benefit from information supplied in-context, but also appears to suggest that data to which the model has been exposed during pretraining may leak into the predictions. While such reliance on knowledge that is obtained from pretraining may be advantageous when predicting or optimizing properties in low-data regimes, these effects are also likely to introduce potentially unwanted biases when searching for chemical novelty.

Discussion

This study evaluated whether pretrained foundation models can predict peptide properties through in-context learning in low-data settings. Across five peptide classification tasks, ICL-based approaches achieved predictive performance that was generally competitive with a strong classical baseline. However, performance remained strongly task-dependent, and no model family consistently outperformed the others. GPT-6.1 Sol achieved the highest mean macro-F1 on three tasks, whereas XGBoost and TabFM each achieved the highest mean on one task. The absence of a universally best model was particularly notable given the large differences in evaluation cost. Endpoint-model API costs, including retries, varied by more than an order of magnitude across the benchmark, from USD 0.97 for Jev 1.14 to USD 27.74 for DeepSeek-V4.1 Flash, yet higher cost did not consistently translate into better predictive performance. Jev 1.14 achieved a higher mean macro-F1 across the five tasks than DeepSeek-V4.1 Flash and Gemma-4 31B (USD 11.80), and the highest mean was achieved by GPT-6.1 Sol (USD 27.62), the second most expensive endpoint model. Table S8 provides additional information regarding costs and token usage for endpoint models.

A notable finding is that tabular foundation models achieved performance comparable to both classical machine-learning models and language-based foundation models despite the absence of peptide-specific pretraining. TabFM achieved the highest performance on the lung-cancer activity task, and both TFMs remained competitive across multiple datasets. These results suggest that strong statistical priors learned during large-scale pretraining on tabular data can be sufficient for many low-data peptide prediction problems.

The inference-mode test showed no significant overall difference between one-by-one and all-at-once prediction. Nevertheless, one-by-one inference was selected for the main evaluation because it improved performance on the self-assembly task and limited the impact of a failed request to a single target sample. Because the inference comparison included only two tasks, this choice should be viewed as a practical decision rather than evidence that one inference mode is generally superior.

Overall, the results indicate that foundation-model-based ICL is a viable alternative to conventional machine-learning workflows for classification tasks in peptide science, where datasets frequently contain only tens to hundreds of labelled sequences. The principal advantage of ICL is the reduced barrier to model deployment. Although ICL still requires feature engineering, it avoids hyperparameter optimization and training, lowering the practical burden of deployment compared to conventional machine learning. While our benchmark suggest slightly better average performance for ICL methods, performance varied substantially across tasks, and the strong drop in zero-shot performance on the surface-tension dataset suggests that pretraining data leakage may inflate language-model performance. Consequently, classical baseline models, such as XGBoost remained highly competitive.

Limitations and future directions

Several limitations should be considered when interpreting these results. First, the benchmark comprises five peptide classification tasks and a single train-test split for each task. The reported replicate statistics therefore quantify replicate-to-replicate variability on a fixed split rather than variation across independent dataset partitions. Second, the endpoint models were evaluated through external APIs. These models can change over time, may not provide deterministic outputs, and cannot always be reproduced exactly.34 Although repeated evaluations were used to estimate variability, exact replication may not be possible in the future. Third, the comparison between one-by-one and all-at-once inference was based on only two tasks and was subsequently used to define the inference mode for the main evaluation. The inference-mode results should therefore be regarded as exploratory. Fourth, the benchmark evaluates classification tasks only. The conclusions may not be generalized to regression problems, generative peptide design, ranking tasks, or optimization settings that are common in peptide discovery workflows. Finally, this study evaluates retrospective prediction only. It does not assess active learning, iterative experimental campaigns, uncertainty-guided acquisition, or prospective prediction on newly generated peptide datasets. These applications are particularly relevant for peptide discovery and materials development and represent important directions for future work.

The datasets used here also cover a limited region of peptide chemical space and predominantly sequence-based representations. Additional benchmarks containing broader peptide chemistries, richer molecular descriptors, and prospectively collected experimental data will be required to determine how broadly the present conclusions generalize.

Conclusions

We systematically evaluated in-context learning for peptide property prediction across five low-data classification tasks spanning self-assembly, surface activity, spectroscopy-derived structural descriptors, and biological activity. Large language models, typed decision models, and tabular foundation models all achieved predictive performance that was competitive with or outperformed an XGBoost classical baseline, demonstrating that useful peptide property prediction is possible without task-specific model training.

The zero-shot and corrupted-label conditions showed that all endpoint models benefited from the supplied reference examples. Corrupted labels produced larger reductions than the removal of examples in 22 of the 25 model–task combinations, which is consistent with the models using the supplied labels rather than only sequence similarity or pretrained knowledge. These analyses cannot exclude contributions from pretraining knowledge, however, and emphasize the need for contamination-aware benchmarking and prospective validation in future studies.

Overall, our results suggest that in-context learning is a practical addition to the peptide-machine-learning toolbox rather than a replacement for conventional methods. For small peptide datasets, foundation models offer a low-barrier route to predictive modelling, but model selection should remain task-specific and should consider predictive performance, reproducibility, and operational cost.

Data Availability

The run records are available in the Zenodo repository at 10.5281/zenodo.23834634. Each record provided in the repository gives the predictions, the confusion matrix, the macro-F1, and the per-class precision, recall, F1, and support. Table S1–Table S7 can be reproduced from these records. The datasets and split files are provided with this article, and Table S9 gives the split counts. The code used to obtain the results is supplied at https://www.github.com/BigChemistry-RobotLab/icl-benchmark.

Acknowledgements

We acknowledge funding from the National Growth Fund project “Big Chemistry” (1420578).

Conflicts of Interest

The authors declare no conflicts of interest.

appendix.pdf

Supplementary methods

Data preparation and split construction

We used five peptide-classification tasks: surface activity[1], β\beta-sheet formation[2], self-assembly[3], breast-cancer activity, and lung-cancer activity[4]. For surface activity, we grouped 4,233 measurements from 154 peptide sequences by sequence and pH. We kept the measurement with the lowest surface tension in each group. When values tied, we kept the first row in the source file. The resulting dataset contained 409 sequence-pH examples. All peptides had a C16 tail, and measurements were retained at pH 3, 7, and 11. The model inputs included sequence, pH, and tail length. We excluded concentration, solvent, purity, instrument metadata, and experiment identifiers. We classified a peptide as surface-active when its surface tension was below 50 mN/m.

For β\beta-sheet formation and self-assembly, we used the source examples and labels. We mapped β\beta-sheet classes 0.0 and 1.0 to false and true, respectively, and self-assembly classes 0 and 1 to false and true. For the breast- and lung-cancer tasks, we retained only experimentally validated entries from the Grisoni studies and kept the original labels: experimentally inactive, moderately active, and very active.

The published Houx et al. dataset includes 16 additional PA sequences that were not in the dataset used here: SKR, SRI, HSH, KPE, QEH, RYK, RQA, QAH, DRL, DHK, KSH, SQH, LEH, NKR, KDD, and QSH. This broader scope does not affect the conclusions of this work.

Case and modification notation were kept as in the source tables: lowercase residues occur in two breast-cancer sequences, and four self-assembly sequences contain the modified-residue notation P′P'. The source tables do not provide a complete modification dictionary, so no chemical meaning was assigned to unverified notation. No sequence normalization or deduplication was applied. No sequence is repeated in the four non-surface tables, so their row-level splits also separate sequences.

The four non-surface tasks used a stratified 80:20 split with seed 42. The surface-activity split used a shuffled, stratified group five-fold split by peptide sequence. The fold that minimized the sum of the absolute test-fraction deviation from 20% and the absolute class-proportion deviations from the cleaned dataset was selected, and ties went to the lower fold number. Class-balanced subsets were drawn from the full training rows with the same fixed seed and contained an equal number of rows per class. Table S9 gives cleaned and per-class counts, the split sizes, and the balanced training sizes.

Models and input representations

Eight models from four families were compared. The endpoint models were three large language models (LLMs; GPT-6.1 Sol, DeepSeek-V4.1 Flash, and Gemma-4 31B) and two typed decision models (TDMs; Jev 1.14 and GPT-6 Luna Decisions). The local models were two tabular foundation models (TabPFN v2 and TabFM) and one gradient-boosted tree baseline (XGBoost). We selected TabPFN v2 instead of its newer versions due to licensing restrictions and access gating on the newer models.

Endpoint models were accessed through OpenRouter with default provider-selected hyperparameters. DeepSeek-V4.1 Flash used low reasoning effort, while the other models used their provider default reasoning effort. The decision endpoints did not accept a temperature or seed. The tabular foundation models ran on one machine with an NVIDIA RTX 4090 GPU and 64 GB of RAM, while XGBoost used the CPU.

Endpoint models received peptide sequences as text. For surface activity, the prompt also included pH and tail length. TabPFN v2 and TabFM received positional integer tokens, with pH and tail length supplied as numeric features for surface activity, while XGBoost received positional one-hot tokens and the same numeric features. The encoder used a 43-token vocabulary, preserved case and modification notation such as P’, and padded sequences to the task maximum length (3, 5, 24, 38, and 38 residues). Padding tokens were encoded separately. TabPFN v2 used its default classifier with categorical feature indices from 0 to the task maximum length minus 1. TabFM used its default row and feature limits. XGBoost used the library defaults with 0.8 row and 0.8 column subsampling and the replicate seed as the random state.

Endpoint models used one-by-one inference. TabPFN v2 and TabFM used the training rows as in-context data without weight updates. XGBoost was fitted to the training rows in each replicate. All local models predicted the same fixed test rows. The costs in Table S8 are provider-reported totals over all requests.

In-context learning protocol and label controls

Each LLM prompt contained the task instruction, the labeled reference set, the target peptides, the allowed labels, and a JSON output schema. The schema required one label per target and a short reasoning field. The TDMs received the reference rows in a structured state and one choice question per target. No retries were allowed during our requests, and each failed request was repeated in a new API call until a valid response was received. In one-by-one mode, each target peptide was submitted in a separate request that repeated the full reference set. In all-at-once mode, all target peptides were submitted in one request, subject to the TDM limit of 200 questions per request. Zero-shot prompts omitted the reference rows. The main evaluation used one-by-one inference. An example request and response for surface activity is described below. The prompt for the other tasks was similar, with the task instruction and allowed labels changed to match the task.

Example prompt and response (surface activity)

Large Language Model

System message

Predict the outputs for every target row, in the same order. Use the reference rows to infer the input to output mapping. If no reference rows are provided, use the task context for zero-shot prediction. Follow all task-specific output labels, units, formats, headers, and schema. Treat empty cells as missing values, not zero. Missing reference outputs are not examples of valid labels or predictions. Return exactly one prediction per target row in the required structured format. After the predictions, briefly state the basis for the predictions and any relevant uncertainty. Use task context and relevant general knowledge to interpret the inputs. When reference rows are available, prefer patterns supported by the examples. State relevant uncertainty. Do not invent evidence or claim validation that was not performed.

User message

Context: Predict the surface activity from the supplied examples. Every candidate is a peptide amphiphile with an N-terminal C16 tail and a tripeptide headgroup. Measurements at pH 3 use citrate buffer, measurements at pH 7 use phosphate buffer, and measurements at pH 11 use Tris buffer. A peptide is considered surface active if it has a surface tension value below 50 mN/m. Allowed output labels: {"surface_active": ["false", "true"]}.

Reference data:

Tail length,Peptide sequence,pH,Surface active 16,SAH,11,false ...

Predict “Surface active” for the following targets in row order. Use each “target_id” value as its key in the predictions object:

target_id,Tail length,Peptide sequence,pH target_0,16,GRH,7 ...

Response format

{
  "predictions": {
    "[target]": {"[label]": "[predicted class]"},
    ...
  },
  "reasoning": "[basis and uncertainty]"
}

Typed Decision Model

State

[
  {
    "role": "user",
    "content": [
      {
        "type": "input_text",
        "text": "Use the labeled reference rows to infer how the
          task inputs map to the output. If no reference rows are
          provided, use the task context for zero-shot prediction.
          Empty cells are missing values, not zero.
          \nTask context:\nPredict the surface activity from the
          supplied examples. Every candidate is a peptide
          amphiphile with an N-terminal C16 tail and a tripeptide
          headgroup. Measurements at pH 3 use citrate buffer,
          measurements at pH 7 use phosphate buffer, and
          measurements at pH 11 use Tris buffer. A peptide is
          considered surface active if it has a surface tension
          value below 50 mN/m."
      },
      {
        "type": "input_json",
        "data": {
          "references": {
            "columns": ["Tail length", "Peptide sequence", "pH",
              "Surface active"],
            "rows": [["16", "SAH", "11", "false"], "..."]
          }
        }
      }
    ]
  }
]

Questions

{
  "target_0": {
    "type": "choice",
    "instructions": "Is the peptide C16-GRH surface active at pH 7?",
    "criteria": {"false": "false", "true": "true"}
  },
  "...": "..."
}

Long string values in the state block are wrapped for display only.

Reference sets were the full training set, the class-balanced training set, a corrupted version of either set, or empty for zero-shot. In the binary tasks, corruption inverted every reference label. In the three-class cancer tasks, corruption shifted every reference label to the next class (inactive → moderately active → very active → inactive). Every reference label changed, every class remained present in the corrupted set, and test labels were unchanged. Balanced training sets were formed by undersampling each training class to the smallest training class.

Endpoint models were run in five independent repeats, and no seed was used. Local models were run in 100 repeats with seeds 42-141. Test rows and labels were fixed across models, reference sets, and repeats. The zero-shot results were shared by the full and balanced comparisons. The complete evaluation comprised 3,725 replicate evaluations.

Evaluation and statistics

Macro-F1 was the unweighted mean of the per-class F1 over the declared classes. Positive-class F1 was the F1 of the true class in the binary tasks (this was not defined for the three-class cancer tasks). Precision, recall, F1, and test support were calculated for each class. Unless stated otherwise, values are means over repeats, and SD is the sample standard deviation.

The one-by-one and all-at-once comparison used 20 matched strata: two tasks (surface activity and self-assembly), five endpoint models, and two reference sets (full and class-balanced). For Table S5, the five replicate-level paired differences in each stratum gave a mean difference and a 95% t interval. The difference was one-by-one minus all-at-once. For Table S2, replicate scores were averaged within each stratum before the 20 matched differences were formed. The summary tests therefore used n=20n = 20 paired differences. These differences were tested with a two-sided paired t-test and a two-sided Wilcoxon signed-rank test. The tests were exploratory, and no correction for multiple comparisons was applied.

Label-control losses were standard ICL minus the condition. The zero-shot loss used the standard full-training score. The corrupted-full and corrupted-balanced losses used the matching standard score. A positive loss indicates that the condition reduced macro-F1.

Supplementary tables

Table S1. Full-training macro-F1 (mean ±\pm SD) for the eight models on the five tasks. Endpoint models used one-by-one inference (n=5n = 5); local models used their training rows directly (n=100n = 100).

Table S2. Summary of the inference mode comparison. Mean differences are macro-F1 (one-by-one minus all-at-once); intervals are 95% t intervals. The tests were two-sided and exploratory.

When a model had no true positives for the very-active class, precision and F1 are reported as 0.000. Under full training on lung cancer, four models had zero recall for the very active class: GPT-6.1 Sol, Gemma-4 31B, GPT-6 Luna Decisions, and TabPFN v2. GPT-6.1 Sol, Gemma-4 31B, and TabPFN v2 emitted the class only as false positives; GPT-6 Luna made no very-active prediction.

Losses were calculated from unrounded values.

Costs are provider-reported totals over all API requests, including retries. The per-run records provided with this article cover completed runs only and therefore sum to a lower total. Totals are calculated from unrounded values.

Supplementary figures

Bibliography

References

  1. [1]Houx, J.; Kong, M. J. W.; Tao, S.; Dankloff, P. F. J.; Martens, G.; Hoeksel, J.; Robinson, W. E.; Huck, W. T. S.; Mabesoone, M. F. J. Data-Driven Discovery of pH-Dependent Surfactant Behavior in a Peptide Amphiphile Library, 2026. https://doi.org/10.26434/chemrxiv.15010226.v1.
  2. [2]Talluri, Y. N.; Sankaranarayanan, S. K.; Fry, H. C.; Batra, R. Discovery of Unconventional and Nonintuitive Self-Assembling Peptide Materials Using Experiment-Driven Machine Learning. Science Advances 2025, 11 (24), eadt9466. https://doi.org/10.1126/sciadv.adt9466.
  3. [3]Njirjak, M.; Žužić, L.; Babić, M.; Janković, P.; Otović, E.; Kalafatovic, D.; Mauša, G. Reshaping the Discovery of Self-Assembling Peptides with Generative AI Guided by Hybrid Deep Learning. Nature Machine Intelligence 2024, 6 (12), 1487–1500. https://doi.org/10.1038/s42256-024-00928-1.
  4. [4]Grisoni, F.; Neuhaus, C. S.; Gabernet, G.; Müller, A. T.; Hiss, J. A.; Schneider, G. Designing Anticancer Peptides by Constructive Machine Learning. ChemMedChem 2018, 13 (13), 1300–1302. https://doi.org/10.1002/cmdc.201800204.

Paper details

vlnc:W_bngcsv67
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents