alphaFeedback

WorksAgricultural and Biological Sciences

Ablating Design Choices for Protein Ensemble Generators Across Structural Regimes

AI contributions
Not recorded for this paper
View details

Abstract

Generative models of protein conformational ensembles inherit much of their design from structure prediction for folded proteins. Ordered and disordered proteins, however, occupy distinct structural regimes, and we investigate how design choices hold up across them. Using a minimal Cαα flow-matching model, we ablate choices of architecture, sequence conditioning, rigid-body alignment and training source, and evaluate each variant against experimental SAXS and NMR chemical-shift data partitioned into ordered, disordered, multi-domain and transitional proteins. Mixture-of-experts routing, a static pair representation and a disorder probe have no detectable effect, a smaller protein language model performs worse on ordered proteins but not on disordered ones, and alignment in the loss degrades agreement with SAXS. Replacing coarse-grained simulations of disordered proteins with fragment-assembled ensembles improves agreement with chemical shifts at a cost to SAXS, while an equal mixture of the two keeps most of the gain at a smaller cost. This points to the training data, rather than the architecture, as the main constraint.

Introduction

Structure prediction is largely solved for proteins with a single well-defined fold [1, 2], yet a substantial fraction of biologically active proteins do not adopt one. Intrinsically disordered proteins and regions (IDPs/IDRs), estimated to make up 20–40% of the human proteome [3], instead populate heterogeneous conformational ensembles whose flexibility underlies their function [4, 5], and many human proteins contain both ordered and disordered regions. Modelling these ensembles underpins applications in drug discovery and protein design [6, 7].

While it is tempting to treat the ordered and disordered regimes as endpoints of a continuum, it is more principled to regard them as distinct, as evident in their scaling behaviour (Rg∼NνR_g \sim N^\nu). Ordered proteins adopt a unique compact native fold, and across the proteome they scale with ν≈1/3\nu\approx1/3–2/52/5, close to the collapsed-globule limit of a polymer in poor solvent [8]. Disordered proteins instead form polymer-like ensembles governed by excluded volume and conformational entropy, from the θ\theta-regime (ν≈1/2\nu\approx1/2) for typical sequences to the self-avoiding-walk limit (ν≈0.59\nu\approx0.59) for highly charged or otherwise expanded ones [9], determined by sequence-dependent physicochemistry such as net charge, charge patterning and hydropathy [10, 5]. Multi-domain proteins, in which folded domains are joined by disordered linkers, reinforce this view rather than softening it: each domain or linker sits in one regime and the protein is their composition.

The experimental methods that underpin structure prediction for folded proteins do not transfer to disorder. Crystallography relies on a well-ordered lattice and single-particle cryo-EM reconstruction requires a small number of alignable states, and as a result the PDB is depleted in proteins with disorder. Instead the probes available for disordered proteins measure ensemble-averaged observables, principally small-angle X-ray scattering (SAXS) [11] and NMR chemical shifts (CS) [12].

As these observables constrain an ensemble only indirectly, ensembles are more often generated by simulation than inferred from data. For molecular dynamics simulations there is an accuracy–efficiency trade-off: all-atom force fields produce high-fidelity trajectories but at an immense computational cost, while coarse-grained methods provide proteome-scale coverage at reduced resolution [13]. Residue-level models such as CALVADOS [14, 15] are calibrated top-down against experimental SAXS and PRE data, and so are well tuned for global chain dimensions while sacrificing local geometry. An alternative approach is fragment assembly [16], which builds ensembles by stitching together short backbone fragments drawn from structural databases. This method aims for broad coverage and accurate local backbone geometry, but its sampled populations do not follow a Boltzmann distribution.

Recent advances in generative modelling have opened another route, in which models are trained to sample conformational ensembles directly, amortising over sequence so that no simulation is required at inference. For disordered regions, idpGAN [17], idpSAM [18] and STARLING [19] learn from coarse-grained or implicit-solvent simulations. AlphaFlow and ESMFlow [20] fine-tune folded-protein structure predictors with flow matching. BioEmu [21] is trained on all-atom molecular dynamics alongside folded structures. PepTron [22] and IDPFold2 [23] instead target the disordered and multi-domain regimes, drawing on proteome-scale ensembles of the kind described above. Several of these models carry over architectural elements from structure prediction, such as pair representations and structure modules, while SimpleFold [24] shows that folding itself can be learned with general-purpose transformer blocks and no pair representation. A complementary line of work predicts ensemble-averaged quantities directly from sequence, without generating an ensemble [25, 26].

In this work we investigate how conformational modelling choices fare across structural regimes, since a choice may matter in one regime and not in another. To this end we ablate a range of design choices on a minimal CαC\alpha flow-matching model, and evaluate each variant against experimental SAXS and chemical shift data partitioned into ordered, disordered, multi-domain and transitional proteins.

Evaluation

Since the conformational ensembles available as training data are inherently approximate, we focus our analysis on agreement with experiment. We evaluate against PeptoneBench [22], which scores generated ensembles against:

  • SAXS (n=439n = 439, from the SASBDB [27]): constrains global dimensions and shape. It is largely insensitive to fold topology, and so agreement on ordered proteins tests compactness rather than precise fold correctness.

  • CS (n=659n = 659, from the BMRB [28]): constrains local backbone geometry, but carries little information about global chain dimensions.

PeptoneBench computes ensemble-averaged observables via forward models, reporting an RMSE normalised by forward-model and experimental uncertainty, so that values near 1 indicate agreement within error. The benchmark’s default summary statistic is LOWESS-AUC, which integrates the RMSE against a continuous disorder metric, the mean G-score (0=0 = ordered, 1=1 = disordered). We score each ensemble as generated, retaining all conformers scored by the forward models and weighting them equally. We do not apply a physicality filter, so that every model is scored on the same proteins and on its full ensemble (see Appendix § 10). PeptoneBench can also refit the conformer weights to the measurement, and we report scores after this reweighting in Appendix § 12. Reweighting requires the measurement itself, which is unavailable in practice, so ensembles as generated are our primary result.

To resolve performance by regime, we partition PeptoneBench into four cells by G-score. To separate multi-domain proteins we define a block as a contiguous run of ≥20\ge20 residues with G<0.3G < 0.3 (rigid) or G>0.7G > 0.7 (flexible), and assign all proteins as follows (cell counts and the scored cohort in Appendix § 10):

  • multi-domain: contains at least one block of each kind.

  • ordered / disordered: mean G<0.3G < 0.3 and G>0.7G > 0.7, respectively.

  • transitional: the remainder, absorbing borderline cases to sharpen the ordered and disordered cells.

The ordered and disordered cells isolate the two regimes of § 1, and the multi-domain cell captures their composition within a single chain.

We compare each variant with the base model protein by protein. For each protein we take the ratio of the variant’s RMSE to the base model’s, each averaged geometrically over training seeds, and summarise a cell by the geometric mean of these ratios, reported as a percentage change. Its 95% confidence interval accounts for variation across both proteins and training seeds (Appendix § 11), and we regard a change as negligible when this interval lies within ±5%.

In addition, we report the Flory scaling exponent ν\nu from a population fit of Rg∼NνR_g \sim N^\nu within each cell. We compute RgR_g per conformer from the Cα coordinates, weighted by residue mass and averaged over the ensemble, and fit ν\nu as the least-squares slope of log⁡Rg\log R_g against log⁡N\log N across the proteins of a cell, with a standard error from a bootstrap over proteins. Unlike the benchmark scores, ν\nu does not depend on a forward model or experimental data, and so provides a consistency check on whether a model reproduces the chain statistics characteristic of each regime (§ 1).

Base model and ablations

Our base model (35.7M parameters) is a sequence-conditioned Cα flow-matching model predicting a per-residue velocity vθv_\theta targeting vt=x1−ϵv_t = x_1 - \epsilon under a linear schedule xt=tx1+(1−t)ϵx_t = t x_1 + (1-t)\epsilon. Noised coordinates and frozen ESM2-650M embeddings [29] are processed by a transformer backbone utilising RoPE [30] and AdaLN-zero time conditioning [31]. We train via a per-residue MSE loss with random SO(3) augmentation. The Cα representation deliberately matches the resolution of the coarse-grained training ensembles, with atomic detail recovered downstream via back-mapping with cg2all [32]. Full architectural, training and sampling details are given in Appendix § 6. An exploration of mini-batch optimal transport pairing yielded no improvement and is detailed in Appendix § 9.

We train on data across the structural regimes of § 1. Batches are sampled uniformly from three sources: high-confidence AFDB structures (pLDDT ≥ 90) as single ordered conformers [33], IDRome-CALVADOS simulated ensembles for disordered chains [5], and AF-CALVADOS simulations for multi-domain proteins [34], in which the whole chain is simulated with CALVADOS and folded domains are restrained towards their AlphaFold2 structures. Additional details are given in Appendix § 7.

Against the minimal base model, we evaluate the following design variants, which are chosen to explore modelling across structural regimes (further details in Appendix § 8). We train the base model with five random seeds and each variant with three, and sample all models with the same settings.

Sequence conditioning. While structure prediction for folded proteins leverages the coevolution of spatial contacts, this is less relevant for disordered regions, which rely instead on bulk physicochemical properties. To test how the required sequence-model capacity differs between regimes, we replace ESM2-650M with ESM2-8M.

Pair representation. Similarly, the dense pair representations central to folded-protein prediction may be redundant for disordered chains. Our base model has no pair bias, and we test adding a static pair representation (Pair) of the kind used by recent ensemble generators such as IDPFold2 [23]. It combines an outer sum of the ESM2 embeddings with relative sequence positions and binned distances of the noised coordinates, projected to attention bias in each layer, and is trained with an auxiliary distogram loss on the AFDB structures.

Disorder adaptation. We consider two variants designed to help the model adapt to ordered versus disordered regions. The first is a frozen per-residue probe on the ESM2 embeddings, trained to predict a pLDDT-derived disorder score and supplied as an additional conditioning signal (Disorder probe). The second is a mixture-of-experts FFN (MoE), motivated by IDPFold2 [23] (five experts under top-2 routing plus a shared always-on expert; 117M parameters, 69M active per token).

Kabsch alignment. To handle rotational invariance without explicit equivariance, models often superimpose prediction and target before computing the loss. While this is used to stabilise gradients for rigid domains, for disordered chains extended transient tails can act as lever arms that dominate the superposition, misaligning the rest of the chain and injecting rotational noise into the gradient. Our base model applies no alignment, and we test adding a loss-side Kabsch superposition.

Training source. CALVADOS is calibrated for global chain dimensions but sacrifices local backbone geometry, whereas fragment assembly inverts this trade-off (§ 1). We replace the IDRome-CALVADOS ensembles with IDRome-o [22], which is generated by fragment assembly (Fragment), and separately train with half of the disordered data drawn from each source (Fragment 50%).

Controls. Finally, we evaluate two controls that deliberately weaken the model, removing RoPE (w/o RoPE) and shrinking it to 11.0M parameters (Size S).

Results

Figure 1 shows the change relative to the base model for each variant, per cell and for the benchmark’s AUC. Table 1 compares the base model, the Kabsch variant and the two training-source variants with three published ensemble generators, BioEmu [21], IDPFold2 [23] and PepTron [22]. Table 2 reports the Flory exponent per cell. Excluding frozen sequence encoders, as in our 35.7M count, BioEmu is comparable in size (∼31\sim31M) but conditions on AlphaFold2 Evoformer representations computed from a multiple sequence alignment, which is considerably more expensive. IDPFold2 (∼330\sim330M total, ∼190\sim190M active per token) and PepTron (∼690\sim690M$, a fine-tuned ESMFold) are roughly an order of magnitude larger, so comparisons with all three are indicative rather than controlled.

Change in geometric-mean RMSE relative to the base model for SAXS and CS across model variants

Figure 1. Change in geometric-mean RMSE of each variant relative to the base model, per cell and for the benchmark’s AUC, with 95% intervals. Ensembles are scored as generated, and lower is better. The shaded band marks ±5%\pm5\%. We regard a change as negligible when its interval lies within the band, and unresolved when its interval contains zero but extends beyond it. The AUC is a single score per run, so its interval reflects variation across training seeds only.

Table 2. Flory scaling exponent ν\nu from a population fit of Rg∼NνR_g \sim N^\nu, with bootstrap SE, for the models of Table 1. The multi-domain cell is omitted, as its narrow length range makes ν\nu unreliable. The other variants are within error of the base model in these cells.

ordereddisorderedtransitional
BioEmu0.41±0.020.41 \pm0.020.69±0.020.69 \pm0.020.48±0.020.48 \pm0.02
IDPFold20.34±0.010.34 \pm0.010.56±0.020.56 \pm0.020.42±0.020.42 \pm0.02
PepTron0.38±0.010.38 \pm0.010.50±0.020.50 \pm0.020.45±0.020.45 \pm0.02
Base0.34±0.010.34 \pm0.010.54±0.010.54 \pm0.010.45±0.020.45 \pm0.02
Kabsch0.35±0.020.35 \pm0.020.58±0.010.58 \pm0.010.48±0.020.48 \pm0.02
Fragment0.33±0.020.33 \pm0.020.61±0.010.61 \pm0.010.49±0.020.49 \pm0.02
Fragment 50%0.34±0.010.34 \pm0.010.55±0.010.55 \pm0.010.47±0.020.47 \pm0.02

The base model is a reasonable reference (Table 1). It is competitive with the three baselines on SAXS, mainly through disordered proteins, but trails IDPFold2 on chemical shifts in every cell, and all three baselines on ordered proteins. Its Flory exponents follow the expected ordering, from 0.34 for ordered to 0.54 for disordered proteins (Table 2).

Table 1. Absolute scores for the three baselines, the base model, the Kabsch variant and the two training-source variants. Each entry is the geometric-mean normalised RMSE over the proteins in a cell, with ensembles scored as generated and without the default physicality filter (Appendix § 10). The base model and variants are first averaged geometrically over training seeds. AUC is the benchmark’s LOWESS-AUC. Lower is better. Cells contain 313/34/42/42 (SAXS) and 237/220/40/150 (CS) proteins.

SAXSCS
ordereddisorderedmulti-domaintransitionalAUCordereddisorderedmulti-domaintransitionalAUC
BioEmu2.063.503.372.542.571.041.481.251.221.24
IDPFold22.512.433.923.212.681.101.241.141.211.17
PepTron2.234.603.662.603.011.031.801.411.461.42
Base2.422.013.062.902.291.241.401.341.311.31
Kabsch3.043.994.674.293.681.271.391.351.331.33
Fragment2.533.263.252.912.681.211.171.201.151.17
Fragment 50%2.452.243.142.802.361.201.201.261.181.19

Reducing the size of the protein language model degrades performance on ordered and multi-domain proteins but not on disordered ones (Figure 1). With ESM2-8M in place of ESM2-650M, ordered proteins are worse on both observables (SAXS +24%, chemical shifts +12%), as are multi-domain proteins on SAXS (+26%), while disordered proteins show no detectable change, in line with the coevolutionary argument of § 3.

None of the added architectural components has a detectable effect. The disorder probe, the mixture of experts, which more than triples the parameter count, and the pair representation, which adds a quadratic pair track, stay within ±5% of the base model wherever the data can resolve it. Only the small multi-domain and transitional SAXS cells are too noisy to rule out a 5% change.

Loss-side Kabsch alignment degrades SAXS in every cell, from +26% on ordered proteins to +99% on disordered ones, and leaves chemical shifts unaffected. It is worse even on ordered proteins, where the gradient stabilisation that motivates it should help most, and the largest effect, on disordered proteins, is consistent with the lever-arm mechanism of § 3. The degradation persists after reweighting (Appendix § 12), suggesting that the ensembles contain too few conformers compatible with the measurement. In the disordered cell its Flory exponent rises above the base model’s (ν=0.58\nu= 0.58 against 0.54), close to the self-avoiding-walk value, indicating over-expansion.

Only the training source improves either observable. Replacing the CALVADOS ensembles with fragment-assembled ones improves chemical shifts by 11–16% outside the ordered cell, bringing the chemical-shift AUC level with IDPFold2’s, but worsens SAXS on disordered proteins by 62% and raises their Flory exponent to 0.61. Fragment 50% keeps most of the chemical-shift gain (−9% against −11% in AUC) at a fraction of the SAXS cost, with the SAXS AUC only slightly worse.

Even deliberately weakening the model changes little. The small model is a third the size of the base model and shows no detectable effect, while removing RoPE costs only a few percent, mostly on SAXS (AUC +8%).

Conclusion

The ablations suggest that, at this scale, the model is limited by its training data rather than its architecture. Adding architectural components has no detectable effect, and even deliberately weakening the model costs little. The choices that do matter act on particular regimes or observables, which only an evaluation resolved by regime can show. Of the two premised on a stable native fold, the size of the protein language model matters only for folded proteins, an effect the chemical-shift AUC barely registers, while alignment in the loss degrades agreement with SAXS in every regime. The source of the disordered training data trades local against global accuracy, a trade-off that an equal mixture of the two sources softens. Only the training source improves either observable, so further progress will need training sources that are accurate at both local and global scales and across structural regimes.

Limitations. Our training data come from coarse-grained simulation, fragment assembly and predicted structures, all of them approximations, and CALVADOS was calibrated against experimental data that may overlap with PeptoneDB-SAXS [34]. Our CαC\alpha model is compared with experiment only after back-mapping, whose error we do not quantify. The ordered data are also restricted to confidently predicted structures (pLDDT≥90pLDDT \ge90), which leaves partially ordered regions underrepresented. Our conclusions about architecture are drawn at a single scale, with ordered, disordered and multi-domain data in fixed equal proportion. Finally, the evaluation rests on two observables. In particular, SAXS on ordered proteins tests compactness rather than fold correctness, and with three seeds per variant, the small non-ordered SAXS cells (34–42 proteins) leave several comparisons unresolved.

Architecture and training

Table Table 3 lists the base model’s architecture and training hyperparameters.

Architecture (35.7M trainable; frozen ESM2-650M excluded)
hidden size / layers / heads / head dim384 / 12 / 6 / 64
feed-forwardSwiGLU, 83\frac{8}{3} expansion (≈1024\approx1024)
per-block norm / gatingRMSNorm on Q, K; sigmoid attention-output gate [35]
conditioningAdaLN-zero (time)
positional encodingRoPE, half of head dims rotated
coordinate scalerdivide by 1.5 nm, centred
Training
optimiserAdamW (β=0.9\beta= 0.9, 0.999; weight decay 0.01)
learning rate5×10−45 \times10^{-4}; 5k-step linear warmup, cosine decay to 1%
batch size / steps / precision128 / 100k / bf16-mixed
grad clip (norm) / EMA decay2.0 / 0.9999
timestep samplinglogit-normal (μ=0.5\mu= 0.5, s=1s = 1; median t≈0.62t \approx0.62), 10% uniform

Table 3. Base model architecture and training hyperparameters.

Sampling and back-mapping. Ensembles are generated by integrating the learned velocity from t=0t=0 to t=1t=1 with a stochastic Euler solver (diffusion coefficient λ(t)=0.5 σ(t)2\lambda(t) = 0.5\,\sigma(t)^2, with σ(t)=1−t\sigma(t) = 1-t the noise scale of the linear schedule) over 100 steps, re-centring at each step, and draw 100 conformers per protein. Cα\alpha ensembles are mapped to all-atom structures with cg2all [32].

Data

Sources and preprocessing. We train on three datasets at Cα\alpha resolution (one bead per residue), restricting to sequences of 30–510 residues:

  • AFDB [33] (ordered): Foldcomp afdb_rep_v4, mean pLDDT ≥90\ge90, a length-stratified subsample of 100,000 proteins with one conformer each.

  • IDRome-CALVADOS [5, 14, 15] (disordered): Zenodo 10251736.

  • AF-CALVADOS [34] (multi-domain): cytosolic subset, from ERDA https://sid.erda.dk/sharelink/FxaWhNrQgp.

Batch composition. Minibatches are stratified across the three datasets in equal 1:1:1 proportion.

Data splits. We cluster the pooled sequences of all training datasets with MMseqs2 at 40% identity and assign each cluster whole to one split (≈\approx90/5/5 train/validation/test). Clusters containing a PeptoneBench sequence are forced into the test split.

AF-CALVADOS Cα\alpha restoration. The source trajectories represent ordered domains by per-residue centre-of-mass beads, which lack physical Cα\alpha geometry. We restore it in each frame. Within every rigid domain (contiguous high-pLDDT residues, confirmed by PAE) we Kabsch-align the AlphaFold2 structure onto the domain’s COM beads and take its Cα\alpha coordinates, leaving disordered residues as their CALVADOS beads. A short healing pass then repairs the Cα\alpha–Cα\alpha bonds at domain–linker junctions.

Variant details

This appendix gives details of the variants not fully specified in § 3.

Disorder probe

The probe is a two-layer MLP (Desm→Desm→1D_{\mathrm{esm}} \to D_{\mathrm{esm}} \to1, SiLU nonlinearity, 0.1 dropout) that maps each ESM2 per-residue embedding to a scalar disorder logit. It is trained separately on AFDB structures to predict Di=1−pLDDTi/100D_i = 1 - \mathrm{pLDDT}_i/100 under a masked binary cross-entropy loss, with proteins sampled in equal proportion across four pLDDT bands ([0, 50), [50, 70), [70, 90), [90, 100)). During generator training the probe is frozen. Its per-residue score is Fourier-embedded (32 dimensions, zero-initialised so the pathway starts inactive) and concatenated with the ESM2 embeddings before the sequence projection.

Mixture of experts

The MoE variant replaces the feed-forward block with five routed experts under top-2 routing plus one always-on shared expert. Routed and shared outputs are combined as (shared + 2 routed)/3, and a Switch-style load-balancing auxiliary loss (coefficient 0.01) discourages routing collapse. This raises the trainable parameter count to 117M.

Static pair representation

The pair track has channel width 128 and is initialised by combining an outer sum of the ESM2 embeddings, a relative-position encoding (∣i−j∣\lvert i-j\rvert clamped at ±32\pm32, 65 bins), and a binned CαC\alpha-distance seed of the noised coordinates (32 bins over 0–4 nm, 0.125 nm wide). The initialisation is time-conditioned through an AdaLN layer, zero-initialised so that the modulation starts as the identity. It is projected to a per-head attention bias at every transformer layer. A distogram head on the pair track is trained with an auxiliary cross-entropy loss (weight 0.1, 64 bins over 0.2–2.2 nm) on AFDB examples only. The static variant applies no triangle-multiplicative updates (zero pair-update blocks).

Kabsch alignment

The Kabsch variant rotates the ground truth x1x_1 onto the model’s current estimate of the clean structure, obtained from the predicted velocity, and re-derives the velocity target as (Rx1−xt)/(1−t)(R x_1 - x_t)/(1-t), so that the implied clean structure is the rigidly rotated Rx1R x_1. The rotation is computed without gradients.

Training source

Fragment replaces IDRome-CALVADOS with IDRome-o [22] in the disordered third of each batch. Fragment 50% fills that third equally from the two sources. The ordered (AFDB) and multi-domain (AF-CALVADOS) sources are unchanged.

Size S

The Size S control reduces the hidden size to 256, depth to 8 layers, and heads to 4 (from the base 384/12/6), giving 11.0M trainable parameters.

On mini-batch optimal-transport pairing

Mini-batch optimal-transport (OT) pairing matches each data sample x1x_1 with a noise sample ϵ\epsilon within a batch to shorten the average flow path and reduce the variance of the velocity field [36]. We tested two cheap approximations to exact OT, and neither helped.

Pair-by-scale. For each data sample we compute its RMS scale, normalised by the RMSD of its parent ensemble to account for how flexible the ensemble is. We then sort the data and noise samples by scale and pair them by rank. Isotropic Gaussian noise in 3L3L dimensions, however, concentrates on a thin shell where all samples have nearly the same scale, so the sort barely changes the pairing.

Pair-by-rotation. For each pair (x1,ϵ)(x_1,\epsilon) we use SO(3) invariance to rotate x1x_1 into the frame of ϵ\epsilon by Kabsch alignment. This made training worse, for which we see three likely reasons. The gain is small to begin with. Two random configurations in 3L3L dimensions are nearly orthogonal, and a rotation can only exploit their 3×33 \times3 cross-covariance, whose singular values grow as L\sqrt{L}. The alignment therefore shortens the squared transport distance by O(L)O(\sqrt{L}) out of O(L)O(L), a fractional gain of order 1/L1/\sqrt{L}, or roughly 10% for a 100-residue chain. Training on aligned pairs also confines the velocity field to a thin, rotationally aligned manifold, from which the trajectory drifts at inference and errors compound. Translational drift is removed by re-centring at each step, but rotational drift is not. Finally, as with loss-side Kabsch alignment, extended tails dominate the superposition and bias it towards matching the tails.

Four-cell partition of PeptoneBench

PeptoneBench [22] benchmarks protein conformational-ensemble generators against experimental observables. We use its two primary datasets: PeptoneDB-CS (n=659n = 659 proteins with NMR chemical shifts from BMRB) and PeptoneDB-SAXS (n=439n = 439 proteins with SAXS profiles from SASBDB). Ensemble-averaged observables are computed via forward models and compared to experiment as a normalised RMSE accounting for forward-model and experimental uncertainty. Per-protein disorder is quantified by a mean G-score in [0,1][0, 1] (00 = ordered, 11 = disordered), derived from secondary chemical shifts for CS proteins and predicted from sequence for SAXS proteins. The summary score for each observable is the LOWESS-AUC, obtained by plotting per-protein RMSE against mean G-score, smoothing with LOWESS and integrating over the range covered by the data.

We partition each dataset into four cells by G-score, as follows. To identify multi-domain proteins, we define blocks as contiguous runs of ≥20\ge20 residues with G<0.3G < 0.3 (rigid blocks) or G>0.7G > 0.7 (flexible blocks). The cells are mutually exclusive. Multi-domain detection takes priority, and the remaining proteins are assigned by mean G-score.

  • multi-domain: ≥1\ge1 rigid block and ≥1\ge1 flexible block (regardless of mean GG).

  • ordered: not multi-domain, G<0.3G < 0.3.

  • disordered: not multi-domain, G>0.7G > 0.7.

  • transitional: not multi-domain, G∈[0.3,0.7]G \in[0.3, 0.7].

The transitional cell is introduced to sharpen the ordered and disordered cells. It may contain borderline cases, such as molten-globule-like proteins (uniform partial flexibility) or mosaic chains (order-disorder transitions at sub-block scales).

Cell counts per dataset, computed under the default thresholds (0.3, 0.7) and 20-residue block-length floor, are below.

observableordereddisorderedmulti-domaintransitional
CS (n=659n = 659)24322042154
SAXS (n=439n = 439)319344343

Table 1 (PDF p. 11).

Scoring cohort. All results use the proteins scored by every model, including the baselines, which gives 431 CS proteins and 647 CS proteins (ordered/disordered/multi-domain/transitional: 313/342/42 and 237/220/40/150). The shortfall from the full partition is due to proteins absent from the published BioEmu and IDPFold2 ensembles. We additionally exclude one SAXS protein (SASDRS5), whose PeptoneBench-distributed scattering curve is sign-inverted and cannot be fit.

Pipeline calibration. With the benchmark’s default settings, our re-implementation reproduces PeptoneBench’s published baseline summary scores. After reweighting, a small residual remains, arising from the unseeded random initialisation of the benchmark’s reweighting optimiser.

Physicality filter. PeptoneBench by default discards conformers that fail the physicality checks of BioEmu [21]. We apply no filter to any model, so that every model is scored on the same proteins and on its full ensemble. Applying the filter reverses none of the variant comparisons in § 4. It removes a larger share of conformers from our Cα ensembles than from the baselines’, so the absolute comparison with the baselines (Table 1) should be read as indicative.

Statistical comparison

We compare each variant VV with the base model BB by combining a paired comparison over proteins with a Welch comparison over training seeds. Let rmspr_{msp} denote the normalised RMSE of model mm, trained with seed ss, on protein pp. The variant has kV=3k_V = 3 seeds and the base model kB=5k_B = 5.

Effect. For each protein in a cell we take the difference of the seed-averaged log RMSEs,

dp=1kV∑slog⁡rVsp−1kB∑slog⁡rBsp,d_p = \frac{1}{k_V} \sum_s \log r_{Vsp} - \frac{1}{k_B} \sum_s \log r_{Bsp},

and average it over the nn proteins to give dˉ\bar{d}. Exponentiating dˉ\bar{d} gives the geometric mean of the per-protein ratios of § 2, which we report as a percentage change, 100(edˉ−1)%100 (e^{\bar{d}} - 1)\%.

Uncertainty. The standard error of dˉ\bar{d} is

SE2=Sd2n+SV2kV+SB2kB,\mathrm{SE}^{2} = \frac{S_d^{2}}{n} + \frac{S_V^{2}}{k_V} + \frac{S_B^{2}}{k_B},

where Sd2S_d^{2} is the variance of dpd_p across proteins, and SV2S_V^{2} and SB2S_B^{2} are the variances across seeds of each model’s log RMSE averaged over the cell. The 95%95\% interval is dˉ±tdf SE\bar{d} \pm t_{\mathrm{df}}\,\mathrm{SE}, with tdft_{\mathrm{df}} from a tt distribution with

df=SE4[(Sd2/n)2n−1+(SV2/kV)2kV−1+(SB2/kB)2kB−1]\mathrm{df} = \mathrm{SE}^{4} \left[ \frac{(S_d^{2}/n)^2}{n-1} + \frac{(S_V^{2}/k_V)^2}{k_V-1} + \frac{(S_B^{2}/k_B)^2}{k_B-1} \right]

degrees of freedom, as in Welch’s test.

Reading. Intervals that contain zero but extend beyond ±5%\pm5\% are unresolved. As a check, we split the base model’s seeds into two groups (three and two) and compared them as if they were different models, and no comparison read as better or worse.

Scores after reweighting

PeptoneBench also scores each ensemble after reweighting it towards the measurement. The conformer weights are refitted by maximum entropy, keeping an effective sample size of at least 10 conformers out of about 100. Because the reweighted score uses the measurement it is evaluated against, it indicates whether an ensemble contains conformers compatible with the measurement, not whether it generates them in the right proportions.

Reweighting lowers every score and narrows the differences between models on SAXS (Table 4). On SAXS the base model is then competitive with the three baselines in AUC and in every cell except the transitional one.

SAXSCS
ordereddisorderedmulti-domaintransitionalAUCordereddisorderedmulti-domaintransitionalAUC
BioEmu1.391.361.481.271.210.931.101.050.900.99
IDPFold21.391.151.451.341.150.960.890.910.840.91
PepTron1.571.761.841.481.470.931.331.081.081.11
Base1.361.151.321.411.141.121.181.161.041.14
Kabsch1.751.702.272.031.691.151.181.181.071.15
Fragment1.371.361.491.471.201.060.901.010.870.97
Fragment 50%1.371.121.361.401.131.060.941.050.890.99

Table 4. As Table 1, after PeptoneBench’s reweighting. Lower is better.

Most comparisons with the base model read as they do without reweighting (Figure 2). The SAXS degradation of ESM2-8M on ordered proteins disappears (+24% to 0%), consistent with suitable conformers being present but under-weighted, while its chemical-shift deficit on ordered proteins remains (+11%). Kabsch remains worse in every SAXS cell (+29% to +71%), although less so on disordered proteins (+99% to +48%), suggesting that its ensembles contain too few conformers compatible with the measurement for reweighting to recover. For Fragment, the SAXS degradation on disordered proteins shrinks (+62% to +19%) and the chemical-shift improvement grows (−16% to −24%). For Fragment 50%, the SAXS change on disordered proteins moves from +12% to −3% (both unresolved), and chemical shifts improve in every cell, including ordered proteins (−5%).

SAXS and chemical-shift changes after reweighting across protein categories and model variants

Figure 2. As Figure 1, after PeptoneBench’s reweighting. Points give the change in geometric-mean RMSE relative to the base model, per cell and for the benchmark’s AUC, with 95% intervals. Lower is better, and the shaded band marks ±5%. The AUC interval reflects variation across training seeds only.

References

  1. [1]John Jumper, Richard Evans, Alexander Pritzel, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596:583–589, 2021. doi: 10.1038/s41586-021-03819-2.DOI
  2. [2]Josh Abramson, Jonas Adler, Jack Dunger, et al. Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, 630:493–500, 2024. doi: 10.1038/s41586-024-07487-w.DOI
  3. [3]Ryan J. Emenecker, Daniel Griffith, and Alex S. Holehouse. Metapredict: a fast, accurate, and easy-to-use predictor of consensus disorder and structure. Biophysical Journal, 120(20):4312–4319, 2021. doi: 10.1016/j.bpj.2021.08.039.DOI
  4. [4]Tamas Lazar, Acadia Connor, Charles F. DeLisle, Virginia Burger, and Peter Tompa. Targeting protein disorder: the next hurdle in drug discovery. Nature Reviews Drug Discovery, 24(10):743–763, 2025. doi: 10.1038/s41573-025-01120-6.DOI
  5. [5]Giulio Tesei, Anna Ida Trolle, Nicolas Jonsson, Johannes Betz, Frederik E. Knudsen, Francesco Pesce, Kristoffer E. Johansson, and Kresten Lindorff-Larsen. Conformational ensembles of the human intrinsically disordered proteome. Nature, 626:897–904, 2024. doi: 10.1038/s41586-023-07004-5.DOI
  6. [6]Anjali Dhar, Thomas R. Sisk, and Paul Robustelli. Ensemble docking for intrinsically disordered proteins. Journal of Chemical Information and Modeling, 65(13), 2025. doi: 10.1021/acs.jcim.5c00370.DOI
  7. [7]Francesco Pesce, Anne Bremer, Giulio Tesei, Jesse B. Hopkins, Christy R. Grace, Tanja Mittag, and Kresten Lindorff-Larsen. Design of intrinsically disordered protein variants with diverse structural properties. Science Advances, 10(35):eadm9926, 2024. doi: 10.1126/sciadv.adm9926.DOI
  8. [8]Liu Hong and Jinzhi Lei. Scaling law for the radius of gyration of proteins and its dependence on hydrophobicity. Journal of Polymer Science Part B: Polymer Physics, 47(2):207–214, 2009. doi: 10.1002/polb.21634.DOI
  9. [9]Hagen Hofmann, Andrea Soranno, Alessandro Borgia, Klaus Gast, Daniel Nettels, and Benjamin Schuler. Polymer scaling laws of unfolded and intrinsically disordered proteins quantified with single-molecule spectroscopy. Proceedings of the National Academy of Sciences, 109(40):16155–16160, 2012. doi: 10.1073/pnas.1207719109.DOI
  10. [10]Rahul K. Das and Rohit V. Pappu. Conformations of intrinsically disordered proteins are influenced by linear sequence distributions of oppositely charged residues. Proceedings of the National Academy of Sciences, 110(33):13392–13397, 2013. doi: 10.1073/pnas.1304749110.DOI
  11. [11]Alexey G. Kikhney and Dmitri I. Svergun. A practical guide to small-angle X-ray scattering (SAXS) of flexible and intrinsically disordered proteins. FEBS Letters, 589(19):2570–2577, 2015. doi: 10.1016/j.febslet.2015.08.027.DOI
  12. [12]Aldo R. Camacho-Zarco, Vincent Schnapka, Serafima Guseva, Anton Abyzov, Wiktor Adamski, Sigrid Milles, Malene Ringkjøbing Jensen, Lukas Zidek, Nicola Salvi, and Martin Blackledge. NMR provides unique insight into the functional dynamics and interactions of intrinsically disordered proteins. Chemical Reviews, 122(10):9331–9356, 2022. doi: 10.1021/acs.chemrev.1c01023.
  13. [13]Sebastian Kmiecik, Dominik Gront, Michal Kolinski, Lukasz Wieteska, Aleksandra Elzbieta Dawid, and Andrzej Kolinski. Coarse-grained protein models and their applications. Chemical Reviews, 116(14):7898–7936, 2016. doi: 10.1021/acs.chemrev.6b00163.DOI
  14. [14]Giulio Tesei, Thea K. Schulze, Ramon Crehuet, and Kresten Lindorff-Larsen. Accurate model of liquid–liquid phase behavior of intrinsically disordered proteins from optimization of single-chain properties. Proceedings of the National Academy of Sciences, 118(44):e2111696118, 2021. doi: 10.1073/pnas.2111696118.DOI
  15. [15]Giulio Tesei and Kresten Lindorff-Larsen. Improved predictions of phase behaviour of intrinsically disordered proteins by tuning the interaction range. Open Research Europe, 2:94, 2023. doi: 10.12688/openreseurope.14967.2.DOI
  16. [16]Valéry Ozenne, Frédéric Bauer, Loïc Salmon, Jie-rong Huang, Malene Ringkjøbing Jensen, Stéphane Segard, Pau Bernadó, Céline Charavay, and Martin Blackledge. Flexible-meccano: a tool for the generation of explicit ensemble descriptions of intrinsically disordered proteins and their associated experimental observables. Bioinformatics, 28(11):1463–1470, 2012. doi: 10.1093/bioinformatics/bts172.DOI
  17. [17]Giacomo Janson, Gilberto Valdes-Garcia, Lim Heo, and Michael Feig. Direct generation of protein conformational ensembles via machine learning. Nature Communications, 14:774, 2023. doi: 10.1038/s41467-023-36443-x.DOI
  18. [18]Giacomo Janson and Michael Feig. Transferable deep generative modeling of intrinsically disordered protein conformations. PLOS Computational Biology, 20(5):e1012144, 2024. doi: 10.1371/journal.pcbi.1012144.DOI
  19. [19]Borna Novak, Jeffrey M. Lotthammer, Ryan J. Emenecker, and Alex S. Holehouse. Accurate predictions of disordered protein ensembles with STARLING. Nature, 652:240–250, 2026. doi: 10.1038/s41586-026-10141-2.DOI
  20. [20]Bowen Jing, Bonnie Berger, and Tommi Jaakkola. AlphaFold meets flow matching for generating protein ensembles. In Proceedings of the 41st International Conference on Machine Learning, 2024.arxiv.org/abs/2402.04845
  21. [21]Sarah Lewis, Tim Hempel, José Jiménez-Luna, Michael Gastegger, Yu Xie, Andrew Y. K. Foong, Victor Garcia Satorras, Osama Abdin, Bastiaan S. Veeling, Iryna Zaporozhets, Yaoyi Chen, Soojung Yang, Adam E. Foster, Arne Schneuing, Jigyasa Nigam, Federico Barbero, Vincent Stimper, Andrew Campbell, Jason Yim, Marten Lienen, Yu Shi, Shuxin Zheng, Hannes Schulz, Usman Munir, Robert Sordillo, Ryota Tomioka, Cecilia Clementi, and Frank Noé. Scalable emulation of protein equilibrium ensembles with generative deep learning. Science, 389(6761):eadv9817, 2025. doi: 10.1126/science.adv9817.DOI
  22. [22]Michele Invernizzi, Sandro Bottaro, Julian O. Streit, B. Trentini, N. A. Venanzi, D. Reidenbach, Y. Lee, Christian Dallago, H. S. Sirelkhatim, Bowen Jing, F. Airoldi, Kresten Lindorff-Larsen, Carlo Fisicaro, and Kamil Tamiola. Advancing protein ensemble predictions across the order–disorder continuum. bioRxiv, 2025. doi: 10.1101/2025.10.18.680935. Preprint.DOI
  23. [23]Junjie Zhu, Sören von Bülow, Hongyi Liu, Kresten Lindorff-Larsen, and Haifeng Chen. Extending conformational ensemble prediction to multidomain proteins and protein complex. bioRxiv, 2026. doi: 10.64898/2026.01.14.699584. Preprint.DOI
  24. [24]Yuyang Wang, Jiarui Lu, Navdeep Jaitly, Josh Susskind, and Miguel Angel Bautista. SimpleFold: Folding proteins is simpler than you think. In International Conference on Learning Representations, 2026.arxiv.org/abs/2509.18480
  25. [25]Jeffrey M. Lotthammer, Garrett M. Ginell, Daniel Griffith, Ryan J. Emenekcer, and Alex S. Holehouse. Direct prediction of intrinsically disordered protein conformational properties from sequence. Nature Methods, 21(3):465–476, 2024. doi: 10.1038/s41592-023-02159-5.DOI
  26. [26]Eoin Quinn, Marco Carobene, Jean Quentin, Sebastien Boyer, Miguel Arbesú, and Oliver Bent. GeoGraph: Geometric and graph-based ensemble descriptors for intrinsically disordered proteins. arXiv preprint arXiv:2510.00774, 2025. doi: 10.48550/arXiv.2510.00774.
  27. [27]Alexey G. Kikhney, Clemente R. Borges, Dmitry S. Molodenskiy, Cy M. Jeffries, and Dmitri I. Svergun. SASBDB: Towards an automatically curated and validated repository for biological scattering data. Protein Science, 29(1):66–75, 2020. doi: 10.1002/pro.3731.DOI
  28. [28]Jeffrey C. Hoch, Kumaran Baskaran, Harrison Burr, et al. Biological Magnetic Resonance Data Bank. Nucleic Acids Research, 51(D1):D368–D376, 2023. doi: 10.1093/nar/gkac1050.DOI
  29. [29]Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123–1130, 2023. doi: 10.1126/science.ade2574.DOI
  30. [30]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. doi: 10.1016/j.neucom.2023.127063.DOI
  31. [31]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. doi: 10.1109/ICCV51070.2023.00387.
  32. [32]Lim Heo and Michael Feig. One head per residue can describe all-atom protein structures. Structure, 32(1):97–111.e6, 2024. doi: 10.1016/j.str.2023.10.013.
  33. [33]Mihaly Varadi, Stephen Anyango, Mandar Deshpande, et al. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Research, 50(D1):D439–D444, 2022. doi: 10.1093/nar/gkab1061.DOI
  34. [34]Sören von Bülow, Kristoffer E. Johansson, and Kresten Lindorff-Larsen. AF-CALVADOS: AlphaFold-guided simulations of multi-domain proteins at the proteome level. Protein Science, 35:e70694, 2026. doi: 10.1002/pro.70694.DOI
  35. [35]Zihan Qiu et al. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.06708
  36. [36]Alexander Tong, Kilian Efrat, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2302.00482

Paper details

vlnc:W_gqk6cju9
v1 · latestHistory (1)

published · Fresh — added to the Hub within the last 30 days

added ·

Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

CC BY 4.0

Original source

  • Agricultural and Biological Sciences

Not recorded for this paper · View statement →

Looking for related papers…
Contents