betaFeedback

WorksMathematics

Does Your Trial Simulator Know What the Drug Does? An Intervention-Blind Test of Interventional Validity on 7,207 Registered Trials

Abstract

Patient world models are increasingly proposed as simulators of clinical trials, and recent surveys rank their capabilities on a ladder from temporal prediction to counterfactual rollouts. Position on that ladder is currently assigned by reading papers rather than by measurement. We turn the counterfactual rung into a measured quantity. From 30,119 completed phase 2/3 trials with results posted to ClinicalTrials.gov we construct 9,996 audited within-trial contrasts from 7,207 trials, and we score a simulator on how much of its accuracy depends on knowing which intervention was tested. The instrument is an intervention-blind reference: the same model, trained on the same trials, with every intervention-identifying field withheld. Interventional skill is the error the full model removes relative to that reference. On a temporal split (878 test contrasts from 655 trials) a registry-trained gradient-boosting simulator removes a fifth of the error of a trivial baseline, but its interventional skill is -0.12 (95% CI -0.21 to -0.05): withholding the intervention makes it better, and no alternative featurisation, stratum, temporal cut or same-molecule retrieval yields an interval above zero. A frontier language model reading only the registered protocol has interventional skill 0.20 [0.08, 0.31]; smaller and open-weight models sit at the limit of detection. Perturbation tests show the large model uses the intervention as such: swapping the arms reverses the sign of its prediction in 86% of contrasts. The blind reference is itself not drug-blind (a language model recovers the withheld drug class 45% of the time from the blind prompt), so interventional skill is a conservative lower bound. A reliability layer shows that ranking by predicted error lowers it but that calibrated error control does not transfer under temporal shift. Benchmark, code, prompts and per-call model outputs are released.

Introduction

A clinical trial is the one place in medicine where a counterfactual is actually observed: two exchangeable groups, one intervention, one contrast. That makes trial simulation the most falsifiable application of a patient world model, a generative or reasoning system that learns from trajectories, protocols and interventions and rolls forward what happens under a treatment it has not seen applied to this population; and it makes current evaluation practice inadequate. Simulators are scored on held-out prediction, but a model can predict a held-out arm well by learning what a responder rate in ulcerative colitis usually looks like or how large a placebo response an enriched population produces, neither of which involves the treatment. What matters for decision support is the difference between arms, and for that quantity held-out accuracy is not evidence of understanding.

The recent survey of world models in healthcare [54] makes this concern explicit. It organises the field on a four-rung ladder of temporal prediction (L1), action-conditioned prediction (L2), counterfactual rollouts (L3) and planning (L4), finds that electronic-health-record systems cluster at L1, and names interventional validity as the property the field cannot yet check: simulated futures “may be realistic but wrong about treatment effects”. Rungs there are assigned by reading the papers; our own screen of 447 clinically grounded NeurIPS 2025 workshop papers (Appendix § 13) finds 26 that model a trajectory or trial arm and none that both represents the intervention and reports, at abstract level, validation against a real trial arm.

This paper. We make interventional validity a measured quantity. The construction has three parts.

  1. A benchmark of real counterfactuals. From every completed phase 2/3 interventional trial with results posted to ClinicalTrials.gov we extract arm-level primary-endpoint results and form within-trial contrasts, treatment arm minus comparator arm, as a risk difference or a standardised mean difference. The audited release holds 9,996 contrasts from 7,207 trials, with a temporal split on the results-posting year (§ 3).

  1. An intervention-blind control. For any simulator we train a second copy on identical trials with every intervention-identifying field withheld but endpoint, condition, eligibility, design, population and comparator type intact. Interventional skill is one minus the ratio of the full model’s mean squared error to the blind model’s (§ 4), the trial-level analogue of the hypothesis-only and question-only baselines of NLI and VQA [52, 30, 29]: if a prediction survives removal of the treatment, it was not a prediction of a treatment effect.

  1. Systems and tests of the measurement. We apply the control to a registry-trained gradient-boosting simulator and to three protocol-reading language models, and test the control itself: how much of the drug the blind reference still implies, whether the models respond to the intervention as an intervention (arm swap, drug substitution), and whether exposure to posted results explains the result (§ 5).

Contributions. (i) A public, audited benchmark of 9,996 real within-trial contrasts with construction code, data card, splits and every result table regenerable by one script.1 (ii) Interventional skill, an intervention-blind measurement that operationalises the L3 rung of [54], to be reported alongside held-out accuracy as a majority-class baseline for a classifier, together with the finding that its blind reference is not drug-blind and the measurement is therefore a lower bound. (iii) The finding that registry-trained simulators have no interventional skill while a frontier language model reading the protocol has a modest, perturbation-consistent amount and smaller models sit at the limit of detection. (iv) A reliability layer showing that confidence ranking helps but calibrated error control does not survive temporal shift.

Related work

Predicting trial success is not predicting effect size. Machine learning for trials has predicted whether a trial succeeds: approval prediction [45, 63], HINT [21], SPOT [Wang et al., 2023b], Trial2Vec [Wang and Sun, 2022], LLM systems that read protocols [23, 55 Zhang et al.,] (Zheng et al. 2025; Zhang et al. 2025b) 2025a, Zheng et al., 2026] and benchmarks built on the success label [5, 24]; a 2025 scoping review finds effect-size prediction essentially absent [65]. Success confounds effect magnitude with power, endpoint choice and sponsor strategy, and none of these systems is scored against a treatment-blind reference. LLMs match HINT on success yet miss negative trials [40]; design features add signal beyond drug and indication [37]. The only prior work predicting posted arm-level results from registrations is Hu et al. [ [38], who predict which arm has more serious adverse events: safety, direction, no blind control.

Digital twins predict the control arm. Industry “digital twins” are prognostic models of the control-arm outcome [18] (Walsh et al. 2020), deployed through PROCOVA, a covariate adjustment that is unbiased because the prognostic score is estimated without the treatment [60, 16] (Vidovszky et al. 2024). External and synthetic control arms make the stronger claim, and every guidance conditions it on comparability checks against concurrent controls [66, 28, 14, 70] (Viele et al. 2014). LLM-based trial simulation [47, 39] (Wang et al. 2024; Xu et al. 2024; Yue et al. 2024) is evaluated on record fidelity, success labels or matching; none reports whether the simulated trial reproduces the posted contrast or whether the output changes when the intervention is hidden.

Causal evaluation lacks a null-by-construction reference. CATE benchmarking [36, 13, 61, 49, 50] has a matching critique: rankings are artefacts of simulation choices and causal models need interventional evaluation on empirical data [8, 46, 27, 9, 17]. Missing is a negative control [44] applied to a learned model: the hypothesis-only baselines of NLI [52, 30], question-only priors in VQA [29] and shortcut learning generally [26]. Target trial emulation [34, 35, 32] and RCT-DUPLICATE [19, 20] (Wang et al. 2023a) judge one bespoke observational study against one RCT; here a learned model is judged on thousands, against a reference that hides the treatment. Meta-research explains why design features carry skill: concealment and blinding [Wood et al., 2008], trial size [10, 51], control-group event rate [59] and a predictable placebo response [22, 69, 15] predict effect estimates, while the prior that a new treatment beats an established one is near even [12] and expert forecasts are modest [3, 4]; the intervention-specific component is what a simulator exists to add.

World models, reliability, registry. Healthcare world models [31, 54, 7, 42, 56, 48] (Waxler et al. 2025; Shmakov et al. 2025; Yang et al. 2025; Xu et al. 2025) are evaluated predictively [Wornow et al., 2023a,b]. Conformal methods [43, 2, 58, 67, 72] (Vovk et al. 2022), selective prediction [25, 6] and abstention [41, 71] supply the reliability layer. The CT.gov results database [68, 64] (Zarin et al. 2011) and its incomplete, sponsor-dependent compliance [53, 1, 11, 33, 57] (Zarin et al. 2016) bound what a benchmark built on it can claim.

A benchmark of real within-trial contrasts

Source and inclusion. We queried the ClinicalTrials.gov v2 API on 9 September 2026 for completed phase 2, 2/3 or 3 interventional studies with posted results (30,119). A study enters if its first primary outcome has a single reported class, a numeric parameter type (mean, least-squares mean, median or geometric mean; count or number of participants) and per-group analysed denominators (23,755). Arms with fewer than 20 analysed participants or a non-finite outcome are dropped, leaving 40,450 arm records from 18,077 trials (Figure 1a).

Figure 1. Held-out accuracy does not separate the systems; the blind control does. (a) Benchmark construction. (b) Skill against the global training mean on the 878 binary test contrasts with the intervention shown or withheld; GBT = registry-trained gradient boosting (five-seed mean), LM = large language model (protocol-only prompt, single run); whiskers 95% trial-bootstrap intervals; dashed line noise ceiling.

Arm records and roles. Each arm record carries the endpoint (title, description, timeframe, unit), the outcome on a common scale, the analysed nn, baseline mean age and fraction female where reported, the arm’s protocol label and description, its intervention names, types and descriptions, and its results-group title and description (join details in Appendix A; 98.7% of contrasts carry substantive intervention text). Each arm is classified as inert control (placebo, sham, no intervention), active comparator, or treatment, from the registered arm type where it can be recovered and the group text otherwise; the comparator’s nature is itself predictive, so the two control kinds are kept distinct.

Contrasts and the pairing audit. Within a trial and outcome, each treatment arm is paired with the largest inert-control arm if one exists, otherwise the largest active comparator (up to three contrasts per trial). Binary endpoints yield the risk difference Δ=pT−pC\Delta= p_T - p_C; continuous endpoints the standardised mean difference d=(yˉT−yˉC)/spooledd = (\bar{y}_T - \bar{y}_C)/s_{\mathrm{pooled}}, with ss reconstructed from the reported SD, SE or 95% interval; contrasts with ∣d∣>5|d| > 5 are removed. The submitted version (v2; 11,274 contrasts) paired arms by text alone; a model-assisted audit of 100 random contrasts found every value correctly re-derived but only 76 pairings valid, and the 24 flagged cases, reviewed by hand, showed three failure modes: a same-molecule comparator at another dose, arms from different stages of a staged or crossover design, and a placebo-labelled double-dummy experimental arm taken as the control. The released version (v2.1) adds three pairing rules (same-molecule exclusion, concurrency, investigational-arm preference; Appendix A) over re-fetched registered arm types; they drop 1,510 v2 contrasts, re-pair 375 and add 240. A fresh audit of 100 contrasts finds 96 valid pairings [0.90, 0.98]; all 125 new or re-paired test contrasts were audited individually and 8 removed by hand. The final table holds 9,996 contrasts from 7,207 trials (4,014 risk differences, 5,982 standardised differences; 7,669 inert-control, 2,327 active-comparator); v2 is kept as a sensitivity set. The repair matters: the removed contrasts were those on which the language model gained most from the intervention (its posted-record IS falls from 0.39 on v2 to 0.27 on v2.1), because in a same-molecule dose comparison the “intervention” names the dose and a posted group description often names the outcome.

Splits and strata. The primary split is temporal on the results-posting year: 7,789 contrasts posted before 2023 for training, 2,207 posted in 2023 or later for testing (binary: 3,136 train, 878 test from 655 trials; continuous: 4,653 and 1,329 from 978). Six therapeutic-area strata plus leave-one-area-out evaluation; design flags (randomised parallel-group allocation, 96% of binary test contrasts; participant-level outcome unit) define sensitivity subsets (Table 15 and Table 16).

systemISseed SD
all test: 878 contrasts, 655 trials
GBT−0.123-0.123 [−0.205-0.205, −0.048-0.048]0.017
Sonnet 5, protocol-only0.201 [0.083, 0.309]–
Sonnet 5, posted-record0.266 [0.156, 0.373]–
Haiku 4.5, protocol-only0.081 [−0.013-0.013, 0.170]–
Llama 3.1 70B, protocol-only0.075 [−0.002-0.002, 0.151]–
randomized parallel-group: 842 contrasts, 627 trials
GBT−0.132-0.132 [−0.214-0.214, −0.052-0.052]0.013
Sonnet 5, protocol-only0.216 [0.100, 0.332]–
Sonnet 5, posted-record0.275 [0.165, 0.379]–
Haiku 4.5, protocol-only0.076 [−0.025-0.025, 0.165]–
Llama 3.1 70B, protocol-only0.083 [0.001, 0.162]–
not randomized parallel-group: 36 contrasts, 28 trials
GBT0.052 [−0.264-0.264, 0.283]0.149
Sonnet 5, protocol-only−0.110-0.110 [−0.691-0.691, 0.163]–
Sonnet 5, posted-record0.131 [−0.121-0.121, 0.416]–
Haiku 4.5, protocol-only0.176 [−0.208-0.208, 0.465]–
Llama 3.1 70B, protocol-only−0.080-0.080 [−0.641-0.641, 0.180]–
participant-level unit: 845 contrasts, 634 trials
GBT−0.109-0.109 [−0.183-0.183, −0.035-0.035]0.017
Sonnet 5, protocol-only0.186 [0.068, 0.295]–
Sonnet 5, posted-record0.261 [0.147, 0.370]–
Haiku 4.5, protocol-only0.072 [−0.025-0.025, 0.156]–
Llama 3.1 70B, protocol-only0.087 [0.014, 0.164]–
participant-level unit & randomized: 821 contrasts, 614 trials
GBT−0.114-0.114 [−0.200-0.200, −0.031-0.031]0.016
Sonnet 5, protocol-only0.197 [0.079, 0.307]–
Sonnet 5, posted-record0.262 [0.145, 0.374]–
Haiku 4.5, protocol-only0.060 [−0.035-0.035, 0.148]–
Llama 3.1 70B, protocol-only0.093 [0.020, 0.168]–
participant-level unit & randomized parallel: 809 contrasts, 606 trials
GBT−0.119-0.119 [−0.199-0.199, −0.034-0.034]0.016
Sonnet 5, protocol-only0.202 [0.077, 0.316]–
Sonnet 5, posted-record0.269 [0.143, 0.383]–
Haiku 4.5, protocol-only0.066 [−0.027-0.027, 0.151]–
Llama 3.1 70B, protocol-only0.096 [0.022, 0.174]–

Table 15. Interventional skill on design subsets of the v2.1 binary temporal test split. Randomized parallel-group: allocation and intervention model from an October 2026 re-fetch of the registry records. Participant-level unit: the binary outcome is a count or percentage of participants. GBT: five-seed mean with one shared set of 1,000 trial resamples, seed SD across the five seeds; language models: single runs, 2,000 trial resamples. Haiku and Llama posted-record rows, seed-0 registry values and the noise ceiling per subset are in results/cr/design.csv and results/cr/design_noise_ceiling.csv.

systemISseed SD
all test: 1,329 contrasts, 978 trials
GBT0.021 [−0.046, 0.082]0.019
Sonnet 5, posted-record0.249 [0.069, 0.410]–
randomized parallel-group: 1,305 contrasts, 961 trials
GBT0.022 [−0.050, 0.081]0.019
Sonnet 5, posted-record0.245 [0.046, 0.404]–
not randomized parallel-group: 24 contrasts, 17 trials
GBT−0.009 [−0.337, 0.164]0.065
Sonnet 5, posted-record0.553 [0.306, 0.762]–
strict: arithmetic MEAN: 484 contrasts, 374 trials
GBT−0.030 [−0.129, 0.041]0.026
Sonnet 5, posted-record0.193 [−0.089, 0.432]–
strict & randomized: 474 contrasts, 366 trials
GBT−0.031 [−0.125, 0.035]0.029
Sonnet 5, posted-record0.186 [−0.094, 0.432]–
strict & randomized parallel: 468 contrasts, 362 trials
GBT−0.029 [−0.127, 0.042]0.028
Sonnet 5, posted-record0.185 [−0.089, 0.425]–
sensitivity: MEAN or LS MEAN & randomized: 1,113 contrasts, 789 trials
GBT0.019 [−0.052, 0.083]0.025
Sonnet 5, posted-record0.251 [0.059, 0.418]–
sensitivity: MEAN or LS MEAN & randomized parallel: 1,102 contrasts, 782 trials
GBT0.020 [−0.060, 0.085]0.025
Sonnet 5, posted-record0.249 [0.046, 0.422]–

Table 16. Interventional skill on design and valid-conversion subsets of the v2.1 continuous temporal test split (posted-record prompt for the language model). Strict: arithmetic means only; sensitivity rows also admit least-squares means. GBT: five-seed mean with one shared set of 1,000 trial resamples, seed SD across the five seeds; language models: single runs, 2,000 trial resamples. Haiku and Llama posted-record rows, seed-0 registry values and the noise ceiling per subset are in results/cr/design.csv and results/cr/design_noise_ceiling.csv.

How much is predictable at all? A reported contrast is the true contrast plus sampling noise. From the arm proportions and sizes, the mean sampling variance of a binary test contrast is 0.0047 against a total variance of 0.0499: reliability 0.906 [0.885, 0.922], the ceiling on skill against a global mean. Relative to the blind registry model’s residual error the attainable interventional skill is at most 0.87, relative to the large language model’s blind prompt 0.82; every model below these, so a null cannot be attributed to noise.

Interventional skill

Skill. For a predictor Δ^\hat{\Delta} and a reference Δ~\tilde{\Delta} on a test set, skill⁡(Δ^∣Δ~)=1−MSE⁡(Δ^)/MSE⁡(Δ~)\operatorname{skill}(\hat{\Delta}\mid\tilde{\Delta}) = 1 - \operatorname{MSE}(\hat{\Delta})/\operatorname{MSE}(\tilde{\Delta}): the fraction of the reference’s squared error the predictor removes. Against the global training mean this is the usual R2R^2-style number. Confidence intervals are percentile bootstraps over trials, because contrasts from one trial share arms; five-seed means use one shared set of resamples.

The blind reference. The reference we care about is a copy of the same model, trained on the same trials with the same procedure and the same endpoint text, condition list, eligibility criteria, design fields, population covariates, therapeutic area and comparator type, and denied every field that identifies the intervention (intervention names, types and descriptions, protocol arm labels and descriptions, results-group titles and descriptions). Call these BLIND and FULL; BLIND−\mathrm{BLIND}^{-} is the same reference without comparator type. Interventional skill is

IS=skill⁡(Δ^FULL∣Δ^BLIND)=1−MSE⁡(Δ^FULL)MSE⁡(Δ^BLIND).\mathrm{IS} = \operatorname{skill}(\hat{\Delta}_{\mathrm{FULL}} \mid\hat{\Delta}_{\mathrm{BLIND}}) = 1 - \frac{\operatorname{MSE}(\hat{\Delta}_{\mathrm{FULL}})}{\operatorname{MSE}(\hat{\Delta}_{\mathrm{BLIND}})}.

A simulator that models the effect of the treatment should have IS>0\mathrm{IS} > 0 whenever the withheld fields are its only route to the intervention; Section 5.5 shows they are not entirely, so IS is a lower bound. A model that has learned only endpoint scale, population prognosis and comparator type has IS = 0 by construction, however high its skill against the global mean; IS<0IS < 0 arises when the intervention fields add variance without information.

Why this is the right control, and what it misses. The global mean credits the model for endpoint scale; shuffling the intervention across trials destroys the legitimate intervention–endpoint co-occurrence; withholding the field with everything else intact isolates the increment that knowing the treatment buys, as the hypothesis-only baseline does in NLI [52]. It does not isolate it completely: condition, endpoint and eligibility text imply the drug class (a biologic-naïve ulcerative-colitis trial with a remission endpoint is a trial of an advanced therapy), so the blind reference already carries class-level knowledge and ISIS measures the increment of the specific drug beyond what the rest of the protocol implies. Section 5.5 quantifies this and shows that ISIS is a conservative lower bound.

Reading IS with a language model. For a prompted system the control is the same prompt with the intervention fields replaced by a withheld marker and intervention-derived tokens removed from the retained text (Appendix § 11). The prompt is built from registered protocol fields only (“protocol-only”: the condition is defined before the trial reads out); the submitted version’s prompt also carried the posted results-group labels and descriptions (“posted-record”), which can leak outcome vocabulary, and is kept as a second row. Two further confounds, pretraining exposure to the trial’s results and redaction leakage, are addressed by stratifying trials on completion, posting and publication dates relative to a model’s documented training cutoff and by auditing every blind prompt.

Experiments

Systems

Registry-trained simulator. The main system is a histogram gradient-boosting regressor (Appendix § 9) over TF-IDF text reduced to 120 SVD components, ordinal-encoded categoricals and raw numerics; FULL text is endpoint plus intervention fields, BLIND text endpoint fields only. Both are trained on the 3,136 binary (4,653 continuous) training contrasts over five seeds. We also report an intervention-only model and six alternative featurisations (§ 5.3).

Protocol-reading language models. Three language models are prompted once per contrast with the protocol fields rendered as text and asked for a point estimate and a 90% interval for the risk difference as strict JSON: a frontier model (Claude Sonnet 5, “large”), a smaller model of the same family (Claude Haiku 4.5, “small”), and an open-weight model with a documented December 2023 knowledge cutoff (Llama 3.1 70B Instruct, quantised BF16 on a dedicated deployment, temperature 0). Condition A (FULL) includes the arms’ labels, intervention names and descriptions and protocol arm descriptions; condition B (BLIND) replaces each of those lines with a withheld marker and passes the retained fields through a redaction that removes intervention-derived name tokens. Both conditions were run on all 878 binary test contrasts with all three models under protocol-only and posted-record prompts, and on the 1,329 continuous contrasts with the large model (posted-record prompt, SMD-scale header); every prompt parsed (Appendix § 11). The redaction audit finds no surviving intervention-derived token in any protocol-only blind prompt and four ordinary English words across the posted-record ones; class-level cues in eligibility text remain available to the blind condition, which makes the reference stronger and the estimate conservative (Section 5.5).

Held-out accuracy looks fine for everyone

Against the global training mean (Table 1, Figure 1b), the registry-trained FULL model removes 21% of baseline error on risk differences and 29% on standardised differences; the large language model removes 57% [0.47, 0.66], the two smaller models 35% and 40%. Read as a leaderboard, every system predicts treatment effects.

Table 1: Skill against the global training mean and interventional skill, v2.1 temporal test split. Binary: 878 contrasts / 655 trials; continuous: 1,329 / 978. GBT: five-seed mean, 95% interval from one shared set of 1,000 trial resamples; language models (large = Claude Sonnet 5, small = Claude Haiku 4.5; Llama 3.1 70B, open weights): single runs, 2,000 trial resamples. Protocol-only prompts hold registered fields only; posted-record prompts (the submitted version’s) add the posted results-group labels and descriptions; the continuous LM row is posted-record. IS = 1−MSE(FULL)/MSE(BLIND)1-\mathrm{MSE}(\mathrm{FULL})/\mathrm{MSE}(\mathrm{BLIND}); v2 values in results/cr/main_v2_sensitivity.csv.

TaskSystemskill(BLIND)skill(FULL)interventional skill
global mean–0–
GBT (5 seeds)0.297 [0.220, 0.373]0.211 [0.141, 0.281]−0.123 [−0.205, −0.048]
Risk differenceLM large, protocol-only0.462 [0.338, 0.565]0.570 [0.470, 0.655]0.201 [0.083, 0.309]
LM large, posted-record0.494 [0.382, 0.587]0.629 [0.547, 0.701]0.266 [0.156, 0.373]
LM small, protocol-only0.293 [0.170, 0.394]0.351 [0.238, 0.449]0.081 [−0.013, 0.170]
LM small, posted-record0.313 [0.177, 0.425]0.389 [0.268, 0.484]0.112 [−0.012, 0.226]
Llama 3.1 70B, protocol-only0.356 [0.270, 0.433]0.404 [0.311, 0.483]0.075 [−0.002, 0.151]
Llama 3.1 70B, posted-record0.360 [0.271, 0.436]0.417 [0.314, 0.499]0.089 [0.018, 0.157]
Standardised differenceGBT (5 seeds)0.273 [0.198, 0.348]0.288 [0.220, 0.354]0.021 [−0.046, 0.082]
LM large, posted-record0.259 [0.125, 0.382]0.443 [0.323, 0.560]0.249 [0.069, 0.410]

Table 1. Skill against the global training mean and interventional skill, v2.1 temporal test split.

The registry-trained simulator has no interventional skill

On v2.1 the registry model’s interventional skill is −0.12 [−0.20, −0.05] over five seeds (Table 1): adding every intervention field lowers held-out skill from 0.30 to 0.21. The change from the submitted version (−0.05 [−0.13, 0.02] on v2) is in the reference, not in the full model: FULL skill is 0.21 on both versions, while BLIND rises from 0.25 to 0.30 because the repaired comparator labels make comparator type more informative (its gain over BLIND) grows from 0.02 to 0.04; Table 7. The negative sign is mostly a cost of placing intervention text in the endpoint’s shared SMD space: with separate text blocks, sparse ridge or PubMedBERT embeddings the estimate lies between −0.05 and −0.01, and no architecture’s interval extends above 0.03 (Table 9). A two-stage model, which can add to the BLIND prediction only a cross-validated and shrunken correction from intervention text and so stays close to zero by construction, gives 0.00 [0.00, 0.01]. We read the row as no detectable interventional skill, and treat IS <0< 0 as a possible outcome of the instrument rather than as a separate finding.

seedBLIND{sup}−`BLINDFULLintervention onlyIS (FULL vs BLIND)
00.2610.2950.2160.030−0.113 [−0.192, −0.037]
10.2500.3010.2100.069−0.130 [−0.238, −0.033]
20.2610.3110.2090.050−0.148 [−0.255, −0.054]
30.2490.2830.2080.043−0.104 [−0.200, −0.025]
40.2620.2950.2120.078−0.118 [−0.207, −0.039]
mean0.2570.2970.2110.054−0.123 [−0.205, −0.048]

Table 7.

architectureseedsskill(BLIND)skill(FULL)IS
Headline GBT, shared SVD (120)50.300.21−0.12 [−0.20, −0.05]
Headline GBT, intervention text only5−0.05−
GBT, separate SVD blocks (120+120)50.310.28−0.04 [−0.08, −0.01]
Ridge on sparse TF-IDF, no SVD (α\alpha 1/1)10.270.24−0.04 [−0.10, 0.02]
Ridge, intervention text only (α\alpha 1)1−0.05−
Two-stage: ridge blind + ridge(intervention) on OOF residuals (α\alpha 100)10.270.270.00 [0.00, 0.01]
GBT on PubMedBERT embeddings (768+768)50.290.29−0.01 [−0.04, 0.02]
Ridge on PubMedBERT embeddings (α\alpha 0.3/1)10.280.27−0.01 [−0.05, 0.03]
Ridge, PubMedBERT intervention embedding only (α\alpha 1)1−0.07−

Table 9.

The null survives every alternative explanation we could test (Figure 2; Appendix § 12). Endpoint family: 1,329 continuous contrasts, IS = 0.02 [−0.05, 0.08]. Temporal cut: retrained splits at 2021, 2022 and 2024 give −0.04, −0.06 and −0.11. Comparator, phase and text strata: retrained within placebo-controlled (n=644n = 644), active-controlled (234), phase-3 (445) and arm-matched rich-text (697) contrasts, IS ∈ [−0.10, −0.01]; phase-2-only trials give 0.08 [−0.05, 0.17]. Arm-level task: predicting each of 4,177 test arms’ own event proportion, BLIND (0.245) again beats FULL (0.212). Seen molecules do not help: on the 286 test contrasts whose intervention name appears in training, IS is −0.23 [−0.39, −0.08], more negative than on the 592 unseen (−0.10 [−0.19, −0.01]; Table 11, Figure 4b). Feature groups (Figure 4a): over a design-only base, endpoint text adds 0.13 [0.04, 0.22] and intervention text −0.01 [−0.08, 0.06]; the intervention-only model reaches 0.05 against the training mean, so the fields carry information, none of it beyond what endpoint, population and comparator already give.

Figure 2. Interventional skill by system and stratum (v2.1). Whiskers are 95% trial-bootstrap intervals; nn = test contrasts. Grey labels: registry-trained models; blue labels: language models with protocol-only prompts (continuous row posted-record), and Llama 3.1 70B on the uncontaminated stratum (trials completed, posted and published after its training cutoff).

Table 11: Seen vs unseen interventions on v2.1 (878 binary test contrasts; five-seed means; headline models evaluated on the stratum). “Seen” = the test contrast’s intervention key appears among training contrasts. Intervals: one shared set of 1,000 trial resamples applied to all seeds. Seen molecules do not help: IS is more negative where the molecule was in training. v2 values in the released CSV.

matchingstratumcontraststrialsskill(BLIND)skill(FULL)IS
exact normalized nameseen2862300.4250.296−0.227 [−0.394, −0.077]
exact normalized nameunseen5924350.2510.180−0.095 [−0.185, −0.007]
token-Jaccard≥0.5seen3873080.3750.288−0.140 [−0.267, −0.019]
token-Jaccard≥0.5unseen4913560.2520.166−0.114 [−0.211, −0.022]

Table 11.

Figure 4. Probes (v2.1). (a) Feature-group gains over a design-only registry model (Table 10). (b) Interventional skill of the registry model for test contrasts whose intervention was or was not seen in training, two matching rules (Table 11): seen molecules do not help. (c) Large language model, protocol-only prompt, interventional skill by therapeutic area (Table 22).

Table 22: Language model (Claude Sonnet 5) by therapeutic area, v2.1 binary test split (878 contrasts / 655 trials). Protocol-only prompts (headline condition) and posted-record prompts. skill(FULL) and skill(BLIND) against the training mean; IS with 95% trial-level bootstrap intervals (2,000 resamples). v2 values in the released CSV.

areacontraststrialsskill(FULL)skill(BLIND)IS
protocol-only prompts
cardiometabolic45340.3320.2890.060 [−0.309, 0.552]
immune/inflam138880.4010.0770.352 [0.092, 0.577]
infectious2051480.6740.5580.262 [0.076, 0.430]
neuro/psych48370.3820.2010.226 [0.019, 0.440]
oncology89810.5630.4700.174 [−0.434, 0.484]
other3532670.5930.5370.119 [−0.093, 0.297]
posted-record prompts
cardiometabolic45340.4730.1980.343 [−0.253, 0.627]
immune/inflam138880.4890.1410.406 [0.194, 0.608]
infectious2051480.6900.5520.307 [0.088, 0.486]
neuro/psych48370.3000.2660.046 [−0.403, 0.407]
oncology89810.6370.4920.287 [−0.171, 0.543]
other3532670.6650.5850.193 [−0.013, 0.365]

Table 22.

Positive control: is interventional skill attainable from the registry at all? If the treatment’s effect is anywhere in public data, it is in earlier trials of the same molecule. For the 246 test contrasts (201 trials) with a training contrast of the same normalised intervention name and comparator type, predicting the blind value plus the shrunk mean residual of the matched trials gives IS = −0.04-0.04 [−0.17,0.05-0.17, 0.05]; the FULL model on the same contrasts gives −0.22-0.22 (Table 13). The registry does not contain, in a form these models can extract, the increment that knowing the drug should provide.

Table 13: Same-molecule retrieval positive control on v2.1 (binary test split, k=3k = 3 matched training contrasts of the same normalised treatment name). “residual” adds the shrunk mean out-of-fold BLIND residual of the matched contrasts to the BLIND prediction; “level” replaces the prediction by their mean delta shrunk towards the training mean. sameCtrl restricts matches to the same comparator kind. 95% trial-level bootstrap intervals (1,000 resamples). Last column: the gradient-boosting FULL model’s IS on the same subset. Variants keyed on intervention MeSH terms are in Table 14.

variantcontraststrialsretrieval ISGBT IS, same subset
name_sameCtrl_k3_residual246201−0.043 [−0.166, 0.052]−0.216 [−0.417, −0.020]
name_sameCtrl_k3_level246201−0.689 [−1.105, −0.385]−0.216 [−0.417, −0.020]
name_anyCtrl_k3_residual286230−0.088 [−0.296, 0.071]−0.255 [−0.440, −0.090]
name_anyCtrl_k3_level286230−0.747 [−1.200, −0.428]−0.255 [−0.440, −0.090]

Table 13.

Language models have interventional skill, in proportion to scale

Applied to the language models, the control gives a different answer. Withholding the intervention drops the large model’s skill from 0.570 to 0.462: IS = 0.20 [0.08,0.310.08, 0.31] with protocol-only prompts, 0.27 [0.16,0.370.16, 0.37] with the submitted version’s posted-record prompts (paired difference 0.07 [−0.03,0.17-0.03, 0.17]). Its blind skill exceeds the registry model’s full skill: it reads eligibility criteria and endpoint definitions better than TF-IDF does, so held-out accuracy ranks it first for a reason unrelated to the drug. The smaller models sit at the limit of detection: Haiku 4.5 0.08 [−0.01,0.17-0.01, 0.17], Llama 3.1 70B 0.08 [0.00,0.150.00, 0.15]. The large model’s skill reappears on continuous endpoints (0.25 [0.07,0.410.07, 0.41], posted-record), on the randomised parallel-group subset (0.22) and on contrasts with a participant-level outcome unit (0.19; Table 15); a second run of its blind condition moves IS by 0.02 [−0.02,0.05-0.02, 0.05] (Table 17).

statisticvaluennpre-registered criterion / outcome
aa prompt condition, v2.1
Claude Sonnet 5: IS, protocol-only0.201 [0.083, 0.309]878
Claude Sonnet 5: IS, posted-record0.266 [0.156, 0.373]878
Claude Sonnet 5: IS difference, posted-record minus protocol-only0.066 [−0.035, 0.166]878
Claude Haiku 4.5: IS, protocol-only0.081 [−0.013, 0.170]878
Claude Haiku 4.5: IS, posted-record0.112 [−0.012, 0.226]878
Claude Haiku 4.5: IS difference, posted-record minus protocol-only0.030 [−0.090, 0.151]878
Llama 3.1 70B: IS, protocol-only0.075 [−0.002, 0.151]878
Llama 3.1 70B: IS, posted-record0.089 [0.018, 0.157]878
Llama 3.1 70B: IS difference, posted-record minus protocol-only0.014 [−0.067, 0.088]878
bb arm swap, v2.1
sign reversal, ∣A∣>0.01|A| > 0.010.861 [0.830, 0.890]742a model that reads the arms returns about −A-A
corr(swapped, −A-A)0.947 [0.921, 0.967]878
mean(swapped + AA) (ideal 0)0.008 [0.003, 0.014]878
mean ∣A∣|A| (scale reference)0.152878
cc blind repeat, v2.1
IS, original A vs original B0.266 [0.156, 0.373]878
IS, original A vs repeat B0.283 [0.158, 0.396]878
IS difference, repeat B minus original B0.017 [−0.024, 0.052]878
corr(repeat B, original B)0.981878
fraction of identical B points0.556878
dd pre-registered negative control, v2.1
H1: mean(∣A∣−∣NC∣|A| - |NC|)0.035 [0.020, 0.049]878>0> 0, CI excludes 0: supported
H1: fraction ∣NC∣<∣A∣|NC| < |A|0.559 [0.519, 0.597]878>0.5> 0.5: supported
H1: fraction ∣NC∣=∣A∣|NC| = |A| (ties)0.151878
H1 overall–878both H1 conditions: supported
H2 (inert control): mean ∣A∣|A|0.186644
H2 (inert control): mean ∣NC∣|NC|0.113644
H2 (inert control): mean ∣A∣|A| - mean ∣NC∣|NC|0.073 [0.057, 0.089]644>0> 0: supported
H2 (inert control): slope of NC on A through origin0.590 [0.498, 0.686]644<0.5< 0.5: inconclusive
H2 overall644both H2 conditions: not supported
H3 (active comparator, A >0> 0): mean(NC - A)−0.118 [−0.178, −0.068]113<0< 0: supported
secondary: skill(NC) vs training mean0.055 [−0.164, 0.247]878
secondary: skill(A ∣| NC)0.607 [0.500, 0.694]878
secondary: skill(NC ∣| B)−0.869 [−1.335, −0.475]878
secondary: corr(NC, A)0.649878
secondary: corr(NC, B)0.583878

Table 17. Language-model revision checks. (a) Interventional skill with protocol-only and posted-record prompts on the v2.1 binary test split, with the paired difference on one shared set of 2,000 trial resamples. (b)–(d) use Claude Sonnet 5 with the original (posted-record) prompts on the 878 v2.1 binary test contrasts (v2 and 816-contrast values in results/cr/lmrevision.csv). (b) Condition A with the arms exchanged; A is the original condition-A prediction. (c) A second run of condition B. (d) Hypotheses H1–H3 exactly as pre-registered before any negative-control call (results/llm/cr/negative_control_preregistration.md): the treatment is replaced by loratadine (cardiometabolic trials) or ezetimibe (all other areas); NC is the substituted prediction; refused calls (3 on v2) are excluded. Intervals are 95% trial-level bootstrap (2,000 resamples).

Does the model use the intervention as an intervention? Two perturbations of the large model’s posted-record prompt test this (Table 17). Arm swap: exchanging the arms reverses the sign of the prediction in 86% of the 742 contrasts with a non-trivial original prediction (correlation 0.95 with the negated original); the registry model, whose bag-of-words features are nearly order-invariant, reverses 7%. Pre-registered negative control: replacing the treatment by an unrelated marketed drug (loratadine in cardiometabolic trials, ezetimibe elsewhere) shrinks the predicted effect (0.035 [0.020, 0.049]; H1 supported), more so in placebo-controlled trials (0.073; H2’s mean criterion supported, its slope criterion of < 0.5 not met at 0.59 [0.50, 0.69]), and in active-comparator trials with a predicted benefit turns it towards harm (−0.12-0.12 [−0.18,−0.07-0.18, -0.07]; H3 supported). The wrong drug is worse than no drug: skill(NC | BLIND) = −0.87-0.87 [−1.34,−0.47-1.34, -0.47].

Exposure to the results. The Claude models’ documented training cutoffs leave no usable post-cutoff stratum (no binary test contrast for Sonnet 5; 5 for Haiku 4.5), so we test exposure with the open-weight model. A contrast is uncontaminated for Llama 3.1 70B when its trial’s primary completion and first results posting both fall on or after 1 January 2024 and no linked results publication predates that date (217 contrasts, 152 trials; Table 5). In the uncontaminated stratum the posted-record interventional skill is unchanged from the full test set (0.09 [−0.01, 0.20] against 0.09 [0.02, 0.16]), so any gain here cannot come from a memorised result, while the protocol-only value is 0.01 [−0.10, 0.12], against 0.10 [0.01, 0.19] in the contaminated complement (difference −0.09 [−0.24, 0.06]). The two instruments say different things, and at this sample size the comparison can neither confirm nor exclude a drop of the size the point estimates suggest (finer splits in Appendix § 12 are equally imprecise). The posted-record result is consistent with interventional skill being attainable without exposure to the results, though its interval touches zero; we make no memorisation claim for the Claude models.

modelboundarystratumcontraststrialsIS protocol-onlyIS posted-recordIS GBT
Llama 3.1 70Btraining cutoff (2024-01-01)uncontaminated2171520.010 [−0.103, 0.123]0.089 [−0.012, 0.195]−0.058 [−0.214, 0.104]
Llama 3.1 70Btraining cutoff (2024-01-01)complement6615030.097 [0.005, 0.188]0.109 [0.006, 0.169]0.144 [−0.232, −0.052]
Haiku 4.5training cutoff (2025-08-01)uncontaminated53−0.102†0.579†−0.759†
Haiku 4.5training cutoff (2025-08-01)complement8736520.083 [−0.016, 0.171]0.107 [−0.021, 0.221]−0.122 [−0.196, −0.046]
Haiku 4.5reliable-knowledge cutoff (2025-03-01)uncontaminated52330.060 [−0.256, 0.331]0.146 [−0.144, 0.337]0.008 [−0.208, 0.152]
Haiku 4.5reliable-knowledge cutoff (2025-03-01)complement8266220.082 [−0.019, 0.174]0.110 [−0.016, 0.221]−0.132 [−0.211, −0.048]
Sonnet 5training / reliable-knowledge cutoff (2026-02-01)uncontaminated00———
Sonnet 5training / reliable-knowledge cutoff (2026-02-01)complement8786550.201 [0.083, 0.309]0.266 [0.156, 0.373]−0.123 [−0.205, −0.048]

Table 5. Interventional skill inside and outside each model’s uncontaminated set, v2.1 binary temporal test split (878 contrasts). A trial is uncontaminated for a model when its primary completion and first results posting both fall on or after the model’s boundary date and no linked results publication predates it. Language-model columns use protocol-only and posted-record prompts (2,000 trial resamples); the registry GBT column (five-seed mean, one shared set of 1,000 trial resamples) has no pretraining exposure and shows the effect of the date statute alone. Strata with fewer than 30 contrasts (†) are not interpretable and are shown without intervals. For Sonnet 5 the training and reliable-knowledge boundaries coincide. Continuous-family strata are in results/cr/contamination.csv.

How blind is blind?

The blind reference withholds the intervention’s name, not its implications. Given only the blind prompt, Haiku 4.5 guesses the withheld treatment’s drug class correctly for 45% [0.41, 0.49] of test contrasts, its mechanism for 45% and the exact agent for 21%, against 2%, 2% and 1% when the same guesses are scored against a randomly permuted contrast’s treatment (Table 18; a Haiku 4.5 judge scored the guesses). The registry model’s blind text behaves the same way: for test contrasts whose molecule occurs in training, the nearest training contrast by blind TF-IDF text is a same-molecule contrast 23% of the time (chance 0.3%), 19% in the 120-dimensional representation the model uses. Both references already encode the drug class and often the drug. IS is the increment of knowing the specific agent beyond what the protocol implies about its class, so it understates the intervention’s total contribution and is a conservative lower bound: the registry model’s zero means “nothing beyond what BLIND already carries”, and the language model’s 0.20 is an increment on a reference that has already placed the trial in a drug class. Any intervention-blind baseline should come with this audit of how much of the withheld variable the remaining inputs reconstruct.

statisticobservedchancerationn (trials)
(a) language model: treatment class guessed from the BLIND prompt (Claude Haiku 4.5, v2.1 binary)
drug class0.449 [0.408, 0.489]0.023 [0.014, 0.033]19.7878 (655)
mechanism0.454 [0.414, 0.494]0.019 [0.011, 0.029]23.4878 (655)
exact agent0.208 [0.179, 0.240]0.007 [0.002, 0.013]30.4878 (655)
(b) registry BLIND text: same-molecule training contrast retrieved (v2.1)
binary, tfidf, top-10.232 [0.172, 0.298]0.00382.2228 (188)
binary, tfidf, top-50.351 [0.279, 0.429]0.01425.1228 (188)
binary, tfidf, top-100.439 [0.366, 0.518]0.02716.0228 (188)
binary, svd120 (model input), top-10.193 [0.136, 0.254]0.00368.2228 (188)
binary, svd120 (model input), top-50.298 [0.232, 0.368]0.01421.4228 (188)
binary, svd120 (model input), top-100.386 [0.314, 0.462]0.02714.1228 (188)
continuous, tfidf, top-10.260 [0.214, 0.309]0.00375.1516 (398)
continuous, tfidf, top-50.370 [0.321, 0.426]0.01721.9516 (398)
continuous, tfidf, top-100.465 [0.413, 0.523]0.03314.1516 (398)
continuous, svd120 (model input), top-10.207 [0.165, 0.255]0.00360.0516 (398)
continuous, svd120 (model input), top-50.320 [0.270, 0.375]0.01718.9516 (398)
continuous, svd120 (model input), top-100.417 [0.363, 0.471]0.03312.6516 (398)

Table 18. How blind is BLIND? (a) Claude Haiku 4.5 reads the original condition-B prompt and guesses the withheld treatment’s drug class, mechanism and an example agent; a Haiku 4.5 judge scores the guess against the true treatment and, as a chance reference, against the treatment of a randomly permuted contrast (judge calls that failed to parse are excluded). On the 878 v2.1 binary test contrasts (v2 values in results/cr/blindness.csv); 95% trial-level bootstrap intervals (2,000 resamples). (b) For v2.1 test contrasts whose normalised treatment name occurs in the training split, the share for which a training contrast with the same molecule is among the kk nearest by BLIND text (endpoint and condition fields), as TF-IDF and as the 120-dimensional SVD the registry model uses; chance is the hypergeometric rate for kk random training contrasts; 1,000 trial resamples.

Reliability: coverage, ranking and error control

Neither system’s point prediction is reliable enough to act on; they fail differently (Figure 3). Cross-fitted conformal 90% intervals for the registry model [43, 2], constant-width and heteroscedastic, cover 0.906 [0.879, 0.933] and 0.867 in-distribution but 0.874 [0.847, 0.899] and 0.845 under the temporal shift, and 0.77 to 0.95 with each therapeutic area held out (Table 19). They are wide (mean width 0.60 in-distribution). The language models’ self-stated 90% intervals are less than half as wide (0.23) and cover 0.70 (intervention shown) and 0.69 (hidden) for the large model and 0.58 to 0.63 for the smaller ones: confidently wrong [Van Canst el al., 2019].

Figure 3. Reliability layer (v2.1). (a) Coverage of 90% cross-fitted conformal intervals for the registry model, in-distribution, under temporal shift and with each therapeutic area held out. (b) Selective prediction on the shifted test set, ranking by cross-fitted error scale (registry model) or self-stated interval width (language model, protocol-only); the marked point is the registry operating point whose threshold was chosen on calibration data for a target MAE of 0.10. (c) Coverage of the large language model’s own 90% intervals.

Table 19: Conformal coverage and mean width (nominal 90%) for the registry model, v2.1 binary contrasts, both calibration recipes, all regimes. Coverage intervals are 95% trial-level bootstrap intervals (1,000 resamples over trials), not Wilson intervals. The last column recalibrates on one randomly chosen contrast per calibration trial, which restores trial-level exchangeability (mean over 200 draws, with the 2.5th–97.5th percentiles across draws); coverage of one random contrast per test trial is in results/cr/coverage.csv.

regimerecipeintervalnncoveragewidthMAEcoverage, one contrast per calibration trial
in_distributionsplitconstant6400.889 [0.860, 0.919]0.6260.1380.880 [0.875, 0.889]
in_distributionsplitnormalized6400.845 [0.813, 0.878]0.5280.1380.858 [0.852, 0.866]
in_distributioncrossfitconstant6400.906 [0.879, 0.933]0.5980.1240.893 [0.891, 0.895]
in_distributioncrossfitnormalized6400.867 [0.834, 0.897]0.5140.1240.868 [0.866, 0.870]
temporal_2023plussplitconstant8780.888 [0.861, 0.914]0.6020.1360.875 [0.868, 0.880]
temporal_2023plussplitnormalized8780.867 [0.839, 0.896]0.5510.1360.854 [0.847, 0.862]
temporal_2023pluscrossfitconstant8780.874 [0.847, 0.899]0.5630.1300.865 [0.859, 0.867]
temporal_2023pluscrossfitnormalized8780.845 [0.814, 0.873]0.5010.1300.844 [0.841, 0.847]
loao_cardiometabolicsplitconstant2540.949 [0.917, 0.977]0.6030.0850.949 [0.949, 0.949]
loao_cardiometabolicsplitnormalized2540.921 [0.880, 0.958]0.4550.0850.919 [0.913, 0.921]
loao_cardiometaboliccrossfitconstant2540.949 [0.914, 0.976]0.5770.0850.944 [0.937, 0.945]
loao_cardiometaboliccrossfitnormalized2540.913 [0.870, 0.951]0.3880.0850.914 [0.913, 0.917]
loao_immune_inflamsplitconstant8740.857 [0.825, 0.887]0.5800.1560.846 [0.843, 0.848]
loao_immune_inflamsplitnormalized8740.810 [0.778, 0.844]0.5320.1560.806 [0.803, 0.811]
loao_immune_inflamcrossfitconstant8740.843 [0.810, 0.875]0.5630.1580.830 [0.824, 0.839]
loao_immune_inflamcrossfitnormalized8740.838 [0.806, 0.868]0.6000.1580.835 [0.827, 0.840]
loao_infectioussplitconstant7270.912 [0.885, 0.938]0.6210.1250.907 [0.906, 0.909]
loao_infectioussplitnormalized7270.856 [0.826, 0.885]0.4710.1250.859 [0.856, 0.861]
loao_infectiouscrossfitconstant7270.883 [0.852, 0.913]0.5710.1260.881 [0.880, 0.882]
loao_infectiouscrossfitnormalized7270.882 [0.853, 0.912]0.5390.1260.882 [0.879, 0.886]
loao_neuro_psychsplitconstant2530.909 [0.868, 0.947]0.6040.1300.901 [0.901, 0.901]
loao_neuro_psychsplitnormalized2530.877 [0.830, 0.923]0.5400.1300.870 [0.870, 0.877]
loao_neuro_psychcrossfitconstant2530.913 [0.869, 0.951]0.5820.1240.893 [0.889, 0.897]
loao_neuro_psychcrossfitnormalized2530.858 [0.805, 0.906]0.5350.1240.858 [0.858, 0.858]
loao_oncologysplitconstant4500.951 [0.931, 0.971]0.6090.1030.942 [0.940, 0.947]
loao_oncologysplitnormalized4500.942 [0.919, 0.962]0.5480.1030.940 [0.940, 0.942]
loao_oncologycrossfitconstant4500.949 [0.927, 0.971]0.5650.0990.942 [0.938, 0.947]
loao_oncologycrossfitnormalized4500.911 [0.884, 0.938]0.4630.0990.911 [0.909, 0.911]
loao_othersplitconstant14560.824 [0.801, 0.846]0.6180.1690.813 [0.808, 0.816]
loao_othersplitnormalized14560.764 [0.738, 0.787]0.4670.1690.762 [0.755, 0.764]
loao_othercrossfitconstant14560.804 [0.778, 0.825]0.5370.1640.793 [0.784, 0.796]
loao_othercrossfitnormalized14560.771 [0.747, 0.794]0.4700.1640.772 [0.769, 0.775]

Table 19.

Ranking by predicted error lowers it; calibrated error control does not transfer. Answering only the quarter of shifted-set contrasts the registry model’s cross-fitted error scale marks most confident cuts MAE from 0.130 to 0.083. But a threshold chosen on calibration data to hold MAE at 0.10 realises 0.106 in-distribution and 0.113 under shift, and at a target of 0.08 realises 0.095 and 0.100 (Table 20; the submitted version chose thresholds on the evaluated set, which meets any target by construction). Ranking the large language model by its stated interval width, the most confident quarter has MAE 0.039 against 0.098 overall (Table 23). Coverage without width, width without coverage, or an answered fraction at a target error can each hide the operating point that matters: the fraction answerable at an error stated before the shift.

Table 20: Answer-or-abstain operating points for the registry model, cross-fitted recipe, v2.1 binary contrasts: answer the contrasts whose predicted error scale is below a threshold. Calibration rows choose the threshold on out-of-fold calibration residuals (largest answered fraction whose calibration MAE does not exceed the target) and report the answered fractions and the realised MAE when that threshold is applied unchanged to the evaluated set. Evaluated (oracle) rows choose the threshold on the evaluated set itself, as in the submitted version, and meet the target by construction. The split recipe is in Table 21.

regimetarget MAEthreshold chosen onanswered (calibration)answered (contrasts)answered (trials)realised MAE, answeredMAE if all answered
in_distribution0.10calibration0.6780.7560.8130.1060.124
in_distribution0.10evaluated (oracle)–0.6580.7080.1000.124
in_distribution0.08calibration0.3970.4830.5380.0950.124
in_distribution0.08evaluated (oracle)–0.2890.3380.0800.124
temporal_2023plus0.10calibration0.7590.8370.8500.1130.130
temporal_2023plus0.10evaluated (oracle)–0.5740.6060.1000.130
temporal_2023plus0.08calibration0.4590.5320.5620.1000.130
temporal_2023plus0.08evaluated (oracle)–0.2560.2760.0780.130

Table 20.

Table 23: Selective prediction, language model (Claude Sonnet 5) condition A on the v2.1 binary test split (878 contrasts), ranked by the model’s stated interval width: MAE at answered fractions 0.10 / 0.25 / 0.50 / 0.75 / 1.00, protocol-only (headline) and posted-record prompts. v2 values in the released CSV.

prompt0.100.250.500.751.00
protocol-only0.0180.0390.0660.0800.098
posted-record0.0130.0380.0560.0770.091

Table 23.

Discussion

What the measurement changes. Every accuracy number in Table 1 would, under current practice, be reported as evidence that a model predicts treatment effects. The blind control shows that for the registry-trained simulator all of it, and for the large language model about two-thirds, is prediction of endpoint, population and comparator: what meta-research says the registry contains [59, 22] (Wood et al. 2008).

On the L3 rung. [54] place decision-supporting counterfactual rollouts at rung L3 and ask for interventional validity to be measured; interventional skill is one such measurement, costing a second training run or prompt. Patient-timeline world models need patient-level EHR that no public benchmark pairs with a posted trial result, but any system that emits an arm-level prediction from a protocol can be scored. By this measure a registry-trained simulator is at L1–L2; a frontier language model has a foothold on L3 (0.20 from the protocol alone, far from the attainable 0.82, overconfident); smaller models are at the limit of detection. We suggest that IS be reported alongside held-out accuracy, as classifiers report a majority-class baseline, with a blindness audit of the reference. The plausible source of the language model’s skill is pretraining exposure to pharmacology and published effects of drug classes, which the registry cannot supply: it records what was tested, not what the drug does. That suggests a route for registry-trained simulators (class-level and mechanism priors as inputs) and a caution: real interventional skill with 0.70 coverage on 90% intervals is more dangerous in a decision loop than none.

Limitations. The benchmark inherits the registry’s failure modes: results are posted for a minority of due trials, with sponsor type the strongest predictor [11, 1], so the contrasts are a selected sample; endpoints are harmonised into two effect families; covariates are trial-level means. Arm pairing is rule-based and audited by sampling (96 of 100 valid in v2.1); 7,690 result groups lack a recoverable registered arm type, and the audits were model-assisted with flagged cases reviewed by the author. Interventional skill is a necessary condition for interventional validity, not a sufficient one: positive IS could exploit a correlate of the intervention rather than its mechanism, and says nothing about individual-level counterfactuals. The exposure test is underpowered (152 uncontaminated trials; interval ±0.11), concerns an open-weight model at temperature 0 while the Claude runs used default sampling, and is not evidence about the Claude models, for which no post-cutoff stratum exists. Two tabs of the submitted version could not be reproduced from preserved code and are reported afresh or flagged provisional (Appendix § 12).

Responsible-use statement

This work evaluates whether models can predict randomised-trial results from protocols. Its negative result argues against using registry-trained simulators to replace or shrink control arms; its positive result for language models is modest (0.20), with intervals covering 0.58–0.70 at nominal 0.90 and no error control under shift. The main misuse risk is the inverse one, quoting held-out accuracy to justify a small trial or an external control arm; any downstream use should report interventional skill against an audited intervention-blind reference, coverage and width of intervals, and the answered fraction at a pre-fixed error target; no such model should inform a design decision without a prespecified comparison to concurrent controls [70].

Benchmark construction and data card

Stepcount
Completed phase 2/3 interventional studies with posted results30,119
Studies with a single-class numeric primary outcome and denominators23,755
Arm records with n≥20n \ge20 and finite outcome (binary / continuous)40,450 (18,548 / 21,902)
trials contributing arm records18,077
v2 contrasts (text-based pairing; after duplicate and ∣d∣>5\lvert d\rvert> 5 removal)11,274
v2 trials (binary / continuous)8,011 (3,225 / 4,786)
v2.1: kept / re-paired / added / relabelled / dropped v2 contrasts9,383 / 375 / 240 / 6 / 1,510
removed by hand after the audit of new and re-paired test contrasts8
v2.1 contrasts (binary / continuous)9,996 (4,014 / 5,982)
trials contributing a contrast (binary / continuous)7,207 (2,950 / 4,257)
Contrasts by comparator: inert control / active comparator7,669 / 2,327
Contrasts by split: train (posted <2023) / test (≥2023\ge2023)7,789 / 2,207
Binary contrasts: train / test (trials)3,136 / 878 (655)
Continuous contrasts: train / test (trials)4,653 / 1,329 (978)

Table 2. Cohort counts at each construction step (harvest of 9 September 2026; v2.1 pairing of 4 October 2026).

Reproducibility and availability. Harvest snapshot, benchmark tables (v2 and v2.1), data card, construction and pairing code, prompt templates, per-call language-model outputs, the pre-registration of the negative control and every result table are released at https://github.com/alicemkyn/ivbench (MIT licence for code, CC BY 4.0 for data and result tables); the camera-ready release is archived with a DOI at doi:10.5281/zenodo.23521618. The repository separates a deterministic layer from an API layer: scripts/run_camera_ready.sh regenerates every table in this paper from committed files with no network or model calls (one script per table under scripts/cr/tables/, writing tables/<label>.tex and a full-precision CSV), and --check re-fits every registry model and diffs the regenerated tables against the committed ones; the API layer (registry harvest and re-fetch, Claude and Llama calls, PubMedBERT embeddings) is documented with its archived outputs. Seeds are fixed; bootstrap intervals use 1,000 trial-level resamples (2,000 for the language-model arms), with one shared set of resamples for five-seed means. Figures are produced by scripts/cr/figures/.

Use of language models. Language models appear in this work in four roles: as the systems under evaluation (§ 5.4); as the judge of drug-class guesses (Section 5.5); as the model-assisted reviewer of arm pairings in the audits, whose flagged cases were reviewed by the author; and as the labeller of abstracts in the workshop screen (Appendix § 13), where 12 failed parses were labelled by hand. They were also used as a coding and writing assistant under the author’s direction and review. Every number in the paper is produced by the released code from the released data and archived model outputs.

Pairing rules (v2.1). The v2 pairing assigned arm roles from group text and paired the three largest treatment groups with the largest control of the preferred kind, with no check of molecule identity, design or stage. v2.1 adds, over registered arm types re-fetched for all 8,011 trials: (a) same-molecule comparator: a comparator whose normalised molecule key (components split on +, /, and, with, then; placebo and standard-care components, doses, units and schedules) matches the treatment’s is excluded and the next admissible comparator used, else the contrast is dropped (235 dropped, 180 re-paired); (b) concurrency: crossover and sequential designs are excluded, pooled and extension groups are excluded, and treatment and comparator must carry the same stage label (part, period, cohort, double-blind or open-label phase), staged groups being paired within stage rather than dropped; (c) investigational arm: where a trial has registered EXPERIMENTAL groups they form the treatment pool and the remaining control-role groups the comparator pool, with the v2 rule as fall-back. With all rules disabled the code reproduces v2 exactly. The universe is restricted to the 8,011 v2 trials; rule (c) would admit 1,516 further contrasts from 1,099 trials absent from v2, which are released separately and not audited. Arm types remain unknown for 7,690 result groups.

Audits. Each audit samples contrasts and asks whether (i) the released values can be re-derived from the current registry record and (ii) the two arms form a valid concurrent treatment-versus-comparator pair of the stated kind; (ii) is model-assisted (Claude Sonnet 5 reading the record) with the same prompt for v2 and v2.1. v2: 100 sampled, 100 re-derived, 76 valid pairings; the 24 flagged cases were reviewed by the author (19 invalid, 5 valid, two of the latter with a wrong comparator-kind label only), and all 19 are absent from v2.1. v2.1: 100 sampled (seed 1), 100 re-derived, 96 valid (Wilson 95% interval 0.90 to 0.98); the 4 failures are sub-group or cohort comparators and an interim randomisation, outside the three rules. All 125 new or re-paired test contrasts were audited individually: 115 valid; 7 pairing failures and 1 contrast whose comparator value cannot be re-derived from the current record were removed by hand, and 3 with a wrong comparator-kind label only were relabelled.

Intervention-field coverage. After the corrected join (type-stripped intervention names, intervention-table arm-label back-references, fuzzy assignment of results-group titles to arm labels at threshold 0.55), 71.9% of treatment arms are matched to a protocol arm and carry intervention names and types, 66.8% carry an intervention description, 62.5% a protocol arm description and 98.9% a results-group description; 98.7% of contrasts have at least 20 characters of intervention text across these fields. A preliminary build with a join defect (61% names, 0% descriptions) gave IS ≈ 0 for the language model too; the measurement is only as good as the completeness of the fields it withholds.

Arm-role rules. Inert control: registered arm type placebo comparator, sham comparator or no intervention, or title/description matching placebo, sham, vehicle, no treatment, no intervention, observation only, untreated, saline control (placebo-labelled EXPERIMENTAL groups excepted under rule (c)). Active comparator: arm type active comparator, or text matching active comparator/control, standard (of) care, usual care, reference drug/treatment/therapy, comparator drug/arm/therapy. Otherwise treatment.

Effect scales. Binary: proportion = count/nanalysedn_{\mathrm{analysed}} (values reported as percentages divided by 100). Continuous: d=(yˉT−yˉC)/spd = (\bar{y}_{T} - \bar{y}_{C})/s_{p} with sp2=[(nT−1)sT2+(nC−1)sC2]/(nT+nC−2)s_{p}^{2} = [(n_{T} - 1)s_{T}^{2} + (n_{C} - 1)s_{C}^{2}]/(n_{T} + n_{C} - 2); ss from SD directly, from SE as s=SEns = \mathrm{SE}\sqrt{n}, or from a 95% CI as s=n(u−l)/3.92s = \sqrt{n}(u - l)/3.92. Table 15 reports the subsets with a participant-level binary unit and with arithmetic means only.

Therapeutic areas. Regular expressions over the condition list, first match in order: oncology (cancer, carcinom, tumo(u)r, neoplas, leukemi, lymphom, myelom, melanom, sarcom, glioma); cardiometabolic (diabet, cardiac, cardiovascular, heart, hypertens, cholesterol, lipid, obesity, coronary, stroke, atrial); infectious; neurology/psychiatry; immune/inflammatory; other.

Noise ceiling. For binary contrasts Var^samp=pT(1−pT)/nT+pC(1−pC)/nC\widehat{\mathrm{Var}}_{\mathrm{samp}} = p_{T}(1-p_{T})/n_{T} + p_{C}(1-p_{C})/n_{C}; for standardised differences Var^samp=(nT+nC)/(nTnC)+d2/(2(nT+nC))\widehat{\mathrm{Var}}_{\mathrm{samp}} = (n_{T} + n_{C})/(n_{T}n_{C}) + d^{2}/(2(n_{T} + n_{C})). Reliability =1−Var^samp/Var(Δ)= 1 - \widehat{\mathrm{Var}}_{\mathrm{samp}}/\mathrm{Var}(\Delta): binary 0.906 [0.885, 0.922] on the v2.1 test split (mean sampling variance 0.0047, total 0.0499). The attainable interventional skill against a reference with error MSEref\mathrm{MSE}_{\mathrm{ref}} is 1−Var^samp/MSEref1 - \widehat{\mathrm{Var}}_{\mathrm{samp}}/\mathrm{MSE}_{\mathrm{ref}}: 0.87 against the registry BLIND model and 0.82 against the large language model’s blind prompt. No test contrast is noise-dominated.

Features, models and hyperparameters

Text fields are concatenated, vectorised with TF-IDF (1–2-grams, English stop words, min_dfdf = 3, sublinear tf, 60,000 features) fitted on the training split, and reduced with truncated SVD to 120 components (random state 0). Categoricals are ordinal-encoded with unknowns mapped to −1; numerics are passed as-is with missing values left to the tree model. The regressor is scikit-learn’s HistGradientBoostingRegressor (max_iter 400, learning rate 0.06, min_samples_leaf 20, ℓ2\ell_{2} regularisation 1.0, early stopping on a 15% validation fraction, random state = seed). Alternative architectures (Table 9): separate 120-component SVD blocks for endpoint and intervention text; ridge on the sparse TF-IDF with a chosen by trial-grouped 5-fold cross-validation; a two-stage model that fits ridge on intervention text to the BLIND model’s out-of-fold residuals; and 768-dimensional sentence embeddings (S-PubMedBert-MS-MARCO) of each text block under gradient boosting and ridge. Same-molecule retrieval: molecule keys are normalised intervention names (type prefix and dose/route/salt tokens stripped; fallback to results-group title); matches are training contrasts sharing a key and the comparator type; the prediction is Δ^BLIND+nn+krˉ\widehat{\Delta}_{\mathrm{BLIND}}+\frac{n}{n+k}\bar{r} with rˉ\bar{r} the mean out-of-fold blind residual of the matched contrasts and k=3k=3. Variants keyed on intervention MeSH terms use terms re-fetched in October 2026 and are reported as provisional (Table 14).

variantcontraststrialsretrieval ISGBT IS, same subset
primary_union_sameCtrl_k3_residual297240−0.040-0.040 [−0.153-0.153, 0.0540.054]−0.198-0.198 [−0.384-0.384, −0.022-0.022]
primary_union_sameCtrl_k3_level297240−0.614-0.614 [−0.940-0.940, −0.358-0.358]−0.198-0.198 [−0.384-0.384, −0.022-0.022]
union_anyCtrl_k3_residual330265−0.089-0.089 [−0.258-0.258, 0.0430.043]−0.202-0.202 [−0.386-0.386, −0.043-0.043]
union_anyCtrl_k3_level330265−0.686-0.686 [−1.093-1.093, −0.406-0.406]−0.202-0.202 [−0.386-0.386, −0.043-0.043]
mesh_sameCtrl_k3_residual193165−0.040-0.040 [−0.163-0.163, 0.0870.087]−0.093-0.093 [−0.349-0.349, 0.0960.096]
mesh_sameCtrl_k3_level193165−0.443-0.443 [−0.809-0.809, −0.164-0.164]−0.093-0.093 [−0.349-0.349, 0.0960.096]
mesh_anyCtrl_k3_residual213183−0.139-0.139 [−0.393-0.393, 0.0730.073]−0.119-0.119 [−0.353-0.353, 0.0850.085]
mesh_anyCtrl_k3_level213183−0.468-0.468 [−0.895-0.895, −0.168-0.168]−0.119-0.119 [−0.353-0.353, 0.0850.085]

Table 14. Same-molecule retrieval, MeSH-keyed variants (provisional). union keys = normalised treatment names ∪\cup intervention MeSH terms; mesh keys = MeSH terms only. The harvest-time MeSH terms were not preserved; the terms used here were re-fetched from ClinicalTrials.gov in October 2026, so the matched subsets differ from those of the submitted version (on v2: 342 vs 358 contrasts for the union variant) and these rows should be read as indicative. Same conventions as Table 13.

GroupFields
Endpoint textoutcome title, outcome description, outcome timeframe, condition list
Intervention texttreatment arm: intervention names, intervention descriptions, protocol arm description, results-group title and description; control arm: intervention names, arm description, group title and description
Population/design (categorical)phase, sponsor class, allocation, masking, primary purpose, therapeutic area, sex eligibility
Population (numeric)nTn_T, nCn_C, mean age, fraction female, results-posting year
Comparator typeinert control vs active comparator
Intervention typedrug / biological / device / procedure / behavioural / . . . (treatment arm)

Table 3. Feature groups. BLIND = endpoint text + population + design + comparator type; BLIND− omits comparator type; FULL = BLIND + intervention text + intervention type.

Conformal and selective prediction

Cross-fitted recipe (main text): 5-fold GroupKFold over the training period supplies out-of-fold residuals for every training contrast and an out-of-fold error-scale model σ^(x)\widehat{\sigma}(x) (gradient boosting on ∣r∣|r|); the constant-width interval is Δ^±q\widehat{\Delta}\pm q with qq the ⌈(n+1)(1−α)⌉/n\lceil(n+1)(1-\alpha)\rceil/n empirical quantile of ∣r∣|r| over the training pool, the normalised interval Δ^±qσσ^(x)\widehat{\Delta}\pm q_{\sigma}\widehat{\sigma}(x) with qσq_{\sigma} the corresponding quantile of ∣r∣/σ^|r|/\widehat{\sigma}. Split recipe (Table 19 and Table 21): the training pool is split 50/50 by trial into a fit set and a calibration set and the same quantities are computed on the calibration set; its error-scale model is fit to in-sample residuals, which are compressed towards zero, so σ^\widehat{\sigma} sits at its floor value for about three quarters of evaluated contrasts and cannot rank them, which is why the main text uses the cross-fitted recipe. Regimes: in-distribution (trial-grouped holdout within the training period, n=640n=640), temporal (≥2023\ge2023, n=878n=878), leave-one-area-out (train on the other five areas, both periods). Coverage intervals are trial-level bootstraps; a trial-level variant that calibrates on one contrast per trial is reported alongside. Selective prediction ranks test contrasts by σ^(x)\widehat{\sigma}(x) (registry model) or by the stated interval width (language model) and reports MAE among the most confident fraction. Operating points (Table 20) choose the answering threshold on calibration data as the largest answered fraction whose calibration MAE does not exceed the target and report the realised MAE when that threshold is applied unchanged to the evaluated set; the submitted version chose thresholds on the evaluated set, which meets any target by construction, and those rows are kept as “oracle”.

Table 21: Answer-or-abstain operating points, split recipe, v2.1. The split recipe fits its error-scale model to in-sample residuals of the proper training half, which are compressed towards zero; the fitted scale therefore sits at its floor value for most contrasts (0.00 to 0.75 of the evaluated set across regimes), ties cannot be broken, and where the tied floor group alone exceeds the target no calibration threshold answers any contrast (answered fraction 0). The recipe is kept for comparability with the submitted version; the cross-fitted recipe (Table 20) is the one the main text uses.

regimetarget MAEthreshold chosen onanswered (calibration)answered (contrasts)answered (trials)realised MAE, answeredMAE if all answered
in_distribution0.10calibration0.0000.0000.000–0.138
in_distribution0.10evaluated (oracle)–0.0090.0110.0470.138
in_distribution0.08calibration0.0000.0000.000–0.138
in_distribution0.08evaluated (oracle)–0.0090.0110.0470.138
temporal_2023plus0.10calibration0.3360.4680.4930.1110.136
temporal_2023plus0.10evaluated (oracle)–0.1580.1710.0980.136
temporal_2023plus0.08calibration0.1150.1610.1740.1020.136
temporal_2023plus0.08evaluated (oracle)–0.1290.1420.0770.136

Table 21.

Language-model arm

Protocol. One call per contrast, no system prompt beyond the provider default, max_tokens = 1500, batched with concurrency 8; Claude models at the provider’s default temperature, Llama 3.1 70B Instruct at temperature 0 with JSON output mode on a dedicated BF16 deployment (2 H200 GPUs, deleted after the run; a 20-prompt gate checked parse rate and prediction diversity first). Contrast identifiers c0000–c1019 are the v2 binary test contrasts sorted by (NCT id, outcome title, treatment group); v2.1 contrasts not in v2 carry c21_/k21_ identifiers. Parse: first JSON object in the response with finite point, low, high and low ≤\leq high, clipped to [−1,1][-1,1] (−5,5 for standardised differences). One retry with the suffix “IMPORTANT: output must be exactly one JSON object with keys point, low, high and nothing else.”; then recorded as failure. Every contrast in every arm received one accepted prediction per condition; the negative-control run on v2 had 3 refusals, which are excluded. Model identifiers returned: claude-sonnet-5, claude-haiku-4-5-20251001, accounts/fireworks/models/llama-v3p1-70b-instruct. Documented cutoffs used for the exposure strata: Llama 3.1, December 2023 (boundary 1 January 2024); Haiku 4.5, training data to July 2025 (reliable knowledge to February 2025); Sonnet 5, January 2026. The continuous-endpoint prompt differs only in its header (defining the standardised mean difference, with ∣d∣|d| of 0.2/0.5/0.8 given as small/medium/large).

Protocol-only and posted-record prompts. The posted-record prompt (the submitted version’s) renders, for each arm, the results-group title and description from the posted results module together with the registered intervention names and descriptions, the analysed nn and baseline summaries. The protocol-only prompt removes every field that exists only after read-out: results-group titles and descriptions, analysed arm sizes and baseline age and sex summaries; it keeps phase, conditions, eligibility criteria, comparator type, the registered intervention names and descriptions and the protocol arm descriptions, and the endpoint title, description, timeframe and unit as registered. Both share the same instruction header, redaction and parser. Outcome-vocabulary audit: 13.7% of v2 posted-record contrasts have a results-group description matching a pattern such as responded, achieved, remission, %; protocol-only prompts contain no results-group text.

Perturbation prompts. Arm swap: the posted-record condition-A prompt with the treatment and control blocks exchanged and the control-type line adjusted. Negative control: condition A with every treatment-arm intervention field replaced by a fixed unrelated marketed drug (loratadine for cardiometabolic trials, so that the substitute is never a plausible treatment, ezetimibe elsewhere) and the arm label and description redacted of the original agent; hypotheses H1–H3 and their criteria were written to the repository before any call (results/llm/cr/negative_control_preregistration.md). Blind repeat: condition B re-run unchanged. Class guess: Haiku 4.5 reads condition B and returns the most likely drug class, mechanism and an example agent as JSON; a second Haiku 4.5 call judges class, mechanism and exact-agent agreement with the true treatment, and, for the chance reference, with the treatment of a randomly permuted contrast (seed 0).

Redaction (condition B). The treatment-arm and control-arm lines are replaced by fixed strings (“the experimental intervention [withheld]”, “[withheld]”, and for the control “the control condition (inert control / active comparator)”). Every retained free-text field is passed through a redactor that removes (i) every token of length ≥ 4 from the contrast’s own intervention/arm fields that is capitalised, contains a digit, or carries a pharmacological suffix (-mab, -nib, -ciclib, -parib, -tinib, -prazole, -sartan, -floxacin, -cycline, -mycin, -cillin, -glutide, -gliptin, -glitazone, -setron, -statin, -afil, -vir, -cept, -pril, -olol, -dipine, …), excluding a list of common trial words, and (ii) any remaining suffix-matching word. Audit: 0 of 1,020 protocol-only blind prompts and 4 of 1,020 posted-record blind prompts retain a token that also occurs in that contrast’s intervention fields (arms, primary, scale, both).

Template (condition A; condition B differs only in the arm block and redaction).

You are simulating the results of a randomized clinical trial. You are given protocol
information for one trial and one binary primary endpoint (an event or responder
proportion). Predict the observed risk difference between the two arms.

Definition: risk difference = (proportion of participants in the TREATMENT arm with the
endpoint) minus (proportion in the CONTROL arm with the endpoint). Both proportions are on a
0–1 fraction scale, so the risk difference lies in [-1, 1]. A higher endpoint value is not
necessarily clinically better - predict the sign as measured, not as desirable.

Respond with STRICT JSON only, no prose, no markdown fences:
{"point": <number>, "low": <number>, "high": <number>}
where point is your best point estimate and [low, high] is a 90% central interval for the
observed risk difference.

--- TRIAL ---
Phase: {phase}
Condition(s): {conditions}
Eligibility criteria (truncated): {eligibility_text[:1500]}

Endpoint title: {outcome_title}
Endpoint description: {outcome_desc[:900]}
Endpoint timeframe: {outcome_timeframe}
Endpoint unit as reported: {unit}

Treatment arm size (n): {n_treat}
Control arm size (n): {n_ctrl}
Control type: {inert control (placebo / sham / no-treatment) |

active comparator (another active treatment)}

Population mean age (years): {age_mean}

Population fraction female: {frac_female}

TREATMENT arm label: {treat_group}

TREATMENT arm intervention name(s): {treat_intervention}

TREATMENT arm intervention description: {treat_intervention_desc[:600]}

TREATMENT arm description: {treat_arm_desc[:600]} {treat_group_desc[:600]}

CONTROL arm label: {ctrl_group}

CONTROL arm intervention name(s): {ctrl_intervention}

CONTROL arm intervention description: {ctrl_intervention_desc[:600]}

CONTROL arm description: {ctrl_arm_desc[:600]} {ctrl_group_desc[:600]}

--- END TRIAL ---

Return only the JSON object.

runfamilytest setnn trialsskill(A)skill(B)IScoverage A / B
Sonnet 5, protocolbinaryv2.18786550.5700.4620.201 [0.083, 0.309]0.70 / 0.69
Sonnet 5, postedbinaryv2.18786550.6290.4940.266 [0.156, 0.373]0.74 / 0.70
Haiku 4.5, protocolbinaryv2.18786550.3510.2930.081 [−0.013, 0.170]0.58 / 0.59
Haiku 4.5, postedbinaryv2.18786550.3890.3130.112 [−0.012, 0.226]0.57 / 0.61
Llama 3.1 70B, protocolbinaryv2.18786550.4040.3560.075 [−0.002, 0.151]0.59 / 0.63
Llama 3.1 70B, postedbinaryv2.18786550.4170.3600.089 [0.018, 0.157]0.66 / 0.69
Sonnet 5, postedcontinuousv2.11,3299780.4430.2590.249 [0.069, 0.440]0.77 / 0.67
Sonnet 5, repeat run A vs original Bbinaryv2.1 kept8166320.6290.4870.277 [0.169, 0.379]0.75 / 0.70
Sonnet 5, original A vs repeat run Bbinaryv2.18786550.6290.4820.283 [0.158, 0.396]0.74 / 0.71
Sonnet 5, neg. control (drug substituted) vs original Bbinaryv2.18786550.0550.494−0.869 [−1.335, −0.475]0.66 / 0.70
Sonnet 5, arms swapped (vs negated truth)binaryv2.18786550.623——0.75 / —

Table 4. All language-model runs. Upper block: benchmark arms on the v2.1 temporal test split; protocol-only prompts contain registered protocol fields only, posted-record prompts are the original prompts (continuous uses the posted-record prompt). Lower block: perturbation runs of Claude Sonnet 5 with the original (posted-record) prompts on the 878 v2.1 binary test contrasts (the 62 new or re-paired contrasts were run separately with identical prompts); the repeat run of condition A exists for v2 only and is reported on the 816 v2.1 contrasts it covers (‘v2.1 kept’). v2 values in results/cr/lmruns.csv. Skill against the family’s training mean; IS = FULL (A) vs BLIND (B) with 95% trial-level bootstrap intervals (2,000 resamples); coverage of the self-stated 90% interval as A / B (MAE and mean width in the CSV). The negative-control row excludes 3 refusals; the arm-swap row is scored against the negated truth. Run names are abbreviated (full names in the CSV). Class guess (Claude Haiku 4.5 from the condition-B prompt, judged by Haiku 4.5; v2.1, 878 contrasts): the drug class matches the true treatment in 0.449 vs 0.023 for a permuted treatment (876 judged), mechanism 0.454 vs 0.019, exact agent 0.208 vs 0.007.

setstratum / splitpromptISnn (trials)
v2.1allprotocol-only0.08 [0.00, 0.15]878 (655)
v2.1allposted-record0.09 [0.02, 0.16]878 (655)
v2.1uncontaminatedprotocol-only0.01 [−0.10, 0.12]217 (152)
v2.1uncontaminatedposted-record0.09 [−0.01, 0.20]217 (152)
v2.1complementprotocol-only0.10 [0.01, 0.19]661 (503)
v2.1complementposted-record0.09 [0.01, 0.17]661 (503)
v2.1uncontaminated, vocabulary unflaggedposted-record0.08 [−0.03, 0.19]196 (134)
v2.1uncontaminated, vocabulary flaggedposted-record0.16 [−0.05, 0.33]21 (18)
v2.1uncontaminated, molecule seenprotocol-only0.14 [−0.13, 0.34]58 (46)
v2.1uncontaminated, molecule unseenprotocol-only−0.01 [−0.15, 0.11]159 (107)
v2.1complement, vocabulary unflaggedposted-record0.09 [−0.01, 0.18]564 (421)
v2.1complement, vocabulary flaggedposted-record0.09 [−0.01, 0.20]97 (83)
v2.1complement, molecule seenprotocol-only0.16 [0.02, 0.29]242 (200)
v2.1complement, molecule unseenprotocol-only0.07 [−0.04, 0.19]419 (310)
v2allprotocol-only0.12 [0.04, 0.20]1020 (740)
v2allposted-record0.12 [0.04, 0.19]1020 (740)
v2uncontaminatedprotocol-only0.03 [−0.08, 0.12]245 (172)
v2uncontaminatedposted-record0.12 [0.03, 0.21]245 (172)
v2complementprotocol-only0.15 [0.05, 0.24]775 (568)
v2complementposted-record0.12 [0.01, 0.21]775 (568)
v2uncontaminated, vocabulary unflaggedposted-record0.13 [0.03, 0.22]215 (149)
v2uncontaminated, vocabulary flaggedposted-record0.05 [−0.13, 0.22]30 (24)
v2uncontaminated, molecule seenprotocol-only0.21 [−0.04, 0.35]57 (46)
v2uncontaminated, molecule unseenprotocol-only0.00 [−0.11, 0.09]188 (127)
v2complement, vocabulary unflaggedposted-record0.12 [0.00, 0.22]665 (478)
v2complement, vocabulary flaggedposted-record0.08 [−0.03, 0.18]110 (92)
v2complement, molecule seenprotocol-only0.14 [−0.02, 0.28]250 (208)
v2complement, molecule unseenprotocol-only0.15 [0.02, 0.26]525 (371)

Table 6. Exposure test with Llama 3.1 70B (knowledge cutoff December 2023). Interventional skill with 95% trial-level bootstrap intervals (2,000 resamples). Uncontaminated: primary completion and first results posting on or after 2024-01-01 and no earlier linked results publication. “Molecule seen”: a normalised treatment name occurs among treatment arms in the registry training split (a proxy for prior knowledge, not a measure of pretraining exposure). “Flagged”: results-group descriptions match an outcome-vocabulary pattern.

Full result tables

All tables are regenerated by the driver script run_camera_ready.sh; v2 (submitted-version) values are kept in the released CSVs and cited where they differ materially. Table 10 and Table 14 rest on definitions that had to be re-derived because the submitted version’s probe code was not preserved; Table 10 is reported afresh on v2.1 and Table 14 is flagged as provisional.

Table 10: Feature-group gains (gradient boosting, seed 0, v2.1 binary test split, 878 contrasts / 655 trials). “design-only” = phase, sponsor class, allocation, masking, purpose, area, sex. 95% trial-level bootstrap intervals (1,000 resamples). Group definitions follow ivbench.specs; this table is computed afresh on v2.1 and supersedes the submitted version, whose probe code was not preserved.

modelreferenceskill
design-onlytrain_mean0.069 [0.018, 0.118]
design + endpoint textdesign-only0.134 [0.035, 0.222]
design + endpoint texttrain_mean0.195 [0.110, 0.280]
design + condition textdesign-only0.063 [−0.047, 0.165]
design + condition texttrain_mean0.128 [0.026, 0.216]
design + population covariatesdesign-only0.028 [−0.043, 0.090]
design + population covariatestrain_mean0.095 [0.032, 0.159]
design + arm sizesdesign-only−0.021 [−0.112, 0.062]
design + arm sizestrain_mean0.050 [−0.039, 0.124]
design + control typedesign-only0.018 [−0.022, 0.057]
design + control typetrain_mean0.086 [0.032, 0.139]
design + intervention textdesign-only−0.009 [−0.079, 0.059]
design + intervention texttrain_mean0.061 [0.002, 0.113]
confound probe: sponsor_class+phase+yeartrain_mean0.008 [−0.023, 0.041]

Table 10.

Table 7: Gradient boosting, binary risk difference, v2.1 temporal test split (878 contrasts / 655 trials), by seed. Columns 2–5: skill against the global training mean. BLIND{sup}−` omits comparator type; BLIND includes it and is the reference for IS. The mean-row interval applies one shared set of 1,000 trial resamples to all seeds. v2 values in the released CSV.

Table 8: Robustness of the registry-model null on v2.1 (headline split: 878 contrasts / 655 trials): alternative temporal cuts (retrained), comparator/phase strata and the rich-intervention-text subset (headline models evaluated on the stratum, and retrained within it). Skill against the training mean; IS = skill(FULL | BLIND) with 95% trial-level bootstrap intervals (1,000 resamples). v2 values in the released CSV.

settingmodecontraststrialsskill(BLIND)skill(FULL)IS
headline (seed 0)headline8786550.2950.216−0.113 [−0.192, −0.037]
cut=2021retrained140610470.2730.245−0.039 [−0.093, 0.019]
cut=2022retrained11028210.2630.219−0.060 [−0.146, 0.015]
cut=2024retrained6454740.3150.240−0.109 [−0.248, 0.022]
ctrl_kind=inert_controlevaluated6444700.3000.215−0.121 [−0.209, −0.028]
ctrl_kind=inert_controlretrained6444700.2680.244−0.032 [−0.127, 0.048]
ctrl_kind=active_comparatorevaluated2341850.2690.222−0.064 [−0.177, 0.046]
ctrl_kind=active_comparatorretrained234185−0.007−0.107−0.099 [−0.229, 0.015]
phase contains PHASE3evaluated4453830.2800.233−0.066 [−0.169, 0.033]
phase contains PHASE3retrained4453830.2450.237−0.011 [−0.115, 0.085]
phase PHASE2 onlyevaluated3662340.3350.216−0.180 [−0.321, −0.032]
phase PHASE2 onlyretrained3662340.1790.2400.075 [−0.047, 0.166]
rich intervention text (arm matched, desc non-empty)evaluated6975280.2930.227−0.095 [−0.188, −0.007]
rich intervention text (arm matched, desc non-empty)retrained6975280.2410.217−0.032 [−0.126, 0.061]

Table 8.

Table 9: Registry-model architectures on the v2.1 binary temporal test split (878 contrasts, 655 trials). Skill columns are against the global training mean; BLIND includes comparator type, and IS is FULL vs BLIND of the same architecture. The headline GBT embeds endpoint and intervention text in one shared 120-dimensional SVD space; the other rows remove that sharing. Five-seed rows report the seed mean with an interval from one shared set of 1,000 trial resamples; ridge rows are deterministic, with α\alpha chosen by trial-grouped 5-fold cross-validation. The two-stage model adds a shrunken correction to the BLIND prediction and so stays near zero by construction. No row has a positive interval. v2 values in the released CSV.

Table 12: Arm-level task on v2.1 (predicting each test arm’s event proportion; 4,177 arms / 1,945 trials; pooled or total groups excluded, repaired arm roles). “blind_norole” additionally omits the arm’s role. 95% trial-level bootstrap intervals (1,000 resamples). v2 values in the released CSV.

modelreferenceskill
blindtrain_mean0.245 [0.204, 0.287]
blind_noroletrain_mean0.225 [0.187, 0.266]
fulltrain_mean0.212 [0.168, 0.255]
fullblind−0.044 [−0.090, −0.003]
fullblind_norole−0.018 [−0.062, 0.024]

Table 12.

Screen of NeurIPS 2025 workshop papers

To ground the claim that interventional validity is unmeasured in current practice we indexed the accepted papers of 19 NeurIPS 2025 workshops in health, causality, world models and reliability (1,648 papers; 1,618 abstracts retrieved), selected the 447 clinically grounded ones by workshop membership or clinical vocabulary, and labelled each abstract on seven axes (data modality, task, intervention representation, causal machinery, reliability mechanism, validation evidence, contribution type) with a language model, hand-labelling 12 that failed to parse. Of the 447, 26 model a patient trajectory or a trial arm; 8 of those represent the intervention beyond a binary flag; 5 add explicit causal machinery; 2 quantify uncertainty; 1 validates outside its training cohort; 0 validate against a real trial arm (Figure 5a). The filter is cumulative, which is why its last row is 0 while Figure 5b shows two papers in the real-trial-arm column: a tumour-dynamics survival model (AI4D3) that reports historical-trial replication but represents no intervention, and so leaves the filter at the second step, and a vasopressor-dosing reinforcement-learning paper (TS4H) scored against trial outcomes, which is a treatment-effect paper rather than a trajectory or arm simulator and never enters the filter. Neither both represents the intervention and validates against a real trial arm. Across all 447, target-trial emulation, conformal prediction and abstention each appear once; 233 validate on an internal split only. Labels are abstract-level and under-report validation detail; the index and labels are released with the code.

Cumulative filter and co-occurrence of task and validation-evidence axes

Figure 5. NeurIPS 2025 workshop screen. (a) Cumulative filter over 447 clinically grounded papers. (b) Co-occurrence of task and validation-evidence axes.

References

  1. [1]Monique L. Anderson, Karen Chiswell, Eric D. Peterson, Asba Tasneem, James Topping, and Robert M. Califf. Compliance with Results Reporting at ClinicalTrials.gov. New England Journal of Medicine, 372(11):1031–1039, 2015. doi: 10.1056/nejmsal409364.
  2. [2]Anastasios N. Angelopoulos and Stephen Bates. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv preprint arXiv:2107.07511, 2021. URL https://arxiv.org/abs/2107.07511.
  3. [3]Pavel Atanasov, Andreas Diamantaras, Amanda MacPherson, Esther Vinarov, Daniel M. Benjamin, Ian Shrier, Friedemann Paul, Ulrich Dirnagl, and Jonathan Kimmelman. Wisdom of the expert crowd prediction of response for 3 neurology randomized trials. Neurology, 95(5), 2020. doi: 10.1212/wnl.0000000000009819.
  4. [4]Daniel Benjamin, David R. Mandel, and Jonathan Kimmelman. Can cancer researchers accurately judge whether preclinical reports will reproduce? PLOS Biology, 15(6):e2002212, 2017. doi: 10.1371/journal.pbio.2002212.
  5. [5]Jintai Chen, Yaojun Hu, Mingchen Cai, Yingzhou Lu, Yue Wang, Xu Cao, Miao Lin, Hongxia Xu, Jian Wu, Xiao Cao, Jimeng Sun, Yuqiang Li, Lucas Glass, Xekin Huang, Marinka Zitnik, and Tianfan Fu. TrialBench: Multi-Modal AI-Ready Datasets for Clinical Trial Prediction. Scientific Data, 12(1):1564, 2025. doi: 10.1038/s41597-025-05680-8.DOI
  6. [6]Tianyi Chen, Yingzhou Lu, Nan Hao, Yuanyuan Zhang, Capucine Van Rechem, Jintai Chen, and Tianfan Fu. Uncertainty Quantification on Clinical Trial Outcome Prediction. arXiv preprint arXiv:2401.03482, 2024. URL https://arxiv.org/abs/2401.03482.
  7. [7]Zhaoyan Chen, Zhongxiu Cong, Zhuanfeng Jin, Wanshu Fan, Dongsheng Zhou, Qi Ai, Haifan Gong, Congyu Liao, Xiaofeng Liu, and Cong Wang. Medical world models in healthcare: foundations, applications, and challenges for trustworthy clinical translation. arXiv preprint arXiv:2607.25242, 2026. URL https://arxiv.org/abs/2607.25242.
  8. [8]Alicia Curth and Mihaela van der Schaar. Doing Great at Estimating CATE? On the Neglected Assumptions in Benchmark Comparisons of Treatment Effect Estimators. arXiv preprint arXiv:2107.13346, 2021. URL https://arxiv.org/abs/2107.13346.
  9. [9]Alicia Curth, Richard W. Peck, Eoin McKinney, James Weatherall, and Mihaela van der Schaar. Using Machine Learning to Individualize Treatment Effect Estimation: Challenges and Opportunities. Clinical Pharmacology & Therapeutics, 115(4):710–719, 2024. doi: 10.1002/cpt.3159.
  10. [10]A. Dechartres, L. Trinquart, I. Boutron, and P. Ravaud. Influence of trial sample size on treatment effect estimates: meta-epidemiological study. BMJ, 346(apr24 14):f2304, 2013. doi: 10.1136/bmj. f2304.
  11. [11]Nicholas J DeVito, Seb Bacon, and Ben Goldacre. Compliance with legal requirement to report clinical trial results on ClinicalTrials.gov: a cohort study. The Lancet, 395(10221):361–369, 2020. doi: 10.1016/S0140-6736(19)33220-9.
  12. [12]Benjamin Djulbegovic, Ambuj Kumar, Paul P Glasziou, Rafael Perera, Tea Reljic, Louise Dent, James Raftery, Marit Johansen, Gian Luca Di Tanna, Branko Miladinovic, Heloisa P Soares, Gunn E Vist, and Iain Chalmers. New treatments compared to established treatments in randomized trials. Cochrane Database of Systematic Reviews, 2012(10), 2012. doi: 10.1002/14651858.mr000024.pub3.
  13. [13]Vincent Dorie, Jennifer Hill, Uri Shalit, Marc Scott, and Dan Cervone. Automated versus Do-It-Yourself Methods for Causal Inference: Lessons Learned from a Data Analysis Competition. Statistical Science, 34(1), 2019. doi: 10.1214/18-sts667.
  14. [14]Severin Elvatun, Daan Knoors, Simon Brant, Christian Jonasson, and Jan F. Nygård. Synthetic data as external control arms in scarce single-arm clinical trials. PLOS Digital Health, 4(1):e0000581, 2025. doi: 10.1371/journal.pdig.0000581.
  15. [15]Paul Enck, Ulrike Bingel, Manfred Schedlowski, and Winfried Rief. The placebo response in medicine: minimize, maximize or personalize? Nature Reviews Drug Discovery, 12(3):191–204, 2013. doi: 10.1038/nrd3923.DOI
  16. [16]European Medicines Agency. Qualification Opinion for Prognostic Covariate Adjustment (PROCOVA). https://www.ema.europa.eu/en/documents/scientific-guideline/qualification-opinion-prognostic-covariate-adjustment-procovatm_en.pdf, 2022. Regulatory guidance document; URL only, no DOI.
  17. [17]Stefan Feuerriegel, Dennis Frauen, Valentyn Melnychuk, Jonas Schweistal, Konstantin Hess, Alicia Curth, Stefan Bauer, Niki Kilbertus, Isaac S. Kohane, and Mihaela van der Schaar. Causal machine learning for predicting treatment outcomes. Nature Medicine, 30(4):958–968, 2024. doi: 10.1038/s41591-024-02902-1.DOI
  18. [18]Charles K. Fisher, Aaron M. Smith, Jonathan R. Walsh, Adam J. Simon, Chris Edgar, Clifford R. Jack, David Holtzman, David Russell, Derek Hill, Donald Grosset, Fred Wood, Hugo Vanderschelle, John Morris, Kaj Blennow, Ken Marek, Leslie M Shaw, Marilyn Albert, Michael Weiner, Nick Fox, Paul Aisen, Patricia E. Cole, Ronald Petersen, Todd Sherer, and Wayne Kubick. Machine learning for comprehensive forecasting of Alzheimer’s Disease progression. Scientific Reports, 9(1):13622, 2019. doi: 10.1038/s41598-019-49656-2.DOI
  19. [19]Jessica M. Franklin, Ajinkya Pawar, David Martin, Robert J. Glynn, Mark Levenson, Robert Temple, and Sebastian Schneeweiss. Nondamized Real-World Evidence to Support Regulatory Decision Making: Process for a Randomized Trial Replication Project. Clinical Pharmacology & Therapeutics, 107(4):817–826, 2020. doi: 10.1002/cpt.1633.DOI
  20. [20]Jessica M. Franklin, Elisabetta Patorno, Rishi J. Desai, Robert J. Glynn, David Martin, Kenneth Quinto, Ajinkya Pawar, Lily G. Bessette, Hemin Lee, Elizabeth M. Garry, Nileesa Gautam, and Sebastian Schneeweiss. Emulating Randomized Clinical Trials With Nonrandomized Real-World Evidence Studies. Circulation, 143(10):1002–1013, 2021. doi: 10.1161/circulationaha.120.051718.DOI
  21. [21]Tianfan Fu, Kexin Huang, Cao Xiao, Lucas M. Glass, and Jimeng Sun. HINT: Hierarchical interaction network for clinical-trial-outcome predictions. Patterns, 3(4):100445, 2022. doi: 10.1016/j.patter.2022.100445.DOI
  22. [22]Toshi A Furukawa, Andrea Cipriani, Lauren Z Atkinson, Stefan Leucht, Yusuke Ogawa, Nozomi Takeshima, Yu Hayasaka, Anna Chaimani, and Georgia Salanti. Placebo response rates in antidepressant trials: a systematic review of published and unpublished double-blind randomised controlled studies. The Lancet Psychiatry, 3(11):1059–1066, 2016. doi: 10.1016/S2215-0366(16)30307-8.DOI
  23. [23]Chufan Gao, Tianfan Fu, and Jimeng Sun. Language Interaction Network for Clinical Trial Approval Estimation. arXiv preprint arXiv:2405.06662, 2024. URL https://arxiv.org/abs/2405.06662.
  24. [24]Chufan Gao, Jathurshan Pradeepkumar, Trisha Das, Shivashankar Thati, and Jimeng Sun. A large-scale database for clinical trial outcomes and features. Nature Health, 1(4):428–436, 2026. doi: 10.1038/s44360-026-00081-6.DOI
  25. [25]Yonatan Geifman and Ran El-Yaniv. Selective Classification for Deep Neural Networks. arXiv preprint arXiv:1705.08500, 2017. URL https://arxiv.org/abs/1705.08500.
  26. [26]Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. doi: 10.1038/s42256-020-00257-z.DOI
  27. [27]Amanda Gentzel, Dan Garant, and David Jensen. The Case for Evaluating Causal Models Using Interventional Measures and Empirical Data. arXiv preprint arXiv:1910.05387, 2019. URL https://arxiv.org/abs/1910.05387.
  28. [28]Mercedeh Ghadessi, Rui Tang, Joey Zhou, Rong Liu, Chenkun Wang, Kiichiro Toyoizumi, Chaoqun Mei, Lixia Zhang, C. Q. Deng, and Robert A. Beckman. A roadmap to using historical controls in clinical trials – by Drug Information Association Adaptive Design Scientific Working Group (DIA-ADSWG). Orphanet Journal of Rare Diseases, 15(1):69, 2020. doi: 10.1186/s13023-020-1332-x.DOI
  29. [29]Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. arXiv preprint arXiv:1612.00837, 2016. URL https://arxiv.org/abs/1612.00837.
  30. [30]Suchin Gururangan, Swabha Swajaypita, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation Artifacts in Natural Language Inference Data. arXiv preprint arXiv:1803.02324, 2018. URL https://arxiv.org/abs/1803.02324.
  31. [31]David Ha and Jürgen Schmidhuber. World Models. arXiv preprint arXiv:1803.10122, 2018. URL https://arxiv.org/abs/1803.10122.
  32. [32]Harrison J. Hansford, Aidan G. Cashin, Matthew D. Jones, Sonja A. Swanson, Nazrul Islam, Susan R. G. Douglas, Rodrigo R. N. Rizzo, Jack J. Devonshire, Sam A. Williams, Issa J. Dahabreh, Barbra A. Dickerman, Matthias Egger, Xabier Garcia-Albeniz, Robert M. Golub, Sara Lodi, Margarita Moreno-Betancur, Sallie-Anne Pearson, Sebastian Schneeweiss, Jonathan A. C. Sterne, Melissa K. Sharp, Elizabeth A. Stuart, Miguel A. Hernán, Hopin Lee, and James H. McAuley. Reporting of Observational Studies Explicitly Aiming to Emulate Randomized Trials. JAMA Network Open, 6(9):e2336023, 2023. doi: 10.1001/jamanetworkopen.2023.36023.DOI
  33. [33]Daniel M. Hartung, Deborah A. Zarin, Jeanne-Marie Guise, Marian McDonagh, Robin Paynter, and Mark Helfand. Reporting Discrepancies Between the ClinicalTrials.gov Results Database and Peer-Reviewed Publications. Annals of Internal Medicine, 160(7):477–483, 2014. doi: 10.7326/m13-0480.DOI
  34. [34]Miguel A. Hernán and James M. Robins. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available. American Journal of Epidemiology, 183(8):758–764, 2016. doi: 10.1093/aje/kwv254.DOI
  35. [35]Miguel A. Hernán, Wei Wang, and David E. Leaf. Target Trial Emulation. JAMA, 328(24):2446, 2022. doi: 10.1001/jama.2022.21383.DOI
  36. [36]Jennifer L. Hill. Bayesian Nonparametric Modeling for Causal Inference. Journal of Computational and Graphical Statistics, 20(1):217–240, 2011. doi: 10.1198/jcgs.2010.08162.DOI
  37. [37]Robert Howard, Beau Bush, Joseph Edelmann, and Diego Sarachaga. Evaluating the impact of clinical trial design features on probability-of-success prediction. Scientific Reports, 2026. doi: 10.1038/s41598-026-64883-0.DOI
  38. [38]Qixuan Hu, Xumou Zhang, Jinman Kim, Florence Bourgeois, and Adam G. Dunn. A novel language model for predicting serious adverse event results in clinical trials from their prospective registrations. Journal of Biomedical Informatics, 181:105088, 2026. doi: 10.1016/j.jbi.2026.105088.DOI
  39. [39]Qiao Jin, Zifeng Wang, Charalampos S. Floudas, Fangyuan Chen, Changlin Gong, Dara Bracken-Clarke, Elisabetta Xue, Yifan Yang, Jimeng Sun, and Zhiyong Lu. Matching patients to clinical trials with large language models. Nature Communications, 15(1):9074, 2024a. doi: 10.1038/s41467-024-53081-z.DOI
  40. [40]Shuyi Jin, Lu Chen, Hongru Ding, Meijie Wang, and Lun Yu. Can artificial intelligence predict clinical trial outcomes? arXiv preprint arXiv:2411.17595, 2024b. URL https://arxiv.org/abs/2411.17595.
  41. [41]Benjamin Kompa, Jasper Snoek, and Andrew L. Beam. Second opinion needed: communicating uncertainty in medical machine learning. npj Digital Medicine, 4(1):4, 2021. doi: 10.1038/s41746-020-00367-3.DOI
  42. [42]Zeljko Kraljevic, Dan Bean, Anthony Shek, Rebecca Bendayan, Harry Hemingway, Joshua Au Yeung, Alexander Deng, Alfred Balston, Jack Ross, Esther Idowu, James T Teo, and Richard J B Dobson. Foresight—a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study. The Lancet Digital Health, 6(4):e281–e290, 2024. doi: 10.1016/S2589-7500(24)00025-6.DOI
  43. [43]Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution-Free Predictive Inference for Regression. Journal of the American Statistical Association, 113 (523):1094–1111, 2018. doi: 10.1080/01621459.2017.1307116.DOI
  44. [44]Marc Lipsitch, Eric Tchetgen Tchetgen, and Ted Cohen. Negative Controls. Epidemiology, 21(3): 383–388, 2010. doi: 10.1097/ede.0b013e3181d61deeb.DOI
  45. [45]Andrew W. Lo, Kien Wei Siah, and Chi Heem Wong. Machine Learning with Statistical Imputation for Predicting Drug Approval. Harvard Data Science Review, 2019. doi: 10.1162/99608f92.5c5f0525.DOI
  46. [46]Divyat Mahajan, Joannis Mitliagkas, Brady Neal, and Vasilis Syrgkanis. Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation. arXiv preprint arXiv:2211.01939, 2022. URL https://arxiv.org/abs/2211.01939.
  47. [47]Nikita Makarov, Maria Bordukova, Lena Voith von Voithenberg, Estrella Pivel-Villanueva, Sabrina Mielke, Jonathan Wickes, Hanchen Wang, Mingyu Derek Ma, Keunwoo Choi, Kyunghyun Cho, Stephen Ra, Raul Rodriguez-Esteban, Fabian Schmich, and Michael Menden. TwinWeaver: An LLM-Based Foundation Model Framework for Pan-Cancer Digital Twins. arXiv preprint arXiv:2601.20906, 2026. URL https://arxiv.org/abs/2601.20906.
  48. [48]Linjie Mu, Zhongzhen Huang, Yannian Gu, Shengqian Qin, Shaoting Zhang, and Xiaofan Zhang. EHRWorld: A Patient-Centric Medical World Model for Long-Horizon Clinical Trajectories. arXiv preprint arXiv:2602.03569, 2026. URL https://arxiv.org/abs/2602.03569.
  49. [49]Brady Neal, Chin-Wei Huang, and Sunand Raghupathi. RealCause: Realistic Causal Inference Benchmarking. arXiv preprint arXiv:2011.15007, 2020. URL https://arxiv.org/abs/2011.15007.
  50. [50]Harsh Parikh, Carlos Varjao, Louise Xu, and Eric Tchetgen Tchetgen. Validating Causal Inference Methods. arXiv preprint arXiv:2202.04208, 2022. URL https://arxiv.org/abs/2202.04208.
  51. [51]Tiago V. Pereira, Ralph I. Horwitz, and John P. A. Ioannidis. Empirical Evaluation of Very Large Treatment Effects of Medical Interventions. JAMA, 308(16):1676, 2012. doi: 10.1001/jama.2012.13444.DOI
  52. [52]Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis Only Baselines in Natural Language Inference. arXiv preprint arXiv:1805.01042, 2018. URL https://arxiv.org/abs/1805.01042.
  53. [53]A. P. Prayle, M. N. Hurley, and A. R. Smyth. Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ, 344(jan03 1):d7373, 2012. doi: 10.1136/bmj.d7373.DOI
  54. [54]Mohammad Areeb Qazi, Maryam Nadeem, and Mohammad Yaqub. Beyond Generative AI: World Models for Clinical Prediction, Counterfactuals, and Planning. arXiv preprint arXiv:2511.16333, 2025. URL https://arxiv.org/abs/2511.16333.
  55. [55]Michael Reinisch, Jianfeng He, Chenxi Liao, Sauleh Ahmad Siddiqui, and Bei Xiao. CTP-LLM: Clinical Trial Phase Transition Prediction Using Large Language Models. arXiv preprint arXiv:2408.10995, 2024. URL https://arxiv.org/abs/2408.10995.
  56. [56]Pawel Renc, Yugang Jia, Anthony E. Samir, Jaroslaw Was, Quanzheng Li, David W. Bates, and Arkadiusz Sitek. Zero shot health trajectory prediction using transformer. npj Digital Medicine, 7 (1):256, 2024. doi: 10.1038/s41746-024-01235-0.DOI
  57. [57]Carolina Riveros, Agnes Dechartres, Elodie Perrodeau, Romana Haneef, Isabelle Boutron, and Philippe Ravaud. Timing and Completeness of Trial Results Posted at ClinicalTrials.gov and Published in Journals. PLoS Medicine, 10(12):e1001566, 2013. doi: 10.1371/journal.pmed.1001566.DOI
  58. [58]Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. Conformalized Quantile Regression. arXiv preprint arXiv:1905.03222, 2019. URL https://arxiv.org/abs/1905.03222.
  59. [59]Christopher H. Schmid, Joseph Lau, Martin W. McIntosh, and Joseph C. Cappelleri. An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Statistics in Medicine, 17(17):1923–1942, 1998. doi: 10.1002/(sici)1097-0258(19980915)17:17<1923::aid-sim874>3.0.co;2-6.doi.org/10.1002/(sici)1097-0258(19980915)17:17<1923::aid-sim874>3.0.co;2-6
  60. [60]Alejandro Schuler, David Walsh, Diana Hall, Jon Walsh, and Charles Fisher. Increasing the efficiency of randomized trial estimates via linear adjustment for a prognostic score. The International Journal of Biostatistics, 18(2):329–356, 2022. doi: 10.1515/ijb-2021-0072.DOI
  61. [61]Yishai Shimoni, Chen Yanover, Ehud Karavani, and Yaara Goldschmnidt. Benchmarking Framework for Performance-Evaluation of Causal Inference Analysis. arXiv preprint arXiv:1802.05046, 2018. URL https://arxiv.org/abs/1802.05046.
  62. [63]Kien Wei Siah, Nicholas W. Kelley, Steffen Ballerstedt, Björn Holzhauer, Tianmeng Lyu, David Mettler, Sophie Sun, Simon Wandel, Yang Zhong, Bin Zhou, Shifeng Pan, Yingyao Zhou, and Andrew W. Lo. Predicting drug approvals: The Novartis data science and artificial intelligence challenge. Patterns, 2(8):100312, 2021. doi: 10.1016/j.patter.2021.100312.DOI
  63. [64]Asba Tasneem, Laura Aberle, Hari Ananth, Swati Chakraborty, Karen Chiswell, Brian J. McCourt, and Ricardo Pietribon. The Database for Aggregate Analysis of ClinicalTrials.gov (AACT) and Subsequent Regrouping by Clinical Specialty. PLoS ONE, 7(3):e33677, 2012. doi: 10.1371/journal.pone.0033677.DOI
  64. [65]Douglas Teodoro, Nona Naderi, Anthony Yazdani, Boya Zhang, and Alban Bornet. A scoping review of artificial intelligence applications in clinical trial risk assessment. npj Digital Medicine, 8(1):486, 2025. doi: 10.1038/s41746-025-01886-7.DOI
  65. [66]Kristian Thorlund, Louis Dron, Jay JH Park, and Edward J Mills. Synthetic and External Controls in Clinical Trials – A Primer for Researchers. Clinical Epidemiology, 12:457–467, 2020. doi: 10.2147/clep.s242097.DOI
  66. [67]Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candes, and Aaditya Ramdas. Conformal Prediction Under Covariate Shift. arXiv preprint arXiv:1904.06019, 2019. URL https://arxiv.org/abs/1904.06019.
  67. [68]Tony Tse, Kevin M Fain, and Deborah A Zarin. How to avoid common problems when using ClinicalTrials.gov in research: 10 issues to consider. BMJ, page k1452, 2018. doi: 10.1136/bmj.k1452.DOI
  68. [69]Alexander H. Tuttle, Sarasa Tohyama, Tim Ramsay, Jonathan Kimmelman, Petra Schweinhardt, Gary J. Bennett, and Jeffrey S. Mogil. Increasing placebo responses over time in U.S. clinical trials of neuropathic pain. Pain, 156(12):2616–2626, 2015. doi: 10.1097/j.pain.0000000000000333.DOI
  69. [70]U.S. Food and Drug Administration. Considerations for the Design and Conduct of Externally Controlled Trials for Drug and Biological Products: Draft Guidance for Industry. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-design-and-conduct-externally-controlled-trials-drug-and-biological-products, 2023. Regulatory guidance document; URL only, no DOI.
  70. [71]Ben Van Calster, David J. McLernon, Maarten van Smeden, Laure Wynants, and Ewout W. Steyerberg. Calibration: the Achilles heel of predictive analytics. BMC Medicine, 17(1):230, 2019. doi: 10.1186/s12916-019-1466-7.DOI
  71. [72]Janette Vazquez and Julio C. Facelli. Conformal Prediction in Clinical Medical Sciences. Journal of Healthcare Informatics Research, 6(3):241–252, 2022. doi: 10.1007/s41666-021-00113-8.DOI

Paper details

vlnc:W_y3jda5v9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents