WorksComputer Science
The Value of System 1 Models in Medicine
Abstract
Many clinical decisions are a choice among a few fixed options, taken quickly and in large numbers. Such decisions also need an uncertainty estimate to know when to defer to a clinician. Generative language models answer slowly in free text and state their confidence poorly. In contrast, System 1 models answer zero-shot with a probability for each answer. However, their performance across clinical tasks is unknown. Four System 1 models, two proprietary and two with open weights, were evaluated zero-shot on 86 public tasks from 14 clinical domains in 20 languages, spanning examination questions and medical imaging. Tasks were assigned to six capability tiers before analysis. The primary outcome was the area under the receiver-operating-characteristic curve (AUROC). Secondary outcomes included calibration, language effects and comparisons with published results and generative models. Across 179,111 cases, the mean AUROC of the best model per task was 0.94 for medical knowledge, 0.93 for text understanding, 0.83 for clinical judgement from text, 0.89 for image perception, 0.84 for 2D image findings and 0.69 for volumetric findings. Imaging tasks scored 0.09 (95% CI, 0.06 to 0.11) lower than text tasks (P < 0.001). The best model matched or exceeded the best published result on 13 of 26 text tasks and on 8 of 35 imaging tasks. Triage nurses outperformed the best model (accuracy, 0.85 vs. 0.56). One decision took 0.34 to 0.44 seconds and 24 to 81 joules on open-weight models run locally. Accuracy on identical questions fell by up to 39 percentage points from English to Yoruba. Generative models of the same family were 11 to 76 times slower, with 3 to 5 percentage points higher balanced accuracy. System 1 models decide quickly, cheaply and with usable confidence when the answer is stated in a text or recalled from medical knowledge, but not when it must be judged or read from an image.
Introduction
Many clinical decisions represent choices among a few fixed options that must be made quickly and in bulk. A triage nurse assigns one of five acuity levels within minutes of arrival, a radiologist decides whether a CT pulmonary angiogram demonstrates embolism, and a coder selects one procedure code for each of thousands of cases. A model that makes such decisions must also state how certain it is, so that confident answers can be acted on and uncertain cases passed to a clinician. [1, 2, 3] Generative large language models perform well on a series of medical tasks, [4, 5] but they write each answer token by token, often following a reasoning trace, which takes several seconds per decision and uses far more energy than a task-specific model. [6] The confidence they state in text is poorly calibrated and barely separates correct from incorrect answers. [7, 8] On the contrary, System 1 models receive a state, such as a report, an image or both, together with one or more typed questions, and return a probability for every answer option in a single forward pass without generating text. Their name refers to dual-process theory, which separates fast, intuitive System 1 judgement from slow, deliberate System 2 reasoning, the mode that generative models resemble when they reason in text. [9, 10]
The open question is which clinical decisions such a model can be trusted with. The answer to a clinical case can be stated in a report, recalled from medical knowledge, judged from several findings, or seen in an image. Of 519 studies of large language models in health care, 44.5% tested medical knowledge such as licensing-1 examination questions, and 5% used real patient-care data. [11] Where other tasks were tested, performance fell from more than 86% on licensing-examination questions [12] to lower accuracy on real clinical cases and conversations. [13, 14] On radiologic images, GPT-4V identified the imaging modality in all 515 images but detected abnormalities in 0% to 90%, depending on the finding, [15] and multimodal models scored lower on volumetric than on two-dimensional images. [16] If performance depends on where the answer lies, a single benchmark score misrepresents a model, a construct-validity problem known from general AI benchmarks. [17]
To evaluate the value of System 1 models in medicine, we tested four models on 86 tasks from 14 clinical domains, with the tasks ordered in six capability tiers from medical knowledge to volumetric findings. We measured which clinical decisions the models can take on, whether their probabilities are calibrated, how they perform outside English, and what their speed costs in time, money, energy and accuracy, against a generative model of the same family.
Methods
Study Design and Tasks. This cross-sectional evaluation used public, de-identified datasets under their respective licences. Candidate tasks came from a taxonomy of 87 clinical decision tasks in 18 domains and from further public datasets with a fixed answer format (Supplementary Appendix, Section S1). A task was included when its data and reference labels were public and its licence allowed transfer to a model hosted on a commercial API. Eighty-six tasks from 14 clinical domains met these criteria. Table S1 lists every task with the publication of its dataset. Eight of them are language versions of another task and were analysed only for language effects, which left 78 primary tasks (Figure 1). Each task used the official test split or a seeded random sample of up to 1,000 cases, in some tasks enriched for rare classes, and volumetric tasks used 81 to 436 studies. The tasks comprised 179,111 cases, with the reference standard released by the dataset authors.

Figure 1. Study Design. Panel A shows how a System 1 model answers. Text, one or more images, or both enters the model with typed questions, and the model returns a probability for each option in one forward pass. The generative model of the same family served as comparator on 32 tasks. Panel B shows the 86 tasks by capability tier and input type. Hatched tiles are language versions of another task and were analysed for language effects only.
Prior to result summarisation, each task was assigned to one of six capability tiers. Medical knowledge covered examination questions and terminology, text understanding covered information stated in a report, abstract or sentence, and clinical judgement from text covered decisions that integrate patient information, such as triage acuity or variant class. The three imaging tiers included image perception (acquisition, anatomy and image quality), two-dimensional (2D) image findings, and volumetric findings on CT or MRI.
System 1 Models. Four System 1 models were evaluated through the OpenRouter decision interface, each in the latest version available in October 2026. Clef (Cloudflare, 27 billion parameters) and Clef-flash (Cloudflare, 9.5 billion parameters) have open weights, and GPT-6 Luna Decisions (OpenAI) and Jev 1.14 (TypeSafe AI) are proprietary. A request held a state of text and up to four images, and one or more typed questions with a yes-or-no answer, a choice of one of options, or an ordered score. The model returned a probability for every option. All questions of a case were sent in one request, zero-shot, with a one-sentence description of the input. Volumetric studies were read as axial slices at a fixed spacing, with one request per slice. The study-level probability was the maximum over slices for yes-or-no questions and the mean over slices for other questions. On four 2D imaging tasks, Jev 1.14 received higher-resolution versions of the same images since its interface accepts larger images. Every request was logged as sent.
Generative Comparison. Clef and Clef-flash were compared with the generative backbones, Qwen3.8-27B [18] and Qwen3.5-9B, [19] and GPT-6 Luna Decisions with the generative GPT-6 Luna. The comparison covered 32 text and 2D imaging tasks. For each task, 100 randomly selected cases were sent to the generative model with the same state and questions as the System 1 model had received, including an instruction to choose one option per question, and an output schema restricted to the answer options. Default reasoning settings were kept. An answer that failed or could not be parsed counted as wrong.
Local Runs and Energy. Clef and Clef-flash were additionally run locally from the released checkpoints, on a single NVIDIA RTX 5090 GPU. Local Clef’s answers were compared with the hosted versions, which confirmed that the two agreed before the local runs were utilised. Energy was measured on a sub-sample of 42 tasks and it followed the AI Research Label of the CEDARS framework for reporting the environmental footprint of clinical AI. [20] The measurements come from the GPU driver’s cumulative energy counter through NVML (power usage effectiveness taken as 1.0 for a single workstation GPU). CodeCarbon (v3.3.1) ran alongside to convert energy to COe at the local grid carbon intensity (Greece, 337 g COe per kWh). Energy is reported per decision and summarised as the median and IQR over tasks. Latency was measured per request. The providers of the hosted models (GPT-6 Luna Decisions, Jev 1.14) publish no per-request energy figures for them.
Reference Results. For every task, published results on the same dataset were collected from the original dataset or challenge publication, from official leaderboards, and from a search of PubMed and the web for later work up to October 2026. Every value was checked against its source and classified as a task-specific model, a generative language model, a rule-based classifier, or clinicians and expert annotators. A published result was compared only when it used the same label set and metric on the matching questions. The highest such result per task is reported as the best published result.
Outcomes
The primary outcome was the area under the receiver-operating-characteristic curve (AUROC), computed per question as a macro average of one-versus-rest curves for more than two options, and averaged over the questions of a task. Secondary outcomes were the balanced accuracy of the most probable option, expressed relative to chance as for classes so that 0 is chance and 1 is perfect, the top-label expected calibration error (ECE), [1, 21] selective accuracy after withholding the least confident answers, [22] cost per 1,000 requests at provider prices, latency per request and energy per decision.
Statistical Analysis. Any question that a model refused or failed was scored as wrong, with the label prevalence as its probability. Task-level 95% confidence intervals (CIs) were estimated using 1,000 bootstrap resamples of cases. Task estimates were pooled within each tier and model using DerSimonian–Laird random-effects models. Two contrasts were specified in advance, imaging versus text tiers and a linear trend across the three imaging tiers, and both were estimated by random-effects meta-regression. AUROC was also summarised by question type. The accuracy of each language version on identical items was compared with English by paired bootstrap and the exact McNemar test. System 1 and generative models were compared by the mean task-level difference in balanced accuracy, with 95% CIs from a task bootstrap. Secondary analyses were not adjusted for multiplicity. Analyses used Python 3.13, pandas 3.0, NumPy 2.5, SciPy 1.18 and scikit-learn 1.9. Code and all model outputs are available in the project repository at https://github.com/twin-x-eu/system1-models-in-medicine.
Results
Tasks and Models. The 86 tasks covered 20 languages and yielded 2.4 million model answers across 179,111 cases (Figure 1). GPT-6 Luna Decisions refused 4,008 case–question pairs (1.3%), Clef and Clef-flash each failed on one pair, and Jev 1.14 answered every question. Refused and failed questions were scored as wrong. Jev 1.14 was the best model on 45 of 78 tasks. Results per task are given in Table S2.
| No. | Task | Clef | Clef-flash | GPT-6 Luna Decisions | Jev 1.14 |
| 001 | MedQA (USMLE) | 0.97 (0.96–0.97) | 0.92 (0.90–0.93) | 0.98 (0.97–0.98) | 0.99 (0.98–0.99) |
| 002 | MedMCQA (India) | 0.89 (0.89–0.90) | 0.87 (0.86–0.88) | 0.92 (0.92–0.93) | 0.93 (0.93–0.94) |
| 003 | HEAD-QA (English) | 0.96 (0.95–0.96) | 0.95 (0.94–0.95) | 0.96 (0.96–0.97) | 0.97 (0.96–0.97) |
| 004 | HEAD-QA (Spanish) | 0.96 (0.95–0.97) | 0.95 (0.94–0.95) | 0.97 (0.96–0.97) | 0.97 (0.97–0.98) |
| 005 | MMLU medical | 0.99 (0.98–0.99) | 0.99 (0.98–0.99) | 0.98 (0.98–0.99) | 0.99 (0.98–0.99) |
| 006 | MMLU-Pro health | 0.97 (0.96–0.97) | 0.96 (0.95–0.97) | 0.96 (0.94–0.96) | 0.97 (0.96–0.98) |
| 007 | MedXpertQA text | 0.72 (0.71–0.73) | 0.63 (0.62–0.64) | 0.76 (0.74–0.77) | 0.77 (0.76–0.78) |
| 008 | Medbullets | 0.92 (0.90–0.94) | 0.83 (0.80–0.86) | 0.94 (0.92–0.96) | 0.94 (0.93–0.96) |
| 009 | AfriMed-QA | 0.94 (0.94–0.95) | 0.92 (0.91–0.93) | 0.95 (0.95–0.96) | 0.95 (0.95–0.96) |
| 010 | MedExpQA | 0.96 (0.95–0.97) | 0.92 (0.91–0.94) | 0.96 (0.94–0.97) | 0.96 (0.95–0.98) |
| 011 | KorMedMCQA | 0.96 (0.96–0.97) | 0.93 (0.92–0.93) | 0.99 (0.99–0.99) | 0.98 (0.98–0.98) |
| 012 | CMExam (China) | 0.97 (0.97–0.97) | 0.95 (0.95–0.96) | 0.97 (0.97–0.98) | 0.98 (0.98–0.99) |
| 013 | MedQA Mainland | 0.98 (0.98–0.98) | 0.96 (0.96–0.97) | 0.98 (0.97–0.98) | 0.98 (0.98–0.99) |
| 014 | MedQA Taiwan | 0.97 (0.96–0.98) | 0.95 (0.95–0.96) | 0.98 (0.98–0.99) | 0.98 (0.98–0.99) |
| 015 | FrenchMedMCQA | 0.99 (0.98–0.99) | 0.97 (0.95–0.98) | 0.99 (0.98–1.00) | 0.99 (0.99–1.00) |
| 016 | MMLU medical (Arabic) | 0.95 (0.94–0.96) | 0.93 (0.92–0.94) | 0.98 (0.97–0.98) | 0.98 (0.98–0.99) |
| 017 | MMLU medical (Hindi) | 0.92 (0.91–0.93) | 0.84 (0.83–0.86) | 0.98 (0.97–0.98) | 0.98 (0.97–0.99) |
| 018 | MMLU medical (Japanese) | 0.97 (0.96–0.97) | 0.96 (0.95–0.97) | 0.98 (0.97–0.98) | 0.98 (0.98–0.99) |
| 019 | MMLU medical (Swahili) | 0.90 (0.89–0.92) | 0.85 (0.83–0.86) | 0.96 (0.95–0.96) | 0.96 (0.95–0.97) |
| 020 | MMLU medical (Yoruba) | 0.85 (0.83–0.86) | 0.79 (0.77–0.81) | 0.91 (0.90–0.93) | 0.93 (0.92–0.94) |
| 021 | MMLU medical (German) | 0.98 (0.97–0.98) | 0.97 (0.96–0.98) | 0.98 (0.97–0.99) | 0.99 (0.98–0.99) |
| 022 | ATC drug class | 0.53 (0.52–0.54) | 0.61 (0.60–0.62) | 0.86 (0.86–0.87) | 0.73 (0.73–0.74) |
| 023 | ICD-9 procedure | 0.65 (0.64–0.65) | 0.64 (0.63–0.65) | 0.81 (0.80–0.81) | 0.72 (0.71–0.72) |
| 024 | CXR report findings | 1.00 (0.99–1.00) | 1.00 (0.99–1.00) | 0.99 (0.99–0.99) | 1.00 (1.00–1.00) |
| 025 | Chest CT report findings | 0.99 (0.98–0.99) | 0.99 (0.98–0.99) | 0.96 (0.96–0.97) | 0.99 (0.99–0.99) |
| 026 | Multilingual report coding | 0.96 (0.96–0.97) | 0.94 (0.93–0.94) | 0.94 (0.94–0.95) | 0.96 (0.96–0.97) |
| 027 | Lung cancer T from report | 0.94 (0.92–0.96) | 0.92 (0.90–0.94) | 0.92 (0.89–0.95) | 0.95 (0.93–0.97) |
| 028 | BI-RADS from report | 0.88 (0.87–0.90) | 0.84 (0.82–0.86) | 0.86 (0.82–0.89) | 0.81 (0.77–0.84) |
| 029 | PT stage from pathology report | 0.96 (0.95–0.97) | 0.95 (0.94–0.96) | 0.96 (0.94–0.97) | 0.96 (0.95–0.97) |
| 030 | Trial report inference | 0.90 (0.88–0.93) | 0.87 (0.84–0.90) | 0.86 (0.83–0.90) | 0.92 (0.90–0.95) |
| 031 | PubMedQA | 0.81 (0.78–0.84) | 0.83 (0.79–0.86) | 0.79 (0.76–0.83) | 0.84 (0.81–0.87) |
| 032 | Drug-drug interaction | 0.89 (0.88–0.91) | 0.90 (0.89–0.91) | 0.87 (0.85–0.89) | 0.93 (0.92–0.94) |
| 033 | Hallmarks of cancer | 0.94 (0.93–0.95) | 0.94 (0.93–0.95) | 0.89 (0.87–0.91) | 0.94 (0.93–0.94) |
| 034 | Public-health claims | 0.85 (0.83–0.87) | 0.81 (0.78–0.83) | 0.81 (0.79–0.84) | 0.82 (0.80–0.84) |
| 035 | Question entailment | 0.88 (0.84–0.92) | 0.88 (0.84–0.92) | 0.90 (0.86–0.94) | 0.92 (0.88–0.95) |
| 036 | Rubric judging | 0.83 (0.79–0.86) | 0.73 (0.69–0.77) | 0.78 (0.75–0.82) | 0.84 (0.81–0.88) |
| 037 | ED triage (KTAS) | 0.80 (0.78–0.81) | 0.75 (0.73–0.77) | 0.82 (0.80–0.84) | 0.80 (0.78–0.82) |
| 038 | Diagnosis from case | 0.83 (0.81–0.85) | 0.74 (0.71–0.76) | 0.85 (0.83–0.87) | 0.82 (0.80–0.84) |
| 039 | ACMG variant class | 0.88 (0.87–0.89) | 0.76 (0.74–0.78) | 0.96 (0.95–0.96) | 0.93 (0.92–0.94) |
| 040 | Trial eligibility | 0.85 (0.83–0.88) | 0.85 (0.82–0.88) | 0.83 (0.80–0.86) | 0.87 (0.84–0.90) |
| 041 | Suicide-risk level | 0.77 (0.75–0.80) | 0.75 (0.73–0.78) | 0.81 (0.78–0.83) | 0.82 (0.79–0.84) |
| 042 | MRI safety of a device | 0.58 (0.56–0.61) | 0.51 (0.49–0.54) | 0.51 (0.49–0.53) | 0.66 (0.64–0.68) |
| 043 | Radiograph body part | 0.95 (0.94–0.96) | 0.96 (0.96–0.97) | 0.96 (0.95–0.96) | 0.95 (0.95–0.96) |
| 044 | CXR view AP/PA | 0.98 (0.97–0.99) | 0.99 (0.98–0.99) | 0.89 (0.86–0.91) | 0.99 (0.99–0.99) |
| 045 | Imaging modality | 1.00 (1.00–1.00) | 1.00 (1.00–1.00) | 0.99 (0.99–1.00) | 1.00 (1.00–1.00) |
Table S2. AUROC per task and model with 95% CI from 1,000 bootstrap resamples of cases. Refused or failed questions were scored as wrong with the label prevalence as probability.
Capability Gradient. For the best model on each task, the random-effects mean AUROC was 0.94 (95% CI, 0.92 to 0.97) for medical knowledge, 0.93 (95% CI, 0.91 to 0.94) for text understanding, 0.89 (95% CI, 0.84 to 0.94) for image perception, 0.84 (95% CI, 0.81 to 0.87) for 2D image findings and 0.69 (95% CI, 0.60 to 0.79) for volumetric findings (Figure 2). Imaging tasks scored 0.09 (95% CI, 0.06 to 0.11) lower than text tasks (), and AUROC fell by 0.09 (95% CI, 0.06 to 0.13) per tier from image perception to volumetric findings (). Both contrasts were significant for each of the four models (Figure 2B). Clinical judgement from text reached 0.83 (95% CI, 0.73 to 0.93), lower than the other two text tiers. Chance-corrected balanced accuracy fell from 0.74 for medical knowledge to 0.20 for volumetric findings, and its trend across the imaging tiers was not significant (, Figure 2C).

Figure 2. Capability Map. Panel A shows the AUROC of each primary task (columns) for the four System 1 models, with tasks grouped by capability tier and sorted by the best model. Grey vertical lines span the four models. Horizontal bars and shaded bands show the random-effects mean AUROC of the best model per task with its 95% CI, and the weakest task of each tier is labelled. Panels B and C show the random-effects mean per tier and model with 95% CIs for AUROC and for chance-corrected balanced accuracy, where 0 is chance and 1 is perfect. Refused or failed questions were scored as wrong.
The best model reached an AUROC of 0.99 on licensing-examination questions (MedQA) [23] and 1.00 on findings stated in chest radiograph reports. On volumetric imaging it reached 0.54 for pulmonary embolism, [24] 0.59 for cervical spine fracture, [25] 0.59 for clinically significant prostate cancer [26] and 0.48 for rib fracture type, and four of nine volumetric tasks stayed below 0.60. Intracranial haemorrhage on head CT reached 0.90. [27] On imaging tasks, questions that assigned the whole image to one of several options reached a median AUROC of 0.87, yes-or-no questions on the presence of a finding reached 0.74, and ordered grades 0.71. On text tasks, yes-or-no questions reached 0.99.
Comparison with Published Results and Clinicians. A published result on the same dataset, label set and metric was available for 61 tasks (Figure 3). On text tasks, the best System 1 model matched (within 0.01) or exceeded the best published result in 13 of 26 tasks, with a median difference of −0.01. On imaging tasks it did so in 8 of 35 tasks, with a median difference of −0.17. Clinicians or expert annotators outperformed the best model in 6 of the 10 tasks for which such a comparison existed. Triage nurses assigned the emergency acuity level with an accuracy of 0.85, compared with 0.56 for the best model, [28] and a points-based rule classified genetic variants with an accuracy of 0.95, compared with 0.73. [29, 30] Zero-shot System 1 models are therefore

Figure 3. System 1 Models, Published Results and Clinicians. Panel A shows text tasks and Panel B imaging tasks. For each task with a published result on the same dataset, label set and metric, the best of the four System 1 models (coloured marker) is shown with the best published result (open circle), which can be a task-specific model, a generative language model or a rule, and with clinicians or expert annotators (open triangle). Values of the System 1 models are computed on the questions that match the published result and can differ from the task-level AUROC. Acc denotes accuracy and BA balanced accuracy.
competitive with purpose-built systems on text, but not on images or where a codified rule already exists.
Calibration and Selective Answering. The median top-label ECE across models was 0.05 to 0.07 on text tasks and 0.11 to 0.17 on imaging tasks (Figure 4A and Figure 4B). On imaging tasks, GPT-6 Luna Decisions was over-confident, and about 80% of its most confident answers were correct. Withholding the less confident half of the answers raised accuracy on text tasks from between 0.76 and 0.83 to between 0.96 and 0.99, and on imaging tasks from between 0.66 and 0.70 to between 0.79 and 0.91 (Figure 4C).

Figure 4. Calibration and Language. Panels A and B show the share of correct answers against the probability the model gave its answer, pooled over text tasks in Panel A and over imaging tasks in Panel B. The diagonal marks perfect calibration. Panel C shows, for each model, the accuracy of all answers (open marker) and of its most confident half (filled marker) on text and imaging tasks. Panel D shows accuracy on the same 1,089 medical examination questions in seven languages.
Language. On the same 1,089 medical examination questions, accuracy in every language version was lower than in English for every model (Figure 4D). The decline was smallest in German and largest in Yoruba, where it ranged from 20.1 (95% CI, 17.6 to 22.5) percentage points for Jev 1.14 to 39.4 (95% CI, 36.3 to 42.4) for Clef-flash, whose accuracy fell from 0.93 to 0.54. The performance measured in English therefore overstates what these models deliver in a low-resource language.
System 1 and Generative Models. On identical cases, the balanced accuracy of each System 1 model was lower than that of the generative model of its family, by 4.3 (95% CI, 2.4 to 6.3) percentage points for GPT-6 Luna Decisions, 3.3 (95% CI, 0.0 to 6.5) for Clef and 4.8 (95% CI, 1.0 to 8.5) for Clef-flash, over 32 tasks each (Figure 5A). The difference was smaller on imaging tasks, where Clef-flash and Qwen3.5-9B differed by 0.4 (95% CI, −1.5 to 2.3) percentage points. Each System 1 model scored higher than its generative counterpart on 8 or 9 of the 32 tasks. Generative models required a median of 4.8 to 30.3 seconds per request, compared with 0.34 to 0.44 seconds for System 1 models, and incurred 2- to 22-fold higher costs (Figure 5B and Figure 5C).

Figure 5. System 1 and Generative Models. Panel A shows, for each task, the balanced accuracy of the System 1 model minus that of the generative model of the same family on identical cases, with the mean and 95% CI (task bootstrap) for text and imaging tasks. The pairs are GPT-6 Luna Decisions and GPT-6 Luna, Clef and Qwen3.8-27B, and Clef-flash and Qwen3.5-9B. Panels B and C show the cost per 1,000 requests and the median latency per request on logarithmic scales, with filled markers for System 1 models and open markers for generative models.
Environmental Impact of System 1 Models. Energy was measured directly on the two open-weight models, Clef and Clef-flash, both run locally on one RTX 5090 GPU over 42 of the tasks and 117,263 decisions each (Methods). The median energy of one decision was 23.5 joules for Clef-flash (IQR, 17.2 to 51.3) and 80.9 joules for Clef (IQR, 68.4 to 125.5). That is 6.5 to 22.5 watt-hours of GPU-board energy per 1,000 decisions and, including CPU and RAM and at the carbon intensity of the local grid, 2.8 to 12.1 g COe per 1,000 decisions (Figure 6). Across capability tiers, the median energy per 1,000 decisions was lowest for medical knowledge (4.8 watt-hours for Clef-flash and 19.2 for Clef) and highest for text understanding (19.3 and 47.6). One kilowatt-hour buys roughly 153,000 decisions from Clef-flash and 4,500 from Clef. Decisions that included an image cost 1.6 to 2.3 times as much energy as text-only decisions, and the length of the input mattered more than its type. On the three tasks with the longest documents, the energy of a decision rose to 2.1 to 5.0 times the median for that model. The two models together took 234,526 decisions for 4.3 kilowatt-hours of GPU-board energy. The energy of a single System 1 decision is therefore small enough that it is not the limiting factor in deploying one at scale.

Figure 6. Energy of Locally Run System 1 Models. Panel A shows the GPU energy per 1,000 requests and Panel B the GPU time per request of Clef (8-bit) and Clef-flash (bf16), run locally on one NVIDIA RTX 5090, for each primary task by capability tier. Energy is the whole-board GPU energy. Bars show the median over tasks, and is the number of tasks per tier. No clinical-judgement task was run locally.
Local versus API Inference. Clef and Clef-flash have open weights and can run inside a hospital network, so patient data do not need to leave the institution. Run locally, they reproduced the hosted interface closely. On chest radiograph report findings, local Clef gave a mean AUROC of 0.99 equal to the version inferenced through the API, and agreed with it on 99.7% of answers, while on visual question answering, the values were 0.947 against 0.942 and 97.2% agreement.
Discussion
System 1 models answered medical examination questions and identified findings described in radiology reports with high accuracy, with inference times below half a second and for a few joules per decision. On text-based tasks, predicted probabilities were well calibrated, closely reflecting observed accuracy. However, performance declined on tasks requiring the integration of multiple findings, such as assigning acuity levels in emergency triage. On imaging tasks, they accurately identified imaging modalities and anatomical regions but frequently failed to detect specific abnormalities. Performance on volumetric CT and MRI tasks was particularly limited, approaching chance levels in four of nine tasks.
For information extraction from clinical text, the case for zero-shot System 1 models is strong. The best System 1 model matched or exceeded the best published result on 13 of 26 text tasks without any training on the task, including results from models that had been fine-tuned for that task. For the T staging of lung cancer from CT reports, the best System 1 model reached an accuracy of 0.78, compared with 0.67 for fine-tuned Japanese BERT models.[31] For the pathological T stage from pathology reports, it reached an AUROC of 0.96, compared with 0.94 for a fine-tuned Clinical-BigBird model.[32] A zero-shot System 1 model could therefore replace models that require task-specific fine-tuning. Each decision took less than half a second and 23.5 to 80.9 joules, with costs ranging from 0.02 to 0.16 USD per 1,000 requests. Withholding the 50% of predictions with the lowest confidence increased accuracy on text-based tasks to 0.96 to 0.99. These findings suggest that System 1 models could support large-scale report labelling, information extraction, and literature screening, while referring uncertain cases to clinicians or slower, more computationally intensive models.[3, 22] Clef and Clef-flash have openly available weights and can be deployed within hospital networks.
Accuracy declined on tasks requiring clinical judgement, such as assigning triage acuity or classifying variants, even when all relevant information was provided in the input. System 1 models generate predictions in a single forward pass, relying on knowledge encoded in their weights. Unlike generative models that produce explicit reasoning traces,[33] they lack intermediate reasoning steps through which multiple findings can be evaluated and integrated before reaching a decision. Across the five tasks requiring clinical judgement, the generative backbones of Clef and Clef-flash achieved accuracies 11 and 15 percentage points higher, respectively, compared with differences of 3 and 5 percentage points across all 32 evaluated tasks. Enabling System 1 models to generate reasoning traces before making predictions could help narrow this performance gap.
An additional limitation of most medical benchmarks is their predominantly English-language content. Similarly, most language models, including System 1 models, are trained primarily on English-language data.[34, 35] Reflecting this, the System 1 models were most accurate in English. On the same 1,089 examination questions, accuracy fell by 2 to 7 percentage points in German, a high-resource language, and by 20 to 39 percentage points in Yoruba, a low-resource language. Generative models lose accuracy in low-resource languages in a similar manner. [35, 36]
Most System 1 models also accept images, but read medical images far less accurately than models trained for the task. The best System 1 model matched or exceeded the best published result in 8 of 35 imaging tasks, compared with 13 of 26 text tasks, with a median difference of −0.17 on imaging tasks. For clinically significant prostate cancer on MRI, it reached an AUROC of 0.59, compared with 0.86 for radiologists. [26] Calibration was also worse on imaging tasks, and only about 80% of the most confident answers of GPT-6 Luna Decisions were correct. Volumetric imaging was most challenging as the current interface of System 1 models accepts at most four images per request and no volumes. CT and MRI studies were therefore sent as single slices every 3 to 10 mm, and a small finding can lie between two slices. The choice of pooling rule over slices did not change the results (Supplementary Appendix, Section S6.3). Furthermore, radiographic images had to be downsampled, for example mammograms at 1,024 pixels on the long side, at which microcalcifications cover few pixels. Generative models also lose accuracy when volumes are given as image sequences, [16] which indicates a property of how vision-language models process images in general.
The open-weight decision models Clef and Clef-flash are based on the Qwen family of generative models, allowing a direct comparison of the two model types. The decision models and their backbones did not differ in accuracy, however the decision models were better calibrated, with a median expected calibration error of 0.12 for Clef-flash against 0.20 for its backbone, and 0.07 for Clef against 0.13. Against the generative backbone with reasoning, the System 1 models were 3 to 5 percentage points less accurate and 11 to 76 times faster.
This study has limitations. Models were used zero-shot with a one-sentence description of the input. Task-specific prompts and fine-tuning were not tested. A System 1 model trained on the records of 3.6 million patients from one health system, CS1-4B, was recently reported to outperform open decision models of up to 27 billion parameters on 13 of 14 tasks in held-out patients from that system.[37] Its weights have not yet been released, so it could not be included here. Public datasets can be part of the training data, which raises results on text tasks in particular. Larger frontier generative models were not tested and are likely more accurate on many tasks, but they are slower and more expensive than the generative models compared herein.
For predefined clinical decisions on text, in particular information extraction, System 1 models are ready for use, with fast, inexpensive and well-calibrated answers. They are as accurate as their backbones but better calibrated, and open weights let a hospital run them sustainably on its own data. However, they are currently limited in questions requiring clinical reasoning and in interpreting medical images, in particular CT and MRI volumes.
Acknowledgements
Funded by the European Union under Horizon Europe grant agreement No 101287679 (TWIN-X). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Health and Digital Executive Agency (HaDEA). Neither the European Union nor the granting authority can be held responsible for them.
Supplementary Appendix
The Value of System 1 Models in Medicine
Skip printed tables of contents, lists of figures and lists of tables entirely: neither their "Contents" heading nor their entries belong in @body
Task Selection
The taxonomy splits radiology into five layers (acquisition and metadata, image quality, anatomy, findings, and examination or patient level). Table S1 lists the evaluated tasks.
Requests
The state contained the case text as released, such as the question stem, the report sections or the triage fields, or a one-sentence description of the input together with the image. Each option of a choice question had a short description, and each level of an ordered score was described in words. Images larger than 4 MiB were downsampled by half until they were below 4 MiB. For Jev 1.14, images with a side below 280 or above 1,568 pixels were rescaled. The four 2D imaging tasks with higher-resolution images for Jev 1.14 were melanoma in ISIC 2020, paediatric pneumonia, retinal OCT and diabetic retinopathy grade in RetinaMNIST. On OpenRouter, Clef and Clef-flash were routed only to the Cloudflare provider, because another provider ignored images. When a model refused a question, the question was removed, the request was repeated with the remaining questions, and the refusal was logged. Volumetric slices were taken within the relevant body region, with the window named in the text.
Scoring
For yes-or-no questions, the AUROC used the probability of the positive option. For other questions, the macro average included only the options present in the reference. Balanced accuracy was the recall of the most probable option, averaged over the reference classes. Probabilities were normalised to sum to one, because two models round probabilities to two decimals. The expected calibration error used 10 equal-width bins of the top-label probability. When a task shared its requests with another task (chest radiograph findings and view, head CT haemorrhage and critical findings), each question was counted in one task only.
Generative Comparison
The generative models replied in JSON. Qwen3.8-27B answered the first 50 of the 100 cases per task. Reasoning tokens were counted and billed.
Reference Results
Table S5 lists every published result with its type.
| No. | Task | Type | Reference | Source | Metric | Ref. | Best S1 |
| 001 | MedQA (USMLE) | Generative language model | o1-preview, zero-shot | Nori 2024 | accuracy | 0.96 | 0.89 |
| 001 | MedQA (USMLE) | Task-specific model | Med-Gemini-L 1.0 (self-training, uncertainty-guided web search) | Saab 2024 | accuracy | 0.91 | 0.89 |
| 002 | MedMCQA (India) | Generative language model | o1-preview, zero-shot | Nori 2024 | accuracy | 0.84 | 0.75 |
| 002 | MedMCQA (India) | Task-specific model | Med-PaLM 2 (ensemble refinement) | Singhal 2025 | accuracy | 0.72 | 0.75 |
| 003 | HEAD-QA (English) | Generative language model | GPT-5, zero-shot | Yang 2026 | accuracy | 0.90 | 0.90 |
| 005 | MMLU medical | Generative language model | GPT-5 | Wang 2025 | accuracy | 0.96 | 0.96 |
| 006 | MMLU-Pro health | Generative language model | Gemini-3.1-Pro | Wang 2024 (leaderboard) | accuracy | 0.86 | 0.83 |
| 007 | MedXpertQA text | Clinicians | Expert (pre-licensed humans) | Zuo 2025 | accuracy | 0.43 | 0.35 |
| 007 | MedXpertQA text | Generative language model | GPT-5 | Wang 2025 | accuracy | 0.56 | 0.35 |
| 008 | Medbullets | Generative language model | GPT-5 (2025-08-07) | Bedi 2025 (MedHELM) | accuracy | 0.87 | 0.78 |
| 009 | AfriMed-QA | Generative language model | GPT-4o, zero-shot | Olatunji 2025 | accuracy | 0.79 | 0.82 |
| 010 | MedExpQA | Generative language model | o4-mini, zero-shot | Yang 2026 | accuracy | 0.88 | 0.87 |
| 010 | MedExpQA | Task-specific model | Mistral-7B (fine-tuned), no RAG | Alonso 2024 | accuracy | 0.68 | 0.87 |
| 011 | KorMedMCQA | Clinicians | Examinees average (licensing exam institute) | Kweon 2024 | accuracy | 0.77 | 0.89 |
| 011 | KorMedMCQA | Generative language model | o1-preview, 5-shot | Kweon 2024 | accuracy | 0.93 | 0.89 |
| 012 | CMExam (China) | Clinicians | Human exam takers | Liu 2023 | accuracy | 0.72 | 0.88 |
| 012 | CMExam (China) | Generative language model | Qwen2.5-72B-Instruct | Baichuan 2025 | accuracy | 0.83 | 0.88 |
| 012 | CMExam (China) | Task-specific model | ChatGLM-CMExam (fine-tuned) | Liu 2023 | accuracy | 0.45 | 0.88 |
| 013 | MedQA Mainland | Generative language model | GPT-4 (5-shot CoT) | Qiu 2024 | accuracy | 0.81 | 0.90 |
| 013 | MedQA Mainland | Task-specific model | MMed-Llama 3 8B (fine-tuned) | Qiu 2024 | accuracy | 0.79 | 0.90 |
| 014 | MedQA Taiwan | Generative language model | GPT-4-base (pre-alignment), 5-shot | Nori 2023 | accuracy | 0.87 | 0.90 |
| 014 | MedQA Taiwan | Task-specific model | BioBERT-large (machine-translated to English + retrieval) | Jin 2021 | accuracy | 0.42 | 0.90 |
| 015 | FrenchMedMCQA | Generative language model | GPT-4, 3-shot | Wang 2024 (Apollo) | accuracy | 0.90 | 0.93 |
| 022 | ATC drug class | Generative language model | GPT-4-0125-preview, 4-shot | Shoham 2024 | accuracy | 0.62 | 0.62 |
| 023 | ICD-9 procedure | Generative language model | GPT-4-0125-preview, 4-shot | Shoham 2024 | accuracy | 0.55 | 0.53 |
| 025 | Chest CT report findings | Generative language model | GPT-4o (gpt-4o-2024-11-20), zero-shot, JSON output | Yamagishi 2025 | accuracy | 0.97 | 0.96 |
| 027 | Lung cancer T from report | Generative language model | GPT-3.5-turbo, zero-shot, full TNM definitions in prompt | Matsuo 2024 | accuracy | 0.57 | 0.78 |
| 027 | Lung cancer T from report | Task-specific model | NAIST3SOCR-2 (fine-tuned Japanese BERTs) | Nakamura 2023 | accuracy | 0.67 | 0.78 |
| 029 | pT stage from pathology report | Task-specific model | BB-TEN (Clinical-BigBird, fine-tuned) | Kefeli 2024 | AUROC | 0.94 | 0.96 |
| 031 | PubMedQA | Clinicians | Single human annotator | Jin 2019 | accuracy | 0.78 | 0.78 |
| 031 | PubMedQA | Generative language model | GPT-4 + Medprompt (kNN few-shot, self-generated CoT, choice-shuffle ensemble) | Nori 2023 (Medprompt) | accuracy | 0.82 | 0.78 |
| 031 | PubMedQA | Task-specific model | MedGemma 27B (text, test-time scaling) | Sellergren 2025 | accuracy | 0.77 | 0.78 |
| 035 | Question entailment | Generative language model | gemma-4-31b-it, five-shot | Wu 2025 (BRIDGE) | accuracy | 0.88 | 0.83 |
| 035 | Question entailment | Task-specific model | PANLP (ensemble, rank 1) | Ben Abacha 2019 | accuracy | 0.75 | 0.83 |
| 037 | ED triage (KTAS) | Clinicians | Triage nurses (KTAS RN) | Moon 2019 (in-data) | accuracy | 0.85 | 0.56 |
| 037 | ED triage (KTAS) | Generative language model | KATE (GPT-4o, 40 kNN shots) | Lee 2026 | accuracy | 0.72 | 0.56 |
| 037 | ED triage (KTAS) | Task-specific model | Llama 3.1 8B, fine-tuned | Lee 2026 | accuracy | 0.65 | 0.56 |
| 038 | Diagnosis from case | Generative language model | o3 | Zhu 2026 | accuracy | 0.69 | 0.65 |
| 039 | ACMG variant class | Rule | ACMG/AMP points system on panel evidence codes | Tavtigian 2020 (in-data) | accuracy | 0.95 | 0.73 |
| 040 | Trial eligibility | Generative language model | TrialGPT-Ranking (GPT-4) | Jin 2024 | AUROC | 0.80 | 0.87 |
| 040 | Trial eligibility | Task-specific model | BioLinkBERT cross-encoder (MedNLI) | Jin 2024 | AUROC | 0.62 | 0.87 |
| 041 | Suicide-risk level | Generative language model | GPT-4, zero-shot (best prompt) | Xu 2024 (Mental-LLM) | balanced accuracy | 0.44 | 0.56 |
Table S5. Published results with the same dataset, label set and metric, which can use other splits, and the best of the four System 1 models on the questions that match the published result (Best S1). Best S1 can therefore differ from the task-level AUROC in Table S2.
Supplementary Results
All Tasks and Models
Figure S1 shows the AUROC of every task for every model. Table S2 gives the AUROC with 95% CIs, and Table S3 the random-effects means per tier.

Figure S1. AUROC of every task (rows, numbered as in Table S1) and model (columns). Blue shades mark values above chance (0.5) and red shades values below chance.
| Tier | Tasks | Metric | Clef | Clef-flash | GPT-6 Luna Decisions | Jev 1.14 | Best |
| Medical knowledge | 16 | AUROC | 0.90 (0.83–0.96) | 0.88 (0.81–0.94) | 0.94 (0.91–0.96) | 0.93 (0.89–0.97) | 0.94 (0.92–0.97) |
| Chance-corrected BA | 0.65 (0.47–0.83) | 0.60 (0.44–0.75) | 0.73 (0.63–0.84) | 0.72 (0.57–0.87) | 0.74 (0.63–0.85) | ||
| Text understanding | 13 | AUROC | 0.92 (0.90–0.93) | 0.89 (0.87–0.92) | 0.89 (0.87–0.92) | 0.92 (0.90–0.94) | 0.93 (0.91–0.94) |
| Chance-corrected BA | 0.63 (0.54–0.73) | 0.56 (0.43–0.69) | 0.61 (0.51–0.71) | 0.65 (0.55–0.75) | 0.66 (0.57–0.75) | ||
| Clinical judgement from text | 6 | AUROC | 0.79 (0.70–0.87) | 0.73 (0.64–0.81) | 0.80 (0.66–0.94) | 0.82 (0.73–0.90) | 0.83 (0.73–0.93) |
| Chance-corrected BA | 0.34 (0.14–0.55) | 0.24 (0.10–0.38) | 0.41 (0.10–0.73) | 0.42 (0.12–0.72) | 0.44 (0.12–0.77) | ||
| Image perception | 12 | AUROC | 0.84 (0.77–0.92) | 0.85 (0.79–0.91) | 0.85 (0.76–0.93) | 0.88 (0.83–0.94) | 0.89 (0.84–0.94) |
| Chance-corrected BA | 0.38 (0.31–0.45) | 0.40 (0.31–0.48) | 0.41 (0.24–0.59) | 0.46 (0.26–0.66) | 0.45 (0.28–0.61) | ||
| 2D image findings | 22 | AUROC | 0.80 (0.75–0.84) | 0.80 (0.75–0.84) | 0.77 (0.73–0.80) | 0.83 (0.79–0.86) | 0.84 (0.81–0.87) |
| Chance-corrected BA | 0.28 (0.18–0.38) | 0.29 (0.18–0.40) | 0.26 (0.16–0.35) | 0.39 (0.24–0.53) | 0.40 (0.27–0.53) | ||
| Volumetric findings | 9 | AUROC | 0.64 (0.55–0.73) | 0.64 (0.56–0.73) | 0.67 (0.56–0.78) | 0.68 (0.58–0.78) | 0.69 (0.60–0.79) |
| Chance-corrected BA | 0.13 (0.05–0.21) | 0.14 (0.04–0.23) | 0.14 (0.07–0.22) | 0.16 (0.04–0.27) | 0.20 (0.09–0.32) |
Table S3. Random-effects mean per capability tier with 95% CI. Best denotes the best of the four models on each task.
System 1 Models and Their Backbones
On 20 tasks, Clef and Clef-flash were also compared with their backbones, Qwen3.8-27B and Qwen3.5-9B, with the backbone read out over the same options (Figure S2). Each task had one primary endpoint. The sign test across tasks was not significant for Clef-flash (better on 14 tasks and worse on 5, Holm-adjusted P=0.13) or Clef (12 and 7, Holm-adjusted P=0.36). Five of the 16 significant advantages of a System 1 model disappeared or reversed after the backbone readout was corrected for its option prior.

Figure S2. Primary endpoint of each System 1 model minus that of its backbone on 20 tasks, with 95% bias-corrected and accelerated bootstrap CIs. Filled markers indicate Holm-adjusted within the task.
Pooling of Slices
For volumetric tasks, the mean study-level AUROC of yes-or-no questions over tasks and models was 0.68 with the maximum over slices, 0.69 with the mean of the three highest slices and 0.69 with the mean over slices (Figure S3). Pulmonary embolism and cervical spine fracture stayed close to chance under every rule.

Figure S3. Study-level AUROC of volumetric tasks with yes-or-no questions, averaged over questions, when the slice probabilities of a study were combined by their maximum (primary rule), the mean of the three highest, or the mean over all slices. Each panel shows one task and each line one model.
Generative Comparison
Table S4 lists the balanced accuracy and cost of each System 1 model and its generative model per task.
| No. | Task | System 1 | Generative | BA S1 | BA gen. | USD/1k S1 | USD/1k gen. |
| 001 | MedQA (USMLE) | Clef | Qwen3.8-27B | 0.84 | 0.92 | 0.08 | 1.40 |
| 027 | Lung cancer T from report | Clef | Qwen3.8-27B | 0.65 | 0.83 | 0.11 | 1.62 |
| 028 | BI-RADS from report | Clef | Qwen3.8-27B | 0.54 | 0.35 | 0.12 | 1.64 |
| 029 | pT stage from pathology report | Clef | Qwen3.8-27B | 0.85 | 0.86 | 0.32 | 0.96 |
| 030 | Trial report inference | Clef | Qwen3.8-27B | 0.90 | 0.89 | 0.15 | 1.27 |
| 031 | PubMedQA | Clef | Qwen3.8-27B | 0.57 | 0.70 | 0.12 | 1.03 |
| 032 | Drug-drug interaction | Clef | Qwen3.8-27B | 0.80 | 0.88 | 0.08 | 1.15 |
| 036 | Rubric judging | Clef | Qwen3.8-27B | 0.82 | 0.64 | 0.26 | 1.70 |
| 037 | ED triage (KTAS) | Clef | Qwen3.8-27B | 0.52 | 0.54 | 0.08 | 2.11 |
| 038 | Diagnosis from case | Clef | Qwen3.8-27B | 0.55 | 0.61 | 0.14 | 2.21 |
| 039 | ACMG variant class | Clef | Qwen3.8-27B | 0.39 | 0.72 | 0.08 | 2.45 |
| 040 | Trial eligibility | Clef | Qwen3.8-27B | 0.65 | 0.66 | 0.24 | 1.69 |
| 041 | Suicide-risk level | Clef | Qwen3.8-27B | 0.48 | 0.62 | 0.44 | 1.82 |
| 042 | MRI safety of a device | Clef | Qwen3.8-27B | 0.25 | 0.25 | 0.07 | 0.72 |
| 043 | Radiograph body part | Clef | Qwen3.8-27B | 0.77 | 0.77 | 0.27 | 0.74 |
| 045 | Imaging modality | Clef | Qwen3.8-27B | 0.97 | 0.97 | 0.17 | 0.53 |
| 046 | Mammography view | Clef | Qwen3.8-27B | 0.61 | 0.81 | 0.19 | 1.73 |
| 048 | CT contrast phase (slice) | Clef | Qwen3.8-27B | 0.52 | 0.53 | 0.15 | 2.21 |
| 052 | Vertebra level | Clef | Qwen3.8-27B | 0.62 | 0.58 | 0.36 | 2.02 |
| 054 | Low-dose CT quality | Clef | Qwen3.8-27B | 0.33 | 0.41 | 0.11 | 1.34 |
| 055 | CXR findings | Clef | Qwen3.8-27B | 0.62 | 0.60 | 0.47 | 2.13 |
| 057 | Line and tube position | Clef | Qwen3.8-27B | 0.57 | 0.63 | 0.62 | 3.07 |
| 058 | Bone-age band | Clef | Qwen3.8-27B | 0.17 | 0.26 | 0.26 | 2.33 |
| 059 | Knee OA KL grade | Clef | Qwen3.8-27B | 0.20 | 0.17 | 0.09 | 2.24 |
| 060 | Lumbar stenosis grade | Clef | Qwen3.8-27B | 0.43 | 0.35 | 0.50 | 2.92 |
| 061 | Head CT haemorrhage (slice) | Clef | Qwen3.8-27B | 0.56 | 0.56 | 0.29 | 2.19 |
| 062 | BI-RADS from mammogram | Clef | Qwen3.8-27B | 0.27 | 0.31 | 0.26 | 2.41 |
| 063 | Breast density | Clef | Qwen3.8-27B | 0.28 | 0.28 | 0.21 | 1.95 |
| 064 | Dermoscopy (HAM10000) | Clef | Qwen3.8-27B | 0.47 | 0.46 | 0.14 | 1.83 |
| 067 | DR severity (IDRiD) | Clef | Qwen3.8-27B | 0.30 | 0.30 | 0.24 | 1.57 |
| 070 | Colorectal tissue type | Clef | Qwen3.8-27B | 0.79 | 0.75 | 0.09 | 1.59 |
| 072 | ECG superclasses | Clef | Qwen3.8-27B | 0.49 | 0.62 | 0.38 | 3.11 |
| 001 | MedQA (USMLE) | Clef-flash | Qwen3.5-9B | 0.72 | 0.92 | 0.01 | 0.65 |
| 027 | Lung cancer T from report | Clef-flash | Qwen3.5-9B | 0.48 | 0.84 | 0.02 | 0.81 |
| 028 | BI-RADS from report | Clef-flash | Qwen3.5-9B | 0.51 | 0.32 | 0.02 | 0.80 |
| 029 | pT stage from pathology report | Clef-flash | Qwen3.5-9B | 0.79 | 0.89 | 0.05 | 0.61 |
| 030 | Trial report inference | Clef-flash | Qwen3.5-9B | 0.79 | 0.90 | 0.03 | 0.72 |
| 031 | PubMedQA | Clef-flash | Qwen3.5-9B | 0.57 | 0.58 | 0.02 | 0.71 |
| 032 | Drug-drug interaction | Clef-flash | Qwen3.5-9B | 0.69 | 0.80 | 0.01 | 0.78 |
| 036 | Rubric judging | Clef-flash | Qwen3.5-9B | 0.65 | 0.65 | 0.04 | 0.72 |
| 037 | ED triage (KTAS) | Clef-flash | Qwen3.5-9B | 0.46 | 0.47 | 0.01 | 0.89 |
| 038 | Diagnosis from case | Clef-flash | Qwen3.5-9B | 0.45 | 0.62 | 0.02 | 0.87 |
| 039 | ACMG variant class | Clef-flash | Qwen3.5-9B | 0.34 | 0.64 | 0.01 | 1.17 |
| 040 | Trial eligibility | Clef-flash | Qwen3.5-9B | 0.67 | 0.68 | 0.04 | 0.83 |
| 041 | Suicide-risk level | Clef-flash | Qwen3.5-9B | 0.28 | 0.53 | 0.07 | 1.00 |
| No. | Task | System 1 | Generative | BA S1 | BA gen. | USD/1k S1 | USD/1k gen. |
| 042 | MRI safety of a device | Clef-flash | Qwen3.5-9B | 0.21 | 0.23 | 0.01 | 0.88 |
| 043 | Radiograph body part | Clef-flash | Qwen3.5-9B | 0.84 | 0.81 | 0.04 | 0.23 |
| 045 | Imaging modality | Clef-flash | Qwen3.5-9B | 0.97 | 0.97 | 0.03 | 0.19 |
| 046 | Mammography view | Clef-flash | Qwen3.5-9B | 0.68 | 0.74 | 0.03 | 0.29 |
| 048 | CT contrast phase (slice) | Clef-flash | Qwen3.5-9B | 0.64 | 0.58 | 0.02 | 0.37 |
| 052 | Vertebra level | Clef-flash | Qwen3.5-9B | 0.53 | 0.47 | 0.06 | 1.27 |
| 054 | Low-dose CT quality | Clef-flash | Qwen3.5-9B | 0.33 | 0.37 | 0.02 | 0.23 |
| 055 | CXR findings | Clef-flash | Qwen3.5-9B | 0.65 | 0.65 | 0.08 | 0.72 |
| 057 | Line and tube position | Clef-flash | Qwen3.5-9B | 0.58 | 0.60 | 0.10 | 1.27 |
| 058 | Bone-age band | Clef-flash | Qwen3.5-9B | 0.23 | 0.24 | 0.04 | 1.50 |
| 059 | Knee OA KL grade | Clef-flash | Qwen3.5-9B | 0.35 | 0.28 | 0.01 | 0.39 |
| 060 | Lumbar stenosis grade | Clef-flash | Qwen3.5-9B | 0.33 | 0.39 | 0.08 | 1.84 |
| 061 | Head CT haemorrhage (slice) | Clef-flash | Qwen3.5-9B | 0.56 | 0.62 | 0.05 | 0.61 |
| 062 | BI-RADS from mammogram | Clef-flash | Qwen3.5-9B | 0.33 | 0.37 | 0.04 | 1.04 |
| 063 | Breast density | Clef-flash | Qwen3.5-9B | 0.33 | 0.30 | 0.03 | 0.44 |
| 064 | Dermoscopy (HAM10000) | Clef-flash | Qwen3.5-9B | 0.52 | 0.53 | 0.02 | 0.49 |
| 067 | DR severity (IDRiD) | Clef-flash | Qwen3.5-9B | 0.32 | 0.35 | 0.04 | 0.83 |
| 070 | Colorectal tissue type | Clef-flash | Qwen3.5-9B | 0.65 | 0.68 | 0.01 | 0.25 |
| 072 | ECG superclasses | Clef-flash | Qwen3.5-9B | 0.51 | 0.51 | 0.06 | 1.75 |
| 001 | MedQA (USMLE) | GPT-6 Luna Decisions | GPT-6 Luna | 0.90 | 0.97 | 0.03 | 0.08 |
| 027 | Lung cancer T from report | GPT-6 Luna Decisions | GPT-6 Luna | 0.76 | 0.91 | 0.04 | 0.09 |
| 028 | BI-RADS from report | GPT-6 Luna Decisions | GPT-6 Luna | 0.32 | 0.37 | 0.04 | 0.09 |
| 029 | pT stage from pathology report | GPT-6 Luna Decisions | GPT-6 Luna | 0.86 | 0.87 | 0.13 | 0.18 |
| 030 | Trial report inference | GPT-6 Luna Decisions | GPT-6 Luna | 0.79 | 0.86 | 0.07 | 0.12 |
| 031 | PubMedQA | GPT-6 Luna Decisions | GPT-6 Luna | 0.55 | 0.63 | 0.05 | 0.08 |
| 032 | Drug-drug interaction | GPT-6 Luna Decisions | GPT-6 Luna | 0.79 | 0.84 | 0.03 | 0.06 |
| 036 | Rubric judging | GPT-6 Luna Decisions | GPT-6 Luna | 0.64 | 0.62 | 0.11 | 0.20 |
| 037 | ED triage (KTAS) | GPT-6 Luna Decisions | GPT-6 Luna | 0.53 | 0.56 | 0.03 | 0.10 |
| 038 | Diagnosis from case | GPT-6 Luna Decisions | GPT-6 Luna | 0.66 | 0.77 | 0.05 | 0.12 |
| 039 | ACMG variant class | GPT-6 Luna Decisions | GPT-6 Luna | 0.85 | 0.82 | 0.03 | 0.13 |
| 040 | Trial eligibility | GPT-6 Luna Decisions | GPT-6 Luna | 0.64 | 0.60 | 0.09 | 0.15 |
| 041 | Suicide-risk level | GPT-6 Luna Decisions | GPT-6 Luna | 0.54 | 0.61 | 0.19 | 0.27 |
| 042 | MRI safety of a device | GPT-6 Luna Decisions | GPT-6 Luna | 0.25 | 0.30 | 0.02 | 0.12 |
| 043 | Radiograph body part | GPT-6 Luna Decisions | GPT-6 Luna | 0.88 | 0.90 | 0.11 | 0.18 |
| 045 | Imaging modality | GPT-6 Luna Decisions | GPT-6 Luna | 0.96 | 0.95 | 0.07 | 0.09 |
| 046 | Mammography view | GPT-6 Luna Decisions | GPT-6 Luna | 0.74 | 0.79 | 0.08 | 0.13 |
| 048 | CT contrast phase (slice) | GPT-6 Luna Decisions | GPT-6 Luna | 0.72 | 0.79 | 0.07 | 0.16 |
| 052 | Vertebra level | GPT-6 Luna Decisions | GPT-6 Luna | 0.64 | 0.58 | 0.13 | 0.60 |
| 054 | Low-dose CT quality | GPT-6 Luna Decisions | GPT-6 Luna | 0.40 | 0.46 | 0.04 | 0.08 |
| 055 | CXR findings | GPT-6 Luna Decisions | GPT-6 Luna | 0.67 | 0.65 | 0.29 | 0.38 |
| 057 | Line and tube position | GPT-6 Luna Decisions | GPT-6 Luna | 0.55 | 0.62 | 0.32 | 0.79 |
| 058 | Bone-age band | GPT-6 Luna Decisions | GPT-6 Luna | 0.34 | 0.45 | 0.10 | 0.30 |
| 059 | Knee OA KL grade | GPT-6 Luna Decisions | GPT-6 Luna | 0.25 | 0.33 | 0.03 | 0.13 |
| 060 | Lumbar stenosis grade | GPT-6 Luna Decisions | GPT-6 Luna | 0.50 | 0.57 | 0.21 | 0.64 |
| 061 | Head CT haemorrhage (slice) | GPT-6 Luna Decisions | GPT-6 Luna | 0.59 | 0.60 | 0.15 | 0.26 |
| 062 | BI-RADS from mammogram | GPT-6 Luna Decisions | GPT-6 Luna | 0.29 | 0.44 | 0.10 | 0.27 |
| 063 | Breast density | GPT-6 Luna Decisions | GPT-6 Luna | 0.33 | 0.32 | 0.08 | 0.14 |
Table S4. System 1 model against the generative model of its family on the same cases, 100 per task and 50 for Clef and Qwen3.8-27B. BA denotes balanced accuracy, USD/1k the cost in US dollars per 1,000 requests, S1 the System 1 model and gen. the generative model.
| Task | Clef | Clef-flash | GPT-6 Luna Decisions | Jev 1.14 |
| 001 MedQA (USMLE) | 0.97 | 0.92 | 0.98 | 0.99 |
| 002 MedMCQA (India) | 0.89 | 0.87 | 0.92 | 0.93 |
| 003 HEAD-QA (English) | 0.96 | 0.95 | 0.96 | 0.97 |
| 004 HEAD-QA (Spanish) | 0.96 | 0.95 | 0.97 | 0.97 |
| 005 MMLU medical | 0.99 | 0.99 | 0.98 | 0.99 |
| 006 MMLU-Pro health | 0.97 | 0.96 | 0.96 | 0.97 |
| 007 MedXpertQA text | 0.72 | 0.63 | 0.76 | 0.77 |
| 008 Medbullets | 0.92 | 0.83 | 0.94 | 0.94 |
| 009 AfriMed-QA | 0.94 | 0.92 | 0.95 | 0.95 |
| 010 MedExpQA | 0.96 | 0.92 | 0.96 | 0.96 |
| 011 KorMedMCQA | 0.96 | 0.93 | 0.99 | 0.98 |
| 012 CMExam (China) | 0.97 | 0.95 | 0.97 | 0.98 |
| 013 MedQA Mainland | 0.98 | 0.96 | 0.98 | 0.98 |
| 014 MedQA Taiwan | 0.97 | 0.95 | 0.98 | 0.98 |
| 015 FrenchMedMCQA | 0.99 | 0.97 | 0.99 | 0.99 |
| 016 MMLU medical (Arabic) | 0.95 | 0.93 | 0.98 | 0.98 |
| 017 MMLU medical (Hindi) | 0.92 | 0.84 | 0.98 | 0.98 |
| 018 MMLU medical (Japanese) | 0.97 | 0.96 | 0.98 | 0.98 |
| 019 MMLU medical (Swahili) | 0.90 | 0.85 | 0.96 | 0.96 |
| 020 MMLU medical (Yoruba) | 0.85 | 0.79 | 0.91 | 0.93 |
| 021 MMLU medical (German) | 0.98 | 0.97 | 0.98 | 0.99 |
| 022 ATC drug class | 0.53 | 0.61 | 0.86 | 0.73 |
| 023 ICD-9 procedure | 0.65 | 0.64 | 0.81 | 0.72 |
| 024 CXR report findings | 1.00 | 1.00 | 0.99 | 1.00 |
| 025 Chest CT report findings | 0.99 | 0.99 | 0.96 | 0.99 |
| 026 Multilingual report coding | 0.96 | 0.94 | 0.94 | 0.96 |
| 027 Lung cancer T from report | 0.94 | 0.92 | 0.92 | 0.95 |
| 028 BI-RADS from report | 0.88 | 0.84 | 0.86 | 0.81 |
| 029 pT stage from pathology report | 0.96 | 0.95 | 0.96 | 0.96 |
| 030 Trial report inference | 0.90 | 0.87 | 0.86 | 0.92 |
| 031 PubMedQA | 0.81 | 0.83 | 0.79 | 0.84 |
| 032 Drug-drug interaction | 0.89 | 0.90 | 0.87 | 0.93 |
| 033 Hallmarks of cancer | 0.94 | 0.94 | 0.89 | 0.94 |
| 034 Public-health claims | 0.85 | 0.81 | 0.81 | 0.82 |
| 035 Question entailment | 0.88 | 0.88 | 0.90 | 0.92 |
| 036 Rubric judging | 0.83 | 0.73 | 0.78 | 0.84 |
| 037 ED triage (KTAS) | 0.80 | 0.75 | 0.82 | 0.80 |
| 038 Diagnosis from case | 0.83 | 0.74 | 0.85 | 0.82 |
| 039 ACMG variant class | 0.88 | 0.76 | 0.96 | 0.93 |
| 040 Trial eligibility | 0.85 | 0.85 | 0.83 | 0.87 |
| 041 Suicide-risk level | 0.77 | 0.75 | 0.81 | 0.82 |
| 042 MRI safety of a device | 0.58 | 0.51 | 0.51 | 0.66 |
| 043 Radiograph body part | 0.95 | 0.96 | 0.96 | 0.95 |
| 044 CXR view AP/PA | 0.98 | 0.99 | 0.89 | 0.99 |
| 045 Imaging modality | 1.00 | 1.00 | 0.99 | 1.00 |
| 046 Mammography view | 0.92 | 0.96 | 0.96 | 0.99 |
| 047 CT contrast phase (series) | 0.82 | 0.82 | 0.90 | 0.90 |
| 048 CT contrast phase (slice) | 0.75 | 0.73 | 0.85 | 0.84 |
| 049 Abdominal organ (3D) | 0.83 | 0.84 | 0.82 | 0.84 |
| 050 Abdominal organ (coronal) | 0.78 | 0.84 | 0.71 | 0.82 |
| 051 Abdominal organ (sagittal) | 0.72 | 0.80 | 0.68 | 0.80 |
| 052 Vertebra level | 0.94 | 0.94 | 0.97 | 0.95 |
| 053 MRI motion artefact | 0.71 | 0.71 | 0.70 | 0.76 |
| 054 Low-dose CT quality | 0.72 | 0.62 | 0.76 | 0.78 |
| 055 CXR findings | 0.72 | 0.77 | 0.73 | 0.77 |
| 056 Paediatric pneumonia | 0.95 | 0.96 | 0.95 | 0.93 |
| 057 Line and tube position | 0.78 | 0.79 | 0.65 | 0.81 |
| 058 Bone-age band | 0.75 | 0.73 | 0.81 | 0.77 |
| 059 Knee OA KL grade | 0.66 | 0.69 | 0.66 | 0.64 |
| 060 Lumbar stenosis grade | 0.68 | 0.66 | 0.81 | 0.73 |
| 061 Head CT haemorrhage (slice) | 0.76 | 0.75 | 0.76 | 0.79 |
| 062 BI-RADS from mammogram | 0.62 | 0.61 | 0.68 | 0.69 |
| 063 Breast density | 0.71 | 0.68 | 0.70 | 0.66 |
| 064 Dermoscopy (HAM10000) | 0.87 | 0.87 | 0.83 | 0.90 |
| 065 Dermoscopy (ISIC 2019) | 0.81 | 0.82 | 0.78 | 0.84 |
| 066 Melanoma (ISIC 2020) | 0.76 | 0.78 | 0.56 | 0.75 |
| 067 DR severity (IDRiD) | 0.75 | 0.82 | 0.61 | 0.83 |
| 068 DR grade (RetinaMNIST) | 0.65 | 0.70 | 0.57 | 0.86 |
| 069 Retinal OCT diagnosis | 0.97 | 0.98 | 0.92 | 1.00 |
| 070 Colorectal tissue type | 0.98 | 0.96 | 0.87 | 0.96 |
| 071 Blood cell type | 0.97 | 0.98 | 0.86 | 0.99 |
| 072 ECG superclasses | 0.73 | 0.67 | 0.74 | 0.68 |
| 073 VQA-RAD yes/no | 0.91 | 0.88 | 0.87 | 0.94 |
| 074 SLAKE yes/no | 0.94 | 0.92 | 0.90 | 0.97 |
| 075 SLAKE yes/no (Chinese) | 0.96 | 0.92 | 0.93 | 0.99 |
| 076 PMC-VQA | 0.84 | 0.83 | 0.82 | 0.86 |
| 077 MedXpertQA multimodal | 0.72 | 0.64 | 0.79 | 0.79 |
| 078 Head CT haemorrhage | 0.83 | 0.85 | 0.90 | 0.89 |
| 079 Critical head CT finding | 0.84 | 0.79 | 0.88 | 0.87 |
| 080 Chest CT abnormalities | 0.69 | 0.71 | 0.76 | 0.74 |
| 081 Pulmonary embolism on CTPA | 0.51 | 0.52 | 0.54 | 0.54 |
| 082 Cervical spine fracture | 0.57 | 0.58 | 0.59 | 0.53 |
| 083 Abdominal trauma on CT | 0.69 | 0.73 | 0.74 | 0.79 |
| 084 Pancreatic cancer on CT | 0.61 | 0.68 | 0.64 | 0.70 |
| 085 Prostate cancer bpMRI | 0.58 | 0.44 | 0.55 | 0.59 |
| 086 Rib fracture type | 0.41 | 0.48 | 0.42 | 0.44 |
Table S1. AUROC of every task (rows, numbered as in Table S1) and model (columns). Blue shades mark values above chance (0.5) and red shades values below chance.
| No. | Task | Dataset | Tier | Input | Language | Cases | Questions |
| 001 | USMLE-style MCQ (MedQA, 4 options) | MedQA [23] | Medical knowledge | text | English | 1,273 | 1 |
| 002 | Indian medical entrance exam MCQ (MedMCQA) | MedMCQA [38] | Medical knowledge | text | English | 4,183 | 1 |
| 003 | Spanish healthcare specialisation exam MCQ (HEAD-QA) | HEAD-QA [39] | Medical knowledge | text | English | 2,675 | 1 |
| 004 | Spanish healthcare specialisation exam MCQ, Spanish original (HEAD-QA) | HEAD-QA [39] | Medical knowledge | text | Spanish | 2,675 | 1 |
| 005 | Medical subsets of MMLU | MMLU [40] | Medical knowledge | text | English | 1,089 | 1 |
| 006 | Health questions of MMLU-Pro | MMLU-Pro [41] | Medical knowledge | text | English | 818 | 1 |
| 007 | Expert-level medical MCQ, text (MedXpertQA Text) | MedXpertQA [42] | Medical knowledge | text | English | 2,450 | 1 |
| 008 | Hard clinical case MCQ (Medbullets op5) | Medbullets op5 [43] | Medical knowledge | text | English | 308 | 1 |
| 009 | Pan-African medical MCQ (AfriMed-QA v2) | AfriMed-QA v2 [35] | Medical knowledge | text | English | 3,600 | 1 |
| 010 | Multilingual clinical-case MCQ (MedExpQA) | MedExpQA [36] | Medical knowledge | text | Several | 500 | 1 |
| 011 | Korean healthcare licensing exam MCQ (KorMedMCQA) | KorMedMCQA [44] | Medical knowledge | text | Korean | 3,009 | 1 |
| 012 | Chinese national medical licensing exam MCQ (CMExam) | CMExam [45] | Medical knowledge | text | Chinese | 6,607 | 1 |
| 013 | Chinese medical licensing exam MCQ (MedQA Mainland) | MedQA [23] | Medical knowledge | text | Chinese | 3,426 | 1 |
| 014 | Taiwanese medical licensing exam MCQ (MedQA Taiwan) | MedQA [23] | Medical knowledge | text | Chinese | 1,413 | 1 |
| 015 | French pharmacy exam MCQ (FrenchMedMCQA) | FrenchMedMCQA [46] | Medical knowledge | text | French | 321 | 1 |
| 016 | Medical subsets of MMLU in Arabic (Global-MMLU) | Global-MMLU [34] | Medical knowledge | text | Arabic | 1,089 | 1 |
| 017 | Medical subsets of MMLU in Hindi (Global-MMLU) | Global-MMLU [34] | Medical knowledge | text | Hindi | 1,089 | 1 |
| 018 | Medical subsets of MMLU in Japanese (Global-MMLU) | Global-MMLU [34] | Medical knowledge | text | Japanese | 1,089 | 1 |
| 019 | Medical subsets of MMLU in Swahili (Global-MMLU) | Global-MMLU [34] | Medical knowledge | text | Swahili | 1,089 | 1 |
| 020 | Medical subsets of MMLU in Yoruba (Global-MMLU) | Global-MMLU [34] | Medical knowledge | text | Yoruba | 1,089 | 1 |
| 021 | Medical subsets of MMLU in German (Global-MMLU) | Global-MMLU [34] | Medical knowledge | text | German | 1,089 | 1 |
| 022 | Drug class from ATC code (MedConceptsQA) | MedConceptsQA [47] | Medical knowledge | text | English | 18,806 | 1 |
| 023 | Procedure description from ICD-9-CM code (MedConceptsQA) | MedConceptsQA [47] | Medical knowledge | text | English | 13,766 | 1 |
| 024 | Chest radiograph report findings (OpenI / IU X-ray) | OpenI / IU X-ray [48] | Text understanding | text | English | 3,832 | 13 |
| 025 | Chest CT findings from Japanese reports (CT-RATE-JPN) | CT-RATE-JPN [49, 50] | Text understanding | text | Japanese | 150 | 18 |
| 026 | Modality and ICD-10 chapters of multilingual reports (PARROT) | PARROT [51] | Text understanding | text | Several | 2,738 | 13 |
| 027 | Lung cancer T category from radiology reports (NTCIR-17 RR-TNM) | NTCIR-17 RR-TNM [31] | Text understanding | text | Japanese | 243 | 1 |
| 028 | BI-RADS category from breast imaging reports (Paraguay corpus) | Paraguay reports [52] | Text understanding | text | English | 1,000 | 1 |
| 029 | Pathologic T stage from a pathology report (TCGA) | TCGA reports [32, 53] | Text understanding | text | English | 1,000 | 1 |
| 030 | Clinical-trial report inference (NLI4CT, SemEval-2024 Task 2) | NLI4CT [54] | Text understanding | text | English | 500 | 1 |
| 031 | Biomedical research QA (PubMedQA PQA-L) | PubMedQA PQA-L [55] | Text understanding | text | English | 500 | 1 |
| 032 | Drug-drug interaction type (DDI-2013) | DDI-2013 [56, 57] | Text understanding | text | English | 1,000 | 1 |
| 033 | Hallmarks of cancer in abstracts (HoC) | HoC [58] | Text understanding | text | English | 371 | 10 |
| 034 | Public-health claim veracity (PUBHEALTH) | PUBHEALTH [59] | Text understanding | text | English | 1,233 | 1 |
| 035 | Consumer health question entailment (MEDIQA-RQE 2019) | MEDIQA-RQE 2019 [60] | Text understanding | text | English | 230 | 1 |
| 036 | Rubric criterion judging (HealthBench meta-evaluation) | HealthBench [61] | Text understanding | text | English | 1,000 | 1 |
| 037 | ED triage acuity (KTAS) | KTAS [28] | Clinical judgement from text | text | English | 1,267 | 1 |
| 038 | Diagnosis selection from a published case (DiagnosisArena) | DiagnosisArena [62] | Clinical judgement from text | text | English | 915 | 1 |
| 039 | ACMG/AMP five-tier variant classification (ClinGen) | ClinGen [63] | Clinical judgement from text | text | English | 1,000 | 1 |
| 040 | Patient-trial eligibility (TREC Clinical Trials 2021/2022) | TREC CT 2021/22 [64] | Clinical judgement from text | text | English | 500 | 1 |
| 041 | Suicide-risk severity of a Reddit user (Reddit C-SSRS) | Reddit C-SSRS [65] | Clinical judgement from text | text | English | 500 | 1 |
| 042 | MRI safety status of a device (FDA AccessGUDID) | FDA AccessGUDID [66] | Clinical judgement from text | text | English | 1,000 | 1 |
| 043 | Radiograph body part | UNIFESP body part [67] | Image perception | 2D | English | 593 | 1 |
| 044 | Chest radiograph view AP vs PA (NIH ChestX-ray14) | NIH ChestX-ray14 [68] | Image perception | 2D | English | 716 | 1 |
| 045 | Imaging modality (ROCOV2) | ROCOV2 [69] | Image perception | 2D | English | 794 | 1 |
| No. | Task | Dataset | Tier | Input | Language | Cases | Questions |
| 046 | Mammography view CC vs MLO (CBIS-DDSM) | CBIS-DDSM [70, 71] | Image perception | 2D | English | 645 | 1 |
| 047 | CT contrast phase (series level, WAW-TACE) | WAW-TACE [72] | Image perception | 3D | English | 160 | 1 |
| 048 | Contrast status and phase (single slice, WAW-TACE) | WAW-TACE [72] | Image perception | 2D | English | 960 | 2 |
| 049 | Abdominal organ on CT, 3D (OrganMNIST3D) | OrganMNIST3D [73, 74] | Image perception | 2D multi-view | English | 610 | 1 |
| 050 | Abdominal organ on coronal CT (OrganCMNIST) | OrganCMNIST [73, 74] | Image perception | 2D | English | 8,216 | 1 |
| 051 | Abdominal organ on sagittal CT (OrganSMNIST) | OrganSMNIST [73, 74] | Image perception | 2D | English | 8,827 | 1 |
| 052 | Marked vertebra level on CT (VerSe) | VerSe [75] | Image perception | 2D multi-view | English | 219 | 2 |
| 053 | MRI motion artefact severity (MR-ART) | MR-ART [76] | Image perception | 3D | English | 436 | 1 |
| 054 | Diagnostic image quality of low-dose CT (LDCTIQAC 2023) | LDCTIQAC 2023 [77] | Image perception | 2D | English | 1,000 | 1 |
| 055 | Chest radiograph multi-label findings (NIH ChestX-ray14) | NIH ChestX-ray14 [68] | 2D image findings | 2D | English | 716 | 14 |
| 056 | Pneumonia on paediatric chest radiographs (PneumoniaMNIST) | PneumoniaMNIST [73, 78] | 2D image findings | 2D | English | 624 | 1 |
| 057 | Line and tube position on chest radiographs | RANZCR CLiP [68, 79] | 2D image findings | 2D | English | 497 | 11 |
| 058 | Paediatric bone-age band | RSNA bone age [80] | 2D image findings | 2D | English | 500 | 1 |
| 059 | Knee osteoarthritis KL grade | OAI [82] [81, 82] | 2D image findings | 2D | English | 1,000 | 1 |
| 060 | Lumbar degenerative stenosis grade | RSNA lumbar 2024 [83] | 2D image findings | 2D multi-view | English | 1,000 | 5 |
| 061 | Head CT haemorrhage and subtype (slice) | RSNA ICH 2019 [84] | 2D image findings | 2D | English | 1,000 | 6 |
| 062 | BI-RADS assessment from mammograms (CBIS-DDSM) | CBIS-DDSM [70, 71] | 2D image findings | 2D multi-view | English | 704 | 1 |
| 063 | Breast density category (CBIS-DDSM) | CBIS-DDSM [70, 71] | 2D image findings | 2D | English | 643 | 1 |
| 064 | Dermoscopic lesion diagnosis (HAM10000 / ISIC 2018 Task 3) | HAM10000 [85, 86] | 2D image findings | 2D | English | 515 | 1 |
| 065 | Dermoscopic lesion diagnosis (ISIC 2019) | ISIC 2019 [86, 87, 88] | 2D image findings | 2D | English | 8,238 | 1 |
| 066 | Melanoma vs other lesion (SIIM-ISIC 2020) | SIIM-ISIC 2020 [89] | 2D image findings | 2D | English | 33,126 | 1 |
| 067 | Diabetic retinopathy severity (IDRiD) | IDRiD [90] | 2D image findings | 2D | English | 516 | 1 |
| 068 | Diabetic retinopathy grade (RetinaMNIST) | RetinaMNIST [73, 91] | 2D image findings | 2D | English | 400 | 1 |
| 069 | Retinal OCT diagnosis (OCTMNIST) | OCTMNIST [73, 78] | 2D image findings | 2D | English | 1,000 | 1 |
| 070 | Colorectal H&E tissue type (CRC-VAL-HE-7K) | CRC-VAL-HE-7K [92] | 2D image findings | 2D | English | 900 | 1 |
| 071 | Blood cell type (BloodMNIST) | BloodMNIST [73, 93] | 2D image findings | 2D | English | 3,421 | 1 |
| 072 | 12-lead ECG superclasses (PTB-XL, rendered as image) | PTB-XL [94] | 2D image findings | 2D | English | 2,198 | 5 |
| 073 | Radiology VQA, closed yes/no questions (VQA-RAD) | VQA-RAD [95] | 2D image findings | text+image | English | 251 | 1 |
| 074 | Radiology VQA, closed yes/no questions (SLAKE) | SLAKE [96] | 2D image findings | text+image | English | 355 | 1 |
| 075 | Radiology VQA in Chinese, closed yes/no questions (SLAKE) | SLAKE [96] | 2D image findings | text+image | Chinese | 347 | 1 |
| 076 | Biomedical figure VQA (PMC-VQA) | PMC-VQA [97] | 2D image findings | text+image | English | 2,000 | 1 |
| 077 | Expert-level medical MCQ with images (MedXpertQA MM) | MedXpertQA [42] | 2D image findings | text+image | English | 2,000 | 1 |
| 078 | Head CT haemorrhage and subtype (scan level, CQ500) | CQ500 [27] | Volumetric findings | 3D | English | 81 | 6 |
| 079 | Critical head CT finding (CQ500) | CQ500 [27] | Volumetric findings | 3D | English | 81 | 4 |
| 080 | Chest CT multi-abnormality (CT-RATE) | CT-RATE [50] | Volumetric findings | 3D | English | 100 | 18 |
| 081 | Pulmonary embolism on CTPA | RSNA PE 2020 [24] | Volumetric findings | 3D | English | 150 | 6 |
| 082 | Cervical spine fracture on CT | RSNA C-spine 2022 [25] | Volumetric findings | 3D | English | 200 | 8 |
| 083 | Traumatic abdominal organ injury on CT | RSNA trauma 2023 [98] | Volumetric findings | 3D | English | 200 | 8 |
| 084 | Pancreatic ductal adenocarcinoma on contrast-enhanced CT (PANORAMA) | PANORAMA [99] | Volumetric findings | 3D | English | 300 | 1 |
| 085 | Clinically significant prostate cancer on bpMRI (PI-CAI) | PI-CAI [26] | Volumetric findings | 3D | English | 200 | 1 |
| 086 | Rib fracture type (FractureMNIST3D, RibFrac) | FractureMNIST3D [73, 100] | Volumetric findings | 2D multi-view | English | 240 | 1 |
Table S1 (PDF p. 17). Table S1 | Tasks. Tasks 004, 016, 017, 018, 019, 020, 021, 075 are language versions of another task. MCQ denotes multiple-choice questions, VQA visual question answering, and ED emergency department.
| No. | Task | Clef | Clef-flash | GPT-6 | Luna Decisions | Jev 1.14 |
| 046 | Mammography view | 0.92 (0.90–0.94) | 0.96 (0.95–0.98) | 0.96 (0.95–0.98) | 0.99 (0.98–0.99) | |
| 047 | CT contrast phase (series) | 0.82 (0.79–0.85) | 0.82 (0.78–0.86) | 0.90 (0.87–0.93) | 0.90 (0.87–0.92) | |
| 048 | CT contrast phase (slice) | 0.75 (0.73–0.77) | 0.73 (0.70–0.75) | 0.85 (0.83–0.86) | 0.84 (0.82–0.86) | |
| 049 | Abdominal organ (3D) | 0.83 (0.82–0.85) | 0.84 (0.83–0.86) | 0.82 (0.81–0.84) | 0.84 (0.82–0.85) | |
| 050 | Abdominal organ (coronal) | 0.78 (0.78–0.79) | 0.84 (0.84–0.84) | 0.71 (0.70–0.71) | 0.82 (0.82–0.83) | |
| 051 | Abdominal organ (sagittal) | 0.72 (0.72–0.73) | 0.80 (0.80–0.81) | 0.68 (0.67–0.68) | 0.80 (0.80–0.81) | |
| 052 | Vertebra level | 0.94 (0.93–0.95) | 0.94 (0.93–0.95) | 0.97 (0.96–0.97) | 0.95 (0.94–0.96) | |
| 053 | MRI motion artefact | 0.71 (0.69–0.74) | 0.71 (0.68–0.73) | 0.70 (0.67–0.72) | 0.76 (0.73–0.78) | |
| 054 | Low-dose CT quality | 0.72 (0.70–0.74) | 0.62 (0.60–0.65) | 0.76 (0.74–0.78) | 0.78 (0.77–0.80) | |
| 055 | CXR findings | 0.72 (0.69–0.74) | 0.77 (0.75–0.79) | 0.73 (0.71–0.76) | 0.77 (0.75–0.79) | |
| 056 | Paediatric pneumonia | 0.95 (0.93–0.96) | 0.96 (0.94–0.97) | 0.95 (0.93–0.96) | 0.93 (0.91–0.95) | |
| 057 | Line and tube position | 0.78 (0.76–0.79) | 0.79 (0.78–0.80) | 0.65 (0.63–0.67) | 0.81 (0.79–0.82) | |
| 058 | Bone-age band | 0.75 (0.73–0.77) | 0.73 (0.71–0.75) | 0.81 (0.79–0.83) | 0.77 (0.75–0.78) | |
| 059 | Knee OA KL grade | 0.66 (0.64–0.68) | 0.69 (0.67–0.70) | 0.66 (0.63–0.69) | 0.64 (0.62–0.66) | |
| 060 | Lumbar stenosis grade | 0.68 (0.66–0.70) | 0.66 (0.65–0.68) | 0.81 (0.80–0.83) | 0.73 (0.71–0.75) | |
| 061 | Head CT haemorrhage (slice) | 0.76 (0.74–0.78) | 0.75 (0.73–0.77) | 0.76 (0.74–0.78) | 0.79 (0.77–0.80) | |
| 062 | BI-RADS from mammogram | 0.62 (0.59–0.65) | 0.61 (0.59–0.65) | 0.68 (0.65–0.70) | 0.69 (0.65–0.72) | |
| 063 | Breast density | 0.71 (0.68–0.73) | 0.68 (0.65–0.71) | 0.70 (0.67–0.73) | 0.66 (0.63–0.69) | |
| 064 | Dermoscopy (HAM10000) | 0.87 (0.85–0.89) | 0.87 (0.85–0.89) | 0.83 (0.81–0.85) | 0.90 (0.89–0.92) | |
| 065 | Dermoscopy (ISIC 2019) | 0.81 (0.81–0.82) | 0.82 (0.82–0.83) | 0.78 (0.77–0.79) | 0.84 (0.83–0.84) | |
| 066 | Melanoma (ISIC 2020) | 0.76 (0.74–0.78) | 0.78 (0.76–0.80) | 0.56 (0.53–0.59) | 0.75 (0.73–0.77) | |
| 067 | DR severity (IDRiD) | 0.75 (0.72–0.78) | 0.82 (0.80–0.85) | 0.61 (0.58–0.64) | 0.83 (0.81–0.85) | |
| 068 | DR grade (RetinaMNIST) | 0.65 (0.61–0.69) | 0.70 (0.66–0.74) | 0.57 (0.54–0.61) | 0.86 (0.83–0.89) | |
| 069 | Retinal OCT diagnosis | 0.97 (0.97–0.98) | 0.98 (0.98–0.99) | 0.92 (0.90–0.93) | 1.00 (1.00–1.00) | |
| 070 | Colorectal tissue type | 0.98 (0.97–0.98) | 0.96 (0.95–0.97) | 0.87 (0.86–0.89) | 0.96 (0.95–0.97) | |
| 071 | Blood cell type | 0.97 (0.97–0.98) | 0.98 (0.98–0.98) | 0.86 (0.85–0.87) | 0.99 (0.98–0.99) | |
| 072 | ECG superclasses | 0.73 (0.72–0.75) | 0.67 (0.66–0.68) | 0.74 (0.73–0.75) | 0.68 (0.66–0.69) | |
| 073 | VQA-RAD yes/no | 0.91 (0.87–0.94) | 0.88 (0.84–0.92) | 0.87 (0.82–0.91) | 0.94 (0.91–0.96) | |
| 074 | SLAKE yes/no | 0.94 (0.92–0.96) | 0.92 (0.89–0.95) | 0.90 (0.87–0.94) | 0.97 (0.96–0.99) | |
| 075 | SLAKE yes/no (Chinese) | 0.96 (0.94–0.98) | 0.92 (0.89–0.95) | 0.93 (0.90–0.96) | 0.99 (0.98–0.99) | |
| 076 | PMC-VQA | 0.84 (0.83–0.86) | 0.83 (0.82–0.85) | 0.82 (0.81–0.84) | 0.86 (0.85–0.88) | |
| 077 | MedXpertQA multimodal | 0.72 (0.70–0.73) | 0.64 (0.63–0.66) | 0.79 (0.78–0.81) | 0.79 (0.78–0.81) | |
| 078 | Head CT haemorrhage | 0.83 (0.75–0.89) | 0.85 (0.80–0.91) | 0.90 (0.85–0.94) | 0.89 (0.84–0.93) | |
| 079 | Critical head CT finding | 0.84 (0.78–0.88) | 0.79 (0.74–0.85) | 0.88 (0.83–0.92) | 0.87 (0.83–0.90) | |
| 080 | Chest CT abnormalities | 0.69 (0.66–0.73) | 0.71 (0.68–0.74) | 0.76 (0.73–0.79) | 0.74 (0.71–0.77) | |
| 081 | Pulmonary embolism on CTPA | 0.51 (0.45–0.56) | 0.52 (0.45–0.59) | 0.54 (0.49–0.59) | 0.54 (0.49–0.59) | |
| 082 | Cervical spine fracture | 0.57 (0.51–0.63) | 0.58 (0.53–0.64) | 0.59 (0.54–0.65) | 0.53 (0.47–0.60) | |
| 083 | Abdominal trauma on CT | 0.69 (0.64–0.74) | 0.73 (0.68–0.77) | 0.74 (0.68–0.78) | 0.79 (0.75–0.83) | |
| 084 | Pancreatic cancer on CT | 0.61 (0.55–0.68) | 0.68 (0.61–0.74) | 0.64 (0.58–0.71) | 0.70 (0.64–0.75) | |
| 085 | Prostate cancer bpMRI | 0.58 (0.50–0.66) | 0.44 (0.36–0.51) | 0.55 (0.46–0.63) | 0.59 (0.51–0.67) | |
| 086 | Rib fracture type | 0.41 (0.37–0.46) | 0.48 (0.43–0.53) | 0.42 (0.37–0.47) | 0.44 (0.39–0.49) |
Table 1.
| No. | Task | System 1 | Generative | BA S1 | BA gen. | USD/1k S1 | USD/1k gen. |
| 064 | Dermoscopy (HAM10000) | GPT-6 Luna Decisions | GPT-6 Luna | 0.44 | 0.51 | 0.05 | 0.16 |
| 067 | DR severity (IDRiD) | GPT-6 Luna Decisions | GPT-6 Luna | 0.33 | 0.42 | 0.10 | 0.22 |
| 070 | Colorectal tissue type | GPT-6 Luna Decisions | GPT-6 Luna | 0.55 | 0.45 | 0.03 | 0.10 |
| 072 | ECG superclasses | GPT-6 Luna Decisions | GPT-6 Luna | 0.57 | 0.62 | 0.19 | 0.75 |
Table 2. System 1 models in medicine
| No. | Task | Type | Reference | Source | Metric | Ref. | Best S1 |
| 041 | Suicide-risk level | Task-specific model | Mental-FLAN-T5 (FLAN-T5-XXL, instruction-finetuned on four mental-health datasets) | Xu 2024 (Mental-LLM) | balanced accuracy | 0.48 | 0.56 |
| 044 | CXR view AP/PA | Task-specific model | ResNet-18 | Kim 2019 | AUROC | 1.00 | 0.99 |
| 046 | Mammography view | Task-specific model | ResNet-50, fine-tuned, 256 × 256 px | Yi 2019 | AUROC | 1.00 | 0.99 |
| 047 | CT contrast phase (series) | Task-specific model | 2dFM_BERT (frozen MAE ViT-L/16 + BERT) | Hou 2025 | AUROC | 0.91 | 0.90 |
| 049 | Abdominal organ (3D) | Task-specific model | 3D CNN with Kinetics-400 video initialization, fine-tuned | Jain 2024 | accuracy | 0.97 | 0.38 |
| 050 | Abdominal organ (coronal) | Generative language model | Qwen3-VL-8B with retrieved exemplars and rules (in-context learning, frozen) | Zhao 2026 | accuracy | 0.94 | 0.40 |
| 050 | Abdominal organ (coronal) | Task-specific model | DenseNet-121, 128 px | Doerrich 2024 | accuracy | 0.94 | 0.40 |
| 051 | Abdominal organ (sagittal) | Task-specific model | DenseNet-121, 64 px | Doerrich 2024 | accuracy | 0.83 | 0.31 |
| 053 | MRI motion artefact | Task-specific model | FewSOME (unsupervised few-shot) | Belton 2024 | accuracy | 0.72 | 0.32 |
| 055 | CXR findings | Task-specific model | Ark+ (Swin-L, 768 px; fine-tuned) | Ma 2025 (Ark+) | AUROC | 0.84 | 0.77 |
| 056 | Paediatric pneumonia | Generative language model | MedGemma-27B, evolved prompt, no fine-tuning | Sviridov 2026 | accuracy | 0.84 | 0.79 |
| 056 | Paediatric pneumonia | Task-specific model | DINO ViT-B/16, linear probing, 224 px | Doerrich 2025 | AUROC | 0.99 | 0.96 |
| 057 | Line and tube position | Clinicians | Radiologist (Reader 2 of 2, board-certified/board-eligible) | Güzel 2026 | balanced accuracy | 0.75 | 0.50 |
| 057 | Line and tube position | Generative language model | Gemini 3.1 Flash Lite Preview, zero-shot (Swan-Ganz) | Güzel 2026 | balanced accuracy | 0.54 | 0.87 |
| 057 | Line and tube position | Task-specific model | Rank 1 (All Data Are Ext) | Kaggle RANZCR CLiP private LB | AUROC | 0.98 | 0.81 |
| 059 | Knee OA KL grade | Task-specific model | Ensemble of 6 ImageNet-pretrained CNNs, per-model optimised input size, mix voting | Pi 2023 | accuracy | 0.77 | 0.42 |
| 061 | Head CT haemorrhage (slice) | Generative language model | Gemini 2.0 Flash, zero-shot | Wang 2025 | accuracy | 0.75 | 0.71 |
| 061 | Head CT haemorrhage (slice) | Task-specific model | Rank-1 Kaggle ensemble | Wang 2021 | AUROC | 1.00 | 0.88 |
| 063 | Breast density | Task-specific model | LG-DPTNet: multi-view (CC + MLO) fusion, dynamic pyramid convolution + cross-transformer | Zhong 2023 | AUROC | 0.97 | 0.71 |
| 064 | Dermoscopy (HAM10000) | Clinicians | Human experts (> 10 years) | Tschandl 2019 | balanced accuracy | 0.81 | 0.62 |
| 064 | Dermoscopy (HAM10000) | Generative language model | Hulu-Med-7B, zero-shot | Liu 2026 (DermAgent) | accuracy | 0.53 | 0.63 |
| 064 | Dermoscopy (HAM10000) | Task-specific model | Rank 1, team ’Minnie’, approach ’ensemble b4 ce+fl+cepos+ds1 b3 ce’ | ISIC 2018 live leaderboard | balanced accuracy | 0.90 | 0.62 |
| 065 | Dermatoscopy (ISIC 2019) | Task-specific model | Rank 1 (DAISYLab, EfficientNet ensemble, external data) | ISIC 2019 leaderboard | balanced accuracy | 0.64 | 0.48 |
| 066 | Melanoma (ISIC 2020) | Task-specific model | Rank 1 of 3,308 entries (CNN ensemble) | Kurtansky 2024 | AUROC | 0.95 | 0.78 |
| 067 | DR severity (IDRiD) | Task-specific model | Multi-stage transfer learning (ImageNet → EyePACS → DDR → IDRiD) + class-balanced loss | Shi 2023 | accuracy | 0.80 | 0.56 |
| 068 | DR grade (RetinaMNIST) | Generative language model | Gemini-3-flash, zero-shot | Zhao 2026 | accuracy | 0.44 | 0.66 |
| 068 | DR grade (RetinaMNIST) | Task-specific model | VGG16, end-to-end training, 224 × 224 | Doerrich 2025 | accuracy | 0.64 | 0.66 |
| 069 | Retinal OCT diagnosis | Task-specific model | Inception V3, ImageNet transfer learning (108,312 training images) | Kermany 2018 | accuracy | 0.97 | 0.97 |
| 070 | Colorectal tissue type | Generative language model | GPT-4V, 5-shot in-context (kNN) | Ferber 2024 | accuracy | 0.78 | 0.74 |
| 070 | Colorectal tissue type | Task-specific model | CONCH, linear probe trained on NCT-CRC-HE-100K | Lu 2024 (CONCH) | balanced accuracy | 0.94 | 0.74 |
| 071 | Blood cell type | Task-specific model | DenseNet-121, end-to-end, 224 px | Doerrich 2025 | accuracy | 0.99 | 0.80 |
| 072 | ECG superclasses | Generative language model | GPT-4o, zero-shot | Liu 2024 (PULSE) | AUROC | 0.56 | 0.74 |
| 072 | ECG superclasses | Task-specific model | Ensemble of all models | Ströder 2021 | AUROC | 0.93 | 0.74 |
| 073 | VQA-RAD yes/no | Generative language model | GPT-5-mini, zero-shot chain-of-thought | Wang 2025 | accuracy | 0.75 | 0.84 |
| 073 | VQA-RAD yes/no | Task-specific model | MedVInT-TD (fine-tuned) | Zhang 2023 | accuracy | 0.87 | 0.84 |
| 074 | SLAKE yes/no | Generative language model | MedGemma 4B, zero-shot | Sellgren 2025 | accuracy | 0.88 | 0.93 |
| 074 | SLAKE yes/no | Task-specific model | M2I2 (supervised) | Li 2023 | accuracy | 0.91 | 0.93 |
| 076 | PMC-VQA | Generative language model | LLaVA-Med, zero-shot | Zhang 2023 | accuracy | 0.35 | 0.64 |
| 076 | PMC-VQA | Task-specific model | MedVInT-TD (trained on PMC-VQA) | Zhang 2023 | accuracy | 0.40 | 0.64 |
| No. | Task | Type | Reference | Source | Metric | Ref. | Best S1 |
| 077 | MedXpertQA multimodal | Clinicians | Expert (pre-licensed humans) | Zuo 2025 | accuracy | 0.46 | 0.49 |
| 077 | MedXpertQA multimodal | Generative language model | GPT-5 | Wang 2025 | accuracy | 0.72 | 0.49 |
| 078 | Head CT haemorrhage | Task-specific model | Qure.ai deep learning algorithms | Chilamkurthy 2018 | AUROC | 0.97 | 0.84 |
| 079 | Critical head CT finding | Task-specific model | Qure.ai deep learning algorithms | Chilamkurthy 2018 | AUROC | 0.97 | 0.99 |
| 080 | Chest CT abnormalities | Task-specific model | CT-CLIP VocabFine | Hamamci 2026 | AUROC | 0.76 | 0.76 |
| 081 | Pulmonary embolism on CTPA | Task-specific model | Attention-based CNN-RNN, strongly supervised (all slice labels) | Hu 2025 | AUROC | 0.93 | 0.58 |
| 082 | Cervical spine fracture | Task-specific model | Mean of top 8 Kaggle algorithms | Lee 2024 | AUROC | 0.96 | 0.63 |
| 083 | Abdominal trauma on CT | Task-specific model | Mean of 8 winning challenge models | Hermans 2025 | AUROC | 0.98 | 0.85 |
| 084 | Pancreatic cancer on CT | Clinicians | 68 radiologists (pooled) | Alves 2026 | AUROC | 0.88 | 0.70 |
| 084 | Pancreatic cancer on CT | Task-specific model | Rank 1 (MountainHill / DTI, DTI_PANORAMA) | PANORAMA leaderboard | AUROC | 0.93 | 0.70 |
| 085 | Prostate cancer bpMRI | Clinicians | 62 radiologists (mpMRI, PI-RADS 2.1) | Saha 2024 | AUROC | 0.86 | 0.59 |
| 085 | Prostate cancer bpMRI | Task-specific model | PI-CAI AI system | Saha 2024 | AUROC | 0.93 | 0.59 |
| 086 | Rib fracture type | Task-specific model | Evolutionary neural architecture search with genetic-algorithm data augmentation | Ali 2024 | accuracy | 0.69 | 0.41 |
Table 27.
References
- [1]Ben Van Calster, David J McLernon, Maarten van Smeden, Laure Wynants, Ewout W Steyerberg, and Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative. Calibration: the Achilles heel of predictive analytics. BMC Med, 17(1):230, 2019. doi: 10.1186/s12916-019-1466-7.
- [2]Benjamin Comanescu, Jasper Snoek, and Andrew L Beam. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digit Med, 4(1):4, 2021. doi: 10.1038/s41746-020-00367-3.DOI
- [3]Krishnamurthy Dj Vijotham, Jim Winkens, Melih Barsbey, Sumedh Ghais̄as, Robert Stanforth, Nick Pawlowski, Patricia Strachan, Zahra Ahmed, Shekoofeh Azizi, Yoram Bachrach, Laura Culp, Mayank Daswani, Jan Freyberg, Christopher Kelly, Atilla Kiraly, et al. Enhancing the reliability and accuracy of AI-enabled diagnosis via complementarity-driven deferral to clinicians. Nat Med, 29(7):1814–1820, 2023. doi: 10.1038/s41591-023-02437-x.
- [4]Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023. doi: 10.1038/s41586-023-06291-2.DOI
- [5]Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of GPT-4 on medical challenge problems. 2023. arXiv:2303.13375.arxiv.org/abs/2303.13375
- [6]Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: watts driving the cost of AI deployment? In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 85–99, 2024. doi: 10.1145/3630106.3658542.DOI
- [7]Thomas Savage, John Wang, Robert Gallo, Abdesselam Boukil, Vishwesh Patel, Seyed Amir Ahmad Safavi-Naini, Ali Soroush, and Jonathan H Chen. Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment. J Am Med Inform Assoc, 32(1):139–149, 2025. doi: 10.1093/jamia/ocae254.DOI
- [8]Mahmud Omar, Reem Agbarea, Benjamin S Glicksberg, Girish N Nadkarni, and Eyal Klang. Benchmarking the confidence of large language models in answering clinical questions: cross-sectional evaluation study. JMIR Med Inform, 13:e66917, 2025. doi: 10.2196/66917.DOI
- [9]Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, New York, 2011. ISBN 9780374275631.
- [10]Pat Croskerry. A universal model of diagnostic reasoning. Acad Med, 84(8):1022–1028, 2009. doi: 10.1097/ACM.0b013e3181ace703.DOI
- [11]Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason A Fries, Michael Wornow, Akshay Swaminathan, Lisa Soleymani Lehmann, Hyo Jung Hong, Mehr Kashyap, Akash R Chaurasia, Nirav R Shah, Karandeep Singh, et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA, 333(4):319–328, 2025. doi: 10.1001/jama.2024.21700.DOI
- [12]Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, Darlene Neal, Qazi Mamunur Rashid, Mike Schaekermann, Amy Wang, Dev Dash, et al. Toward expert-level medical question answering with large language models. Nat Med, 31(3):943–950, 2025. doi: 10.1038/s41591-024-03423-7.DOI
- [13]Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, and Daniel Rueckert. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med, 30(9):2613–2622, 2024. doi: 10.1038/s41591-024-03097-1.DOI
- [14]Shreya Johri, Jaehwan Jeong, Benjamin A Tran, Daniel I Schlessinger, Shannon Wongvibulsin, Leandra A Barnes, Hong-Yu Zhou, Zhuo Ran Cai, Eliezer M Van Allen, David Kim, Roxana Daneshjou, and Pranav Rajpurkar. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat Med, 31(1):77–86, 2025. doi: 10.1038/s41591-024-03328-5.DOI
- [15]Quirin D Strotzer, Felix Nieberle, Laura S Kupke, Gerardo Napodano, Anna Katharina Muertz, Stefanie Meiler, Ingo Einspieler, Janine Rennert, Michael Strotzer, Isabel Wiesinger, Christina Wendl, Christian Stroszczynski, Okka W Hamer, and Andreas Schicho. Toward foundation models in radiology? Quantitative assessment of GPT-4V’s multimodal and multianatomical region capabilities. Radiology, 313(2):e240955, 2024. doi: 10.1148/radiol.240955.DOI
- [16]Qingxia Wu, Qingxia Wu, Peipei Zhang, Zhifeng Yi, Yu Shen, Yan Bai, Hongna Tan, Pei Dong, Zhong Xue, Neil Roberts, and Meiyun Wang. A bilingual benchmark for evaluating diagnostic performance of multimodal large language models in radiology (RadM-Bench): evaluation development and validation. J Med Internet Res, 28:e92183, 2026. doi: 10.2196/92183.DOI
- [17]Inioluwa Deborah Raji, Emily M. Bender, Amandalynn Paullada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. doi: 10.48550/arXiv.2111.15366. URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/084b6fb b10729ed4da8c3d3f5a3ae7c9-Abstract-round2.html. arXiv:2111.15366.DOI
- [18]Qwen Team. Qwen3.8-Max: A new bar for coding and cowork, August 2026. URL https://qwen.ai/blog?id=qwen3.8.
- [19]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5.
- [20]Tugba Akincı D’Antoni and Florence Doo. CEDARS: Carbon, energy diagnostics and reporting for sustainability. Software, 2026. URL https://github.com/takinci/cedars. Apache-2.0. Accessed October 2026.
- [21]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330, 2017. doi: 10.48550/arXiv.1706.04599. URL https:// proceedings.mlr.press/v70/guo17a.html. arXiv:1706.04599.DOI
- [22]Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017. doi: 10.48550/arXiv.1705.08500. URL https:// proceedings.neurips.cc/paper/2017/hash/4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html. arXiv:1705.08500.DOI
- [23]Di Jin, Eileen Pan, Nassim Oufatolle, Wei-Hung Weng, Hanxi Fang, and Peter Szolovits. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl Sci, 11(14):6421, 2021. doi: 10.3390/app11146421.DOI
- [24]Errol Colak, Felipe C Kitamura, Stephen B Hobbs, Carol C Wu, Matthew P Lungren, Luciano M Prevedello, Jayashree Kalpathy-Cramer, Robyn L Ball, George Shih, Anouk Stein, Sarfwan S Halabi, Emre Altinmakas, Meng Law, Parveen Kumar, Karam A Manzalawi, et al. The RSNA Pulmonary Embolism CT dataset. Radiol Artif Intell, 3(2):e200254, 2021. doi: 10.1148/ryai.2021200254.DOI
- [25]Hui Ming Lin, Errol Colak, Tyler Richards, Felipe C Kitamura, Luciano M Prevedello, Jason Talbott, Robyn L Ball, Ekim Gumeler, Kristen W Yeom, Mohammad Hamghalam, Amber L Simpson, Jasna Strika, Deniz Bulja, Salita Angkruwaranon, Almudena Pérez-Lara, et al. The RSNA Cervical Spine Fracture CT dataset. Radiol Artif Intell, 5(5):e2300034, 2023. doi: 10.1148/ryai.2300034.DOI
- [26]Anindo Saha, Joeran S Bosma, Jasper J Twilt, Bram van Ginneken, Anders Bjartell, Anwar R Padhani, David Bonekamp, Geert Villeirs, Georg Salomon, Gianluca Giannarini, Jayashree Kalpathy-Cramer, Jelle Barentsz, Klaus H Maier-Hein, Mirabela Rusu, Olivier Rouvière, et al. Artificial intelligence and radiologists in prostate cancer detection on MRI (PI-CAI): an international, paired, non-inferiority, confirmatory study. Lancet Oncol, 25(7):879–887, 2024. doi: 10.1016/S1470-2045(24)00020-1.DOI
- [27]Sasanik Chlamtchuk, Rohit Ghosh, Swetha Tanamala, Mustafa Biviji, Norbert G Campeau, Vasantha Kumar Venugopal, Vidur Mahajan, Pooja Rao, and Prashant Warier. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet, 392(10162):2388–2396, 2018. doi: 10.1016/S0140-6736(18)31645-3.
- [28]Sun-Hee Moon, Jae Lan Shim, Keun-Sook Park, and Chon-Suk Park. Triage accuracy and causes of mistriage using the Korean Triage and Acuity Scale. PLoS One, 14(9):e0216972, 2019. doi: 10.1371/journal.pone.0216972.DOI
- [29]Sue Richards, Nazneen Aziz, Sherri Bale, David Bick, Soma Das, Julie Gastier-Foster, Wayne W Grody, Madhuri Hegde, Elaine Lyon, Elaine Specter, Karl Veelkenring, Heidi L Rehm, and ACMG Laboratory Quality Assurance Committee. Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genet Med, 17(5):405–424, 2015. doi: 10.1038/gim.2015.30.DOI
- [30]Sean V Tavtigian, Steven M Harrison, Kenneth M Boucher, and Leslie G Bieseker. Fitting a naturally scaled point system to the ACMG/AMP variant classification guidelines. Hum Mutat, 41(10):1734–1737, 2020. doi: 10.1002/humu.24088.DOI
- [31]Syuta Nakamura, Shouhei Hanaoka, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. NTCIR-17 MedNLP-SC radiology report subtask overview: dataset and solutions for automated lung cancer staging. In Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, pages 145–151, 2023. doi: 10.20736/0002001328.DOI
- [32]Jenna Kefeli, Jacob Berkowitz, Jose M Acitores Cortina, Kevin K Tsang, and Nicholas P Tatonetti. Generalizable and automated classification of TNM stage from pathology reports with external validation. Nat Commun, 15(1):8916, 2024. doi: 10.1038/s41467-024-53190-9.DOI
- [33]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. 2023. URL https://arxiv.org/abs/2201.11903.
- [34]Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, PeeraT Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, et al. Global MMLU: understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, 2025. doi: 10.18653/v1/2025.acl-long.919.DOI
- [35]Charles Nimo, Tobi Olatunji, Abraham Toluwase Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, Ezinwanne C. Aka, Folafunmi Omofoye, Foutse Yuehogh, Timothy Faniran, Bonaventure F. P. Dossou, Moshood O. Yekini, Jonas Kemp, Katherine A Heller, Jude Chidubem Omeke, et al. AfriMed-QA: a Pan-African, multi-specialty, medical question-answering benchmark dataset. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 1948–1973, 2025. doi: 10.18653/v1/2025.acl-long.96.DOI
- [36]Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. MedExpQA: multilingual benchmarking of large language models for medical question answering. Artif Intell Med, 155:102938, 2024. doi: 10.1016/j.artmed.2024.102938.DOI
- [37]Andrew Zhang, Trinh T. L. Vuong, Sophia J. Wagner, Dongmin Bang, Tong Ding, Yuchang Su, Mary Cipperman, Christina S. Chen, Judith Lefkes, Ramin Kahidi, Ming Y. Lu, Cuiwei Tian, Tianhao Luo, Jingnan Ma, Rowland Pettit, Long Phi Le, and Faisal Mahmood. A general-purpose system one decision model for clinical records, 2026. URL https://arxiv.org/abs/2610.27099.
- [38]Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankar subbu. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 248–260, 2022. doi: 10.48550/arXiv.2203.14371. URL https://proceedings.mlr.press/v174/pal22a.html. arXiv:2203.14371.DOI
- [39]David Vilares and Carlos Gómez-Rodríguez. HEAD-QA: a healthcare dataset for complex reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 960–966, 2019. doi: 10.18653/v1/P19-1092.DOI
- [40]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021. doi: 10.48550/arXiv.2009.03300. URL https://iclr.cc/virtual/2021/poster/2962. arXiv:2009.03300.
- [41]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, et al. MMLU-Pro: a more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 37, pages 95266–95290, 2024. doi: 10.48550/arXiv.2406.01574. URL https://papers.nips.cc/paper_files/paper/2024/hash/ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets_and_Benchmarks_Track.html. arXiv:2406.01574.DOI
- [42]Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. MedXpertQA: benchmarking expert-level medical reasoning and understanding. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 80961–80990, 2025. doi: 10.48550/arXiv.2501.18362. URL https://proceedings.mlr.press/v267/zuo25a.html. arXiv:2501.18362.DOI
- [43]Hanjie Chen, Xiaozhuang Fang, Yash Singla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3563–3599, 2025. doi: 10.18653/v1/2025.naacl-long.182.
- [44]Sunjun Kweon, Byungjin Choi, Gyouk Chu, Junyeong Song, Daeun Hyeon, Sujin Gan, Jueon Kim, Minkyu Kim, Rae Woong Park, and Edward Choi. KorMedMCQA: multi-choice question answering benchmark for Korean healthcare professional licensing examinations, 2024. arXiv:2403.01469.arxiv.org/abs/2403.01469
- [45]Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, and Michael Lengzhi Li. Benchmarking large language models on CMExam – a comprehensive Chinese medical exam dataset. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 36, pages 52430–52452, 2023. doi: 10.48550/arXiv.2306.03030. URL https://papers.nips.cc/paper_files/paper/2023/hash/a48ad12d588c597f4725a8b84af647b5-Abstract-Datasets_and_Benchmarks.html. arXiv:2306.03030.DOI
- [46]Yanis Labrak, Adrien Bazoge, Richard Dufour, Béatrice Daille, Pierre-Antoine Gourraud, Emmanuel Morin, and Mickael Rouvier. FrenchMedMCQA: a French multiple-choice question answering dataset for medical domain. In Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), pages 41–46, 2022. doi: 10.18653/v1/2022.louhi-1.5.DOI
- [47]Ofir Ben Shoham and Nadav Rappoport. MedConceptsQA: open source medical concepts QA benchmark. Comput Biol Med, 182:109089, 2024. doi: 10.1016/j.compbiomed.2024.109089.DOI
- [48]Dina Denner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval. J Am Med Inform Assoc, 23(2):304–310, 2016. doi: 10.1093/jamia/ocv080.
- [49]Yosuke Yamagishi, Yuta Nakamura, Tomohiro Kikuchi, Yuki Sonoda, Hiroshi Hirakawa, Shintaro Kano, Satoshi Nakamura, Shouhei Hanaoka, Takeharu Yoshikawa, and Osamu Abe. Development of a large-scale dataset of chest computed tomography reports in Japanese and a high-performance finding classification model: dataset development and validation study. JMIR Med Inform, 13:e71137, 2025. doi: 10.2196/71137.DOI
- [50]Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Seval Nil Esirgun, İrem Dogan, Omer Faruk Durugol, Benjamin Hou, Suprosanna S. Wicheng Dai, Murong Xu, Hadrien Reynaud, Muhammed Furkan Dasdelen, Bastian Wittmann, et al. General foundation models from a multimodal dataset for 3D computed tomography. Nat Biomed Eng, 10(8):1610–1628, 2026. doi: 10.1038/s41551-025-01599-y.DOI
- [51]Bastien Le Guellec, Kokou Adambounou, Lisa C Adams, Thibault Agripnidis, Sung Soo Ahn, Radhia Ait Chalal, Tugba Akinci D’Antonioli, Philippe Amouyel, Henrick Andersson, Raphaël Bentegac, Claudio Benzoni, Antonino Andrea Blandino, Felix Busch, Elif Can, Riccardo Cau, et al. PARROT, an open multilingual radiology reports dataset. Eur J Radiol Artif Intell, 5:100066, 2026. doi: 10.1016/j.ejrai.2025.100066.DOI
- [52]José Luis Vázquez Noguera, Alejandro Torres-Hurtado, Helena Gómez-Adorno, Julio César Mello-Román, Enrique Javier Fleitas-Alvarez, Federico Fernando Espinola Schulze, Miguel Garcia-Torres, Carlos Domingo Méndez Gaona, Pedro Esteban Gardel Sotomayor, Silvia Vázquez Noguera, Norma Elizabeth Zaracho Amarilla, and Oxades Wilfrido Gamarra Esquivel. Mammography reporting dataset with BI-RADS system for natural language processing applications: addressing public data gaps in Spanish. Data Brief, 61:111761, 2025. doi: 10.1016/j.dib.2025.111761.DOI
- [53]Jenna Kefeli and Nicholas Tatonetti. TCGA-Reports: a machine-readable pathology report resource for benchmarking text-based AI models. Patterns (N Y), 5(3):100933, 2024. doi: 10.1016/j.patter.2024.100933.DOI
- [54]Mael Jullien, Marco Valentino, and André Freitas. SemEval-2024 task 2: safe biomedical natural language inference for clinical trials. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1947–1962, 2024. doi: 10.18653/v1/2024.semeval-1.271. URL https://aclanthology.org/2024.semeval-1.271/.DOI
- [55]Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: a dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, 2019. doi: 10.18653/v1/D19-1259.DOI
- [56]María Herrero-Zazo, Isabel Segura-Bedmar, Paloma Martínez, and Thierry Declerck. The DDI corpus: an annotated corpus with pharmacological substances and drug-drug interactions. J Biomed Inform, 46(5):914–920, 2013. doi: 10.1016/j.jbi.2013.07.011.
- [57]Isabel Segura-Bedmar, Paloma Martínez, and María Herrero-Zazo. SemEval-2013 task 9: extraction of drug-drug interactions from biomedical texts (DDIExtraction 2013). In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 341–350, 2013. URL https://aclanthology.org/S13-2056/.
- [58]Simon Baker, Ilona Silins, Yufan Guo, Imran Ali, Johan Högberg, Ulla Stenius, and Anna Korhonen. Automatic semantic classification of scientific literature according to the hallmarks of cancer. Bioinformatics, 32(3):432–440, 2016. doi: 10.1093/bioinformatics/btv585.DOI
- [59]Neema Kotonya and Francesca Toni. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7740–7754, 2020. doi: 10.18653/v1/2020.emnlp-main.623. URL https://aclanthology.org/2020.emnlp-main.623/.DOI
- [60]Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. Overview of the MEDIQA 2019 shared task on textual inference, question entailment and question answering. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 370–379, 2019. doi: 10.18653/v1/W19-5039. URL https://aclanthology.org/W19-5039/.DOI
- [61]Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. HealthBench: evaluating large language models towards improved human health, 2025. arXiv:2505.08775.arxiv.org/abs/2505.08775
- [62]Yakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang, Wei Nie, Jiaji Liu, Shaoting Zhang, Pengfei Liu, and Xiao-fan Zhang. DiagnosisArena: benchmarking diagnostic reasoning for large language models. In Findings of the Association for Computational Linguistics: ACL 2026, pages 3074–3098, 2026. doi: 10.18653/v1/2026.findings-acl.151. URL https://aclanthology.org/2026.findings-acl.151/. arXiv:2505.14107.DOI
- [63]Heidi L Rehm, Jonathan S Berg, Lisa D Brooks, Carlos D Bustamante, James P Evans, Melissa J Landrum, David H Ledbetter, Donna R Maglott, Christa Lese Martin, Robert L Nussbaum, Sharon P Oplen, Erin M Ramos, Stephen T Sherry, Michael S Watson, and ClinGen. ClinGen–the Clinical Genome Resource. N Engl J Med, 372(23):2235–2242, 2015. doi: 10.1056/NEJMsr1406261.DOI
- [64]Kirk Roberts, Dina Demner-Fushman, Ellen M Voorhees, Steven Bedrick, and William R Hersh. Overview of the TREC 2022 Clinical Trials track. In The Thirty-First Text REtrieval Conference Proceedings (TREC 2022), NIST Special Publication 500-338, 2022. doi: 10.6028/NIST.SP.500-338.trials-overview. URL https://trec.nist.gov/pubs/trec31/papers/Overview_trials.pdf.
- [65]Manas Gaur, Amanuel Alambo, Joy Prakash Sain, Ugur Kursuncu, Krishnaprasad Thirunarayan, Ramakanth Kavuluru, Amit Sheth, Randy Welton, and Jyotishman Pathak. Knowledge-aware assessment of severity of suicide risk for early intervention. In The World Wide Web Conference (WWW ’19), pages 514–525, 2019. doi: 10.1145/3308558.3313698.DOI
- [66]U.S. Food and Drug Administration. AccessGUDID: Global Unique Device Identification Database (GUDID), 2026. URL https://accessgudid.nlm.nih.gov/. Accessed October 2026.
- [67]Universidade Federal de São Paulo (UNIFESP). UNIFESP X-ray Body Part Classifier Competition, 2022. URL https://www.kaggle.com/datasets/felipkamitura/unifesp-xray-bodypart-classification. Accessed October 2026.
- [68]Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadihadi Bagheri, and Ronald M. Summers. ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3462–3471, 2017. doi: 10.1109/CVPR.2017.369.DOI
- [69]Johannes Rückert, Louise Bloch, Raphael Brü̈ngel, Ahmad Idrissi-Yaghir, Henning Schäfer, Cynthia S Schmidt, Sven Koitka, Obioma Pelka, Asma Ben Abacha, Alba G Seco de Herrera, Henning Müller, Peter A Horn, Felix Nensa, and Christoph M Friedrich. ROCov2: Radiology Objects in Context version 2, an updated multimodality image dataset. Sci Data, 11(1):688, 2024. doi: 10.1038/s41597-024-03496-6.DOI
- [70]Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, Kanae Kawai Miyake, Mia Gorovoy, and Daniel L Rubin. A curated mammography data set for use in computer-aided detection and diagnosis research. Sci Data, 4:170177, 2017. doi: 10.1038/sdata.2017.177.
- [71]Rebecca Sawyer-Lee, Francisco Gimenez, Assaf Hoogi, and Daniel Rubin. Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM), 2016. Data set.
- [72]Krzysztof Bartnik, Tomasz Bartczak, Mateusz Krzyziński, Krzysztof Korzeniowski, Krzysztof Lamparski, Piotr Węgrzyn, Eric Lam, Mateusz Bartkowiak, Tadeusz Wróblewski, Katarzyna Mech, Magdalena Januszewicz, and Przemysław Bięcek. WAW-TACE: a hepatocellular carcinoma multiphase CT dataset with segmentations, radiomics features, and clinical data. Radiol Artif Intell, 6(6):e240296, 2024. doi: 10.1148/ryai.240296.DOI
- [73]Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification. Sci Data, 10(1):41, 2023. doi: 10.1038/s41597-022-01721-8.DOI
- [74]Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kainis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Maman, Gabriel Chartrand, Fabian Löhöfler, Julian Walter Holch, Wieland Sommer, Felix Hofmann, Alexandre Hostettler, et al. The Liver Tumor Segmentation Benchmark (LiTS). Med Image Anal, 84:102680, 2023. doi: 10.1016/j.media.2022.102680.DOI
- [75]Anjany Sekuboyina, Malek E Husseini, Amirhossein Bayat, Maximilian Löffler, Hans Liebl, Hongwei Li, Giles Tetteh, Jan Kukačka, Christian Payer, Darko Štern, Martin Urschler, Maodong Chen, Dalong Cheng, Nikolas Lessmann, Yujin Hu, et al. VerSe: a vertebrae labelling and segmentation benchmark for multi-detector CT images. Med Image Anal, 73:102166, 2021. doi: 10.1016/j.media.2021.102166.DOI
- [76]Ádám Nárai, Petra Hermann, Tibor Auer, Péter Kemenczky, János Szalma, István Homolya, Eszter Somogyi, Pál Vági, Béla Weiss, and Zoltán Vidnyánszky. Movement-related artefacts (MR-ART) dataset of matched motion-corrupted and clean structural MRI brain scans. Sci Data, 9(1):630, 2022. doi: 10.1038/s41597-022-01694-8.DOI
- [77]Wonkyeong Lee, Fabian Wagner, Adrian Galdran, Yongyi Shi, Wenjun Xia, Ge Wang, Xuanqin Mou, Md Atik Ahamed, Abdullah Al Zubaer Imran, Ji Eun Oh, Kyungsang Kim, Jong Tak Baek, Dongheon Lee, Boowhi Hong, Philip Templeman, et al. Low-dose computed tomography perceptual image quality assessment. Med Image Anal, 99:103343, 2025. doi: 10.1016/j.media.2024.103343.
- [78]Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina G S Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, Justin Dong, Made K Prasadha, Jacqueline Pei, Magdalene Y L Ting, Jie Zhu, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell, 172(5):1122–1131.e9, 2018. doi: 10.1016/j.cell.2018.02.010.DOI
- [79]Jennifer S N Tang, Jarrel C Y Seah, Adil Zia, Jay Gajera, Richard N Schlegel, Aaron J N Wong, Dayu Gai, Shu Su, Tony Bose, Marcus L Kok, Alex Jarema, George N Harisis, Chris-Tin Cheng, Helen Kavnoudias, Wayland Wang, et al. CLIP, catheter and line position dataset. Sci Data, 8(1):285, 2021. doi: 10.1038/s41597-021-01066-8.DOI
- [80]Safwan S Halabi, Luciano M Prevedello, Jayashree Kalpathy-Cramer, Artem B Mamonov, Alexander Bilbily, Mark Cicero, Ian Pan, Lucas Araújo Pereira, Rafael Teixeira Sousa, Nitamar Abdala, Felipe Campos Kitamura, Hans H Thodberg, Leon Chen, George Shih, Katherine Andriole, et al. The RSNA pediatric bone age machine learning challenge. Radiology, 290(2):498–503, 2019. doi: 10.1148/radiol.2018180736.DOI
- [81]Pingjun Chen, Linlin Gao, Xiaoshuang Shi, Kyle Allen, and Lin Yang. Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss. Computerized Medical Imaging and Graphics, 75:84–92, 2019. doi: 10.1016/j.compmedimag.2019.06.002.DOI
- [82]Pingjun Chen. Knee osteoarthritis severity grading dataset, 2018. Data set.DOI
- [83]Tyler J Richards, Adam E Flanders, Errol Colak, Luciano M Prevedello, Robyn L Ball, Felipe Kitamura, John Mongan, Maryam Vazirabad, Hui-Ming Lin, Anne Kendall, Thanat Kanthawang, Salita Angkurawaranon, Erme Altınmakas, Hakan Dogan, Paulo Eduardo de Aguiar Kuriki, et al. The RSNA Lumbar Degenerative Imaging Spine Classification (LumbarDISC) dataset. Radiol Artif Intell, 8(2):e250480, 2026. doi: 10.1148/ryai.250480.DOI
- [84]Adam E Flanders, Luciano M Prevedello, George Shih, Safwan S Halabi, Jayashree Kalpathy-Cramer, Robyn Ball, John T Mongan, Anouk Stein, Felipe C Kitamura, Matthew P Lungren, Gagandeep Choudhary, Lesley Cala, Luiz Coelho, Monique Mogensen, Fanny Morón, et al. Construction of a machine learning dataset through collaboration: the RSNA 2019 Brain CT Hemorrhage Challenge. Radiol Artif Intell, 2(3):e190211, 2020. doi: 10.1148/ryai.2020190211.DOI
- [85]Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection 2018: a challenge hosted by the International Skin Imaging Collaboration (ISIC), 2019. arXiv:1902.03368.arxiv.org/abs/1902.03368
- [86]Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci Data, 5:180161, 2018. doi: 10.1038/sdata.2018.161.DOI
- [87]Noel C F Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection: a challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hosted by the International Skin Imaging Collaboration (ISIC). In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 168–172, 2018. doi: 10.1109/ISBI.2018.8363547. arXiv:1710.05006.DOI
- [88]Carlos Hernández-Pérez, Marc Comas, Sebastian Podlipnik, Noel C F Codella, Veronica Rotemberg, Allan C Halpern, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Brian Helba, Susana Puig, Veronica Vilaplana, and Josep Malvehy. BCN20000: dermoscopic lesions in the wild. Sci Data, 11(1):641, 2024. doi: 10.1038/s41597-024-03387-w.
- [89]Veronica Rotemberg, Nicholas Kurtansky, Brigid Betz-Stablein, Liam Caffery, Emmanouil Chousakos, Noel Codella, Marc Combalía, Stephen Dusza, Pascale Gussetra, David Gutman, Allan Halpern, Brian Helba, Harald Kittler, Kivanc Kose, Steve Langer, et al. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Sci Data, 8(1):34, 2021. doi: 10.1038/s41597-021-00815-z.DOI
- [90]Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Deshmukh, Vivek Sahasrabuddhe, and Fabrice Meriaudeau. Indian Diabetic Retinopathy Image Dataset (IDRiD): a database for diabetic retinopathy screening research. Data, 3(3):25, 2018. doi: 10.3390/data3030025.DOI
- [91]Ruhan Liu, Xiangning Wang, Qiang Wu, Ling Dai, Xi Fang, Tao Yan, Jaemin Son, Shiqi Tang, Jiang Li, Zijian Gao, Adrian Galdran, J M Poorneshwaran, Hao Liu, Jie Wang, Yerui Chen, et al. DeepDRiD: diabetic retinopathy-grading and image quality estimation challenge. Patterns (N Y), 3(6):100512, 2022. doi: 10.1016/j.patter.2022.100512.DOI
- [92]Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, Lina Jansen, Constantino Carlos Reyes-Aldasoro, Inka Zörníg, Dirk Jäger, Hermann Brenner, et al. Predicting survival from colorectal cancer histology slides using deep learning: a retrospective multicenter study. PLoS Med, 16(1):e1002730, 2019. doi: 10.1371/journal.pmed.1002730.DOI
- [93]Andrea Acevedo, Anna Merino, Santiago Alférez, Ángel Molina, Laura Boldú, and José Rodelar. A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data Brief, 30:105474, 2020. doi: 10.1016/j.dib.2020.105474.DOI
- [94]Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojciech Samek, and Tobias Schaeffter. PTB-XL, a large publicly available electrocardiography dataset. Sci Data, 7(1):154, 2020. doi: 10.1038/s41597-020-0495-6.DOI
- [95]Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Sci Data, 5:180251, 2018. doi: 10.1038/sdata.2018.251.
- [96]Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. SLAKE: a semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654, 2021. doi: 10.1109/ISBI48211.2021.9434010. arXiv:2102.09542.DOI
- [97]Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Development of a large-scale medical visual question-answering dataset. Commun Med (Lond), 4(1):277, 2024. doi: 10.1038/s43856-024-00709-z.DOI
- [98]Jeffrey D Rudie, Hui-Ming Lin, Robyn L Ball, Sabeena Jalal, Luciano M Prevedello, Savvas Nicolaou, Brett S Marinelli, Adam E Flanders, Kirti Magudia, George Shih, Melissa A Davis, John Mongan, Peter D Chang, Ferco H Berger, Sebastiaan Hermans, et al. The RSNA Abdominal Traumatic Injury CT (RATIC) dataset. Radiol Artif Intell, 6(6):e240101, 2024. doi: 10.1148/ryai.240101.DOI
- [99]Natalia Alves, Megan Schuurmans, Dawid Rutkowski, Anindo Saha, Pierpaolo Vendittelli, Nancy Obuchowski, Marjolein H Liedenbaum, Ingrid S Haldorsen, Anders Molven, Derya Yakar, Jeroen Geerdink, Sebastiaan van Koeverden, Deniece M Riviere, Wulphert Venderink, Robbert de Haas, et al. Artificial intelligence and radiologists in pancreatic cancer detection using standard of care CT scans (PANORAMA): an international, paired, non-inferiority, confirmatory, observational study. Lancet Oncol, 27(1):116–124, 2026. doi: 10.1016/S1470-2045(25)00567-4.DOI
- [100]Liang Jin, Jiancheng Yang, Kaiming Kuang, Bingbing Ni, Yiyi Gao, Yingli Sun, Pan Gao, Weiling Ma, Mingyu Tan, Hui Kang, Jiajun Chen, and Ming Li. Deep-learning-assisted detection and segmentation of rib fractures from CT scans: development and validation of FracNet. EBioMedicine, 62:103106, 2020. doi: 10.1016/j.ebiom.2020.103106.DOI