betaFeedback

Reading v1View latest version

WorksComputer Science

The Value of System 1 Models in Medicine

Abstract

Many clinical decisions are a choice among a few fixed options, taken quickly and in large numbers. Such decisions also need an uncertainty estimate to know when to defer to a clinician. Generative language models answer slowly in free text and state their confidence poorly. In contrast, System 1 models answer zero-shot with a probability for each answer. However, their performance across clinical tasks is unknown. Four System 1 models, two proprietary and two with open weights, were evaluated zero-shot on 86 public tasks from 14 clinical domains in 20 languages, spanning examination questions and medical imaging. Tasks were assigned to six capability tiers before analysis. The primary outcome was the area under the receiver-operating-characteristic curve (AUROC). Secondary outcomes included calibration, language effects and comparisons with published results and generative models. Across 179,111 cases, the mean AUROC of the best model per task was 0.94 for medical knowledge, 0.93 for text understanding, 0.83 for clinical judgement from text, 0.89 for image perception, 0.84 for 2D image findings and 0.69 for volumetric findings. Imaging tasks scored 0.09 (95% CI, 0.06 to 0.11) lower than text tasks (P < 0.001). The best model matched or exceeded the best published result on 13 of 26 text tasks and on 8 of 35 imaging tasks. Triage nurses outperformed the best model (accuracy, 0.85 vs. 0.56). One decision took 0.34 to 0.44 seconds and 24 to 81 joules on open-weight models run locally. Accuracy on identical questions fell by up to 39 percentage points from English to Yoruba. Generative models of the same family were 11 to 76 times slower, with 3 to 5 percentage points higher balanced accuracy. System 1 models decide quickly, cheaply and with usable confidence when the answer is stated in a text or recalled from medical knowledge, but not when it must be judged or read from an image.

Introduction

Many clinical decisions represent choices among a few fixed options that must be made quickly and in bulk. A triage nurse assigns one of five acuity levels within minutes of arrival, a radiologist decides whether a CT pulmonary angiogram demonstrates embolism, and a coder selects one procedure code for each of thousands of cases. A model that makes such decisions must also state how certain it is, so that confident answers can be acted on and uncertain cases passed to a clinician. [1, 2, 3] Generative large language models perform well on a series of medical tasks, [4, 5] but they write each answer token by token, often following a reasoning trace, which takes several seconds per decision and uses far more energy than a task-specific model. [6] The confidence they state in text is poorly calibrated and barely separates correct from incorrect answers. [7, 8] On the contrary, System 1 models receive a state, such as a report, an image or both, together with one or more typed questions, and return a probability for every answer option in a single forward pass without generating text. Their name refers to dual-process theory, which separates fast, intuitive System 1 judgement from slow, deliberate System 2 reasoning, the mode that generative models resemble when they reason in text. [9, 10]

The open question is which clinical decisions such a model can be trusted with. The answer to a clinical case can be stated in a report, recalled from medical knowledge, judged from several findings, or seen in an image. Of 519 studies of large language models in health care, 44.5% tested medical knowledge such as licensing-1 examination questions, and 5% used real patient-care data. [11] Where other tasks were tested, performance fell from more than 86% on licensing-examination questions [12] to lower accuracy on real clinical cases and conversations. [13, 14] On radiologic images, GPT-4V identified the imaging modality in all 515 images but detected abnormalities in 0% to 90%, depending on the finding, [15] and multimodal models scored lower on volumetric than on two-dimensional images. [16] If performance depends on where the answer lies, a single benchmark score misrepresents a model, a construct-validity problem known from general AI benchmarks. [17]

To evaluate the value of System 1 models in medicine, we tested four models on 86 tasks from 14 clinical domains, with the tasks ordered in six capability tiers from medical knowledge to volumetric findings. We measured which clinical decisions the models can take on, whether their probabilities are calibrated, how they perform outside English, and what their speed costs in time, money, energy and accuracy, against a generative model of the same family.

Methods

Study Design and Tasks. This cross-sectional evaluation used public, de-identified datasets under their respective licences. Candidate tasks came from a taxonomy of 87 clinical decision tasks in 18 domains and from further public datasets with a fixed answer format (Supplementary Appendix, Section S1). A task was included when its data and reference labels were public and its licence allowed transfer to a model hosted on a commercial API. Eighty-six tasks from 14 clinical domains met these criteria. Table S1 lists every task with the publication of its dataset. Eight of them are language versions of another task and were analysed only for language effects, which left 78 primary tasks (Figure 1). Each task used the official test split or a seeded random sample of up to 1,000 cases, in some tasks enriched for rare classes, and volumetric tasks used 81 to 436 studies. The tasks comprised 179,111 cases, with the reference standard released by the dataset authors.

Study design showing System 1 model inputs, typed questions, model architecture, probability per option, comparator generative model, and task categories

Figure 1. Study Design. Panel A shows how a System 1 model answers. Text, one or more images, or both enters the model with typed questions, and the model returns a probability for each option in one forward pass. The generative model of the same family served as comparator on 32 tasks. Panel B shows the 86 tasks by capability tier and input type. Hatched tiles are language versions of another task and were analysed for language effects only.

Prior to result summarisation, each task was assigned to one of six capability tiers. Medical knowledge covered examination questions and terminology, text understanding covered information stated in a report, abstract or sentence, and clinical judgement from text covered decisions that integrate patient information, such as triage acuity or variant class. The three imaging tiers included image perception (acquisition, anatomy and image quality), two-dimensional (2D) image findings, and volumetric findings on CT or MRI.

System 1 Models. Four System 1 models were evaluated through the OpenRouter decision interface, each in the latest version available in October 2026. Clef (Cloudflare, 27 billion parameters) and Clef-flash (Cloudflare, 9.5 billion parameters) have open weights, and GPT-6 Luna Decisions (OpenAI) and Jev 1.14 (TypeSafe AI) are proprietary. A request held a state of text and up to four images, and one or more typed questions with a yes-or-no answer, a choice of one of nn options, or an ordered score. The model returned a probability for every option. All questions of a case were sent in one request, zero-shot, with a one-sentence description of the input. Volumetric studies were read as axial slices at a fixed spacing, with one request per slice. The study-level probability was the maximum over slices for yes-or-no questions and the mean over slices for other questions. On four 2D imaging tasks, Jev 1.14 received higher-resolution versions of the same images since its interface accepts larger images. Every request was logged as sent.

Generative Comparison. Clef and Clef-flash were compared with the generative backbones, Qwen3.8-27B [18] and Qwen3.5-9B, [19] and GPT-6 Luna Decisions with the generative GPT-6 Luna. The comparison covered 32 text and 2D imaging tasks. For each task, 100 randomly selected cases were sent to the generative model with the same state and questions as the System 1 model had received, including an instruction to choose one option per question, and an output schema restricted to the answer options. Default reasoning settings were kept. An answer that failed or could not be parsed counted as wrong.

Local Runs and Energy. Clef and Clef-flash were additionally run locally from the released checkpoints, on a single NVIDIA RTX 5090 GPU. Local Clef’s answers were compared with the hosted versions, which confirmed that the two agreed before the local runs were utilised. Energy was measured on a sub-sample of 42 tasks and it followed the AI Research Label of the CEDARS framework for reporting the environmental footprint of clinical AI. [20] The measurements come from the GPU driver’s cumulative energy counter through NVML (power usage effectiveness taken as 1.0 for a single workstation GPU). CodeCarbon (v3.3.1) ran alongside to convert energy to CO2_{2}e at the local grid carbon intensity (Greece, 337 g CO2_{2}e per kWh). Energy is reported per decision and summarised as the median and IQR over tasks. Latency was measured per request. The providers of the hosted models (GPT-6 Luna Decisions, Jev 1.14) publish no per-request energy figures for them.

Reference Results. For every task, published results on the same dataset were collected from the original dataset or challenge publication, from official leaderboards, and from a search of PubMed and the web for later work up to October 2026. Every value was checked against its source and classified as a task-specific model, a generative language model, a rule-based classifier, or clinicians and expert annotators. A published result was compared only when it used the same label set and metric on the matching questions. The highest such result per task is reported as the best published result.

Outcomes

The primary outcome was the area under the receiver-operating-characteristic curve (AUROC), computed per question as a macro average of one-versus-rest curves for more than two options, and averaged over the questions of a task. Secondary outcomes were the balanced accuracy of the most probable option, expressed relative to chance as (BA−1/k)/(1−1/k)(\mathrm{BA}-1/k)/(1-1/k) for kk classes so that 0 is chance and 1 is perfect, the top-label expected calibration error (ECE), [1, 21] selective accuracy after withholding the least confident answers, [22] cost per 1,000 requests at provider prices, latency per request and energy per decision.

Statistical Analysis. Any question that a model refused or failed was scored as wrong, with the label prevalence as its probability. Task-level 95% confidence intervals (CIs) were estimated using 1,000 bootstrap resamples of cases. Task estimates were pooled within each tier and model using DerSimonian–Laird random-effects models. Two contrasts were specified in advance, imaging versus text tiers and a linear trend across the three imaging tiers, and both were estimated by random-effects meta-regression. AUROC was also summarised by question type. The accuracy of each language version on identical items was compared with English by paired bootstrap and the exact McNemar test. System 1 and generative models were compared by the mean task-level difference in balanced accuracy, with 95% CIs from a task bootstrap. Secondary analyses were not adjusted for multiplicity. Analyses used Python 3.13, pandas 3.0, NumPy 2.5, SciPy 1.18 and scikit-learn 1.9. Code and all model outputs are available in the project repository at https://github.com/twin-x-eu/system1-models-in-medicine.

Results

Tasks and Models. The 86 tasks covered 20 languages and yielded 2.4 million model answers across 179,111 cases (Figure 1). GPT-6 Luna Decisions refused 4,008 case–question pairs (1.3%), Clef and Clef-flash each failed on one pair, and Jev 1.14 answered every question. Refused and failed questions were scored as wrong. Jev 1.14 was the best model on 45 of 78 tasks. Results per task are given in Table S2.

No.TaskClefClef-flashGPT-6 Luna DecisionsJev 1.14
001MedQA (USMLE)0.97 (0.96–0.97)0.92 (0.90–0.93)0.98 (0.97–0.98)0.99 (0.98–0.99)
002MedMCQA (India)0.89 (0.89–0.90)0.87 (0.86–0.88)0.92 (0.92–0.93)0.93 (0.93–0.94)
003HEAD-QA (English)0.96 (0.95–0.96)0.95 (0.94–0.95)0.96 (0.96–0.97)0.97 (0.96–0.97)
004HEAD-QA (Spanish)0.96 (0.95–0.97)0.95 (0.94–0.95)0.97 (0.96–0.97)0.97 (0.97–0.98)
005MMLU medical0.99 (0.98–0.99)0.99 (0.98–0.99)0.98 (0.98–0.99)0.99 (0.98–0.99)
006MMLU-Pro health0.97 (0.96–0.97)0.96 (0.95–0.97)0.96 (0.94–0.96)0.97 (0.96–0.98)
007MedXpertQA text0.72 (0.71–0.73)0.63 (0.62–0.64)0.76 (0.74–0.77)0.77 (0.76–0.78)
008Medbullets0.92 (0.90–0.94)0.83 (0.80–0.86)0.94 (0.92–0.96)0.94 (0.93–0.96)
009AfriMed-QA0.94 (0.94–0.95)0.92 (0.91–0.93)0.95 (0.95–0.96)0.95 (0.95–0.96)
010MedExpQA0.96 (0.95–0.97)0.92 (0.91–0.94)0.96 (0.94–0.97)0.96 (0.95–0.98)
011KorMedMCQA0.96 (0.96–0.97)0.93 (0.92–0.93)0.99 (0.99–0.99)0.98 (0.98–0.98)
012CMExam (China)0.97 (0.97–0.97)0.95 (0.95–0.96)0.97 (0.97–0.98)0.98 (0.98–0.99)
013MedQA Mainland0.98 (0.98–0.98)0.96 (0.96–0.97)0.98 (0.97–0.98)0.98 (0.98–0.99)
014MedQA Taiwan0.97 (0.96–0.98)0.95 (0.95–0.96)0.98 (0.98–0.99)0.98 (0.98–0.99)
015FrenchMedMCQA0.99 (0.98–0.99)0.97 (0.95–0.98)0.99 (0.98–1.00)0.99 (0.99–1.00)
016MMLU medical (Arabic)0.95 (0.94–0.96)0.93 (0.92–0.94)0.98 (0.97–0.98)0.98 (0.98–0.99)
017MMLU medical (Hindi)0.92 (0.91–0.93)0.84 (0.83–0.86)0.98 (0.97–0.98)0.98 (0.97–0.99)
018MMLU medical (Japanese)0.97 (0.96–0.97)0.96 (0.95–0.97)0.98 (0.97–0.98)0.98 (0.98–0.99)
019MMLU medical (Swahili)0.90 (0.89–0.92)0.85 (0.83–0.86)0.96 (0.95–0.96)0.96 (0.95–0.97)
020MMLU medical (Yoruba)0.85 (0.83–0.86)0.79 (0.77–0.81)0.91 (0.90–0.93)0.93 (0.92–0.94)
021MMLU medical (German)0.98 (0.97–0.98)0.97 (0.96–0.98)0.98 (0.97–0.99)0.99 (0.98–0.99)
022ATC drug class0.53 (0.52–0.54)0.61 (0.60–0.62)0.86 (0.86–0.87)0.73 (0.73–0.74)
023ICD-9 procedure0.65 (0.64–0.65)0.64 (0.63–0.65)0.81 (0.80–0.81)0.72 (0.71–0.72)
024CXR report findings1.00 (0.99–1.00)1.00 (0.99–1.00)0.99 (0.99–0.99)1.00 (1.00–1.00)
025Chest CT report findings0.99 (0.98–0.99)0.99 (0.98–0.99)0.96 (0.96–0.97)0.99 (0.99–0.99)
026Multilingual report coding0.96 (0.96–0.97)0.94 (0.93–0.94)0.94 (0.94–0.95)0.96 (0.96–0.97)
027Lung cancer T from report0.94 (0.92–0.96)0.92 (0.90–0.94)0.92 (0.89–0.95)0.95 (0.93–0.97)
028BI-RADS from report0.88 (0.87–0.90)0.84 (0.82–0.86)0.86 (0.82–0.89)0.81 (0.77–0.84)
029PT stage from pathology report0.96 (0.95–0.97)0.95 (0.94–0.96)0.96 (0.94–0.97)0.96 (0.95–0.97)
030Trial report inference0.90 (0.88–0.93)0.87 (0.84–0.90)0.86 (0.83–0.90)0.92 (0.90–0.95)
031PubMedQA0.81 (0.78–0.84)0.83 (0.79–0.86)0.79 (0.76–0.83)0.84 (0.81–0.87)
032Drug-drug interaction0.89 (0.88–0.91)0.90 (0.89–0.91)0.87 (0.85–0.89)0.93 (0.92–0.94)
033Hallmarks of cancer0.94 (0.93–0.95)0.94 (0.93–0.95)0.89 (0.87–0.91)0.94 (0.93–0.94)
034Public-health claims0.85 (0.83–0.87)0.81 (0.78–0.83)0.81 (0.79–0.84)0.82 (0.80–0.84)
035Question entailment0.88 (0.84–0.92)0.88 (0.84–0.92)0.90 (0.86–0.94)0.92 (0.88–0.95)
036Rubric judging0.83 (0.79–0.86)0.73 (0.69–0.77)0.78 (0.75–0.82)0.84 (0.81–0.88)
037ED triage (KTAS)0.80 (0.78–0.81)0.75 (0.73–0.77)0.82 (0.80–0.84)0.80 (0.78–0.82)
038Diagnosis from case0.83 (0.81–0.85)0.74 (0.71–0.76)0.85 (0.83–0.87)0.82 (0.80–0.84)
039ACMG variant class0.88 (0.87–0.89)0.76 (0.74–0.78)0.96 (0.95–0.96)0.93 (0.92–0.94)
040Trial eligibility0.85 (0.83–0.88)0.85 (0.82–0.88)0.83 (0.80–0.86)0.87 (0.84–0.90)
041Suicide-risk level0.77 (0.75–0.80)0.75 (0.73–0.78)0.81 (0.78–0.83)0.82 (0.79–0.84)
042MRI safety of a device0.58 (0.56–0.61)0.51 (0.49–0.54)0.51 (0.49–0.53)0.66 (0.64–0.68)
043Radiograph body part0.95 (0.94–0.96)0.96 (0.96–0.97)0.96 (0.95–0.96)0.95 (0.95–0.96)
044CXR view AP/PA0.98 (0.97–0.99)0.99 (0.98–0.99)0.89 (0.86–0.91)0.99 (0.99–0.99)
045Imaging modality1.00 (1.00–1.00)1.00 (1.00–1.00)0.99 (0.99–1.00)1.00 (1.00–1.00)

Table S2. AUROC per task and model with 95% CI from 1,000 bootstrap resamples of cases. Refused or failed questions were scored as wrong with the label prevalence as probability.

Capability Gradient. For the best model on each task, the random-effects mean AUROC was 0.94 (95% CI, 0.92 to 0.97) for medical knowledge, 0.93 (95% CI, 0.91 to 0.94) for text understanding, 0.89 (95% CI, 0.84 to 0.94) for image perception, 0.84 (95% CI, 0.81 to 0.87) for 2D image findings and 0.69 (95% CI, 0.60 to 0.79) for volumetric findings (Figure 2). Imaging tasks scored 0.09 (95% CI, 0.06 to 0.11) lower than text tasks (P<0.001P < 0.001), and AUROC fell by 0.09 (95% CI, 0.06 to 0.13) per tier from image perception to volumetric findings (P<0.001P < 0.001). Both contrasts were significant for each of the four models (Figure 2B). Clinical judgement from text reached 0.83 (95% CI, 0.73 to 0.93), lower than the other two text tiers. Chance-corrected balanced accuracy fell from 0.74 for medical knowledge to 0.20 for volumetric findings, and its trend across the imaging tiers was not significant (P=0.10P = 0.10, Figure 2C).

Capability Map

Figure 2. Capability Map. Panel A shows the AUROC of each primary task (columns) for the four System 1 models, with tasks grouped by capability tier and sorted by the best model. Grey vertical lines span the four models. Horizontal bars and shaded bands show the random-effects mean AUROC of the best model per task with its 95% CI, and the weakest task of each tier is labelled. Panels B and C show the random-effects mean per tier and model with 95% CIs for AUROC and for chance-corrected balanced accuracy, where 0 is chance and 1 is perfect. Refused or failed questions were scored as wrong.

The best model reached an AUROC of 0.99 on licensing-examination questions (MedQA) [23] and 1.00 on findings stated in chest radiograph reports. On volumetric imaging it reached 0.54 for pulmonary embolism, [24] 0.59 for cervical spine fracture, [25] 0.59 for clinically significant prostate cancer [26] and 0.48 for rib fracture type, and four of nine volumetric tasks stayed below 0.60. Intracranial haemorrhage on head CT reached 0.90. [27] On imaging tasks, questions that assigned the whole image to one of several options reached a median AUROC of 0.87, yes-or-no questions on the presence of a finding reached 0.74, and ordered grades 0.71. On text tasks, yes-or-no questions reached 0.99.

Comparison with Published Results and Clinicians. A published result on the same dataset, label set and metric was available for 61 tasks (Figure 3). On text tasks, the best System 1 model matched (within 0.01) or exceeded the best published result in 13 of 26 tasks, with a median difference of −0.01. On imaging tasks it did so in 8 of 35 tasks, with a median difference of −0.17. Clinicians or expert annotators outperformed the best model in 6 of the 10 tasks for which such a comparison existed. Triage nurses assigned the emergency acuity level with an accuracy of 0.85, compared with 0.56 for the best model, [28] and a points-based rule classified genetic variants with an accuracy of 0.95, compared with 0.73. [29, 30] Zero-shot System 1 models are therefore

System 1 Models, Published Results and Clinicians

Figure 3. System 1 Models, Published Results and Clinicians. Panel A shows text tasks and Panel B imaging tasks. For each task with a published result on the same dataset, label set and metric, the best of the four System 1 models (coloured marker) is shown with the best published result (open circle), which can be a task-specific model, a generative language model or a rule, and with clinicians or expert annotators (open triangle). Values of the System 1 models are computed on the questions that match the published result and can differ from the task-level AUROC. Acc denotes accuracy and BA balanced accuracy.

competitive with purpose-built systems on text, but not on images or where a codified rule already exists.

Calibration and Selective Answering. The median top-label ECE across models was 0.05 to 0.07 on text tasks and 0.11 to 0.17 on imaging tasks (Figure 4A and Figure 4B). On imaging tasks, GPT-6 Luna Decisions was over-confident, and about 80% of its most confident answers were correct. Withholding the less confident half of the answers raised accuracy on text tasks from between 0.76 and 0.83 to between 0.96 and 0.99, and on imaging tasks from between 0.66 and 0.70 to between 0.79 and 0.91 (Figure 4C).

Calibration and Language

Figure 4. Calibration and Language. Panels A and B show the share of correct answers against the probability the model gave its answer, pooled over text tasks in Panel A and over imaging tasks in Panel B. The diagonal marks perfect calibration. Panel C shows, for each model, the accuracy of all answers (open marker) and of its most confident half (filled marker) on text and imaging tasks. Panel D shows accuracy on the same 1,089 medical examination questions in seven languages.

Language. On the same 1,089 medical examination questions, accuracy in every language version was lower than in English for every model (Figure 4D). The decline was smallest in German and largest in Yoruba, where it ranged from 20.1 (95% CI, 17.6 to 22.5) percentage points for Jev 1.14 to 39.4 (95% CI, 36.3 to 42.4) for Clef-flash, whose accuracy fell from 0.93 to 0.54. The performance measured in English therefore overstates what these models deliver in a low-resource language.

System 1 and Generative Models. On identical cases, the balanced accuracy of each System 1 model was lower than that of the generative model of its family, by 4.3 (95% CI, 2.4 to 6.3) percentage points for GPT-6 Luna Decisions, 3.3 (95% CI, 0.0 to 6.5) for Clef and 4.8 (95% CI, 1.0 to 8.5) for Clef-flash, over 32 tasks each (Figure 5A). The difference was smaller on imaging tasks, where Clef-flash and Qwen3.5-9B differed by 0.4 (95% CI, −1.5 to 2.3) percentage points. Each System 1 model scored higher than its generative counterpart on 8 or 9 of the 32 tasks. Generative models required a median of 4.8 to 30.3 seconds per request, compared with 0.34 to 0.44 seconds for System 1 models, and incurred 2- to 22-fold higher costs (Figure 5B and Figure 5C).

System 1 and Generative Models

Figure 5. System 1 and Generative Models. Panel A shows, for each task, the balanced accuracy of the System 1 model minus that of the generative model of the same family on identical cases, with the mean and 95% CI (task bootstrap) for text and imaging tasks. The pairs are GPT-6 Luna Decisions and GPT-6 Luna, Clef and Qwen3.8-27B, and Clef-flash and Qwen3.5-9B. Panels B and C show the cost per 1,000 requests and the median latency per request on logarithmic scales, with filled markers for System 1 models and open markers for generative models.

Environmental Impact of System 1 Models. Energy was measured directly on the two open-weight models, Clef and Clef-flash, both run locally on one RTX 5090 GPU over 42 of the tasks and 117,263 decisions each (Methods). The median energy of one decision was 23.5 joules for Clef-flash (IQR, 17.2 to 51.3) and 80.9 joules for Clef (IQR, 68.4 to 125.5). That is 6.5 to 22.5 watt-hours of GPU-board energy per 1,000 decisions and, including CPU and RAM and at the carbon intensity of the local grid, 2.8 to 12.1 g CO2_2e per 1,000 decisions (Figure 6). Across capability tiers, the median energy per 1,000 decisions was lowest for medical knowledge (4.8 watt-hours for Clef-flash and 19.2 for Clef) and highest for text understanding (19.3 and 47.6). One kilowatt-hour buys roughly 153,000 decisions from Clef-flash and 4,500 from Clef. Decisions that included an image cost 1.6 to 2.3 times as much energy as text-only decisions, and the length of the input mattered more than its type. On the three tasks with the longest documents, the energy of a decision rose to 2.1 to 5.0 times the median for that model. The two models together took 234,526 decisions for 4.3 kilowatt-hours of GPU-board energy. The energy of a single System 1 decision is therefore small enough that it is not the limiting factor in deploying one at scale.

Energy of Locally Run System 1 Models

Figure 6. Energy of Locally Run System 1 Models. Panel A shows the GPU energy per 1,000 requests and Panel B the GPU time per request of Clef (8-bit) and Clef-flash (bf16), run locally on one NVIDIA RTX 5090, for each primary task by capability tier. Energy is the whole-board GPU energy. Bars show the median over tasks, and nn is the number of tasks per tier. No clinical-judgement task was run locally.

Local versus API Inference. Clef and Clef-flash have open weights and can run inside a hospital network, so patient data do not need to leave the institution. Run locally, they reproduced the hosted interface closely. On chest radiograph report findings, local Clef gave a mean AUROC of 0.99 equal to the version inferenced through the API, and agreed with it on 99.7% of answers, while on visual question answering, the values were 0.947 against 0.942 and 97.2% agreement.

Discussion

System 1 models answered medical examination questions and identified findings described in radiology reports with high accuracy, with inference times below half a second and for a few joules per decision. On text-based tasks, predicted probabilities were well calibrated, closely reflecting observed accuracy. However, performance declined on tasks requiring the integration of multiple findings, such as assigning acuity levels in emergency triage. On imaging tasks, they accurately identified imaging modalities and anatomical regions but frequently failed to detect specific abnormalities. Performance on volumetric CT and MRI tasks was particularly limited, approaching chance levels in four of nine tasks.

For information extraction from clinical text, the case for zero-shot System 1 models is strong. The best System 1 model matched or exceeded the best published result on 13 of 26 text tasks without any training on the task, including results from models that had been fine-tuned for that task. For the T staging of lung cancer from CT reports, the best System 1 model reached an accuracy of 0.78, compared with 0.67 for fine-tuned Japanese BERT models.[31] For the pathological T stage from pathology reports, it reached an AUROC of 0.96, compared with 0.94 for a fine-tuned Clinical-BigBird model.[32] A zero-shot System 1 model could therefore replace models that require task-specific fine-tuning. Each decision took less than half a second and 23.5 to 80.9 joules, with costs ranging from 0.02 to 0.16 USD per 1,000 requests. Withholding the 50% of predictions with the lowest confidence increased accuracy on text-based tasks to 0.96 to 0.99. These findings suggest that System 1 models could support large-scale report labelling, information extraction, and literature screening, while referring uncertain cases to clinicians or slower, more computationally intensive models.[3, 22] Clef and Clef-flash have openly available weights and can be deployed within hospital networks.

Accuracy declined on tasks requiring clinical judgement, such as assigning triage acuity or classifying variants, even when all relevant information was provided in the input. System 1 models generate predictions in a single forward pass, relying on knowledge encoded in their weights. Unlike generative models that produce explicit reasoning traces,[33] they lack intermediate reasoning steps through which multiple findings can be evaluated and integrated before reaching a decision. Across the five tasks requiring clinical judgement, the generative backbones of Clef and Clef-flash achieved accuracies 11 and 15 percentage points higher, respectively, compared with differences of 3 and 5 percentage points across all 32 evaluated tasks. Enabling System 1 models to generate reasoning traces before making predictions could help narrow this performance gap.

An additional limitation of most medical benchmarks is their predominantly English-language content. Similarly, most language models, including System 1 models, are trained primarily on English-language data.[34, 35] Reflecting this, the System 1 models were most accurate in English. On the same 1,089 examination questions, accuracy fell by 2 to 7 percentage points in German, a high-resource language, and by 20 to 39 percentage points in Yoruba, a low-resource language. Generative models lose accuracy in low-resource languages in a similar manner. [35, 36]

Most System 1 models also accept images, but read medical images far less accurately than models trained for the task. The best System 1 model matched or exceeded the best published result in 8 of 35 imaging tasks, compared with 13 of 26 text tasks, with a median difference of −0.17 on imaging tasks. For clinically significant prostate cancer on MRI, it reached an AUROC of 0.59, compared with 0.86 for radiologists. [26] Calibration was also worse on imaging tasks, and only about 80% of the most confident answers of GPT-6 Luna Decisions were correct. Volumetric imaging was most challenging as the current interface of System 1 models accepts at most four images per request and no volumes. CT and MRI studies were therefore sent as single slices every 3 to 10 mm, and a small finding can lie between two slices. The choice of pooling rule over slices did not change the results (Supplementary Appendix, Section S6.3). Furthermore, radiographic images had to be downsampled, for example mammograms at 1,024 pixels on the long side, at which microcalcifications cover few pixels. Generative models also lose accuracy when volumes are given as image sequences, [16] which indicates a property of how vision-language models process images in general.

The open-weight decision models Clef and Clef-flash are based on the Qwen family of generative models, allowing a direct comparison of the two model types. The decision models and their backbones did not differ in accuracy, however the decision models were better calibrated, with a median expected calibration error of 0.12 for Clef-flash against 0.20 for its backbone, and 0.07 for Clef against 0.13. Against the generative backbone with reasoning, the System 1 models were 3 to 5 percentage points less accurate and 11 to 76 times faster.

This study has limitations. Models were used zero-shot with a one-sentence description of the input. Task-specific prompts and fine-tuning were not tested. A System 1 model trained on the records of 3.6 million patients from one health system, CS1-4B, was recently reported to outperform open decision models of up to 27 billion parameters on 13 of 14 tasks in held-out patients from that system.[37] Its weights have not yet been released, so it could not be included here. Public datasets can be part of the training data, which raises results on text tasks in particular. Larger frontier generative models were not tested and are likely more accurate on many tasks, but they are slower and more expensive than the generative models compared herein.

For predefined clinical decisions on text, in particular information extraction, System 1 models are ready for use, with fast, inexpensive and well-calibrated answers. They are as accurate as their backbones but better calibrated, and open weights let a hospital run them sustainably on its own data. However, they are currently limited in questions requiring clinical reasoning and in interpreting medical images, in particular CT and MRI volumes.

Acknowledgements

Funded by the European Union under Horizon Europe grant agreement No 101287679 (TWIN-X). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Health and Digital Executive Agency (HaDEA). Neither the European Union nor the granting authority can be held responsible for them.

Supplementary Appendix

The Value of System 1 Models in Medicine

Skip printed tables of contents, lists of figures and lists of tables entirely: neither their "Contents" heading nor their entries belong in @body

Task Selection

The taxonomy splits radiology into five layers (acquisition and metadata, image quality, anatomy, findings, and examination or patient level). Table S1 lists the evaluated tasks.

Requests

The state contained the case text as released, such as the question stem, the report sections or the triage fields, or a one-sentence description of the input together with the image. Each option of a choice question had a short description, and each level of an ordered score was described in words. Images larger than 4 MiB were downsampled by half until they were below 4 MiB. For Jev 1.14, images with a side below 280 or above 1,568 pixels were rescaled. The four 2D imaging tasks with higher-resolution images for Jev 1.14 were melanoma in ISIC 2020, paediatric pneumonia, retinal OCT and diabetic retinopathy grade in RetinaMNIST. On OpenRouter, Clef and Clef-flash were routed only to the Cloudflare provider, because another provider ignored images. When a model refused a question, the question was removed, the request was repeated with the remaining questions, and the refusal was logged. Volumetric slices were taken within the relevant body region, with the window named in the text.

Scoring

For yes-or-no questions, the AUROC used the probability of the positive option. For other questions, the macro average included only the options present in the reference. Balanced accuracy was the recall of the most probable option, averaged over the reference classes. Probabilities were normalised to sum to one, because two models round probabilities to two decimals. The expected calibration error used 10 equal-width bins of the top-label probability. When a task shared its requests with another task (chest radiograph findings and view, head CT haemorrhage and critical findings), each question was counted in one task only.

Generative Comparison

The generative models replied in JSON. Qwen3.8-27B answered the first 50 of the 100 cases per task. Reasoning tokens were counted and billed.

Reference Results

Table S5 lists every published result with its type.

No.TaskTypeReferenceSourceMetricRef.Best S1
001MedQA (USMLE)Generative language modelo1-preview, zero-shotNori 2024accuracy0.960.89
001MedQA (USMLE)Task-specific modelMed-Gemini-L 1.0 (self-training, uncertainty-guided web search)Saab 2024accuracy0.910.89
002MedMCQA (India)Generative language modelo1-preview, zero-shotNori 2024accuracy0.840.75
002MedMCQA (India)Task-specific modelMed-PaLM 2 (ensemble refinement)Singhal 2025accuracy0.720.75
003HEAD-QA (English)Generative language modelGPT-5, zero-shotYang 2026accuracy0.900.90
005MMLU medicalGenerative language modelGPT-5Wang 2025accuracy0.960.96
006MMLU-Pro healthGenerative language modelGemini-3.1-ProWang 2024 (leaderboard)accuracy0.860.83
007MedXpertQA textCliniciansExpert (pre-licensed humans)Zuo 2025accuracy0.430.35
007MedXpertQA textGenerative language modelGPT-5Wang 2025accuracy0.560.35
008MedbulletsGenerative language modelGPT-5 (2025-08-07)Bedi 2025 (MedHELM)accuracy0.870.78
009AfriMed-QAGenerative language modelGPT-4o, zero-shotOlatunji 2025accuracy0.790.82
010MedExpQAGenerative language modelo4-mini, zero-shotYang 2026accuracy0.880.87
010MedExpQATask-specific modelMistral-7B (fine-tuned), no RAGAlonso 2024accuracy0.680.87
011KorMedMCQACliniciansExaminees average (licensing exam institute)Kweon 2024accuracy0.770.89
011KorMedMCQAGenerative language modelo1-preview, 5-shotKweon 2024accuracy0.930.89
012CMExam (China)CliniciansHuman exam takersLiu 2023accuracy0.720.88
012CMExam (China)Generative language modelQwen2.5-72B-InstructBaichuan 2025accuracy0.830.88
012CMExam (China)Task-specific modelChatGLM-CMExam (fine-tuned)Liu 2023accuracy0.450.88
013MedQA MainlandGenerative language modelGPT-4 (5-shot CoT)Qiu 2024accuracy0.810.90
013MedQA MainlandTask-specific modelMMed-Llama 3 8B (fine-tuned)Qiu 2024accuracy0.790.90
014MedQA TaiwanGenerative language modelGPT-4-base (pre-alignment), 5-shotNori 2023accuracy0.870.90
014MedQA TaiwanTask-specific modelBioBERT-large (machine-translated to English + retrieval)Jin 2021accuracy0.420.90
015FrenchMedMCQAGenerative language modelGPT-4, 3-shotWang 2024 (Apollo)accuracy0.900.93
022ATC drug classGenerative language modelGPT-4-0125-preview, 4-shotShoham 2024accuracy0.620.62
023ICD-9 procedureGenerative language modelGPT-4-0125-preview, 4-shotShoham 2024accuracy0.550.53
025Chest CT report findingsGenerative language modelGPT-4o (gpt-4o-2024-11-20), zero-shot, JSON outputYamagishi 2025accuracy0.970.96
027Lung cancer T from reportGenerative language modelGPT-3.5-turbo, zero-shot, full TNM definitions in promptMatsuo 2024accuracy0.570.78
027Lung cancer T from reportTask-specific modelNAIST3SOCR-2 (fine-tuned Japanese BERTs)Nakamura 2023accuracy0.670.78
029pT stage from pathology reportTask-specific modelBB-TEN (Clinical-BigBird, fine-tuned)Kefeli 2024AUROC0.940.96
031PubMedQACliniciansSingle human annotatorJin 2019accuracy0.780.78
031PubMedQAGenerative language modelGPT-4 + Medprompt (kNN few-shot, self-generated CoT, choice-shuffle ensemble)Nori 2023 (Medprompt)accuracy0.820.78
031PubMedQATask-specific modelMedGemma 27B (text, test-time scaling)Sellergren 2025accuracy0.770.78
035Question entailmentGenerative language modelgemma-4-31b-it, five-shotWu 2025 (BRIDGE)accuracy0.880.83
035Question entailmentTask-specific modelPANLP (ensemble, rank 1)Ben Abacha 2019accuracy0.750.83
037ED triage (KTAS)CliniciansTriage nurses (KTAS RN)Moon 2019 (in-data)accuracy0.850.56
037ED triage (KTAS)Generative language modelKATE (GPT-4o, 40 kNN shots)Lee 2026accuracy0.720.56
037ED triage (KTAS)Task-specific modelLlama 3.1 8B, fine-tunedLee 2026accuracy0.650.56
038Diagnosis from caseGenerative language modelo3Zhu 2026accuracy0.690.65
039ACMG variant classRuleACMG/AMP points system on panel evidence codesTavtigian 2020 (in-data)accuracy0.950.73
040Trial eligibilityGenerative language modelTrialGPT-Ranking (GPT-4)Jin 2024AUROC0.800.87
040Trial eligibilityTask-specific modelBioLinkBERT cross-encoder (MedNLI)Jin 2024AUROC0.620.87
041Suicide-risk levelGenerative language modelGPT-4, zero-shot (best prompt)Xu 2024 (Mental-LLM)balanced accuracy0.440.56

Table S5. Published results with the same dataset, label set and metric, which can use other splits, and the best of the four System 1 models on the questions that match the published result (Best S1). Best S1 can therefore differ from the task-level AUROC in Table S2.

Supplementary Results

All Tasks and Models

Figure S1 shows the AUROC of every task for every model. Table S2 gives the AUROC with 95% CIs, and Table S3 the random-effects means per tier.

AUROC heatmap of every task and model

Figure S1. AUROC of every task (rows, numbered as in Table S1) and model (columns). Blue shades mark values above chance (0.5) and red shades values below chance.

TierTasksMetricClefClef-flashGPT-6 Luna DecisionsJev 1.14Best
Medical knowledge16AUROC0.90 (0.83–0.96)0.88 (0.81–0.94)0.94 (0.91–0.96)0.93 (0.89–0.97)0.94 (0.92–0.97)
Chance-corrected BA0.65 (0.47–0.83)0.60 (0.44–0.75)0.73 (0.63–0.84)0.72 (0.57–0.87)0.74 (0.63–0.85)
Text understanding13AUROC0.92 (0.90–0.93)0.89 (0.87–0.92)0.89 (0.87–0.92)0.92 (0.90–0.94)0.93 (0.91–0.94)
Chance-corrected BA0.63 (0.54–0.73)0.56 (0.43–0.69)0.61 (0.51–0.71)0.65 (0.55–0.75)0.66 (0.57–0.75)
Clinical judgement from text6AUROC0.79 (0.70–0.87)0.73 (0.64–0.81)0.80 (0.66–0.94)0.82 (0.73–0.90)0.83 (0.73–0.93)
Chance-corrected BA0.34 (0.14–0.55)0.24 (0.10–0.38)0.41 (0.10–0.73)0.42 (0.12–0.72)0.44 (0.12–0.77)
Image perception12AUROC0.84 (0.77–0.92)0.85 (0.79–0.91)0.85 (0.76–0.93)0.88 (0.83–0.94)0.89 (0.84–0.94)
Chance-corrected BA0.38 (0.31–0.45)0.40 (0.31–0.48)0.41 (0.24–0.59)0.46 (0.26–0.66)0.45 (0.28–0.61)
2D image findings22AUROC0.80 (0.75–0.84)0.80 (0.75–0.84)0.77 (0.73–0.80)0.83 (0.79–0.86)0.84 (0.81–0.87)
Chance-corrected BA0.28 (0.18–0.38)0.29 (0.18–0.40)0.26 (0.16–0.35)0.39 (0.24–0.53)0.40 (0.27–0.53)
Volumetric findings9AUROC0.64 (0.55–0.73)0.64 (0.56–0.73)0.67 (0.56–0.78)0.68 (0.58–0.78)0.69 (0.60–0.79)
Chance-corrected BA0.13 (0.05–0.21)0.14 (0.04–0.23)0.14 (0.07–0.22)0.16 (0.04–0.27)0.20 (0.09–0.32)

Table S3. Random-effects mean per capability tier with 95% CI. Best denotes the best of the four models on each task.

System 1 Models and Their Backbones

On 20 tasks, Clef and Clef-flash were also compared with their backbones, Qwen3.8-27B and Qwen3.5-9B, with the backbone read out over the same options (Figure S2). Each task had one primary endpoint. The sign test across tasks was not significant for Clef-flash (better on 14 tasks and worse on 5, Holm-adjusted P=0.13) or Clef (12 and 7, Holm-adjusted P=0.36). Five of the 16 significant advantages of a System 1 model disappeared or reversed after the backbone readout was corrected for its option prior.

Forest plots comparing Clef and Clef-flash with their backbones across tasks

Figure S2. Primary endpoint of each System 1 model minus that of its backbone on 20 tasks, with 95% bias-corrected and accelerated bootstrap CIs. Filled markers indicate Holm-adjusted P<0.05P<0.05 within the task.

Pooling of Slices

For volumetric tasks, the mean study-level AUROC of yes-or-no questions over tasks and models was 0.68 with the maximum over slices, 0.69 with the mean of the three highest slices and 0.69 with the mean over slices (Figure S3). Pulmonary embolism and cervical spine fracture stayed close to chance under every rule.

Study-level AUROC of volumetric tasks under different slice-probability aggregation rules

Figure S3. Study-level AUROC of volumetric tasks with yes-or-no questions, averaged over questions, when the slice probabilities of a study were combined by their maximum (primary rule), the mean of the three highest, or the mean over all slices. Each panel shows one task and each line one model.

Generative Comparison

Table S4 lists the balanced accuracy and cost of each System 1 model and its generative model per task.

No.TaskSystem 1GenerativeBA S1BA gen.USD/1k S1USD/1k gen.
001MedQA (USMLE)ClefQwen3.8-27B0.840.920.081.40
027Lung cancer T from reportClefQwen3.8-27B0.650.830.111.62
028BI-RADS from reportClefQwen3.8-27B0.540.350.121.64
029pT stage from pathology reportClefQwen3.8-27B0.850.860.320.96
030Trial report inferenceClefQwen3.8-27B0.900.890.151.27
031PubMedQAClefQwen3.8-27B0.570.700.121.03
032Drug-drug interactionClefQwen3.8-27B0.800.880.081.15
036Rubric judgingClefQwen3.8-27B0.820.640.261.70
037ED triage (KTAS)ClefQwen3.8-27B0.520.540.082.11
038Diagnosis from caseClefQwen3.8-27B0.550.610.142.21
039ACMG variant classClefQwen3.8-27B0.390.720.082.45
040Trial eligibilityClefQwen3.8-27B0.650.660.241.69
041Suicide-risk levelClefQwen3.8-27B0.480.620.441.82
042MRI safety of a deviceClefQwen3.8-27B0.250.250.070.72
043Radiograph body partClefQwen3.8-27B0.770.770.270.74
045Imaging modalityClefQwen3.8-27B0.970.970.170.53
046Mammography viewClefQwen3.8-27B0.610.810.191.73
048CT contrast phase (slice)ClefQwen3.8-27B0.520.530.152.21
052Vertebra levelClefQwen3.8-27B0.620.580.362.02
054Low-dose CT qualityClefQwen3.8-27B0.330.410.111.34
055CXR findingsClefQwen3.8-27B0.620.600.472.13
057Line and tube positionClefQwen3.8-27B0.570.630.623.07
058Bone-age bandClefQwen3.8-27B0.170.260.262.33
059Knee OA KL gradeClefQwen3.8-27B0.200.170.092.24
060Lumbar stenosis gradeClefQwen3.8-27B0.430.350.502.92
061Head CT haemorrhage (slice)ClefQwen3.8-27B0.560.560.292.19
062BI-RADS from mammogramClefQwen3.8-27B0.270.310.262.41
063Breast densityClefQwen3.8-27B0.280.280.211.95
064Dermoscopy (HAM10000)ClefQwen3.8-27B0.470.460.141.83
067DR severity (IDRiD)ClefQwen3.8-27B0.300.300.241.57
070Colorectal tissue typeClefQwen3.8-27B0.790.750.091.59
072ECG superclassesClefQwen3.8-27B0.490.620.383.11
001MedQA (USMLE)Clef-flashQwen3.5-9B0.720.920.010.65
027Lung cancer T from reportClef-flashQwen3.5-9B0.480.840.020.81
028BI-RADS from reportClef-flashQwen3.5-9B0.510.320.020.80
029pT stage from pathology reportClef-flashQwen3.5-9B0.790.890.050.61
030Trial report inferenceClef-flashQwen3.5-9B0.790.900.030.72
031PubMedQAClef-flashQwen3.5-9B0.570.580.020.71
032Drug-drug interactionClef-flashQwen3.5-9B0.690.800.010.78
036Rubric judgingClef-flashQwen3.5-9B0.650.650.040.72
037ED triage (KTAS)Clef-flashQwen3.5-9B0.460.470.010.89
038Diagnosis from caseClef-flashQwen3.5-9B0.450.620.020.87
039ACMG variant classClef-flashQwen3.5-9B0.340.640.011.17
040Trial eligibilityClef-flashQwen3.5-9B0.670.680.040.83
041Suicide-risk levelClef-flashQwen3.5-9B0.280.530.071.00
No.TaskSystem 1GenerativeBA S1BA gen.USD/1k S1USD/1k gen.
042MRI safety of a deviceClef-flashQwen3.5-9B0.210.230.010.88
043Radiograph body partClef-flashQwen3.5-9B0.840.810.040.23
045Imaging modalityClef-flashQwen3.5-9B0.970.970.030.19
046Mammography viewClef-flashQwen3.5-9B0.680.740.030.29
048CT contrast phase (slice)Clef-flashQwen3.5-9B0.640.580.020.37
052Vertebra levelClef-flashQwen3.5-9B0.530.470.061.27
054Low-dose CT qualityClef-flashQwen3.5-9B0.330.370.020.23
055CXR findingsClef-flashQwen3.5-9B0.650.650.080.72
057Line and tube positionClef-flashQwen3.5-9B0.580.600.101.27
058Bone-age bandClef-flashQwen3.5-9B0.230.240.041.50
059Knee OA KL gradeClef-flashQwen3.5-9B0.350.280.010.39
060Lumbar stenosis gradeClef-flashQwen3.5-9B0.330.390.081.84
061Head CT haemorrhage (slice)Clef-flashQwen3.5-9B0.560.620.050.61
062BI-RADS from mammogramClef-flashQwen3.5-9B0.330.370.041.04
063Breast densityClef-flashQwen3.5-9B0.330.300.030.44
064Dermoscopy (HAM10000)Clef-flashQwen3.5-9B0.520.530.020.49
067DR severity (IDRiD)Clef-flashQwen3.5-9B0.320.350.040.83
070Colorectal tissue typeClef-flashQwen3.5-9B0.650.680.010.25
072ECG superclassesClef-flashQwen3.5-9B0.510.510.061.75
001MedQA (USMLE)GPT-6 Luna DecisionsGPT-6 Luna0.900.970.030.08
027Lung cancer T from reportGPT-6 Luna DecisionsGPT-6 Luna0.760.910.040.09
028BI-RADS from reportGPT-6 Luna DecisionsGPT-6 Luna0.320.370.040.09
029pT stage from pathology reportGPT-6 Luna DecisionsGPT-6 Luna0.860.870.130.18
030Trial report inferenceGPT-6 Luna DecisionsGPT-6 Luna0.790.860.070.12
031PubMedQAGPT-6 Luna DecisionsGPT-6 Luna0.550.630.050.08
032Drug-drug interactionGPT-6 Luna DecisionsGPT-6 Luna0.790.840.030.06
036Rubric judgingGPT-6 Luna DecisionsGPT-6 Luna0.640.620.110.20
037ED triage (KTAS)GPT-6 Luna DecisionsGPT-6 Luna0.530.560.030.10
038Diagnosis from caseGPT-6 Luna DecisionsGPT-6 Luna0.660.770.050.12
039ACMG variant classGPT-6 Luna DecisionsGPT-6 Luna0.850.820.030.13
040Trial eligibilityGPT-6 Luna DecisionsGPT-6 Luna0.640.600.090.15
041Suicide-risk levelGPT-6 Luna DecisionsGPT-6 Luna0.540.610.190.27
042MRI safety of a deviceGPT-6 Luna DecisionsGPT-6 Luna0.250.300.020.12
043Radiograph body partGPT-6 Luna DecisionsGPT-6 Luna0.880.900.110.18
045Imaging modalityGPT-6 Luna DecisionsGPT-6 Luna0.960.950.070.09
046Mammography viewGPT-6 Luna DecisionsGPT-6 Luna0.740.790.080.13
048CT contrast phase (slice)GPT-6 Luna DecisionsGPT-6 Luna0.720.790.070.16
052Vertebra levelGPT-6 Luna DecisionsGPT-6 Luna0.640.580.130.60
054Low-dose CT qualityGPT-6 Luna DecisionsGPT-6 Luna0.400.460.040.08
055CXR findingsGPT-6 Luna DecisionsGPT-6 Luna0.670.650.290.38
057Line and tube positionGPT-6 Luna DecisionsGPT-6 Luna0.550.620.320.79
058Bone-age bandGPT-6 Luna DecisionsGPT-6 Luna0.340.450.100.30
059Knee OA KL gradeGPT-6 Luna DecisionsGPT-6 Luna0.250.330.030.13
060Lumbar stenosis gradeGPT-6 Luna DecisionsGPT-6 Luna0.500.570.210.64
061Head CT haemorrhage (slice)GPT-6 Luna DecisionsGPT-6 Luna0.590.600.150.26
062BI-RADS from mammogramGPT-6 Luna DecisionsGPT-6 Luna0.290.440.100.27
063Breast densityGPT-6 Luna DecisionsGPT-6 Luna0.330.320.080.14

Table S4. System 1 model against the generative model of its family on the same cases, 100 per task and 50 for Clef and Qwen3.8-27B. BA denotes balanced accuracy, USD/1k the cost in US dollars per 1,000 requests, S1 the System 1 model and gen. the generative model.

TaskClefClef-flashGPT-6 Luna DecisionsJev 1.14
001 MedQA (USMLE)0.970.920.980.99
002 MedMCQA (India)0.890.870.920.93
003 HEAD-QA (English)0.960.950.960.97
004 HEAD-QA (Spanish)0.960.950.970.97
005 MMLU medical0.990.990.980.99
006 MMLU-Pro health0.970.960.960.97
007 MedXpertQA text0.720.630.760.77
008 Medbullets0.920.830.940.94
009 AfriMed-QA0.940.920.950.95
010 MedExpQA0.960.920.960.96
011 KorMedMCQA0.960.930.990.98
012 CMExam (China)0.970.950.970.98
013 MedQA Mainland0.980.960.980.98
014 MedQA Taiwan0.970.950.980.98
015 FrenchMedMCQA0.990.970.990.99
016 MMLU medical (Arabic)0.950.930.980.98
017 MMLU medical (Hindi)0.920.840.980.98
018 MMLU medical (Japanese)0.970.960.980.98
019 MMLU medical (Swahili)0.900.850.960.96
020 MMLU medical (Yoruba)0.850.790.910.93
021 MMLU medical (German)0.980.970.980.99
022 ATC drug class0.530.610.860.73
023 ICD-9 procedure0.650.640.810.72
024 CXR report findings1.001.000.991.00
025 Chest CT report findings0.990.990.960.99
026 Multilingual report coding0.960.940.940.96
027 Lung cancer T from report0.940.920.920.95
028 BI-RADS from report0.880.840.860.81
029 pT stage from pathology report0.960.950.960.96
030 Trial report inference0.900.870.860.92
031 PubMedQA0.810.830.790.84
032 Drug-drug interaction0.890.900.870.93
033 Hallmarks of cancer0.940.940.890.94
034 Public-health claims0.850.810.810.82
035 Question entailment0.880.880.900.92
036 Rubric judging0.830.730.780.84
037 ED triage (KTAS)0.800.750.820.80
038 Diagnosis from case0.830.740.850.82
039 ACMG variant class0.880.760.960.93
040 Trial eligibility0.850.850.830.87
041 Suicide-risk level0.770.750.810.82
042 MRI safety of a device0.580.510.510.66
043 Radiograph body part0.950.960.960.95
044 CXR view AP/PA0.980.990.890.99
045 Imaging modality1.001.000.991.00
046 Mammography view0.920.960.960.99
047 CT contrast phase (series)0.820.820.900.90
048 CT contrast phase (slice)0.750.730.850.84
049 Abdominal organ (3D)0.830.840.820.84
050 Abdominal organ (coronal)0.780.840.710.82
051 Abdominal organ (sagittal)0.720.800.680.80
052 Vertebra level0.940.940.970.95
053 MRI motion artefact0.710.710.700.76
054 Low-dose CT quality0.720.620.760.78
055 CXR findings0.720.770.730.77
056 Paediatric pneumonia0.950.960.950.93
057 Line and tube position0.780.790.650.81
058 Bone-age band0.750.730.810.77
059 Knee OA KL grade0.660.690.660.64
060 Lumbar stenosis grade0.680.660.810.73
061 Head CT haemorrhage (slice)0.760.750.760.79
062 BI-RADS from mammogram0.620.610.680.69
063 Breast density0.710.680.700.66
064 Dermoscopy (HAM10000)0.870.870.830.90
065 Dermoscopy (ISIC 2019)0.810.820.780.84
066 Melanoma (ISIC 2020)0.760.780.560.75
067 DR severity (IDRiD)0.750.820.610.83
068 DR grade (RetinaMNIST)0.650.700.570.86
069 Retinal OCT diagnosis0.970.980.921.00
070 Colorectal tissue type0.980.960.870.96
071 Blood cell type0.970.980.860.99
072 ECG superclasses0.730.670.740.68
073 VQA-RAD yes/no0.910.880.870.94
074 SLAKE yes/no0.940.920.900.97
075 SLAKE yes/no (Chinese)0.960.920.930.99
076 PMC-VQA0.840.830.820.86
077 MedXpertQA multimodal0.720.640.790.79
078 Head CT haemorrhage0.830.850.900.89
079 Critical head CT finding0.840.790.880.87
080 Chest CT abnormalities0.690.710.760.74
081 Pulmonary embolism on CTPA0.510.520.540.54
082 Cervical spine fracture0.570.580.590.53
083 Abdominal trauma on CT0.690.730.740.79
084 Pancreatic cancer on CT0.610.680.640.70
085 Prostate cancer bpMRI0.580.440.550.59
086 Rib fracture type0.410.480.420.44

Table S1. AUROC of every task (rows, numbered as in Table S1) and model (columns). Blue shades mark values above chance (0.5) and red shades values below chance.

No.TaskDatasetTierInputLanguageCasesQuestions
001USMLE-style MCQ (MedQA, 4 options)MedQA [23]Medical knowledgetextEnglish1,2731
002Indian medical entrance exam MCQ (MedMCQA)MedMCQA [38]Medical knowledgetextEnglish4,1831
003Spanish healthcare specialisation exam MCQ (HEAD-QA)HEAD-QA [39]Medical knowledgetextEnglish2,6751
004Spanish healthcare specialisation exam MCQ, Spanish original (HEAD-QA)HEAD-QA [39]Medical knowledgetextSpanish2,6751
005Medical subsets of MMLUMMLU [40]Medical knowledgetextEnglish1,0891
006Health questions of MMLU-ProMMLU-Pro [41]Medical knowledgetextEnglish8181
007Expert-level medical MCQ, text (MedXpertQA Text)MedXpertQA [42]Medical knowledgetextEnglish2,4501
008Hard clinical case MCQ (Medbullets op5)Medbullets op5 [43]Medical knowledgetextEnglish3081
009Pan-African medical MCQ (AfriMed-QA v2)AfriMed-QA v2 [35]Medical knowledgetextEnglish3,6001
010Multilingual clinical-case MCQ (MedExpQA)MedExpQA [36]Medical knowledgetextSeveral5001
011Korean healthcare licensing exam MCQ (KorMedMCQA)KorMedMCQA [44]Medical knowledgetextKorean3,0091
012Chinese national medical licensing exam MCQ (CMExam)CMExam [45]Medical knowledgetextChinese6,6071
013Chinese medical licensing exam MCQ (MedQA Mainland)MedQA [23]Medical knowledgetextChinese3,4261
014Taiwanese medical licensing exam MCQ (MedQA Taiwan)MedQA [23]Medical knowledgetextChinese1,4131
015French pharmacy exam MCQ (FrenchMedMCQA)FrenchMedMCQA [46]Medical knowledgetextFrench3211
016Medical subsets of MMLU in Arabic (Global-MMLU)Global-MMLU [34]Medical knowledgetextArabic1,0891
017Medical subsets of MMLU in Hindi (Global-MMLU)Global-MMLU [34]Medical knowledgetextHindi1,0891
018Medical subsets of MMLU in Japanese (Global-MMLU)Global-MMLU [34]Medical knowledgetextJapanese1,0891
019Medical subsets of MMLU in Swahili (Global-MMLU)Global-MMLU [34]Medical knowledgetextSwahili1,0891
020Medical subsets of MMLU in Yoruba (Global-MMLU)Global-MMLU [34]Medical knowledgetextYoruba1,0891
021Medical subsets of MMLU in German (Global-MMLU)Global-MMLU [34]Medical knowledgetextGerman1,0891
022Drug class from ATC code (MedConceptsQA)MedConceptsQA [47]Medical knowledgetextEnglish18,8061
023Procedure description from ICD-9-CM code (MedConceptsQA)MedConceptsQA [47]Medical knowledgetextEnglish13,7661
024Chest radiograph report findings (OpenI / IU X-ray)OpenI / IU X-ray [48]Text understandingtextEnglish3,83213
025Chest CT findings from Japanese reports (CT-RATE-JPN)CT-RATE-JPN [49, 50]Text understandingtextJapanese15018
026Modality and ICD-10 chapters of multilingual reports (PARROT)PARROT [51]Text understandingtextSeveral2,73813
027Lung cancer T category from radiology reports (NTCIR-17 RR-TNM)NTCIR-17 RR-TNM [31]Text understandingtextJapanese2431
028BI-RADS category from breast imaging reports (Paraguay corpus)Paraguay reports [52]Text understandingtextEnglish1,0001
029Pathologic T stage from a pathology report (TCGA)TCGA reports [32, 53]Text understandingtextEnglish1,0001
030Clinical-trial report inference (NLI4CT, SemEval-2024 Task 2)NLI4CT [54]Text understandingtextEnglish5001
031Biomedical research QA (PubMedQA PQA-L)PubMedQA PQA-L [55]Text understandingtextEnglish5001
032Drug-drug interaction type (DDI-2013)DDI-2013 [56, 57]Text understandingtextEnglish1,0001
033Hallmarks of cancer in abstracts (HoC)HoC [58]Text understandingtextEnglish37110
034Public-health claim veracity (PUBHEALTH)PUBHEALTH [59]Text understandingtextEnglish1,2331
035Consumer health question entailment (MEDIQA-RQE 2019)MEDIQA-RQE 2019 [60]Text understandingtextEnglish2301
036Rubric criterion judging (HealthBench meta-evaluation)HealthBench [61]Text understandingtextEnglish1,0001
037ED triage acuity (KTAS)KTAS [28]Clinical judgement from texttextEnglish1,2671
038Diagnosis selection from a published case (DiagnosisArena)DiagnosisArena [62]Clinical judgement from texttextEnglish9151
039ACMG/AMP five-tier variant classification (ClinGen)ClinGen [63]Clinical judgement from texttextEnglish1,0001
040Patient-trial eligibility (TREC Clinical Trials 2021/2022)TREC CT 2021/22 [64]Clinical judgement from texttextEnglish5001
041Suicide-risk severity of a Reddit user (Reddit C-SSRS)Reddit C-SSRS [65]Clinical judgement from texttextEnglish5001
042MRI safety status of a device (FDA AccessGUDID)FDA AccessGUDID [66]Clinical judgement from texttextEnglish1,0001
043Radiograph body partUNIFESP body part [67]Image perception2DEnglish5931
044Chest radiograph view AP vs PA (NIH ChestX-ray14)NIH ChestX-ray14 [68]Image perception2DEnglish7161
045Imaging modality (ROCOV2)ROCOV2 [69]Image perception2DEnglish7941
No.TaskDatasetTierInputLanguageCasesQuestions
046Mammography view CC vs MLO (CBIS-DDSM)CBIS-DDSM [70, 71]Image perception2DEnglish6451
047CT contrast phase (series level, WAW-TACE)WAW-TACE [72]Image perception3DEnglish1601
048Contrast status and phase (single slice, WAW-TACE)WAW-TACE [72]Image perception2DEnglish9602
049Abdominal organ on CT, 3D (OrganMNIST3D)OrganMNIST3D [73, 74]Image perception2D multi-viewEnglish6101
050Abdominal organ on coronal CT (OrganCMNIST)OrganCMNIST [73, 74]Image perception2DEnglish8,2161
051Abdominal organ on sagittal CT (OrganSMNIST)OrganSMNIST [73, 74]Image perception2DEnglish8,8271
052Marked vertebra level on CT (VerSe)VerSe [75]Image perception2D multi-viewEnglish2192
053MRI motion artefact severity (MR-ART)MR-ART [76]Image perception3DEnglish4361
054Diagnostic image quality of low-dose CT (LDCTIQAC 2023)LDCTIQAC 2023 [77]Image perception2DEnglish1,0001
055Chest radiograph multi-label findings (NIH ChestX-ray14)NIH ChestX-ray14 [68]2D image findings2DEnglish71614
056Pneumonia on paediatric chest radiographs (PneumoniaMNIST)PneumoniaMNIST [73, 78]2D image findings2DEnglish6241
057Line and tube position on chest radiographsRANZCR CLiP [68, 79]2D image findings2DEnglish49711
058Paediatric bone-age bandRSNA bone age [80]2D image findings2DEnglish5001
059Knee osteoarthritis KL gradeOAI [82] [81, 82]2D image findings2DEnglish1,0001
060Lumbar degenerative stenosis gradeRSNA lumbar 2024 [83]2D image findings2D multi-viewEnglish1,0005
061Head CT haemorrhage and subtype (slice)RSNA ICH 2019 [84]2D image findings2DEnglish1,0006
062BI-RADS assessment from mammograms (CBIS-DDSM)CBIS-DDSM [70, 71]2D image findings2D multi-viewEnglish7041
063Breast density category (CBIS-DDSM)CBIS-DDSM [70, 71]2D image findings2DEnglish6431
064Dermoscopic lesion diagnosis (HAM10000 / ISIC 2018 Task 3)HAM10000 [85, 86]2D image findings2DEnglish5151
065Dermoscopic lesion diagnosis (ISIC 2019)ISIC 2019 [86, 87, 88]2D image findings2DEnglish8,2381
066Melanoma vs other lesion (SIIM-ISIC 2020)SIIM-ISIC 2020 [89]2D image findings2DEnglish33,1261
067Diabetic retinopathy severity (IDRiD)IDRiD [90]2D image findings2DEnglish5161
068Diabetic retinopathy grade (RetinaMNIST)RetinaMNIST [73, 91]2D image findings2DEnglish4001
069Retinal OCT diagnosis (OCTMNIST)OCTMNIST [73, 78]2D image findings2DEnglish1,0001
070Colorectal H&E tissue type (CRC-VAL-HE-7K)CRC-VAL-HE-7K [92]2D image findings2DEnglish9001
071Blood cell type (BloodMNIST)BloodMNIST [73, 93]2D image findings2DEnglish3,4211
07212-lead ECG superclasses (PTB-XL, rendered as image)PTB-XL [94]2D image findings2DEnglish2,1985
073Radiology VQA, closed yes/no questions (VQA-RAD)VQA-RAD [95]2D image findingstext+imageEnglish2511
074Radiology VQA, closed yes/no questions (SLAKE)SLAKE [96]2D image findingstext+imageEnglish3551
075Radiology VQA in Chinese, closed yes/no questions (SLAKE)SLAKE [96]2D image findingstext+imageChinese3471
076Biomedical figure VQA (PMC-VQA)PMC-VQA [97]2D image findingstext+imageEnglish2,0001
077Expert-level medical MCQ with images (MedXpertQA MM)MedXpertQA [42]2D image findingstext+imageEnglish2,0001
078Head CT haemorrhage and subtype (scan level, CQ500)CQ500 [27]Volumetric findings3DEnglish816
079Critical head CT finding (CQ500)CQ500 [27]Volumetric findings3DEnglish814
080Chest CT multi-abnormality (CT-RATE)CT-RATE [50]Volumetric findings3DEnglish10018
081Pulmonary embolism on CTPARSNA PE 2020 [24]Volumetric findings3DEnglish1506
082Cervical spine fracture on CTRSNA C-spine 2022 [25]Volumetric findings3DEnglish2008
083Traumatic abdominal organ injury on CTRSNA trauma 2023 [98]Volumetric findings3DEnglish2008
084Pancreatic ductal adenocarcinoma on contrast-enhanced CT (PANORAMA)PANORAMA [99]Volumetric findings3DEnglish3001
085Clinically significant prostate cancer on bpMRI (PI-CAI)PI-CAI [26]Volumetric findings3DEnglish2001
086Rib fracture type (FractureMNIST3D, RibFrac)FractureMNIST3D [73, 100]Volumetric findings2D multi-viewEnglish2401

Table S1 (PDF p. 17). Table S1 | Tasks. Tasks 004, 016, 017, 018, 019, 020, 021, 075 are language versions of another task. MCQ denotes multiple-choice questions, VQA visual question answering, and ED emergency department.

No.TaskClefClef-flashGPT-6Luna DecisionsJev 1.14
046Mammography view0.92 (0.90–0.94)0.96 (0.95–0.98)0.96 (0.95–0.98)0.99 (0.98–0.99)
047CT contrast phase (series)0.82 (0.79–0.85)0.82 (0.78–0.86)0.90 (0.87–0.93)0.90 (0.87–0.92)
048CT contrast phase (slice)0.75 (0.73–0.77)0.73 (0.70–0.75)0.85 (0.83–0.86)0.84 (0.82–0.86)
049Abdominal organ (3D)0.83 (0.82–0.85)0.84 (0.83–0.86)0.82 (0.81–0.84)0.84 (0.82–0.85)
050Abdominal organ (coronal)0.78 (0.78–0.79)0.84 (0.84–0.84)0.71 (0.70–0.71)0.82 (0.82–0.83)
051Abdominal organ (sagittal)0.72 (0.72–0.73)0.80 (0.80–0.81)0.68 (0.67–0.68)0.80 (0.80–0.81)
052Vertebra level0.94 (0.93–0.95)0.94 (0.93–0.95)0.97 (0.96–0.97)0.95 (0.94–0.96)
053MRI motion artefact0.71 (0.69–0.74)0.71 (0.68–0.73)0.70 (0.67–0.72)0.76 (0.73–0.78)
054Low-dose CT quality0.72 (0.70–0.74)0.62 (0.60–0.65)0.76 (0.74–0.78)0.78 (0.77–0.80)
055CXR findings0.72 (0.69–0.74)0.77 (0.75–0.79)0.73 (0.71–0.76)0.77 (0.75–0.79)
056Paediatric pneumonia0.95 (0.93–0.96)0.96 (0.94–0.97)0.95 (0.93–0.96)0.93 (0.91–0.95)
057Line and tube position0.78 (0.76–0.79)0.79 (0.78–0.80)0.65 (0.63–0.67)0.81 (0.79–0.82)
058Bone-age band0.75 (0.73–0.77)0.73 (0.71–0.75)0.81 (0.79–0.83)0.77 (0.75–0.78)
059Knee OA KL grade0.66 (0.64–0.68)0.69 (0.67–0.70)0.66 (0.63–0.69)0.64 (0.62–0.66)
060Lumbar stenosis grade0.68 (0.66–0.70)0.66 (0.65–0.68)0.81 (0.80–0.83)0.73 (0.71–0.75)
061Head CT haemorrhage (slice)0.76 (0.74–0.78)0.75 (0.73–0.77)0.76 (0.74–0.78)0.79 (0.77–0.80)
062BI-RADS from mammogram0.62 (0.59–0.65)0.61 (0.59–0.65)0.68 (0.65–0.70)0.69 (0.65–0.72)
063Breast density0.71 (0.68–0.73)0.68 (0.65–0.71)0.70 (0.67–0.73)0.66 (0.63–0.69)
064Dermoscopy (HAM10000)0.87 (0.85–0.89)0.87 (0.85–0.89)0.83 (0.81–0.85)0.90 (0.89–0.92)
065Dermoscopy (ISIC 2019)0.81 (0.81–0.82)0.82 (0.82–0.83)0.78 (0.77–0.79)0.84 (0.83–0.84)
066Melanoma (ISIC 2020)0.76 (0.74–0.78)0.78 (0.76–0.80)0.56 (0.53–0.59)0.75 (0.73–0.77)
067DR severity (IDRiD)0.75 (0.72–0.78)0.82 (0.80–0.85)0.61 (0.58–0.64)0.83 (0.81–0.85)
068DR grade (RetinaMNIST)0.65 (0.61–0.69)0.70 (0.66–0.74)0.57 (0.54–0.61)0.86 (0.83–0.89)
069Retinal OCT diagnosis0.97 (0.97–0.98)0.98 (0.98–0.99)0.92 (0.90–0.93)1.00 (1.00–1.00)
070Colorectal tissue type0.98 (0.97–0.98)0.96 (0.95–0.97)0.87 (0.86–0.89)0.96 (0.95–0.97)
071Blood cell type0.97 (0.97–0.98)0.98 (0.98–0.98)0.86 (0.85–0.87)0.99 (0.98–0.99)
072ECG superclasses0.73 (0.72–0.75)0.67 (0.66–0.68)0.74 (0.73–0.75)0.68 (0.66–0.69)
073VQA-RAD yes/no0.91 (0.87–0.94)0.88 (0.84–0.92)0.87 (0.82–0.91)0.94 (0.91–0.96)
074SLAKE yes/no0.94 (0.92–0.96)0.92 (0.89–0.95)0.90 (0.87–0.94)0.97 (0.96–0.99)
075SLAKE yes/no (Chinese)0.96 (0.94–0.98)0.92 (0.89–0.95)0.93 (0.90–0.96)0.99 (0.98–0.99)
076PMC-VQA0.84 (0.83–0.86)0.83 (0.82–0.85)0.82 (0.81–0.84)0.86 (0.85–0.88)
077MedXpertQA multimodal0.72 (0.70–0.73)0.64 (0.63–0.66)0.79 (0.78–0.81)0.79 (0.78–0.81)
078Head CT haemorrhage0.83 (0.75–0.89)0.85 (0.80–0.91)0.90 (0.85–0.94)0.89 (0.84–0.93)
079Critical head CT finding0.84 (0.78–0.88)0.79 (0.74–0.85)0.88 (0.83–0.92)0.87 (0.83–0.90)
080Chest CT abnormalities0.69 (0.66–0.73)0.71 (0.68–0.74)0.76 (0.73–0.79)0.74 (0.71–0.77)
081Pulmonary embolism on CTPA0.51 (0.45–0.56)0.52 (0.45–0.59)0.54 (0.49–0.59)0.54 (0.49–0.59)
082Cervical spine fracture0.57 (0.51–0.63)0.58 (0.53–0.64)0.59 (0.54–0.65)0.53 (0.47–0.60)
083Abdominal trauma on CT0.69 (0.64–0.74)0.73 (0.68–0.77)0.74 (0.68–0.78)0.79 (0.75–0.83)
084Pancreatic cancer on CT0.61 (0.55–0.68)0.68 (0.61–0.74)0.64 (0.58–0.71)0.70 (0.64–0.75)
085Prostate cancer bpMRI0.58 (0.50–0.66)0.44 (0.36–0.51)0.55 (0.46–0.63)0.59 (0.51–0.67)
086Rib fracture type0.41 (0.37–0.46)0.48 (0.43–0.53)0.42 (0.37–0.47)0.44 (0.39–0.49)

Table 1.

No.TaskSystem 1GenerativeBA S1BA gen.USD/1k S1USD/1k gen.
064Dermoscopy (HAM10000)GPT-6 Luna DecisionsGPT-6 Luna0.440.510.050.16
067DR severity (IDRiD)GPT-6 Luna DecisionsGPT-6 Luna0.330.420.100.22
070Colorectal tissue typeGPT-6 Luna DecisionsGPT-6 Luna0.550.450.030.10
072ECG superclassesGPT-6 Luna DecisionsGPT-6 Luna0.570.620.190.75

Table 2. System 1 models in medicine

No.TaskTypeReferenceSourceMetricRef.Best S1
041Suicide-risk levelTask-specific modelMental-FLAN-T5 (FLAN-T5-XXL, instruction-finetuned on four mental-health datasets)Xu 2024 (Mental-LLM)balanced accuracy0.480.56
044CXR view AP/PATask-specific modelResNet-18Kim 2019AUROC1.000.99
046Mammography viewTask-specific modelResNet-50, fine-tuned, 256 × 256 pxYi 2019AUROC1.000.99
047CT contrast phase (series)Task-specific model2dFM_BERT (frozen MAE ViT-L/16 + BERT)Hou 2025AUROC0.910.90
049Abdominal organ (3D)Task-specific model3D CNN with Kinetics-400 video initialization, fine-tunedJain 2024accuracy0.970.38
050Abdominal organ (coronal)Generative language modelQwen3-VL-8B with retrieved exemplars and rules (in-context learning, frozen)Zhao 2026accuracy0.940.40
050Abdominal organ (coronal)Task-specific modelDenseNet-121, 128 pxDoerrich 2024accuracy0.940.40
051Abdominal organ (sagittal)Task-specific modelDenseNet-121, 64 pxDoerrich 2024accuracy0.830.31
053MRI motion artefactTask-specific modelFewSOME (unsupervised few-shot)Belton 2024accuracy0.720.32
055CXR findingsTask-specific modelArk+ (Swin-L, 768 px; fine-tuned)Ma 2025 (Ark+)AUROC0.840.77
056Paediatric pneumoniaGenerative language modelMedGemma-27B, evolved prompt, no fine-tuningSviridov 2026accuracy0.840.79
056Paediatric pneumoniaTask-specific modelDINO ViT-B/16, linear probing, 224 pxDoerrich 2025AUROC0.990.96
057Line and tube positionCliniciansRadiologist (Reader 2 of 2, board-certified/board-eligible)Güzel 2026balanced accuracy0.750.50
057Line and tube positionGenerative language modelGemini 3.1 Flash Lite Preview, zero-shot (Swan-Ganz)Güzel 2026balanced accuracy0.540.87
057Line and tube positionTask-specific modelRank 1 (All Data Are Ext)Kaggle RANZCR CLiP private LBAUROC0.980.81
059Knee OA KL gradeTask-specific modelEnsemble of 6 ImageNet-pretrained CNNs, per-model optimised input size, mix votingPi 2023accuracy0.770.42
061Head CT haemorrhage (slice)Generative language modelGemini 2.0 Flash, zero-shotWang 2025accuracy0.750.71
061Head CT haemorrhage (slice)Task-specific modelRank-1 Kaggle ensembleWang 2021AUROC1.000.88
063Breast densityTask-specific modelLG-DPTNet: multi-view (CC + MLO) fusion, dynamic pyramid convolution + cross-transformerZhong 2023AUROC0.970.71
064Dermoscopy (HAM10000)CliniciansHuman experts (> 10 years)Tschandl 2019balanced accuracy0.810.62
064Dermoscopy (HAM10000)Generative language modelHulu-Med-7B, zero-shotLiu 2026 (DermAgent)accuracy0.530.63
064Dermoscopy (HAM10000)Task-specific modelRank 1, team ’Minnie’, approach ’ensemble b4 ce+fl+cepos+ds1 b3 ce’ISIC 2018 live leaderboardbalanced accuracy0.900.62
065Dermatoscopy (ISIC 2019)Task-specific modelRank 1 (DAISYLab, EfficientNet ensemble, external data)ISIC 2019 leaderboardbalanced accuracy0.640.48
066Melanoma (ISIC 2020)Task-specific modelRank 1 of 3,308 entries (CNN ensemble)Kurtansky 2024AUROC0.950.78
067DR severity (IDRiD)Task-specific modelMulti-stage transfer learning (ImageNet → EyePACS → DDR → IDRiD) + class-balanced lossShi 2023accuracy0.800.56
068DR grade (RetinaMNIST)Generative language modelGemini-3-flash, zero-shotZhao 2026accuracy0.440.66
068DR grade (RetinaMNIST)Task-specific modelVGG16, end-to-end training, 224 × 224Doerrich 2025accuracy0.640.66
069Retinal OCT diagnosisTask-specific modelInception V3, ImageNet transfer learning (108,312 training images)Kermany 2018accuracy0.970.97
070Colorectal tissue typeGenerative language modelGPT-4V, 5-shot in-context (kNN)Ferber 2024accuracy0.780.74
070Colorectal tissue typeTask-specific modelCONCH, linear probe trained on NCT-CRC-HE-100KLu 2024 (CONCH)balanced accuracy0.940.74
071Blood cell typeTask-specific modelDenseNet-121, end-to-end, 224 pxDoerrich 2025accuracy0.990.80
072ECG superclassesGenerative language modelGPT-4o, zero-shotLiu 2024 (PULSE)AUROC0.560.74
072ECG superclassesTask-specific modelEnsemble of all modelsStröder 2021AUROC0.930.74
073VQA-RAD yes/noGenerative language modelGPT-5-mini, zero-shot chain-of-thoughtWang 2025accuracy0.750.84
073VQA-RAD yes/noTask-specific modelMedVInT-TD (fine-tuned)Zhang 2023accuracy0.870.84
074SLAKE yes/noGenerative language modelMedGemma 4B, zero-shotSellgren 2025accuracy0.880.93
074SLAKE yes/noTask-specific modelM2I2 (supervised)Li 2023accuracy0.910.93
076PMC-VQAGenerative language modelLLaVA-Med, zero-shotZhang 2023accuracy0.350.64
076PMC-VQATask-specific modelMedVInT-TD (trained on PMC-VQA)Zhang 2023accuracy0.400.64
No.TaskTypeReferenceSourceMetricRef.Best S1
077MedXpertQA multimodalCliniciansExpert (pre-licensed humans)Zuo 2025accuracy0.460.49
077MedXpertQA multimodalGenerative language modelGPT-5Wang 2025accuracy0.720.49
078Head CT haemorrhageTask-specific modelQure.ai deep learning algorithmsChilamkurthy 2018AUROC0.970.84
079Critical head CT findingTask-specific modelQure.ai deep learning algorithmsChilamkurthy 2018AUROC0.970.99
080Chest CT abnormalitiesTask-specific modelCT-CLIP VocabFineHamamci 2026AUROC0.760.76
081Pulmonary embolism on CTPATask-specific modelAttention-based CNN-RNN, strongly supervised (all slice labels)Hu 2025AUROC0.930.58
082Cervical spine fractureTask-specific modelMean of top 8 Kaggle algorithmsLee 2024AUROC0.960.63
083Abdominal trauma on CTTask-specific modelMean of 8 winning challenge modelsHermans 2025AUROC0.980.85
084Pancreatic cancer on CTClinicians68 radiologists (pooled)Alves 2026AUROC0.880.70
084Pancreatic cancer on CTTask-specific modelRank 1 (MountainHill / DTI, DTI_PANORAMA)PANORAMA leaderboardAUROC0.930.70
085Prostate cancer bpMRIClinicians62 radiologists (mpMRI, PI-RADS 2.1)Saha 2024AUROC0.860.59
085Prostate cancer bpMRITask-specific modelPI-CAI AI systemSaha 2024AUROC0.930.59
086Rib fracture typeTask-specific modelEvolutionary neural architecture search with genetic-algorithm data augmentationAli 2024accuracy0.690.41

Table 27.

References

  1. [1]Ben Van Calster, David J McLernon, Maarten van Smeden, Laure Wynants, Ewout W Steyerberg, and Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative. Calibration: the Achilles heel of predictive analytics. BMC Med, 17(1):230, 2019. doi: 10.1186/s12916-019-1466-7.
  2. [2]Benjamin Comanescu, Jasper Snoek, and Andrew L Beam. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digit Med, 4(1):4, 2021. doi: 10.1038/s41746-020-00367-3.DOI
  3. [3]Krishnamurthy Dj Vijotham, Jim Winkens, Melih Barsbey, Sumedh Ghais̄as, Robert Stanforth, Nick Pawlowski, Patricia Strachan, Zahra Ahmed, Shekoofeh Azizi, Yoram Bachrach, Laura Culp, Mayank Daswani, Jan Freyberg, Christopher Kelly, Atilla Kiraly, et al. Enhancing the reliability and accuracy of AI-enabled diagnosis via complementarity-driven deferral to clinicians. Nat Med, 29(7):1814–1820, 2023. doi: 10.1038/s41591-023-02437-x.
  4. [4]Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023. doi: 10.1038/s41586-023-06291-2.DOI
  5. [5]Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of GPT-4 on medical challenge problems. 2023. arXiv:2303.13375.arxiv.org/abs/2303.13375
  6. [6]Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: watts driving the cost of AI deployment? In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 85–99, 2024. doi: 10.1145/3630106.3658542.DOI
  7. [7]Thomas Savage, John Wang, Robert Gallo, Abdesselam Boukil, Vishwesh Patel, Seyed Amir Ahmad Safavi-Naini, Ali Soroush, and Jonathan H Chen. Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment. J Am Med Inform Assoc, 32(1):139–149, 2025. doi: 10.1093/jamia/ocae254.DOI
  8. [8]Mahmud Omar, Reem Agbarea, Benjamin S Glicksberg, Girish N Nadkarni, and Eyal Klang. Benchmarking the confidence of large language models in answering clinical questions: cross-sectional evaluation study. JMIR Med Inform, 13:e66917, 2025. doi: 10.2196/66917.DOI
  9. [9]Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, New York, 2011. ISBN 9780374275631.
  10. [10]Pat Croskerry. A universal model of diagnostic reasoning. Acad Med, 84(8):1022–1028, 2009. doi: 10.1097/ACM.0b013e3181ace703.DOI
  11. [11]Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason A Fries, Michael Wornow, Akshay Swaminathan, Lisa Soleymani Lehmann, Hyo Jung Hong, Mehr Kashyap, Akash R Chaurasia, Nirav R Shah, Karandeep Singh, et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA, 333(4):319–328, 2025. doi: 10.1001/jama.2024.21700.DOI
  12. [12]Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, Darlene Neal, Qazi Mamunur Rashid, Mike Schaekermann, Amy Wang, Dev Dash, et al. Toward expert-level medical question answering with large language models. Nat Med, 31(3):943–950, 2025. doi: 10.1038/s41591-024-03423-7.DOI
  13. [13]Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, and Daniel Rueckert. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med, 30(9):2613–2622, 2024. doi: 10.1038/s41591-024-03097-1.DOI
  14. [14]Shreya Johri, Jaehwan Jeong, Benjamin A Tran, Daniel I Schlessinger, Shannon Wongvibulsin, Leandra A Barnes, Hong-Yu Zhou, Zhuo Ran Cai, Eliezer M Van Allen, David Kim, Roxana Daneshjou, and Pranav Rajpurkar. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat Med, 31(1):77–86, 2025. doi: 10.1038/s41591-024-03328-5.DOI
  15. [15]Quirin D Strotzer, Felix Nieberle, Laura S Kupke, Gerardo Napodano, Anna Katharina Muertz, Stefanie Meiler, Ingo Einspieler, Janine Rennert, Michael Strotzer, Isabel Wiesinger, Christina Wendl, Christian Stroszczynski, Okka W Hamer, and Andreas Schicho. Toward foundation models in radiology? Quantitative assessment of GPT-4V’s multimodal and multianatomical region capabilities. Radiology, 313(2):e240955, 2024. doi: 10.1148/radiol.240955.DOI
  16. [16]Qingxia Wu, Qingxia Wu, Peipei Zhang, Zhifeng Yi, Yu Shen, Yan Bai, Hongna Tan, Pei Dong, Zhong Xue, Neil Roberts, and Meiyun Wang. A bilingual benchmark for evaluating diagnostic performance of multimodal large language models in radiology (RadM-Bench): evaluation development and validation. J Med Internet Res, 28:e92183, 2026. doi: 10.2196/92183.DOI
  17. [17]Inioluwa Deborah Raji, Emily M. Bender, Amandalynn Paullada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. doi: 10.48550/arXiv.2111.15366. URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/084b6fb b10729ed4da8c3d3f5a3ae7c9-Abstract-round2.html. arXiv:2111.15366.DOI
  18. [18]Qwen Team. Qwen3.8-Max: A new bar for coding and cowork, August 2026. URL https://qwen.ai/blog?id=qwen3.8.
  19. [19]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5.
  20. [20]Tugba Akincı D’Antoni and Florence Doo. CEDARS: Carbon, energy diagnostics and reporting for sustainability. Software, 2026. URL https://github.com/takinci/cedars. Apache-2.0. Accessed October 2026.
  21. [21]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330, 2017. doi: 10.48550/arXiv.1706.04599. URL https:// proceedings.mlr.press/v70/guo17a.html. arXiv:1706.04599.DOI
  22. [22]Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017. doi: 10.48550/arXiv.1705.08500. URL https:// proceedings.neurips.cc/paper/2017/hash/4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html. arXiv:1705.08500.DOI
  23. [23]Di Jin, Eileen Pan, Nassim Oufatolle, Wei-Hung Weng, Hanxi Fang, and Peter Szolovits. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl Sci, 11(14):6421, 2021. doi: 10.3390/app11146421.DOI
  24. [24]Errol Colak, Felipe C Kitamura, Stephen B Hobbs, Carol C Wu, Matthew P Lungren, Luciano M Prevedello, Jayashree Kalpathy-Cramer, Robyn L Ball, George Shih, Anouk Stein, Sarfwan S Halabi, Emre Altinmakas, Meng Law, Parveen Kumar, Karam A Manzalawi, et al. The RSNA Pulmonary Embolism CT dataset. Radiol Artif Intell, 3(2):e200254, 2021. doi: 10.1148/ryai.2021200254.DOI
  25. [25]Hui Ming Lin, Errol Colak, Tyler Richards, Felipe C Kitamura, Luciano M Prevedello, Jason Talbott, Robyn L Ball, Ekim Gumeler, Kristen W Yeom, Mohammad Hamghalam, Amber L Simpson, Jasna Strika, Deniz Bulja, Salita Angkruwaranon, Almudena Pérez-Lara, et al. The RSNA Cervical Spine Fracture CT dataset. Radiol Artif Intell, 5(5):e2300034, 2023. doi: 10.1148/ryai.2300034.DOI
  26. [26]Anindo Saha, Joeran S Bosma, Jasper J Twilt, Bram van Ginneken, Anders Bjartell, Anwar R Padhani, David Bonekamp, Geert Villeirs, Georg Salomon, Gianluca Giannarini, Jayashree Kalpathy-Cramer, Jelle Barentsz, Klaus H Maier-Hein, Mirabela Rusu, Olivier Rouvière, et al. Artificial intelligence and radiologists in prostate cancer detection on MRI (PI-CAI): an international, paired, non-inferiority, confirmatory study. Lancet Oncol, 25(7):879–887, 2024. doi: 10.1016/S1470-2045(24)00020-1.DOI
  27. [27]Sasanik Chlamtchuk, Rohit Ghosh, Swetha Tanamala, Mustafa Biviji, Norbert G Campeau, Vasantha Kumar Venugopal, Vidur Mahajan, Pooja Rao, and Prashant Warier. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet, 392(10162):2388–2396, 2018. doi: 10.1016/S0140-6736(18)31645-3.
  28. [28]Sun-Hee Moon, Jae Lan Shim, Keun-Sook Park, and Chon-Suk Park. Triage accuracy and causes of mistriage using the Korean Triage and Acuity Scale. PLoS One, 14(9):e0216972, 2019. doi: 10.1371/journal.pone.0216972.DOI
  29. [29]Sue Richards, Nazneen Aziz, Sherri Bale, David Bick, Soma Das, Julie Gastier-Foster, Wayne W Grody, Madhuri Hegde, Elaine Lyon, Elaine Specter, Karl Veelkenring, Heidi L Rehm, and ACMG Laboratory Quality Assurance Committee. Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genet Med, 17(5):405–424, 2015. doi: 10.1038/gim.2015.30.DOI
  30. [30]Sean V Tavtigian, Steven M Harrison, Kenneth M Boucher, and Leslie G Bieseker. Fitting a naturally scaled point system to the ACMG/AMP variant classification guidelines. Hum Mutat, 41(10):1734–1737, 2020. doi: 10.1002/humu.24088.DOI
  31. [31]Syuta Nakamura, Shouhei Hanaoka, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. NTCIR-17 MedNLP-SC radiology report subtask overview: dataset and solutions for automated lung cancer staging. In Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, pages 145–151, 2023. doi: 10.20736/0002001328.DOI
  32. [32]Jenna Kefeli, Jacob Berkowitz, Jose M Acitores Cortina, Kevin K Tsang, and Nicholas P Tatonetti. Generalizable and automated classification of TNM stage from pathology reports with external validation. Nat Commun, 15(1):8916, 2024. doi: 10.1038/s41467-024-53190-9.DOI
  33. [33]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. 2023. URL https://arxiv.org/abs/2201.11903.
  34. [34]Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, PeeraT Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, et al. Global MMLU: understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, 2025. doi: 10.18653/v1/2025.acl-long.919.DOI
  35. [35]Charles Nimo, Tobi Olatunji, Abraham Toluwase Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, Ezinwanne C. Aka, Folafunmi Omofoye, Foutse Yuehogh, Timothy Faniran, Bonaventure F. P. Dossou, Moshood O. Yekini, Jonas Kemp, Katherine A Heller, Jude Chidubem Omeke, et al. AfriMed-QA: a Pan-African, multi-specialty, medical question-answering benchmark dataset. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 1948–1973, 2025. doi: 10.18653/v1/2025.acl-long.96.DOI
  36. [36]Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. MedExpQA: multilingual benchmarking of large language models for medical question answering. Artif Intell Med, 155:102938, 2024. doi: 10.1016/j.artmed.2024.102938.DOI
  37. [37]Andrew Zhang, Trinh T. L. Vuong, Sophia J. Wagner, Dongmin Bang, Tong Ding, Yuchang Su, Mary Cipperman, Christina S. Chen, Judith Lefkes, Ramin Kahidi, Ming Y. Lu, Cuiwei Tian, Tianhao Luo, Jingnan Ma, Rowland Pettit, Long Phi Le, and Faisal Mahmood. A general-purpose system one decision model for clinical records, 2026. URL https://arxiv.org/abs/2610.27099.
  38. [38]Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankar subbu. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 248–260, 2022. doi: 10.48550/arXiv.2203.14371. URL https://proceedings.mlr.press/v174/pal22a.html. arXiv:2203.14371.DOI
  39. [39]David Vilares and Carlos Gómez-Rodríguez. HEAD-QA: a healthcare dataset for complex reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 960–966, 2019. doi: 10.18653/v1/P19-1092.DOI
  40. [40]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021. doi: 10.48550/arXiv.2009.03300. URL https://iclr.cc/virtual/2021/poster/2962. arXiv:2009.03300.
  41. [41]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, et al. MMLU-Pro: a more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 37, pages 95266–95290, 2024. doi: 10.48550/arXiv.2406.01574. URL https://papers.nips.cc/paper_files/paper/2024/hash/ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets_and_Benchmarks_Track.html. arXiv:2406.01574.DOI
  42. [42]Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. MedXpertQA: benchmarking expert-level medical reasoning and understanding. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 80961–80990, 2025. doi: 10.48550/arXiv.2501.18362. URL https://proceedings.mlr.press/v267/zuo25a.html. arXiv:2501.18362.DOI
  43. [43]Hanjie Chen, Xiaozhuang Fang, Yash Singla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3563–3599, 2025. doi: 10.18653/v1/2025.naacl-long.182.
  44. [44]Sunjun Kweon, Byungjin Choi, Gyouk Chu, Junyeong Song, Daeun Hyeon, Sujin Gan, Jueon Kim, Minkyu Kim, Rae Woong Park, and Edward Choi. KorMedMCQA: multi-choice question answering benchmark for Korean healthcare professional licensing examinations, 2024. arXiv:2403.01469.arxiv.org/abs/2403.01469
  45. [45]Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, and Michael Lengzhi Li. Benchmarking large language models on CMExam – a comprehensive Chinese medical exam dataset. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 36, pages 52430–52452, 2023. doi: 10.48550/arXiv.2306.03030. URL https://papers.nips.cc/paper_files/paper/2023/hash/a48ad12d588c597f4725a8b84af647b5-Abstract-Datasets_and_Benchmarks.html. arXiv:2306.03030.DOI
  46. [46]Yanis Labrak, Adrien Bazoge, Richard Dufour, Béatrice Daille, Pierre-Antoine Gourraud, Emmanuel Morin, and Mickael Rouvier. FrenchMedMCQA: a French multiple-choice question answering dataset for medical domain. In Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), pages 41–46, 2022. doi: 10.18653/v1/2022.louhi-1.5.DOI
  47. [47]Ofir Ben Shoham and Nadav Rappoport. MedConceptsQA: open source medical concepts QA benchmark. Comput Biol Med, 182:109089, 2024. doi: 10.1016/j.compbiomed.2024.109089.DOI
  48. [48]Dina Denner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval. J Am Med Inform Assoc, 23(2):304–310, 2016. doi: 10.1093/jamia/ocv080.
  49. [49]Yosuke Yamagishi, Yuta Nakamura, Tomohiro Kikuchi, Yuki Sonoda, Hiroshi Hirakawa, Shintaro Kano, Satoshi Nakamura, Shouhei Hanaoka, Takeharu Yoshikawa, and Osamu Abe. Development of a large-scale dataset of chest computed tomography reports in Japanese and a high-performance finding classification model: dataset development and validation study. JMIR Med Inform, 13:e71137, 2025. doi: 10.2196/71137.DOI
  50. [50]Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Seval Nil Esirgun, İrem Dogan, Omer Faruk Durugol, Benjamin Hou, Suprosanna S. Wicheng Dai, Murong Xu, Hadrien Reynaud, Muhammed Furkan Dasdelen, Bastian Wittmann, et al. General foundation models from a multimodal dataset for 3D computed tomography. Nat Biomed Eng, 10(8):1610–1628, 2026. doi: 10.1038/s41551-025-01599-y.DOI
  51. [51]Bastien Le Guellec, Kokou Adambounou, Lisa C Adams, Thibault Agripnidis, Sung Soo Ahn, Radhia Ait Chalal, Tugba Akinci D’Antonioli, Philippe Amouyel, Henrick Andersson, Raphaël Bentegac, Claudio Benzoni, Antonino Andrea Blandino, Felix Busch, Elif Can, Riccardo Cau, et al. PARROT, an open multilingual radiology reports dataset. Eur J Radiol Artif Intell, 5:100066, 2026. doi: 10.1016/j.ejrai.2025.100066.DOI
  52. [52]José Luis Vázquez Noguera, Alejandro Torres-Hurtado, Helena Gómez-Adorno, Julio César Mello-Román, Enrique Javier Fleitas-Alvarez, Federico Fernando Espinola Schulze, Miguel Garcia-Torres, Carlos Domingo Méndez Gaona, Pedro Esteban Gardel Sotomayor, Silvia Vázquez Noguera, Norma Elizabeth Zaracho Amarilla, and Oxades Wilfrido Gamarra Esquivel. Mammography reporting dataset with BI-RADS system for natural language processing applications: addressing public data gaps in Spanish. Data Brief, 61:111761, 2025. doi: 10.1016/j.dib.2025.111761.DOI
  53. [53]Jenna Kefeli and Nicholas Tatonetti. TCGA-Reports: a machine-readable pathology report resource for benchmarking text-based AI models. Patterns (N Y), 5(3):100933, 2024. doi: 10.1016/j.patter.2024.100933.DOI
  54. [54]Mael Jullien, Marco Valentino, and André Freitas. SemEval-2024 task 2: safe biomedical natural language inference for clinical trials. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1947–1962, 2024. doi: 10.18653/v1/2024.semeval-1.271. URL https://aclanthology.org/2024.semeval-1.271/.DOI
  55. [55]Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: a dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, 2019. doi: 10.18653/v1/D19-1259.DOI
  56. [56]María Herrero-Zazo, Isabel Segura-Bedmar, Paloma Martínez, and Thierry Declerck. The DDI corpus: an annotated corpus with pharmacological substances and drug-drug interactions. J Biomed Inform, 46(5):914–920, 2013. doi: 10.1016/j.jbi.2013.07.011.
  57. [57]Isabel Segura-Bedmar, Paloma Martínez, and María Herrero-Zazo. SemEval-2013 task 9: extraction of drug-drug interactions from biomedical texts (DDIExtraction 2013). In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 341–350, 2013. URL https://aclanthology.org/S13-2056/.
  58. [58]Simon Baker, Ilona Silins, Yufan Guo, Imran Ali, Johan Högberg, Ulla Stenius, and Anna Korhonen. Automatic semantic classification of scientific literature according to the hallmarks of cancer. Bioinformatics, 32(3):432–440, 2016. doi: 10.1093/bioinformatics/btv585.DOI
  59. [59]Neema Kotonya and Francesca Toni. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7740–7754, 2020. doi: 10.18653/v1/2020.emnlp-main.623. URL https://aclanthology.org/2020.emnlp-main.623/.DOI
  60. [60]Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. Overview of the MEDIQA 2019 shared task on textual inference, question entailment and question answering. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 370–379, 2019. doi: 10.18653/v1/W19-5039. URL https://aclanthology.org/W19-5039/.DOI
  61. [61]Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. HealthBench: evaluating large language models towards improved human health, 2025. arXiv:2505.08775.arxiv.org/abs/2505.08775
  62. [62]Yakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang, Wei Nie, Jiaji Liu, Shaoting Zhang, Pengfei Liu, and Xiao-fan Zhang. DiagnosisArena: benchmarking diagnostic reasoning for large language models. In Findings of the Association for Computational Linguistics: ACL 2026, pages 3074–3098, 2026. doi: 10.18653/v1/2026.findings-acl.151. URL https://aclanthology.org/2026.findings-acl.151/. arXiv:2505.14107.DOI
  63. [63]Heidi L Rehm, Jonathan S Berg, Lisa D Brooks, Carlos D Bustamante, James P Evans, Melissa J Landrum, David H Ledbetter, Donna R Maglott, Christa Lese Martin, Robert L Nussbaum, Sharon P Oplen, Erin M Ramos, Stephen T Sherry, Michael S Watson, and ClinGen. ClinGen–the Clinical Genome Resource. N Engl J Med, 372(23):2235–2242, 2015. doi: 10.1056/NEJMsr1406261.DOI
  64. [64]Kirk Roberts, Dina Demner-Fushman, Ellen M Voorhees, Steven Bedrick, and William R Hersh. Overview of the TREC 2022 Clinical Trials track. In The Thirty-First Text REtrieval Conference Proceedings (TREC 2022), NIST Special Publication 500-338, 2022. doi: 10.6028/NIST.SP.500-338.trials-overview. URL https://trec.nist.gov/pubs/trec31/papers/Overview_trials.pdf.
  65. [65]Manas Gaur, Amanuel Alambo, Joy Prakash Sain, Ugur Kursuncu, Krishnaprasad Thirunarayan, Ramakanth Kavuluru, Amit Sheth, Randy Welton, and Jyotishman Pathak. Knowledge-aware assessment of severity of suicide risk for early intervention. In The World Wide Web Conference (WWW ’19), pages 514–525, 2019. doi: 10.1145/3308558.3313698.DOI
  66. [66]U.S. Food and Drug Administration. AccessGUDID: Global Unique Device Identification Database (GUDID), 2026. URL https://accessgudid.nlm.nih.gov/. Accessed October 2026.
  67. [67]Universidade Federal de São Paulo (UNIFESP). UNIFESP X-ray Body Part Classifier Competition, 2022. URL https://www.kaggle.com/datasets/felipkamitura/unifesp-xray-bodypart-classification. Accessed October 2026.
  68. [68]Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadihadi Bagheri, and Ronald M. Summers. ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3462–3471, 2017. doi: 10.1109/CVPR.2017.369.DOI
  69. [69]Johannes Rückert, Louise Bloch, Raphael Brü̈ngel, Ahmad Idrissi-Yaghir, Henning Schäfer, Cynthia S Schmidt, Sven Koitka, Obioma Pelka, Asma Ben Abacha, Alba G Seco de Herrera, Henning Müller, Peter A Horn, Felix Nensa, and Christoph M Friedrich. ROCov2: Radiology Objects in Context version 2, an updated multimodality image dataset. Sci Data, 11(1):688, 2024. doi: 10.1038/s41597-024-03496-6.DOI
  70. [70]Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, Kanae Kawai Miyake, Mia Gorovoy, and Daniel L Rubin. A curated mammography data set for use in computer-aided detection and diagnosis research. Sci Data, 4:170177, 2017. doi: 10.1038/sdata.2017.177.
  71. [71]Rebecca Sawyer-Lee, Francisco Gimenez, Assaf Hoogi, and Daniel Rubin. Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM), 2016. Data set.
  72. [72]Krzysztof Bartnik, Tomasz Bartczak, Mateusz Krzyziński, Krzysztof Korzeniowski, Krzysztof Lamparski, Piotr Węgrzyn, Eric Lam, Mateusz Bartkowiak, Tadeusz Wróblewski, Katarzyna Mech, Magdalena Januszewicz, and Przemysław Bięcek. WAW-TACE: a hepatocellular carcinoma multiphase CT dataset with segmentations, radiomics features, and clinical data. Radiol Artif Intell, 6(6):e240296, 2024. doi: 10.1148/ryai.240296.DOI
  73. [73]Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification. Sci Data, 10(1):41, 2023. doi: 10.1038/s41597-022-01721-8.DOI
  74. [74]Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kainis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Maman, Gabriel Chartrand, Fabian Löhöfler, Julian Walter Holch, Wieland Sommer, Felix Hofmann, Alexandre Hostettler, et al. The Liver Tumor Segmentation Benchmark (LiTS). Med Image Anal, 84:102680, 2023. doi: 10.1016/j.media.2022.102680.DOI
  75. [75]Anjany Sekuboyina, Malek E Husseini, Amirhossein Bayat, Maximilian Löffler, Hans Liebl, Hongwei Li, Giles Tetteh, Jan Kukačka, Christian Payer, Darko Štern, Martin Urschler, Maodong Chen, Dalong Cheng, Nikolas Lessmann, Yujin Hu, et al. VerSe: a vertebrae labelling and segmentation benchmark for multi-detector CT images. Med Image Anal, 73:102166, 2021. doi: 10.1016/j.media.2021.102166.DOI
  76. [76]Ádám Nárai, Petra Hermann, Tibor Auer, Péter Kemenczky, János Szalma, István Homolya, Eszter Somogyi, Pál Vági, Béla Weiss, and Zoltán Vidnyánszky. Movement-related artefacts (MR-ART) dataset of matched motion-corrupted and clean structural MRI brain scans. Sci Data, 9(1):630, 2022. doi: 10.1038/s41597-022-01694-8.DOI
  77. [77]Wonkyeong Lee, Fabian Wagner, Adrian Galdran, Yongyi Shi, Wenjun Xia, Ge Wang, Xuanqin Mou, Md Atik Ahamed, Abdullah Al Zubaer Imran, Ji Eun Oh, Kyungsang Kim, Jong Tak Baek, Dongheon Lee, Boowhi Hong, Philip Templeman, et al. Low-dose computed tomography perceptual image quality assessment. Med Image Anal, 99:103343, 2025. doi: 10.1016/j.media.2024.103343.
  78. [78]Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina G S Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, Justin Dong, Made K Prasadha, Jacqueline Pei, Magdalene Y L Ting, Jie Zhu, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell, 172(5):1122–1131.e9, 2018. doi: 10.1016/j.cell.2018.02.010.DOI
  79. [79]Jennifer S N Tang, Jarrel C Y Seah, Adil Zia, Jay Gajera, Richard N Schlegel, Aaron J N Wong, Dayu Gai, Shu Su, Tony Bose, Marcus L Kok, Alex Jarema, George N Harisis, Chris-Tin Cheng, Helen Kavnoudias, Wayland Wang, et al. CLIP, catheter and line position dataset. Sci Data, 8(1):285, 2021. doi: 10.1038/s41597-021-01066-8.DOI
  80. [80]Safwan S Halabi, Luciano M Prevedello, Jayashree Kalpathy-Cramer, Artem B Mamonov, Alexander Bilbily, Mark Cicero, Ian Pan, Lucas Araújo Pereira, Rafael Teixeira Sousa, Nitamar Abdala, Felipe Campos Kitamura, Hans H Thodberg, Leon Chen, George Shih, Katherine Andriole, et al. The RSNA pediatric bone age machine learning challenge. Radiology, 290(2):498–503, 2019. doi: 10.1148/radiol.2018180736.DOI
  81. [81]Pingjun Chen, Linlin Gao, Xiaoshuang Shi, Kyle Allen, and Lin Yang. Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss. Computerized Medical Imaging and Graphics, 75:84–92, 2019. doi: 10.1016/j.compmedimag.2019.06.002.DOI
  82. [82]Pingjun Chen. Knee osteoarthritis severity grading dataset, 2018. Data set.DOI
  83. [83]Tyler J Richards, Adam E Flanders, Errol Colak, Luciano M Prevedello, Robyn L Ball, Felipe Kitamura, John Mongan, Maryam Vazirabad, Hui-Ming Lin, Anne Kendall, Thanat Kanthawang, Salita Angkurawaranon, Erme Altınmakas, Hakan Dogan, Paulo Eduardo de Aguiar Kuriki, et al. The RSNA Lumbar Degenerative Imaging Spine Classification (LumbarDISC) dataset. Radiol Artif Intell, 8(2):e250480, 2026. doi: 10.1148/ryai.250480.DOI
  84. [84]Adam E Flanders, Luciano M Prevedello, George Shih, Safwan S Halabi, Jayashree Kalpathy-Cramer, Robyn Ball, John T Mongan, Anouk Stein, Felipe C Kitamura, Matthew P Lungren, Gagandeep Choudhary, Lesley Cala, Luiz Coelho, Monique Mogensen, Fanny Morón, et al. Construction of a machine learning dataset through collaboration: the RSNA 2019 Brain CT Hemorrhage Challenge. Radiol Artif Intell, 2(3):e190211, 2020. doi: 10.1148/ryai.2020190211.DOI
  85. [85]Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection 2018: a challenge hosted by the International Skin Imaging Collaboration (ISIC), 2019. arXiv:1902.03368.arxiv.org/abs/1902.03368
  86. [86]Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci Data, 5:180161, 2018. doi: 10.1038/sdata.2018.161.DOI
  87. [87]Noel C F Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection: a challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hosted by the International Skin Imaging Collaboration (ISIC). In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 168–172, 2018. doi: 10.1109/ISBI.2018.8363547. arXiv:1710.05006.DOI
  88. [88]Carlos Hernández-Pérez, Marc Comas, Sebastian Podlipnik, Noel C F Codella, Veronica Rotemberg, Allan C Halpern, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Brian Helba, Susana Puig, Veronica Vilaplana, and Josep Malvehy. BCN20000: dermoscopic lesions in the wild. Sci Data, 11(1):641, 2024. doi: 10.1038/s41597-024-03387-w.
  89. [89]Veronica Rotemberg, Nicholas Kurtansky, Brigid Betz-Stablein, Liam Caffery, Emmanouil Chousakos, Noel Codella, Marc Combalía, Stephen Dusza, Pascale Gussetra, David Gutman, Allan Halpern, Brian Helba, Harald Kittler, Kivanc Kose, Steve Langer, et al. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Sci Data, 8(1):34, 2021. doi: 10.1038/s41597-021-00815-z.DOI
  90. [90]Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Deshmukh, Vivek Sahasrabuddhe, and Fabrice Meriaudeau. Indian Diabetic Retinopathy Image Dataset (IDRiD): a database for diabetic retinopathy screening research. Data, 3(3):25, 2018. doi: 10.3390/data3030025.DOI
  91. [91]Ruhan Liu, Xiangning Wang, Qiang Wu, Ling Dai, Xi Fang, Tao Yan, Jaemin Son, Shiqi Tang, Jiang Li, Zijian Gao, Adrian Galdran, J M Poorneshwaran, Hao Liu, Jie Wang, Yerui Chen, et al. DeepDRiD: diabetic retinopathy-grading and image quality estimation challenge. Patterns (N Y), 3(6):100512, 2022. doi: 10.1016/j.patter.2022.100512.DOI
  92. [92]Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, Lina Jansen, Constantino Carlos Reyes-Aldasoro, Inka Zörníg, Dirk Jäger, Hermann Brenner, et al. Predicting survival from colorectal cancer histology slides using deep learning: a retrospective multicenter study. PLoS Med, 16(1):e1002730, 2019. doi: 10.1371/journal.pmed.1002730.DOI
  93. [93]Andrea Acevedo, Anna Merino, Santiago Alférez, Ángel Molina, Laura Boldú, and José Rodelar. A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data Brief, 30:105474, 2020. doi: 10.1016/j.dib.2020.105474.DOI
  94. [94]Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojciech Samek, and Tobias Schaeffter. PTB-XL, a large publicly available electrocardiography dataset. Sci Data, 7(1):154, 2020. doi: 10.1038/s41597-020-0495-6.DOI
  95. [95]Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Sci Data, 5:180251, 2018. doi: 10.1038/sdata.2018.251.
  96. [96]Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. SLAKE: a semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654, 2021. doi: 10.1109/ISBI48211.2021.9434010. arXiv:2102.09542.DOI
  97. [97]Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Development of a large-scale medical visual question-answering dataset. Commun Med (Lond), 4(1):277, 2024. doi: 10.1038/s43856-024-00709-z.DOI
  98. [98]Jeffrey D Rudie, Hui-Ming Lin, Robyn L Ball, Sabeena Jalal, Luciano M Prevedello, Savvas Nicolaou, Brett S Marinelli, Adam E Flanders, Kirti Magudia, George Shih, Melissa A Davis, John Mongan, Peter D Chang, Ferco H Berger, Sebastiaan Hermans, et al. The RSNA Abdominal Traumatic Injury CT (RATIC) dataset. Radiol Artif Intell, 6(6):e240101, 2024. doi: 10.1148/ryai.240101.DOI
  99. [99]Natalia Alves, Megan Schuurmans, Dawid Rutkowski, Anindo Saha, Pierpaolo Vendittelli, Nancy Obuchowski, Marjolein H Liedenbaum, Ingrid S Haldorsen, Anders Molven, Derya Yakar, Jeroen Geerdink, Sebastiaan van Koeverden, Deniece M Riviere, Wulphert Venderink, Robbert de Haas, et al. Artificial intelligence and radiologists in pancreatic cancer detection using standard of care CT scans (PANORAMA): an international, paired, non-inferiority, confirmatory, observational study. Lancet Oncol, 27(1):116–124, 2026. doi: 10.1016/S1470-2045(25)00567-4.DOI
  100. [100]Liang Jin, Jiancheng Yang, Kaiming Kuang, Bingbing Ni, Yiyi Gao, Yingli Sun, Pan Gao, Weiling Ma, Mingyu Tan, Hui Kang, Jiajun Chen, and Ming Li. Deep-learning-assisted detection and segmentation of rib fractures from CT scans: development and validation of FracNet. EBioMedicine, 62:103106, 2020. doi: 10.1016/j.ebiom.2020.103106.DOI

Paper details

vlnc:W_6br4vxjy
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents