betaFeedback

WorksComputer Science

The Value of System 1 Models in Medicine

Abstract

Many clinical decisions are a choice among a few fixed options, taken quickly and in large numbers. Such decisions also need an uncertainty estimate to know when to defer to a clinician. Generative language models answer slowly in free text and state their confidence poorly. In contrast, System 1 models answer zero-shot with a probability for each answer. However, their performance across clinical tasks is unknown. Four System 1 models, two proprietary and two with open weights, were evaluated zero-shot on 86 public tasks from 14 clinical domains in 20 languages, spanning examination questions and medical imaging. Tasks were assigned to six capability tiers before analysis. The primary outcome was the area under the receiver-operating-characteristic curve (AUROC). Secondary outcomes included calibration, language effects and comparisons with published results and generative models. Across 179,111 cases, the mean AUROC of the best model per task was 0.94 for medical knowledge, 0.93 for text understanding, 0.83 for clinical judgement from text, 0.89 for image perception, 0.84 for 2D image findings and 0.69 for volumetric findings. Imaging tasks scored 0.09 (95% CI, 0.06 to 0.11) lower than text tasks (P < 0.001). The best model matched or exceeded the best published result on 13 of 26 text tasks and on 8 of 35 imaging tasks. Triage nurses outperformed the best model (accuracy, 0.85 vs. 0.56). One decision took 0.34 to 0.44 seconds and 24 to 81 joules on open-weight models run locally. Accuracy on identical questions fell by up to 39 percentage points from English to Yoruba. Generative models of the same family were 11 to 76 times slower, with 3 to 5 percentage points higher balanced accuracy. System 1 models decide quickly, cheaply and with usable confidence when the answer is stated in a text or recalled from medical knowledge, but not when it must be judged or read from an image.

Introduction

Many clinical decisions represent choices among a few fixed options that must be made quickly and in bulk. A triage nurse assigns one of five acuity levels within minutes of arrival, a radiologist decides whether a CT pulmonary angiogram demonstrates embolism, and a coder selects one procedure code for each of thousands of cases. A model that makes such decisions must also state how certain it is, so that confident answers can be acted on and uncertain cases passed to a clinician. [1, 2, 3].

Generative large language models perform well on a series of medical tasks, [4, 5] but they write each answer token by token, often following a reasoning trace, which takes several seconds per decision and uses far more energy than a task-specific model. [6] The confidence they state in text is poorly calibrated and barely separates correct from incorrect answers. [7, 8] On the contrary, System 1 models receive a state, such as a report, an image or both, together with one or more typed questions, and return a probability for every answer option in a single forward pass without generating text. Their name refers to dual-process theory, which separates fast, intuitive System 1 judgement from slow, deliberate System 2 reasoning, the mode that generative models resemble when they reason in text. [9, 10]

The open question is which clinical decisions such a model can be trusted with. The answer to a clinical case can be stated in a report, recalled from medical knowledge, judged from several findings, or seen in an image. Of 519 studies of large language models in health care, 44.5% tested medical knowledge such as licensing-examination questions, and 5% used real patient-care data.2 Where other tasks were tested, performance fell from more than 86% on licensing-examination questions3 to lower accuracy on real clinical cases and conversations.4 On radiologic images, GPT-4V identified the imaging modality in all 515 images but detected abnormalities in 0% to 90%, depending on the finding,5 and multimodal models scored lower on volumetric than on two-dimensional images.6 If performance depends on where the answer lies, a single benchmark score misrepresents a model, a construct-validity problem known from general AI benchmarks.7

To evaluate the value of System 1 models in medicine, we tested four models on 86 tasks from 14 clinical domains, with the tasks ordered in six capability tiers from medical knowledge to volumetric findings. We measured which clinical decisions the models can take on, whether their probabilities are calibrated, how they perform outside English, and what their speed costs in time, money, energy and accuracy, against a generative model of the same family.

METHODS

Study Design and Tasks. This cross-sectional evaluation used public, de-identified datasets under their respective licences. Candidate tasks came from a taxonomy of 87 clinical decision tasks in 18 domains and from further public datasets with a fixed answer format (Supplementary Appendix, Section S1). A task was included when its data and reference labels were public and its licence allowed transfer to a model hosted on a commercial API. Eighty-six tasks from 14 clinical domains met these criteria. Table S1 lists every task with the publication of its dataset. Eight of them are language versions of another task and were analysed only for language effects, which left 78 primary tasks (Figure 1). Each task used the official test split or a seeded random sample of up to 1,000 cases, in some tasks enriched for rare classes, and volumetric tasks used 81 to 436 studies. The tasks comprised 179,111 cases, with the reference standard released by the dataset authors.

Figure 1. Study Design. Panel A shows how a System 1 model answers. Text, one or more images, or both enters the model with typed questions, and the model returns a probability for each option in one forward pass. The generative model of the same family served as comparator on 32 tasks. Panel B shows the 86 tasks by capability tier and input type. Hatched tiles are language versions of another task and were analysed for language effects only.

No.TaskDatasetTierInputLanguageCasesQuestions
001USMLE-style MCQ (MedQA, 4 options)MedQA [23]Medical knowledgetextEnglish1,2731
002Indian medical entrance exam MCQ (MedMCQA)MedMCQA [38]Medical knowledgetextEnglish4,1831
003Spanish healthcare specialisation exam MCQ (HEAD-QA)HEAD-QA [39]Medical knowledgetextEnglish2,6751
004Spanish healthcare specialisation exam MCQ, Spanish original (HEAD-QA)HEAD-QA [39]Medical knowledgetextSpanish2,6751
005Medical subsets of MMLUMMLU [40]Medical knowledgetextEnglish1,0891
006Health questions of MMLU-ProMMLU-Pro [41]Medical knowledgetextEnglish8181
007Expert-level medical MCQ, text (MedXpertQA Text)MedXpertQA [42]Medical knowledgetextEnglish2,4501
008Hard clinical case MCQ (Medbullets op5)Medbullets op5 [43]Medical knowledgetextEnglish3081
009Pan-African medical MCQ (AfriMed-QA v2)AfriMed-QA v2 [35]Medical knowledgetextEnglish3,6001
010Multilingual clinical-case MCQ (MedExpQA)MedExpQA [36]Medical knowledgetextSeveral5001
011Korean healthcare licensing exam MCQ (KorMedMCQA)KorMedMCQA [44]Medical knowledgetextKorean3,0091
012Chinese national medical licensing exam MCQ (CMExam)CMExam [45]Medical knowledgetextChinese6,6071
013Chinese medical licensing exam (MedQA Mainland)MedQA [23]Medical knowledgetextChinese3,4261
014Taiwanese medical licensing exam MCQ (MedQA Taiwan)MedQA [23]Medical knowledgetextChinese1,4131
015French pharmacy exam MCQ (FrenchMedMCQA)FrenchMedMCQA [46]Medical knowledgetextFrench3211
016Medical subsets of MMLU in Arabic (Global-MMLU)Global-MMLU [34]Medical knowledgetextArabic1,0891
017Medical subsets of MMLU in Hindi (Global-MMLU)Global-MMLU [34]Medical knowledgetextHindi1,0891
018Medical subsets of MMLU in Japanese (Global-MMLU)Global-MMLU [34]Medical knowledgetextJapanese1,0891
019Medical subsets of MMLU in Swahili (Global-MMLU)Global-MMLU [34]Medical knowledgetextSwahili1,0891
020Medical subsets of MMLU in Yoruba (Global-MMLU)Global-MMLU [34]Medical knowledgetextYoruba1,0891
021Medical subsets of MMLU in German (Global-MMLU)Global-MMLU [34]Medical knowledgetextGerman1,0891
022Drug class from ATC code (MedConceptsQA)MedConceptsQA [47]Medical knowledgetextEnglish18,8061
023Procedure description from ICD-9-CM code (MedConceptsQA)MedConceptsQA [47]Medical knowledgetextEnglish13,7661
024Chest radiograph report findings (OpenI / IU X-ray)OpenI / IU X-ray [48]Text understandingtextEnglish3,83213
025Chest CT findings from Japanese reports (CT-RATE-JPN)CT-RATE-JPN [49, 50]Text understandingtextJapanese15018
026Modality and ICD-10 chapters of multilingual reports (PARROT)PARROT [51]Text understandingtextSeveral2,73813
027Lung cancer T category from radiology reports (NTCIR-17 RR-TNM)NTCIR-17 RR-TNM [31]Text understandingtextJapanese2431
028BI-RADS category from breast imaging reports (Paraguay corpus)Paraguay reports [52]Text understandingtextEnglish1,0001
029Pathologic T stage from a pathology report (TCGA)TCGA reports [32, 53]Text understandingtextEnglish1,0001
030Clinical-trial report inference (NLI4CT, SemEval-2024 Task 2)NLI4CT [54]Text understandingtextEnglish5001
031Biomedical research QA (PubMedQA PQA-L)PubMedQA PQA-L [55]Text understandingtextEnglish5001
032Drug-drug interaction type (DDI-2013)DDI-2013 [56] [57]Text understandingtextEnglish1,0001
033Hallmarks of cancer in abstracts (HoC)HoC 58Text understandingtextEnglish37110
034Public-health claim veracity (PUBHEALTH)PUBHEALTH 59Text understandingtextEnglish1,2331
035Consumer health question entailment (MEDIQA-RQE 2019)MEDIQA-RQE 2019 60Text understandingtextEnglish2301
036Rubric criterion judging (HealthBench meta-evaluation)HealthBench 61Text understandingtextEnglish1,0001
037ED triage acuity (KTAS)KTAS [28]Clinical judgement from texttextEnglish1,2671
038Diagnosis selection from a published case (DiagnosisArena)DiagnosisArena 62Clinical judgement from texttextEnglish9151
039ACMG/AMP five-tier variant classification (ClinGen)ClinGen 63Clinical judgement from texttextEnglish1,0001
040Patient-trial eligibility (TREC Clinical Trials 2021/2022)TREC CT 2021/22 64Clinical judgement from texttextEnglish5001
041Suicide-risk severity of a Reddit user (Reddit C-SSRS)Reddit C-SSRS 65Clinical judgement from texttextEnglish5001
042MRI safety status of a device (FDA AccessGUDID)FDA AccessGUDID 66Clinical judgement from texttextEnglish1,0001
043Radiograph body partUNIFESP body part 67Image perception2DEnglish5931
044Chest radiograph view AP vs PA (NIH ChestX-ray14)NIH ChestX-ray14 68Image perception2DEnglish7161
045Imaging modality (ROCOv2)ROCOv2 69Image perception2DEnglish7941
No.TaskDatasetTierInputLanguageCasesQuestions
046Mammography view CC vs MLO (CBIS-DDSM)CBIS-DDSM 70;71Image perception2DEnglish6451
047CT contrast phase (series level, WAW-TACE)WAW-TACE 72Image perception3DEnglish1601
048Contrast status and phase (single slice, WAW-TACE)WAW-TACE 72Image perception2DEnglish9602
049Abdominal organ on CT, 3D (OrganMNIST3D)OrganMNIST3D 73;74Image perception2D multi-viewEnglish6101
050Abdominal organ on coronal CT (OrganCMNIST)OrganCMNIST 73;74Image perception2DEnglish8,2161
051Abdominal organ on sagittal CT (OrganSMNIST)OrganSMNIST 73;74Image perception2DEnglish8,8271
052Marked vertebra level on CT (VerSe)VerSe 75Image perception2D multi-viewEnglish2192
053MRI motion artefact severity (MR-ART)MR-ART 76Image perception3DEnglish4361
054Diagnostic image quality of low-dose CT (LDCTIQAC 2023)LDCTIQAC 2023 77Image perception2DEnglish1,0001
055Chest radiograph multi-label findings (NIH ChestX-ray14)NIH ChestX-ray14 682D image findings2DEnglish71614
056Pneumonia on paediatric chest radiographs (PneumoniaMNIST)PneumoniaMNIST 73;782D image findings2DEnglish6241
057Line and tube position on chest radiographsRANZCR CLiP 68;792D image findings2DEnglish49711
058Paediatric bone-age bandRSNA bone age [80]2D image findings2DEnglish5001
059Knee osteoarthritis KL gradeOAI [82] [81, 82]2D image findings2DEnglish1,0001
060Lumbar degenerative stenosis gradeRSNA lumbar 2024 [83]2D image findings2D multi-viewEnglish1,0005
061Head CT haemorrhage and subtype (slice)RSNA ICH 2019 [84]2D image findings2DEnglish1,0006
062BI-RADS assessment from mammograms (CBIS-DDSM)CBIS-DDSM 70;712D image findings2D multi-viewEnglish7041
063Breast density category (CBIS-DDSM)CBIS-DDSM 70;712D image findings2DEnglish6431
064Dermoscopic lesion diagnosis (HAM10000 / ISIC 2018 Task 3)HAM10000 [85, 86]2D image findings2DEnglish5151
065Dermoscopic lesion diagnosis (ISIC 2019)ISIC 2019 [86, 87, 88]2D image findings2DEnglish8,2381
066Melanoma vs other lesion (SIIM-ISIC 2020)SIIM-ISIC 2020 [89]2D image findings2DEnglish33,1261
067Diabetic retinopathy severity (IDRiD)IDRiD [90]2D image findings2DEnglish5161
068Diabetic retinopathy grade (RetinaMNIST)RetinaMNIST [91] [73]2D image findings2DEnglish4001
069Retinal OCT diagnosis (OCTMNIST)OCTMNIST 73;782D image findings2DEnglish1,0001
070Colorectal H&E tissue type (CRC-VAL-HE-7K)CRC-VAL-HE-7K [92]2D image findings2DEnglish9001
071Blood cell type (BloodMNIST)BloodMNIST [93] [73]2D image findings2DEnglish3,4211
07212-lead ECG superclasses (PTB-XL, rendered as image)PTB-XL [94]2D image findings2DEnglish2,1985
073Radiology VQA, closed yes/no questions (VQA-RAD)VQA-RAD [95]2D image findingstext+imageEnglish2511
074Radiology VQA, closed yes/no questions (SLAKE)SLAKE [96]2D image findingstext+imageEnglish3551
075Radiology VQA in Chinese, closed yes/no questions (SLAKE)SLAKE [96]2D image findingstext+imageChinese3471
076Biomedical figure VQA (PMC-VQA)PMC-VQA [97]2D image findingstext+imageEnglish2,0001
077Expert-level medical MCQ with images (MedXpertQA MM)MedXpertQA [42]2D image findingstext+imageEnglish2,0001
078Head CT haemorrhage and subtype (scan level, CQ500)CQ500 [27]Volumetric findings3DEnglish816
079Critical head CT finding (CQ500)CQ500 [27]Volumetric findings3DEnglish814
080Chest CT multi-abnormality (CT-RATE)CT-RATE [50]Volumetric findings3DEnglish10018
081Pulmonary embolism on CTPARSNA PE 2020 [24]Volumetric findings3DEnglish1506
082Cervical spine fracture on CTRSNA C-spine 2022 [25]Volumetric findings3DEnglish2008
083Traumatic abdominal organ injury on CTRSNA trauma 2023 [98]Volumetric findings3DEnglish2008
084Pancreatic ductal adenocarcinoma on contrast-enhanced CT (PANORAMA)PANORAMA [99]Volumetric findings3DEnglish3001
085Clinically significant prostate cancer on bpMRI (PI-CAI)PI-CAI [26]Volumetric findings3DEnglish2001
086Rib fracture type (FractureMNIST3D, RibFrac)FractureMNIST3D [100] [73]Volumetric findings2D multi-viewEnglish2401

Table S1. Tasks. Tasks 004, 016, 017, 018, 019, 020, 021, 075 are language versions of another task. MCQ denotes multiple-choice questions, VQA visual question answering, and ED emergency department.

Prior to result summarisation, each task was assigned to one of six capability tiers. Medical knowledge covered examination questions and terminology, text understanding covered information stated in a report, abstract or sentence, and clinical judgement from text covered decisions that integrate patient information, such as triage acuity or variant class. The three imaging tiers included image perception (acquisition, anatomy and image quality), two-dimensional (2D) image findings, and volumetric findings on CT or MRI.

System 1 Models. Four System 1 models were evaluated through the OpenRouter decision interface, each in the latest version available in October 2026. Clef (Cloudflare, 27 billion parameters) and Clef-flash (Cloudflare, 9.5 billion parameters) have open weights, and GPT-6 Luna Decisions (OpenAI) and Jev 1.14 (TypeSafe AI) are proprietary. A request held a state of text and up to four images, and one or more typed questions with a yes-or-no answer, a choice of one of nn options, or an ordered score. The model returned a probability for every option. All questions of a case were sent in one request, zero-shot, with a one-sentence description of the input. Volumetric studies were read as axial slices at a fixed spacing, with one request per slice. The study-level probability was the maximum over slices for yes-or-no questions and the mean over slices for other questions. On four 2D imaging tasks, Jev 1.14 received higher-resolution versions of the same images since its interface accepts larger images. Every request was logged as sent.

Generative Comparison. Clef and Clef-flash were compared with their generative backbones, Qwen3.8-27B8 and Qwen3.5-9B,9 and GPT-6 Luna Decisions with the generative GPT-6 Luna. The comparison covered 32 text and 2D imaging tasks. For each task, 100 randomly selected cases were sent to the generative model with the same state and questions as the System 1 model had received, including an instruction to choose one option per question, and an output schema restricted to the answer options. Default reasoning settings were kept. An answer that failed or could not be parsed counted as wrong.

Local Runs and Energy. Clef and Clef-flash were additionally run locally from the released checkpoints, on a single NVIDIA RTX 5090 GPU. Local Clef’s answers were compared with the hosted versions, which confirmed that the two agreed before the local runs were utilised. Energy was measured on a sub-sample of 42 tasks and it followed the AI Research Label of the GEDARS framework for reporting the environmental footprint of clinical AI.10 The measurements come from the GPU driver’s cumulative energy counter through NVML (power usage effectiveness taken as 1.0 for a single workstation GPU). CodeCarbon (v3.3.1) ran alongside to convert energy to CO2_2e at the local grid carbon intensity (Greece, 337 g CO2_2e per kWh). Energy is reported per decision and summarised as the median and IQR over tasks. Latency was measured per request. The providers of the hosted models (GPT-6 Luna Decisions, Jev 1.14) publish no per-request energy figures for them.

Reference Results. For every task, published results on the same dataset were collected from the original dataset or challenge publication, from official leaderboards, and from a search of PubMed and the web for later work up to October 2026. Every value was checked against its source and classified as a task-specific model, a generative language model, a rule-based classifier, or clinicians and expert annotators. A published result was compared only when it used the same label set and metric on the matching questions. The highest such result per task is reported as the best published result.

Outcomes. The primary outcome was the area under the receiver-operating-characteristic curve (AUROC), computed per question as a macro average of one-versus-rest curves for more than two options, and averaged over the questions of a task. Secondary outcomes were the balanced accuracy of the most probable option, expressed relative to chance as (BA−1/k)/(1−1/k)(\mathrm{BA}-1/k)/(1-1/k) for kk classes so that 0 is chance and 1 is perfect, the top-label expected calibration error (ECE), [1, 21] selective accuracy after withholding the least confident answers, [22] cost per 1,000 requests at provider prices, latency per request and energy per decision.

Statistical Analysis. Any question that a model refused or failed was scored as wrong, with the label prevalence as its probability. Task-level 95% confidence intervals (CIs) were estimated using 1,000 bootstrap resamples of cases. Task estimates were pooled within each tier and model using DerSimonian–Laird random-effects models. Two contrasts were specified in advance, imaging versus text tiers and a linear trend across the three imaging tiers, and both were estimated by random-effects meta-regression. AUROC was also summarised by question type. The accuracy of each language version on identical items was compared with English by paired bootstrap and the exact McNemar test. System 1 and generative models were compared by the mean task-level difference in balanced accuracy, with 95% CIs from a task bootstrap. Secondary analyses were not adjusted for multiplicity. Analyses used Python 3.13, pandas 3.0, NumPy 2.5, SciPy 1.18 and scikit-learn 1.9. Code and all model outputs are available in the project repository at https://github.com/twin-x-eu/system1-models-in-medicine.

Results

Tasks and Models. The 86 tasks covered 20 languages and yielded 2.4 million model answers across 179,111 cases (Figure 1). GPT-6 Luna Decisions refused 4,008 case–question pairs (1.3%), Clef and Clef-flash each failed on one pair, and Jev 1.14 answered every question. Refused and failed questions were scored as wrong. Jev 1.14 was the best model on 45 of 78 tasks. Results per task are given in Table S2.

Capability Gradient. For the best model on each task, the random-effects mean AUROC was 0.94 (95% CI, 0.92 to 0.97) for medical knowledge, 0.93 (95% CI, 0.91 to 0.94) for text understanding, 0.89 (95% CI, 0.84 to 0.94) for image perception, 0.84 (95% CI, 0.81 to 0.87) for 2D image findings and 0.69 (95% CI, 0.60 to 0.79) for volumetric findings (Figure 2). Imaging tasks scored 0.09 (95% CI, 0.06 to 0.11) lower than text tasks (P<0.001P < 0.001), and AUROC fell by 0.09 (95% CI, 0.06 to 0.13) per tier from image perception to volumetric findings (P<0.001P < 0.001). Both contrasts were significant for each of the four models (Figure 2B). Clinical judgement from text reached 0.83 (95% CI, 0.73 to 0.93), lower than the other two text tiers. Chance-corrected balanced accuracy fell from 0.74 for medical knowledge to 0.20 for volumetric findings, and its trend across the imaging tiers was not significant (P=0.10P = 0.10, Figure 2C).

The best model reached an AUROC of 0.99 on licensing-examination questions (MedQA) [23] and 1.00 on findings stated in chest radiograph reports. On volumetric imaging it reached 0.54 for pulmonary embolism, [24] 0.59 for cervical spine fracture, [25] 0.59 for clinically significant prostate cancer [26] and 0.48 for rib fracture type, and four of nine volumetric tasks stayed below 0.60. Intracranial haemorrhage on head CT reached 0.90. [27] On imaging tasks, questions that assigned the whole image to one of several options reached a median AUROC of 0.87, yes-or-no questions on the presence of a finding reached 0.74, and ordered grades 0.71. On text tasks, yes-or-no questions reached 0.99.

Comparison with Published Results and Clinicians. A published result on the same dataset, label set and metric was available for 61 tasks (Figure 3). On text tasks, the best System 1 model matched (within 0.01) or exceeded the best published result in 13 of 26 tasks, with a median difference of −0.01. On imaging tasks it did so in 8 of 35 tasks, with a median difference of −0.17. Clinicians or expert annotators outperformed the best model in 6 of the 10 tasks for which such a comparison existed. Triage nurses assigned the emergency acuity level with an accuracy of 0.85, compared with 0.56 for the best model, [28] and a points-based rule classified genetic variants with an accuracy of 0.95, compared

with 0.73. [29, 30] Zero-shot System 1 models are therefore competitive with purpose-built systems on text, but not on images or where a codified rule already exists.

Calibration and Selective Answering. The median top-label ECE across models was 0.05 to 0.07 on text tasks and 0.11 to 0.17 on imaging tasks (Figure 4A and 4B). On imaging tasks, GPT-6 Luna Decisions was over-confident, and about 80% of its most confident answers were correct. Withholding the less confident half of the answers raised accuracy on text tasks from between 0.76 and 0.83 to between 0.96 and 0.99, and on imaging tasks from between 0.66 and 0.70 to between 0.79 and 0.91 (Figure 4C).

Language. On the same 1,089 medical examination questions, accuracy in every language version was lower than in English for every model (Figure 4D). The decline was smallest in German and largest in Yoruba, where it ranged from 20.1 (95% CI, 17.6 to 22.5) percentage points for Jev 1.14 to 39.4 (95% CI, 36.3 to 42.4) for Clef-flash, whose accuracy fell from 0.93 to 0.54. The performance measured in English therefore overstates what these models deliver in a low-resource language.

System 1 and Generative Models. On identical cases, the balanced accuracy of each System 1 model was lower than that of the generative model of its family, by 4.3 (95% CI, 2.4 to 6.3) percentage points for GPT-6 Luna Decisions, 3.3 (95% CI, 0.0 to 6.5) for Clef and 4.8 (95% CI, 1.0 to 8.5) for Clef-flash, over 32 tasks each (Figure 5A). The difference was smaller on imaging tasks, where Clef-flash and Qwen3.5-9B differed by 0.4 (95% CI, −1.5 to 2.3) percentage points. Each System 1 model scored higher than its generative counterpart on 8 or 9 of the 32 tasks. Generative models required a median of 4.8 to 30.3 seconds per request, compared with 0.34 to 0.44 seconds for System 1 models, and incurred 2- to 22-fold higher costs (Figure 5B and 5C).

Environmental Impact of System 1 Models. Energy was measured directly on the two open-weight models, Clef and Clef-flash, both run locally on one RTX 5090 GPU over 42 of the tasks and 117,263 decisions each (Methods). The median energy of one decision was 23.5 joules for Clef-flash (IQR, 17.2 to 51.3) and 80.9 joules for Clef (IQR, 68.4 to 125.5). That is 6.5 to 22.5 watt-hours of GPU-board energy per 1,000 decisions and, including GPU and RAM and at the carbon intensity of the local grid, 2.8 to 12.1 g CO2_2e per 1,000 decisions (Figure 6). Across capability tiers, the median energy per 1,000 decisions was lowest for medical knowledge (4.8 watt-hours for Clef-flash and 19.2 for Clef) and highest for text understanding (19.3 and 47.6). One kilowatt-hour buys roughly 153,000 decisions from Clef-flash and 44,500 from Clef. Decisions that included an image cost 1.6 to 2.3 times as much energy as text-only decisions, and the length of the input mattered more than its type. On the three tasks with the longest documents, the energy of a decision rose to 2.1 to 5.0 times the median for that model. The two models together took 234,526 decisions for 4.3 kilowatt-hours of GPU-board energy. The energy of a single System 1 decision is therefore small enough that it is not the limiting factor in deploying one at scale.

Figure 2. Panel A shows the AUROC of each primary task (columns) for the four System 1 models, with tasks grouped by capability tier and sorted by the best model. Grey vertical lines span the four models. Horizontal bars and shaded bands show the random-effects mean AUROC of the best model per task with its 95% CI, and the weakest task of each tier is labelled. Panels B and C show the random-effects mean per tier and model with 95% CIs for AUROC and for chance-corrected balanced accuracy, where 0 is chance and 1 is perfect. Refused or failed questions were scored as wrong.

Local versus API Inference. Clef and Clef-flash have open weights and can run inside a hospital network, so patient data do not need to leave the institution. Run locally, they reproduced the hosted interface closely. On chest radiograph report findings, local Clef gave a mean AUROC of 0.99 equal to the version inferenced through the API, and agreed with it on 99.7% of answers, while on visual question answering, the values were 0.947 against 0.942 and 97.2% agreement.

Discussion

System 1 models answered medical examination questions and identified findings described in radiology reports with high accuracy, with inference times below half a second and for a few joules per decision. On text-based tasks, predicted probabilities were well calibrated, closely reflecting observed accuracy. However, performance declined on tasks requiring the integration of multiple findings, such as assigning acuity levels in emergency triage. On imaging tasks, they accurately identified imaging modalities and anatomical regions but frequently failed to detect specific abnormalities. Performance on volumetric CT and MRI tasks was particularly limited, approaching chance levels in four of nine tasks.

For information extraction from clinical text, the case for zero-shot System 1 models is strong. The best System 1 model matched or exceeded the best published result on 13 of 26 text tasks without any training on the task, including results from models that had been fine-tuned for that task. For the T staging of lung cancer from CT reports, the best System 1 model reached an accuracy of 0.78, compared with 0.67 for fine-tuned Japanese BERT models.[31] For the pathological T stage from pathology reports, it reached an AUROC of 0.96, compared with 0.94 for a fine-tuned Clinical-BigBird model.[32] A zero-shot System 1 model could therefore replace models that require task-specific fine-tuning. Each decision took less than half a second and 23.5 to 80.9 joules, with costs ranging from 0.02 to 0.16 USD per 1,000 requests. Withholding the 50% of predictions with the lowest confidence increased accuracy on text-based tasks to 0.96 to 0.99. These findings suggest that System 1 models could support large-scale report labelling, information extraction, and literature screening, while referring uncertain cases to clinicians or slower, more computationally intensive models.[3, 22] Clef and Clef-flash have openly available weights and can be deployed within hospital networks.

Figure 3. System 1 Models, Published Results and Clinicians. Panel A shows text tasks and Panel B imaging tasks. For each task with a published result on the same dataset, label set and metric, the best of the four System 1 models (coloured marker) is shown with the best published result (open circle), which can be a task-specific model, a generative language model or a rule, and with clinicians or expert annotators (open triangle). Values of the System 1 models are computed on the questions that match the published result and can differ from the task-level AUROC. Acc denotes accuracy and BA balanced accuracy.

Accuracy declined on tasks requiring clinical judgement, such as assigning triage acuity or classifying variants, even when all relevant information was provided in the input. System 1 models generate predictions in a single forward pass, relying on knowledge encoded in their weights. Unlike generative models that produce explicit reasoning traces,[33] they lack intermediate reasoning steps through which multiple findings can be evaluated and integrated before reaching a decision. Across the five tasks requiring clinical judgement, the generative backbones of Clef and Clef-flash achieved accuracies 11 and 15 percentage points higher, respectively, compared with differences of 3 and 5 percentage points across all 32 evaluated tasks. Enabling System 1 models to generate reasoning traces before making predictions could help narrow this performance gap.

An additional limitation of most medical benchmarks is their predominantly English-language content. Similarly, most language models, including System 1 models, are trained primarily on English-language data.[34, 35] Reflecting this, the System 1 models were most accurate in English. On the same 1,089 examination questions, accuracy fell by 2 to 7 percentage points in German, a high-resource language, and by 20 to 39 percentage points in Yoruba, a low-resource language. Generative models lose accuracy in low-resource languages in a similar manner. [35, 36]

Figure 4. Calibration and Language. Panels A and B show the share of correct answers against the probability the model gave its answer, pooled over text tasks in Panel A and over imaging tasks in Panel B. The diagonal marks perfect calibration. Panel C shows, for each model, the accuracy of all answers (open marker) and of its most confident half (filled marker) on text and imaging tasks. Panel D shows accuracy on the same 1,089 medical examination questions in seven languages.

Figure 5. System 1 and Generative Models. Panel A shows, for each task, the balanced accuracy of the System 1 model minus that of the generative model of the same family on identical cases, with the mean and 95% CI (task bootstrap) for text and imaging tasks. The pairs are GPT-6 Luna Decisions and GPT-6 Luna, Clef and Qwen3.8-27B, and Clef-flash and Qwen3.5-9B. Panels B and C show the cost per 1,000 requests and the median latency per request on logarithmic scales, with filled markers for System 1 models and open markers for generative models.

Most System 1 models also accept images, but read medical images far less accurately than models trained for the task. The best System 1 model matched or exceeded the best published result in 8 of 35 imaging tasks, compared with 13 of 26 text tasks, with a median difference of −0.17 on imaging tasks. For clinically significant prostate cancer on MRI, it reached an AUROC of 0.59, compared with 0.86 for radiologists. [26] Calibration was also worse on imaging tasks, and only about 80% of the most confident answers of GPT-6 Luna Decisions were correct. Volumetric imaging was most challenging as the current interface of System 1 models accepts at most four images per request and no volumes. CT and MRI studies were therefore sent as single slices every 3 to 10 mm, and a small finding can lie between two slices. The choice of pooling rule over slices did not change the results (Supplementary Appendix, Section S6.3). Furthermore, radiographic images had to be downsampled, for example mammograms at 1,024 pixels on the long side, at which microcalcifications cover few pixels. Generative models also lose accuracy when volumes are given as image sequences, [16] which indicates a property of how vision-language models process images in general.

The open-weight decision models Clef and Clef-flash are based on the Qwen family of generative models, allowing a direct comparison of the two model types. The decision models and their backbones did not differ in accuracy, however the decision models were better calibrated, with a median expected calibration error of 0.12 for Clef-flash against 0.20 for its backbone, and 0.07 for Clef against 0.13. Against the generative backbone with reasoning, the System 1 models were 3 to 5 percentage points less accurate and 11 to 76 times faster.

This study has limitations. Models were used zero-shot with a one-sentence description of the input. Task-specific prompts and fine-tuning were not tested. A System 1 model trained on the records of 3.6 million patients from one health system, CS1-4B, was recently reported to outperform open decision models of up to 27 billion parameters on 13 of 14 tasks in held-out patients from that system.[37] Its weights have not yet been released, so it could not be included here. Public datasets can be part of the training data, which raises results on text tasks in particular. Larger frontier generative models were not tested and are likely more accurate on many tasks, but they are slower and more expensive than the generative models compared herein.

Energy of Locally Run System 1 Models

Figure 6. Panel A shows the GPU energy per 1,000 requests and Panel B the GPU time per request of Clef (8-bit) and Clef-flash (bf16), run locally on one NVIDIA RTX 5090, for each primary task by capability tier. Energy is the whole-board GPU energy. Bars show the median over tasks, and nn is the number of tasks per tier. No clinical-judgement task was run locally.

For predefined clinical decisions on text, in particular information extraction, System 1 models are ready for use, with fast, inexpensive and well-calibrated answers. They are as accurate as their backbones but better calibrated, and open weights let a hospital run them sustainably on its own data. However, they are currently limited in questions requiring clinical reasoning and in interpreting medical images, in particular CT and MRI volumes.

Acknowledgements

Funded by the European Union under Horizon Europe grant agreement No 101287679 (TWIN-X). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Health and Digital Executive Agency (HaDEA). Neither the European Union nor the granting authority can be held responsible for them.

Supplementary Appendix

The Value of System 1 Models in Medicine

Task Selection

The taxonomy splits radiology into five layers (acquisition and metadata, image quality, anatomy, findings, and examination or patient level). Table S1 lists the evaluated tasks.

Requests

The state contained the case text as released, such as the question stem, the report sections or the triage fields, or a one-sentence description of the input together with the image. Each option of a choice question had a short description, and each level of an ordered score was described in words. Images larger than 4 MiB were downsampled by half until they were below 4 MiB. For Jev 1.14, images with a side below 280 or above 1,568 pixels were rescaled. The four 2D imaging tasks with higher-resolution images for Jev 1.14 were melanoma in ISIC 2020, paediatric pneumonia, retinal OCT and diabetic retinopathy grade in RetinaMNIST. On OpenRouter, Clef and Clef-flash were routed only to the Cloudflare provider, because another provider ignored images. When a model refused a question, the question was removed, the request was repeated with the remaining questions, and the refusal was logged. Volumetric slices were taken within the relevant body region, with the window named in the text.

Scoring

For yes-or-no questions, the AUROC used the probability of the positive option. For other questions, the macro average included only the options present in the reference. Balanced accuracy was the recall of the most probable option, averaged over the reference classes. Probabilities were normalised to sum to one, because two models round probabilities to two decimals. The expected calibration error used 10 equal-width bins of the top-label probability. When a task shared its requests with another task (chest radiograph findings and view, head CT haemorrhage and critical findings), each question was counted in one task only.

Generative Comparison

The generative models replied in JSON. Qwen3.8-27B answered the first 50 of the 100 cases per task. Reasoning tokens were counted and billed.

Reference Results

Table S5 lists every published result with its type.

No.TaskTypeReferenceSourceMetricRef.Best S1
001MedQA (USMLE)Generative language modelo1-preview, zero-shotNori 2024accuracy0.960.89
001MedQA (USMLE)Task-specific modelMed-Gemini-L 1.0 (self-training, uncertainty-guided web search)Saab 2024accuracy0.910.89
002MedMCQA (India)Generative language modelo1-preview, zero-shotNori 2024accuracy0.840.75
002MedMCQA (India)Task-specific modelMed-PaLM 2 (ensemble refinement)Singhal 2025accuracy0.720.75
003HEAD-QA (English)Generative language modelGPT-5, zero-shotYang 2026accuracy0.900.90
005MMLU medicalGenerative language modelGPT-5Wang 2025accuracy0.960.96
006MMLU-Pro healthGenerative language modelGemini-3.1-ProWang 2024 (leaderboard)accuracy0.860.83
007MedXpertQA textCliniciansExpert (pre-licensed humans)Zuo 2025accuracy0.430.35
007MedXpertQA textGenerative language modelGPT-5Wang 2025accuracy0.560.35
008MedbulletsGenerative language modelGPT-5 (2025-08-07)Bedi 2025 (MedHELM)accuracy0.870.78
009AfriMed-QAGenerative language modelGPT-4o, zero-shotOlatunji 2025accuracy0.790.82
010MedExpQAGenerative language modelo4-mini, zero-shotYang 2026accuracy0.880.87
010MedExpQATask-specific modelMistral-7B (fine-tuned), no RAGAlonso 2024accuracy0.680.87
011KorMedMCQACliniciansExaminees average (licensing exam institute)Kweon 2024accuracy0.770.89
011KorMedMCQAGenerative language modelo1-preview, 5-shotKweon 2024accuracy0.930.89
012CMExam (China)CliniciansHuman exam takersLiu 2023accuracy0.720.88
012CMExam (China)Generative language modelQwen2.5-72B-InstructBaichuan 2025accuracy0.830.88
012CMExam (China)Task-specific modelChatGLM-CMExam (fine-tuned)Liu 2023accuracy0.450.88
013MedQA MainlandGenerative language modelGPT-4 (5-shot CoT)Qiu 2024accuracy0.810.90
013MedQA MainlandTask-specific modelMMed-Llama 3 8B (fine-tuned)Qiu 2024accuracy0.790.90
014MedQA TaiwanGenerative language modelGPT-4-base (pre-alignment), 5-shotNori 2023accuracy0.870.90
014MedQA TaiwanTask-specific modelBioBERT-large (machine-translated to English + retrieval)Jin 2021accuracy0.420.90
015FrenchMedMCQAGenerative language modelGPT-4, 3-shotWang 2024 (Apollo)accuracy0.900.93
022ATC drug classGenerative language modelGPT-4-0125-preview, 4-shotShoham 2024accuracy0.620.62
023ICD-9 procedureGenerative language modelGPT-4-0125-preview, 4-shotShoham 2024accuracy0.550.53
025Chest CT report findingsGenerative language modelGPT-4o (gpt-4o-2024-11-20), zero-shot, JSON outputYamagishi 2025accuracy0.970.96
027Lung cancer T from reportGenerative language modelGPT-3.5-turbo, zero-shot, full TNM definitions in promptMatsuo 2024accuracy0.570.78
027Lung cancer T from reportTask-specific modelNAIST3SCORE-R2 (fine-tuned Japanese BERTs)Nakamura 2023accuracy0.670.78
029pT stage from pathology reportTask-specific modelBB-TEN (Clinical-BigBird, fine-tuned)Kefeli 2024AUROC0.940.96
031PubMedQACliniciansSingle human annotatorJin 2019accuracy0.780.78
031PubMedQAGenerative language modelGPT-4 + Medprompt (kNN few-shot, self-generated CoT, choice-shuffle ensemble)Nori 2023 (Medprompt)accuracy0.820.78
031PubMedQATask-specific modelMedGemma 27B (text, test-time scaling)Sellergren 2025accuracy0.770.78
035Question entailmentGenerative language modelgemma-4-31b-it, five-shotWu 2025 (BRIDGE)accuracy0.880.83
035Question entailmentTask-specific modelPANLP (ensemble, rank 1)Ben Abacha 2019accuracy0.750.83
037ED triage (KTAS)CliniciansTriage nurses (KTAS_RN)Moon 2019 (in-data)accuracy0.850.56
037ED triage (KTAS)Generative language modelKATE (GPT-4o, 40 kNN shots)Lee 2026accuracy0.720.56
037ED triage (KTAS)Task-specific modelLlama 3.1 8B, fine-tunedLee 2026accuracy0.650.56
038Diagnosis from caseGenerative language modelo3Zhu 2026accuracy0.690.65
039ACMG variant classRuleACMG/AMP points system on panel evidence codesTavtigian 2020 (in-data)accuracy0.950.73
040Trial eligibilityGenerative language modelTrialGPT-Ranking (GPT-4)Jin 2024AUROC0.800.87
040Trial eligibilityTask-specific modelBioLinkBERT cross-encoder (MedNLI)Jin 2024AUROC0.620.87
041Suicide-risk levelGenerative language modelGPT-4, zero-shot (best prompt)Xu 2024 (Mental-LLM)balanced accuracy0.440.56
No.TaskTypeReferenceSourceMetricRef.Best S1
041Suicide-risk levelTask-specific modelMental-FLAN-T5 (FLAN-T5-XXL, instruction-finetuned on four mental-health datasets)Xu 2024 (Mental-LLM)balanced accuracy0.480.56
044CXR view AP/PATask-specific modelResNet-18Kim 2019AUROC1.000.99
046Mammography viewTask-specific modelResNet-50, fine-tuned, 256×256256 \times256 pxYi 2019AUROC1.000.99
047CT contrast phase (series)Task-specific model2dFM_BERT (frozen MAE ViT-L/16 + BERT)Hou 2025AUROC0.910.90
049Abdominal organ (3D)Task-specific model3D CNN with Kinetics-400 video initialization, fine-tunedJain 2024accuracy0.970.38
050Abdominal organ (coronal)Generative language modelQwen3-VL-8B with retrieved exemplars and rules (in-context learning, frozen)Zhao 2026accuracy0.940.40
050Abdominal organ (coronal)Task-specific modelDenseNet-121, 128 pxDoerrich 2024accuracy0.940.40
051Abdominal organ (sagittal)Task-specific modelDenseNet-121, 64 pxDoerrich 2024accuracy0.830.31
053MRI motion artefactTask-specific modelFewSOME (unsupervised few-shot)Belton 2024accuracy0.720.32
055CXR findingsTask-specific modelArk+ (Swin-L, 768 px; fine-tuned)Ma 2025 (Ark+)AUROC0.840.77
056Paediatric pneumoniaGenerative language modelMedGemma-27B, evolved prompt, no fine-tuningSviridov 2026accuracy0.840.79
056Paediatric pneumoniaTask-specific modelDINO ViT-B/16, linear probing, 224 pxDoerrich 2025AUROC0.990.96
057Line and tube positionCliniciansRadiologist (Reader 2 of 2, board-certified/board-eligible)Güzel 2026balanced accuracy0.750.50
057Line and tube positionGenerative language modelGemini 3.1 Flash Lite Preview, zero-shot (Swan-Ganz)Güzel 2026balanced accuracy0.540.87
057Line and tube positionTask-specific modelRank 1 (All Data Are Ext)Kaggle RANZCR CLiP private LBAUROC0.980.81
059Knee OA KL gradeTask-specific modelEnsemble of 6 ImageNet-pretrained CNNs, per-model optimised input size, mix votingPi 2023accuracy0.770.42
061Head CT haemorrhage (slice)Generative language modelGemini 2.0 Flash, zero-shotWang 2025accuracy0.750.71
061Head CT haemorrhage (slice)Task-specific modelRank-1 Kaggle ensembleWang 2021AUROC1.000.88
063Breast densityTask-specific modelLG-DPTNet: multi-view (CC + MLO) fusion, dynamic pyramid convolution + cross-transformerZhong 2023AUROC0.970.71
064Dermoscopy (HAM10000)CliniciansHuman experts (> 10 years)Tschandl 2019balanced accuracy0.810.62
064Dermoscopy (HAM10000)Generative language modelHulu-Med-7B, zero-shotLiu 2026 (DermAgent)accuracy0.530.63
064Dermoscopy (HAM10000)Task-specific modelRank 1, team ’Minjie’, approach ’ensemble b4 ce+fl+cepos+ds1 b3 ce’ISIC 2018 live leaderboardbalanced accuracy0.900.62
065Dermoscopy (ISIC 2019)Task-specific modelRank 1 (DAISYLab, EfficientNet ensemble, external data)ISIC 2019 leaderboardbalanced accuracy0.640.48
066Melanoma (ISIC 2020)Task-specific modelRank 1 of 3,308 entries (CNN ensemble)Kurtsanaky 2024AUROC0.950.78
067DR severity (IDRiD)Task-specific modelMulti-stage transfer learning (ImageNet → EyePACS → DDR → IDRiD) + class-balanced lossShi 2023accuracy0.800.56
068DR grade (RetinaMNIST)Generative language modelGemini-3-flash, zero-shotZhao 2026accuracy0.440.66
068DR grade (RetinaMNIST)Task-specific modelVGG16, end-to-end training, 224×224224 \times224Doerrich 2025accuracy0.640.66
069Retinal OCT diagnosisTask-specific modelInception V3, ImageNet transfer learning (108,312 training images)Kermany 2018accuracy0.970.97
070Colorectal tissue typeGenerative language modelGPT-4V, 5-shot in-context (kNN)Ferber 2024accuracy0.780.74
070Colorectal tissue typeTask-specific modelCONCH, linear probe trained on NCT-CRC-HE-100KLu 2024 (CONCH)balanced accuracy0.940.74
071Blood cell typeTask-specific modelDenseNet-121, end-to-end, 224 pxDoerrich 2025accuracy0.990.80
072ECG superclassesGenerative language modelGPT-4o, zero-shotLiu 2024 (PULSE)AUROC0.560.74
072ECG superclassesTask-specific modelEnsemble of all modelsStronoff 2021AUROC0.930.74
073VQA-RAD yes/noGenerative language modelGPT-5-mini, zero-shot chain-of-thoughtWang 2025accuracy0.750.84
073VQA-RAD yes/noTask-specific modelMedVInT-TD (fine-tuned)Zhang 2023accuracy0.870.84
074SLAKE yes/noGenerative language modelMedGemma 4B, zero-shotSellgren 2025accuracy0.880.93
074SLAKE yes/noTask-specific modelM2I2 (supervised)Li 2023accuracy0.910.93
076PMC-VQAGenerative language modelLLaVA-Med, zero-shotZhang 2023accuracy0.350.64
076PMC-VQATask-specific modelMedVInT-TD (trained on PMC-VQA)Zhang 2023accuracy0.400.64
No.TaskTypeReferenceSourceMetricRef.Best S1
077MedXpertQA multimodalCliniciansExpert (pre-licensed humans)Zuo 2025accuracy0.460.49
077MedXpertQA multimodalGenerative language modelGPT-5Wang 2025accuracy0.720.49
078Head CT haemorrhageTask-specific modelQure.ai deep learning algorithmsChilamkurthy 2018AUROC0.970.84
079Critical head CT findingTask-specific modelQure.ai deep learning algorithmsChilamkurthy 2018AUROC0.970.99
080Chest CT abnormalitiesTask-specific modelCT-CLIP VocabFineHamamci 2026AUROC0.760.76
081Pulmonary embolism on CTPATask-specific modelAttention-based CNN-RNN, strongly supervised (all slice labels)Hu 2025AUROC0.930.58
082Cervical spine fractureTask-specific modelMean of top 8 Kaggle algorithmsLee 2024AUROC0.960.63
083Abdominal trauma on CTTask-specific modelMean of 8 winning challenge modelsHermans 2025AUROC0.980.85
084Pancreatic cancer on CTClinicians68 radiologists (pooled)Alves 2026AUROC0.880.70
084Pancreatic cancer on CTTask-specific modelRank 1 (MountainHill / DTI, DTI_PANORAMA)PANORAMA leaderboardAUROC0.930.70
085Prostate cancer bpMRIClinicians62 radiologists (mpMRI, PI-RADS 2.1)Saha 2024AUROC0.860.59
085Prostate cancer bpMRITask-specific modelPI-CAI AI systemSaha 2024AUROC0.930.59
086Rib fracture typeTask-specific modelEvolutionary neural architecture search with genetic-algorithm data augmentationAli 2024accuracy0.690.41

Table S5. Published results with the same dataset, label set and metric, which can use other splits, and the best of the four System 1 models on the questions that match the published result (Best S1). Best S1 can therefore differ from the task-level AUROC in Table S2.

Supplementary Results

All Tasks and Models

Figure S1 shows the AUROC of every task for every model. Table S2 gives the AUROC with 95% CIs, and Table S3 the random-effects means per tier.

Figure S1. AUROC of every task (rows, numbered as in Table S1) and model (columns). Blue shades mark values above chance (0.5) and red shades values below chance.

No.TaskClefClef-flashGPT-6Luna DecisionsJev 1.14
001MedQA (USMLE)0.97 (0.96–0.97)0.92 (0.90–0.93)0.98 (0.97–0.98)0.99 (0.98–0.99)
002MedMCQA (India)0.89 (0.89–0.90)0.87 (0.86–0.88)0.92 (0.92–0.93)0.93 (0.93–0.94)
003HEAD-QA (English)0.96 (0.95–0.96)0.95 (0.94–0.95)0.96 (0.96–0.97)0.97 (0.96–0.97)
004HEAD-QA (Spanish)0.96 (0.95–0.97)0.95 (0.94–0.95)0.97 (0.96–0.97)0.97 (0.97–0.98)
005MMLU medical0.99 (0.98–0.99)0.99 (0.98–0.99)0.98 (0.98–0.99)0.99 (0.98–0.99)
006MMLU-Pro health0.97 (0.96–0.97)0.96 (0.95–0.97)0.96 (0.94–0.96)0.97 (0.96–0.98)
007MedXpertQA text0.72 (0.71–0.73)0.63 (0.62–0.64)0.76 (0.74–0.77)0.77 (0.76–0.78)
008Medbullets0.92 (0.90–0.94)0.83 (0.80–0.86)0.94 (0.92–0.96)0.94 (0.93–0.96)
009AfriMed-QA0.94 (0.94–0.95)0.92 (0.91–0.93)0.95 (0.95–0.96)0.95 (0.95–0.96)
010MedExpQA0.96 (0.95–0.97)0.92 (0.91–0.94)0.96 (0.94–0.97)0.96 (0.95–0.98)
011KorMedMCQA0.96 (0.96–0.97)0.93 (0.92–0.93)0.99 (0.99–0.99)0.98 (0.98–0.98)
012CMExam (China)0.97 (0.97–0.97)0.95 (0.95–0.96)0.97 (0.97–0.98)0.98 (0.98–0.99)
013MedQA Mainland0.98 (0.98–0.98)0.96 (0.96–0.97)0.98 (0.97–0.98)0.98 (0.98–0.99)
014MedQA Taiwan0.97 (0.96–0.98)0.95 (0.95–0.96)0.98 (0.98–0.99)0.98 (0.98–0.99)
015FrenchMedMCQA0.99 (0.98–0.99)0.97 (0.95–0.98)0.99 (0.98–1.00)0.99 (0.99–1.00)
016MMLU medical (Arabic)0.95 (0.94–0.96)0.93 (0.92–0.94)0.98 (0.97–0.98)0.98 (0.98–0.99)
017MMLU medical (Hindi)0.92 (0.91–0.93)0.84 (0.83–0.86)0.98 (0.97–0.98)0.98 (0.97–0.99)
018MMLU medical (Japanese)0.97 (0.96–0.97)0.96 (0.95–0.97)0.98 (0.97–0.98)0.98 (0.98–0.99)
019MMLU medical (Swahili)0.90 (0.89–0.92)0.85 (0.83–0.86)0.96 (0.95–0.96)0.96 (0.95–0.97)
020MMLU medical (Yoruba)0.85 (0.83–0.86)0.79 (0.77–0.81)0.91 (0.90–0.93)0.93 (0.92–0.94)
021MMLU medical (German)0.98 (0.97–0.98)0.97 (0.96–0.98)0.98 (0.97–0.99)0.99 (0.98–0.99)
022ATC drug class0.53 (0.52–0.54)0.61 (0.60–0.62)0.86 (0.86–0.87)0.73 (0.73–0.74)
023ICD-9 procedure0.65 (0.64–0.65)0.64 (0.63–0.65)0.81 (0.80–0.81)0.72 (0.71–0.72)
024CXR report findings1.00 (0.99–1.00)1.00 (0.99–1.00)0.99 (0.99–0.99)1.00 (1.00–1.00)
025Chest CT report findings0.99 (0.98–0.99)0.99 (0.98–0.99)0.96 (0.96–0.97)0.99 (0.99–0.99)
026Multilingual report coding0.96 (0.96–0.97)0.94 (0.93–0.94)0.94 (0.94–0.95)0.96 (0.96–0.97)
027Lung cancer T from report0.94 (0.92–0.96)0.92 (0.90–0.94)0.92 (0.89–0.95)0.95 (0.93–0.97)
028BI-RADS from report0.88 (0.87–0.90)0.84 (0.82–0.86)0.86 (0.82–0.89)0.81 (0.77–0.84)
029PT stage from pathology report0.96 (0.95–0.97)0.95 (0.94–0.96)0.96 (0.94–0.97)0.96 (0.95–0.97)
030Trial report inference0.90 (0.88–0.93)0.87 (0.84–0.90)0.86 (0.83–0.90)0.92 (0.90–0.95)
031PubMedQA0.81 (0.78–0.84)0.83 (0.79–0.86)0.79 (0.76–0.83)0.84 (0.81–0.87)
032Drug-drug interaction0.89 (0.88–0.91)0.90 (0.89–0.91)0.87 (0.85–0.89)0.93 (0.92–0.94)
033Hallmarks of cancer0.94 (0.93–0.95)0.94 (0.93–0.95)0.89 (0.87–0.91)0.94 (0.93–0.94)
034Public-health claims0.85 (0.83–0.87)0.81 (0.78–0.83)0.81 (0.79–0.84)0.82 (0.80–0.84)
035Question entailment0.88 (0.84–0.92)0.88 (0.84–0.92)0.90 (0.86–0.94)0.92 (0.88–0.95)
036Rubric judging0.83 (0.79–0.86)0.73 (0.69–0.77)0.78 (0.75–0.82)0.84 (0.81–0.88)
037ED triage (KTAS)0.80 (0.78–0.81)0.75 (0.73–0.77)0.82 (0.80–0.84)0.80 (0.78–0.82)
038Diagnosis from case0.83 (0.81–0.85)0.74 (0.71–0.76)0.85 (0.83–0.87)0.82 (0.80–0.84)
039ACMG variant class0.88 (0.87–0.89)0.76 (0.74–0.78)0.96 (0.95–0.96)0.93 (0.92–0.94)
040Trial eligibility0.85 (0.83–0.88)0.85 (0.82–0.88)0.83 (0.80–0.86)0.87 (0.84–0.90)
041Suicide-risk level0.77 (0.75–0.80)0.75 (0.73–0.78)0.81 (0.78–0.83)0.82 (0.79–0.84)
042MRI safety of a device0.58 (0.56–0.61)0.51 (0.49–0.54)0.51 (0.49–0.53)0.66 (0.64–0.68)
043Radiograph body part0.95 (0.94–0.96)0.96 (0.96–0.97)0.96 (0.95–0.96)0.95 (0.95–0.96)
044CXR view AP/PA0.98 (0.97–0.99)0.99 (0.98–0.99)0.89 (0.86–0.91)0.99 (0.99–0.99)
045Imaging modality1.00 (1.00–1.00)1.00 (1.00–1.00)0.99 (0.99–1.00)1.00 (1.00–1.00)

Table S2. AUROC per task and model with 95% CI from 1,000 bootstrap resamples of cases. Refused or failed questions were scored as wrong with the label prevalence as probability.

TierTasksMetricClefClef-flashGPT-6 Luna DecisionsJev 1.14Best
Medical knowledge16AUROC0.90 (0.83–0.96)0.88 (0.81–0.94)0.94 (0.91–0.96)0.93 (0.89–0.97)0.94 (0.92–0.97)
Medical knowledge16Chance-corrected BA0.65 (0.47–0.83)0.60 (0.44–0.75)0.73 (0.63–0.84)0.72 (0.57–0.87)0.74 (0.63–0.85)
Text understanding13AUROC0.92 (0.90–0.93)0.89 (0.87–0.92)0.89 (0.87–0.92)0.92 (0.90–0.94)0.93 (0.91–0.94)
Text understanding13Chance-corrected BA0.63 (0.54–0.73)0.56 (0.43–0.69)0.61 (0.51–0.71)0.65 (0.55–0.75)0.66 (0.57–0.75)
Clinical judgement from text6AUROC0.79 (0.70–0.87)0.73 (0.64–0.81)0.80 (0.66–0.94)0.82 (0.73–0.90)0.83 (0.73–0.93)
Clinical judgement from text6Chance-corrected BA0.34 (0.14–0.55)0.24 (0.10–0.38)0.41 (0.10–0.73)0.42 (0.12–0.72)0.44 (0.12–0.77)
Image perception12AUROC0.84 (0.77–0.92)0.85 (0.79–0.91)0.85 (0.76–0.93)0.88 (0.83–0.94)0.89 (0.84–0.94)
Image perception12Chance-corrected BA0.38 (0.31–0.45)0.40 (0.31–0.48)0.41 (0.24–0.59)0.46 (0.26–0.66)0.45 (0.28–0.61)
2D image findings22AUROC0.80 (0.75–0.84)0.80 (0.75–0.84)0.77 (0.73–0.80)0.83 (0.79–0.86)0.84 (0.81–0.87)
2D image findings22Chance-corrected BA0.28 (0.18–0.38)0.29 (0.18–0.40)0.26 (0.16–0.35)0.39 (0.24–0.53)0.40 (0.27–0.53)
Volumetric findings9AUROC0.64 (0.55–0.73)0.64 (0.56–0.73)0.67 (0.56–0.78)0.68 (0.58–0.78)0.69 (0.60–0.79)
Volumetric findings9Chance-corrected BA0.13 (0.05–0.21)0.14 (0.04–0.23)0.14 (0.07–0.22)0.16 (0.04–0.27)0.20 (0.09–0.32)

Table S3. Random-effects mean per capability tier with 95% CI. Best denotes the best of the four models on each task.

System 1 Models and Their Backbones

On 20 tasks, Clef and Clef-flash were also compared with their backbones, Qwen3.8-27B and Qwen3.5-9B, with the backbone read out over the same options (Figure S2). Each task had one primary endpoint. The sign test across tasks was not significant for Clef-flash (better on 14 tasks and worse on 5, Holm-adjusted P=0.13) or Clef (12 and 7, Holm-adjusted P=0.36). Five of the 16 significant advantages of a System 1 model disappeared or reversed after the backbone readout was corrected for its option prior.

Figure S2. Primary endpoint of each System 1 model minus that of its backbone on 20 tasks, with 95% bias-corrected and accelerated bootstrap CIs. Filled markers indicate Holm-adjusted P<0.05 within the task.

Pooling of Slices

For volumetric tasks, the mean study-level AUROC of yes-or-no questions over tasks and models was 0.68 with the maximum over slices, 0.69 with the mean of the three highest slices and 0.69 with the mean over slices (Figure S3). Pulmonary embolism and cervical spine fracture stayed close to chance under every rule.

Figure S3. Study-level AUROC of volumetric tasks with yes-or-no questions, averaged over questions, when the slice probabilities of a study were combined by their maximum (primary rule), the mean of the three highest, or the mean over all slices. Each panel shows one task and each line one model.

Generative Comparison

Table S4 lists the balanced accuracy and cost of each System 1 model and its generative model per task.

No.TaskSystem 1GenerativeBA S1BA gen.USD/1k S1USD/1k gen.
001MedQA (USMLE)ClefQwen3.8-27B0.840.920.081.40
027Lung cancer T from reportClefQwen3.8-27B0.650.830.111.62
028BI-RADS from reportClefQwen3.8-27B0.540.350.121.64
029pT stage from pathology reportClefQwen3.8-27B0.850.860.320.96
030Trial report inferenceClefQwen3.8-27B0.900.890.151.27
031PubMedQAClefQwen3.8-27B0.570.700.121.03
032Drug-drug interactionClefQwen3.8-27B0.800.880.081.15
036Rubric judgingClefQwen3.8-27B0.820.640.261.70
037ED triage (KTAS)ClefQwen3.8-27B0.520.540.082.11
038Diagnosis from caseClefQwen3.8-27B0.550.610.142.21
039ACMG variant classClefQwen3.8-27B0.390.720.082.45
040Trial eligibilityClefQwen3.8-27B0.650.660.241.69
041Suicide-risk levelClefQwen3.8-27B0.480.620.441.82
042MRI safety of a deviceClefQwen3.8-27B0.250.250.070.72
043Radiograph body partClefQwen3.8-27B0.770.770.270.74
045Imaging modalityClefQwen3.8-27B0.970.970.170.53
046Mammography viewClefQwen3.8-27B0.610.810.191.73
048CT contrast phase (slice)ClefQwen3.8-27B0.520.530.152.21
052Vertebra levelClefQwen3.8-27B0.620.580.362.02
054Low-dose CT qualityClefQwen3.8-27B0.330.410.111.34
055CXR findingsClefQwen3.8-27B0.620.600.472.13
057Line and tube positionClefQwen3.8-27B0.570.630.623.07
058Bone-age bandClefQwen3.8-27B0.170.260.262.33
059Knee OA KL gradeClefQwen3.8-27B0.200.170.092.24
060Lumbar stenosis gradeClefQwen3.8-27B0.430.350.502.92
061Head CT haemorrhage (slice)ClefQwen3.8-27B0.560.560.292.19
062BI-RADS from mammogramClefQwen3.8-27B0.270.310.262.41
063Breast densityClefQwen3.8-27B0.280.280.211.95
064Dermoscopy (HAM10000)ClefQwen3.8-27B0.470.460.141.83
067DR severity (IDRiD)ClefQwen3.8-27B0.300.300.241.57
070Colorectal tissue typeClefQwen3.8-27B0.790.750.091.59
072ECG superclassesClefQwen3.8-27B0.490.620.383.11
001MedQA (USMLE)Clef-flashQwen3.5-9B0.720.920.010.65
027Lung cancer T from reportClef-flashQwen3.5-9B0.480.840.020.81
028BI-RADS from reportClef-flashQwen3.5-9B0.510.320.020.80
029pT stage from pathology reportClef-flashQwen3.5-9B0.790.890.050.61
030Trial report inferenceClef-flashQwen3.5-9B0.790.900.030.72
031PubMedQAClef-flashQwen3.5-9B0.570.580.020.71
032Drug-drug interactionClef-flashQwen3.5-9B0.690.800.010.78
036Rubric judgingClef-flashQwen3.5-9B0.650.650.040.72
037ED triage (KTAS)Clef-flashQwen3.5-9B0.460.470.010.89
038Diagnosis from caseClef-flashQwen3.5-9B0.450.620.020.87
039ACMG variant classClef-flashQwen3.5-9B0.340.640.011.17
040Trial eligibilityClef-flashQwen3.5-9B0.670.680.040.83
041Suicide-risk levelClef-flashQwen3.5-9B0.280.530.071.00
No.TaskSystem 1GenerativeBA S1BA gen.USD/1k S1USD/1k gen.
042MRI safety of a deviceClef-flashQwen3.5-9B0.210.230.010.88
043Radiograph body partClef-flashQwen3.5-9B0.840.810.040.23
045Imaging modalityClef-flashQwen3.5-9B0.970.970.030.19
046Mammography viewClef-flashQwen3.5-9B0.680.740.030.29
048CT contrast phase (slice)Clef-flashQwen3.5-9B0.640.580.020.37
052Vertebra levelClef-flashQwen3.5-9B0.530.470.061.27
054Low-dose CT qualityClef-flashQwen3.5-9B0.330.370.020.23
055CXR findingsClef-flashQwen3.5-9B0.650.650.080.72
057Line and tube positionClef-flashQwen3.5-9B0.580.600.101.27
058Bone-age bandClef-flashQwen3.5-9B0.230.240.041.50
059Knee OA KL gradeClef-flashQwen3.5-9B0.350.280.010.39
060Lumbar stenosis gradeClef-flashQwen3.5-9B0.330.390.081.84
061Head CT haemorrhage (slice)Clef-flashQwen3.5-9B0.560.620.050.61
062BI-RADS from mammogramClef-flashQwen3.5-9B0.330.370.041.04
063Breast densityClef-flashQwen3.5-9B0.330.300.030.44
064Dermoscopy (HAM10000)Clef-flashQwen3.5-9B0.520.530.020.49
067DR severity (IDRiD)Clef-flashQwen3.5-9B0.320.350.040.83
070Colorectal tissue typeClef-flashQwen3.5-9B0.650.680.010.25
072ECG superclassesClef-flashQwen3.5-9B0.510.510.061.75
001MedQA (USMLE)GPT-6 Luna DecisionsGPT-6 Luna0.900.970.030.08
027Lung cancer T from reportGPT-6 Luna DecisionsGPT-6 Luna0.760.910.040.09
028BI-RADS from reportGPT-6 Luna DecisionsGPT-6 Luna0.320.370.040.09
029pT stage from pathology reportGPT-6 Luna DecisionsGPT-6 Luna0.860.870.130.18
030Trial report inferenceGPT-6 Luna DecisionsGPT-6 Luna0.790.860.070.12
031PubMedQAGPT-6 Luna DecisionsGPT-6 Luna0.550.630.050.08
032Drug-drug interactionGPT-6 Luna DecisionsGPT-6 Luna0.790.840.030.06
036Rubric judgingGPT-6 Luna DecisionsGPT-6 Luna0.640.620.110.20
037ED triage (KTAS)GPT-6 Luna DecisionsGPT-6 Luna0.530.560.030.10
038Diagnosis from caseGPT-6 Luna DecisionsGPT-6 Luna0.660.770.050.12
039ACMG variant classGPT-6 Luna DecisionsGPT-6 Luna0.850.820.030.13
040Trial eligibilityGPT-6 Luna DecisionsGPT-6 Luna0.640.600.090.15
041Suicide-risk levelGPT-6 Luna DecisionsGPT-6 Luna0.540.610.190.27
042MRI safety of a deviceGPT-6 Luna DecisionsGPT-6 Luna0.250.300.020.12
043Radiograph body partGPT-6 Luna DecisionsGPT-6 Luna0.880.900.110.18
045Imaging modalityGPT-6 Luna DecisionsGPT-6 Luna0.960.950.070.09
046Mammography viewGPT-6 Luna DecisionsGPT-6 Luna0.740.790.080.13
048CT contrast phase (slice)GPT-6 Luna DecisionsGPT-6 Luna0.720.790.070.16
052Vertebra levelGPT-6 Luna DecisionsGPT-6 Luna0.640.580.130.60
054Low-dose CT qualityGPT-6 Luna DecisionsGPT-6 Luna0.400.460.040.08
055CXR findingsGPT-6 Luna DecisionsGPT-6 Luna0.670.650.290.38
057Line and tube positionGPT-6 Luna DecisionsGPT-6 Luna0.550.620.320.79
058Bone-age bandGPT-6 Luna DecisionsGPT-6 Luna0.340.450.100.30
059Knee OA KL gradeGPT-6 Luna DecisionsGPT-6 Luna0.250.330.030.13
060Lumbar stenosis gradeGPT-6 Luna DecisionsGPT-6 Luna0.500.570.210.64
061Head CT haemorrhage (slice)GPT-6 Luna DecisionsGPT-6 Luna0.590.600.150.26
062BI-RADS from mammogramGPT-6 Luna DecisionsGPT-6 Luna0.290.440.100.27
063Breast densityGPT-6 Luna DecisionsGPT-6 Luna0.330.320.080.14

Table S4. System 1 model against the generative model of its family on the same cases, 100 per task and 50 for Clef and Qwen3.8-27B. BA denotes balanced accuracy, USD/1k the cost in US dollars per 1,000 requests, S1 the System 1 model and gen. the generative model.

No.TaskClefClef-flashGPT-6Luna DecisionsJev 1.14
046Mammography view0.92 (0.90–0.94)0.96 (0.95–0.98)0.96 (0.95–0.98)0.99 (0.98–0.99)
047CT contrast phase (series)0.82 (0.79–0.85)0.82 (0.78–0.86)0.90 (0.87–0.93)0.90 (0.87–0.92)
048CT contrast phase (slice)0.75 (0.73–0.77)0.73 (0.70–0.75)0.85 (0.83–0.86)0.84 (0.82–0.86)
049Abdominal organ (3D)0.83 (0.82–0.85)0.84 (0.83–0.86)0.82 (0.81–0.84)0.84 (0.82–0.85)
050Abdominal organ (coronal)0.78 (0.78–0.79)0.84 (0.84–0.84)0.71 (0.70–0.71)0.82 (0.82–0.83)
051Abdominal organ (sagittal)0.72 (0.72–0.73)0.80 (0.80–0.81)0.68 (0.67–0.68)0.80 (0.80–0.81)
052Vertebra level0.94 (0.93–0.95)0.94 (0.93–0.95)0.97 (0.96–0.97)0.95 (0.94–0.96)
053MRI motion artefact0.71 (0.69–0.74)0.71 (0.68–0.73)0.70 (0.67–0.72)0.76 (0.73–0.78)
054Low-dose CT quality0.72 (0.70–0.74)0.62 (0.60–0.65)0.76 (0.74–0.78)0.78 (0.77–0.80)
055CXR findings0.72 (0.69–0.74)0.77 (0.75–0.79)0.73 (0.71–0.76)0.77 (0.75–0.79)
056Paediatric pneumonia0.95 (0.93–0.96)0.96 (0.94–0.97)0.95 (0.93–0.96)0.93 (0.91–0.95)
057Line and tube position0.78 (0.76–0.79)0.79 (0.78–0.80)0.65 (0.63–0.67)0.81 (0.79–0.82)
058Bone-age band0.75 (0.73–0.77)0.73 (0.71–0.75)0.81 (0.79–0.83)0.77 (0.75–0.78)
059Knee OA KL grade0.66 (0.64–0.68)0.69 (0.67–0.70)0.66 (0.63–0.69)0.64 (0.62–0.66)
060Lumbar stenosis grade0.68 (0.66–0.70)0.66 (0.65–0.68)0.81 (0.80–0.83)0.73 (0.71–0.75)
061Head CT haemorrhage (slice)0.76 (0.74–0.78)0.75 (0.73–0.77)0.76 (0.74–0.78)0.79 (0.77–0.80)
062BI-RADS from mammogram0.62 (0.59–0.65)0.61 (0.59–0.65)0.68 (0.65–0.70)0.69 (0.65–0.72)
063Breast density0.71 (0.68–0.73)0.68 (0.65–0.71)0.70 (0.67–0.73)0.66 (0.63–0.69)
064Dermoscopy (HAM10000)0.87 (0.85–0.89)0.87 (0.85–0.89)0.83 (0.81–0.85)0.90 (0.89–0.92)
065Dermoscopy (ISIC 2019)0.81 (0.81–0.82)0.82 (0.82–0.83)0.78 (0.77–0.79)0.84 (0.83–0.84)
066Melanoma (ISIC 2020)0.76 (0.74–0.78)0.78 (0.76–0.80)0.56 (0.53–0.59)0.75 (0.73–0.77)
067DR severity (IDRiD)0.75 (0.72–0.78)0.82 (0.80–0.85)0.61 (0.58–0.64)0.83 (0.81–0.85)
068DR grade (RetinaMNIST)0.65 (0.61–0.69)0.70 (0.66–0.74)0.57 (0.54–0.61)0.86 (0.83–0.89)
069Retinal OCT diagnosis0.97 (0.97–0.98)0.98 (0.98–0.99)0.92 (0.90–0.93)1.00 (1.00–1.00)
070Colorectal tissue type0.98 (0.97–0.98)0.96 (0.95–0.97)0.87 (0.86–0.89)0.96 (0.95–0.97)
071Blood cell type0.97 (0.97–0.98)0.98 (0.98–0.98)0.86 (0.85–0.87)0.99 (0.98–0.99)
072ECG superclasses0.73 (0.72–0.75)0.67 (0.66–0.68)0.74 (0.73–0.75)0.68 (0.66–0.69)
073VQA-RAD yes/no0.91 (0.87–0.94)0.88 (0.84–0.92)0.87 (0.82–0.91)0.94 (0.91–0.96)
074SLAKE yes/no0.94 (0.92–0.96)0.92 (0.89–0.95)0.90 (0.87–0.94)0.97 (0.96–0.99)
075SLAKE yes/no (Chinese)0.96 (0.94–0.98)0.92 (0.89–0.95)0.93 (0.90–0.96)0.99 (0.98–0.99)
076PMC-VQA0.84 (0.83–0.86)0.83 (0.82–0.85)0.82 (0.81–0.84)0.86 (0.85–0.88)
077MedXpertQA multimodal0.72 (0.70–0.73)0.64 (0.63–0.66)0.79 (0.78–0.81)0.79 (0.78–0.81)
078Head CT haemorrhage0.83 (0.75–0.89)0.85 (0.80–0.91)0.90 (0.85–0.94)0.89 (0.84–0.93)
079Critical head CT finding0.84 (0.78–0.88)0.79 (0.74–0.85)0.88 (0.83–0.92)0.87 (0.83–0.90)
080Chest CT abnormalities0.69 (0.66–0.73)0.71 (0.68–0.74)0.76 (0.73–0.79)0.74 (0.71–0.77)
081Pulmonary embolism on CTPA0.51 (0.45–0.56)0.52 (0.45–0.59)0.54 (0.49–0.59)0.54 (0.49–0.59)
082Cervical spine fracture0.57 (0.51–0.63)0.58 (0.53–0.64)0.59 (0.54–0.65)0.53 (0.47–0.60)
083Abdominal trauma on CT0.69 (0.64–0.74)0.73 (0.68–0.77)0.74 (0.68–0.78)0.79 (0.75–0.83)
084Pancreatic cancer on CT0.61 (0.55–0.68)0.68 (0.61–0.74)0.64 (0.58–0.71)0.70 (0.64–0.75)
085Prostate cancer bpMRI0.58 (0.50–0.66)0.44 (0.36–0.51)0.55 (0.46–0.63)0.59 (0.51–0.67)
086Rib fracture type0.41 (0.37–0.46)0.48 (0.43–0.53)0.42 (0.37–0.47)0.44 (0.39–0.49)

Table 1. System 1 models in medicine

No.TaskSystem 1GenerativeBA S1BA gen.USD/1k S1USD/1k gen.
064Dermoscopy (HAM10000)GPT-6 Luna DecisionsGPT-6 Luna0.440.510.050.16
067DR severity (IDRiD)GPT-6 Luna DecisionsGPT-6 Luna0.330.420.100.22
070Colorectal tissue typeGPT-6 Luna DecisionsGPT-6 Luna0.550.450.030.10
072ECG superclassesGPT-6 Luna DecisionsGPT-6 Luna0.570.620.190.75

Table 2. System 1 models in medicine

References

  1. [1]Ben Van Calster, David J McLernon, Maarten van Smeden, Laure Wynants, Ewout W Steyerberg, and Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative. Calibration: the Achilles heel of predictive analytics. BMC Med, 17(1):230, 2019. doi: 10.1186/s12916-019-1466-7.
  2. [2]Benjamin Coma, Jasper Snoek, and Andrew L Beam. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digit Med, 4(1):4, 2021. doi: 10.1038/s41746-020-00367-3.DOI
  3. [3]Krishnamurthy Dj Vijotham, Jim Winkens, Melih Barsbey, Sumedh Ghaisas, Robert Stanforth, Nick Pawlowski, Patricia Strachan, Zahra Ahmed, Shekoofeh Azizi, Yoram Bachrach, Laura Culp, Mayank Daswani, Jan Freyberg, Christopher Kelly, Atilla Kiraly, et al. Enhancing the reliability and accuracy of AI-enabled diagnosis via complementarity-driven deferral to clinicians. Nat Med, 29(7):1814–1820, 2023. doi: 10.1038/s41591-023-02437-x.
  4. [4]Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023. doi: 10.1038/s41586-023-06291-2.DOI
  5. [5]Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of GPT-4 on medical challenge problems. 2023. arXiv:2303.13375.arxiv.org/abs/2303.13375
  6. [6]Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: watts driving the cost of AI deployment? In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 85–99, 2024. doi: 10.1145/3630106.3658542.DOI
  7. [7]Thomas Savage, John Wang, Robert Gallo, Abdesselam Boukil, Vishwesh Patel, Seyed Amir Ahmad Safavi-Naini, Ali Soroush, and Jonathan H Chen. Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment. J Am Med Inform Assoc, 32(1):139–149, 2025. doi: 10.1093/jamia/ocae254.DOI
  8. [8]Mahmud Omar, Reem Agebarea, Benjamin S Glicksberg, Girish N Nadkarni, and Eyal Klang. Benchmarking the confidence of large language models in answering clinical questions: cross-sectional evaluation study. JMIR Med Inform, 13:e66917, 2025. doi: 10.2196/66917.DOI
  9. [9]Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, New York, 2011. ISBN 9780374275631.
  10. [10]Pat Croskerry. A universal model of diagnostic reasoning. Acad Med, 84(8):1022–1028, 2009. doi: 10.1097/ACM.0b013e3181ace703.DOI
  11. [16]Qingxia Wu, Qingxia Wu, Peipei Zhang, Zhifeng Yi, Yu Shen, Yan Bai, Hongna Tan, Pei Dong, Zhong Xue, Neil Roberts, and Meiyun Wang. A bilingual benchmark for evaluating diagnostic performance of multimodal large language models in radiology (RadM-Bench): evaluation development and validation. J Med Internet Res, 28:e92183, 2026. doi: 10.2196/92183.DOI
  12. [21]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 70 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330, 2017. doi: 10.48550/arXiv.1706.04599. URL https://proceedings.mlr.press/v70/guo17a.html. arXiv:1706.04599.DOI
  13. [22]Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017. doi: 10.48550/arXiv.1705.08500. URL https://proceedings.neurips.cc/paper/2017/hash/4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html. arXiv:1705.08500.DOI
  14. [23]Di Jin, Eileen Pan, Nassim Oufatolle, Wei-Hung Weng, Hanvi Fang, and Peter Szolovits. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl Sci, 11(14):6421, 2021. doi: 10.3390/app11146421.DOI
  15. [24]Errol Colak, Felipe C Kitamura, Stephen B Hobbs, Carol C Wu, Matthew P Lungren, Luciano M Prevedello, Jayashree Kalpathy-Cramer, Robyn L Ball, George Shih, Anouk Stein, Sarwan S Halabi, Emre Altinmakas, Meng Law, Parveen Kumar, Karam A Manzalawi, et al. The RSNA Pulmonary Embolism CT dataset. Radiol Artif Intell, 3(2):e200254, 2021. doi: 10.1148/ryai.2021200254.DOI
  16. [25]Hui Ming Lin, Errol Colak, Tyler Richards, Felipe C Kitamura, Luciano M Prevedello, Jason Talbott, Robyn L Ball, Ekim Gumeler, Kristen W Yeom, Mohammad Hamghalam, Amber L Simpson, Jasna Strika, Deniz Bulja, Salita Angkruwaranon, Almudena Perez-Lara, et al. The RSNA Cervical Spine Fracture CT dataset. Radiol Artif Intell, 5(5):e2300034, 2023. doi: 10.1148/ryai.2300034.
  17. [26]Anindo Saha, Joeran S Bosma, Jasper J Twilt, Bram van Ginneken, Anders Bjartell, Anwar R Padhani, David Bonen-kamp, Geert Villeirs, Georg Salomon, Gianluca Giannarini, Jayashree Kalpathy-Cramer, Jelle Barentsz, Klaus H Maier-Hein, Mirabela Rusu, Olivier Rouvière, et al. Artificial intelligence and radiologists in prostate cancer detection on MRI (PI-CAI): an international, paired, non-inferiority, confirmatory study. Lancet Oncol, 25(7):879–887, 2024. doi: 10.1016/S1470-2045(24)00020-1.DOI
  18. [27]Sasan Klamchukrthy, Rohit Ghosh, Swetha Tanamala, Mustafa Biviji, Norbert G Campeau, Vasantha Kumar Venugopal, Vidur Mahajan, Pooja Rao, and Prashant Warier. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet, 392(10162):2388–2396, 2018. doi: 10.1016/S0140-6736(18)31645-3.
  19. [28]Sun-Hee Moon, Jae Lan Shim, Keun-Sook Park, and Chon-Suk Park. Triage accuracy and causes of mistriage using the Korean Triage and Acuity Scale. PLoS One, 14(9):e0216972, 2019. doi: 10.1371/journal.pone.0216972.DOI
  20. [29]Sue Richards, Nazneen Aziz, Sherri Bale, David Bick, Soma Das, Julie Gastier-Foster, Wayne W Grody, Madhuri Hegde, Elaine Lyon, Elaine Spector, Karl Voelkerding, Heidi L Rehm, and ACMG Laboratory Quality Assurance Committee. Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genet Med, 17(5):405–424, 2015. doi: 10.1038/gim.2015.30.DOI
  21. [30]Sean V Tavtigian, Steven M Harrison, Kenneth M Boucher, and Leslie G Biesecker. Fitting a naturally scaled point system to the ACMG/AMP variant classification guidelines. Hum Mutat, 41(10):1734–1737, 2020. doi: 10.1002/humu.24088.DOI
  22. [31]Yuta Nakamura, Shouhei Hanaoka, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. NTCIR-17 MedNLP-SC radiology report subtask overview: dataset and solutions for automated lung cancer staging. In Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies, pages 145–151, 2023. doi: 10.20736/0002001328.DOI
  23. [32]Jenna Kefeli, Jacob Berkowitz, Jose M Acitores Cortina, Kevin K Tsang, and Nicholas P Tatonetti. Generalizable and automated classification of TNM stage from pathology reports with external validation. Nat Commun, 15(1):8916, 2024. doi: 10.1038/s41467-024-53190-9.DOI
  24. [33]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903.
  25. [34]Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeol u wa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Péter Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, et al. Global MMLU: understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, 2025. doi: 10.18653/v1/2025.acl-long.919.DOI
  26. [35]Charles Nimo, Tobi Olatunji, Abraham Toluwase Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, Ezinwanne C. Aka, Folafunmi Omof o ye, Foutse Yueh g oh, Timothy Faniran, Bonaventure F. P. Dossou, Moshood O. Yekini, Jonas Kemp, Katherine A Heller, Jude Chidubem Omeke, et al. AfriMed-QA: a Pan-African, multi-specialty, medical question-answering benchmark dataset. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics tics (Volume 1: Long Papers), pages 1948–1973, 2025. doi: 10.18653/v1/2025.acl-long.96.
  27. [36]Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. MedExpQA: multilingual benchmarking of large language models for medical question answering. Artif Intell Med, 155:102938, 2024. doi: 10.1016/j.artmed.2024.102938.DOI
  28. [37]Andrew Zhang, Trinh T. L. Vuong, Sophia J. Wagner, Dongmin Bang, Tong Ding, Yuchang Su, Mary Cipperman, Christina S. Chen, Judith Lefkes, Ramin Kahidi, Ming Y. Lu, Caiwei Tian, Tianhao Luo, Jingnan Ma, Rowland Pettit, Long Phile, and Faisal Mahmood. A general-purpose system one decision model for clinical records, 2026. URL https://arxiv.org/abs/2610.27099.
  29. [38]Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankar subbu. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 248–260, 2022. doi: 10.48550/arXiv.2203.14371. URL https://proceedings.mlr.press/v174/pal22za.html. arXiv:2203.14371.DOI
  30. [39]David Vilares and Carlos Gómez-Rodríguez. HEAD-QA: a healthcare dataset for complex reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 960–966, 2019. doi: 10.18653/v1/P19-1092.DOI
  31. [40]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021. doi: 10.48550/arXiv.2009.03300. URL https://iclr.cc/virtual/2021/poster/2962. arXiv:2009.03300.
  32. [41]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shuangyu Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, et al. MMLU-Pro: a more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 37, pages 95266–95290, 2024. doi: 10.48550/arXiv.2406.01574. URL https://papers.nips.cc/paper_files/paper/2024/hash/ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets_and_Benchmarks_Track.html. arXiv:2406.01574.DOI
  33. [42]Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. MedXpertQA: benchmarking expert-level medical reasoning and understanding. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 80961–80990, 2025. doi: 10.48550/arXiv.2501.18362. URL https://proceedings.mlr.press/v267/zuo25a.html. arXiv:2501.18362.DOI
  34. [43]HanJie Chen, Xiaohu Zhang, Yaw Shinla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3563–3599, 2025. doi: 10.18653/v1/2025.naacl-long.182.DOI
  35. [44]Sunjun Kweon, Byungjin Choi, Gyouk Chu, Junyeong Song, Daeun Jeon, Sujin Gan, Jueon Kim, Minkyu Kim, Rae Woong Park, and Edward Choi. KorMedMCQA: multi-choice question answering benchmark for Korean healthcare professional licensing examinations, 2024. arXiv:2403.01469.arxiv.org/abs/2403.01469
  36. [45]Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, and Michael Lignizi Li. Benchmarking large language models on CMExam – a comprehensive Chinese medical exam dataset. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 36, pages 52430–52452, 2023. doi: 10.48550/arXiv.2306.03030. URL https://papers.nips.cc/paper_files/paper/2023/hash/a484d12d588c597f4725a8b84af647b5-Abstract-Datasets_and_Benchmarks.html. arXiv:2306.03030.DOI
  37. [46]Yanis Labrak, Adrien Bazoge, Richard Dufour, Béatrice Daille, Pierre-Antoine Gourraud, Emmanuel Morin, and Mickael Rouvier. FrenchMedMCQA: a French multiple-choice question answering dataset for medical domain. In Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), pages 41–46, 2022. doi: 10.18653/v1/2022.louhi-1.5.DOI
  38. [47]Ofir Ben Shoham and Nadav Rappoport. MedConceptsQA: open source medical concepts QA benchmark. Comput Biol Med, 182:109089, 2024. doi: 10.1016/j.compbiomed.2024.109089.DOI
  39. [48]Dina Denner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval. J Am Med Inform Assoc, 23(2):304–310, 2016. doi: 10.1093/jamia/ocv080.
  40. [49]Yosuke Yamagishi, Yuta Nakamura, Tomohiro Kikuchi, Yuki Sonoda, Hiroshi Hirakawa, Shintaro Kano, Satoshi Nakamura, Shouhei Hanaoka, Takeharu Yoshikawa, and Osamu Abe. Development of a large-scale dataset of chest computed tomography reports in Japanese and a high-performance finding classification model: dataset development and validation study. JMIR Med Inform, 13:e71137, 2025. doi: 10.2196/71137.
  41. [50]Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Seval Nil Erisgun, Irem Dogan, Omer Faruk Durugol, Benjamin Hou, Suprosanna Shit, Weicheng Dai, Murong Xu, Hadrien Reynaud, Muhammed Furkan Dasdelen, Bastian Wittmann, et al. Generalist foundation models from a multimodal dataset for 3D computed tomography. Nat Biomed Eng, 10(8):1610–1628, 2026. doi: 10.1038/s41551-025-01599-y.DOI
  42. [51]Bastien Le Guellec, Kokou Adambounou, Lisa C Adams, Thibault Agripnidis, Sung Soo Ahn, Radhia Ait Chalal, Tugba Akinci D’Antonioli, Philippe Amouyel, Henrik Andersson, Raphaël Bentégeac, Claudio Benzoni, Antonino Andrea Blandino, Felix Busch, Elif Can, Riccardo Cau, et al. PARROT, an open multilingual radiology reports dataset. Eur J Radiol Artif Intell, 5:100066, 2026. doi: 10.1016/j.ejrai.2025.100066.DOI
  43. [52]José Luis Vázquez Noguera, Alejandro Torres-Hurtado, Helena Gómez-Adorno, Julio César Mello-Román, Enrique Javier Fleitas-Alvarez, Federico Fernando Espinola Schulze, Miguel Garcia-Torres, Carlos Domingo Méndez Gaona, Pedro Esteban Gardel Sotomayor, Silvia Vázquez Noguera, Norma Elizabeth Zaracho Amarilla, and Oxades Wilfrido Gamarra Esquivel. Mammography reporting dataset with BI-RADS system for natural language processing applications: addressing public data gaps in Spanish. Data Brief, 61:111761, 2025. doi: 10.1016/j.dib.2025.111761.DOI
  44. [53]Jenna Kefeli and Nicholas Tatonetti. TCGA-Reports: a machine-readable pathology report resource for benchmarking text-based AI models. Patterns (N Y), 5(3):100933, 2024. doi: 10.1016/j.patter.2024.100933.DOI
  45. [54]Mael Jullien, Marco Valentino, and André Freitas. SemEval-2024 task 2: safe biomedical natural language inference for clinical trials. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1947–1962, 2024. doi: 10.18653/v1/2024.semeval-1.271. URL https://aclanthology.org/2024.semeval-1.271/.DOI
  46. [55]Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: a dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, 2019. doi: 10.18653/v1/D19-1259.DOI
  47. [56]María Herrero-Zazo, Isabel Segura-Bedmar, Paloma Martínez, and Thierry Declerck. The DDI corpus: an annotated corpus with pharmacological substances and drug-drug interactions. J Biomed Inform, 46(5):914–920, 2013. doi: 10.1016/j.jbi.2013.07.011.
  48. [80]Safwan S Halabi, Luciano M Prevedello, Jayshree Kalpathy-Cramer, Artem B Mamonov, Alexander Bibilly, Mark Cicero, Ian Pan, Lucas Araújo Pereira, Rafael Teixeira Sousa, Nitamar Abdala, Felipe Campos Kitamura, Hans H Thodberg, Leon Chen, George Shih, Katherine Andriole, et al. The RSNA pediatric bone age machine learning challenge. Radiology, 290(2):498–503, 2019. doi: 10.1148/radiol.2018180736.DOI
  49. [81]Pingjun Chen, Linlin Gao, Xiaoshuang Shi, Kyle Allen, and Lin Yang. Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss. Comput Med Imaging Graph, 75:84–92, 2019. doi: 10.1016/j.compmedimag.2019.06.002.DOI
  50. [82]Pingjun Chen. Knee osteoarthritis severity grading dataset, 2018. Data set.DOI
  51. [83]Tyler J Richards, Adam E Flanders, Errol Colak, Luciano M Prevedello, Robyn L Ball, Felipe Kitamura, John Mongan, Maryam Vazirabad, Hui-Ming Lin, Anne Kendall, Thanat Kanthawang, Salita Angkuranwaranon, Erme Altinmakas, Hakan Dogan, Paulo Eduardo de Aguiar Kuriki, et al. The RSNA Lumbar Degenerative Imaging Spine Classification (LumbarDISC) dataset. Radiol Artif Intell, 8(2):e250480, 2026. doi: 10.1148/ryai.250480.DOI
  52. [84]Adam E Flanders, Luciano M Prevedello, George Shih, Safwan S Halabi, Jayshree Kalpathy-Cramer, Robyn Ball, John T Mongan, Anouk Stein, Felipe C Kitamura, Matthew P Lungren, Gagandeep Choudhary, Lesley Cala, Luiz Coelho, Monique Mogensen, Fanny Morón, et al. Construction of a machine learning dataset through collaboration: the RSNA 2019 Brain CT Hemorrhage Challenge. Radiol Artif Intell, 2(3):e190211, 2020. doi: 10.1148/ryai.2020190211.DOI
  53. [85]Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection 2018: a challenge hosted by the International Skin Imaging Collaboration (ISIC), 2019. arXiv:1902.03368.arxiv.org/abs/1902.03368
  54. [86]Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci Data, 5:180161, 2018. doi: 10.1038/sdata.2018.161.DOI
  55. [87]Noel C F Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection: a challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hosted by the International Skin Imaging Collaboration (ISIC). In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 168–172, 2018. doi: 10.1109/ISBI.2018.8363547. arXiv:1710.05006.DOI
  56. [88]Carlos Hernández-Pérez, Marc Combalia, Sebastian Podlipnik, Noel C F Codella, Veronica Rotemberg, Allan C Halpern, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Brian Helba, Susana Puig, Veronica Vilaplana, and Josep Malvehy. BCN20000: dermoscopic lesions in the wild. Sci Data, 11(1):641, 2024. doi: 10.1038/s41597-024-03387-w.
  57. [89]Veronica Rotemberg, Nicholas Kurtansky, Brigid Betz-Stablein, Liam Caffery, Emmanouil Chousakos, Noel Codella, Marc Combalia, Stephen Dusza, Pascale Gusset, David Gutman, Allan Halpern, Brian Helba, Harald Kittler, Kivanc Kose, Steve Langer, et al. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Sci Data, 8(1):34, 2021. doi: 10.1038/s41597-021-00815-z.DOI
  58. [90]Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Deshmukh, Vivek Sahasrabuddhe, and Fabrice Meriaudeau. Indian Diabetic Retinopathy Image Dataset (IDRiD): a database for diabetic retinopathy screening research. Data, 3(3):25, 2018. doi: 10.3390/data3030025.DOI
  59. [91]Ruhan Liu, Xiangning Wang, Qiang Wu, Ling Dai, Xi Fang, Tao Yan, Jaemin Son, Shiqi Tang, Jiang Li, Zijian Gao, Adrian Galdran, J M Poorneshwaran, Hao Liu, Jie Wang, Yerui Chen, et al. DeepDRiD: diabetic retinopathy-grading and image quality estimation challenge. Patterns (N Y), 3(6):100512, 2022. doi: 10.1016/j.patter.2022.100512.DOI
  60. [92]Jakob Nikolas Kather, Johannes Krisam, Pompimol Charoentong, Tom Luedde, Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, Lina Jansen, Constantino Carlos Reyes-Aldasoro, Inka Zörgiebel, Dirk Jäger, Hermann Brenner, et al. Predicting survival from colorectal cancer histology slides using deep learning: a retrospective multicenter study. PLoS Med, 16(1):e1002730, 2019. doi: 10.1371/journal.pmed.1002730.DOI
  61. [93]Andrea Acevedo, Anna Merino, Santiago Alférez, Ángel Molina, Laura Boldú, and José Rodellar. A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data Brief, 30:105474, 2020. doi: 10.1016/j.dib.2020.105474.DOI
  62. [94]Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojciech Samek, and Tobias Schaeffter. PTB-XL, a large publicly available electrocardiography dataset. Sci Data, 7(1):154, 2020. doi: 10.1038/s41597-020-0495-6.DOI
  63. [95]Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Sci Data, 5:180251, 2018. doi: 10.1038/sdata.2018.251.DOI
  64. [96]Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. SLAKE: a semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654, 2021. doi: 10.1109/ISBI48211.2021.9434010. arXiv:2102.09542.DOI
  65. [97]Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Development of a large-scale medical visual question-answering dataset. Commun Med (Lond), 4(1):277, 2024. doi: 10.1038/s43856-024-00709-z.DOI
  66. [98]Jeffrey D Rudie, Hui-Ming Lin, Robyn L Ball, Sabeena Jalal, Luciano M Prevedello, Savvas Nicolaou, Brett S Marinelli, Adam E Flanders, Kirti Magudia, George Shih, Melissa A Davis, John Mongan, Peter D Chang, Ferco H Berger, Sebastiaan Hermans, et al. The RSNA Abdominal Traumatic Injury CT (RATIC) dataset. Radiol Artif Intell, 6(6):e240101, 2024. doi: 10.1148/ryai.240101.DOI
  67. [99]Natalia Alves, Megan Schuurmans, Dawid Rutkowski, Anindo Saha, Pierpaolo Vendittelli, Nancy Obuchowski, Marjolein H Liedenbaum, Ingfrid S Haldorsen, Anders Molven, Derya Yakar, Jeroen Geerdink, Sebastiaan van Koeverden, Deniece M Riviere, Wulphert Venderink, Robbert de Haas, et al. Artificial intelligence and radiologists in pancreatic cancer detection using standard of care CT scans (PANORAMA): an international, paired, non-inferiority, confirmatory, observational study. Lancet Oncol, 27(1):116–124, 2026. doi: 10.1016/S1470-2045(25)00567-4.DOI
  68. [100]Liang Jin, Jiancheng Yang, Kaiming Kuang, Bingbing Ni, Yiyi Gao, Yingli Sun, Pan Gao, Weiling Ma, Mingyu Tan, Hui Kang, Jiajun Chen, and Ming Li. Deep-learning-assisted detection and segmentation of rib fractures from CT scans: development and validation of FracNet. EBioMedicine, 62:103106, 2020. doi: 10.1016/j.ebiom.2020.103106.DOI

Paper details

vlnc:W_6br4vxjy
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents