WorksComputer Science
MIMIC-Note-Extract: A Large-Scale Structured Dataset Extracted from MIMIC-IV-Note Discharge Summaries
Abstract
MIMIC is one of the most widely used public hospital datasets for machine learning on electronic health records, yet clinically important information, such as past medical history, allergies, and diagnostic findings, remains confined to free-text discharge summaries. Studies requiring this information must therefore build their own extraction pipelines, duplicating substantial effort across projects. We release MIMIC-Note-Extract (MNE), a large-scale dataset that uses a hybrid extraction pipeline combining deterministic parsing with schema-guided large language model generation to transform 16 sections from 300,720 discharge summaries into structured tables. MNE adds information that has no structured MIMIC counterpart and enriches related tables with broader coverage and greater clinical detail, including 10 times more patient-history entries and medication histories for more than twice as many admissions. Validation by medical doctors confirms extraction accuracy above 92% across sections. Using MNE, we construct a clinical decision-making benchmark for LLM agents, demonstrating how reusable structured representations of clinical narratives reduce data curation effort.
Keywords: clinical information extraction, electronic health records, MIMIC, LLMs
Data and Code Availability
MNE is available on PhysioNet (Kuntzer et al., [24]; Pollard et al., 2026) under the same credentialed-access controls as its source dataset, MIMIC-IV-Note v2.2 (Johnson et al., [22]). Code for reproduction is available at https://github.com/kschwethelm/mimic-note-extract.
Institutional Review Board (IRB)
No IRB approval required. Our work is a secondary analysis of deidentified MIMIC-IV-Note v2.2 data (Johnson et al., [22]) and involves no interaction with individuals and no access to identifiable private information.
Introduction
The Medical Information Mart for Intensive Care (MIMIC) Johnson et al., [23] has enabled thousands of peer-reviewed publications Zhang et al., 2025. However, the highly structured tabular formats that underpin much of this research omit the rich narrative context of patient care. Elements such as a patient’s past medical history, allergies, and pertinent findings are only found in unstructured text. While MIMIC-IV-Note Johnson et al., [22] provides these free-text narratives through a collection of clinical documents, including discharge summaries and radiology reports, this wealth of information remains difficult to scale or integrate into research workflows without extensive preprocessing.
This leads to a fragmented ecosystem of “individual preprocessing” where, in the absence of a standardized pipeline, each project must build its own scripts to extract the specific variables needed for its use case. These isolated preprocessing pipelines are rarely shared or unified, resulting in duplicated engineering effort and inconsistent extraction quality across the community. But even if such an effort were made, the resulting datasets are typically tailored so closely to individual projects that they lack the generality needed to support broader research questions. At the core of this lies a methodological limitation: most existing projects focus on information extraction. Instead, we posit that the community requires information conversion—preserving the underlying information of the raw data while transforming it into an easily queryable, structured format.
Our design philosophy reflects this distinction by prioritizing the completeness of the clinical record. We introduce MIMIC-Note-Extract (MNE), a structured dataset derived from 300,720 MIMIC-IV-Note discharge summaries. We employ a hybrid processing framework that pairs deterministic parsing with schema-guided LLM generation to translate 16 major clinical sections of the discharge summary into structured tables. By providing structured representations for 13 of these sections—ranging from pre-admission history to detailed physical exams—we eliminate the preprocessing barrier while preserving the semantic nuances of the original narratives (see Figure 1).
Figure 1. A raw discharge summary flowing through our extraction pipeline, showing a representative subset of four sections. The example is synthetic and does not correspond to any real admission. The discharge summary (left) is split into sections (middle) and decomposed into relational rows (right). Badges denote the extraction method of each section (rule-based, hybrid, or LLM).
To demonstrate the dataset’s utility, we construct a large-scale clinical decision-making benchmark that evaluates LLM agents on a real-world diagnostic task. It supports more pathologies and more specific clinical actions with substantially less curation effort.
Contributions.
We introduce and publicly release MIMIC-Note-Extract, a clinical database containing data from 16 discharge summary sections of 300,720 admissions, validated via expert medical review to >92% accuracy and shown to complement existing MIMIC tables with novel information.
Together with medical experts, we design an adaptive extraction pipeline that balances computational efficiency and clinical accuracy by dynamically choosing rule- or LLM-based extraction.
We construct an agentic benchmark with 10,000 patient cases to evaluate LLMs’ capabilities for diagnosing 19 different pathologies.
Related Work
MIMIC-derived tabular datasets. Several works derive task-specific datasets by selecting, merging, and engineering features from MIMIC’s existing structured tables: MIMIC-Extract builds time-series features from MIMIC-III chart events Wang et al., 2020, the MIMIC-III multitask benchmark packages vital signs and lab results for clinical prediction Harutyunyan et al., [19], the MIMIC-IV emergency-department (ED) benchmark assembles triage outcomes from ED tables Xie et al., 2022, and Gupta et al. [16] standardize cohort construction across these sources. Instead of selecting from existing tables, we convert free-text discharge summaries into new structured tables, adding clinically important information that is absent from MIMIC’s tabular data.
Radiology reports. Beyond the structured tables, MIMIC provides two free-text document types: radiology reports and discharge summaries Johnson et al., [22]. The radiology reports are widely studied and have been structured into entity-relation graphs, standardized report formats, disease labels, or question-answer pairs [21, 7] (Müller et al. 2026). We instead work on discharge summaries, which cover the whole admission rather than a single imaging study. However, our extraction can easily be combined with extractions of the radiology report sections.
Manual discharge-summary annotation. Several datasets share our focus on discharge summaries, but manually label passages in them for a single, narrow purpose: DiReCT Wang et al., 2024 marks findings that support a diagnosis to annotate the diagnostic reasoning chain, CLIP (Mullenbach et al., 2021) flags post-discharge to-dos (e.g., “follow up with cardiology in two weeks”), MedDec (Elgaar et al., [8]) marks where a clinical decision was made (e.g., “started IV antibiotics”), and the 2018 n2c2 task (Henry et al., [20]) tags medications and the adverse drug events they cause. Because manual annotation is costly, each covers only a few hundred notes (718 for CLIP, 511 for DiReCT, and 451 for MedDec), orders of magnitude smaller than our cohort.
Automated discharge-summary extraction. Prior automated approaches extract text from notes without structuring it: MIMIC-IV-Ext-BHC (Ali et al., 2025) releases the verbatim brief-hospital-course section of 270,033 summaries as a summarization target, clinical decision-making (CDM) benchmarks (Hager et al., [18]; Gaber et al., [10]; Ferber et al., [9]) use regular expressions (regex) to assemble diagnostic cases from selected note sections such as the patient history (500–9k cases), and MIMIC-IV-Ext-22MCTS (Wang et al., 2025) uses Llama-3.1-8B to decompose ~267k summaries into a list of short clinical events (e.g., “fever” or “pain control”). In contrast, we do not target a single task, section, output type, or extract only verbatim text: we convert all 16 sections of every discharge summary across ~300k admissions into general, structured tables with well-defined fields, creating a general-purpose dataset that supports a wide range of downstream applications.
Dataset and Extraction Pipeline
We derive our cohort from the 331,793 raw discharge summaries available in MIMIC-IV-Note v2.2, which are released de-identified, with sensitive information redacted by “___”. The redaction method also removed some non-identifying clinical content, including clinical terms and parts of section headers. The remaining text retains all patient histories, symptoms, diagnoses, treatments, findings, and descriptions of the hospital course, although some keywords are lost through redaction. Following quality filtering1, our final dataset comprises 300,720 admissions. The resultMIMIC-Note-Extract dataset preserves high data availability across nearly all clinical categories (Table 1), with most sections exceeding 90% coverage. Coverage denotes the percentage of admissions with a non-empty section; for example, the low coverage of “Major Procedures” directly reflects the availability of procedure information in the source data. We also preserve three high-complexity narrative sections in their original form to avoid losing information where a structured breakdown was not feasible. The following paragraphs detail our extraction pipeline; the full database schema and comprehensive technical details are provided in Appendices § 11 and § 12.
| Clinical Section | Key Fields | Method | Admissions | Coverage |
| Gender | Patient sex assigned at birth | Rule | 300,720 | 100.0% |
| Services | Clinical service at discharge | Rule | 300,720 | 100.0% |
| Chief Complaint | Symptom category and clinical description | Hybrid | 296,132 | 98.5% |
| History of Present Illness | Chronological narrative of events prior to admission | Verbatim | 296,620 | 98.6% |
| Past Medical History | Prior conditions and interventions by category, chronicity, and subject | LLM | 290,212 | 96.5% |
| Medications on Admission | Active home medications with dose, route, frequency, and strength | LLM | 275,035 | 91.5% |
| Allergies | Allergen name | Hybrid | 296,378 | 98.6% |
| Physical Examination | Physiological measurements and findings by organ system | LLM | 294,708 | 98.0% |
| Pertinent Results | Lab, imaging, and diagnostic findings with key values | LLM | 241,508 | 80.3% |
| Brief Hospital Course | Narrative of clinical course and medical management | Verbatim | 267,164 | 88.8% |
| Major Procedures | Procedural interventions with current vs. historical flags | Hybrid | 159,791 | 53.1% |
| Discharge Diagnosis | Discharge diagnoses preserving clinical priority | Hybrid | 297,537 | 98.9% |
| Discharge Condition | Clinical status indicators | Hybrid | 300,109 | 99.8% |
| Discharge Medications | Active outpatient prescriptions at discharge with dose, route, and frequency | LLM | 285,926 | 95.1% |
| Discharge Instructions | Outbound instructions, follow-up directives, and education | Verbatim | 291,647 | 97.0% |
| Discharge Disposition | Discharge destination | Rule | 300,667 | 100.0% |
Table 1. Clinical sections, key concepts, extraction methodologies, and cohort-wide coverage in MNE across 300,720 admissions. Coverage is the percentage of admissions with a non-empty section.
Splitting into sections. The first step in our pipeline divides monolithic discharge reports into their constituent sections. We identify 24 distinct sections, dropping 8 that are fully redacted (e.g., “Name” in Figure 1). De-identification is, at times, over-inclusive and also removes tokens that carry no identifying information, including words within section headers. Splitting is therefore complicated by partially redacted headers (e.g., “Discharge ___”, which could indicate “Discharge Condition” or “Discharge Disposition”). To resolve this, our deterministic splitting algorithm tracks common header variants. It further exploits the consistent relative ordering of clinical sections to infer headers that remain ambiguous due to redaction.2
Information modeling. Following section splitting, the next step in our pipeline is defining the target schemas for information extraction. The difficulty lies in the massive structural and content diversity within individual clinical sections. For instance, the Pertinent Results section can range from a single-line negative cell culture report to a multi-page unfiltered output of microbiology panels and longitudinal imaging findings. Therefore, we must carefully decide what information to retain and how to structure it; the goal is to capture essential clinical depth without introducing hyper-specific fields that remain empty across most samples. Keeping the schemas concise also ensures they remain manageable for downstream human verification. To achieve this balance, we itera- ends_mid=1 tively developed target models in close consultation with clinicians, analyzing source texts for recurring patterns to refine our field definitions. The schemas preserve temporal context where it is available in the source, including distinctions between current and historical information.
Adaptive extraction. To process large volumes of clinical notes efficiently, we follow the principle of choosing the simplest tool capable of handling the task. Simple, highly structured sections can be reliably processed using deterministic rules, such as regular expressions and string matching, whereas complex narratives require the contextual understanding of LLMs3. Our pipeline thus categorizes sections into three distinct operational modes (see Table 1): some rely entirely on rule-based extraction, others require LLMs exclusively, and a hybrid group uses rule-based methods by default, falling back to an LLM only when section-specific validation criteria fail. For example, for Discharge Condition, we extract and validate status indicators deterministically; any residual text triggers LLM fallback (Appendix § 12). This adaptive approach ensures optimal computational throughput for straightforward data while reserving LLM inference for genuinely complex data. Furthermore, we optimized token efficiency by mapping long clinical identifiers to single-token codes (e.g., mapping subject to s). By instructing the model to output these short abbreviations, we trade off expensive output tokens for much cheaper input tokens. This approach reduced overall inference costs (see Appendix D.§ 1). Across the entire cohort, this pipeline was executed for a total cost of 1,894.61€, yielding substantial cost savings of 321.24€ (14.5% of total cost) compared to a naive LLM-only baseline (see Table 6 in Appendix D.§ 2).
| Clinical Section | Operational Mode | Production Model | LLM Baseline (€) | Production Cost (€) | Optimized Savings (€) |
| LLM-Only Extraction | |||||
| Past Medical History | LLM | Gemini 3.1 Flash-Lite | 138.07 | 138.07 | — |
| Medications on Admission | LLM | Gemini 3.1 Flash-Lite | 487.00 | 487.00 | — |
| Physical Examination | LLM | Gemini 3.1 Flash-Lite | 345.00 | 345.00 | — |
| Pertinent Results | LLM | Gemini 3.1 Flash-Lite | 179.00 | 179.00 | — |
| Discharge Medications | LLM | Gemini 3.1 Flash-Lite | 581.00 | 581.00 | — |
| Hybrid (Rule-First with LLM Fallback) | |||||
| Chief Complaint | Hybrid (61.2% Auto) | Gemini 2.5 Flash | 62.78 | 24.36 | 38.42 |
| Allergies | Hybrid (99.1% Auto) | Gemini 2.5 Flash | 124.90 | 1.07 | 123.83 |
| Major Procedures | Hybrid (20.0% Auto) | Gemini 2.5 Flash | 53.42 | 42.75 | 10.67 |
| Discharge Diagnosis | Hybrid (27.6% Auto) | Gemini 2.5 Flash | 63.68 | 46.11 | 17.57 |
| Discharge Condition | Hybrid (72.2% Auto) | Gemini 3.1 Flash-Lite | 181.00 | 50.25 | 130.75 |
| Deterministic (Rule / Verbatim) | |||||
| Gender | Rule-Based | — | 0.00 | 0.00 | — |
| Services | Rule-Based | — | 0.00 | 0.00 | — |
| Discharge Disposition | Rule-Based | — | 0.00 | 0.00 | — |
| History of Present Illness | Verbatim | — | 0.00 | 0.00 | — |
| Brief Hospital Course | Verbatim | — | 0.00 | 0.00 | — |
| Discharge Instructions | Verbatim | — | 0.00 | 0.00 | — |
| Aggregate Total | Mixed | 2,215.85 | 1,894.61 | 321.24 |
Table 6. Production extraction methodology, LLM model, API expenditures, and rule-first optimization savings by clinical section.
Validation and constraints. To ensure the final output aligns with our target data models, the pipeline concludes with a strict validation layer. We use formal schemas to programmatically constrain LLM generation ([3, 2]) and to run post-hoc data validation. Any output that violates these structural constraints or custom validation rules triggers an automated retry loop, ensuring a guaranteed fit with the target data model.
Quality Validation
We evaluate extraction quality through a manual, side-by-side review of the source text against the extractions by three medical doctors. Annotations were performed at the item level to allow the distinction between a single item-level error and an entirely incorrect sample. The error taxonomy, concrete examples, and additional metrics are shown in Appendix § 14.
Setup. Rule-based extractions were excluded from manual review due to their deterministic, auditable nature. Final validation was performed after the extraction pipeline had been frozen. Extractions reviewed during this validation were not manually corrected. For LLM-based extractions, we determined the required sample size using the standard normal approximation (Lohr, 2021) formula for binomial proportions. Assuming an expected accuracy of at a 95% confidence level and a conservative target margin of error (MoE) of 6%, the formula dictated a minimum of 171 unique cases per section. To assess inter-annotator agreement (IAA), we utilized 121 single-annotated and 50 dual-annotated cases. The dual-annotated cases were randomly distributed among pairs of our three practitioners. This setup ensured that each overlapping case was independently reviewed by two people, resulting in a total workload of cases per annotator per section. In total, the annotators completed 2,210 case-level tasks (13,744 individual item-level decisions). Alongside accuracy, we report Gwet’s AC1 ([17] ) for IAA, since raw agreement is inflated by the high prevalence of correct extractions.
Results. As summarized in Table 2, item-level accuracy is consistently high across all sections, ranging from 92.2% to 99.8%, with realized margins of error between 0.32% and 3.81%, well inside our 6% target. IAA is similarly high: AC1 exceeds 95% in seven of the ten sections. The three exceptions are pertinent results at 88.4%, past medical history at 90.1%, and physical examination at 90.4%.
| Section | Acc. (%) | AC1 |
| Allergies | 99.8 | |
| Chief Complaint | 98.0 | |
| Discharge Condition | 98.0 | |
| Discharge Diagnosis | 95.8 | |
| Major Procedures | 98.0 | |
| Past Medical History | 90.1 | |
| Physical Examination | 90.4 | |
| Pertinent Results | 88.4 | |
| Discharge Medications | 97.1 | |
| Medications on Admission | 97.2 |
Table 2. Extraction accuracy (item-level, 95% confidence margins of error) and inter-annotator agreement (Gwet’s AC1) per section.
Comparison to MIMIC Tables
The information we extract from the discharge notes overlaps partially with MIMIC’s existing tables. We compare each MNE section with its closest structured counterpart(s) in MIMIC to determine whether it validates, extends, or adds to existing data. Exact one-to-one matching is unrealistic for most sections: MNE fields are extracted from unstructured text, whereas MIMIC tables contain coded events.
Table 3 groups the sections by how they relate to MIMIC’s tables and gives a short summary of what each section contributes. Three sections largely reproduce an existing MIMIC field; the redundancy can be used to flag mismatching records. The discharge-disposition extraction additionally covers 44k admissions with a missing MIMIC value. The remaining sections either extend a related but non-equivalent MIMIC table with broader coverage or richer detail (e.g., allergies, past medical history, medications on admission) or have no MIMIC counterpart at all (discharge condition, discharge medications, physical exam, and pertinent results). Overall, MNE complements MIMIC’s existing structured data by adding substantial new clinical information and providing broader or richer representations where related tables already exist. See Appendix § 15 for detailed results.
| Section | Contribution |
| Already exists in MIMIC, but can flag mismatches | |
| Gender | Flag mismatches |
| Services | Flag mismatches |
| Discharge Disposition | Flag mismatches; adds 44k missing values |
| Related to MIMIC information, but broader or richer | |
| Discharge Diagnosis | Clinician-written diagnoses with primary/secondary structure |
| Chief Complaint | more admissions; cleaner terms and additional clinical context |
| Allergies | Comprehensive allergen list covering far more admissions than MIMIC’s ICD codes |
| Past Medical History | more history rows; contains broader history, family, substance, and functional history, compared to MIMIC’s ICD codes |
| Medications on Admission | more admissions; more detail by providing dose, route, frequency, PRN, and hold status |
| Major Procedures | Clinical procedure label and more surgical details compared to ICD/HCPCS billing |
| No structured MIMIC counterpart | |
| Discharge Condition | Provides structured discharge state, including mental status and functional activity |
| Discharge Medications | Provides discharge-medication list |
| Physical Examination | Provides system-level examination findings with normal/abnormal status |
| Pertinent Results | Provides important clinical results, e.g., radiology interpretations |
Table 3. Relationship of extracted sections to MIMIC’s existing tabular data.
Benchmarking Agents in Clinical Decision-Making
As a concrete downstream application of our dataset, we build an agentic benchmark for clinical decision-making (CDM) based on MIMIC-CDM ([18]). Such benchmarks are a natural application for our dataset because they require clinical information that is not available in MIMIC’s tabular data. MIMIC-CDM simulates real-world clinical decision-making from patient presentation to diagnosis: starting from patient history, the doctor agent decides which examinations and tests to perform before making a diagnosis. Like a human doctor would examine a patient or order a laboratory test or imaging study, the agent initiates these actions through tools and receives the corresponding findings. To supply these findings, the authors parsed discharge summaries with regex, a laborious process that yielded verbatim free-text blocks (similar to the “Section-split text” in Figure 1). We ask how the construction of MIMIC-CDM would change if MNE had been available at the time.
We first validate that MNE can replace the regex-derived texts without reducing accuracy, while requiring far less curation effort. We then leverage MNE’s fields to build a larger agentic CDM benchmark with more admissions and pathologies, stricter case filtering, and finer control over the agent’s context.
Full-Information CDM Benchmark
We reproduce the full-information setting of MIMIC-CDM, in which the LLM receives all patient information up front instead of requesting it through tools. This resembles a second-reader setting: the model interprets evidence already selected and collected by the treating clinicians to reach a diagnosis. Specifically, the LLM receives a single message containing the patient’s history of present illness, physical examinations, laboratory tests, and radiology reports. This setting evaluates whether the model can reach the correct diagnosis when the relevant clinical information has already been selected.
As described above, Hager et al. [18] had to pre-process discharge summaries to construct this context. We instead can query the relevant fields from MNE and apply simple filters where necessary. For example, the timepoint column in MNE’s physical-examination table distinguishes admission from discharge examinations. This allows us to exclude findings that were available only at discharge, i.e., after the diagnosis had been made (Figure 1). As shown in Table 4, accuracy matches or improves for every model. Thus, MNE reproduces the original benchmark context through simple table queries, substantially lowering the curation effort required to construct the benchmark.
| Model | Reasoning | Full info | Agentic |
| Gemini 3.5 Flash | minimal | 88.7 (+0.1) | 87.9 |
| Gemini 3.1 Flash Lite | minimal | 87.4 (+0.7) | 81.9 |
| Gemini 3.1 Flash Lite | high | 88.6 (+0.0) | 84.2 |
| Gemma 4 31B | 86.7 (+0.3) | 83.7 | |
| Qwen3.6 35B-A3B | 90.4 (+0.3) | 83.6 | |
| Qwen3.6 35B-A3B | 89.3 (+0.2) | 82.6 | |
| Qwen3.5 2B | 82.0 (+0.9) | 51.3 |
Table 4. Diagnosis accuracy (%) on the full-information and agentic CDM benchmarks. Parentheses report how accuracy changes when using our data instead of the original regex-derived data. Appendix Table 13 reports 95% confidence intervals.
Agentic CDM Benchmark
We use MNE’s structured representation to develop an agentic CDM benchmark in which agents perform clinical actions to reach a diagnosis (Figure 2). For example, the agent can read a section of the patient history or request a physical examination of a specific body part. This level of granularity was not possible in the original MIMIC-CDM benchmark, where the corresponding tools return entire verbatim text blocks of the whole history or the whole examination notes. The minimal curation effort also allows us to construct a larger benchmark with 10,000 cases across 19 pathologies, compared with four pathologies and 2,400 cases in MIMIC-CDM.
Figure 2. Overview of the agentic CDM benchmark built from MIMIC-Note-Extract. The LLM agent starts from the patient presentation and gathers evidence through five tools before making a diagnosis. Green tools are sourced from MNE, blue tools from existing MIMIC data, and hatched from both. In contrast to prior benchmarks, the agent must name the exact item it wants, e.g., a single examination system or a patient-record section, and receives only this requested part.
We create five agent tools representing a subset of real-world clinical actions: reviewing patient history, performing physical examinations, ordering laboratory tests, and requesting imaging or other clinical studies (Table 15). Agents specify the target of each action, such as examining the abdomen, ordering a troponin test, or requesting a chest X-ray. In clinical practice, a human doctor would make those requests and receive real findings. Here, the agent makes the requests through tool calls, and recorded MIMIC and MNE data simulate the findings. If the information is unavailable, the tool returns a message such as “The lab cannot perform the troponin test for this patient”. Overall, the model must decide which clinical action to take next based on the findings obtained so far, determine when it has sufficient evidence, and interpret that evidence to reach a diagnosis.
| Tool | Source | Requestable items |
| Physical examination | MNE | Six examination groups: vital signs, general, cardiopulmonary, abdominal, neurologic, and skin/extremities. |
| Patient record | MNE | Eight sections: history of present illness, conditions, interventions, substance history, functional status, family history, allergies, and admission medications. |
| Laboratory | MIMIC | Panels (CBC, basic metabolic, liver function, coagulation, urinalysis, blood gas, urine electrolytes), body-fluid analyses (CSF, ascitic, synovial, pleural), or individual tests from a catalog. |
| Imaging | MIMIC + MNE | A region (abdomen, chest, head, neck, spine, extremity, venous) paired with a modality (CT, MRI, ultrasound, radiograph, nuclear medicine, fluoroscopy, ERCP). |
| Clinical study | MNE | One study: echocardiogram, cardiac catheterization, electrocardiogram, stress test, upper endoscopy, or colonoscopy. |
Table 15. The five tools exposed to the agent. Three are backed by MIMIC-Note-Extract (MNE); laboratory and imaging reuse existing MIMIC tables, with imaging additionally drawing on extracted pertinent-result imaging.
To accurately simulate patient cases, admissions need sufficient diagnostic evidence. For example, diagnosing myocardial infarction may require a recorded electrocardiogram (ECG). MNE enables precise case selection through structured fields, allowing us, e.g., to require an ECG, specific examination systems, or complete vital signs. A clinician first selected pathologies that could be diagnosed using the available tools, after which 10,000 admissions were automatically selected using filters over the MIMIC and MNE tables.
Overall, MNE simplified case filtering and data curation while providing greater flexibility in how information is exposed to the agent.
Results. Table 4 reports diagnosis accuracy on our benchmark. The larger models reached 82–88% accuracy, while the small Qwen3.5 2B only achieved 51.3%, showing that the agentic benchmark is more challenging for most models than the full-information benchmark. Reasoning reduced accuracy for Qwen3.6 and improved it for Gemini 3.1 Flash Lite, in line with prior findings that chain-of-thought does not reliably improve clinical performance (McCoy et al., 2025; Wu et al., 2025). See Appendix G for more details.
Discussion
Downstream Applications
Our dataset supports many downstream applications. Here, we sketch three examples.
Drug-allergy labels and antibiotic prescribing. Reported penicillin allergies are associated with higher rates of antibiotic-resistant infections, such as MRSA, and C. difficile infections, partly because they lead to greater use of broad-spectrum alternative antibiotics, even though the reported allergy history is often inaccurate [4] (Macy and Contreras 2014). Studying this relationship in MIMIC is difficult because it lacks allergy information. Our dataset provides detailed allergy information for nearly 170,000 admissions, which can be combined with the broader clinical record to study how reported allergy labels influence antibiotic prescribing and subsequent infection outcomes during hospitalization.
Medication reconciliation. Cornish et al. [6] showed that more than half of patients admitted on four or more regular medications have at least one unintended discrepancy between their pre-admission medications and their admission orders, and that nearly 40% of these discrepancies had the potential to cause moderate to severe harm. MNE provides structured medication lists for admissions. These lists can be compared with inpatient prescription records to identify medication discrepancies at the cohort scale, enabling large-scale studies of medication reconciliation and polypharmacy (a patient taking multiple medications at the same time).
Risk adjustment with documented function. The Hospital Frailty Risk Score estimates patients’ vulnerability to poor health outcomes by assessing their reduced ability to recover from illness or stress, using ICD-10 codes because functional status is often unavailable in routine administrative data [11]. Such code-based measures can be affected by incomplete or inaccurate coding (O’Malley et al., 2005). MNE instead provides directly documented information on patients’ functional and mental status, including detailed medical history. These fields can be used in outcome models and to assess code-based frailty measures.
Clinical Text Extraction with LLMs
Extracting structured information from clinical notes at the scale of MIMIC-IV surfaced several practical lessons that are likely common to similar efforts.
Schema design. Most of our design effort went into defining categories that cleanly partition the clinical notes without being overly broad or narrow. We refined categories iteratively until they covered all common cases; capturing rare edge cases was infeasible. Designing an effective schema required close collaboration with medical experts to decide what is worth extracting and to verify correctness. Attempts to automate schema design with an LLM-in-the-loop did not produce satisfactory results.
Prompt design. Current LLMs have strong medical knowledge and can interpret clinical terminology, abbreviations, and reasoning without explicit instruction. The main value of prompt design was to resolve ambiguity: clarifying how borderline cases should be handled, describing dataset-specific formatting patterns, handling documentation errors, or disambiguating fields. However, every rule we added, no matter how precise, gave the model more to think over, making it more likely to overthink simple extractions and overapply exception-handling logic. Finding the right balance was difficult: too few rules left genuine ambiguity unresolved, too many made the model second-guess straightforward cases. This was especially pronounced for reasoning models, which would produce lengthy reasoning traces before their response, even when the correct extraction was obvious. Iterative prompt testing with validation sets was essential.
Automatic validation. Even the best LLMs make extraction errors. We used Pydantic validators [5] to catch and automatically correct common mistakes like malformed dates, out-of-range values, or type mismatches. Fixing these predictable errors through deterministic postprocessing was more reliable than overly specific prompts.
Managing costs. Extracting 16 sections across 300,720 admissions cost 1,894.61€ in API fees (Appendix D). We kept costs down through concise prompts and output formats, selecting the cheapest model that met quality requirements, using low thinking budgets for reasoning models, and using rule-based extraction wherever LLMs were not necessary.
Zero data retention (ZDR). Clinical notes contain sensitive patient information that should not be shared with third parties through cloud services or APIs. We require strict ZDR policies that prohibit saving data, e.g., for model training or abuse monitoring. We considered running local models, but found the hardware requirements infeasible at our scale.
Limitations and Future Work
Our dataset inherits limitations from the source discharge summaries. The original MIMIC developers de-identified the notes by removing names, contact details, identifying numbers, and explicit dates and times. As described in § 3, the de-identification method also redacted some non-identifying content, resulting in loss of clinically relevant information. The removal of dates and times limits temporal reconstruction, although MNE uses retained relative expressions such as “at discharge”, “three years ago”, and “since yesterday” to capture temporal context. More generally, discharge summaries reflect clinicians’ retrospective selection of information, may contain text copied from earlier encounters, and may omit distracting observations, competing hypotheses, and uncertainty present during care.
Extraction quality was not systematically compared with open-weight and specialized medical models. The released pipeline, schemas, prompts, and model configurations support such comparisons in future work.
To preserve clinician wording, we did not fully standardize clinical terminology. Fields such as diagnoses and procedures retain source wording and abbreviations, so identifying patient cohorts may require terminology mapping or additional filters. Broader terminology normalization remains future work.
Finally, Brief Hospital Course, History of Present Illness, and Discharge Instructions are too diverse to find a reliable pattern for splitting them into fields. Future work may partition these free-text sections into temporal event chains, following Wang et al. (2025).
Conclusion
We presented MIMIC-Note-Extract, a large-scale structured dataset that extends MIMIC’s tabular data across 300,720 admissions with important clinical information, including diagnoses, home medications, allergies, patient history, and physical examination findings. Overall, our dataset adds 16 new tables to MIMIC that easily integrate with existing data through admission identifiers. The information was extracted from the MIMIC-IV-Note discharge summaries using regex rules and LLMs. Our extraction pipeline was built in close collaboration with clinicians and achieves extraction accuracy across sections, as confirmed by medical expert review.
Acknowledgments
KS and GK received support from the Bavarian Collaborative Research Project PRIPREKI of the Free State of Bavaria Funding Programme “Artificial Intelligence – Data Science”.
JK was supported by the German Medical Informatics Initiative grant ID 01ZZ2316C.
The authors gratefully acknowledge the scientific support and resources of the AI service infrastructure LRZ AI Systems provided by the Leibniz Supercomputing Centre (LRZ) of the Bavarian Academy of Sciences and Humanities (BAdW), funded by the Bayerisches Staatsministerium für Wissenschaft und Kunst (StMWK), primarily through the BayernKI initiative.
Prompt and Pydantic Schema Examples
Discharge Diagnosis
The discharge diagnosis section illustrates a minimal LLM-assisted extraction. Listings 1 and 2 show the pydantic extraction schema and the system prompt.
class DischargeDiagnosis(BaseModel):
id: int = Field(description="Case ID")
primary: list[str] = Field(default_factory=list)
secondary: list[str] = Field(default_factory=list)
Listing 1: Pydantic model for the discharge diagnosis section.
You are a medical data extraction specialist. Your task is to extract individual medical diagnoses and
conditions from unstructured clinical notes. Categorize as primary or secondary when explicitly
labeled, otherwise place all in primary (except for special cases). Preserve exact medical
terminology, order and remove enumeration/bullets.
SPECIAL CASES:
- Past medical history (pmh) labeled explicitly with a separate header goes to secondary.
- Preserve relevant ’[REDACTED]’ placeholders.
- Preserve time identifiers in diagnoses (e.g., ’chronic’, ’acute’, ’history of’).
- Remove duplicates.
- Use context when parsing; a few linebreaks are formatting errors, not semantic breaks.
- Special axis structure: preserve as SINGLE primary diagnosis with linebreaks between axes.
- Clear ’with’ descriptions are a single diagnosis (e.g., ’Right-sided rib fractures 8 and 9 with small
anterior pneumothorax’).
Listing 2: System prompt for the discharge diagnosis section.
Physical Examination
The physical examination section is one of the most complex sections. Each admission can have multiple exam events (admission, discharge, interim) and most events contain a vital-sign object plus a list of per-system assessments. Enums are encoded with short three-letter codes to keep the LLM’s JSON output compact. Listings 3 and 4 show the extraction model and prompt in abbreviated form.
class VitalSigns(BaseModel):
temperature: float | "[REDACTED]" | None
heart_rate: int | "[REDACTED]" | None
blood_pressure_systolic: int | "[REDACTED]" | None
blood_pressure_diastolic: int | "[REDACTED]" | None
respiratory_rate: int | "[REDACTED]" | None
o2_saturation: int | "[REDACTED]" | None
weight: Weight | None
respiratory_support: OxygenSupport | None
class SystemAssessment(BaseModel):
system: SystemName # GEN, HNT, CVR, RSP, GIT, EXT, NEU, SKN, PSY, GUR, ACC
status: ExamStatus # NRM, ABN, NAS
abnormal_findings: list[str]
class PhysicalExamEvent(BaseModel):
timepoint: Timepoint # ADM, DSC, INT, UNK
vital_signs: VitalSigns | None
systems: list[SystemAssessment]
class PhysicalExams(BaseModel):
id: int
exams: list[PhysicalExamEvent]
Listing 3: Abbreviated Pydantic model for the physical examination section.
Role: Medical Data Extraction Specialist
Objective: Extract physical examination data from clinical discharge reports into a structured format
while adhering to strict logic for data integrity and medical accuracy.
I. Data Integrity & Constraints
1. System Uniqueness: For any single timepoint (e.g., Admission), a body system must appear only once. If
the source contains multiple observations for the same system, synthesize them into a single
comprehensive entry.
2. Omission Policy: Only include systems explicitly mentioned in the text. Do not generate entries for
systems that were not documented or assessed.
3. Timepoint Separation: Maintain strict separation between distinct exam events (e.g., Admission vs.
Discharge). Do not merge findings across different clinical timeframes.
II. The Mutual Exclusivity Rule
Every entry must strictly follow status-finding dependencies.
- Normal Status: "abnormal_findings" must be empty.
- Abnormal Status: if any specific abnormal finding, observation, or indwelling device is extracted,
status is "Abnormal".
- Access Systems: presence of any indwelling device (IV, Foley, Chest Tube) dictates status "Abnormal",
with the device type listed in "abnormal_findings".
- Negative Indicators: phrases like "No wheezing", "Clear", "Supple", "S1/S2", "Soft", "Non-tender",
"Non-distended", "Intact", "Alert", "Oriented" map to Normal status with no abnormal findings.
III. Vital Sign Extraction & Clustering
- Aggressive Re-scan: Vital signs (BP, HR, RR, Temp, SpO2) often appear in clusters. If one metric is
identified (e.g., via a "/" for BP or "%" for SpO2), re-scan the surrounding text for unlabeled values.
- Physiological Range Mapping: Use context to map unlabeled numbers. BP is "120/80"; a value of 98 in a
cluster is likely saturation or temperature; use standard physiological ranges to differentiate heart
rate from respiratory rate.
IV. Clinical Data Transformation
- Terminology Standardization: Translate descriptive observations into standard medical terminology in
"abnormal_findings" (e.g., "Rapid breathing" → Tachypnea, "Redness and swelling" → Erythema and Edema).
- Synthesized Descriptions: Observations should be concise but clinically accurate.
Listing 4: System prompt for the physical examination section.
Complete Relational Database Schema
We present the complete database schema of MIMIC-Note-Extract in Table 5. This schema includes all 16 tables, including administrative hub tables and narrative text tables. To construct the final relational database, we flatten the Pydantic extraction models. Simple fields map directly to standard SQL types, and categorical values convert to SQL enums, while complex polymorphic structures (such as varied medication frequencies) are serialized into JSONB columns to prevent excessive table sparsity.
| Table | Cardinality | Key fields |
| Hub | ||
admissions | 1 | subject_id, hadm_id, note_id |
| Single-row attributes | ||
gender | 0–1 | enum |
services | 0–1 | enum (20 services) |
discharge_disposition | 0–1 | enum |
discharge_condition | 0–1 | 14 enums + 2 booleans |
| Multi-row clinical lists | ||
allergies | 0–n | title, title_std |
chief_complaint | 0–n | category (enum), text |
discharge_diagnosis | 0–n | title, is_primary |
major_procedures | 0–n | title, is_previous |
past_medical_history | 0–n | concept_name, category, temporal_status, subject (patient/family), family_relation |
medications_on_admission | 0–n | name, presentation_form, dose (value + unit), route, duration (value + unit), strength and frequency (JSONB), PRN/hold flags |
discharge_medications | 0–n | name, presentation_form, dose (value + unit), route, duration (value + unit), strength and frequency (JSONB), PRN/hold flags |
| Per-timepoint structured exam | ||
physical_exam_vitals | 0–4 | timepoint (enum), temperature, heart rate, blood pressure, respiratory rate, saturation, weight, support |
physical_exam_systems | 0–n | timepoint (enum), system (enum), status (enum), abnormal_findings |
| Subsection-typed list | ||
pertinent_results | 0–n | content type (enum), line range, verbatim text passage, per-type enum arrays, topic label (clinical notes only) |
| Raw text | ||
discharge_sections | 1 | one text column per section (16) |
Table 5. All 16 tables in the MIMIC-Note-Extract schema. “Cardinality” refers to rows per admission.
Pipeline Implementation Details
This section provides technical implementation details for the pipeline’s section splitting, extraction, and validation stages.
Structural parsing Twenty-four distinct clinical sections were identified as appearing in consistent relative order across the MIMIC-IV-Note discharge summaries. Eight of these sections (e.g., Name, Admission Date, Date of Birth) are entirely redacted by upstream de-identification and are consequently dropped from the cohort. Deterministic mappings were established between common header variants and canonical section names. The relative ordering of sections was utilized to automatically resolve all but 5,461 ambiguous headers that could not be parsed automatically due to upstream de-identification (e.g., distinguishing between Discharge Condition and Discharge Disposition given “Discharge ___”). Those cases were manually resolved by inspecting their content.
Hybrid extraction heuristics We use section-specific deterministic rules to reduce computational cost, with a fallback to an LLM only when the complexity exceeds the capabilities of regular expressions. In the Allergies, Chief Complaint, Discharge Diagnosis, and Major Procedure sections, the LLM is bypassed for simple cases: single-line entries that match a curated list of terms or restricted character sets. In more detailed sections, such as Discharge Condition, success relies on a “zero-remainder” policy. Status indicators are extracted and validated using enums. These indicators are then removed from the original text. If any residual content remains, the entire section is processed by the LLM instead.
Validation and post-processing Throughout the pipeline, section-specific Pydantic validators enforce structural and clinical consistency on the extracted data. We showcase two sections here as an example. For the Pertinent Results section, validators verify that extracted document coordinates represent non-overlapping, chronologically ordered line ranges to maintain the original document’s sequence. In the Physical Examination section, more complex logic is applied: validators ensure clinical consistency by flagging assessments marked as “Normal” that contain abnormal findings, while automatically merging fragmented extractions that share the same clinical timepoint. Global post-processing includes cleaning whitespace, deduplicating redundant clinical list items, and standardizing clinical units, such as normalizing temperature values to Fahrenheit. Additionally, the pipeline identifies physiological outliers for researcher review while preserving all verbatim values to maintain a faithful representation of the original clinical text.
Extraction Cost Analysis
Exploiting API Pricing Asymmetries with Compressed Output Schemas
Because the LLM generates a structured JSON response for each submitted sample, structural elements such as field names and enum values are repeated in every output. While these keys are strictly necessary for parsing, they are essentially discarded after extraction—yet we still have to pay for the tokens to generate them. At a cohort size of over 300,000 samples, this repetitive JSON overhead becomes a significant driver of total costs.
This presents an excellent opportunity for optimization by exploiting the asymmetric pricing model of LLM APIs, where output tokens are much more expensive than input tokens (e.g., Gemini 3.1 Flash-Lite pricing of $1.50 per 1M output tokens versus $0.25 per 1M input tokens). Input tokens—i.e., the static system instructions and schemas repeated across all samples—are cheaper still at a rate of $0.025 per 1M tokens, representing a 60-fold cost reduction compared to output tokens. Consequently, minimizing the number of output tokens is a great lever to save costs.
To exploit this asymmetry, we designed our Pydantic schemas to use short, abbreviated field aliases and enum codes during model inference (e.g., mapping subject s, and PATIENT P). These brief aliases are resolved back to their canonical, human-readable forms downstream to maintain usability for subsequent tasks and human validation.
To validate the efficiency of this approach, we conducted a comparative analysis of 20 samples from the Past Medical History section using Gemini 3.1 Flash-Lite for the cost analysis. The full Pydantic models for both the original verbose schema and the optimized version are presented in Listings 5 and 6, respectively. The experimental results demonstrate a 9.87% reduction in total output tokens compared to the verbose baseline. This resulted in a 5.99% reduction in total execution cost under standard pricing.4
Implementation note. This optimization strategy was developed midway through the project, meaning it is not used across all sections. For sections that had already been processed in earlier stages of development, we elected to retain the original verbose schemas rather than incurring the additional cost of re-running the full extraction pipeline with the optimized Pydantic schemas.
class SubjectType(str, Enum):
PATIENT = "PATIENT"
FAMILY = "FAMILY"
class CategoryType(str, Enum):
CONDITION = "CONDITION"
INTERVENTION = "INTERVENTION"
SUBSTANCE = "SUBSTANCE"
FUNCTIONAL_STATUS = "FUNCTIONAL_STATUS"
class TemporalStatus(str, Enum):
CURRENT = "CURRENT"
PAST = "PAST"
class MedicalHistoryItem(BaseModel):
subject: SubjectType
family_relation: str | None = None
concept_name: str
category: CategoryType
temporal_status: TemporalStatus | None = None
Listing 5: Original verbose Pydantic schema.
class SubjectType(str, Enum):
P = "P"
F = "F"
class CategoryType(str, Enum):
C = "C"
I = "I"
S = "S"
FS = "FS"
class TemporalStatus(str, Enum):
U = "U"
T = "T"
class MedicalHistoryItem(BaseModel):
s : SubjectType
r : str | None = None
n : str
cat : CategoryType
tmp : TemporalStatus | None = None
Listing 6: Optimized Pydantic schema with 1-token aliases. Red text indicates the fields and values modified to minimize output token consumption.
Detailed Cost and Savings Calculations
A detailed breakdown of the extraction costs and savings afforded by our hybrid, rule-first extraction strategy is presented. To calculate these figures, a baseline cost for each clinical section, assuming a fully LLM-driven pipeline, is first estimated. This baseline is established by calculating the average per-case cost from representative pilot runs using Gemini pricing models. This unit cost is then multiplied by the total number of cases in the cohort to project the baseline expense.
The baseline is compared with actual production expenditures, and the resulting net savings are shown in Table 6. The table structure is defined as follows:
LLM Baseline: The projected total cost of extracting the entire section using only LLMs.
Production Cost: The actual cost incurred using our optimized pipeline, where deterministic rules bypassed LLM calls for structured cases.
Optimized Savings: The savings achieved by our rule-first strategy.
Full Validation Results
This section provides the full details of the manual validation, including the error taxonomy, concrete examples, metric definitions, and the complete performance breakdown.
Error taxonomy
Ambiguous The source text does not unambiguously support either approval or rejection (e.g., shorthand that could be interpreted in multiple valid ways).
Fabrication The extracted item is not supported by the source text.
Omission An item present in the source text is missing from the extracted set.
Clinically Altered Meaning The extracted item is textually close to the source but semantically different in a way that would affect a clinical reader.
Incorrect Attribute / Classification Structured attribute fields (e.g., dosage or category) are incorrect.
Incorrect Boundaries The detected line boundaries are incorrect. Exclusive to the “Pertinent Results” section.
Concrete examples Table 7 illustrates the clinician-assigned labels with source excerpts and the corresponding extracted fields. We selected illustrative examples, one for each category. Each explanation follows the clinicians’ annotations.
| Category / section | Source excerpt | Extracted fields | Error explanation |
| Ambiguous Medications on admission | Simethicone 40-80 mg PO QID:PRN abd pain | strength: range 40–80 mg dose: 80 mg | The source gives a 40–80 mg range without separately specifying strength and administered dose. The extraction records both a strength range and a single 80 mg dose. |
| Fabrication Past medical history | R THA / L THA / Stents x2 | concept_name: Coronary Artery Stents | The extraction adds a coronary location to the stents, although their anatomical location is not specified in the source. |
| Omission Medications on admission | 18. Insulin SC / Sliding Scale Insulin SC Sliding Scale using HUM Insulin | medications: no insulin entry | Sliding-scale insulin is explicitly listed in the source but is absent from the complete extracted medication list. |
| Clinically altered meaning Discharge medications | Calcium 500 + D (calcium carbonate-vitamin D3) 500 mg(1,250mg) -400 unit oral DAILY | strength: [500 mg, 400 units] dose: 1,250 mg | The parenthetical 1,250 mg quantity is incorrectly interpreted as the administered dose. |
| Incorrect attribute / classification Discharge medications | furosemide 40 mg Tablet ___: One (1) Tablet PO ONCE (Once) for 1 doses. | name: furosemide frequency: 1 per DAY duration: null | A one-time dose is incorrectly represented as daily administration without a duration limit. |
| Incorrect boundaries Pertinent results | 19: collection. 20: Findings were concerning for early or partial small bowel 21: obstruction. The NG tube was kept in place, and medications were | start_line: 6 end_line: 19 Clinician span: 6–21 | The extracted span ends at line 19 instead of line 21, omitting the following finding of suspected small-bowel obstruction. |
Table 7. Examples of clinician-assigned error and ambiguity labels. Bold text highlights the source phrases and extracted values relevant to each judgment; field names are set in monospace. Source excerpts retain the original wording, with line breaks represented by spaces or slashes except in the boundary example; extracted fields are abbreviated for display. Boundary line numbers are zero-based and inclusive.
An overview of extraction errors and ambiguous judgments across all categories is detailed in Table 8. Validation highlights that omissions (231 errors) and incorrect attributes (157 errors) are the predominant error modes, localized mainly within “Physical Examination” and “Past Medical History”. In contrast, critical failures such as fabrications (14 total) and clinically altered meanings (42 total) are rare across the 13,744 validated items.
| Section | Judged Items | Omiss. | Attr. | Alter. | Fabr. | Bound. | Ambig. |
| Allergies | 1,488 | 0 (0.0%) | 0 (0.0%) | 1 (0.1%) | 2 (0.1%) | 0 (0.0%) | 6 (0.4%) |
| Chief Complaint | 222 | 1 (0.5%) | 4 (1.8%) | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) | 3 (1.4%) |
| Discharge Condition | 222 | 4 (1.8%) | 0 (0.0%) | 0 (0.0%) | 1 (0.5%) | 0 (0.0%) | 1 (0.5%) |
| Discharge Diagnosis | 222 | 1 (0.5%) | 5 (2.3%) | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| Major Procedures | 222 | 1 (0.5%) | 1 (0.5%) | 2 (0.9%) | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| Past Medical History | 2,761 | 66 (2.4%) | 46 (1.7%) | 26 (0.9%) | 3 (0.1%) | 0 (0.0%) | 36 (1.3%) |
| Physical Examination | 3,715 | 103 (2.8%) | 79 (2.1%) | 3 (0.1%) | 0 (0.0%) | 0 (0.0%) | 6 (0.2%) |
| Pertinent Results | 564 | 23 (4.1%) | 4 (0.7%) | 6 (1.1%) | 0 (0.0%) | 10 (1.8%) | 1 (0.2%) |
| Discharge Medications | 2,281 | 21 (0.9%) | 13 (0.6%) | 4 (0.2%) | 7 (0.3%) | 0 (0.0%) | 551 (24.2%) |
| Medications on Admission | 2,047 | 11 (0.5%) | 5 (0.2%) | 0 (0.0%) | 1 (0.0%) | 0 (0.0%) | 388 (19.0%) |
| Total | 13,744 | 231 (1.7%) | 157 (1.1%) | 42 (0.3%) | 14 (0.1%) | 10 (0.1%) | 992 (7.2%) |
Table 8. Counts and percentages of extraction errors and ambiguous judgments by clinical section. Percentages are calculated relative to all judged items within each section.
Accuracy is calculated using an ambiguity-excluded mapping. Because an ambiguous judgment indicates that the source text does not support a reliable decision about correctness, we exclude it from the accuracy denominators rather than treating it as either correct or incorrect. At the item level, each remaining extracted item contributes one decision, and recorded omissions are added to the denominator as incorrect missing items. Precision, recall, and F1-score use the same mapping, treating omissions as false negatives and fabrications or semantic alterations as false positives. At the case level, we evaluate the remaining item decisions and exclude cases with no evaluable items after removing ambiguous judgments; a case with no items and no recorded omissions is scored as correct by convention.
Judgments are clustered within cases: the 50 dual-annotated cases contribute two judgments each, and the items of a case are reviewed together rather than drawn independently. We therefore take the case as the unit of analysis, letting each case contribute the mean of the judgments made about it, so that a disagreement between two annotators enters as 0.5. Margins of error come from a cluster bootstrap that resamples the 171 cases with replacement over 10,000 replicates and reports the 2.5th and 97.5th percentiles, which anchors the effective sample size to the independent cases rather than to the judgments made about them.
The item-level accuracy column is the Jaccard index rather than conventional accuracy. A true negative would be an item correctly not extracted, which carries no meaning for a task whose objective is to extract everything present in the source text, so the taxonomy has no such category and none can enter a denominator. The case-level columns are macro-averaged over cases while the item-level columns are micro-averaged over items, and a case with no items and no recorded omissions is scored as correct by convention.
To assess inter-annotator agreement, both raw agreement and Gwet’s AC1 are reported using a tolerant mapping that treats Approve and Ambiguous as the same label. This measures agreement on substantive extraction correctness without penalizing differences in annotators’ thresholds for assigning the ambiguity label. Raw agreement can be misleadingly high in datasets with high-prevalence categories (as observed here).
Gwet’s AC1 is more robust to this effect and provides a more stable estimate of agreement beyond chance. It is reported at the item level only; a case-level label is the full pattern of item labels across the case, which leaves nearly every case in a category of its own and reduces the chance correction to nothing.
Error counts are reported per annotator judgment rather than per case, so an error flagged independently by both annotators of a dual-annotated case contributes twice. The “Judged Items” column is the matching denominator, leaving the relative figures unaffected.
The gap between case-level and item-level accuracy (Table 9 and Table 10) is expected in multi-item sections: a single disputed medication or history item can cause the entire case to fail. This is most evident in complex sections such as past medical history or medications. Item-level quality is treated as the primary estimate for cohort-level reuse, while case-level accuracy provides a conservative lower bound.
| Section | Acc. | Prec. | Rec. | F1 |
| Ambiguity-excluded case-level (%) | ||||
| Allergies | 98.5 ± 2.06 | 99.6 | 100.0 | 99.8 |
| Chief Complaint | 97.6 ± 2.37 | 97.9 | 97.8 | 97.8 |
| Discharge Condition | 97.9 ± 2.35 | 97.9 | 97.9 | 97.9 |
| Discharge Diagnosis | 97.1 ± 2.63 | 97.7 | 97.4 | 97.5 |
| Major Procedures | 98.2 ± 2.05 | 98.5 | 98.4 | 98.4 |
| Past Medical History | 72.8 ± 6.43 | 97.7 | 97.4 | 97.2 |
| Physical Examination | 80.7 ± 5.56 | 97.3 | 95.2 | 96.1 |
| Pertinent Results | 89.5 ± 4.39 | 95.6 | 96.3 | 95.7 |
| Discharge Medications | 84.8 ± 5.15 | 98.4 | 98.9 | 98.4 |
| Medications on Admission | 95.4 ± 2.78 | 99.5 | 99.6 | 99.5 |
Table 9. Model extraction performance by clinical section at the case level (with 95% confidence margins of error).
| Section | Ambiguity-excluded Item-level (%) | |||
| Section | Acc. | Prec. | Rec. | F1 |
| Allergies | 99.8 ± 0.32 | 99.8 | 100.0 | 99.9 |
| Chief Complaint | 97.6 ± 2.37 | 97.9 | 99.7 | 98.8 |
| Discharge Condition | 97.4 ± 3.15 | 99.4 | 97.9 | 98.7 |
| Discharge Diagnosis | 97.1 ± 2.65 | 97.7 | 99.4 | 98.5 |
| Major Procedures | 98.3 ± 2.04 | 98.5 | 99.7 | 99.1 |
| Past Medical History | 95.2 ± 1.48 | 97.5 | 97.5 | 97.5 |
| Physical Examination | 95.2 ± 1.83 | 98.1 | 97.0 | 97.5 |
| Pertinent Results | 92.2 ± 3.81 | 96.2 | 95.7 | 96.0 |
| Discharge Medications | 97.2 ± 1.17 | 98.6 | 98.6 | 98.6 |
| Medications on Admission | 99.3 ± 0.47 | 99.7 | 99.5 | 99.6 |
Table 10. Model extraction performance by clinical section at the item level (with 95% confidence margins of error).
The agreement gap in the medication sections stems from how annotators handled incomplete dosage and strength details in the raw notes. When the source text omitted medication strengths, one clinician routinely flagged the extraction as “Ambiguous” while others marked it as “Approved”. While this difference in labeling threshold lowered the exact raw agreement, both annotators agreed that the LLM successfully extracted everything present in the source.
Table 9 and Table 10 report the complete per-section validation breakdown of model performance at the case and item levels, while Table 11 provides the detailed inter-annotator agreement results. Table 8 provides the counts and percentages of extraction errors and ambiguous judgments across all clinical sections.
| Section | Case IAA (tolerant) | Item IAA (tolerant) | AC1 (tolerant) |
| Allergies | 98.0 | 99.8 | 99.8 |
| Chief Complaint | 96.0 | 98.0 | 98.0 |
| Discharge Condition | 98.0 | 98.0 | 98.0 |
| Discharge Diagnosis | 96.0 | 96.0 | 95.8 |
| Major Procedures | 96.0 | 98.0 | 98.0 |
| Past Medical History | 56.0 | 90.4 | 90.1 |
| Physical Examination | 68.0 | 90.9 | 90.4 |
| Pertinent Results | 80.0 | 88.7 | 88.4 |
| Discharge Medications | 76.0 | 97.1 | 97.1 |
| Medications on Admission | 80.0 | 97.3 | 97.2 |
Table 11. Inter-Annotator Agreement (Raw IAA and Gwet’s AC1) by clinical section.
Details on Comparison to MIMIC Tables
In this section, we provide details on the comparison of each MNE section to the existing MIMIC tables. MIMIC tables span different modules, which differ in both content and cohort. The hospital-wide hosp module covers all admissions, while the emergency-department (ED) and intensive-care (ICU) modules add finer triage and bedside detail but only for the admissions treated in those units.
Gender The extracted gender field maps directly to hosp.patients.gender: both use the values M and F. 99.96% of documented genders match exactly, leaving 122 admissions with differences. Manual review of these mismatching admissions shows that most are explained by transgender, gender-identity, or transition-related documentation in the discharge notes. Thus, MNE’s gender field can serve as a sanity check for correctness.
Services The extracted services field corresponds to hosp.services.curr_service. However, MIMIC stores abbreviated transfer-level service codes rather than the broad labels used in our table. hosp.services represents service history as timed transfer rows, using transfer time and abbreviated codes such as MED, SURG, and NSURG. We map these codes to our extraction labels before comparison. Many mappings are one-to-one, but some MIMIC codes map to the same extracted category (e.g. MED, CMED, and OMED to MEDICINE), and a few extracted labels have no direct counterpart. We test whether the discharge-summary header records the service at discharge by comparing it with the last MIMIC service entry. Among 300,221 comparable admissions, 99.16% match exactly. Allowing any mapped service during the admission to count as a match increases the overlap by only 175 cases. Thus, while MNE largely contains service information already available in MIMIC, it also provides a useful sanity check by flagging cases with mismatched service assignments.
Discharge disposition The extracted discharge disposition field contains four broad categories: HOME, HOME_WITH_SERVICE, EXTENDED_CARE, and EXPIRED. hosp.admissions stores the more detailed discharge_location, together with death indicators such as hospital_expire_flag and deathtime. We map MIMIC discharge locations to extraction labels where the correspondence is clear, for example HOME to HOME, HOME HEALTH CARE to HOME_WITH_SERVICE, and skilled nursing or rehabilitation facilities to EXTENDED_CARE. The main remaining unmapped MIMIC destinations are HOSPICE and AGAINST ADVICE, which occur in roughly 5,000 admissions and are more specific than our four-category schema. Among 251,208 admissions with a mapped non-missing MIMIC disposition, 96.87% match the extracted value.
| Section | Related MIMIC table(s) | Contribution |
| Already exists in MIMIC, but can flag mismatches | ||
| Gender | h.patients.gender | Flag mismatches |
| Services | h.services.curr_service | Flag mismatches |
| Discharge Disposition | h.admissions.discharge_location | Flag mismatches; adds 44k missing values |
| Related to MIMIC information, but broader or richer | ||
| Discharge Diagnosis | h.diagnoses_icd | Concise clinician-written diagnoses with primary/secondary structure, preserving note wording instead of dense, generic ICD codes |
| Chief Complaint | e.triage.chiefcomplaint | more admissions; cleaner terms and additional clinical context |
| Allergies | h.diagnoses_icd (allergy codes) | Dedicated structured allergen list covering far more admissions than MIMIC’s sparse ICD codes |
| Past Medical History | h.diagnoses_icd (history codes) | more history rows; contains broader prior conditions, family, substance, and functional history, compared to MIMIC’s ICD codes |
| Medications on Admission | e.medrecon | more admissions; more detail by providing dose, route, frequency, PRN, and hold status |
| Major Procedures | h.procedures_icd; h.hcpcsevents | Contains compact clinical procedure label and more surgical details compared to coded ICD/HCPCS billing tables |
| No structured MIMIC counterpart | ||
| Discharge Condition | none | Provides structured discharge state, including mental status and functional activity |
| Discharge Medications | none | Provides discharge-medication list |
| Physical Examination | none | Provides system-level examination findings with normal/abnormal status |
| Pertinent Results | none | Provides important clinical results, e.g., radiology interpretations |
Table 12. Extracted sections grouped by how they relate to MIMIC’s existing tables, with the closest related table(s) and the contribution of our extraction. Table prefixes: h=hosp, e=ED, i=ICU.
However, the MIMIC table is missing a discharge location for 44,209 admissions, which are in MNE. These cases represent added structured information rather than disagreement with MIMIC. Thus, the extracted field validates well against mapped MIMIC values and fills many missing location values.
Allergies MIMIC has no dedicated structured allergy table, but hosp.diagnoses_icd contains 36 allergy-status billing codes for only 2,211 admissions such as V140 (“personal history of allergy to penicillin”), V1507 (“latex allergy”), and broader codes such as V1505 (“Allergy to other foods”). MNE’s allergy table covers 169,595 admissions with at least one documented allergen and contains 606,069 allergy rows overall. We compare the overlapping subset by selecting ICD-9 allergy-status codes V14*, V1504, V1505, V1507, V1508, and V1509, and ICD-10 codes Z88* and Z910*. We then map both sources to broad allergen categories where a correspondence is possible, such as penicillin, sulfonamide, contrast, latex, seafood, antibiotic, analgesic, or narcotic allergy, using substring matching. Using this approximate category mapping, about 70% of 2,211 admissions with ICD allergy-status codes have a matching extracted allergen category in our table. Overall, we contribute a new table with detailed structured allergy lists, which are not part of MIMIC’s tables.
Chief complaint. The extracted chief complaint field has a partial structured counterpart in ed.triage.chiefcomplaint, but the coverage and semantics differ. The ED triage table captures the presenting complaint at ED arrival and is only available for ED stays, whereas the discharge summary can contain admission complaints, hospital-stay complaints, and additional clinical context. MNE’s chief complaint table covers 296,132 admissions, while ed.triage.chiefcomplaint is available for 140,466 admissions in the discharge-summary cohort. There are 137,035 admissions with both sources. For these paired admissions, we concatenate multiple extracted chief complaint rows and multiple ED triage complaints within each admission, normalize both strings by lowercasing, removing non-alphanumeric characters, and normalizing whitespace, and compare the resulting strings. Normalized exact agreement is 11.0%, while token-set string similarity at the threshold of 90 reaches 29.2%. These low agreement rates reflect that the two fields are related but not interchangeable: ED triage values are short arrival complaints and abbreviations, while discharge-summary chief complaints are often more comprehensive with additional clinical context. The extracted field therefore adds broader hospital-level chief-complaint coverage rather than duplicating ED triage.
Major procedures. The extracted major procedures field is related to the coded procedure sources in the hosp module (procedures_icd and hcpcsevents), but these tables use standardized procedure codes for billing and reporting, while our field contains concise clinician-written procedure summaries. Considering only procedures from the current admission, the extracted major-procedure table covers 146,515 admissions, while 175,599 admissions in our cohort have ICD procedure codes and 44,514 have HCPCS events. The current-admission extracted procedure subset contains 250,504 procedure rows, compared with 564,069 ICD or HCPCS rows in the related MIMIC tables for the same cohort. In practice, the sources are only partially comparable: ICD and HCPCS often encode approach, site, device, material, and billing-specific detail, while the discharge note usually gives a compact clinical procedure label or surgical details. Overall, major procedures provide clinician-written procedure summaries to MIMIC.
Admission and discharge medications. MIMIC contains several medication-related tables, but they answer different questions. hosp.prescriptions, hosp.pharmacy, hosp.emar, and icu.inputevents describe inpatient orders, administrations, or ICU inputs, not a reconciled pre-admission home-medication list. The closest structured counterpart for medications on admission is ed.medrecon, which contains ED medication reconciliation rows but covers only the ED subset. In our discharge-summary cohort, ed.medrecon is available for 116,387 admissions, while our extracted admission-medication table covers 275,035 admissions. There are 112,294 admissions with both sources and 162,741 admissions with extracted admission medications but no mapped ED medication reconciliation. For the overlap subset, we first compare medication names with conservative normalized exact matching, which gives a lower-bound agreement of 44.72% from extracted medications to ED medrecon and 43.61% in the reverse direction. We then map each medication name to an RxNorm ingredient key when a mapping is available (Nelson et al., 2011); unmapped names retain their canonicalized string as fallback. This increases row-level agreement to 74.25% from extracted medications to ED medrecon and 72.63% in the reverse direction. These values should be interpreted as same-admission agreement with ED medication reconciliation, not as extraction accuracy, because the two sources can differ in timing, completeness, and medication-name granularity. The extracted admission-medication table also provides structured attributes not exposed by ed.medrecon, including dose, strength, route, frequency, duration, PRN status, and hold instructions. The extracted discharge-medication table covers 285,926 admissions and adds a structured discharge-state medication list. No standard MIMIC table exposes discharge medications as a single reconciled list.
Discharge diagnosis. The extracted discharge diagnoses are related to hosp.diagnoses_icd, but the two sources differ in purpose. Our table captures clinician-written diagnoses from the discharge summary, whereas diagnoses_icd contains standardized billing-coded diagnoses, including comorbidities, complications, history/status codes, and ICD-specific granularity. Coverage is high in both sources: 297,537 admissions have extracted discharge diagnoses, 300,546 have ICD diagnoses, and 297,375 have both. However, ICD is much denser, with 3,720,215 diagnosis rows and a mean of 12.38 rows per admission, compared with 982,748 extracted discharge-diagnosis rows and a mean of 3.30 rows per admission. We first compare same-admission strings directly against ICD long titles. Raw exact matching links only 4.32% of extracted diagnosis rows to an ICD title, and conservative normalization increases this to 15.79%. This normalization expands common abbreviations such as HTN to hypertension and removes generic ICD title wording such as “unspecified”, “other and unspecified”, and “not elsewhere classified”. We then use token-set similarity as an approximate string-matching analysis. At threshold 80, 44.87% of extracted diagnosis rows have a sufficiently similar same-admission ICD title. Thus, fewer than half of the diagnoses in our extracted discharge-diagnosis list can be linked to a single billed ICD diagnosis title using this approximate string comparison. Manual inspection shows that many non-links are representational differences rather than obvious errors: clinician-written diagnoses may use concise phrases, etiologic combinations, post-procedure context, or problem-level labels that require combining several ICD rows or translating ICD-specific wording. The ICD table remains the appropriate source when standardized billing codes are required. Our extracted table contributes a compact, directly interpretable clinical diagnosis list with primary versus secondary structure, preserving the wording and grouping used in the discharge summary rather than requiring users to reconstruct this summary from multiple coded rows.
Discharge condition The discharge condition fields, such as mental status and functional activity, have no direct structured counterpart in MIMIC hosp, ICU, or ED. This section therefore adds new structured discharge-state information for nearly the full cohort.
Past medical history The closest structured MIMIC counterpart for past medical history is the subset of history/status billing codes in hosp.diagnoses_icd, but this is only an indirect comparison. Our extracted past-medical-history table contains 3,616,698 rows across 290,212 admissions, including clinician-written prior conditions, family history, prior interventions, substance history, and functional status. The selected ICD history/status subset contains 373,062 rows across 181,890 admissions, and both sources are present for 178,619 admissions. These ICD codes mainly represent selected billable history and status concepts, such as prior tobacco use, prior thromboembolism, malignancy history, cardiac bypass or angioplasty status, dialysis status, and implanted devices. Thus, ICD history/status codes provide useful context, but they are not a direct validation target for the extracted past-medical-history section. Our table adds substantially broader structured clinical and family-history content.
Physical exam The physical exam section has no direct structured counterpart in MIMIC. ED triage, ED vitalsign, and ICU chartevents contain measured vital or bedside observations, often tracked repeatedly over time and only for ED or ICU subsets. In contrast, our extracted physical-exam tables cover 294,708 admissions and represent note-derived examinations at documented timepoints, mainly admission and discharge. They contain 434,633 vital-sign rows and 4,780,963 system-level exam rows with normal/abnormal status and abnormal findings. The system-level clinician assessment is the main contribution, because it is not available as an equivalent structured MIMIC hospital table. Therefore, the related MIMIC measurement tables are orthogonal rather than comparable.
Pertinent results The pertinent results table has no direct structured counterpart in MIMIC. hosp.labevents and hosp.microbiologyevents contain detailed raw laboratory and microbiology events, but these are not part of our extracted table. We deliberately do not extract laboratory and microbiology event values from discharge summaries because all note timepoints are redacted and the note text is noisy and incomplete compared with the structured MIMIC event tables. Instead, pertinent results provides a structured index over the clinician-selected results. We segment the section into typed subsections each annotated with its line-range boundaries in the source text, the verbatim text they delimit, and per-type categorical tags such as imaging modality and body part. We do not summarize or reinterpret the underlying values: the clinician’s original wording, including any interpretation of radiology findings, is preserved verbatim. However, a brief (3–5 word) topic label is added. The section therefore adds a structured classification and indexing layer over clinically relevant results.
Experimental Details on CDM Benchmarks
| Model | Reasoning | Full information | Agentic |
| Gemini 3.5 Flash | minimal | 88.7 [87.4, 89.9] | 87.9 [87.2, 88.5] |
| Gemini 3.1 Flash Lite | minimal | 87.4 [86.1, 88.7] | 81.9 [81.2, 82.7] |
| Gemini 3.1 Flash Lite | high | 88.6 [87.3, 89.8] | 84.2 [83.5, 84.9] |
| Gemma 4 31B | 86.7 [85.3, 88.0] | 83.7 [82.9, 84.4] | |
| Qwen3.6 35B-A3B | 90.4 [89.1, 91.5] | 83.6 [82.9, 84.3] | |
| Qwen3.6 35B-A3B | ✓ | 89.3 [88.0, 90.5] | 82.6 [81.8, 83.3] |
| Qwen3.5 2B | 82.0 [80.4, 83.5] | 51.3 [50.2, 52.2] |
Table 13. Diagnosis accuracy (%) on the full-information and agentic CDM benchmarks, with 95% Wilson confidence intervals in brackets.
Cohort Construction
To construct the agentic CDM benchmark, we first identify admissions with a usable extracted primary discharge diagnosis and then apply a sequence of tabular filters to obtain a high-quality pool of diagnostic cases. After filtering, 27,963 admissions remain as the candidate pool, from which we draw cases over the 19 classes (Table 14).
| Class | Cases | Class | Cases |
| Heart failure exacerbation | 731 | Acute cholecystitis | 646 |
| Gastrointestinal bleeding | 731 | Appendicitis | 566 |
| Cellulitis | 731 | Pancreatitis | 434 |
| Small bowel obstruction | 731 | Cholangitis | 346 |
| Pneumonia | 731 | Diverticulitis | 219 |
| Urinary tract infection | 731 | Diabetic ketoacidosis | 172 |
| COPD exacerbation | 730 | Aseptic meningitis | 159 |
| Myocardial infarction | 730 | Pancreatic cancer | 120 |
| Pulmonary embolism | 730 | Spontaneous bacterial peritonitis | 67 |
| Colitis | 695 |
Table 14. The 19 benchmark classes and the number of cases drawn.
Agent Harness
Interaction loop. Each case is evaluated separately with no memory retained between cases. The agent is initialized with a system prompt that frames the diagnostic task and fixes the answer format, followed by the patient demographics and the triage chief complaint. It then runs as a reason-and-act loop: it reasons, optionally issues one or more tool calls, reads the returned results, and repeats until it commits a final answer. The candidate class names are supplied to the model, and the committed “Final diagnosis:” line is scored by case- and punctuation-insensitive substring matching. We cap at ten tool-calling rounds. On reaching the cap the agent is instructed to stop calling tools and output a diagnosis.
Tools and rendering. The agent interacts with each case through the five tools listed in Table 15. Each tool returns only the items the agent names, so the case is assembled incrementally. Two tools additionally support a coarse-to-fine request pattern that keeps the initial view compact: the patient-record tool can return the list of available sections with their item counts (e.g. conditions (7), allergies (2)) before the agent asks for the content of a chosen section, and the laboratory tool can list the individual tests within a panel or body-fluid group before any are ordered.
Tool results are rendered as compact, uniform text: laboratory values as (Fluid) Test Label: Value | RR: [low – high] with the reference range appended when available; physical-examination systems as -
System: Status - findings lines preceded by a vital-signs line; patient-record sections as headed bullet lists; and imaging and clinical studies as the study title followed by the clinician-written report or result summary. When a case never underwent a requested imaging or clinical study, the tool returns an explicit not-available message, and an already-returned study cannot be requested again.
Pertinent results. The pertinent results section mixes study findings with physician decisions and diagnostic interpretations. Fields containing diagnostic interpretation were excluded to prevent diagnosis leakage. The remaining findings were divided between the imaging and clinical studies tools according to study type.
Models and Inference
The models and inference code are shared by the agentic benchmark and the full-information setting. We access the Gemini models [13 b] through the Google Cloud Vertex AI API with zero data retention and serve the open-weight models locally with vLLM [25]: Qwen3.6-35B-A3B (Qwen Team, 2026a), a mixture of experts with 35B total and 3B active parameters, Gemma 4 31B [15], and Qwen3.5-2B (Qwen Team, 2026b), all obtained from the Hugging Face Hub (Wolf et al., 2020). Qwen3.6-35B-A3B is served in FP8 (Qwen/Qwen3.6-35B-A3B-FP8), Gemma 4 31B in W4A16 (google/gemma-4-31B-it-qat-w4a16-ct), and Qwen3.5-2B unquantized.
Agent Tool-Use Analysis
We analyze how our fine-grained tools are used. We find that tool-call behavior differs substantially across models. Interestingly, increasing or enabling reasoning reduced the mean number of tool calls for both Gemini 3.1 Flash Lite and Qwen3.6 (Table 16). In terms of tool call errors, Qwen3.5 2B frequently repeated requests and had the highest rates of malformed and excessive requests.
| Model | Reasoning | Mean | Median | Malformed | Excessive |
| Gemini 3.5 Flash | minimal | 6.95 | 7 | 2.01 | 1.85 |
| Gemini 3.1 Flash Lite | minimal | 7.21 | 7 | 1.89 | 10.16 |
| Gemini 3.1 Flash Lite | high | 3.73 | 4 | 0.92 | 0.39 |
| Gemma 4 31B | × | 5.54 | 5 | 0.56 | 0.59 |
| Qwen3.6 35B-A3B | × | 3.61 | 4 | 2.75 | 0.72 |
| Qwen3.6 35B-A3B | ✓ | 2.95 | 3 | 5.30 | 0.16 |
| Qwen3.5 2B | × | 9.09 | 11 | 8.10 | 30.64 |
Table 16. Mean and median attempted tool calls per completed case, and percentage of calls with malformed or excessive requests. Excessive requests include repetitions and calls denied by request limits.
Across all models, the agent’s first tool call is almost always the patient record (88.7–100% of cases) and the physical examination (92.2–100%), and it queries each only once or twice per case. Both are never requested in full: every physical-examination call names specific body systems, and each record read returns a median of three of the eight sections, typically after an initial call listing which sections exist.
The remaining tools are requested selectively, tracking what each disease requires (Figure 3, for the most accurate model, Gemini 3.5 Flash). Imaging is ordered in 98.0% of appendicitis, small-bowel obstruction, acute cholecystitis, cholangitis, diverticulitis, and pneumonia cases, where it is diagnostically decisive, and in 47% of myocardial-infarction cases. The clinical-study tool shows the sharpest targeting: 99.6% of myocardial-infarction and 97.5% of heart-failure cases, where an electrocardiogram is the decisive test, against 2–8% of abdominal and skin pathologies. This tool exists only because MIMIC-Note-Extract contains electrocardiograms and other studies from the pertinent-results extraction; the regex-derived benchmark could not expose them and cardiac pathologies were absent there.
Figure 3. Fraction of cases in which Gemini 3.5 Flash queries each tool at least once, by pathology. The patient record and physical examination are queried almost universally, whereas imaging and the clinical-study (electrocardiogram, endoscopy) tool are requested selectively.
Disease-specific laboratory coverage. To assess whether agents obtained relevant diagnostic evidence, laboratory tests were grouped into disease-specific categories following Hager et al. [18]. For pancreatitis, for example, these categories cover inflammation, pancreatic enzymes, and disease severity. A case was counted when the agent obtained at least one laboratory result from each category. Table 17 reports the percentage of cases meeting this criterion for four pathologies.
| Model | Reasoning | Appendicitis () | Acute cholecystitis () | Diverticulitis () | Pancreatitis () |
| Gemini 3.5 Flash | minimal | 99.1 | 98.8 | 99.1 | 94.4 |
| Gemini 3.1 Flash Lite | minimal | 94.5 | 97.7 | 94.9 | 78.7 |
| Gemini 3.1 Flash Lite | high | 90.4 | 84.8 | 83.3 | 47.1 |
| Gemma 4 31B | × | 89.6 | 87.7 | 81.9 | 57.9 |
| Qwen3.6 35B-A3B | × | 88.4 | 80.3 | 76.4 | 52.3 |
| Qwen3.6 35B-A3B | ✓ | 63.8 | 69.5 | 57.9 | 19.9 |
| Qwen3.5 2B | × | 59.7 | 19.7 | 37.0 | 3.2 |
Table 17. Percentage of cases in which at least one laboratory result was obtained from every disease-specific category. Only cases completed by all model configurations and with results available from every category were included; counts are shown in the headings.
Coverage varied across models and diseases, with the largest differences observed for pancreatitis. Interestingly, increasing reasoning in Gemini 3.1 Flash Lite reduced laboratory coverage while improving diagnosis accuracy, whereas enabling reasoning in Qwen3.6 reduced both coverage and accuracy (Table 4). These results show that lower laboratory coverage does not necessarily imply lower diagnostic accuracy.
References
- [2]Debangshu Banerjee, Tarun Suresh, Shubham Ugare, Sasa Milailovic, and Kagandeep Singh. CRANE: Reasoning with constrained LLM generation. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=wKs9fHYxCV.arxiv.org/abs/2502.09061
- [3]Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Guiding llms the right way: fast, non-invasive constrained generation. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.
- [4]Kimberly G. Blumenthal, Na Lu, Yuqing Zhang, Yu Li, Rochelle P. Walensky, and Hyon K. Choi. Risk of meticillin resistant Staphylococcus aureus and Clostridium difficile in patients with a documented penicillin allergy: population based matched cohort study. BMJ, 361:k2400, 2018. doi: 10.1136/bmj.k2400.DOI
- [5]Samuel Colvin, Eric Jolibois, Hasan Ramezani, Adrian Garcia Badaracco, Terrence Dorsey, David Montague, Serge Matveenko, Marcelo Trylesinski, Sydney Runkle, David Hewitt, Alex Hall, and Victorien Plot. Pydantic Validation, May 2026. URL https://github.com/pydantic/pydantic. Version 2.13.3.
- [6]Patricia L. Cornish, Sandra R. Knowles, Romina Marchesano, Vincent Tam, Steven Shadowitz, David N. Juurlink, and Edward E. Etchells. Unintended medication discrepancies at the time of hospital admission. Archives of Internal Medicine, 165 (4):424–429, 2005. doi: 10.1001/archinte.165.4.424.DOI
- [7]Jean-Benoit Delbrouck, Justin Xu, Johannes Moll, Alois Thomas, Zhihong Chen, Sophie Ostmeier, Asfandyar Azhar, Kelvin Zhenghao Li, Andrew Johnston, Christian Bluethgen, Eduardo Pontes Reis, Mohamed S Muneer, Maya Varma, and Curtis Langlotz. Automated structured radiology report generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26813–26829, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1301. URL https://aclanthology.org/2025.acl-long.1301/.DOI
- [8]Mohamed Elgaar, Jiali Cheng, Nidhi Vakil, Hadi Amiri, and Leo Anthony Celi. MedDec: A dataset for extracting medical decisions from discharge summaries. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 16442–16455, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.975. URL https://aclanthology.org/2024.findings-acl.975/.DOI
- [9]Dyke Ferber, Lars Hilgers, Christiane Höper, Benedict Kinny-Köster, Jan-Niklas Eckardt, Katharina Egger-Heidrich, Marius Bill, Martin M. K. Schneider, Jan Clusmann, Lejla Kadric, Marcel Oehme, Maximilian Mayrhofer-Schmid, Alexander Oeser, Georg Wölflein, Isabella C. Wiest, Jan Moritz Middeke, A. John Iafrate, Daniel Truhn, Dirk Jäger, and Jakob Nikolas Kather. Towards autonomous medical artificial intelligence agents. Nature, 2026. doi: 10.1038/s41586-026-10675-5. URL https://doi.org/10.1038/s41586-026-10675-5.
- [10]Farieda Gaber, Maqsood Shaik, Fabio Allegra, Agnes Julia Bilecz, Felix Busch, Kelsey Goon, Vedran Franke, and Altuna Akalin. Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis. npj Digital Medicine, 8(1):263, 2025. doi: 10.1038/s41746-025-01684-1. URL https://doi.org/10.1038/s41746-025-01684-1.
- [11]Thomas Gilbert, Jenny Neuburger, Joshua Kraindler, Ellis Keeble, Paul Smith, Cono Ariti, Sandeepa Arora, Andrew Street, Stuart Parker, Helen C. Sam Henry, Kevin Buchan, Michele Filannino, Amber Roberts, Martin Bardsley, and Simon Conroy. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic hospital records: an observational study. The Lancet, 391(10132):1775–1782, 2018. doi: 10.1016/S0140-6736(18)30668-8.DOI
- [12]Google DeepMind. Gemini 2.5 Flash, 2025. URL https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-2-5-Flash-Model-Card.pdf. Accessed: 16.06.26.
- [13]Google DeepMind. Gemini 3.5 Flash, 2026a. URL https://deepmind.google/models/model-cards/gemini-3-5-flash/. Accessed: 16.06.26.
- [14]Google DeepMind. Gemini 3.1 Flash Lite, 2026b. URL https://deepmind.google/models/model-cards/gemini-3-1-flash-lite/. Accessed: 16.06.26.
- [15]Google DeepMind. Gemma 4, 2026c. URL https://deepmind.google/models/gemma/gemma-4/. Accessed: 16.06.26.
- [16]Mehak Gupta, Brennan Gallamoza, Nicolas Cutrona, Pranjal Dhakal, Raphael Poulain, and Rahmatollah Beheshti. An extensive data processing pipeline for mimic-iv. Proceedings of Machine Learning Research, 193:311–325, 2022.arxiv.org/abs/2204.13841
- [17]Kilem Li Gwet. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48, 2008. doi: 10.1348/000711006X126600.DOI
- [18]Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, and Daniel Rueckert. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine, 30(9):2613–2622, 2024. doi: 10.1038/s41591-024-03097-1. URL https://doi.org/10.1038/s41591-024-03097-1.
- [19]Hrayer Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1):96, 2019. doi: 10.1038/s41597-019-0103-9. URL https://doi.org/10.1038/s41597-019-0103-9.
- [21]Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P. Lungren, Andrew Y. Ng, Curtis Langlotz, and Pranav Rajpurkar. Radgraph: Extracting clinical entities and relations from radiology reports. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https://openreview.net/forum?id=pMWtc5NKd7V.
- [22]Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV-Note: Deidentified free-text clinical notes. PhysioNet, January 2023a. doi: 10.13026/n174-ne17. URL https://doi.org/10.13026/n174-ne17. Version 2.2.
- [23]Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. Mimic-iv, a freely accessible electronic health record dataset. Scientific Data, 10(1):1, 2023b. doi: 10.1038/s41597-022-01899-x. URL https://doi.org/10.1038/s41597-022-01899-x.
- [24]Jonas Kuntzer, Kristian Schwethelm, Conrad Ketzer, Eva Wende, Moritz Schlager, Daniel Rueckert, Georgios Kaissis, Friederike Jungmann, and Paul Hager. MIMIC-IV-Ext MIMIC Note Extract: A Large-Scale Structured Dataset Extracted from MIMIC-IV-Note Discharge Summaries. PhysioNet, October 2026. doi: 10.13026/cq50-rd11. URL https://doi.org/10.13026/cq50-rd11. Version 1.0.
- [25]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611–626, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 978400702297. doi: 10.1145/3600006.3613165. URL https://doi.org/10.1145/3600006.3613165.


