WorksComputer Science
OmitEHR: Scalable Omission Detection in LLM-Generated Clinical Summaries
Abstract
Health systems increasingly rely on large language models (LLMs) to draft summaries of patient medical records, yet evaluations now find omissions, the absence of clinically important information, to be the most common failure mode. Detecting omissions in an LLM-generated summary requires knowing which facts in the record are salient, a signal that existing methods hand-specify, ignore, or recompute at inference with costly LLM judges. We present OmitEHR, a reference-free omission detector that learns clinical salience from historical clinician-written summaries predating widespread LLM use. OmitEHR learns to predict which source-note sentences clinicians include in their summaries, then surfaces high-salience sentences not covered by an LLM-generated summary as candidate omissions. We apply OmitEHR to brief hospital course (BHC) summarization using 5,000 admissions from a large academic medical center. On held-out admissions, OmitEHR outperformed all evaluated baselines at predicting salient source sentences, including a prompted LLM judge. The full omission detection pipeline took 17.7 to 19.5 seconds and cost $0.03 to $0.13 per summary depending on the base model, 1.6 to 1.7 times faster and 5.6 to 6.7 times cheaper than scoring every source sentence with an LLM judge. In a blinded review of 25 admissions, two board-certified clinicians confirmed a mean of 5.6 clinically meaningful omissions per summary surfaced by OmitEHR, compared with 3.3 for a full LLM pipeline, counting candidates confirmed by at least one clinician. The efficiency and validity of OmitEHR make it a practical tool for health systems to flag omissions before a summary reaches the clinician.
Data and Code Availability
This work uses de-identified EHR data from an academic medical center. The clinical notes may contain protected health information and cannot be shared publicly. Code for the labeling pipeline, salience model, and omission detector is available at https://github.com/alsentzerlab/omitehr.
Institutional Review Board (IRB)
This study was conducted using de-identified electronic health record data under a protocol approved by the Stanford University Institutional Review Board (protocol #57916).
Introduction
Chart review and documentation occupy a substantial share of clinicians’ time, with studies finding that nearly half is consumed by desk work [22, 25, 4]. In response, health systems are increasingly deploying tools powered by large language models (LLMs) that summarize the electronic health record (EHR) at the point of care, including Epic’s integrated chart summarization ([12]), Stanford’s ChatEHR ([21]), and Penn Medicine’s Chart Hero ([19]). Although LLM-generated summaries may improve clinician productivity, they exhibit two key failure modes: 1) hallucinations, statements in the summary not supported by the patient’s record, and 2) omissions, clinically important information from the record that is missing from the summary. Existing research primarily targets hallucinations, where each claim in a summary can be evaluated for support against the source (e.g., [7]). However, clinician evaluations identify omissions as the more frequent failure mode. Across 147 AI-generated chart summaries, clinicians flagged 46 omissions and only 5 hallucinations ([12]), and in a separate evaluation of GPT-4 emergency department summaries, 47% of summaries contained omissions and 42% hallucinations ([26]).
Omissions can disrupt handoffs, delay care, and increase the risk of adverse events ([23, 13]), and they are harder to detect than hallucinations because there is no fixed set of claims to verify. A detector must review the patient chart, decide which facts are salient, and then determine whether the summary covers them. Salience is central to omission detection, yet existing methods either ignore it or estimate it with LLM judges at inference time ([20, 5] (Oukeloun et al. 2025)). Judging salience with an LLM is expensive, as it requires reviewing the entire source record, and frequently diverges from physicians’ judgments of relevance ([10]). LLM services charge by token usage, so repeatedly processing full records makes omission detection more expensive as records grow longer. As a result, health systems now monitor token spending as a budget line and discontinue AI tools whose cost exceeds their value [8].
Supervision to train a clinical salience model, however, already exists at scale. Clinicians have written summaries of hospitalizations for decades, and these summaries provide evidence of which information they considered worth including. Across thousands of admissions, these inclusion decisions provide a training signal for clinical salience that requires no manual annotation.
We introduce OmitEHR, a reference-free omission detector that learns clinical salience from these historical inclusion decisions (Figure 1). OmitEHR separates omission detection into two steps, identifying salient information in the source record and determining whether the generated summary covers it. For the first step, a model trained on historical record–summary pairs predicts, for a new admission, which source sentences a clinician’s summary would have covered, effectively predicting the content of the reference summary that is unavailable at deployment. For the second, an LLM judge checks only the highest-ranked sentences for coverage by the generated summary, and uncovered sentences are surfaced as candidate omissions. Since salience is learned offline, the only LLM computation at deployment is a fixed number of coverage checks, so the cost of detection does not grow with the length of the record.
Figure 1. The OmitEHR pipeline. Labels indicate content absent from (0), indirectly covered by (1), or directly stated in (2) the BHC. (a) At training time, an LLM aligns each source-note sentence to the clinician-written BHC, and the resulting inclusion labels train a contextual salience model. (b) At deployment, the model ranks the sentences of a new admission by salience, only the highest-ranked sentences are checked for coverage against the LLM-written BHC, and uncovered sentences are surfaced as candidate omissions. Examples are illustrative.
We apply OmitEHR to brief hospital course (BHC) summarization, which synthesizes documentation from across a hospitalization into the narrative hospital-course section of the discharge summary [1]. Using 5,000 admissions comprising 2.9 million source sentences, we train a bidirectional Transformer over source-sentence embeddings to predict each sentence’s salience in the context of the full admission. Training labels indicate whether each source sentence is represented in the clinician-written BHC. At inference, OmitEHR screens an LLM-generated BHC for omissions in 17.7 to 19.5 seconds and at a cost of $0.03 to $0.13 on average, 1.6 to 1.7 times faster and 5.6 to 6.7 times cheaper than judging every source sentence with an LLM. In a blinded review of 25 admissions, OmitEHR surfaced a mean of 5.64 omissions per summary confirmed by at least one clinician, compared with 3.32 for a full LLM pipeline.
This paper contributes (1) a reframing of reference-free omission detection as learned clinical salience followed by summary-coverage assessment, (2) a scalable pipeline for deriving salience labels from historical clinician-written BHCs and a contextual salience model that outperforms structural, lexical, and prompted LLM-judge baselines, and (3) a low-cost omission detection pipeline whose surfaced omissions clinicians confirmed in a blinded review.
Related Work
Reference-free omission detection. Omissions in LLM-generated clinical summaries have generally been identified through manual clinician review [12, 26] or by comparison against a clinician-written reference, but manual review does not scale and the reference is unavailable at deployment, since it is exactly the document the LLM was asked to produce. Reference-free methods instead detect omissions from the source record and the generated summary alone. Embed-KDECheck (Oukelmoon et al., 2025) embeds source and summary text and uses kernel density estimation to flag regions of the source the summary does not cover, which is inexpensive but does not model clinical importance. MED-OMIT [20] extracts atomic facts from dialogue, flags those an LLM judge finds excluded from a generated SOAP note, and weights each by its effect on a simulated differential diagnosis as a proxy for importance. The most closely related formulation is CARE [5], which uses LLM judges at inference time to assign a clinical importance and summary coverage score to each source sentence, and calibrates thresholds on both scores with conformal risk control. Scoring every source sentence for every summary is expensive, and LLM estimates of clinical relevance frequently diverge from those of physicians [10]. OmitEHR adopts the same importance-and-coverage decomposition as CARE but learns salience from historical clinician-written BHCs, so no LLM importance judgments are made at inference and salience is grounded in observed clinician decisions rather than in a judge’s reading of a prompt.
Learning salience from clinician behavior. Clinician behavior has been utilized to provide a relevance signal for related tasks. Jiang et al. [11] derive relevance labels from EHR audit logs, treating the notes a clinician opened before documenting as evidence of notes worth reading. In general-domain summarization, content selection is classically learned from sentence- or entity-level inclusion labels obtained from reference summaries [17, 16, 15]. SPEER (Adams et al., [1]) applies this idea to BHC summarization, labeling medical entities from the source note as salient when they appear in the reference BHC and training a classifier whose predictions guide a fine-tuned LLM’s sentence planning. OmitEHR uses the same supervision but differs in unit and purpose. It predicts salience over whole sentences rather than isolated terms, and it audits summaries rather than generating them, so the salience model can check the output of any summarizer for omitted content.
Problem Formulation
Our goal is to identify clinically important information from the source notes of a hospitalization that a corresponding LLM-generated summary fails to include. Formally, let denote the sentences from an admission’s source record, and let be an LLM-written summary of that record. OmitEHR identifies an omission separately from two judgments: (i) whether is salient, meaning it belongs in the summary, and (ii) whether it is covered, meaning its content appears in .
We define the salience of as the probability that a clinician would include its clinical content in the summary, , conditioned on the entire record. We learn this function from an offline set of historical clinician-written summaries . For each source record and its summary , each source sentence receives an inclusion label indicating whether its content is absent from, indirectly reflected in, or directly stated in . Treating labels 1 and 2 as positive, we train to predict .
We define the coverage of a sentence as its inclusion label in an LLM-generated summary, , using the same inclusion scale. Coverage is a property of a specific sentence–summary pair and must be computed at inference. OmitEHR retains the most salient sentences from the source records, defined as , and surfaces those not covered by . The tunable budget controls the tradeoff between omission recall, review burden, and inference cost. The resulting set of candidate omissions is defined as:
Methods
Dataset Construction
We assemble a dataset of 5,000 internal medicine admissions at a large academic medical center. We restrict the cohort to stays longer than 24 hours, where longitudinal summarization is more useful. For each admission, we concatenate all notes from the hospitalization, excluding the discharge summary, to form the source. Notes are also the primary input to deployed summarization systems, though the exact input construction may vary (Shah et al., [21]; Kahl et al., [12]; Penn Medicine, [19]). The target summary is the brief hospital course (BHC) section of the discharge summary. Whereas other sections are often templated, the BHC is a clinician-written narrative of the admission’s key events and therefore provides naturally occurring supervision for which source information clinicians considered salient. Additional dataset characteristics are reported in Appendix § 7.
Sentence segmentation. We segment the source notes into sentences using the medspaCy clinical sentence splitter (Eyre et al., [9]). We then apply rule-based preprocessing that removes templated content (Cahoon et al., [6]) and splits bulleted and numbered list items, which medspaCy often merges, into separate segments. We use sentences rather than LLM-extracted atomic facts because segmentation is deterministic, auditable, and preserves the note’s wording, whereas decomposition requires an LLM call over every note and can drop or alter clinical details.
Salience labels. Given each source sentence, its source note, and the clinician-written BHC, we prompt GPT-5-mini to assign one of three labels describing how the sentence’s content is reflected in the BHC (Table 1). Across the 2.9 million sentences, 63.5% are labeled not related, 30.5% indirectly related, and 5.9% directly included (Appendix § 7).
To validate the labeling scheme, two internal medicine clinicians independently annotated 300 sentences from 15 admissions using the same rubric, blinded to the LLM labels and to each other, with each sentence shown alongside its source note and the BHC. A third annotator adjudicated disagreements. The two clinicians agreed with each other at a linearly weighted Cohen’s kappa of 0.647, and GPT-5-mini’s labels showed moderate agreement with each clinician (0.454 and 0.414). Appendix § 8 details the labeling pipeline, annotation interface, and full agreement results.
| Label | Definition | Source-note example |
| Not related (0) | The BHC makes no claim this sentence supports. | “All other systems reviewed and are negative.” |
| Indirectly related (1) | Supports a BHC claim but adds a value, timing, or method the BHC does not state. | “Chronic bronchiectasis, presented with 3 days of worsening SOB.” |
| Directly included (2) | States the same fact as the BHC, in the same or different words. | “Etiology of symptoms include bronchiectasis flare, PNA, or viral process.” |
caption: Inclusion-label rubric used for training and inference. Examples. The review of systems supports no BHC claim (0). The bronchiectasis history relates to the BHC’s etiology claim but adds history and symptom timing the BHC does not state (1). The etiology line states the same candidate causes as the BHC (2).
:::
Salience Model
We train a class of models to predict the salience of each source sentence. We collapse the three inclusion labels of Table 1 into a binary target, treating indirectly related and directly included sentences as salient and not related sentences as non-salient.
Our primary model, OmitEHR-contextual, represents each sentence as its mean-pooled BioClinRoBERTa token embeddings [14] projected to 256 dimensions, plus a learned positional embedding and a 256-dimensional projection of structural features. These structural features consist of the sentence’s position within the note, the note’s position within the admission, the admission’s length in days, the time since the first note and until discharge, how often the same sentence recurs in other notes, its token length, the note’s title, and the section header it falls under.
The admission’s sentence representations then pass through a two-layer, eight-head Transformer encoder with hidden size 256 and a linear classification head. A sentence’s salience can therefore depend on its context, for example whether it is repeated or rendered unimportant by its surrounding course. We fine-tune the final eight layers of BioClinRoBERTa jointly with the rest of the model and train with binary cross-entropy, weighting positive examples by the class ratio of each block.
To isolate the contributions of fine-tuning and of the Transformer encoder, we train two sentence-level ablations that share the same per-sentence representation but score each sentence on its own, without attending to the admission’s other sentences. OmitEHR-frozen keeps BioClinRoBERTa frozen and trains only the feature projections and classification head, while OmitEHR-fine-tuned also updates its final eight layers. We repeat all three variants with BioClinicalBERT ([3]) and BioClinical ModernBERT ([24]) as the base encoder. Appendix § 9 provides further architectural and training details.
Baselines and Evaluation Protocol
Baselines. We compare against three families of baselines. Structural heuristics score a sentence with a single scalar (relative position in the record, length in tokens, or repetition count) and test whether salience reduces to document structure. Lexical models test whether salience reduces to wording. We use a token-level naive Bayes classifier, with and without length normalization, and logistic-regression and MLP classifiers over TF-IDF sentence vectors. Finally, the LLM judge mirrors the per-sentence importance judging of inference-time detectors ([5]). For each sentence, we prompt GPT-5-mini to estimate the probability that a physician would include its content in the BHC. Each call explicitly supplies up to 20 consecutive source sentences to score, together with shared admission context consisting of the first 40,000 characters of the admission’s notes. The judge always receives each sentence it scores together with neighboring sentences in the same batch, regardless of its position in the record; the 40,000-character limit applies only to the shared admission context. This shared context gives the judge admission-level information to promote a fair comparison to OmitEHR-contextual.
Evaluation. All models are evaluated on a held-out test set of 750 admissions containing over 446K sentences, 37% of which the labeling pipeline marks salient. For each admission we compute AUROC and AUPRC over that admission’s sentences alone, then average across admissions. Confidence intervals come from a hierarchical bootstrap over admissions. We test whether the contextual model outperforms the strongest sentence-level model with a paired bootstrap over admissions with 20,000 replicates.
Omission Detection
OmitEHR can be applied to any system that generates a BHC. At inference time, we apply the same preprocessing used during training and score every source sentence with the trained salience model. We retain the highest-scoring unique sentences, where controls the size of the candidate pool. We use throughout our experiments to provide a broad candidate pool for coverage assessment, though the cutoff can be tuned to deployment needs. An LLM judge then checks whether each retained sentence is already covered by the generated summary, using the same inclusion rubric used in salience label generation (Table 1), now applied to the LLM-generated BHC. As defined in § 3, sentences among the highest-ranked candidates that receive a coverage label of 0 are surfaced as candidate omissions. Additional details on the omission detection pipeline are provided in Appendix § 11.
Clinician Evaluation of Omissions
To evaluate whether the candidates OmitEHR surfaces are omissions a clinician would act on in practice, and to compare them with the candidates an LLM pipeline surfaces at the same review budget, we ran a blinded clinician evaluation. We sampled 25 admissions from the test set, generated a BHC for each with Gemini-2.5-pro, and ran both OmitEHR and the full LLM-judge pipeline. The pipelines differed in salience scoring: OmitEHR used its learned salience model, while the full LLM pipeline used GPT-5-mini to score clinical importance; both used Gemini-2.5-pro for coverage assessment with . The 10 highest-ranked candidate omissions from each pipeline per admission were reviewed independently by two board-certified clinicians. For each admission, we provided the candidate omission, along with the complete source notes from the admission and the LLM-generated BHC (Figure 2). Since this was a retrospective annotation study where reviewers had not participated in the patients’ care, each candidate omission was accompanied by a brief LLM-generated rationale derived from the source notes. This rationale situates the sentence in the admission’s timeline, cause, and history. It withholds any verdict on inclusion and is not part of the detection pipeline. Prompts are included in Appendix § 14.

Figure 2. Example candidate omissions with completed fields: source sentence and note type, verdict buttons, LLM-generated rationale, and the coverage judge’s reason. Patient content is synthetic.
Results
Salience Classification
Table 2 reports the AUROC, AUPRC, and recall at 100 (R@100) for salience prediction, computed within each admission and then averaged, so each model is judged on how well it ranks the sentences of a single record. R@100 is the fraction of an admission’s salient sentences ranked in its top 100, matching our coverage-labeling budget. Structural heuristics perform near chance, with the best, sentence length, reaching only 0.593 AUROC and 0.360 R@100. This suggests that salience is not reducible to document structure. Lexical models perform markedly better (TF–IDF with logistic regression: 0.767 AUROC, 0.621 AUPRC), suggesting that vocabulary carries substantial signal. The GPT-5-mini judge reaches 0.806 AUROC and 0.668 AUPRC, better than every lexical model and OmitEHR-frozen.
| Family | Model | AUROC | AUPRC | R@100 |
| Heuristic | Position | 0.537 ± .004 | 0.394 ± .007 | 0.313 ± .014 |
| Length | 0.593 ± .004 | 0.452 ± .006 | 0.360 ± .014 | |
| Repetition | 0.506 ± .004 | 0.379 ± .007 | 0.292 ± .013 | |
| Lexical | Token naive Bayes | 0.730 ± .004 | 0.588 ± .008 | 0.458 ± .016 |
| Token naive Bayes (length-normalized) | 0.740 ± .004 | 0.567 ± .008 | 0.448 ± .017 | |
| TF–IDF + logistic regression | 0.767 ± .003 | 0.621 ± .007 | 0.476 ± .016 | |
| TF–IDF + MLP | 0.730 ± .004 | 0.574 ± .007 | 0.451 ± .016 | |
| LLM judge | GPT-5-mini (zero-shot) | 0.806 ± .004 | 0.668 ± .008 | 0.519 ± .018 |
| OmitEHR-frozen | BioClinRoBERTa | 0.789 ± .003 | 0.647 ± .007 | 0.492 ± .017 |
| BioClinicalBERT | 0.780 ± .003 | 0.635 ± .007 | 0.485 ± .016 | |
| BioClinical ModernBERT | 0.784 ± .003 | 0.639 ± .007 | 0.488 ± .017 | |
| OmitEHR-fine-tuned | BioClinRoBERTa | 0.825 ± .003 | 0.691 ± .007 | 0.519 ± .018 |
| BioClinicalBERT | 0.819 ± .003 | 0.682 ± .007 | 0.515 ± .018 | |
| BioClinical ModernBERT | 0.816 ± .003 | 0.677 ± .007 | 0.512 ± .018 | |
| OmitEHR-contextual | BioClinRoBERTa | 0.828 ± .003 | 0.697 ± .007 | 0.522 ± .018 |
| BioClinicalBERT | 0.821 ± .003 | 0.685 ± .007 | 0.517 ± .018 | |
| BioClinical ModernBERT | 0.816 ± .004 | 0.677 ± .007 | 0.513 ± .018 |
Table 2. Admission-level salience classification on the held-out test set (750 admissions, over 446K sentences). R@100 is salience recall among the 100 highest-ranked sentences per admission. Each cell reports the point estimate ± the half-width of the 95% hierarchical-bootstrap confidence interval. Bold marks the best result.
Fine-tuning the encoder accounts for most of the improvement among the OmitEHR variants. Tuning the final eight layers of BioClinRoBERTa while still scoring each sentence independently raises performance by 3.6 AUROC points and 4.4 AUPRC points over the frozen encoder and also surpasses the LLM judge when evaluated against the LLM-derived inclusion label. Adding the contextual encoder provides an additional gain of 0.3 AUROC points and 0.6 AUPRC points. This improvement, while modest, is statistically significant under a paired bootstrap (), but it comes at additional computational cost, so the fine-tuned sentence-level model remains a viable lighter-weight option. Across encoders, BioClinRoBERTa is the strongest for every model class, fine-tuning improves every encoder, and context adds a small gain for BioClinRoBERTa and BioClinicalBERT but none for BioClinical ModernBERT. Additional salience results, including evaluation against clinician annotations, are reported in Appendix § 10.
Clinician Evaluation
Two board-certified clinicians independently reviewed the ten highest-ranked candidate omissions from OmitEHR and the full LLM-judge pipeline on the same 25 LLM-generated BHCs (Figure 3). Of the 250 OmitEHR candidates reviewed, 141 (56.4%) were judged to be clinically meaningful omissions by at least one clinician and 71 (28.4%) by both, compared with 83 (33.2%) and 17 (6.8%), respectively, for the full LLM pipeline (Figure 3a). Per summary, OmitEHR surfaced a mean of 5.64 omissions confirmed by at least one clinician and 2.84 confirmed by both, compared with 3.32 and 0.68 for the full LLM pipeline. OmitEHR also surfaced at least one omission confirmed by both clinicians in 24 of 25 admissions, compared with 12 of 25 for the full LLM pipeline. The two pipelines performed similarly when clinicians reviewed only the top-ranked candidate, and OmitEHR’s advantage widened as the review budget grew (Figure 3b). Precision at each review budget and per-clinician decisions are reported in Appendix F.
Figure 3. Blinded clinician review of candidate omissions surfaced by OmitEHR and the full LLM pipeline (GPT-5-mini) on 25 LLM-generated BHCs. (a) Verdicts from two board-certified clinicians on the ten highest-ranked candidates per admission (250 per pipeline), each reviewed independently. (b) Mean number of omissions per summary confirmed by at least one clinician among the top ranked candidates. At , OmitEHR surfaced 2.3 more confirmed omissions per summary than the full LLM pipeline. Additional precision results reporting confirmation by both clinicians are reported in Table 11.
Cost and Latency
Averaged over the 25 evaluation admissions, screening one generated BHC with OmitEHR takes 17.7 s for $0.03 with GPT-5-mini for coverage labeling and 19.5 s for $0.13 with Gemini-2.5-pro. We compare OmitEHR against a full LLM pipeline that uses the same model for both salience and coverage (Figure 4).
Figure 4. Mean cost and latency of screening one generated BHC, by pipeline stage, across the 25 evaluation admissions. Each row uses the specified model for all LLM calls. Within each row, OmitEHR and the full LLM pipeline differ only in salience scoring: OmitEHR uses its learned salience model, whereas the full LLM pipeline scores every source sentence with the LLM. OmitEHR’s salience step incurs approximately $0.002 in GPU cost and is too small to be visible. Latencies are wall-clock times with up to 16 concurrent API calls. Details in Appendix § 13.
The first three stages of OmitEHR—sentence splitting, salience scoring, and top- selection with —take 6.3 s and require no LLM API calls. Coverage labeling is the only LLM-dependent stage.
The full LLM pipeline differs only in how salience is assessed: rather than using OmitEHR’s learned salience model, it asks an LLM to score every source sentence in the admission. LLM-based salience assessment alone consumes 10 to 11 times as many input tokens as OmitEHR’s entire pipeline and takes 12.7–15.9 s, compared with 2.0 s for OmitEHR’s salience model. End to end, OmitEHR is 5.6 to 6.7 times cheaper and 1.6 to 1.7 times faster, and its LLM cost is capped at whereas the full pipeline’s grows with the number of sentences. Additional comparisons of cost and latency are located in Appendix § 13.
Discussion
This work demonstrates that reference-free omission detection need not infer clinical importance from scratch at inference time. Instead, salience can be learned from historical clinician-written summaries and used to identify important source content. Using
this supervision, a compact contextual salience model outperforms structural, lexical, and LLM-judge baselines at predicting which source content belongs in a BHC. OmitEHR combines this learned notion of salience with coverage assessment to surface clinically important information omitted from an LLM-generated summary.
Learning salience also changes the computational structure of omission detection. Inference-time LLM judges must repeatedly inspect the source record to determine what matters, causing salience-assessment cost to grow with record length. OmitEHR instead predicts salience with a learned model and reserves LLM-based coverage assessment for a fixed set of top- candidates. As a result, the expensive LLM component is bounded by rather than by the number of source sentences. This distinction matters for real-time clinical use: an omission detector must add minimal latency if it is to screen summaries without slowing clinical workflows. For long longitudinal records, reading the full chart can otherwise dominate both latency and cost.
The clinician evaluation suggests that clinically meaningful omissions remain present in LLM-generated summaries. OmitEHR surfaced a mean of 5.64 omissions per summary confirmed by at least one clinician, compared with 3.32 for the full LLM-judge pipeline. It also surfaced at least one omission confirmed by both clinicians in 24 of 25 admissions, compared with 12 of 25 for the baseline. These results suggest that learned salience can more effectively direct clinicians to clinically important missing information.
The OmitEHR recipe can extend to any summary type with a history of clinician-written references, such as specialist consult notes or oncology progress notes, and conditioning the salience model on the querying provider, specialty, and clinical question is a direct next step. Beyond auditing, the same framework could support summary generation itself. Surfaced candidates can be fed back to the summarizer for revision, and predicted salience can guide generation directly, as SPEER does with entity-level labels [1].
Limitations. Our study has several limitations that motivate future work. First, we focus only on brief hospital course generation from internal medicine admissions at a single academic medical center. Future work should evaluate generalization to other sites, services, and summary types. Second, our salience evaluation relies on silver-standard labels that show moderate agreement with clinician annotations ( to ). Accordingly, the results in Table 2 measure agreement with an LLM-derived target rather than with clinical judgment directly. These labels allow for evaluation at scale but inherit limitations from both the LLM-based labeling procedure and the clinician-written summaries themselves, which may not capture all clinically important information. We therefore complement the salience evaluation with a blinded clinician evaluation to provide an independent downstream assessment of whether the surfaced candidates are clinically meaningful. Third, our omission evaluation measures precision among the surfaced candidates, not recall over all clinically important omissions. Measuring recall would require clinicians to review entire records, about 580 sentences per admission, which was beyond our annotation budget. The salience Recall@K and candidate-count analyses capture the trade-off between computational cost and salience coverage but do not establish omission recall, which remains an experiment for future work. Fourth, our clinician evaluation was limited to 25 admissions and two reviewers, neither of whom had prior knowledge of the cases they reviewed. LLM-generated rationales were provided to contextualize each candidate omission but may have influenced judgments despite instruction to withhold an explicit omission verdict. Additionally, the absence of covered or low-salience control candidates in our clinician evaluation prevents estimation of reviewers’ false-positive rate. Future work should expand clinician evaluation to familiar cases, include blinded control candidates, and compare judgments with and without rationales.
Dataset and Label Distribution
The 5,000 admissions are split by patient into training, development, and held-out test sets, so no patient contributes to more than one split. The training, development and test sets contain 3500, 750 and 750 admissions respectively. Table 3 reports the distribution of silver inclusion labels in each split. Class proportions are stable across splits, with 36–38% of sentences salient under the binary target (labels 1 and 2).
| Split | Sentences | 0: Not related | 1: Indirectly related | 2: Directly included |
| Train | 2,054,630 | 1,312,014 (63.9%) | 620,859 (30.2%) | 121,757 (5.9%) |
| Dev | 397,998 | 248,090 (62.3%) | 125,557 (31.5%) | 24,351 (6.1%) |
| Test | 446,467 | 282,259 (63.2%) | 138,679 (31.1%) | 25,529 (5.7%) |
Table 3. Silver inclusion-label distribution by split. Values are sentence counts with within-split percentages in parentheses.
Entailment Label Validation
To assess the entailment labeling quality of the LLM Judge, two internal medicine clinicians independently annotated the same 300 sentences sampled from 15 distinct admissions. Both were blinded to the LLM labels and to each other’s annotations, and each sentence was presented alongside the full source note and BHC. An example screen of the annotation platform is shown in Figure 5.

Figure 5. Interface used by clinicians to review source chunks and assign entailment labels relative to the corresponding clinician-written BHC.
The two clinicians agreed with each other on 81.0% of sentences (linearly weighted , 95% CI 0.560 to 0.728). We treat this as the practical ceiling for the task, since a labeler cannot be expected to agree with either clinician more closely than the clinicians agree with each other. The clinicians differ on 57 of 300 sentences, most often at the boundary between labels 0 and 1 (33 of the 57). GPT-5-mini reaches 70.7% and 68.7% exact agreement ( and 0.414) against the two clinicians respectively (Table 4).
| Raters | Exact agr. | Wtd. |
| Clinician 1 vs. 2 | 81.0% | 0.647 |
| Clinician 1 vs. GPT-5-mini | 70.7% | 0.454 |
| Clinician 2 vs. GPT-5-mini | 68.7% | 0.414 |
Table 4. Pairwise annotation agreement on the 300-sentence blinded review set. Weighted uses linear weights across the ordered three-class label scheme.
Out of the 300 reviewed sentences, the labels for 185 were agreed upon by all three parties (Clinician 1, Clinician 2, and GPT-5-mini). A third annotator independently reviewed the remaining 115 disputed sentences in a blinded review. Among the 58 cases in which Clinicians 1 and 2 agreed with each other but disagreed with GPT-5-mini, the adjudicator agreed with the LLM in 26 cases (44.8%; Table 5). This near-even split suggests that the remaining disagreements are mostly sentences that sit between two labels, where annotators themselves differ, rather than systematic errors by the LLM.
| Initial disagreement pattern | Adjudicator agrees with LLM | |
| R1 and R2 agree against LLM | 58 | 26 (44.8%) |
| R1 and R2 split, one agrees with LLM | 48 | 31 (64.6%) |
| R1 and R2 split, neither agrees with LLM | 9 | 4 (44.4%) |
| All initially disputed sentences | 115 | 61 (53.0%) |
Table 5. Adjudicator review of the 115 sentences for which at least one clinician disagreed with GPT-5-mini. R1 and R2 denote the two clinicians.
Salience Model and Training Details
Architecture. Figure 6 shows the architecture of OmitEHR-contextual. Each source sentence is encoded independently with RoBERTa-large-PM-M3-Voc (Lewis et al., [14]). The final-layer token representations are mean-pooled into one sentence embedding and projected to 256 dimensions. A second 256-dimensional vector is projected from the structural features and added to it, along with a learned sentence-position embedding. The structural features are relative position within the admission and within the note, admission length, time since the first note and until discharge, within-admission repetition count, token length, and learned embeddings of the note title (hashed into 128 buckets) and of a rule-based section category (9 categories). The resulting sequence passes through a two-layer, eight-head Transformer encoder with hidden dimension 256, GELU activations, and dropout 0.1. Admissions are divided into contiguous blocks of at most 1,024 sentences that retain source order. A layer-normalized linear head produces one logit per sentence, converted by a sigmoid into a salience probability.
Figure 6. The contextual salience model. Each sentence embedding is combined with projected structural features and a position embedding, contextualized by an admission-level Transformer encoder, and scored by a per-sentence head.
Training. The model is trained with class-weighted binary cross-entropy, treating silver labels 1 and 2 as salient and label 0 as non-salient, and weighting positive examples by the negative-to-positive ratio of the training block. The final eight encoder layers are fine-tuned jointly with the projections, contextual encoder, and head, while earlier encoder layers are frozen. The final checkpoint is selected by admission-level AUPRC on the development set. The sentence-level variants (OmitEHR-frozen and OmitEHR-fine-tuned) use the same sentence representation, structural features, and head, but omit the contextual encoder and score each sentence independently. We use AdamW with weight decay 0.01 and gradient clipping at 1.0, with a learning rate of for the fine-tuned encoder layers and for the contextual encoder, projections, and head. Training runs for three epochs on a single NVIDIA H100 GPU with mixed precision and gradient checkpointing.
Additional Salience Results
This appendix supplements Table 2 in three ways. First, we report recall at for every model. Second, we repeat salience training and evaluation with direct inclusion as the only positive class, meaning label 2 of the rubric in Table 1, content stated directly in the BHC, rather than labels 1 and 2 together. Third, we rescore OmitEHR-contextual and the LLM judge on the 300 test sentences the clinicians annotated in Appendix § 8 to check whether a ranking learned from LLM labels holds up against clinician judgment.
Salience recall. Table 6 reports the recall of the salience model at (R@K), the fraction of an admission’s salient sentences ranked among its top , averaged over admissions, for . corresponds to the number of candidate sentences passed to coverage labeling in the main pipeline (§ 5.3). At every cutoff, the fine-tuned and contextual OmitEHR models outperform all heuristic and lexical baselines, and contextual BioClinRoBERTa has the highest or tied-highest recall.
| Family | Model | R@10 | R@50 | R@100 | R@200 | R@400 |
| Heuristic | Position | 0.031 ± .002 | 0.158 ± .009 | 0.313 ± .014 | 0.567 ± .020 | 0.832 ± .018 |
| Length | 0.047 ± .003 | 0.200 ± .009 | 0.360 ± .014 | 0.606 ± .018 | 0.847 ± .017 | |
| Repetition | 0.030 ± .002 | 0.148 ± .008 | 0.292 ± .013 | 0.547 ± .019 | 0.822 ± .018 | |
| Lexical | Token naive Bayes | 0.058 ± .003 | 0.267 ± .011 | 0.458 ± .016 | 0.696 ± .017 | 0.888 ± .015 |
| Token naive Bayes (length-normalized) | 0.051 ± .003 | 0.247 ± .011 | 0.448 ± .017 | 0.703 ± .019 | 0.896 ± .015 | |
| TF–IDF + logistic regression | 0.065 ± .003 | 0.275 ± .012 | 0.476 ± .016 | 0.720 ± .018 | 0.901 ± .014 | |
| TF–IDF + MLP | 0.057 ± .003 | 0.254 ± .011 | 0.451 ± .016 | 0.696 ± .018 | 0.889 ± .015 | |
| LLM judge | GPT-5-mini (zero-shot) | 0.071 ± .004 | 0.307 ± .013 | 0.519 ± .018 | 0.748 ± .018 | 0.905 ± .013 |
| OmitEHR-frozen | BioClinRoBERTa | 0.066 ± .003 | 0.286 ± .012 | 0.492 ± .017 | 0.736 ± .018 | 0.908 ± .014 |
| BioClinicalBERT | 0.066 ± .003 | 0.280 ± .012 | 0.485 ± .016 | 0.729 ± .018 | 0.905 ± .014 | |
| BioClinical ModernBERT | 0.064 ± .003 | 0.283 ± .012 | 0.488 ± .017 | 0.733 ± .018 | 0.906 ± .014 | |
| OmitEHR-fine-tuned | BioClinRoBERTa | 0.071 ± .004 | 0.305 ± .013 | 0.519 ± .018 | 0.761 ± .018 | 0.918 ± .013 |
| BioClinicalBERT | 0.069 ± .004 | 0.301 ± .013 | 0.515 ± .018 | 0.757 ± .018 | 0.916 ± .014 | |
| BioClinical ModernBERT | 0.069 ± .004 | 0.298 ± .013 | 0.512 ± .018 | 0.754 ± .018 | 0.915 ± .014 | |
| OmitEHR-contextual | BioClinRoBERTa | 0.071 ± .004 | 0.308 ± .013 | 0.522 ± .018 | 0.763 ± .017 | 0.919 ± .013 |
| BioClinicalBERT | 0.069 ± .004 | 0.302 ± .013 | 0.517 ± .018 | 0.758 ± .018 | 0.917 ± .014 | |
| BioClinical ModernBERT | 0.069 ± .004 | 0.298 ± .013 | 0.513 ± .018 | 0.755 ± .018 | 0.916 ± .013 |
Table 6. Admission-level salience recall at on the held-out test set (750 admissions). R@K is the fraction of an admission’s salient sentences ranked among its top , averaged over admissions. Each cell reports the point estimate ± the half-width of the 95% hierarchical-bootstrap confidence interval. Bold marks the best result in each column.
Training on direct inclusion labels. Table 2 treats labels 1 and 2 (see Table 1) as salient, so its positive class is cut at the boundary between labels 0 and 1, which is where the clinicians disagree most often (Appendix § 8). We therefore repeated the experiment with label 2 alone, content stated directly in the BHC, as both the training target and the evaluation target. We retrained the lexical and OmitEHR models on label-2 training labels and scored every model, including the heuristics and the GPT-5-mini judge, against label-2 test labels. Table 7 reports the best heuristic and lexical baselines, the judge, and the two trained OmitEHR variants. On the test set, 5.7% of sentences have label 2. Contextual BioClinRoBERTa reaches 0.886 AUROC and 0.356 AUPRC, and its top 100 sentences capture 78.5% of an admission’s label-2 sentences. The GPT-5-mini judge falls below the lexical baseline on this target.
| Model | AUROC | AUPRC | R@10 | R@50 | R@100 | R@200 |
| Position | 0.589 ± .010 | 0.087 ± .005 | 0.025 ± .006 | 0.172 ± .016 | 0.366 ± .022 | 0.629 ± .023 |
| TF–IDF + logistic regression | 0.807 ± .007 | 0.246 ± .011 | 0.145 ± .011 | 0.460 ± .019 | 0.657 ± .019 | 0.844 ± .016 |
| GPT-5-mini judge (zero-shot) | 0.749 ± .007 | 0.149 ± .007 | 0.080 ± .009 | 0.365 ± .020 | 0.577 ± .021 | 0.775 ± .018 |
| OmitEHR-fine-tuned | 0.882 ± .005 | 0.347 ± .013 | 0.216 ± .015 | 0.590 ± .019 | 0.777 ± .017 | 0.910 ± .012 |
| OmitEHR-contextual | 0.886 ± .005 | 0.356 ± .014 | 0.223 ± .015 | 0.592 ± .019 | 0.785 ± .017 | 0.913 ± .012 |
Table 7. Admission-level salience classification and recall at with label 2 as the only positive class, on the held-out test set (750 admissions, 5.7% positive). OmitEHR models use BioClinRoBERTa. Position and TF–IDF + logistic regression are the best heuristic and lexical baselines. R@K is defined as in Table 6. Each cell reports the point estimate ± the half-width of the 95% hierarchical-bootstrap confidence interval. Bold marks the best result.
Evaluation against clinician labels. Table 2 evaluates models against GPT-5-mini’s labels, which show only moderate agreement with clinician annotations. To assess alignment with physician judgment, Table 8 rescores OmitEHR-contextual and the LLM judge on the 300 clinician-annotated test sentences described in Appendix § 8, against both the original GPT-5-mini labels (silver standard) and the clinician majority labels (gold standard). Against gold labels the original OmitEHR model reaches 0.820 AUROC and 0.702 AUPRC, close to its silver-label results on the same sentences (0.842 and 0.717 respectively). The LLM Judge’s point estimates are higher against the gold labels, consistent with silver-label evaluation favoring the model trained on those labels. However, neither AUROC nor AUPRC differs significantly between the two models, leaving insufficient evidence to distinguish their ranking performance. With direct inclusion as the target, the model trained on label 2 only outperforms the judge reprompted for that target against both silver and gold labels. The difference is significant for AUROC against silver labels (0.876 versus 0.692, ) and for AUPRC against gold labels (0.502 versus 0.318, ).
| Training target | Evaluation labels | AUROC | AUPRC | ||||
| OmitEHR | LLM judge | OmitEHR | LLM judge | ||||
| Labels 1 or 2 | Silver (GPT-5-mini) | 0.842 | 0.819 | 0.313 | 0.717 | 0.699 | 0.621 |
| Gold (clinicians) | 0.820 | 0.857 | 0.137 | 0.702 | 0.764 | 0.203 | |
| Label 2 only | Silver (GPT-5-mini) | 0.876 | 0.692 | 0.031 | 0.478 | 0.330 | 0.078 |
| Gold (clinicians) | 0.846 | 0.691 | 0.095 | 0.502 | 0.318 | 0.018 |
Table 8. Admission-level salience ranking of OmitEHR-contextual and the LLM judge on the 300 clinician-annotated sentences of Appendix § 8, scored against GPT-5-mini labels (silver) and clinician majority labels (gold) with the positive class matching the training target. The gold label-2 evaluation uses the 292 sentences with a resolved majority. In the second block the judge is reprompted for direct inclusion. -values are from paired admission-level sign-flip tests.
Omission Candidate Identification
At inference, OmitEHR ranks source sentences by predicted salience and sends the top to the coverage judge, as defined in § 3. Here, is a configurable inference-time budget controlling the number of sentences evaluated for coverage. For our evaluation, we sampled 25 admissions from the test set and used Gemini 2.5 Pro to generate a brief hospital course (BHC) for each admission. Since notes within an admission often repeat sentences verbatim, we deduplicate identical sentences before ranking so that the top candidates are unique. We selected for all admissions to provide a sufficiently large candidate pool for the subsequent coverage assessment. This yielded 2,500 source sentences for coverage evaluation. Gemini-2.5-pro was used to label each sentence as directly covered, indirectly related, or not covered by the generated BHC using Prompt C in Appendix § 14. As shown in Table 9, the coverage model classified 49.9% as directly included, 25.1% as indirectly related, and 25.0% as not represented in the generated BHC. Sentences judged not covered are treated as candidate omissions and ranked by salience.
| Coverage outcome | (%) |
| Directly included evidence | 1,248 (49.9%) |
| Indirectly related | 628 (25.1%) |
| Not related (omitted content) | 624 (25.0%) |
| Total | 2,500 (100.0%) |
Table 9. Coverage of the top 100 deduplicated salient source sentences per admission in the review set ( admissions; 2,500 sentences).
Clinician Omission Review Details
For each of the 25 admissions, the two clinician annotators were shown the 10 highest-ranked candidate omissions from both OmitEHR and full LLM pipeline in separate documents, the complete set of notes from the admission, and the generated BHC. Each candidate was additionally accompanied by an LLM-generated rationale (Prompt D in Appendix § 14) describing what event the sentence refers to, when it occurred, why it happened, and relevant history. This rationale was provided purely as supplementary material to assist the clinicians in familiarizing themselves with the care trajectories of these patients who were not under their care, and it withheld any judgment about whether the sentence should have been included. For each candidate sentence, the clinicians first decided whether the candidate was a true, clinically meaningful omission and, for rejected candidates, whether the information was actually covered by the BHC or not important enough to include. Table 10 summarizes paired clinician decisions on the 250 candidates from each pipeline. For OmitEHR, the clinicians agreed on 180 candidates, both confirming 71 as true omissions and rejecting 109 (72.0% agreement; Cohen’s ). The remaining 70 candidates were confirmed by only one clinician, yielding 141 candidates (56.4%) confirmed by at least one clinician. For the full LLM pipeline, both clinicians confirmed 17 candidates and rejected 167 (73.6% agreement; Cohen’s ), with 83 candidates (33.2%) confirmed by at least one clinician. Among the 109 OmitEHR candidates rejected by both clinicians, they agreed on the reason for rejection in 68 instances (62.4%), including 59 salience errors and 9 coverage errors.
| Clinician A | Clinician B | OmitEHR | LLM pipeline |
| True omission | True omission | 71 (28.4) | 17 (6.8) |
| Not omission | Not omission | 109 (43.6) | 167 (66.8) |
| True omission | Not omission | 46 (18.4) | 42 (16.8) |
| Not omission | True omission | 24 (9.6) | 24 (9.6) |
| Confirmed by either clinician | 141 (56.4) | 83 (33.2) | |
| Raw agreement | 180 (72.0) | 184 (73.6) | |
| Cohen’s | 0.43 | 0.18 |
Table 10. Paired clinician decisions on 250 candidates per pipeline across the same 25 admissions. Entries report counts (percentages), except for Cohen’s . The one missing clinician judgment for the LLM pipeline was conservatively treated as a not-omission decision.
Precision by review budget. Holding the coverage-assessment budget fixed at , we evaluated clinician-confirmed precision among the top ranked candidate omissions per admission across 25 admissions. We use two confirmation criteria. Under the either-clinician criterion, a candidate counts as a confirmed omission if at least one of the two clinicians judged it to be one. Under the both-clinician criterion, both clinicians must have judged it to be one. Under the either-clinician criterion, the full LLM-judge pipeline achieved a Precision@10 of 33.2% (83/250), compared with 56.4%
(141/250) for OmitEHR, an absolute improvement of 23.2 percentage points. OmitEHR also achieved higher precision at top-3 (61.3% vs. 44.0%) and top-5 (64.0% vs. 38.4%). Under the stricter both-clinician criterion, Precision@10 was 28.4% (71/250) for OmitEHR and 6.8% (17/250) for the full LLM-judge pipeline. Table 11 reports all cutoffs under both criteria.
| Confirmation | OmitEHR | LLM pipeline | |
| Either clinician | 1 | 48.0% | 52.0% |
| 3 | 61.3% | 44.0% | |
| 5 | 64.0% | 38.4% | |
| 10 | 56.4% | 33.2% | |
| Both clinicians | 1 | 20.0% | 16.0% |
| 3 | 29.3% | 8.0% | |
| 5 | 31.2% | 8.0% | |
| 10 | 28.4% | 6.8% |
Table 11. Clinician-confirmed Precision@n across 25 admissions, with source sentences checked for coverage. Either-clinician confirmation requires at least one clinician to confirm an omission; both clinician confirmation requires both.
Rejection reasons. Table 12 summarizes reasons for rejecting candidates under each pipeline. It includes all candidates rejected by at least one clinician. For both pipelines, clinicians cited insufficient clinical importance more often than existing BHC coverage, suggesting that salience errors were the more common source of rejected candidates.
| OmitEHR | Full LLM pipeline | |
| Rejected by either clinician | 179 | 233 |
| Rejection reasons, n (% of rejected) | ||
| Clinically unimportant | 146 (81.6) | 185 (79.4) |
| Already covered by BHC | 66 (36.9) | 98 (42.1) |
| Unspecified | 8 (4.5) | 35 (15.0) |
Table 12. Rejection reasons for candidates rejected by at least one clinician across 25 admissions. Percentages use each pipeline’s rejected candidates as the denominator. A candidate counts toward every reason cited by either clinician, so reasons are not mutually exclusive. Unspecified: rejected without a reason given.
Cost and Latency Details
The estimated one-time cost of offline labeling all 5,000 admissions with GPT-5-mini was $678.14, or approximately $0.14 per admission (Table 13). All inference-time measurements were taken over the 25 admissions used in the clinician evaluation. Sentence splitting ran on CPU, and salience classification and top-100 selection ran on a single NVIDIA H100 GPU (12 CPU cores, 96 GB of RAM), with timings excluding model initialization. Coverage labeling sent the top-100 candidates in batches of up to 10, grouped by source note, resulting in about 26 calls per admission. LLM salience scoring supplied up to 20 consecutive source sentences per call, together with shared admission context consisting of the first 40,000 characters of the admission’s notes, resulting in about 42 calls per admission. Both processes issued up to 16 requests concurrently within an admission, so latencies are wall-clock times rather than serialized per-request times. Every LLM step was run with both GPT-5-mini and Gemini-2.5-pro (Figure 4). GPT-5-mini salience predictions with Gemini-2.5-pro coverage is the configuration reviewed in the clinician evaluation. Table 14 gives per-admission means and Table 15 the spread of latencies. Costs are computed from recorded token counts at list prices, $0.25 per million input tokens and $2.00 per million output tokens for GPT-5-mini and 10.00 for Gemini-2.5-pro. Coverage cost per admission ranged from 0.035 with GPT-5-mini (median $0.027) and from 0.166 with Gemini-2.5-pro (median $0.127), driven by input tokens that ranged from about 25K to 74K depending on the length of the candidates’ source notes. The GPU-time estimate for OmitEHR assumes $3 per H100-hour over the 49.7 GPU-seconds of salience classification across the 25 admissions.
| Split | Input tokens | Output tokens | Total cost (USD) |
| Train | 734.5M | 114.4M | $412.37 |
| Dev | 233.4M | 30.4M | $119.20 |
| Test | 303.8M | 35.3M | $146.57 |
| Total | 1,271.8M | 180.1M | $678.14 |
Table 13. Token consumption and estimated cost of the one-time offline labeling of all 5,000 admissions with GPT-5-mini, using $0.25 per million input tokens and $2.00 per million output tokens.
| Model | Calls | Tokens (in / out) | Cost | Latency |
| Salience scoring task | ||||
| OmitEHR | 0 | 0 / 0 | $0.0021 | 2.0 s | |
| GPT-5-mini | 41.6 | 492.2K / 5.2K | $0.134 | 12.7 s | |
| Gemini-2.5-pro | 41.9 | 526.6K / 7.8K | $0.737 | 15.9 s | |
| Coverage labeling task | ||||
| GPT-5-mini | 25.5 | 47.4K / 7.8K | $0.027 | 11.4 s | |
| Gemini-2.5-pro | 25.6 | 46.5K / 7.0K | $0.128 | 13.2 s |
Table 14. Mean per-admission API calls, tokens, cost, and latency of each pipeline step by model over 25 admissions. Both coverage runs judge the same top-100 candidates, and their call counts differ only through retries.
| Stage | Median | Min to Max |
| Shared stages (no API calls) | ||
| Sentence splitting | 2.9 | 0.6 to 11.3 |
| Top-100 candidate selection | 0.8 | 0.2 to 3.3 |
| Salience scoring | ||
| OmitEHR | 1.9 | 0.4 to 5.8 |
| GPT-5-mini, every sentence | 9.3 | 2.8 to 50.1 |
| Gemini-2.5-pro, every sentence | 11.7 | 4.0 to 55.4 |
| Coverage labeling of the top-100 candidates | ||
| GPT-5-mini | 9.3 | 8.0 to 21.3 |
| Gemini-2.5-pro | 11.2 | 8.1 to 43.9 |
Table 15. Spread of per-admission latency in seconds for each pipeline stage over 25 admissions. Mean sentence splitting and top-100 selection times are 3.3 s and 1.0 s, and the other means are in Table 14. A pipeline’s end-to-end latency is the sum of the two shared stages, one salience row, and one coverage row.
LLM Prompt Templates
We provide the complete prompt templates used in our experiments. Prompt A was used to assign sentence-level salience labels by measuring the relationship between source-note content and clinician-written BHCs; these labels were used to train the OmitEHR salience model. Prompt B was used to generate BHCs from the source notes of each evaluation admission. Prompt C was used to determine whether salient source-note content was covered by the generated BHCs and to identify candidate omissions. Prompt D was used to generate relevant context for each candidate omission. Prompt E was used as the zero-shot LLM baseline for predicting the clinical salience of source sentences without access to a reference BHC. Placeholders enclosed in braces were replaced with admission-specific inputs at runtime.
Prompt A: Salience Labeling of Clinician-Written BHCs (GPT-5-mini)
SYSTEM_PROMPT = "You are a senior clinician auditing evidence used to construct a Brief Hospital Course (BHC)."
USER_PROMPT = "You receive a BHC, one complete source note, and specific chunks taken from that note. Judge each chunk in
the context of that note. Do not judge a chunk as an isolated sentence and do not rely on medical knowledge to fill
gaps in the BHC.
The task is evidence selection for faithful compression. Imagine selecting the smallest set of source facts from which a
clinician could write this BHC without adding unsupported detail. A chunk can be clinically true and important yet be
0 if it does not support a specific BHC claim.
First resolve the chunk’s subject and scope from the complete source note. Words such as "complications", "stable", "
negative", or "improved" refer to the event, procedure, test, or patient described by that note not automatically to
the hospital course as a whole. The same words can receive different labels in a radiology report, procedure note,
and progress note.
Assign exactly one label:
2 = DIRECTLY INCLUDED. The BHC directly states the same clinical proposition as the chunk, allowing ordinary paraphrase
but preserving material details such as subject, event, certainty, timing, severity, and location. Cite the exact BHC
text. Do not give 2 merely because the BHC mentions the same admission,
diagnosis, procedure, organ system, or test.
1 = INDIRECTLY RELATED. The chunk relates to one identifiable BHC claim, but is not itself directly included. It may
provide method/detail/context for that claim, be broader or narrower, conditional, temporally different, or require
an inference. For example, a procedure note describing HOW the patient was extubated is 1 if the BHC says only that
the patient was extubated. A 1 must name a specific BHC claim; being from the same note, test, or clinical topic is
not enough.
0 = NOT RELATED TO A BHC CLAIM. The chunk does not support a specific BHC claim at the same scope. This includes
boilerplate, unrelated findings, and details from a study/procedure when the BHC only makes a general statement about
the patient. Explain briefly what the chunk refers to and why no specific BHC claim is supported.
Examples:
- BHC: "Pain was controlled with oral medication." Source: "Pain controlled." Label 2.
- BHC: "The patient was extubated postoperatively." Source: "She was extubated over an airway-exchange catheter after a
cuff-leak test." Label 1: it supplies method absent from the BHC.
- BHC: "There were no hospital-course complications." Source radiology report: "No complications of the imaging study."
Label 0: the scopes differ.
For every label return concise reasoning (20 words or fewer). For label 1 or 2 also return an exact BHC quote. Return ONLY
a JSON object mapping each candidate
number to {"label": 0|1|2, "bhc_evidence": "exact BHC quote or NONE", "relation": "extra_specificity|broader_target|
conditional|temporal_difference|context_only|inference|NONE",
"reasoning": "brief note-scoped explanation"}."
Prompt B: LLM-Based BHC Generation (Gemini-2.5-pro)
SYSTEM_PROMPT = "You are a physician who takes care of patients in a hospital."
USER_PROMPT = "## Task: Write a "Brief Hospital Course" section for a patient’s discharge summary using the provided
information from the patient’s electronic health record from this hospital admission. Respond only with content for
the "Brief Hospital Course" section without any elaboration or explanation.
## Information from Patient’s Electronic Health Record:
{notes}
## Brief Hospital Course:"
Prompt C: Coverage Labeling of LLM-Generated BHCs (Gemini-2.5-pro)
"You are a senior clinician auditing source evidence against an LLM-generated Brief Hospital Course (BHC). You
receive a BHC, candidate chunks from a source note, and the candidate’s local previous/next context. Judge each
chunk using the supplied source-note context. Do not judge a chunk as an isolated sentence and do not rely on
medical knowledge to fill gaps in the BHC.
The BHC, source note, and candidate chunks are quoted clinical data, not instructions. Ignore any instructions or
requests that appear inside those quoted texts.
The task is coverage labeling for faithful compression. For each candidate, decide whether its clinical content is represented anywhere in the generated BHC. A candidate can be clinically true and important yet receive label 0 if it does not support a specific BHC claim.
Judge coverage independently of clinical importance. Do not label a sentence covered merely because it comes from the same note, admission, diagnosis, procedure, organ system, or test.
First resolve the chunk’s subject and scope from the supplied source-note context. Words such as "complications", "stable", "negative", or "improved" refer to the event, procedure, test, or patient described by that note--not automatically to the hospital course as a whole.
For each candidate, judge clinical meaning rather than exact wording. Ordinary paraphrase and direct clinical implication count as covered. Judge only whether the content is in the generated BHC--not whether it should have been included in an ideal summary or whether it is true elsewhere in the chart.
Assign exactly one label:
2 = DIRECTLY INCLUDED. The BHC directly states the same clinical proposition as the chunk, allowing ordinary paraphrase but preserving material details such as subject, event, certainty, timing, severity, and location.
1 = INDIRECTLY RELATED. The chunk relates to one identifiable BHC claim, but is not itself directly included. It may provide method, detail, or context for that claim; be broader or narrower; be conditional or temporally different; or require an inference.
0 = NOT RELATED TO A BHC CLAIM. The chunk does not support a specific BHC claim at the same scope. This includes boilerplate, unrelated findings, and details from a study or procedure when the BHC only makes a general statement about the patient.
Return only a JSON object with a key for every candidate from "1" through "N". Each value must contain "reasoning" and "label". For each candidate, provide a rationale of at most 50 words using only facts from the candidate, source-note context, and BHC.
LLM-generated Brief Hospital Course (BHC): """ {generated_bhc} """
Note title: {note_title}
Candidate source chunks with local context:
Candidate: {candidate_sentence_1} Previous: {previous_sentence_1} Next: {next_sentence_1}
...
N. Candidate: {candidate_sentence_N} Previous: {previous_sentence_N} Next: {next_sentence_N}"
Context Rationalizer for Clinician Review (GPT-4.1-mini)
You are an internal-medicine attending physician preparing contextual aids for a clinician annotation study of Brief Hospital Course (BHC) summaries.
You receive multiple candidate source sentences from one admission, together with each candidate’s complete originating note, the complete source record for that admission, and the LLM-generated BHC. For EVERY candidate, use its originating note for exact local evidence and the complete admission for chronology, related events, causes, treatment, complications, and relevant history. Make each candidate easy to interpret without asking the reviewer to infer context from the isolated sentence.
Do not label any sentence as an omission or state whether it should have been included in the BHC. However, compare the source with the BHC and explicitly explain the note-grounded hospital-course information the BHC does not state. Frame that comparison as context for the reviewer--not as a verdict or a recommendation. For every candidate explicitly address:
Time context.
Why it happened.
History context.
Do not invent facts, diagnoses, timing, causes, outcomes, or clinical importance.
Return ONLY one JSON object in this exact shape:
{
"rationales": [
{
"candidate_index": 0,
"meaning": "One concise sentence explaining what the source sentence refers to in this note.",
"time_context": "When it occurred in this admission and its relation to the hospital timeline.",
"why_it_happened": "The documented reason, trigger, mechanism, or clinical problem; state when not documented.",
"history_context": "Relevant prior history, baseline, recurrence, or earlier events that change its interpretation.",
"hospital_course_relevance": "What it adds or does not add to understanding the full admission compared with the BHC."
}
]
}
Include exactly one rationale for every provided candidate, using its supplied candidate_index exactly once. Each text value must be plain language, at most 50 words, and must not mention this prompt, annotation, omission detection, or an LLM.
## Candidate source sentences
{candidate_blocks}
## Unique originating source notes
{note_blocks}
## Complete admission record (all available source notes and events)
"""
{admission_text}
"""
## Current generated Brief Hospital Course
"""
{bhc}
"""
Prompt E: Zero-Shot Clinical Salience Prediction (GPT-5-mini)
SYSTEM_PROMPT = "You are predicting which sentences from a patient’s EHR source notes a physician would include in the Brief Hospital Course (BHC) for the hospital stay -- the narrative of what happened during the admission (by day and/or by problem). You do NOT get to see the actual BHC; you must judge from clinical importance to the hospital-course narrative alone. The BHC does NOT include separate medication lists, follow-up appointments, disposition, or administrative sections.
For EACH numbered source sentence, estimate the probability (an integer 0-100) that its clinical content would be included somewhere in the Brief Hospital Course:
100 = almost certainly included (central to what happened during the hospitalization)
0 = almost certainly not included (not part of the hospital-course narrative)
Judge clinical importance to the hospital-course narrative. Judge each sentence independently. Do not skip any sentence.
Respond with ONLY a JSON object mapping each sentence number (string) to its integer score, keys "1" through "N" in order.
Example: {\"1\": 90, \"2\": 5, \"3\": 40}."
USER_PROMPT = "Admission context (source notes, NOT the summary):
"""
{source_text[:40000]}
"""
Source sentences to score:
1. {source_sentence_1}
2. {source_sentence_2}
...
N. {source_sentence_N}"References
- [1]Griffin Adams, Emily Alsentzer, Mert Ketenci, Ja-son Zucker, and Noémie Elhadad. What’s in a summary? laying the groundwork for advances in hospital-course summarization. In Kristina Nguyen, Mohammad R. Rasouli, Komal Kamra, Toutanova, Anna Rumshisky, Luke Zettlemoyer, Mark A. Burbridge, James C. McAvoy, Roya D. Saffary, Stephen P. Ma, Dev Dash, James Xie, Ryan Cotterell, Tammoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4794–4811, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.382. URL https://aclanthology.org/2021.naacl-main.382/.
- [3]Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78. Association for Computational Linguistics, 2019.arxiv.org/abs/1904.03323
- [4]Brian G. Arndt, John W. Beasley, Michelle D. Watkinson, Jonathan L. Temte, Wen-Jan Tuan, Christine A. Sinsky, and Valerie J. Gilchrist. Tethered to the EHR: Primary care physician workload assessment using EHR event log data and time-motion observations. Annals of Family Medicine, 15(5):419–426, 2017. doi: 10.1370/afm.2121.DOI
- [5]Suhana Bedi, Bridget Lin, Anson Y. Zhou, Chloe O. Stanwyck, Jenelle A. Jindal, Sanmi Koyejo, David Stutz, and Nigam H. Shah. CARE: A conformal safety layer for medical summarization. arXiv preprint arXiv:2606.08969, 2026. doi: 10.48550/arXiv.2606.08969. URL https://arxiv.org/abs/2606.08969.
- [6]Jordan L. Cahoon, Chloe Stanwyck, Asad Aali, Rachel Madding, Emma Sun, Yixing Jiang, Renu-mathy Dhanasekaran, and Emily Alsentzer. Clinical note bloat reduction for efficient llm use, 2026.arxiv.org/abs/2604.16364
- [7]Philip Chung, Akshay Swaminathan, Alex J. Good-ell, Yeasul Kim, S. M. Momsen Reinke, Lichy Han, Ben Deverett, Mohammad Amin Sadeghi, Abdel-Badih Ariss, Marc Ghanem, David Seong, An-drew A. Lee, Caitlin E. Coombes, Brad Brad-shaw, Mahir A. Sufian, Hyo Jung Hong, Teresa P. In Kristina Nguyen, Mohammad R. Rasouli, Komal Kamra, Mark A. Burbridge, James C. McAvoy, Roya Saffary, Stephen P. Ma, Dev Dash, James Xie, Ellen Y. Wang, Clifford A. Schmesing, Nigam Shah, and Nima Aghaee-pour. VeriFact: Verifying facts in LLM-generated clinical text with electronic health records. arXiv preprint arXiv:2501.16672, 2025.arxiv.org/abs/2501.16672
- [9]Hannah Eyre, Alec B. Chapman, Kelly S. Peterson, Jianlin Shi, Patrick R. Alba, Makoto M. Jones, Tamara L. Box, Scott L. DuVall, and Olga V. Patterson. Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python. AMIA Annual Symposium Proceedings, 2021:438–447, 2021.arxiv.org/abs/2106.07799
- [10]Yuexing Hao, Kumail Alhamoud, Hyewon Jeong, Haoran Zhang, Isha Puri, Philip Torr, Mike Schaekermann, Ariel D. Stern, and Marzyeh Ghassemi. MedPAIR: Measuring physicians and AI relevance alignment in medical question answering, 2025. URL https://arxiv.org/abs/2505.24040.
- [11]Sharon Jiang, Barbara D. Lam, Monica Agrawal, Shannon Shen, Nicholas Kurtzman, Steven Horng, David R. Karger, and David Sontag. Machine learning to predict notes for chart review in the oncology setting: a proof of concept strategy for improving clinician note-writing. Journal of the American Medical Informatics Association, 31(7):1578–1582, 2024. doi: 10.1093/jamia/ocae092.DOI
- [12]Nicolas M. Kahl, Marshall J. Frieden, Zach R. Pope, Marlene M. Millen, Vaishal M. Tolia, Theodore C. Chan, Christopher A. Longhurst, Karandeep Singh, and Alan X. You. Evaluation of electronic health record-integrated artificial intelligence chart review. npj Health Systems, 3:6, 2026. doi: 10.1038/s44401-025-00064-x. Article no. 6.DOI
- [13]Sunil Kripalani, Frank LeFevre, Christopher O. Phillips, Mark V. Williams, Preetha Basaviah, and David W. Baker. Deficits in communication and information transfer between hospital-based and primary care physicians: Implications for patient safety and continuity of care. JAMA, 297(8):831–841, 2007. doi: 10.1001/jama.297.8.831.
- [14]Patrick Lewis, Myle Ott, Jingfei Du, and Veselin Stoyanov. Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 146–157. Association for Computational Linguistics, 2020.DOI
- [15]Jessica Lin and Amir Zeldes. GUM-SAGE: A novel dataset and approach for graded entity salience prediction. In Findings of the Association for Computational Linguistics: ACL 2025, pages 438–455, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-acl.24. URL https://aclanthology.org/2025.findings-acl.24/.DOI
- [16]Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740. Association for Computational Linguistics, 2019.arxiv.org/abs/1908.08345
- [17]Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. SummaRuNNer: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017.arxiv.org/abs/1611.04230
- [19]Penn Medicine. Chart Hero. Center for Health Care Transformation and Innovation, https://chti.upenn.edu/chart-hero, 2026. Accessed September 8, 2026.
- [20]Elliot Schumacher, Daniel Rosenthal, Dhruv Naik, Varun Nair, Luladay Price, Geoffrey Tso, and Anitha Kannan. MED-OMIT: Extrinsically-focused evaluation metric for omissions in medical summarization. In Proceedings of the 4th Machine Learning for Health Symposium, volume 259 of Proceedings of Machine Learning Research, pages 897–922. PMLR, 2025.
- [21]Nigam H. Shah, Niraj Sehgal, Euan A. Ashley, and Michael A. Pfeffer. Lessons from deploying the ChatEHR system at Stanford Medicine. Nature Medicine, 2026. doi: 10.1038/s41591-026-04574-5. Comment.DOI
- [22]Christine Sinsky, Lacey Colligan, Ling Li, Mirela Prgomet, Sam Reynolds, Lindsey Goeders, Johanna Westbrook, Michael Tutty, and George Blike. Allocation of physician time in ambulatory practice: A time and motion study in 4 specialties. Annals of Internal Medicine, 165(11):753–760, 2016. doi: 10.7326/M16-0961.DOI
- [23]Peter C. Smith, Rodrigo Araya-Guerra, Caroline Bublitz, Bennett Parnes, L. Miriam Dickinson, Rebecca Van Vorst, John M. Westfall, and Wilson D. Pace. Missing clinical information during primary care visits. JAMA, 293(5):565–571, 2005. doi: 10.1001/jama.293.5.565.DOI
- [24]Thomas Sounack, Joshua Davis, Brigitte Durieux, Antoine Chaffin, Tom J. Pollard, Eric Lehman, Alistair E. W. Johnson, Matthew McDermott, Tristan Naumann, and Charlotta Lindvall. BioClinical ModernBERT: A state-of-the-art long-context encoder for biomedical and clinical NLP. arXiv preprint arXiv:2506.10896, 2025.arxiv.org/abs/2506.10896
- [25]Aaron A. Tierney, Gregg Gayre, Brian Hoberman, Britt Matter, Manuel Ballesca, Patricia Kipnis, Vincent Liu, and Kristine Lee. Ambient artificial intelligence scribes to alleviate the burden of clinical documentation. NEJM Catalyst Innovations in Care Delivery, 5(3), 2024. doi: 10.1056/CAT.23.0404.DOI
- [26]Christopher Y. K. Williams, Jaskaran Bains, Tianyu Tang, Kishan Patel, Alexa N. Lucas, Fiona Chen, Brenda Y. Miao, Atul J. Butte, and Aaron E. Kornblith. Evaluating large language models for drafting emergency department encounter summaries. PLOS Digital Health, 4(6):e0000899, 2025. doi: 10.1371/journal.pdig.0000899.DOI



