WorksComputer Science
Evaluating Pretrained Language Semantics for Structured EHR Prediction Tasks
Abstract
Structured Electronic Health Record (EHR) models typically represent clinical concepts as dataset-specific tokens, discarding their associated natural-language semantics. Consequently, relationships between concepts must largely be learned from EHR co-occurrence without directly exploiting pretrained linguistic knowledge. We investigate the contribution of pretrained language semantics to longitudinal EHR prediction through tEHR, a framework that converts clinical events into natural-language descriptions and adapts pretrained autoregressive language models through EHR-specific continued pretraining and task-specific discriminative fine-tuning. We evaluate tEHR across four MIMIC-IV prediction tasks, where Mistral-7B achieves the strongest overall performance against structured-data, code-based, and LLM-based baselines. Controlled ablations demonstrate the importance of lexical grounding, numerical values, and both adaptation stages. A capacity-controlled experiment further shows that replacing clinical descriptions with atomic code tokens substantially reduces performance despite retaining the same LLM backbone. Alternative approaches incorporating pretrained semantic embeddings offer lower computational costs but do not match full textualisation. These results highlight the importance of both semantic representation and downstream adaptation when applying pretrained language models to structured EHR prediction.
Data and Code Availability We use the publicly available MIMIC-IV dataset (Johnson et al., [14]), processed into the Medical Event Data Standard (MEDS) format (McDermott and Xu, [18]) with downstream cohorts assembled using the ACES framework (Xu et al., 2025). Code can be found at https://github.com/Ashvin-Gupta/tEHR.
Institutional Review Board (IRB) This research uses the publicly available MIMIC-IV dataset, which has been previously approved for research use and does not require additional IRB approval.
Introduction
Electronic Health Records (EHRs) provide longitudinal records of diagnoses, medications, procedures, laboratory measurements, and other clinical events. Transformer-based architectures have achieved strong performance across tasks including early pancreatic cancer detection ([23, 31]) and ICU outcome prediction ([10]), with recent work extending to generative sequence models ([15, 25, 33] (Wornow et al. 2023; Shmaktoko et al. 2025; Ramjohan et al. 2025)), state-space models ([8]), and prompt-based prediction ([3]).
Despite their architectural differences, these approaches largely rely on each clinical event being expressed as unique tokens. Consequently, models must learn representations and relationships between concepts from their EHR co-occurrence, and cannot exploit any prior semantic knowledge. These clinical concepts are recorded using a range of coding systems, including SNOMED-CT ([30]), ICD (World Health Organization), and Read Codes ([4]), with the choice of vocabulary varying across healthcare systems. Even within a single coding system, the subset of codes used can differ between sites and evolve over time as coding practices and terminologies change. Consequently, the same underlying clinical concept may be represented by multiple codes across datasets, while the overall vocabulary can contain thousands of possible concepts: Non-Type 1 Diabetes Mellitus is represented by 768 out of 360,000 active concepts in SNOMED-CT (NHS England Primary Care Domain, 2025). This heterogeneity often requires ontology mapping, code collapsing, or vocabulary refinement before modelling ([32, 17]), creating a dataset-specific vocabulary and model. Transferring a model to a new dataset therefore requires mapping clinical codes to the expected vocabulary of the model, which can lead to information loss. Furthermore, models take different approaches to representing the time intervals between events or continuous values of recordings ([9, 1, 16]). These preprocessing requirements make structured-EHR modelling difficult to reuse consistently across datasets and healthcare systems.
Replacing symbolic clinical codes with their natural-language descriptions provides a direct way to exploit the semantic knowledge encoded in pretrained language models and reduce the need for vocabulary-specific preprocessing. Recent work suggests that these representations can improve structured EHR prediction, but it remains unclear how pretrained language semantics are beneficial for longitudinal modelling and how the language model should be incorporated into the prediction pipeline. Prompt-based approaches present patient histories directly to an LLM, either without parameter updates (Zhu et al., 2024) or with supervised fine-tuning ([28]), treating prediction primarily as a language-generation problem. Other work uses pretrained language models as representation encoders. Event-level embedding approaches encode each clinical event and pass these through a separate longitudinal aggregator, typically a Transformer, for the final prediction ([13, 22]). Patient-level embedding approaches encode the full patient trajectory into a single representation that is subsequently used by a lightweight classifier, using either clinically pretrained encoders ([19]) or general-purpose embedding-specific models ([11]). These studies establish that natural-language representations can provide useful inductive bias for clinical prediction tasks, but leave unresolved how much performance comes from the pretrained semantics themselves.
To investigate these questions, we develop tEHR, a two-stage framework for adapting pretrained autoregressive LLMs to longitudinal EHR prediction. tEHR converts structured clinical events into natural-language descriptions, applies continued pretraining on unlabelled EHR trajectories, and performs task-specific discriminative fine-tuning using the LLM as a longitudinal patient encoder.
We use this framework to investigate whether pretrained lexical knowledge contributes to predictive performance beyond model capacity alone, which components of the clinical representation are important, and whether alternative approaches to incorporating pretrained semantic information can achieve comparable performance at lower computational cost.
Our contributions are:
We present tEHR, a framework for adapting pretrained autoregressive LLMs to structured EHR prediction through continued pretraining on longitudinal patient trajectories followed by task-specific discriminative fine-tuning.
Through controlled ablations, we investigate the contributions of pretrained lexical grounding, numerical and temporal information, model capacity, and the two stages of adaptation to predictive performance.
We evaluate alternative approaches to incorporating pretrained language semantics into longitudinal EHR models, comparing predictive performance and computational cost across four MIMIC-IV prediction tasks.
Methodology
Patient Trajectory Representation
A patient trajectory, , is represented as a time-ordered sequence of clinical events, , where denotes the number of clinical events for patient . Each event is represented as , where is a discrete medical code, is an optional continuous numerical value, and is the event timestamp. Static patient attributes, such as sex and year of birth, are retained at the beginning of the patient sequence.
Clinical Narrative Textualisation
The EHR dataset is first transformed into the Medical Event Data Standard (MEDS) ([2]), which provides a common event-based representation across coding systems. We then apply a deterministic textualisation function that converts each structured patient trajectory into a sequence of natural-language event descriptions:

Figure 1. Structured EHRs are converted into the MEDS event-based representation ([2]) and textualised using natural-language descriptions of clinical events, numerical values, and elapsed time between events. A pretrained large language model (LLM) is adapted to unlabelled patient trajectories using next-token prediction, with low-rank adaptation (LoRA) while keeping the backbone frozen. For downstream prediction tasks, patient trajectories are truncated at the prediction time and appended with a [PREDICT] token. The adapted LLM produces a contextual representation of this token, which is passed to a linear classification head to estimate the probability of the target outcome using binary cross-entropy loss.
where denotes the textual representation of event .
For each event , the symbolic concept is replaced by its human-readable description using official coding-system translation metadata. Numerical measurements are retained in the resulting text and rounded to one decimal place. Elapsed time between consecutive events is also included in the text stream, , using the nearest clinically interpretable unit, such as minutes, hours, days, weeks, months, or years. Conceptually, the representation remains a time-ordered event sequence, but with all events expressed through their natural-language description together with any associated numerical value or time delta, rather than as a single code token. Unlike code-based approaches, it does not require a dataset-specific clinical vocabulary, frequency-based code pruning, or specialised temporal and numerical embedding modules.
Two-Stage Training Framework
The tEHR framework adapts a causal LLM in two stages. First, the LLM is fine-tuned on the unlabelled patient trajectories through continued pretraining. The second involves discriminative fine-tuning with a prediction head on top of a specific hidden representation for the downstream task of interest. Both stages use Low-Rank Adaptation (LoRA) ([12, 26]) applied to the self-attention and linear projection layers. For a pre-trained weight matrix , LoRA represents the update as the product of two low-rank matrices and , such that the modified forward pass is: , where is the scaling constant and is the rank. During both stages of training, remains frozen while only and are updated, substantially reducing the number of trainable parameters while retaining the pretrained representations of the backbone model.
Stage 1: Domain-adaptive continued pretraining. The model is trained on the textualised training set patient trajectories using the standard next-token prediction loss, updating the LoRA parameters . The textual sequence is tokenised using the LLM’s built-in tokeniser, producing , where denotes the number of LLM tokens in the tokenised trajectory for patient . Note that differs from , since each clinical event description may be represented by multiple language-model tokens.
This stage is intended to align the pretrained model with the distribution of longitudinal EHR trajectories without downstream outcome labels.
Stage 2: Discriminative fine-tuning. For each downstream task, the adapted model receives the patient narrative truncated at the corresponding prediction time, appended with a special [PREDICT] token. The hidden representation of this token, , is input to a linear classifier layer to produce the probability with the model further fine-tuned under the Binary Cross-Entropy objective:
for patients. This stage updates the task-specific classification head and the LoRA adapters inherited from Stage 1, while the original backbone weights remain frozen. We use a linear head so that downstream performance primarily reflects the quality of the learned patient representation. The hidden state of the [PREDICT] token is used as it outperformed alternative pooling strategies such as mean pooling in preliminary experiments. Both adaptation stages are implemented with Unsloth (Daniel Han and team, [5]) and trained with early stopping on validation loss using a fixed 80:10:10 patient-level split and three random seeds.
Experiments
Dataset and Prediction Tasks
MIMIC-IV: We use MIMIC-IV version 3.1 ([14]), a large critical-care dataset containing events recorded during hospital and intensive-care admissions. We assess four binary prediction tasks: ICU mortality, ICU readmission, hospital mortality, and 30-day hospital readmission. MIMIC-IV contains clinical concepts represented primarily using ICD and NDC coding systems. In our processed dataset, this corresponds to roughly 15k unique medical codes. It provides a high-acuity evaluation setting characterised by dense event sequences and comparatively short prediction horizons.
Baselines
We group our baselines into structured code-based models and semantic representation-based prediction approaches.
Structured Code-Based Models: XGBoost is trained on bag-of-codes frequency features. Transformer Encoder following Placido et al. [23] is trained on the corresponding symbolic event sequences with an analogous pretraining and supervised fine-tuning procedure. CLMBR-T (Wornow et al., 2023), is a 141M-parameter structured-EHR Transformer pretrained on 2.57 million patients. Applying CLMBR-T to our datasets requires mapping the dataset codes to the model’s vocabulary. Using the public MIMIC-IV-to-OMOP mapping pipeline (OHDSI, 79.4% of event occurrences are mapped). We evaluate both a frozen CLMBR-T representation with a trainable linear-classifier head and a task-specific fine-tuned variant.
Semantic Representation Models: BioClinical ModernBERT is fine-tuned on the same textualised patient trajectories as tEHR (Sounack et al., [29]). We evaluate zero-shot LLM prompting of an LLM following Zhu et al. (2024) and prompt-based supervised fine-tuning of an LLM following (Shoham and Rappoport, [28]). We investigate three alternative approaches to incorporating pretrained language semantics into longitudinal EHR prediction. First, LLM-initialised Transformer, motivated by Park et al. [22], uses a code-based Transformer whose clinical-code embeddings are initialised from pretrained LLM representations of their corresponding natural-language descriptions. Second, Event LLM embeddings, using a hierarchical architecture similar to Hur et al. [13], represents individual clinical events with frozen pretrained LLM embeddings, which are passed to a trainable Transformer for longitudinal prediction. Third, LLM projection projects pretrained event-level LLM embeddings into the input embedding space of a longitudinal LLM. The latter two approaches use task-specific supervised training without the autoregressive next-token continued pretraining stage employed by tEHR. We report AUROC, AUPRC and computational cost in the main text with full implementation details provided in Appendix § 7.2.
Ablations
We conduct controlled ablations across the four MIMIC-IV prediction tasks to investigate three questions: (1) which stages of the tEHR training procedure are necessary, (2) which components of the textualised EHR representation contribute predictive information, and (3) how sensitive performance is to backbone choice and model capacity.
Two-stage training framework. We evaluate three training variants to disentangle the contributions of domain-adaptive continued pretraining and task-specific adaptation. (1) we remove Stage 1 and initialise directly from the base LLM, applying the same supervised LoRA adaptation and linear classification head used in the full model. (2) we retain Stage 1 but freeze the adapted backbone during downstream training, optimising only the linear classification head. (3) we freeze the initial unadapted base LLM and train only the classification head. Together, these variants separate the contribution of unlabelled EHR adaptation, task-specific backbone adaptation, and generic pretrained representations.
Clinical information representation. We next test which components of the textualised EHR representation contribute to predictive performance using controlled perturbations. In the hashed-concept condition, each clinical concept is replaced by a deterministic, unique, and semantically meaningless identifier, preserving concept identity and recurrence structure while removing lexical meaning. In the scrambled-concept condition, concept descriptions are deterministically permuted within event categories for the events that occurred for that patient, preserving plausible medical language and broad event type while breaking the correspondence between each concept and its true clinical meaning. In the no-time condition, explicit textualised elapsed-time intervals are removed while chronological event order is preserved. In the no-value condition, numerical measurement values are removed while retaining the identity of the corresponding laboratory or vital-sign event. Across all conditions, demographics, hospital-transfer events and event ordering are preserved, and only the targeted component of the representation is modified.
Backbone and model capacity. We evaluate the full two-stage tEHR framework using Mistral-7B-v0.3, Qwen3-8B, and Qwen3-0.6B to assess robustness across LLM families and sensitivity to model capacity. To distinguish the contribution of pretrained lexical information from model capacity, we additionally construct a capacity-controlled variant using the Mistral-7B backbone, Mistral-7B atomic codes, replacing natural-language event descriptions with atomic clinical-code tokens. The vocabulary is extended with the clinical codes used by the structured baselines, whose embeddings are newly initialised. This variant follows the same two-stage training framework as tEHR, preserving backbone capacity and the two-stage adaptation procedure while replacing pretrained lexical representations with newly initialised code embeddings.
We use a maximum context length of 16,000 language-model tokens for tEHR and 2,048 clinical events for the code-based models where supported by their input representations and context limits. To ensure matched clinical histories, we apply a shared event-level truncation: for each patient, we retain the most recent events that fit within both limits, while preserving demographic information. The resulting identical event subset is provided to each model using its corresponding representation. BioClinical ModernBERT and CLMBR-T are the exceptions due to their 8,192 and 496 context length respectively. Token expansion and context-length characteristics are detailed in Appendix § 7.5.
Results
tEHR-Mistral achieved the highest AUROC and AUPRC across all four MIMIC-IV prediction tasks (Table 1). Improvements were particularly pronounced for ICU mortality, where tEHR achieved an AUPRC of 0.424 compared with 0.406 for fine-tuned CLMBR-T and 0.370 for the code-based Transformer. In contrast, zero-shot prompting performed substantially worse, with AUROC ranging from 0.528 to 0.627.
| Model | ICU Mortality AUROC | ICU Mortality AUPRC | ICU Readm. AUROC | ICU Readm. AUPRC | Hospital Mortality AUROC | Hospital Mortality AUPRC | Hospital Readm. AUROC | Hospital Readm. AUPRC |
| XGBoost | 0.817 | 0.306 | 0.713 | 0.333 | 0.868 | 0.235 | 0.742 | 0.483 |
| Transformer Encoder | 0.866 | 0.370 | 0.773 | 0.456 | 0.911 | 0.281 | 0.765 | 0.531 |
| CLMBR-T (frozen) | 0.793 | 0.235 | 0.724 | 0.340 | 0.881 | 0.204 | 0.708 | 0.409 |
| CLMBR-T (fine-tuned) | 0.861 | 0.406 | 0.762 | 0.428 | 0.915 | 0.328 | 0.762 | 0.515 |
| BioClinical ModernBERT | 0.824 | 0.300 | 0.739 | 0.359 | 0.860 | 0.197 | 0.766 | 0.532 |
| Zero-shot LLM (Mistral-7B) | 0.559 | 0.109 | 0.539 | 0.147 | 0.627 | 0.048 | 0.528 | 0.232 |
| Prompt SFT (Mistral-7B) | 0.811 | 0.221 | 0.668 | 0.219 | 0.786 | 0.073 | 0.724 | 0.437 |
| Mistral-7B (atomic codes) | 0.772 | 0.221 | 0.703 | 0.323 | 0.871 | 0.226 | 0.766 | 0.511 |
| LLM-initialised Transformer | 0.843 | 0.351 | 0.778 | 0.467 | 0.897 | 0.275 | 0.766 | 0.532 |
| Event LLM embeddings | 0.739 | 0.194 | 0.547 | 0.166 | 0.876 | 0.215 | 0.739 | 0.480 |
| LLM projection | 0.737 | 0.210 | 0.651 | 0.273 | 0.817 | 0.156 | 0.737 | 0.483 |
| tEHR-Qwen3-0.6B | 0.822 | 0.289 | 0.738 | 0.414 | 0.903 | 0.273 | 0.759 | 0.522 |
| tEHR-Qwen3-8B | 0.870 | 0.391 | 0.780 | 0.480 | 0.927 | 0.324 | 0.764 | 0.527 |
| tEHR-Mistral-7B | 0.887 | 0.424 | 0.794 | 0.498 | 0.932 | 0.333 | 0.775 | 0.556 |
Table 1. AUROC and AUPRC across four MIMIC-IV prediction tasks, averaged over three seeds. Best results are shown in bold. Standard deviations are reported in the appendix.
The training ablations demonstrate the contribution of both adaptation stages (Figure 3). Removing EHR-specific continued pretraining reduced AUROC across all four tasks, while freezing the backbone during downstream training also reduced performance. Using frozen representations from the unadapted base

Figure 3. Change in AUROC relative to the full tEHR-Mistral-7B model across four downstream tasks on MIMIC-IV for the ablation experiments.
LLM produced the weakest results among the training variants, demonstrating the importance of adapting the pretrained model to longitudinal clinical data and downstream prediction.
The clinical representation ablations further demonstrate the importance of lexical grounding. Replacing clinical descriptions with hashed identifiers or scrambling their correspondence to the underlying events substantially reduced performance across all tasks. Removing numerical values also consistently reduced AUROC, whereas removing explicit elapsed-time descriptions had little effect.
Performance varied across pretrained backbones, with Mistral-7B outperforming Qwen3-8B and Qwen3-0.6B. However, the capacity-controlled Mistral-7B experiment provides further evidence that model size alone does not explain tEHR’s performance. Replacing natural-language descriptions with atomic clinical-code tokens reduced ICU mortality AUROC from 0.887 to 0.772, despite retaining the same backbone and two-stage training framework. Similar reductions were observed across the remaining tasks.
All alternative approaches to incorporating pretrained semantic representations performed below full tEHR. The LLM-initialised Transformer was the strongest of these alternatives, achieving an ICU mortality AUROC of 0.843 compared with 0.887 for tEHR-Mistral. Event-level LLM embeddings and projected event representations yielded further reductions, suggesting that compact semantic representations did not
recover the performance of full textualisation under the evaluated setups.
We additionally evaluated calibration across the four MIMIC-IV tasks (Figure 2). tEHR-Mistral achieved the lowest Brier score across all tasks, although this metric reflects both discrimination and calibration. Calibration was particularly strong for ICU and hospital readmission, with slopes of 1.01 and 1.03 and ECE values of 0.006 and 0.005, respectively. Other models achieved better calibration on some metrics, particularly for the mortality tasks (Appendix § 7.4).

Figure 2. Calibration of predicted risks across four MIMIC-IV prediction tasks. Points show mean predicted probability and observed outcome frequency within 10 equal-count bins; connecting lines are included for visualisation only. The grey diagonal indicates perfect calibration. Axes use task-specific limits with equal ranges on both dimensions.
These performance differences were accompanied by computational trade-offs (Table 2). tEHR-Mistral required 85.11 GPU-hours for EHR-specific continued pretraining and 26.09 GPU-hours for ICU mortality fine-tuning, compared with 96.11 and 15.65 GPU-hours for the code-based Transformer. However, single-patient inference was substantially slower (301 versus 8.8 ms), while the LLM-initialised Transformer achieved comparable latency to the code-based Transformer. Thus, more compact representations offered lower inference costs, but did not match tEHR’s predictive performance.
| Model | EHR PT GPU-h | Task FT GPU-h | Latency ms |
| XGBoost | – | – | 0.21 |
| Transformer Encoder | 96.11 | 15.65 | 8.8 |
| CLMBR-T (frozen) | Ext. | 0.14 | 3.9 |
| CLMBR-T (fine-tuned) | Ext. | 1.52 | 4.0 |
| BioClinical-ModernBERT | Ext. | 3.09 | 487 |
| Zero-shot LLM | – | – | 373 |
| Prompt SFT | – | 28.68 | 385 |
| Mistral-7B (atomic codes) | 5.45 | 4.67 | 87.7 |
| LLM-initialised Transformer | 96.10 | 15.65 | 8.75 |
| Event LLM embeddings | – | 16.03 | 8.86 |
| LLM projection | – | 4.73 | 77.7 |
| tEHR-Qwen3-0.6B | 19.66 | 6.62 | 68.0 |
| tEHR-Qwen3-8B | 90.70 | 28.04 | 328 |
| tEHR-Mistral-7B | 85.11 | 26.09 | 301 |
Table 2. Computational cost on one NVIDIA H200. EHR PT denotes EHR-specific pretraining, and Task FT denotes supervised task training (ICU mortality), both in GPU-hours. Latency is measured per patient at batch size 1. External pretraining is excluded.
Discussion
We present tEHR, a framework for adapting pretrained autoregressive LLMs to EHR prediction tasks. tEHR can in principle be applied to different autoregressive LLMs and relies on clinical events being represented with their natural-language descriptions rather than as symbolic tokens, allowing the model to reuse semantic information learnt during large-scale pretraining. tEHR requires simple data preprocessing, avoids the construction of a model-specific vocabulary and can still be applied to large models through parameter-efficient fine-tuning with LoRA. However, the other LLM-based experiments show that simply introducing pretrained semantic embeddings does not recover the performance of full textualisation: zero-shot prompting performed poorly, while alternative approaches incorporating pretrained semantic embeddings also failed to match full tEHR. This suggests that access to pretrained language knowledge alone is insufficient, and that how this knowledge is incorporated into longitudinal modelling matters.
The ablations provide a possible explanation for the success of tEHR. Continued pretraining adapts the model to the distribution and sequential structure of the EHR data, while task-specific fine-tuning further shapes the representation for the downstream prediction task. The frozen base-model ablation assesses the predictive utility of generic pretrained representations without EHR-specific adaptation. The improvements obtained through continued pretraining and task-specific fine-tuning suggest that both stages contribute to learning effective longitudinal patient representations. The lexical ablations further demonstrate the importance of correctly grounding clinical events in natural-language. In hashed-concept, the model is unable to identify the semantic meaning of the events occurring and performs worse. Similarly, the scrambled-concept ablation indicates that it is not sufficient for the input to contain plausible clinical language but that the correspondence between a clinical event and its surrounding context matters. Together, these perturbations suggest that the model benefits from the correspondence between clinical concepts and their pretrained lexical representations. This is further supported by the capacity-controlled Mistral-7B experiment, where replacing natural-language descriptions with atomic clinical-code tokens substantially reduced performance despite retaining the same backbone and two-stage training framework. Together, these findings suggest that pretrained lexical knowledge contributes to tEHR’s performance beyond model capacity alone.
A possible explanation for the drop in performance when the clinical values are removed is that the model can no longer directly distinguish normal from abnormal measurement values. This is not the same case for time, consistent with previous work [1] and suggesting that chronological ordering may therefore capture much of the required temporal information. Performance also varied across pretrained backbones. Differences between Mistral-7B and Qwen3-8B, despite similar parameter counts, suggest that pretraining differences may influence downstream performance, although these experiments do not isolate their source. The weaker performance of Qwen3-0.6B further indicates sensitivity to model capacity. These performance improvements come with a computational trade-off as tEHR was an expensive method for training and slower for inference, albeit with one patient prediction still comfortably under one second at 0.3s. More compact semantic representation strategies offered lower training costs, albeit with reduced predictive performance.
The CLMBR-T experiment illustrates a practical limitation of structured-EHR foundation models with fixed vocabularies. Although CLMBR-T performs strongly within its intended representation setting, applying it to our MIMIC-IV cohorts required mapping source concepts into the vocabulary expected by the model. Only 79.4% of event occurrences were successfully mapped, which may partly explain the remaining performance gap. The mapping issue is of high concern and as such, the tEHR experiments were also conducted using a private UK primary care dataset for 12-month pancreatic cancer prediction. Primary-care is a substantially different setting from MIMIC-IV as is longitudinal data recorded across many practices and therefore results in a much larger and more heterogeneous clinical vocabulary; over 241k unique medical codes. A common preprocessing step is to map codes to higher-level clinical concepts, which is manually intensive and can lead to incorrect medical timelines if there are mistakes. tEHR showed the strongest performance, showing that predictive models can be developed without extensive code simplification. A mapping of CLMBR-T’s vocabulary to the private dataset vocabulary was not found meaning the model could not be run, highlighting the issue with many of the current code-based models. In contrast, tEHR operates on natural-language concept descriptions, avoiding explicit alignment to a pretrained symbolic vocabulary. An interesting future experiment would be to evaluate the cross-dataset transfer of a tEHR model which would complement recent work by Ding et al. [7] on unifying vocabularies across datasets.
Several limitations remain. First, evaluation is restricted to four prediction tasks within MIMIC-IV and a private UK dataset which could not be shared. Further evaluation across independent healthcare systems, coding vocabularies, and prediction tasks is necessary to establish generalisability. Second, textualisation expands clinical events into multiple subword tokens, increasing computational cost and requiring truncation of longer patient histories. Future work could investigate cross-dataset transfer and the interpretability of learned representations.
Acknowledgements
This work is supported by UK Research and Innovation (UKRI AI Centre for Doctoral Training in Digital Healthcare grant number EP/Y030974/1), NIHR Imperial Biomedical Research Centre and the NIHR Invention for Innovation grant “Artificial Intelligence to support cancer early diagnosis in general practice" (NIHR207533).
Appendix
Dataset information
MIMIC-IV (version 3.1) contains longitudinal hospital and ICU records from Beth Israel Deaconess Medical Center. Each hospital admission or ICU stay constitutes a separate prediction instance, meaning that a patient may contribute multiple instances if they have multiple admissions or stays.
We evaluate four binary prediction tasks. For ICU mortality, prediction is made at ICU admission, with the outcome defined as death during the corresponding ICU stay. For ICU readmission, prediction is made at ICU discharge, with the outcome indicating a subsequent ICU admission. Similarly, hospital mortality is predicted at hospital admission, with the outcome defined as death during that hospitalisation, while 30-day hospital readmission is predicted at hospital discharge, with the outcome indicating readmission within 30 days.
For each task, only clinical events available up to the corresponding prediction time are included. Train, validation, and test partitions are constructed at the patient level using unique subject IDs, ensuring that all admissions and stays belonging to the same patient remain within a single partition.
| Metric / Task | Train Count | Train % | Validation Count | Validation % | Test Count | Test % |
| Unique Patients | 291,702 | 80.0% | 36,463 | 10.0% | 36,462 | 10.0% |
| ICU Mortality | ||||||
| Case (Positive) | 5,881 | 7.8% | 705 | 7.7% | 715 | 7.7% |
| Control (Negative) | 69,169 | 92.2% | 8,508 | 92.3% | 8,628 | 92.3% |
| ICU Readmission | ||||||
| Case (Positive) | 10,745 | 14.2% | 1,345 | 14.5% | 1,350 | 14.3% |
| Control (Negative) | 64,978 | 85.8% | 7,947 | 85.5% | 8,079 | 85.7% |
| Hospital Mortality | ||||||
| Case (Positive) | 8,061 | 2.9% | 1,014 | 2.9% | 988 | 2.8% |
| Control (Negative) | 271,056 | 97.1% | 34,271 | 97.1% | 33,840 | 97.2% |
| Hospital Readmission | ||||||
| Case (Positive) | 86,859 | 20.4% | 11,074 | 20.6% | 11,480 | 21.2% |
| Control (Negative) | 339,559 | 79.6% | 42,659 | 79.4% | 42,676 | 78.8% |
Table 3. MIMIC-IV Population and Task-Specific Distribution
Baselines and Ablations
XGBoost We implement XGBoost using bag-of-codes frequency features constructed from all discrete medical concepts observed in each patient’s available history. Each feature corresponds to a clinical code and records the number of times that code occurs before the task-specific prediction time. This provides a non-sequential structured-data baseline that retains event frequency but discards temporal ordering, continuous values, and relationships between events.
Transformer Encoder We implement a code-based Transformer encoder, similar to Placido et al. [23], with 12 encoder layers, 8 attention heads, and a maximum context length of 2,048 events. The model operates directly on symbolic medical codes rather than natural-language descriptions, with each code becoming a unique token. Temporal information and continuous measurements are discretised into additional learned representations following the code-based preprocessing pipeline.
The encoder follows an analogous two-stage training procedure to tEHR. In Stage 1, it is pretrained on trajectories from the training split using masked language modelling with a masking probability of 15%; of selected tokens, 80% are replaced by the mask token, 10% by a random token, and 10% are left unchanged, following BERT (Devlin et al., [6]). In Stage 2, a linear classification head is added to the patient representation and the encoder is fine-tuned jointly for each downstream prediction task using binary cross-entropy. We train both stages for 50 epochs with early stopping on validation loss.
CLMBR-T We evaluate CLMBR-T (Wornow et al., 2023), a 141M-parameter structured-EHR Transformer pretrained on 2.57 million patients, on the four MIMIC-IV tasks. Because CLMBR-T operates over a predefined structured vocabulary, MIMIC-IV concepts were mapped using the public MIMIC-IV-to-OMOP pipeline (OHDSI). This mapped 79.4% of observed event occurrences, with unmapped events omitted from the input sequence. We evaluate two variants. In the frozen setting, CLMBR-T is used as a fixed patient representation and only a logistic-regression classification head is trained. In the fine-tuned setting, the pretrained model is additionally adapted to each downstream task before classification. Because the mapping is incomplete, these results should be interpreted as a transfer benchmark under partial vocabulary alignment rather than as a comparison under fully matched input representations.
BioClinical-ModernBERT We use BioClinical-ModernBERT-base (150M parameters) Sounack et al. [29] as a pretrained medical text encoder and fine-tune all model parameters for each downstream prediction task. BioClinical-ModernBERT receives the same textualised EHR representation as tEHR. Its maximum context length is 8,192 tokens; trajectories exceeding this limit are truncated from the left to retain the most recent clinical history, while demographic information is preserved at the beginning of the sequence. A linear binary classification head is applied to the patient-level representation and all parameters are optimised jointly with the encoder using binary cross-entropy.
Zero-shot LLM To evaluate whether pretrained LLM knowledge alone is sufficient for clinical prediction, we use the instruction-tuned version of the corresponding LLM without any parameter updates. The model is provided with the textualised patient trajectory together with a task-specific instruction requesting a binary prediction, encoded as 0 for control and 1 for case. Predictions are obtained from the model’s output logits for the two candidate labels, which are normalised to produce a continuous case score for AUROC and AUPRC evaluation. No examples or demonstrations are included in the prompt. The patient trajectory is truncated only when it exceeds the context window of the underlying LLM.
Prompt-based Supervised Fine-tuning The prompt-based SFT baseline uses the same textualised input representation and binary prediction prompt as the zero-shot setting, but adapts the LLM to the labelled downstream cohort using LoRA. Unlike tEHR, no continued pretraining on unlabelled EHR trajectories is performed, and no discriminative classification head is introduced. Instead, the model is trained to generate the correct binary outcome token from the patient trajectory. This evaluates generative supervised fine-tuning as an alternative to tEHR’s discriminative adaptation of the patient-level representation.
The prompts for both are as follows:
You are an expert in clinical prediction from longitudinal electronic health record (EHR) data.
Your task is to predict whether the patient will {task} based only on the EHR information provided below.
Demographic information appears at the beginning, followed by clinical events in chronological order. Events
↔ are separated by semicolons. Numerical measurements are reported to one decimal place with their units,
↔ and elapsed time between events is included.
Consider the patient's longitudinal history, including relevant clinical events, measurements, and trends over
↔ time.
Answer with exactly one digit on a single line:
1 = yes
0 = no
Do not output any additional text, explanation, probability, or symbols.
Patient's medical history:
{ehr_text}
Answer:
Mistral-7B with Atomic Clinical-Code Tokens To assess whether tEHR’s performance can be explained by model capacity rather than its use of natural-language event descriptions, we construct a capacity-controlled variant using the same Mistral-7B backbone. Each unique clinical concept is represented by an atomic token, rather than its natural-language description. We extend the Mistral vocabulary with the clinical-code vocabulary used by the code-based Transformer baseline, initialising new embeddings for these codes.
The model follows the same two-stage training procedure as tEHR. During Stage 1, the LoRA adapters and newly introduced code embeddings are updated through autoregressive next-token continued pretraining on patient trajectories. During Stage 2, a linear classification head is introduced, and the LoRA adapters, code embeddings, and classification head are optimised jointly using binary cross-entropy. This preserves the backbone and adaptation procedure while replacing natural-language descriptions with learned symbolic code representations.
LLM-initialised Transformer Motivated by the use of pretrained language-model embeddings for medical concepts in Park et al. [22], we evaluate whether pretrained language semantics can be incorporated into a conventional code-based EHR Transformer through its input embedding layer. This variant uses the same architecture and structured event representation as the Transformer Encoder baseline, but initialises clinical-code embeddings using pretrained LLM representations of their corresponding natural-language descriptions. Each description is independently encoded by the LLM to obtain a semantic representation used to initialise the corresponding code embedding, which remains trainable during subsequent adaptation.
Following initialisation, the model uses the same two-stage training procedure as the Transformer Encoder baseline: masked language modelling on unlabelled patient trajectories followed by task-specific supervised fine-tuning using binary cross-entropy. The remaining architecture and training procedure are unchanged, allowing pretrained semantic information to be introduced without processing full natural-language trajectories through the longitudinal model.
Event LLM Embeddings To evaluate the utility of pretrained language representations without adapting the LLM itself, we encode each textualised clinical event independently using the frozen base LLM to obtain event-level embeddings. These embeddings are then passed to a trainable six-layer, eight-head Transformer encoder that models the longitudinal patient sequence and produces a patient-level representation for binary classification. The LLM parameters remain fixed throughout, while the Transformer aggregator and classification head are trained on the labelled cohort. This provides an event-level representation baseline analogous to Hur et al. [13].
LLM Projection We evaluate an alternative approach in which pretrained event-level representations are supplied directly to the longitudinal LLM rather than processing clinical descriptions as natural-language token sequences. Each clinical event is independently encoded using a frozen pretrained LLM to obtain an event-level embedding. These embeddings are projected into the input embedding space of Mistral-7B, allowing each clinical event to occupy a single position in the longitudinal sequence rather than multiple subword tokens.
The projected event sequence is processed by Mistral-7B to produce a patient-level representation for binary classification. Unlike tEHR, this variant does not undergo autoregressive next-token continued pretraining on longitudinal EHR trajectories. Instead, it is adapted directly to each downstream prediction task using a supervised classification objective. This separates event-level semantic encoding from longitudinal aggregation while reducing the sequence expansion associated with natural-language textualisation.
Hyperparameter search
The same patient-level train/validation/test splits are used as for tEHR and the different hyperparameters for each of the models can be seen in Table 4.
| Model | Hyperparameter | Search Space / Values |
| XGBoost | Learning Rate () | |
| Max Depth | ||
| Subsample Ratio | ||
| Gamma | log-uniform | |
| BioClinical-ModernBERT | Learning Rate | |
| Weight Decay | ||
| Dropout Rate | ||
| Batch Size | ||
| Transformer Encoder | Hidden Dimension () | |
| Number of Layers | ||
| Attention Heads | ||
| Main experiments (All LoRA models) | LoRA Rank () | |
| LoRA Alpha () | ||
| Learning Rate | ||
| Weight Decay | ||
| Dropout |
Table 4. Hyperparameter Search Space for Baseline Models.
Calibration
We additionally assess the calibration of predicted risks on the held-out MIMIC-IV test sets. Table 5 reports Brier scores and calibration slopes for tEHR and the strongest competing baselines. tEHR achieves the lowest Brier score across all four tasks, although this metric reflects both discrimination and calibration. Its calibration slopes are closest to the ideal value of 1 on three of the four tasks, with particularly good calibration for ICU and hospital readmission.
| Task | tEHR Mistral-7B | Transformer Encoder | CLMBR-T fine-tuned | LLM-initialised Transformer | XGBoost |
| Brier score | |||||
| ICU mortality | 0.055 | 0.067 | 0.063 | 0.060 | 0.061 |
| ICU readmission | 0.091 | 0.102 | 0.107 | 0.096 | 0.106 |
| Hospital mortality | 0.022 | 0.025 | 0.024 | 0.023 | 0.024 |
| Hospital readmission | 0.131 | 0.146 | 0.181 | 0.137 | 0.142 |
| Calibration slope | |||||
| ICU mortality | 0.90 | 0.66 | 0.49 | 0.70 | 1.12 |
| ICU readmission | 1.01 | 1.25 | 0.57 | 0.70 | 1.32 |
| Hospital mortality | 0.85 | 1.07 | 0.66 | 1.01 | 1.06 |
| Hospital readmission | 1.03 | 0.78 | 0.74 | 0.84 | 1.16 |
Table 5. Brier scores and calibration slopes across four MIMIC-IV prediction tasks. Lower Brier scores are better, while calibration slopes closer to 1 indicate better calibration slope. Best values are shown in bold.
Tokenisation and context length
Textualising structured EHR events expands each clinical event into multiple language-model subword tokens. On MIMIC-IV training trajectories, we measured a median expansion of 7.41 Mistral tokens per clinical event (Table 6). To prevent differences in available clinical history from confounding model comparisons, we apply a shared event-level truncation procedure. For each patient, we retain the most recent clinical events, up to a maximum of 2,048 events, subject to the corresponding textualised sequence fitting within tEHR’s 16,000-token context window. The same retained clinical events are then supplied to the code-based and semantic representation models using their respective input representations. Demographic information is preserved at the beginning of each trajectory. The exception is BioClinical-ModernBERT, whose 8,192-token context window may require additional truncation of the matched textualised trajectory, and CLMBR-T, whose 496 token context window limits the history of events. For this model, demographic information and the most recent events are retained within its context limit.
| 25th pct. | Median | 75th pct. | |
| BPE / event | 7.04 | 7.41 | 7.81 |
Table 6. Mistral BPE tokens per clinical event on MIMIC-IV training trajectories, measured after textualising the most recent 2,048 events with temporal information and numerical values retained. Percentiles are computed within each task and averaged equally across the four tasks.
MIMIC-IV Full results
| Model / experiment | ICU Mortality AUROC | ICU Mortality AUPRC | ICU Readmission AUROC | ICU Readmission AUPRC | Hospital Mortality AUROC | Hospital Mortality AUPRC | Hospital Readmission AUROC | Hospital Readmission AUPRC |
| Baselines | ||||||||
| XGBoost | ||||||||
| Transformer Encoder | ||||||||
| CLMBR-T (frozen) | ||||||||
| CLMBR-T (fine-tuned) | ||||||||
| BioClinical-ModernBERT | ||||||||
| Zero-shot LLM | ||||||||
| Prompt SFT – Mistral-7B-v0.3 | ||||||||
| Ablations: alternative semantic injection | ||||||||
| LLM-initialised Transformer | ||||||||
| Event LLM embeddings | ||||||||
| LLM projection | ||||||||
| Ablations: training framework | ||||||||
| No Stage 1 | ||||||||
| Frozen stage 1 model | ||||||||
| Frozen base LLM | ||||||||
| Mistral-7B atomic codes | ||||||||
| Ablations: clinical information representation | ||||||||
| Hashed concepts | ||||||||
| Scrambled concepts | ||||||||
| No elapsed-time text | ||||||||
| No numerical values | ||||||||
| tEHR | ||||||||
| tEHR-Qwen3-0.6B | ||||||||
| tEHR-Qwen3-8B | ||||||||
| tEHR-Mistral-7B-v0.3 |
Table 7. MIMIC-IV baseline, ablation, and tEHR results. Point estimates are test-set AUROC / AUPRC presented as mean ± standard deviation across 3 seeds. Ablations use tEHR-Mistral-7B-v0.3 unless otherwise specified.
Additional Metrics
We report precision, recall, F1, balanced accuracy, accuracy, and Matthews correlation coefficient (MCC) for all MIMIC-IV tasks, as indicated in the table below. Because these metrics require a decision threshold, the threshold for each model and task was selected exclusively on the validation set by maximising F1 score, and was then fixed before evaluation on the test set. This avoids tuning the operating point on the test data. Across the four tasks, tEHR-Mistral-7B achieves the highest F1 and MCC in all four settings, while other models occasionally attain higher recall or accuracy depending on the selected operating point.
| Model | Thr. | Prec. | Rec. | F1 | Bal. acc. | Acc. | MCC |
| XGBoost | 0.180 | 0.331 | 0.399 | 0.362 | 0.666 | 0.892 | 0.305 |
| BioClinical-ModernBERT | 0.222 | 0.323 | 0.411 | 0.362 | 0.670 | 0.889 | 0.304 |
| Transformer Encoder | 0.156 | 0.363 | 0.484 | 0.415 | 0.707 | 0.895 | 0.363 |
| Zero-shot LLM | 0.150 | 0.168 | 0.179 | 0.174 | 0.553 | 0.870 | 0.103 |
| Prompt SFT | 0.400 | 0.223 | 0.660 | 0.333 | 0.735 | 0.798 | 0.298 |
| CLMBR-T (frozen) | 0.722 | 0.273 | 0.392 | 0.322 | 0.653 | 0.874 | 0.259 |
| CLMBR-T (fine-tuned) | 0.097 | 0.337 | 0.580 | 0.426 | 0.743 | 0.880 | 0.382 |
| tEHR-Mistral-7B | 0.245 | 0.422 | 0.485 | 0.451 | 0.715 | 0.910 | 0.403 |
Table 8. ICU mortality. Thresholds maximise validation F1 and are then fixed and applied to the test set. Best test-set value for each metric is shown in bold.
| Model | Thr. | Prec. | Rec. | F1 | Bal. acc. | Acc. | MCC |
| XGBoost | 0.165 | 0.285 | 0.435 | 0.344 | 0.633 | 0.777 | 0.224 |
| BioClinical-ModernBERT | 0.154 | 0.305 | 0.521 | 0.385 | 0.668 | 0.776 | 0.273 |
| Transformer Encoder | 0.149 | 0.403 | 0.483 | 0.440 | 0.686 | 0.835 | 0.346 |
| Zero-shot LLM | 0.240 | 0.151 | 0.727 | 0.250 | 0.546 | 0.414 | 0.066 |
| Prompt SFT | 0.480 | 0.222 | 0.548 | 0.316 | 0.625 | 0.681 | 0.181 |
| CLMBR-T (frozen) | 0.594 | 0.319 | 0.453 | 0.375 | 0.652 | 0.797 | 0.263 |
| CLMBR-T (fine-tuned) | 0.284 | 0.381 | 0.454 | 0.414 | 0.670 | 0.827 | 0.316 |
| tEHR-Mistral-7B | 0.217 | 0.430 | 0.514 | 0.468 | 0.704 | 0.843 | 0.379 |
Table 9. ICU readmission. Thresholds maximise validation F1 and are then fixed and applied to the test set. Best test-set value for each metric is shown in bold.
| Model | Thr. | Prec. | Rec. | F1 | Bal. acc. | Acc. | MCC |
| XGBoost | 0.185 | 0.340 | 0.292 | 0.314 | 0.638 | 0.964 | 0.297 |
| BioClinical-ModernBERT | 0.093 | 0.228 | 0.385 | 0.286 | 0.673 | 0.946 | 0.270 |
| Transformer Encoder | 0.134 | 0.331 | 0.351 | 0.341 | 0.665 | 0.962 | 0.321 |
| Zero-shot LLM | 0.230 | 0.049 | 0.381 | 0.087 | 0.584 | 0.776 | 0.068 |
| Prompt SFT | 0.234 | 0.088 | 0.555 | 0.151 | 0.694 | 0.825 | 0.167 |
| CLMBR-T (frozen) | 0.858 | 0.237 | 0.387 | 0.294 | 0.675 | 0.948 | 0.277 |
| CLMBR-T (fine-tuned) | 0.068 | 0.338 | 0.444 | 0.384 | 0.709 | 0.960 | 0.367 |
| tEHR-Mistral-7B | 0.202 | 0.363 | 0.447 | 0.401 | 0.712 | 0.962 | 0.384 |
Table 10. Hospital mortality. Thresholds maximise validation F1 and are then fixed and applied to the test set. Best test-set value for each metric is shown in bold.
| Model | Thr. | Prec. | Rec. | F1 | Bal. acc. | Acc. | MCC |
| XGBoost | 0.203 | 0.370 | 0.638 | 0.468 | 0.673 | 0.693 | 0.293 |
| BioClinical-ModernBERT | 0.232 | 0.418 | 0.607 | 0.495 | 0.690 | 0.737 | 0.336 |
| Transformer Encoder | 0.155 | 0.403 | 0.638 | 0.494 | 0.692 | 0.723 | 0.332 |
| Zero-shot LLM | 0.240 | 0.235 | 0.772 | 0.361 | 0.549 | 0.420 | 0.087 |
| Prompt SFT | 0.321 | 0.326 | 0.736 | 0.452 | 0.663 | 0.621 | 0.267 |
| CLMBR-T (frozen) | 0.540 | 0.361 | 0.583 | 0.446 | 0.653 | 0.693 | 0.264 |
| CLMBR-T (fine-tuned) | 0.521 | 0.416 | 0.613 | 0.496 | 0.691 | 0.736 | 0.337 |
| tEHR-Mistral-7B | 0.266 | 0.457 | 0.594 | 0.517 | 0.702 | 0.764 | 0.370 |
Table 11. Hospital readmission. Thresholds maximise validation F1 and are then fixed and applied to the test set. Best test-set value for each metric is shown in bold.
References
- [1]Rafi Al Attrach, Rajna Fani, David Restrepo, Yugang Jia, and Peter Schüffler. Rethinking Tokenization for Clinical Time Series: When Less is More. 12 2025.
- [2]Bert Arnrich, Edward Choi, Jason Alan Fries, Matthew B.A. McDermott, Jungwoo Oh, Tom Pollard, Nigam Shah, Ethan Steinberg, Michael Wornow, and Robin van de Water. Medical Event Data Standard ({MEDS}): Facilitating Machine Learning for Health. ICLR 2024 Workshop on Learning from Time Series For Health, 2024.
- [3]Payal Chandak, Gregory Kondas, Liat Antwarg Friedman, Isaac Kohane, and Matthew McDermott. Evr yQuery: Zero-Shot Clinical Prediction via Task-Conditioned Pretraining over Electronic Health Records. 5 2026.
- [4]J Chisholm. The Read clinical classification. BMJ, 300(6732):1092–1092, 4 1990. ISSN 0959-8138. doi: 10.1136/bmj.300.6732.1092.DOI
- [5]Michael Han Daniel Han and Unsloth team. Unsloth, 2023. URL https://github.com/unslothai/unsloth.
- [6]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT. In Proceedings of the 2019 Conference of the North, pages 4171–4186, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423.DOI
- [7]Tong Ding, Caiwei Tian, Dongmin Bang, Ming Y. Lu, Sophia J. Wagner, Andrew Zhang, Rowland Pettit, Long Phi Le, and Faisal Mahmood. Discrete Neural Interlingua for Ontology-Agnostic EHR Modeling. 10 2026.arxiv.org/abs/2610.27047
- [8]Adibvafa Fallahpour, Mahshid Alinoori, Wenqian Ye, Xu Cao, Arash Afkanpour, and Amrit Krishnan. EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Records. 11 2024.arxiv.org/abs/2405.14567
- [9]Lin Lawrence Guo, Santiago Eduardo Arciniegas, Joseph Jihyung Lee, Adam Paul Yan, George Tomlinson, Jason Fries, and Lillian Sung. Tokenization Tradeoffs in Structured EHR Foundation Models. 3 2026.arxiv.org/abs/2603.15644
- [10]Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. 8 2019. doi: 10.1038/s41597-019-0103-9.DOI
- [11]Stefan Hegselmann, Georg von Arnim, Tillmann Rheude, Noel Kronenberg, David Sontag, Gerhard Hindricks, Roland Eils, and Benjamin Wild. Large language models are powerful electronic health record coders. npj Digital Medicine, 9 (1):530, 7 2026. ISSN 2398-6352. doi: 10.1038/s41746-026-02915-9.
- [12]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. 10 2021.arxiv.org/abs/2106.09685
- [13]Kyunghoon Hur, Jungwoo Oh, Junu Kim, Jiyoun Kim, Min Jae Lee, Eunbyeol Cho, Seong-Eun Moon, Young-Hak Kim, and Edward Choi. UniHPF : Universal Healthcare Predictive Framework with Zero Domain Knowledge. 9 2024.
- [14]Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Brian Gow, Benjamin Moody, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV. PhysioNet, 10 2024. doi: 10.13026/kpb9-mt58. URL https://doi.org/10.13026/kpb9-mt58.
- [15]Zeljko Kraljevic, Dan Bean, Anthony Shek, Rebecca Bendayan, Harry Hemingway, Joshua Au Yeung, Alexander Deng, Alfie Baston, Jack Ross, Esther Idowu, James T Teo, and Richard J Dobson. Foresight – Generative Pretrained Transformer (GPT) for Modelling of Patient Timelines using EHRs. 1 2023.arxiv.org/abs/2212.08072
- [16]Inhyeok Lee, Luke Solo, Michael C. Burkhart, Bashar Ramadan, William F. Parker, and Brett K. Beaulieu-Jones. Representation Before Training: A Fixed-Budget Benchmark for Generative Medical Event Models. 4 2026.
- [17]Nikolai Makarov and Mikhail Lipkovich. A transformer-based model for next disease prediction using electronic health records. The European Physical Journal Special Topics, 1 2025. ISSN 1951-6355. doi: 10.1140/epjs/s11734-024-01447-1.DOI
- [18]Matthew McDermott and Justin Xu. MIMIC-IV MEDS: An ETL pipeline to extract MIMIC-IV data into the MEDS format, 4 2025. URL https://github.com/Medical-Event-Data-Standard/MIMIC_IV_MEDS.DOI
- [19]Anna Munoz-Farre, Harry Rose, and Sera Aylin Cakiroglu. sEHR-CE: Language modelling of structured EHR data for efficient and generalizable patient cohort expansion. 11 2022.
- [22]Jiheum Park, Jason Patterson, Jose M. Aci-tores Cortina, Tian Gu, Chin Hur, and Nicholas Tatonetti. Enhancing EHR-based pancreatic cancer prediction with LLM-derived embeddings. npj Digital Medicine, 8(1):465, 7 2025. ISSN 2398-6352. doi: 10.1038/s41746-025-01869-8.DOI
- [23]Davide Placido, Bo Yuan, Jessica X. Hjaltelin, Chun-lei Zheng, Amalie D. Haue, Piotr J. Chmura, Chen Yuan, Jihye Kim, Renato Umeton, Gregory Antell, Alexander Chowdhury, Alexandra Franz, Lauren Brais, Elizabeth Andrews, Debora S. Marks, Aviv Regev, Siamak Ayandeh, Mary T. Brophy, Nhan V. Do, Peter Kraft, Brian M. Wolpin, Michael H. Rosenthal, Nathanael R. Fillmore, Søren Brunak, and Chris Sander. A deep learning algorithm to predict risk of pancreatic cancer from disease trajectories. Nature Medicine, 29(5):1113–1122, 5 2023. ISSN 1078-8956. doi: 10.1038/s41591-023-02332-5.DOI
- [25]Pawel Renc, Yugang Jia, Anthony E. Samir, Jaroslaw Was, Quanzheng Li, David W. Bates, and Arkadiusz Sitek. Zero shot health trajectory prediction using transformer. npj Digital Medicine, 7(1):256, 9 2024. ISSN 2398-6352. doi: 10.1038/s41746-024-01235-0.DOI
- [26]John Schulman and Thinking Machines Lab. LoRA Without Regret. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20250929.DOI
- [28]Ofir Ben Shoham and Nadav Rappoport. CPLM: Clinical Prediction with Large Language Models. 5 2024.
- [29]Thomas Sounack, Joshua Davis, Brigitte Durieux, Antoine Chaffin, Tom J. Pollard, Eric Lehman, Alistair E. W. Johnson, Matthew McDermott, Tristan Naumann, and Charlotta Lindvall. BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP. 6 2025.arxiv.org/abs/2506.10896
- [30]M Q Stearns, C Price, K A Spackman, and A Y Wang. SNOMED clinical terms: overview of the development process and project status. Proceedings. AMIA Symposium, pages 662–6, 2001. ISSN 1531-605X.pubmed.ncbi.nlm.nih.gov/11825268
- [31]Ethan Steinberg, Jason Fries, Yizhe Xu, and Nigam Shah. MOTOR: A Time-To-Event Foundation Model For Structured Medical Records. 12 2023.arxiv.org/abs/2301.03150
- [32]Lan Wang, Yonghua Yin, Ben Glampson, Robert Peach, Mauricio Barahona, Brendan C. Delaney, and Erik K. Mayer. Transformer-based deep learning model for the diagnosis of suspected lung cancer in primary care based on electronic health record data. eBioMedicine, 110:105442, 12 2024. ISSN 23523964. doi: 10.1016/j.ebiom.2024.105442.DOI
- [33]Shane Waxler, Paul Blazek, Davis White, Daniel Sneider, Kevin Chung, Mani Nagarathnam, Patrick Williams, Hank Voeller, Karen Wong, Matthew Swanhorst, Sheng Zhang, Naoto Usuyama, Cliff Wong, Tristan Naumann, Hoifung Poon, Andrew Loza, Daniella Meeker, Seth Hain, and Rahul Shah. Generative Medical Event Models Improve with Scale. 11 2025.arxiv.org/abs/2508.12104