betaFeedback

Reading v1View latest version

WorksComputer Science

Evaluating Pretrained Language Semantics for Structured EHR Prediction Tasks

Abstract

Structured Electronic Health Records (EHR) models typically treat each clinical concept as a dataset-specific token, discarding the natural-language semantics associated with the event. As a result, relationships between concepts must largely be learned from EHR co-occurrence, without directly exploiting any prior semantic knowledge. We investigate how pretrained causal language models can be adapted for longitudinal EHR prediction by developing tEHR. Here, we represent events with their natural language description and use this text stream to apply EHR-adaptive continued pretraining, followed by task-specific discriminative fine-tuning using LoRA, utilising the LLM as a longitudinal patient-encoder. We evaluate this approach across four prediction tasks on the MIMIC-IV dataset and pancreatic cancer prediction on the CPRD dataset at 1 to 12 month horizons, with tEHR using the Mistral-7B-v0.3 backbone achieving the strongest overall performance. Ablations show that performance declines when clinical descriptions are removed or misaligned, when numerical values are omitted, and when either stage of adaptation is removed, while explicit elapsed-time descriptions provide little additional benefit. Analysis of the learned patient representations further reveals clinically coherent structure associated with distinct patterns of disease risk and clinical history.

Data and Code Availability We use the publicly available MIMIC-IV dataset [14] and private Clinical Practice Research Datalink (CPRD) dataset [40], processed into the Medical Event Data Standard (MEDS) format [18] with downstream cohorts assembled using the ACES framework [43]. For confidentiality constraints, the CPRD dataset cannot be shared. Code can be found at https://github.com/Ashvin-Gupta/tEHR.

Institutional Review Board (IRB) This research uses the publicly available MIMIC-IV dataset, which has been previously approved for research use and does not require additional IRB approval. A research protocol, reference 23_002840, was submitted and approved by the Clinical Practice Research Datalink (CPRD) Independent Scientific Advisory Committee before the study was conducted.

Introduction

Electronic Health Records (EHRs) provide longitudinal records of diagnoses, medications, procedures, laboratory measurements, and other clinical events. Transformer-based architectures have achieved strong performance across tasks including early pancreatic cancer detection (Placido et al., 2023; Steinberg et al., 2023) and ICU outcome prediction [10], with recent work extending to generative sequence models [15, 42, 29, 32, 28, 39], state-space models (Hafner et al., 2024), and prompt-based prediction [3].

Despite their architectural differences, these approaches largely rely on each clinical event being expressed as unique tokens. Consequently, models must learn representations and relationships between concepts from their EHR co-occurrence, and cannot exploit any prior semantic knowledge. These clinical concepts are recorded using a range of coding systems, including SNOMED-CT [35], ICD (World Health Organization), and Read Codes [4], with the choice of vocabulary varying across healthcare systems. Even within a single coding system, the subset of codes used can differ between sites and evolve over time as coding practices and terminologies change. Consequently, the same underlying clinical concept may be represented by multiple codes across datasets, while the overall vocabulary can contain thousands of possible concepts: Non-Type 1 Diabetes Mellitus is represented by 768 out of > 360, 000 active concepts in SNOMED-CT (NHS England Primary Care Domain, 2025). This heterogeneity often requires ontology mapping, code collapsing, or vocabulary refinement before modelling [38, 17], creating a dataset-specific vocabulary and model. Transferring a model to a new dataset therefore requires mapping clinical codes to the expected vocabulary of the model, which can lead to information loss. Furthermore, models take different approaches to representing the time intervals between events or continuous values of recordings [9, 1, 16]. These preprocessing requirements make structured-EHR modelling difficult to reuse consistently across datasets and healthcare systems.

Replacing symbolic clinical codes with their natural-language descriptions provides a direct way to exploit the semantic knowledge encoded in pretrained language models and avoid these preprocessing issues. Recent work suggests that these representations can improve structured EHR prediction, but it remains unclear how pretrained language semantics are beneficial for longitudinal modelling and how the language model should be incorporated into the prediction pipeline. Prompt-based approaches present patient histories directly to an LLM, either without parameter updates [44] or with supervised fine-tuning [33], treating prediction primarily as a language-generation problem. Other work uses pretrained language models as representation encoders. Event-level embedding approaches encode each clinical event and pass these through a separate longitudinal aggregator, typically a Transformer, for the final prediction [13, 25]. Patient-level embedding approaches encode the full patient trajectory into a single representation that is subsequently used by a lightweight classifier, using either clinically pretrained encoders [21] or general-purpose embedding-specific models [11]. These studies establish that natural-language representations can provide useful inductive bias for clinical prediction tasks, but leave unresolved how much performance comes from the pretrained semantics themselves.

To investigate these questions, we develop tEHR, a two-stage framework applicable to causal LLMs. tEHR converts structured clinical events into a natural-language event stream, applies continued pretraining on unlabelled EHR trajectories, and then performs task-specific discriminative fine-tuning using the LLM as the longitudinal patient-encoder. We use this framework to study which aspects of pretrained language representations and downstream adaptation are responsible for predictive performance in structured EHR tasks.

Our contributions are:

  1. We present tEHR, a framework for adapting pretrained autoregressive LLMs to structured EHR prediction through continued pretraining on longitudinal patient trajectories followed by task-specific discriminative fine-tuning.

  1. We investigate how pretrained language semantics contribute to EHR prediction through controlled ablations of lexical grounding, model capacity, and adaptation, alongside comparisons with alternative semantic representations and their computational trade-offs.

  1. We demonstrate strong predictive performance across four MIMIC-IV tasks and pancreatic cancer prediction in CPRD, showing that tEHR can be independently applied across distinct healthcare settings and clinical coding systems.

Methodology

Patient Trajectory Representation

A patient trajectory, X(p)X^{(p)}, is represented as a time-ordered sequence of clinical events, X(p)=(e1(p),e2(p),…,eLp(p))X^{(p)} = (e_1^{(p)}, e_2^{(p)}, \ldots, e_{L_p}^{(p)}), where LpL_p denotes the number of clinical events for patient pp. Each event is represented as ei(p)=(ci(p),vi(p),ti(p))e_i^{(p)} = (c_i^{(p)}, v_i^{(p)}, t_i^{(p)}), where ci(p)∈Vc_i^{(p)} \in\mathcal{V} is a discrete medical code, vi(p)∈Rv_i^{(p)} \in\mathbb{R} is an optional continuous numerical value, and ti(p)t_i^{(p)} is the event timestamp. Static patient attributes, such as sex and year of birth, are retained at the beginning of the patient sequence.

Clinical Narrative Textualisation

All datasets are first transformed into the Medical Event Data Standard (MEDS) [2], which provides a common event-based representation across coding systems. We then apply a deterministic textualisation function that converts each structured patient trajectory into a sequence of natural-language event descriptions:

Structured EHRs are converted into the MEDS event-based representation and textualised for pretrained language model adaptation and downstream prediction

Figure 1. Structured EHRs are converted into the MEDS event-based representation ([2]) and textualised using natural-language descriptions of clinical events, numerical values, and elapsed time between events. A pretrained large language model (LLM) is adapted to unlabelled patient trajectories using next-token prediction, with low-rank adaptation (LoRA) while keeping the backbone frozen. For downstream prediction tasks, patient trajectories are truncated at the prediction time and appended with a [PREDICT] token. The adapted LLM produces a contextual representation of this token, which is passed to a linear classification head to estimate the probability of the target outcome using binary cross-entropy loss.

X(p)⟶X~(p)=(x~1(p),…,x~Lp(p)),(1)X^{(p)} \longrightarrow\widetilde{X}^{(p)} = \left(\widetilde{x}^{(p)}_1,\ldots,\widetilde{x}^{(p)}_{L_p}\right), \tag*{(1)}

where x~i(p)\widetilde{x}^{(p)}_i denotes the textual representation of event ei(p)e^{(p)}_i. For each event ei(p)e^{(p)}_i, the symbolic concept ci(p)c^{(p)}_i is replaced by its human-readable description using official coding-system translation metadata. Numerical measurements are retained in the resulting text and rounded to one decimal place. Elapsed time between consecutive events is also included in the text stream, Δti(p)=ti(p)−ti−1(p)\Delta t^{(p)}_i = t^{(p)}_i - t^{(p)}_{i-1}, using the nearest clinically interpretable unit, such as minutes, hours, days, weeks, months, or years. Conceptually, the representation remains a time-ordered event sequence, but with all events expressed through their natural-language description together with any associated numerical value or time delta, rather than as a single code token. Unlike code-based approaches, it does not require a dataset-specific clinical vocabulary, frequency-based code pruning, or specialised temporal and numerical embedding modules.

Two-Stage Training Framework

The tEHR framework adapts a causal LLM in two stages. First, the LLM is fine-tuned on the unlabelled patient trajectories through continued pretraining. The second involves discriminative fine-tuning with a prediction head on top of a specific hidden representation for the downstream task of interest. Both stages use Low-Rank Adaptation (LoRA) [12, 31] applied to the self-attention and linear projection layers. For a pre-trained weight matrix W0∈Rd×kW_0 \in\mathbb{R}^{d \times k}, LoRA represents the update ΔW\Delta W as the product of two low-rank matrices B∈Rd×rB \in\mathbb{R}^{d \times r} and A∈Rr×kA \in\mathbb{R}^{r \times k}, such that the modified forward pass is:

h=(W0+ΔW)x=(W0+αrBA)x,h = (W_0 + \Delta W)x = \left(W_0 + \frac{\alpha}{r}BA\right)x,

where α\alpha is the scaling constant and rr is the rank. During both stages of training, W0W_0 remains frozen while only AA and BB are updated, substantially reducing the number of trainable parameters while retaining the pretrained representations of the backbone model.

Stage 1: Domain-adaptive continued pretraining. The model is trained on the textualised training set patient trajectories using the standard next-token prediction loss, updating the LoRA parameters θ\theta. The textual sequence X~(p)\tilde{X}^{(p)} is tokenised using the LLM’s built-in tokeniser, producing Z(p)=z1(p),…,zTp(p)Z^{(p)} = z_1^{(p)}, \ldots, z_{T_p}^{(p)} where TpT_p denotes the number of LLM tokens in the tokenised trajectory for patient pp. Note that TpT_p differs from LpL_p, since each clinical event description may be represented by multiple language-model tokens.

LCLM(θ)=−1Tp∑t=1Tplog⁡Pθ(zt(p)∣z<t(p)).(2)\mathcal{L}_{\mathrm{CLM}}(\theta) = -\frac{1}{T_p}\sum_{t=1}^{T_p}\log P_\theta\left(z_t^{(p)} \mid z_{<t}^{(p)}\right). \tag*{(2)}

This stage is intended to align the pretrained model with the distribution of longitudinal EHR trajectories without downstream outcome labels.

Stage 2: Discriminative fine-tuning. For each downstream task, the adapted model receives the patient narrative truncated at the corresponding prediction time, appended with a special [PREDICT] token. The hidden representation of this token, hPREDICT(p)∈Rdh_{\mathrm{PREDICT}}^{(p)} \in\mathbb{R}^{d}, is input to a linear classifier layer to produce the probability y^p=σ(w⊤hPREDICT(p)+b)\hat{y}_p = \sigma(w^\top h_{\mathrm{PREDICT}}^{(p)} + b) with the model further fine-tuned under the Binary Cross-Entropy objective:

LBCE=−1N[∑p=1Nyplog⁡y^p+(1−yp)log⁡(1−y^p)](3)\mathcal{L}_{\mathrm{BCE}} = -\frac{1}{N}\left[\sum_{p=1}^{N} y_p \log\hat{y}_p + (1-y_p)\log(1-\hat{y}_p)\right] \tag*{(3)}

for NN patients. This stage updates the task-specific classification head and the LoRA adapters inherited from Stage 1, while the original backbone weights remain frozen. We use a linear head so that downstream performance primarily reflects the quality of the learned patient representation. The hidden state of the [PREDICT] token is used as it outperformed alternative pooling strategies such as mean pooling in preliminary experiments. Both adaptation stages are implemented with Unsloth (Daniel Han and team, [5]) and trained with early stopping on validation loss using a fixed 80:10:10 patient-level split and three random seeds. Performance is reported using AUROC and AUPRC.

Experiments

Datasets and Prediction Tasks

MIMIC-IV: We use MIMIC-IV version 3.1 ([14]), a large critical-care dataset containing events recorded during hospital and intensive-care admissions. We assess four binary prediction tasks: ICU mortality, ICU readmission, hospital mortality, and 30-day hospital readmission. MIMIC-IV contains clinical concepts represented primarily using ICD and NDC coding systems. In our processed dataset, this corresponds to roughly 15k unique medical codes. It provides a high-acuity evaluation setting characterised by dense event sequences and comparatively short prediction horizons.

CPRD: The Clinical Practice Research Datalink (CPRD) ([40]) is a private UK primary-care database containing over 16 million patients, accessible only via a secure environment. Its clinical events are represented using Read Codes. Due to the heterogeneous nature of primary care coding practices in the UK, this dataset contains over 241k unique medical codes. We construct a task-specific cohort for pancreatic cancer prediction. Eligible pancreatic cancer cases are matched to up to five controls on sex, birth year within ±3\pm3 years, and general-practice region, resulting in 73,732 patients with up to 10 years of preceding clinical history (Appendix § 6.2). This design focuses evaluation on clinically comparable cases and controls rather than the highly imbalanced full CPRD population. To assess prediction at different lead times, patient histories are right-censored at 1, 3, 6, and 12 months before the index date.

Baselines

We group our baselines into structured code-based models and text-based prediction approaches.

Structured Code-Based Models: XGBoost which is trained on bag-of-codes frequency features. Transformer Encoder following Pla ́cido et al. (2023), is trained on the corresponding subsymbolic event sequences with an analogous pretraining and supervised fine-tuning procedure. CLMBR-T ([42]), is a 141M-parameter structured-EHR Transformer pretrained on 2.57 million patients. Applying CLMBR-T to our datasets requires mapping the dataset codes to the model’s vocabulary. Using the public MIMIC-IV-to-OMOP mapping pipeline ((OHDSI)), 79.4% of event occurrences are mapped.

We did not identify an equivalent mapping for CPRD Read Codes, so CLMBR-T is evaluated only on MIMIC-IV. We evaluate both a frozen CLMBR-T representation with a trainable linear-classifier head and a task-specific fine-tuned variant.

Text-based Models: BioClinical ModernBERT is fine-tuned on the same textualised patient trajectories as tEHR ([34]). We evaluate zero-shot LLM prompting of an LLM following Zhu et al. [44] and prompt-based supervised fine-tuning of an LLM following (Shoham and Rappoport, [33]) (only performed on MIMIC-IV due to compute restrictions in the CPRD secure environment).

Ablations

All ablations are conducted on the MIMIC-IV tasks because computational restrictions in the CPRD secure environment prevented an equivalent sweep on CPRD. We investigate four questions: (1) which stages of the tEHR training procedure are necessary, (2) which components of the textualised EHR representation contribute predictive information, (3) how sensitive performance is to backbone choice and model capacity, and (4) whether pretrained language semantics can be incorporated through alternative event representations.

Two-stage training framework. We evaluate three training variants to disentangle the contributions of domain-adaptive continued pretraining and task-specific adaptation. (1) we remove Stage 1 and initialise directly from the base LLM, applying the same supervised LoRA adaptation and linear classification head used in the full model. (2) we retain Stage 1 but freeze the adapted backbone during downstream training, optimising only the linear classification head. (3) we freeze the initial unadapted base LLM and train only the classification head. Together, these variants separate the contribution of unlabelled EHR adaptation, task-specific backbone adaptation, and generic pretrained representations.

Clinical information representation. We next test which components of the textualised EHR representation contribute to predictive performance using controlled perturbations. In the hashed-concept condition, each clinical concept is replaced by a deterministic, unique, and semantically meaningless identifier, preserving concept identity and recurrence structure while removing lexical meaning. In the scrambled-concept condition, concept descriptions are deterministically permuted within event categories for the events that occurred for that patient, preserving plausible medical language and broad event type while breaking the correspondence between each concept and its true clinical meaning. In the no-time condition, explicit textualised elapsed-time intervals are removed while chronological event order is preserved. In the no-value condition, numerical measurement values are removed while retaining the identity of the corresponding laboratory or vital-sign event. Across all conditions, demographics, hospital-transfer events and event ordering are preserved, and only the targeted component of the representation is modified.

Backbone and model capacity. We evaluate the full two-stage tEHR framework using Mistral-7B-v0.3, Qwen3-8B, and Qwen3-0.6B to assess robustness across LLM families and sensitivity to model capacity. To distinguish the contribution of pretrained lexical information from model capacity, we additionally construct a capacity-controlled variant using the Mistral-7B backbone, Mistral-7B atomic codes, replacing natural-language event descriptions with atomic clinical-code tokens. The vocabulary is extended with the clinical codes used by the structured baselines, whose embeddings are newly initialised. This variant follows the same two-stage training framework as tEHR, preserving backbone capacity and the two-stage adaptation procedure while replacing pretrained lexical representations with newly initialised code embeddings.

Alternative semantic representations. We investigate three alternative approaches to incorporating pretrained language semantics into longitudinal EHR prediction. First, LLM-initialised Transformer uses a code-based Transformer whose clinical-code embeddings are initialised from pretrained LLM representations of their corresponding natural-language descriptions, Hur et al. [13]. Second, Event LLM embeddings, following Park et al. [25], represents individual clinical events using frozen LLM embeddings, which are passed to a trainable Transformer for longitudinal prediction. Third, LLM projection projects pretrained event-level LLM embeddings into the input embedding space of a longitudinal LLM. The latter two approaches use task-specific supervised training without the autoregressive next-token continued pretraining stage employed by tEHR. We report AUROC and computational cost in the main text, with

AUPRC and full implementation details provided in Appendix § 6.3 and Appendix § 6.9 respectively.

Representation Analysis and Clinical Characterisation

To investigate the patient representations learned by tEHR, we extract the 4,096-dimensional [PREDICT] embeddings from the test set of the 1-month pancreatic cancer model. We focus on this setting because it provides a strong predictive signal and pancreatic cancer has well-characterised clinical risk factors that facilitate interpretation of the learned representations. Embeddings are first ℓ2\ell_2-normalised and reduced to 10 dimensions using UMAP ([19]). We use UMAP rather than PCA to preserve potentially nonlinear local neighbourhood structure in the learned representations, rather than prioritising linear explained variance and allow us to use density-based clustering. HDBSCAN is applied in this 10-dimensional space to identify dense groups of patients without specifying the number of clusters in advance. A separate two-dimensional UMAP projection is used solely for visualisation and does not affect cluster assignment. Clusters are characterised by their size, pancreatic cancer prevalence, categorical-event enrichment, and distributions of continuous laboratory measurements and hyperparameters are selected using silhouette scores on the validation set and fixed before test-set analysis (Appendix § 6.4). For categorical events, enrichment is defined as the prevalence of an event within a cluster relative to its prevalence in the full population. This analysis is intended to assess whether the task-adapted representation organises patients into clinically coherent phenotypes beyond the binary case-control label.

Results

tEHR-Mistral achieved the strongest overall performance across all four MIMIC-IV tasks (Table 1) and all pancreatic cancer prediction horizons in CPRD (Figure 2). On MIMIC-IV, the strongest competing baselines were the code-based Transformer and fine-tuned CLMBR-T, while zero-shot prompting showed near chance-level discrimination on several tasks. On CPRD, tEHR also outperformed all baselines, with XGBoost being the strongest competitor. Performance declined across all models as the prediction horizon increased, reflecting the greater difficulty of earlier risk prediction.

ModelICU Mort.ICU Read.Hosp. Mort.Hosp. Read.
XGBoost0.8170.7130.8680.742
Transformer Encoder0.8660.7730.9110.765
CLMBR-T (frozen)0.7930.7240.8810.708
CLMBR-T (fine-tuned)0.8610.7620.9150.762
BioClinical ModernBERT0.8240.7390.8600.766
Zero-shot LLM (Mistral-7B)0.5590.5390.6270.528
Prompt SFT (Mistral-7B)0.8110.6680.7860.724
Mistral-7B (atomic codes)0.7720.7030.8710.766
LLM-initialised Transformer0.8430.7780.8970.766
Event LLM embeddings0.7390.5470.8760.739
LLM projection0.7370.6510.8170.737
tEHR-Qwen3-0.6B0.8220.7380.9030.759
tEHR-Qwen3-8B0.8700.7800.9270.764
tEHR-Mistral-7B0.8870.7940.9320.775

Table 1. AUROC across ICU Mortality, ICU Readmission, Hospital Mortality and Hospital Readmission MIMIC-IV prediction tasks, averaged over three seeds. Best results are shown in bold. Standard deviations and AUPRC are reported in the appendix.

AUROC for pancreatic cancer prediction at horizons of 1, 3, 6 and 12 months preceding diagnosis for tEHR, XGBoost, Transformer Encoder, BioClinical-ModernBERT and Zero-shot LLM

Figure 2. AUROC for pancreatic cancer prediction at horizons of 1, 3, 6 and 12 months preceding diagnosis for tEHR, XGBoost, Transformer Encoder, BioClinical-ModernBERT and Zero-shot LLM.

The ablations demonstrate the importance of both adaptation stages (Figure 3). Removing continued pretraining reduced AUROC across all four MIMIC-IV tasks, while restricting downstream adaptation to only the linear head led to further deterioration. Perturbing lexical grounding through hashed or scrambled concept representations also substantially reduced performance. Numerical values contributed consistently, whereas explicit elapsed-time descriptions provided little additional benefit.

Bar chart showing changes in AUROC for ablation experiments

Figure 3. Change in AUROC relative to the full tEHR-Mistral-7B model across four downstream tasks on MIMIC-IV for the ablation experiments.

Performance varied across LLM backbones, with Mistral-7B outperforming both Qwen3-8B and Qwen3-0.6B. The capacity-controlled Mistral-7B variant using atomic clinical-code tokens performed substantially below full tEHR despite retaining the same backbone and two-stage adaptation framework. Together, these findings indicate that performance depends on both backbone choice and clinical event representation, rather than model capacity alone.

All three alternative approaches to incorporating pretrained language semantics performed below full tEHR across the MIMIC-IV tasks. The LLM-initialised Transformer was the strongest alternative, offering a more computationally efficient approach at the cost of predictive performance.

Table 2 shows the computational cost for the experiments on MIMIC-IV. Although tEHR required greater computational resources, its training cost remained of the same order of magnitude as the code-based Transformer (85.11 versus 96.11 GPU-hours for EHR-pretraining, and 26.09 versus 15.65 GPU-hours for ICU mortality fine-tuning). Inference was substantially slower (301 versus 8.8 ms per patient), although remaining below one second per patient on a single NVIDIA H200.

ModelEHR PT GPU-hTask FT GPU-hLatency ms
XGBoost––0.21
Transformer Encoder96.1115.658.8
CLMBR-T (frozen)Ext.0.143.9
CLMBR-T (fine-tuned)Ext.1.524.0
BioClinical-ModernBERTExt.3.09487
Zero-shot LLM––373
Prompt SFT–28.68385
Mistral-7B (atomic codes)5.454.6787.7
LLM-initialised Transformer96.1015.658.75
Event LLM embeddings–16.038.86
LLM projection–4.7377.7
tEHR-Qwen3-0.6B19.666.6268.0
tEHR-Qwen3-8B90.7028.04328
tEHR-Mistral-7B85.1126.09301

Table 2. Computational cost on one NVIDIA H200. EHR PT denotes EHR-specific pretraining, and Task FT denotes supervised task training (ICU mortality), both in GPU-hours. Latency is measured per patient at batch size 1. External pretraining is excluded.

UMAP projection followed by HDBSCAN clustering of the [PREDICT] patient embeddings (Figure 4) identified four clinically distinct regions with markedly different case prevalence (Appendix § 6.5, Figure 5). Cluster 1 was strongly enriched for pancreatic cancer cases and exhibited the clearest disease-associated clinical profile. It showed substantial enrichment of steatorrhoea, pancreatitis, jaundice, and abdominal mass, alongside higher distributions of HbA1c, CA125, and ferritin. These features are consistent with clinical manifestations and biomarkers associated with pancreatic malignancy [37, 30, 20]. The remaining clusters captured distinct clinical structure rather than simply reflecting case–control status. Cluster 0 was predominantly composed of controls and showed comparatively little enrichment of the selected pancreatic-cancer-associated features. Cluster 2 contained a mixture of cases and controls and was distinguished by enrichment of lupus erythematosus and non-alcoholic fatty liver disease, together with higher inflammatory-marker distributions, including ESR, consistent with systemic inflammation or hepatic dysfunction [22]. Cluster 3 was also predominantly control-associated and showed a distinct reproductive and gynaecological profile, with particularly strong enrichment of male infertility and dysmenorrhoea and more modest enrichment of leiomyoma [36].

Two-dimensional UMAP visualisation of pancreatic-cancer patient embeddings

Figure 4. Two-dimensional UMAP visualisation of pancreatic-cancer patient embeddings. Colour denotes case-control status and opacity denotes predicted cancer probability.

Clinical phenotypes of interest and their fold enrichment for each cluster

Figure 5. Clinical phenotypes of interest and their fold enrichment for each cluster. Cluster 0: baseline, cluster 1: pancreatic malignancy, cluster 2: inflammatory phenotype, cluster 3: gynaecological and urological conditions.

Discussion

We present tEHR, a framework for adapting pretrained autoregressive LLMs to EHR prediction tasks. tEHR can in principle be applied to different autoregressive LLMs and relies on clinical events being represented with their natural-language descriptions rather than as symbolic tokens, allowing the model to reuse semantic information learnt during large-scale pretraining. tEHR requires simple data preprocessing, avoids the construction of a model-specific vocabulary and can still be applied to large models through parameter-efficient fine-tuning with LoRA. However, the other LLM-based experiments show that simply introducing pretrained semantic embeddings does not recover the performance of full textualisation. Neither zero-shot, description-initialised code embeddings nor event-level LLM representations matched tEHR, suggesting that the manner in which semantic information is incorporated into longitudinal modelling is also important.

The ablations provide a possible explanation for the success of tEHR. Continued pretraining adapts the model to the distribution and sequential structure of the EHR data, while task-specific fine-tuning further shapes the representation for the downstream prediction task. The frozen base-model ablation assesses the predictive utility of generic pretrained representations without EHR-specific adaptation. The improvements obtained through continued pretraining and task-specific fine-tuning suggest that both stages contribute to learning effective longitudinal patient representations. The lexical ablations further demonstrate the importance of correctly grounding clinical events in natural-language. In hashed-concept, the model is unable to identify the semantic meaning of the events occurring and performs worse. Similarly, the scrambled-concept ablation indicates that it is not sufficient for the input to contain plausible clinical language but that the correspondence between a clinical event and its surrounding context matters. Together, these perturbations suggest that the model benefits from the correspondence between clinical concepts and their pretrained lexical representations. This is further supported by the capacity-controlled Mistral-7B experiment, where replacing natural-language descriptions with atomic clinical-code tokens substantially reduced performance despite retaining the same backbone and two-stage training framework. Together, these findings suggest that pretrained lexical knowledge contributes to tEHR’s performance beyond model capacity alone.

A possible explanation for the drop in performance when the clinical values are removed is that the model can no longer directly distinguish normal from abnormal measurement values. This is not the same case for time, consistent with previous work (Attrach et al., [1]) and suggesting that chronological ordering may therefore capture much of the required temporal information. Performance also varied across pretrained backbones. Differences between Mistral-7B and Qwen3-8B, despite similar parameter counts, suggest that pretraining differences may influence downstream performance, although these experiments do not isolate their source. The weaker performance of Qwen3-0.6B further indicates sensitivity to model capacity. These performance improvements come with a computational trade-off as tEHR was an expensive method for both training and inference. More compact semantic representation strategies offered lower training costs, albeit with reduced predictive performance.

The CLMBR-T experiment illustrates a practical limitation of structured-EHR foundation models with fixed vocabularies. Although CLMBR-T performs strongly within its intended representation setting, applying it to our MIMIC-IV cohorts required mapping source concepts into the vocabulary expected by the model. Only 79.4% of event occurrences were successfully mapped, which may partly explain the remaining performance gap. We did not identify an equivalent mapping suitable for CPRD Read Codes, preventing a corresponding evaluation there. In contrast, tEHR operates on natural-language concept descriptions, avoiding explicit alignment to a pretrained symbolic vocabulary. An interesting future experiment would be to evaluate the cross-dataset transfer of a tEHR model which would complement recent work by Ding et al. [7] on unifying vocabularies across datasets.

The CPRD experiments further evaluate the tEHR method on primary-care, which is a substantially different setting from MIMIC-IV. CPRD contains longitudinal primary-care data recorded across many practices and therefore results in a much larger and more heterogeneous clinical vocabulary. A common preprocessing step is to map codes to higher-level clinical concepts, which is manually intensive and can lead to incorrect medical timelines if there are mistakes. The strong performance of tEHR in this setting suggests that predictive models can be developed without extensive code simplification. The strong performance of XGBoost also shows that model complexity alone is not sufficient for strong prediction in this setting. The latent-space analysis provides additional evidence that the learned representation contains clinically meaningful structure. The pancreatic-cancer-enriched region contains features associated with the disease, while other regions capture distinct inflammatory, metabolic, reproductive, and urological patterns. However, clustering does not establish causal mechanisms, and the identified groups may be sensitive to the choice of dimensionality-reduction and clustering parameters.

Several limitations remain. First, CPRD evaluation was restricted by the approved study protocol and computational constraints to pancreatic cancer prediction, with the full ablation suite therefore performed only on MIMIC-IV. Evaluation across additional diseases and prediction tasks would provide a stronger test of generality. Second, textualisation expands clinical events into multiple language-model tokens, increasing sequence length and computational cost. We therefore limit the LLM context window to 16K tokens and retain the most recent clinical history for longer trajectories (Appendix § 6.6). Finally, further work could investigate cross-dataset transfer, retention of pretrained knowledge, and the interpretability of learned patient representations.

Appendix

Acknowledgements

This work is supported by UK Research and Innovation (UKRI AI Centre for Doctoral Training in Digital Healthcare grant number EP/Y030974/1), NIHR Imperial Biomedical Research Centre and the NIHR Invention for Innovation grant “Artificial Intelligence to support cancer early diagnosis in general practice" (NIHR207533).

Dataset information

The Clinical Practice Research Datalink (CPRD) Aurum is a UK-based longitudinal primary care database derived from routinely collected electronic health records in general practice. CPRD contains detailed patient-level information including diagnoses, prescriptions, laboratory results, referrals, and demographic variables, with linkage options to additional datasets such as hospital admissions and mortality records. The database is broadly representative of the UK population.

CPRD pancreatic cancer cohort construction. Pancreatic cancer cases were identified from CPRD and assigned an index date corresponding to their recorded date of diagnosis. For each case, we sought up to five controls matched on sex, birth year within ±3\pm3 years, and general-practice region. Controls were required to have no recorded pancreatic cancer diagnosis throughout their available follow-up, and patients who subsequently developed pancreatic cancer were not eligible to serve as controls. Each control was used at most once. For controls, the index date was defined as two years before their final recorded clinical event to reduce the likelihood of including patients with an undiagnosed pancreatic cancer near the end of follow-up. Consequently, matching was performed on birth year rather than age at index, and differences in age at index between cases and controls are expected because their index dates are defined differently. Matching was performed before the cohort was divided into training, validation, and test sets. Up to five eligible controls were selected for each case; cases were retained when fewer than five matching controls were available. Due to case-control matching, AUPRC values should not be interpreted as population screening precision.

FeatureTrainValidationTest
CaseControlCaseControlCaseControl
Total Patients (NN)10725482461336601413486063
Gender
Male5397 (50.3%)23750 (49.2%)659 (49.3%)2975 (49.5%)658 (48.8%)3051 (50.3%)
Female5328 (49.7%)24496 (50.8%)677 (50.7%)3039 (50.5%)690 (51.2%)3012 (49.7%)
Age at Index (Years)69.0 [61.0–77.0]72.0 [62.0–79.0]70.0 [60.0–78.0]72.0 [63.0–79.0]69.0 [61.0–78.0]72.0 [63.0–79.0]

Table 3. CPRD Patient Demographics and Clinical Event Statistics by Dataset Split. Values are reported as Count (%) or Median [IQR].

MIMIC-IV (version 3.1) is a dataset sourced from the Beth Israel Deaconess Medical Center, containing data from ICU admissions. In our tasks, ICU mortality is evaluated using ICU admission as the appropriate final time point, ICU readmission is evaluated with ICU discharge, hospital mortality with hospital admission and hospital readmission with hospital discharge. Splitting is by unique subject ID, all admissions/stays belonging to a patient remain in the same partition, but one patient may contribute multiple datapoints depending on how many stays they have.

Metric / TaskTrain CountTrain %Validation CountValidation %Test CountTest %
Unique Patients291,70280.0%36,46310.0%36,46210.0%
ICU Mortality
Case (Positive)5,8817.8%7057.7%7157.7%
Control (Negative)69,16992.2%8,50892.3%8,62892.3%
ICU Readmission
Case (Positive)10,74514.2%1,34514.5%1,35014.3%
Control (Negative)64,97885.8%7,94785.5%8,07985.7%
Hospital Mortality
Case (Positive)8,0612.9%1,0142.9%9882.8%
Control (Negative)271,05697.1%34,27197.1%33,84097.2%
Hospital Readmission
Case (Positive)86,85920.4%11,07420.6%11,48021.2%
Control (Negative)339,55979.6%42,65979.4%42,67678.8%

Table 4. MIMIC-IV Population and Task-Specific Distribution

Baselines and Ablations

XGBoost We implement XGBoost using bag-of-codes frequency features constructed from all discrete medical concepts observed in each patient’s available history. Each feature corresponds to a clinical code and records the number of times that code occurs before the task-specific prediction time. This provides a non-sequential structured-data baseline that retains event frequency but discards temporal ordering, continuous values, and relationships between events.

Transformer Encoder We implement a code-based Transformer encoder, similar to Placido et al. (2023), with 12 encoder layers, 8 attention heads, and a maximum context length of 2,048 events. The model operates directly on symbolic medical codes rather than natural-language descriptions, with each code becoming a unique token. Temporal information and continuous measurements are discretised into additional learned representations following the code-based preprocessing pipeline.

The encoder follows an analogous two-stage training procedure to tEHR. In Stage 1, it is pretrained on trajectories from the training split using masked language modelling with a masking probability of 15%; of selected tokens, 80% are replaced by the mask token, 10% by a random token, and 10% are left unchanged, following BERT (Devlin et al. [6]). In Stage 2, a linear classification head is added to the patient representation and the encoder is fine-tuned jointly for each downstream prediction task using binary cross-entropy. We train both stages for 50 epochs with early stopping on validation loss.

CLMBR-T We evaluate CLMBR-T (Wornow et al. [42]), a 141M-parameter structured-EHR Transformer pretrained on 2.57 million patients, on the four MIMIC-IV tasks. Because CLMBR-T operates over a predefined structured vocabulary, MIMIC-IV concepts were mapped using the public MIMIC-IV-to-OMOP pipeline (OHDSI). This mapped 79.4% of observed event occurrences, with unmapped events omitted from the input sequence. We evaluate two variants. In the frozen setting, CLMBR-T is used as a fixed patient representation and only a logistic-regression classification head is trained. In the fine-tuned setting, the pretrained model is additionally adapted to each downstream task before classification. Because the mapping is incomplete, these results should be interpreted as a transfer benchmark under partial vocabulary alignment rather than as a comparison under fully matched input representations.

BioClinical-ModernBERT We use BioClinical-ModernBERT-base (150M parameters) Sounack et al. [34] as a pretrained medical text encoder and fine-tune all model parameters for each downstream prediction task. BioClinical-ModernBERT receives the same textualised EHR representation as tEHR. Its maximum context length is 8,192 tokens; trajectories exceeding this limit are truncated from the left to retain the most recent clinical history, while demographic information is preserved at the beginning of the sequence. A linear binary classification head is applied to the patient-level representation and all parameters are optimised jointly with the encoder using binary cross-entropy.

Zero-shot LLM. To evaluate whether pretrained LLM knowledge alone is sufficient for clinical prediction, we use the instruction-tuned version of the corresponding LLM without any parameter updates. The model is provided with the textualised patient trajectory together with a task-specific instruction requesting a binary prediction, encoded as 0 for control and 1 for case. Predictions are obtained from the model’s output logits for the two candidate labels, which are normalised to produce a continuous case score for AUROC and AUPRC evaluation. No examples or demonstrations are included in the prompt. The patient trajectory is truncated only when it exceeds the context window of the underlying LLM.

Prompt-based Supervised Fine-tuning. The prompt-based SFT baseline uses the same input representation and binary prediction prompt as the zero-shot setting, but adapts the LLM to the labelled downstream cohort using LoRA. Unlike tEHR, no continued pretraining on unlabelled EHR trajectories is performed and no discriminative classification head is introduced. Instead, the model is trained directly to generate the correct binary outcome token from the textualised patient trajectory. This baseline therefore evaluates whether conventional generative supervised fine-tuning of an LLM is sufficient for EHR prediction, relative to directly optimising the patient-level hidden representation as performed by tEHR.

The prompts for both are as follows:

You are an expert in clinical prediction from longitudinal electronic health record (EHR) data.
Your task is to predict whether the patient will {task} based only on the EHR information provided below.
Demographic information appears at the beginning, followed by clinical events in chronological order. Events are separated by semicolons. Numerical measurements are reported to one decimal place with their units, and elapsed time between events is included.
Consider the patient's longitudinal history, including relevant clinical events, measurements, and trends over time.
Answer with exactly one digit on a single line:
1 = yes
0 = no
Do not output any additional text, explanation, probability, or symbols.

Patient's medical history:
{ehr_text}

Answer:

Mistral-7B with Atomic Clinical-Code Tokens. To assess whether tEHR’s performance can be explained by the capacity of the pretrained LLM rather than its use of NL event descriptions, we construct a capacity-controlled variant using the same Mistral-7B backbone. Instead of textualising clinical codes, each unique medical concept is represented by an atomic token. We extend the Mistral vocabulary with the same clinical-code vocabulary used by the code-based Transformer baselines, with newly initialised embeddings for these codes.

The model follows the same two-stage training procedure as tEHR. During Stage 1, the LoRA adapters and newly introduced code embeddings are updated through autoregressive next-token continued pretraining on patient trajectories. During Stage 2, the same linear classification head is introduced, and the LoRA adapters and code embeddings are further optimised jointly with the classification head using binary cross-entropy. This preserves the pretrained backbone, model capacity, and adaptation procedure while replacing natural-language descriptions with learned symbolic code representations.

LLM-initialised Transformer. We evaluate whether pretrained language semantics can be incorporated into a conventional code-based EHR Transformer through its input embedding layer. This variant uses the same architecture and structured event representation as the Transformer Encoder baseline, but initialises its clinical-code embedding table using representations extracted from a pretrained LLM. Each clinical code is mapped to its corresponding natural-language description, which is independently encoded by the LLM to obtain a semantic embedding used to initialise that code’s representation, which can be further adapted in subsequent training. Following initialisation, the model uses the same two-stage training procedure as the standard Transformer Encoder, consisting of masked language modelling on unlabelled patient trajectories followed by task-specific supervised fine-tuning using binary cross-entropy. The remaining model architecture and training procedure are kept consistent with the original code-based Transformer baseline. This allows pretrained lexical information to be introduced through the code embeddings without requiring full natural-language trajectories to be processed by the longitudinal model.

Event LLM Embeddings To evaluate the utility of pretrained language representations without adapting the LLM itself, we encode each textualised clinical event independently using the frozen base LLM to obtain event-level embeddings. These embeddings are then passed to a trainable six-layer, eight-head Transformer encoder that models the longitudinal patient sequence and produces a patient-level representation for binary classification. The LLM parameters remain fixed throughout, while the Transformer aggregator and classification head are trained on the labelled cohort. This provides an event-level representation baseline analogous to Park et al. [25].

LLM Projection We evaluate an alternative approach in which pretrained event-level semantic representations are supplied directly to the longitudinal LLM rather than processing clinical event descriptions as natural-language token sequences. Each clinical event is independently encoded using a frozen pretrained LLM to obtain an event-level embedding. These embeddings are then projected into the input embedding space of Mistral-7B, allowing each clinical event to occupy a single position in the longitudinal sequence rather than multiple subword tokens.

The resulting sequence of projected event representations is processed by Mistral-7B to obtain a patient-level representation for downstream binary classification. Next-token prediction is not done, instead it is adapted directly to each downstream prediction task using supervised training. This separates event-level semantic encoding from longitudinal aggregation while reducing the sequence expansion associated with textualising clinical concepts.

The same patient-level train/validation/test splits are used as for tEHR and the different hyperparameters for each of the models can be seen in Table 5.

ModelHyperparameterSearch Space / Values
XGBoostLearning Rate (η\eta){0.01,0.05,0.1,0.2}\{0.01, 0.05, 0.1, 0.2\}
XGBoostMax Depth{5,6,7,8,9,10}\{5, 6, 7, 8, 9, 10\}
XGBoostSubsample Ratio{0.6,0.8,1.0}\{0.6, 0.8, 1.0\}
XGBoostGammalog-uniform [0.1,5.0][0.1, 5.0]
BioClinical-ModernBERTLearning Rate{1e−5,2e−5,5e−5}\{1e-5, 2e-5, 5e-5\}
BioClinical-ModernBERTWeight Decay{0.01,0.1}\{0.01, 0.1\}
BioClinical-ModernBERTDropout Rate{0.1,0.2,0.3}\{0.1, 0.2, 0.3\}
BioClinical-ModernBERTBatch Size{16,32}\{16, 32\}
Transformer EncoderHidden Dimension (dmodeld_{model}){256,512,768,1024}\{256, 512, 768, 1024\}
Transformer EncoderNumber of Layers{4,6,8,12}\{4, 6, 8, 12\}
Transformer EncoderAttention Heads{4,8}\{4, 8\}
Main experiments (All LoRA models)LoRA Rank (rr){8,16,32}\{8, 16, 32\}
Main experiments (All LoRA models)LoRA Alpha (α\alpha){16,32,64}\{16, 32, 64\}
Main experiments (All LoRA models)Learning Rate{5e−5,1e−4,2e−4}\{5e-5, 1e-4, 2e-4\}
Main experiments (All LoRA models)Weight Decay{0.01,0.1}\{0.01, 0.1\}
Main experiments (All LoRA models)Dropout{0.1,0.2}\{0.1, 0.2\}

Table 5. Hyperparameter Search Space for Baseline Models.

UMAP and HDBSCAN Implementation

Patient representations were analysed using UMAP followed by HDBSCAN clustering. Hyperparameters were selected on the validation set and then fixed before applying the analysis to the held-out test set. All embeddings were ℓ2\ell_2-normalised before dimensionality reduction.

Clustering. For hyperparameter selection, validation-set embeddings were reduced from 4,096 to 10 dimensions using UMAP. We searched over UMAP and HDBSCAN configurations, including n_neighbors = 10–50, min_cluster_size = 50–200, and min_samples = 10–40. Candidate configurations were required to produce 4–12 non-noise clusters with a noise fraction between 0.5% and 20%, and were ranked by silhouette score on the non-noise validation samples.

The selected configuration used UMAP with n_neighbors = 22, min_dist = 0.0, and Euclidean distance, followed by HDBSCAN with min_cluster_size = 127, min_samples = 12, Euclidean distance, and the eom cluster-selection method. These hyperparameters were then fixed and applied to embeddings from the held-out test set. HDBSCAN noise points were excluded from the clinical phenotype analysis.

Visualisation. For visualisation only, the test-set embeddings were independently projected to two dimensions using UMAP with n_neighbors = 15 and min_dist = 0.1. Cluster assignments from the 10-dimensional representation were overlaid on this projection; the two-dimensional projection was not used for clustering.

ComponentHyperparameterSearch space
UMAPnneighborsn_{\mathrm{neighbors}}10 - 50
UMAPncomponentsn_{\mathrm{components}}10
UMAPmin_dist0.0
UMAPMetricEuclidean
HDBSCANmin_cluster_size50–200
HDBSCANmin_samples10–40
HDBSCANCluster selection methodeom
HDBSCANMetricEuclidean
SearchNumber of trials50
SearchAllowed number of clusters4–12
SearchAllowed noise fraction0.5–20%
SearchOptimisation metricSilhouette score

Table 6. Hyperparameter search space for UMAP and HDBSCAN clustering. Candidate configurations were filtered by cluster-count and noise-fraction constraints, and valid configurations were ranked using silhouette score in the 10-dimensional UMAP space.

Clinical characterisation.

For each resulting cluster kk, we quantify the enrichment of a categorical clinical concept cc using the fold-enrichment ratio

FEc,k=P(c∣z=k)P(c)=mc,k/nkMc/N,(4)\mathrm{FE}_{c,k} = \frac{P(c \mid z = k)}{P(c)} = \frac{m_{c,k}/n_k}{M_c/N}, \tag*{(4)}

where mc,km_{c,k} is the number of patients in cluster kk containing concept cc, nkn_k is the total number of patients in cluster kk, McM_c is the number of patients containing concept cc in the full test population, and NN is the total number of patients in the test set. Values of FEc,k>1\mathrm{FE}_{c,k} > 1 indicate that concept cc is more prevalent within cluster kk than in the overall test population.

Lab value distribution by cluster

Figure 6. Lab value distribution by cluster. Red dotted lines indicate clinically healthy range.

Tokenisation and context length

Textualising structured EHR events expands each clinical event into multiple language-model subword tokens. On MIMIC-IV training trajectories, we measured a median expansion of 8.41 Mistral tokens per event (Table 7). At this rate, tEHR’s 16,000-token context window corresponds to approximately 1,900 clinical events at the median expansion rate. This is close to, but slightly below, the 2,048-event context used by the code-based Transformer. Consequently, tEHR does not gain an advantage from access to longer longitudinal histories. For sufficiently long trajectories, the Transformer Encoder can retain somewhat more clinical events than tEHR. For shorter trajectories that fit within both context limits, both models receive the full available history.

25th pct.Median75th pct.
BPE / event8.048.418.81

Table 7. Mistral BPE tokens per clinical event on MIMIC-IV training trajectories, measured after textualising the most recent 2,048 events with temporal information and numerical values retained. Percentiles are computed within each task and averaged equally across the four tasks.

MIMIC-IV Full results

Model / experimentICU Mortality AUROCICU Mortality AUPRCICU Readmission AUROCICU Readmission AUPRCHospital Mortality AUROCHospital Mortality AUPRCHospital Readmission AUROCHospital Readmission AUPRC
Baselines
XGBoost0.817±0.00090.817 \pm0.00090.306±0.00530.306 \pm0.00530.713±0.00080.713 \pm0.00080.333±0.00180.333 \pm0.00180.868±0.00070.868 \pm0.00070.235±0.00310.235 \pm0.00310.742±0.00100.742 \pm0.00100.483±0.00280.483 \pm0.0028
Transformer Encoder0.866±0.00100.866 \pm0.00100.370±0.00560.370 \pm0.00560.773±0.00090.773 \pm0.00090.456±0.00160.456 \pm0.00160.911±0.00070.911 \pm0.00070.281±0.00400.281 \pm0.00400.765±0.00130.765 \pm0.00130.531±0.00250.531 \pm0.0025
CLMBR-T (frozen)0.793±0.00040.793 \pm0.00040.235±0.00310.235 \pm0.00310.724±0.00060.724 \pm0.00060.340±0.00090.340 \pm0.00090.881±0.00040.881 \pm0.00040.204±0.00220.204 \pm0.00220.708±0.00050.708 \pm0.00050.409±0.00150.409 \pm0.0015
CLMBR-T (fine-tuned)0.861±0.00050.861 \pm0.00050.406±0.00380.406 \pm0.00380.762±0.00090.762 \pm0.00090.428±0.00110.428 \pm0.00110.915±0.00060.915 \pm0.00060.328±0.00290.328 \pm0.00290.762±0.00070.762 \pm0.00070.515±0.00220.515 \pm0.0022
BioClinical-ModernBERT0.824±0.00110.824 \pm0.00110.300±0.00520.300 \pm0.00520.739±0.00100.739 \pm0.00100.359±0.00210.359 \pm0.00210.860±0.00080.860 \pm0.00080.197±0.00410.197 \pm0.00410.766±0.00120.766 \pm0.00120.532±0.00300.532 \pm0.0030
Zero-shot LLM0.559±0.00190.559 \pm0.00190.109±0.00740.109 \pm0.00740.539±0.00190.539 \pm0.00190.147±0.00350.147 \pm0.00350.627±0.00150.627 \pm0.00150.048±0.00500.048 \pm0.00500.528±0.00200.528 \pm0.00200.232±0.00460.232 \pm0.0046
Prompt SFT – Mistral-7B-v0.30.811±0.00170.811 \pm0.00170.221±0.00630.221 \pm0.00630.668±0.00180.668 \pm0.00180.219±0.00320.219 \pm0.00320.786±0.00120.786 \pm0.00120.073±0.00440.073 \pm0.00440.724±0.00160.724 \pm0.00160.437±0.00430.437 \pm0.0043
Ablations: alternative semantic injection
LLM-initialised Transformer0.843±0.00070.843 \pm0.00070.351±0.00490.351 \pm0.00490.778±0.00080.778 \pm0.00080.467±0.00200.467 \pm0.00200.897±0.00080.897 \pm0.00080.275±0.00390.275 \pm0.00390.766±0.00110.766 \pm0.00110.532±0.00280.532 \pm0.0028
Event LLM embeddings0.739±0.00080.739 \pm0.00080.194±0.00600.194 \pm0.00600.547±0.00140.547 \pm0.00140.166±0.00280.166 \pm0.00280.876±0.00090.876 \pm0.00090.215±0.00490.215 \pm0.00490.739±0.00120.739 \pm0.00120.480±0.00290.480 \pm0.0029
LLM projection0.737±0.00090.737 \pm0.00090.210±0.00550.210 \pm0.00550.651±0.00090.651 \pm0.00090.273±0.00260.273 \pm0.00260.817±0.00100.817 \pm0.00100.156±0.00360.156 \pm0.00360.737±0.00090.737 \pm0.00090.483±0.00330.483 \pm0.0033
Ablations: training framework
No Stage 10.843±0.00140.843 \pm0.00140.347±0.00160.347 \pm0.00160.727±0.01070.727 \pm0.01070.326±0.00200.326 \pm0.00200.900±0.00090.900 \pm0.00090.265±0.00140.265 \pm0.00140.768±0.00050.768 \pm0.00050.522±0.00090.522 \pm0.0009
Frozen stage 1 model0.794±0.00150.794 \pm0.00150.240±0.00180.240 \pm0.00180.735±0.00180.735 \pm0.00180.366±0.00210.366 \pm0.00210.871±0.00100.871 \pm0.00100.207±0.00150.207 \pm0.00150.726±0.00050.726 \pm0.00050.462±0.00110.462 \pm0.0011
Frozen base LLM0.778±0.00160.778 \pm0.00160.222±0.00190.222 \pm0.00190.718±0.00190.718 \pm0.00190.335±0.00230.335 \pm0.00230.848±0.00110.848 \pm0.00110.189±0.00160.189 \pm0.00160.703±0.00050.703 \pm0.00050.418±0.00100.418 \pm0.0010
Mistral-7B atomic codes0.772±0.00080.772 \pm0.00080.221±0.00550.221 \pm0.00550.703±0.00090.703 \pm0.00090.323±0.00190.323 \pm0.00190.871±0.00060.871 \pm0.00060.226±0.00320.226 \pm0.00320.766±0.00100.766 \pm0.00100.511±0.00230.511 \pm0.0023
Ablations: clinical information representation
Hashed concepts0.766±0.00160.766 \pm0.00160.245±0.00200.245 \pm0.00200.702±0.00180.702 \pm0.00180.299±0.00220.299 \pm0.00220.873±0.00090.873 \pm0.00090.218±0.00150.218 \pm0.00150.736±0.00050.736 \pm0.00050.463±0.00100.463 \pm0.0010
Scrambled concepts0.791±0.00150.791 \pm0.00150.285±0.00190.285 \pm0.00190.705±0.00180.705 \pm0.00180.295±0.00220.295 \pm0.00220.869±0.00100.869 \pm0.00100.226±0.00150.226 \pm0.00150.728±0.00050.728 \pm0.00050.450±0.00100.450 \pm0.0010
No elapsed-time text0.887±0.00120.887 \pm0.00120.428±0.00150.428 \pm0.00150.794±0.00160.794 \pm0.00160.499±0.00180.499 \pm0.00180.931±0.00080.931 \pm0.00080.346±0.00120.346 \pm0.00120.772±0.00050.772 \pm0.00050.540±0.00090.540 \pm0.0009
No numerical values0.851±0.00130.851 \pm0.00130.346±0.00170.346 \pm0.00170.759±0.00170.759 \pm0.00170.411±0.00210.411 \pm0.00210.913±0.00090.913 \pm0.00090.297±0.00140.297 \pm0.00140.752±0.00050.752 \pm0.00050.481±0.00100.481 \pm0.0010
tEHR
tEHR-Qwen3-0.6B0.822±0.00100.822 \pm0.00100.289±0.00520.289 \pm0.00520.738±0.00110.738 \pm0.00110.414±0.00230.414 \pm0.00230.903±0.00090.903 \pm0.00090.273±0.00410.273 \pm0.00410.759±0.00100.759 \pm0.00100.522±0.00370.522 \pm0.0037
tEHR-Qwen3-8B0.870±0.00110.870 \pm0.00110.391±0.00440.391 \pm0.00440.780±0.00090.780 \pm0.00090.480±0.00160.480 \pm0.00160.927±0.00150.927 \pm0.00150.324±0.00390.324 \pm0.00390.764±0.00120.764 \pm0.00120.527±0.00250.527 \pm0.0025
tEHR-Mistral-7B-v0.30.887±0.00080.887 \pm0.00080.424±0.00480.424 \pm0.00480.794±0.00080.794 \pm0.00080.498±0.00150.498 \pm0.00150.932±0.00040.932 \pm0.00040.333±0.00350.333 \pm0.00350.775±0.00090.775 \pm0.00090.556±0.00240.556 \pm0.0024

Table 8. MIMIC-IV baseline, ablation, and tEHR results. Point estimates are test-set AUROC / AUPRC presented as mean ± standard deviation across 3 seeds. Ablations use tEHR-Mistral-7B-v0.3 unless otherwise specified.

Additional Metrics

We report precision, recall, F1, balanced accuracy, accuracy, and Matthews correlation coefficient (MCC) for all MIMIC-IV tasks, as indicated in the table below. Because these metrics require a decision threshold, the threshold for each model and task was selected exclusively on the validation set by maximising F1 score, and was then fixed before evaluation on the test set. This avoids tuning the operating point on the test data. Across the four tasks, tEHR-Mistral-7B achieves the highest F1 and MCC in all four settings, while other models occasionally attain higher recall or accuracy depending on the selected operating point.

ModelThr.Prec.Rec.F1Bal. acc.Acc.MCC
XGBoost0.1800.3310.3990.3620.6660.8920.305
BioClinical-ModernBERT0.2220.3230.4110.3620.6700.8890.304
Transformer Encoder0.1560.3630.4840.4150.7070.8950.363
Zero-shot LLM0.1500.1680.1790.1740.5530.8700.103
Prompt SFT0.4000.2230.6600.3330.7350.7980.298
CLMBR-T (frozen)0.7220.2730.3920.3220.6530.8740.259
CLMBR-T (fine-tuned)0.0970.3370.5800.4260.7430.8800.382
tEHR-Mistral-7B0.2450.4220.4850.4510.7150.9100.403

Table 9. ICU mortality. Thresholds maximise validation F1 and are then fixed and applied to the test set. Best test-set value for each metric is shown in bold.

ModelThr.Prec.Rec.F1Bal. acc.Acc.MCC
XGBoost0.1650.2850.4350.3440.6330.7770.224
BioClinical-ModernBERT0.1540.3050.5210.3850.6680.7760.273
Transformer Encoder0.1490.4030.4830.4400.6860.8350.346
Zero-shot LLM0.2400.1510.7270.2500.5460.4140.066
Prompt SFT0.4800.2220.5480.3160.6250.6810.181
CLMBR-T (frozen)0.5940.3190.4530.3750.6520.7970.263
CLMBR-T (fine-tuned)0.2840.3810.4540.4140.6700.8270.316
tEHR-Mistral-7B0.2170.4300.5140.4680.7040.8430.379

Table 10. ICU readmission. Thresholds maximise validation F1 and are then fixed and applied to the test set. Best test-set value for each metric is shown in bold.

ModelThr.Prec.Rec.F1Bal. acc.Acc.MCC
XGBoost0.1850.3400.2920.3140.6380.9640.297
BioClinical-ModernBERT0.0930.2280.3850.2860.6730.9460.270
Transformer Encoder0.1340.3310.3510.3410.6650.9620.321
Zero-shot LLM0.2300.0490.3810.0870.5840.7760.068
Prompt SFT0.2340.0880.5550.1510.6940.8250.167
CLMBR-T (frozen)0.8580.2370.3870.2940.6750.9480.277
CLMBR-T (fine-tuned)0.0680.3380.4440.3840.7090.9600.367
tEHR-Mistral-7B0.2020.3630.4470.4010.7120.9620.384

Table 11. Hospital mortality. Thresholds maximise validation F1 and are then fixed and applied to the test set. Best test-set value for each metric is shown in bold.

ModelThr.Prec.Rec.F1Bal. acc.Acc.MCC
XGBoost0.2030.3700.6380.4680.6730.6930.293
BioClinical-ModernBERT0.2320.4180.6070.4950.6900.7370.336
Transformer Encoder0.1550.4030.6380.4940.6920.7230.332
Zero-shot LLM0.2400.2350.7720.3610.5490.4200.087
Prompt SFT0.3210.3260.7360.4520.6630.6210.267
CLMBR-T (frozen)0.5400.3610.5830.4460.6530.6930.264
CLMBR-T (fine-tuned)0.5210.4160.6130.4960.6910.7360.337
tEHR-Mistral-7B0.2660.4570.5940.5170.7020.7640.370

Table 12. Hospital readmission. Thresholds maximise validation F1 and are then fixed and applied to the test set. Best test-set value for each metric is shown in bold.

CPRD AUPRC

AUPRC for pancreatic cancer prediction at horizons of 1, 3, 6 and 12 months preceding diagnosis for tEHR, XGBoost, Transformer Encoder and BioClinical ModernBERT and zero-shot LLM

Figure 7. AUPRC for pancreatic cancer prediction at horizons of 1, 3, 6 and 12 months preceding diagnosis for tEHR, XGBoost, Transformer Encoder and BioClinical ModernBERT and zero-shot LLM.

References

  1. [1]Rafi Al Attrach, Rajna Fani, David Restrepo, Yugang Jia, and Peter Schüffler. Rethinking Tokenization for Clinical Time Series: When Less is More. 12 2025.
  2. [2]Bert Arnrich, Edward Choi, Jason Alan Fries, Matthew B.A. McDermott, Jungwoo Oh, Tom Pollard, Nigam Shah, Ethan Steinberg, Michael Wornow, and Robin van de Water. Medical Event Data Standard ({MEDS}): Facilitating Machine Learning for Health. ICLR 2024 Workshop on Learning from Time Series For Health, 2024.
  3. [3]Payal Chandak, Gregory Kondas, Liat Antwarg Friedman, Isaac Kohane, and Matthew McDermott. EveryQuery: Zero-Shot Clinical Prediction via Task-Conditioned Pretraining over Electronic Health Records. 5 2026.
  4. [4]J Chisholm. The Read clinical classification. BMJ, 300(6732):1092–1092, 4 1990. ISSN 0959-8138. doi: 10.1136/bmj.300.6732.1092.DOI
  5. [5]Michael Han Daniel Han and Unsloth team. Unsloth, 2023. URL https://github.com/unslothai/unsloth.
  6. [6]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT. In Proceedings of the 2019 Conference of the North, pages 4171–4186, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423.DOI
  7. [7]Tong Ding, Caiwei Tian, Dongmin Bang, Ming Y. Lu, Sophia J. Wagner, Andrew Zhang, Rowland Pettit, Long Phi Le, and Faisal Mahmood. Discrete Neural Interlingua for Ontology-Agnostic EHR Modeling. 10 2026.arxiv.org/abs/2610.27047
  8. [9]Lin Lawrence Guo, Santiago Eduardo Arciniegas, Joseph Jihyung Lee, Adam Paul Yan, George Tomlinson, Jason Fries, and Lillian Sung. Tokenization Tradeoffs in Structured EHR Foundation Models. 3 2026.arxiv.org/abs/2603.15644
  9. [10]Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. 8 2019. doi: 10.1038/s41597-019-0103-9.DOI
  10. [11]Stefan Hegselmann, Georg von Arnim, Tillmann Rheude, Noel Kronenberg, David Sontag, Gerhard Hindricks, Roland Eils, and Benjamin Wild. Large language models are powerful electronic health record encoders. npj Digital Medicine, 9 (1):530, 7 2026. ISSN 2398-6352. doi: 10.1038/s41746-026-02915-9.DOI
  11. [12]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. 10 2021.arxiv.org/abs/2106.09685
  12. [13]Kyung-hoon Hur, Jungwoo Oh, Junu Kim, Jiyoun Kim, Min Jae Lee, Eunbyeol Cho, Seong-Eun Moon, Young-Hak Kim, and Edward Choi. UniHPF : Universal Healthcare Predictive Framework with Zero Domain Knowledge. 9 2024.
  13. [14]Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Brian Gow, Benjamin Moody, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV. PhysioNet, 10 2024. doi: 10.13026/kpb9-mt58. URL https://doi.org/10.13026/kpb9-mt58.
  14. [15]Zeljko Kraljevic, Dan Bean, Anthony Shek, Rebecca Bendayan, Harry Hemingway, Joshua Au Yeung, Alexander Deng, Alfie Baston, Jack Ross, Esther Idowu, James T Teo, and Richard J Dobson. Foresight – Generative Pretrained Transformer (GPT) for Modelling of Patient Timelines using EHRs. 1 2023.arxiv.org/abs/2212.08072
  15. [16]Inhyeok Lee, Luke Solo, Michael C. Burkhart, Bashar Ramadan, William F. Parker, and Brett K. Beaulieu-Jones. Representation Before Training: A Fixed-Budget Benchmark for Generative Medical Event Models. 4 2026.
  16. [17]Nikolai Makarov and Mikhail Lipkovich. A transformer-based model for next disease prediction using electronic health records. The European Physical Journal Special Topics, 1 2025. ISSN 1951-6355. doi: 10.1140/epjs/s11734-024-01447-1.DOI
  17. [18]Matthew McDermott and Justin Xu. MIMIC-IV MEDS: An ETL pipeline to extract MIMIC-IV data into the MEDS format, 4 2025. URL https://github.com/Medical-Event-Data-Standard/MIMIC_IV_MEDS.
  18. [19]Leland McInnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. 9 2020.
  19. [20]Qingcai Meng, Si Shi, Chen Liang, Jinfeng Xiang, Dingkong Liang, Bo Zhang, Yi Qin, Shunrong Ji, Wenyan Xu, Jin Xu, Quanxing Ni, and Xianjun Yu. Diagnostic Accuracy of a CA125-Based Biomarker Panel in Patients with Pancreatic Cancer: A Systematic Review and Meta-Analysis. Journal of Cancer, 8(17):3615–3622, 2017. ISSN 1837-9664. doi: 10.7150/jca.18901.DOI
  20. [21]Anna Munoz-Farre, Harry Rose, and Sera Aylin Cakiroglu. sEHR-CE: Language modelling of structured EHR data for efficient and generalizable patient cohort expansion. 11 2022.
  21. [22]Philip N Newsome, Rob Cramb, Suzanne M Davison, John F Dillon, Mark Foulerton, Edmund M Godfrey, Richard Hall, Ulrike Harrower, Mark Hudson, Andrew Langford, Anne Mackie, Robert Mitchell-Thain, Karen Sennett, Nicholas C Sheron, Julia Verne, Martine Walmsley, and Andrew Yeoman. Guidelines on the management of abnormal liver blood tests. Gut, 67(1):6–19, 1 2018. ISSN 0017-5749. doi: 10.1136/gutjnl-2017-314924.DOI
  22. [25]Jiheum Park, Jason Patterson, Jose M. Aci-tores Cortina, Tian Gu, Chin Hur, and Nicholas Tatonetti. Enhancing EHR-based pancreatic cancer prediction with LLM-derived embeddings. npj Digital Medicine, 8(1):465, 7 2025. ISSN 2398-6352. doi: 10.1038/s41746-025-01869-8.DOI
  23. [28]Haresh Rengaraj Rajamohan, Xiang Gao, Weicheng Zhu, Shih-Lun Huang, Long Chen, Kyunghyun Cho, Cem M. Deniz, and Narges Razavian. Foundation Models for Clinical Records at Health System Scale. 7 2025.
  24. [29]Pawel Renc, Yugang Jia, Anthony E. Samir, Jaroslaw Was, Quanzheng Li, David W. Bates, and Arkadiusz Sitek. Zero shot health trajectory prediction using transformer. npj Digital Medicine, 7(1):256, 9 2024. ISSN 2398-6352. doi: 10.1038/s41746-024-01235-0.DOI
  25. [30]Mia Schmidt-Hansen, Sabine Berendse, and William Hamilton. Symptoms of Pancreatic Cancer in Primary Care. Pancreas, 45(6):814–818, 7 2016. ISSN 0885-3177. doi: 10.1097/MPA.0000000000000527.
  26. [31]John Schulman and Thinking Machines Lab. LoRA Without Regret. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20250929.DOI
  27. [32]Artem Shmatko, Alexander Wolfgang Jung, Kumar Gaurav, Søren Brunak, Laust Hvas Mortensen, Ewan Birney, Tom Fitzgerald, and Moritz Gerstung. Learning the natural history of human disease with generative transformers. Nature, 9 2025. ISSN 0028-0836. doi: 10.1038/s41586-025-09529-3.DOI
  28. [33]Ofir Ben Shoham and Nadav Rappoport. CPLLM: Clinical Prediction with Large Language Models. 5 2024.
  29. [34]Thomas Sounack, Joshua Davis, Brigitte Durieux, Antoine Chaffin, Tom J. Pollard, Eric Lehman, Alistair E. W. Johnson, Matthew McDermott, Tristan Naumann, and Charlotta Lindvall. BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP. 6 2025.arxiv.org/abs/2506.10896
  30. [35]M Q Stearns, C Price, K A Spackman, and A Y Wang. SNOMED clinical terms: overview of the development process and project status. Proceedings. AMIA Symposium, pages 662–6, 2001. ISSN 1531-605X. Ethan Steinberg, Jason Fries, Yizhe Xu, and Nigam Shah. MOTOR: A Time-To-Event Foundation Model For Structured Medical Records. 12 2023.
  31. [36]Elizabeth A Stewart. Uterine fibroids. The Lancet, 357(9252):293–298, 1 2001. ISSN 01406736. doi: 10.1016/S0140-6736(00)03622-9.DOI
  32. [37]Lan Valerie Tao, Joanna M. Karasinska, Vanessa G. P. Souza, Jonathan M. Loree, James D. Johnson, Daniel J. Renouf, and David F. Schaeffer. New-onset diabetes as an emerging risk group for early detection of pancreatic cancer: current evidence, clinical challenge, and future directions. Frontiers in Gastroenterology, 4, 1 2026. ISSN 2813-1169. doi: 10.3389/f гастr.2025.1645459.DOI
  33. [38]Lan Wang, Yonghua Yin, Ben Glampson, Robert Peach, Mauricio Barahona, Brendan C. Delaney, and Erik K. Mayer. Transformer-based deep learning model for the diagnosis of suspected lung cancer in primary care based on electronic health record data. eBioMedicine, 110:105442, 12 2024. ISSN 23523964. doi: 10.1016/j.ebiom.2024.105442.DOI
  34. [39]Shane Waxler, Paul Blazek, Davis White, Daniel Sneider, Kevin Chung, Mani Nagarathnam, Patrick Williams, Hank Voeller, Karen Wong, Matthew Swanhорst, Sheng Zhang, Naoto Usuyama, Cliff Wong, Tristan Naumann, Hoifung Poon, Andrew Loza, Daniella Meeker, Seth Hain, and Rahul Shah. Generative Medical Event Models Improve with Scale. 11 2025.arxiv.org/abs/2508.12104
  35. [40]Achim Wolf, Daniel Dedman, Jennifer Campbell, Helen Booth, Darren Lunn, Jennifer Chapman, and Puja Myles. Data resource profile: Clinical Practice Research Datalink (CPRD) Aurum. International Journal of Epidemiology, 48(6):1740–1740, 12 2019. ISSN 0300-5771. doi: 10.1093/ije/dyz034.DOI
  36. [42]Michael Wornow, Rahul Thapa, Ethan Steinberg, Jason A. Fries, and Nigam H. Shah. EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models. 12 2023.arxiv.org/abs/2307.02028
  37. [43]Justin Xu, Jack Gallifant, Alistair E. W. Johnson, and Matthew B. A. McDermott. ACES: Automatic Co-hort Extraction System for Event-Stream Datasets.arxiv.org/abs/2406.19653
  38. [44]Yinghao Zhu, Zixiang Wang, Junyi Gao, Yuning Tong, Jingkun An, Weibin Liao, Ewen M. Harrison, Liantao Ma, and Chengwei Pan. Prompting Large Language Models for Zero-Shot Clinical Prediction with Structured Longitudinal Electronic Health Record Data. 2 2024.arxiv.org/abs/2402.01713

Paper details

vlnc:W_2r73qpmu
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents