WorksComputer Science
SLICES: A Benchmark and Evaluation Framework for Self-Supervised Learning on Sparse ICU Time Series
Abstract
Self-supervised learning (SSL) results for intensive care unit (ICU) time series are hard to compare when data processing, encoders, and adaptation vary together. We present SLICES, a benchmark and evaluation framework for comparing and extending SSL methods under a shared clinical setup. It fixes 24-hour value and observation-mask inputs, a Transformer encoder, patient-level splits, four endpoints, two adaptation protocols, and five matched seeds across MIMIC-IV, eICU-CRD, and their pooled union. Masked autoencoding (MAE), joint-embedding prediction (JEPA), and contrastive learning provide reference implementations. Among these objectives, MAE leads mean AUPRC in all nine frozen classification comparisons, also under converged probes, and in seven of nine fine-tuning comparisons. Fine-tuned MAE leads hospital mortality at full labels and scores higher than a supervised Transformer of the same architecture at every label fraction, and SSL fine-tuning gives the lowest length-of-stay error. Supervised baselines remain strong elsewhere: GRU-D leads 24-hour mortality and XGBoost leads acute kidney injury in all three regimes. Rankings depend on the adaptation protocol, the comparator set, and whether the probe loss matches the metric. A two-seed MIMIC-IV rerun with the maintained code stays within two across-seed SDs of the reference in all 48 downstream runs. The code repository supports new objectives, shared downstream evaluation, and regeneration of the reference results, so new SSL methods can be compared with these references under the same conditions. Generalization to unseen hospitals is not tested here and is the natural next extension.
Keywords: self-supervised learning, clinical time series, intensive care, benchmark, missing data
Data and Code Availability We use deidentified MIMIC-IV v3.1 and eICU-CRD v2.0 data from PhysioNet under credentialed access [10, 15]. Code, run-level aggregate results, post-review diagnostics, and executable audits are available in the code repository at github.com/hannesill/SLICES (tag ml4h-2026). The repository excludes patient records, split identities, credentials, and trained patient-data weights. Aggregate auditing needs no data access. Training reproduction requires independent credentialing, training, and a data-use agreement.
Institutional Review Board (IRB) This secondary analysis of deidentified, credentialed-access data under the PhysioNet data-use agreements does not require IRB approval.
Introduction
ICU records are sparse because measurements are taken at different frequencies and often in response to a patient’s evolving condition. The absence and timing of a measurement can therefore carry information about both physiology and care processes [19]. This makes ICU time series a relevant test bed for SSL. Unlabeled measurements are abundant, but the measurement process complicates the choice of learning signal.
Prior clinical SSL studies use reconstruction, forecasting, contrastive learning, or combined pretext tasks [20, 13, 24] (Oufattole et al. 2024). Their differences are hard to attribute when cohorts, representations, encoders, augmentations, and evaluation also vary.
a systematic review of structured-EHR foundation models found similar heterogeneity in features, temporal representation, SSL objectives, and adaptation, with compute often underreported [7]. Frozen probes test whether task information is linearly accessible. Fine-tuning tests whether a pretrained initialization remains useful after label-driven adaptation. Either alone can hide a protocol-dependent ranking.
SLICES makes the evaluation contract reusable (Figure 1). A new objective consumes the same value–mask tensors, returns a loss through a common interface, and exports the same encoder for downstream evaluation. A machine-readable specification and result validators make changes to the comparison explicit. The controlled unit is the complete implementation: masking geometry, auxiliary networks, encoder passes, and pretraining parameter counts remain objective specific.
Figure 1. Benchmark overview. Cohorts, hourly inputs, encoder architecture, patient splits, tasks, and evaluation protocols are fixed. SSL implementations vary in their masks and auxiliary modules. Equal epochs do not imply equal compute. Baselines provide context, and TS2Vec-style illustrates extension. LOS uses mean absolute error in days.
Our contributions are:
A benchmark framework: a fixed sparse-ICU specification, an objective/encoder-transfer interface with MAE, JEPA, and contrastive reference implementations, result validators, and a documented extension workflow. It fixes one representation, hourly values with observation masks.
A first study run with the framework: 2,455 training runs across two databases and their union, with five matched seeds. GRU-D and XGBoost lead six of twelve full-label cells, fine-tuned SSL leads hospital mortality narrowly and LOS, and MAE leads all frozen SSL classification comparisons. Rankings depend on the adaptation protocol, label fraction, comparator set, and probe loss.
A code repository with the historical training implementation, framework code, checksummed aggregate results, and tools to regenerate results and check comparability without patient-data access.
Related Work
ICU benchmarks. The MIMIC-III benchmark formalized common ICU prediction tasks and evaluation procedures [8]. YAIB provided a modular multi-center clinical framework over several ICU databases using harmonized preprocessing [21]. SLICES builds on explicit cohorts, labels, and leakage controls with a setup designed for SSL comparisons. It holds the clinical inputs, encoder, and downstream evaluation fixed across objectives, reducing variation from these sources.
Clinical time-series SSL. STraTS represents sparse observations as time–variable–value triplets and uses forecasting-style pretraining [20]. DuETT uses dual event-time attention and self-supervised prediction tasks [13]. Neighborhood and event-based contrastive methods use clinical structure to define positive pairs or anchors [24] (Oufattole et al. 2024). Sequential multi-dimensional SSL evaluates contrastive and non-contrastive losses at multiple levels of clinical trajectories [17]. SMART combines missing-aware temporal and variable attention with latent reconstruction [25]. AID-MAE is a recent masked-modeling approach for incomplete EHR time series [22]. These systems motivate the objectives studied here but change either the representation, architecture, positive-pair rule, or task contract.
Closest comparative work. TS-JEPA compares JEPA, masked autoencoding, autoregression, and TS2Vec-style contrastive pretraining with a common Transformer, on sensor and forecasting datasets, with frozen-head evaluation and a low-label study [6]. It is the closest objective-level predecessor. SLICES differs in its missingness-preserving ICU interface, patient-level clinical splits, paired probe and fine-tuning evaluation, and matched MIMIC-IV–eICU-CRD regimes. Clin-JEPA uses text-serialized hourly states and an LLM-based encoder, so it changes the model and input contract rather than testing a shared small-encoder interface [23]. General MAE, I-JEPA, SimCLR, and TS2Vec formulations provide the objective context [9, 1, 5, 26].
Scores from these studies are not directly comparable with SLICES because cohorts, representations, and tasks differ.
Benchmark Design
Cohorts and Hourly Representation
We use MIMIC-IV v3.1 and eICU-CRD v2.0 [11, 16]. Common extraction and unit harmonization are based on RICU [2]. Each sample is one ICU stay with valid patient and stay metadata, ICU duration of at least 24 hours, and at least one observed time-series value in the first 24 hours. The processed cohorts contain 74,829 MIMIC-IV stays and 132,686 eICU-CRD stays. We form their 207,515-stay union, Combined, after schema compatibility checks and source-specific identifier namespacing. MIMIC-IV, eICU-CRD, and Combined are the three regimes; Combined is pooled, not a separate cohort.
Each stay is represented as normalized values and an observation mask . Feature means and standard deviations are fitted on the training partition only. Missing cells are zero filled after normalization while preserves whether a value was observed. RICU aggregates repeated measurements within an hour by concept-specific rules, by default the median for numeric values and “any” for logical indicators, with exceptions such as summed urine output. The mask records whether at least one value fell in an hour, not how many, so within-hour sampling intensity is lost. The 84 retained variables are shared across regimes. Only 10.7% to 15.3% of tensor cells are observed, although most stays have at least one measurement in nearly every hour.
Patients are assigned to 70/15/15 train, validation, and test partitions for each seed. All stays for one patient remain in one partition. Task-specific missing labels are removed only after this assignment, and the same split is used for pretraining and downstream evaluation. Input normalization uses training data only. Direct length-of-stay variables, future-duration helpers, and derived features that expose downstream outcomes are excluded from the input.
Endpoints and Horizons
All tasks use the first 24 ICU hours as input. The endpoints follow task families of established ICU benchmarks (Harutyunyan et al., [8]; van de Water et al., [21]). They differ in prevalence, horizon, label construction, and target type, which can change an objective ranking. Classification uses area under the precision–recall curve (AUPRC), computed as average precision, because prevalence is low and endpoint dependent (Saito and Rehmsmeier, [18]). Mean test prevalence spans 1.6%–20.3% (Table 4). Twenty-four-hour mortality predicts death in hours 24–48. Hospital mortality predicts in-hospital death after hour 24. Deaths within the input window receive null labels. Creatinine-defined acute kidney injury (AKI) uses the minimum in hours 0–24 as baseline and seeks KDIGO stage evidence in hours 24–48: a later value at least 1.5 times baseline or at least 0.3 mg/dL above an earlier value within 48 hours (KDIGO AKI Work Group, 2012). Urine output is excluded. A missing baseline or forward measurement yields a null label. Remaining ICU length of stay (LOS) is ICU duration minus one day, clipped at zero. We retain this continuous target rather than impose a binary threshold. Absolute error keeps prediction misses interpretable in days.
| Data | Task | Test samples | Positive | Prevalence |
| Combined | 24h mortality | 30,863.4 | 492.4 | 0.0160 |
| Combined | Hospital mortality | 30,869.0 | 3,135.8 | 0.1016 |
| Combined | AKI | 21,724.0 | 3,565.2 | 0.1641 |
| Combined | Remaining LOS | 31,092.2 | – | – |
| eICU-CRD | 24h mortality | 19,788.6 | 315.2 | 0.0159 |
| eICU-CRD | Hospital mortality | 19,788.6 | 1,823.8 | 0.0922 |
| eICU-CRD | AKI | 13,297.4 | 1,850.0 | 0.1391 |
| eICU-CRD | Remaining LOS | 19,966.4 | – | – |
| MIMIC-IV | 24h mortality | 11,150.0 | 185.4 | 0.0166 |
| MIMIC-IV | Hospital mortality | 11,157.0 | 1,310.0 | 0.1174 |
| MIMIC-IV | AKI | 8,423.2 | 1,710.6 | 0.2031 |
| MIMIC-IV | Remaining LOS | 11,194.8 | – | – |
Table 4. Mean test support across the five patient splits. Positive counts and prevalence apply to classification tasks.
Shared Encoder and Objective-Specific Components
Appendix Figure 4 shows the shared interface. Each hour is one continuous attention token: 84 normalized values and 84 observation indicators are concatenated and projected from to , without value quantization or vocabulary lookup. The shared small Transformer has sinusoidal time encoding, two pre-layer-normalized layers, four heads, feed-forward width 256, and dropout .1. Empty hours are excluded from attention and SSL losses. Downstream heads mean-pool valid outputs. The same 159,808-parameter encoder transfers for every SSL method.
Figure 4. Shared hourly tokenization and encoder. Values and masks form 168 inputs per hour, projected to 64 dimensions. Empty hours are excluded from attention and losses. SSL uses timestep outputs, with objective-specific pooling for contrastive learning. Downstream heads mean-pool valid hours.
Figure 2 contrasts the learning signals. MAE masks random non-empty hours, encodes visible tokens, and uses a two-layer Transformer decoder to reconstruct observed normalized values at masked hours. Missing cells do not enter the loss. JEPA masks up to three contiguous blocks. An online encoder and predictor estimate layer-normalized embeddings from a stop-gradient exponential-moving-average target encoder. Contrastive learning makes two complementary masked views, encodes each separately, mean-pools visible hours, and applies an MLP projection with an NT-Xent instance loss at temperature .07. Only the common encoder transfers downstream.
Figure 2. Core objective geometries. MAE reconstructs observed values at masked hours. JEPA’s online encoder receives visible valid hours. Its EMA target encoder receives the full unmasked valid sequence. JEPA predicts layer-normalized, stop-gradient targets, with loss only on masked valid hours. Contrastive learning aligns projected pooled views. Auxiliary modules are discarded before downstream transfer.
The implementations preserve reconstruction, latent prediction, and view agreement as distinct learning signals while sharing inputs, transferred encoder, and evaluation. Forecasting objectives are out of scope. Objective-specific masks and auxiliary modules retain each method’s learning structure. MAE has 269,460 trainable pretraining parameters, JEPA has 189,440 plus a non-trainable 159,808-parameter target copy, and contrastive learning has 193,408 with two encoder passes. These are comparisons of complete implementations, not causal loss-function effects. Appendix Table 5 and Table 6 summarize the pretraining contract and modules.
| Objective | Masking | Target | Ratio |
| MAE | random hours | observed values | .5 |
| JEPA | blocks | target embeddings | .5 |
| Contrastive | two views | view agreement | .5 |
Table 5. Core pretraining contract. Ratio is the configured timestep-mask ratio. All rows use AdamW, batch size 256, 100 epochs, and the same transferred encoder.
| Objective | Geometry | Learning target | Auxiliary module | Trainable | Transferred |
| MAE | random masked hours | observed normalized cells | Transformer decoder | 269,460 | 159,808 |
| JEPA | up to three masked blocks | EMA target embeddings | predictor + target encoder | 189,440 | 159,808 |
| Contrastive | complementary views | instance agreement | projection MLP | 193,408 | 159,808 |
Table 6. Objective-specific modules and transferred capacity. Shared settings are listed in § 4.
Experimental Setup
Using and Extending the Benchmark
The code repository supports auditing the archived results (the aggregates of all 2,455 runs), regenerating tables and figures without retraining, reproducing the comparison with credentialed data access, and evaluating additional SSL objectives (Appendix Figure 8). A machine-readable specification records clinical inputs, splits, tasks, encoder, training and checkpoint policies, and metrics. Comparability checks flag changed settings.
Figure 8. SLICES benchmark. The reusable path from fixed specification through objective extension and encoder transfer to auditable results. Comparison checks enforce declared settings and export coverage. They do not prove that a training run used its declared private data. Objective-specific modules and compute remain different.
New objectives use the shared value–mask input and export an encoder for both frozen probing and fine-tuning. Pretraining retains hourly representations. Downstream evaluation averages valid hours into one stay-level vector because each endpoint assigns one target to a stay. Auxiliary pretraining networks are discarded. The TS2Vec-style extension demonstrates this path. A short run on artificial data checks the workflow without patient-data access. Its scores are not clinical evidence. Appendix B.§ 6 gives implementation and validation details.
Historical source, current framework, archived results, and later diagnostics remain separately traceable. The framework comprises the specification, interface, validators, and extension workflow. The reference evaluation below is one study run through it.
Reference Evaluation
The study comprises 2,455 training runs: 145 pretraining and 2,310 downstream or baseline runs. It tests objective and protocol choice, reduced-label performance, cross-database adaptation, hyperparameter and encoder-capacity sensitivity, classical baselines, and the TS2Vec-style extension. The core comparison accounts for 45 pretraining runs and 420 evaluations.
Pretraining. All methods use 100 epochs, batch size 256, AdamW learning rate , weight decay .01, gradient clipping at 1.0, ten warmup epochs, and cosine decay to , without early stopping. Batches contain only training-partition stays, and validation loss is monitored. Equal epochs give an equal data-pass schedule within each regime, not equal FLOPs, wall time, or objective-specific module cost. Configurations, shared across datasets, were fixed before the final rerun but informed by earlier development passes, whose test metrics we saw while fixing label and input defects. Test partitions were never used for training, normalization, or checkpoint selection, and objective-specific changes followed label-free pretraining diagnostics (Appendix § 11.§ 4).
Adaptation. Frozen probing trains a linear head for at most 50 epochs with learning rate while keeping the encoder fixed. Full fine-tuning trains the encoder and a width-64 ReLU head for at most 100 epochs with learning rate . Both use batch size 64, weight decay .05, cosine scheduling, gradient clipping at 1.0, and validation patience 10. Neural classification uses square-root-balanced class weights and label smoothing of .1. LOS trains with mean squared error but uses mean absolute error in days for evaluation and checkpoint selection. Classification checkpoint selection maximizes validation AUPRC. The two protocols differ in trainable scope, head capacity, learning rate, and epoch cap.
Converged and loss-matched probes. The capped probe is not trained to convergence, and its LOS training loss differs from the reported metric. We therefore refit all 180 core full-label frozen probes on the same embeddings, splits, and test rows: L2 logistic regression fitted to convergence, and for LOS a Huber probe approximating absolute error and a ridge probe. Validation selects regularization, and the protocol was fixed before any of its test metrics were computed (Appendix C.§ 1). Appendix C collects all analyses added during peer review. We recommend the converged probe for new objectives; its script is released with the diagnostics but is not yet a framework protocol.
Baselines. The supervised Transformer uses the same encoder family and full-fine-tuning trainer from random initialization. GRU-D provides a recurrent, missingness-aware comparator ([3]). XGBoost uses per-variable summary features, including extrema, first and last values, counts, and observation fractions ([4]). TS2Vec is a temporal-contrastive extension, not a fourth core objective.
Seeds, metrics, and checkpoints. All reported core cells contain five runs with matched seeds. A seed jointly determines the patient split and stochastic training. We report mean and sample standard deviation across runs, without confirmatory significance claims (§ 7).
Every downstream SSL run starts from the encoder saved after the final pretraining epoch, not one selected on objective-specific validation loss. This gives all objectives the same 100-epoch data-pass rule but may disadvantage an objective whose useful representations peak earlier. Selecting encoders on downstream validation would make SSL checkpoint choice label-dependent and task-specific. The final epoch is a label-free rule that treats objectives alike. Downstream and baseline test metrics use validation-selected downstream checkpoints. § 5.5 tests this rule.
Results
Core Classification Results
Table 1 reports all full-label classification cells. Under the capped linear probe, MAE has the highest mean AUPRC in all nine cells. This ordering holds in all 90 matched-seed comparisons against JEPA and contrastive learning. Mean paired advantages over the stronger SSL alternative range from .026 to .084 AUPRC (Appendix Figure 6). Under fine-tuning, the SSL ranking is less uniform and paired differences often cross zero: MAE leads seven cells, ties JEPA on Combined AKI (.626 each), and trails contrastive learning on MIMIC-IV 24-hour mortality. The supervised Transformer has the highest fine-tuned mean in all three AKI cells.
| Data | Endpoint | Probe, capped | Probe, converged | Fine-tune | Supervised baselines | ||||||||
| MAE | JEPA | Con. | MAE | JEPA | Con. | MAE | JEPA | Con. | Transf. | GRU-D | XGBoost | ||
| Combined | 24h mortality | 0.284 | 0.188 | 0.205 | 0.309 | 0.208 | 0.239 | 0.409 | 0.391 | 0.399 | 0.389 | 0.422 | 0.374 |
| Combined | Hospital mortality | 0.470 | 0.357 | 0.392 | 0.470 | 0.363 | 0.395 | 0.531 | 0.521 | 0.514 | 0.518 | 0.527 | 0.525 |
| Combined | AKI | 0.447 | 0.327 | 0.396 | 0.451 | 0.332 | 0.402 | 0.626 | 0.626 | 0.614 | 0.652 | 0.630 | 0.668 |
| eICU-CRD | 24h mortality | 0.276 | 0.199 | 0.195 | 0.287 | 0.219 | 0.215 | 0.377 | 0.363 | 0.360 | 0.351 | 0.397 | 0.315 |
| eICU-CRD | Hospital mortality | 0.439 | 0.355 | 0.350 | 0.439 | 0.359 | 0.356 | 0.495 | 0.471 | 0.471 | 0.474 | 0.484 | 0.482 |
| eICU-CRD | AKI | 0.361 | 0.268 | 0.286 | 0.366 | 0.275 | 0.293 | 0.519 | 0.515 | 0.494 | 0.554 | 0.510 | 0.569 |
| MIMIC-IV | 24h mortality | 0.312 | 0.236 | 0.254 | 0.357 | 0.298 | 0.322 | 0.451 | 0.451 | 0.456 | 0.449 | 0.480 | 0.402 |
| MIMIC-IV | Hospital mortality | 0.510 | 0.415 | 0.458 | 0.520 | 0.440 | 0.471 | 0.579 | 0.564 | 0.568 | 0.565 | 0.577 | 0.577 |
| MIMIC-IV | AKI | 0.504 | 0.410 | 0.478 | 0.510 | 0.433 | 0.492 | 0.667 | 0.642 | 0.645 | 0.693 | 0.661 | 0.729 |
Table 1. Core classification AUPRC, mean over five matched seeds. Capped probe: linear probe trained for at most 50 epochs. Converged probe: L2 logistic regression fitted to convergence on the same embeddings. Baselines are trained from scratch with full labels (Transf.: supervised Transformer). Bold marks the best mean per row within each probe block and across fine-tuning and baselines, ties at three decimals included. Appendix Tables 8 and 17 give SDs.
Probe optimization does not drive the frozen ranking (Table 1; Appendix C.§ 1). The converged probe scores higher in 132 of 135 runs, by .017 AUPRC on average. Gains are largest for 24-hour mortality, where 82% of capped probes stopped early on a low-prevalence validation metric, and .004–.012 per objective elsewhere, where 93–96% reached the epoch cap. The objective order is unchanged in all nine cells by AUPRC and AUROC. MAE beats both alternatives in all 90 matched-seed comparisons, with a mean paired advantage of .018–.081.
SSL Versus Supervised and Classical Baselines
At full labels, baselines lead six of the twelve cells (Table 1 and the fine-tune rows of Table 2; Appendix Table 7 gives the compact matrix). GRU-D has the highest 24-hour-mortality AUPRC and XGBoost the highest AKI AUPRC in all three regimes. Fine-tuned MAE leads hospital mortality in all three, narrowly (.531, .495, and .579 against at most .527, .484, and .577 on Combined, eICU-CRD, and MIMIC-IV). The best SSL model has the lowest LOS error in all three regimes, by .03–.06 days. The baselines share cohorts, splits, and metrics but differ in input summaries and inductive biases.
| Data | Probe/protocol | MAE | JEPA | Con. | Transf. | GRU-D | XGBoost | Const. |
| Combined | Probe, abs. | 2.193 (0.028) | 2.215 (0.027) | 2.200 (0.031) | – | – | – | 2.403 (0.025) |
| Combined | Probe, sq. | 2.469 (0.025) | 2.506 (0.024) | 2.486 (0.031) | – | – | – | 2.403 (0.025) |
| Combined | Fine-tune, sq. | 2.118 (0.029) | 2.135 (0.013) | 2.152 (0.024) | 2.190 (0.031) | 2.174 (0.031) | 2.340 (0.020) | 2.403 (0.025) |
| eICU-CRD | Probe, abs. | 1.992 (0.010) | 2.024 (0.011) | 2.018 (0.008) | – | – | – | 2.176 (0.006) |
| eICU-CRD | Probe, sq. | 2.228 (0.011) | 2.283 (0.015) | 2.272 (0.006) | – | – | – | 2.176 (0.006) |
| eICU-CRD | Fine-tune, sq. | 1.964 (0.017) | 1.977 (0.034) | 2.008 (0.031) | 1.997 (0.030) | 2.003 (0.018) | 2.163 (0.015) | 2.176 (0.006) |
| MIMIC-IV | Probe, abs. | 2.439 (0.062) | 2.420 (0.059) | 2.416 (0.062) | – | – | – | 2.733 (0.074) |
| MIMIC-IV | Probe, sq. | 2.774 (0.046) | 2.750 (0.048) | 2.757 (0.051) | – | – | – | 2.733 (0.074) |
| MIMIC-IV | Fine-tune, sq. | 2.368 (0.051) | 2.371 (0.069) | 2.359 (0.069) | 2.397 (0.066) | 2.394 (0.046) | 2.613 (0.038) | 2.733 (0.074) |
Table 2. Remaining-LOS absolute error in days, mean (sample SD), ; lower is better. Probe rows are linear probes on frozen final-epoch embeddings, trained with an absolute-error-type Huber loss (abs.) or ridge squared error (sq.). Fine-tune rows are the reference squared-error runs beside full-label baselines (Transf.: supervised Transformer). Const. predicts each seed’s training-label median. Bold marks the lowest learned-model mean per row; in sq. probe rows the constant is lower.
Length of Stay and Loss Matching
The capped LOS probes trained with squared error but were selected and evaluated on absolute error. The target is right-skewed, with training means of 2.7–3.4 days and medians of 1.3–1.5 days. Absolute error therefore favors the training-median constant, while a squared-error model estimates the conditional mean. With an absolute-error loss, all three objectives beat the median constant in 5/5 seeds per regime, by 7–12% (Table 2). Ridge probes lose to this constant on absolute error but beat a training-mean constant on squared error in 5/5 seeds. Objective differences are small: MAE is lowest on eICU-CRD and Combined, and the three are within .023 days on MIMIC-IV, where MAE is last in 5/5 seeds. The capped-probe ordering, in which contrastive learning was lowest and frozen MAE was above the constant (Appendix Table 18), does not survive loss matching. Its contrastive heads came from the first few epochs. Fine-tuning still trains with squared error and beats the constant in every regime, but was not rerun with an absolute-error loss.
Label Efficiency
Does pretraining help when labels are scarce? We reduce only the training labels, to 1–50% for hospital mortality, 5–50% for 24-hour mortality, and 10% for AKI and LOS, giving 33 reduced-label cells per protocol. Among the three SSL implementations, MAE leads 30/33 fine-tuning and 28/33 capped-probe means (28/30 classification cells under each protocol), and 28/33 fine-tuning means with the supervised Transformer included. For hospital mortality, fine-tuned MAE exceeds the supervised Transformer at every fraction in every regime, by .033 (paired SD .012) at 1% labels on Combined. No single fraction marks where pretraining stops paying off: for MIMIC-IV 24-hour mortality, differences above 5% labels change sign, and at 5% the unstratified subsets have 31–43 training positives per seed. Reduced-label GRU-D and XGBoost runs are not compared. Appendix Figure 7 and Table 10 give all fractions.
Figure 7. Hospital-mortality label efficiency under full fine-tuning, including the supervised Transformer at each label fraction. Points are five-seed means. Bars are sample SD across seeds. Only training labels are reduced.
| Data | Task | 1% | 5% | 10% | 25% | 50% | 100% |
| Combined | 24h mortality | – | 0.010 | 0.021 | 0.033 | 0.020 | 0.020 |
| Combined | Hospital mortality | 0.033 | 0.036 | 0.036 | 0.028 | 0.016 | 0.012 |
| Combined | AKI | – | – | 0.006 | – | – | -0.026 |
| Combined | Remaining LOS | – | – | -0.023 | – | – | -0.072 |
| eICU-CRD | 24h mortality | – | 0.011 | 0.010 | 0.026 | 0.035 | 0.026 |
| eICU-CRD | Hospital mortality | 0.029 | 0.032 | 0.022 | 0.024 | 0.020 | 0.021 |
| eICU-CRD | AKI | – | – | -0.006 | – | – | -0.034 |
| eICU-CRD | Remaining LOS | – | – | -0.014 | – | – | -0.032 |
| MIMIC-IV | 24h mortality | – | 0.026 | -0.020 | -0.016 | -0.008 | 0.002 |
| MIMIC-IV | Hospital mortality | 0.008 | 0.021 | 0.014 | 0.017 | 0.015 | 0.014 |
| MIMIC-IV | AKI | – | – | 0.016 | – | – | -0.026 |
| MIMIC-IV | Remaining LOS | – | – | 0.004 | – | – | -0.029 |
Table 10. Mean paired difference between fine-tuned MAE and the supervised Transformer at each label fraction, five seeds. Classification uses AUPRC, where positive favors MAE. LOS uses absolute error in days, where negative favors MAE. Dashes mark fractions that were not run.
Secondary Diagnostics
Sources and databases. Scored separately on the MIMIC-IV and eICU-CRD portions of the Combined test split, converged MAE probes lead both alternatives on both portions for all classification tasks in 5/5 seeds. Fine-tuned MAE leads five of six portion–task means among SSL objectives, and supervised training leads AKI on both portions. Pooling hides a difference in absolute level, with higher AUPRC on the MIMIC-IV portion, not a reversal (Appendix C.§ 2). With source pretraining and target-label fine-tuning, MAE leads or ties all four MIMIC-IV-to-eICU-CRD tasks and leads three reverse tasks, and stays within .004–.010 AUPRC of target pretraining for mortality (Appendix B.§ 3). This is adaptation with target labels, not out-of-domain validation (§ 7).
Compute. The 45 core pretraining runs took an estimated 55–63 L4-hours. JEPA costs the most on every GPU measure, with 2.2 times the forward FLOPs of MAE and 1.5 times its idle-GPU step time, yet ranks last in 7 of 9 frozen classification cells. MAE uses the fewest FLOPs, while a contrastive step takes 18% less time on an idle GPU. Cost alone therefore does not explain the ranking, although we ran no compute-matched ablation (Appendix C.§ 3).
Checkpoint selection. Selecting MIMIC-IV encoders on SSL validation loss instead of the final epoch worsens JEPA capped probes on hospital mortality and LOS by .041 AUPRC and .26 days, changes MAE and contrastive learning by at most .001, and leaves both leaders unchanged (Appendix B.§ 8).
Missingness. A linear control on MIMIC-IV hospital mortality reaches .301 (.025) AUPRC from masks alone and .404 (.013) from values plus masks, evidence of input signal, not an optimized baseline (Appendix B.§ 9).
Other diagnostics. Medium and large MAE encoders beat matched supervised models on MIMIC-IV 24-hour mortality at three label fractions, and a random encoder trails SSL probes in 118/120 seed-level comparisons. Appendices B.4–B.7 add the TS2Vec-style extension and subgroup audits.
Discussion
What can be reused. Researchers can audit run aggregates or add an objective while retaining clinical inputs, encoder transfer, and evaluation. A new objective then faces two references: the three SSL implementations and the supervised, recurrent, and tabular baselines, which lead half of the full-label cells.
Why reconstruction may fit the interface. The tensor exposes a value and observation mask at each hour. MAE reconstructs observed cells, JEPA predicts an evolving target embedding, and core contrastive learning compares pooled stay vectors. An interface preference for reconstruction is one hypothesis for the classification ordering. The controls do not identify the learned information or establish that continuous values are better than quantization.
Interpreting missingness. Measurement patterns carry outcome signal: masks alone give a linear model .301 AUPRC for MIMIC-IV hospital mortality at 11.7% prevalence. The signal is ambiguous, because an unmeasured laboratory value may reflect clinical judgment, workflow, availability, or data loss, so a model can learn site-specific measurement policy, not only physiology. Preserving the mask avoids treating missing cells as observed zeros but does not make missingness causal or transportable. The frozen classification ranking held on both source portions of Combined, which does not show transfer to a new hospital.
Evaluation practice. Both probes and fine-tuning remain necessary because they test different representation properties, and baselines show whether an SSL ordering matters for task performance. Frozen probes should be trained to convergence, and regression probes should use a loss matched to the reported metric. Checkpoint and cost policies must be explicit, because equal epochs do not imply equal compute and checkpoint selection affected JEPA.
Limitations
The benchmark holds clinical inputs and downstream evaluation fixed while retaining the components needed by each objective. Objective-specific heads, target networks, encoder passes, and trainable parameter counts therefore differ, so results describe these implementations, not objective families. Configurations reflect unequal development histories and a small targeted sensitivity analysis, not matched compute or tuning budgets or objective-specific optima, and the benchmark was not preregistered.
No out-of-domain validation. No model is evaluated on a database or hospital without training on its labels. Cross-database analyses fine-tune on target labels, and per-source results stay within the pooled regime. The results therefore do not show that rankings or predictors hold in a new hospital, for instance one with different ordering practices. Such a test needs source normalization applied to target data, which the benchmark does not yet provide. Combined is a union, not an external cohort, and both databases are from the US, retrospective, and deidentified.
Representation and architecture. Rankings may change with observation-triplet, quantized, event-token, or patch representations, which also change the masking unit. Waveforms, notes, images, and full medication histories are out of scope. All objectives use one small Transformer. SSL on decay-based recurrent encoders such as GRU-D is the first architecture extension, since masked hours must then be skipped or fed as missing. Shared masked views may underrepresent contrastive methods with clinically defined positive pairs.
Statistics and labels. All comparisons are descriptive. Five matched seeds are few and reuse overlapping source populations. They describe split and optimization variation but do not establish population uncertainty, statistical equivalence, calibration quality, subgroup fairness, clinical utility, or deployment readiness. Reduced-label subsets are not class-stratified, and reduced-label GRU-D and XGBoost runs are not compared. The AKI label uses creatinine only. Fine-tuned LOS models train with squared error. Final-epoch checkpoints may miss better earlier states. The random-encoder, raw/mask, and post-review diagnostics are post hoc.
Ethics, Access, and Reproducibility
The study analyzes retrospective, deidentified ICU records under the credentialing and data-use terms of PhysioNet. It involves no prospective intervention, patient contact, or automated clinical decision. Benchmark results come from the unchanged archived run aggregates; diagnostics are stored separately.
We reran a MIMIC-IV slice with the maintained framework, starting from the processed tensors (Appendix C.§ 4). From archived reference encoders, 12 frozen probes reproduce every reference test metric to numerical precision, and 12 fine-tuning runs lie within one across-seed SD. The full pipeline, with 6 pretraining and 48 downstream runs over two seeds, differs from the reference by .008 mean absolute AUPRC for probes and .010 for fine-tuning, all within two SDs, and keeps the objective order in 13 of 16 cells. LOS fine-tuning error was higher in all nine comparisons, by .03–.04 days on average, for reasons we have not identified, and other databases and seeds were not rerun.
Conclusion
SLICES provides a fixed sparse-ICU benchmark specification, interchangeable SSL objectives, shared downstream evaluation, and auditable reference results. At full labels, baselines lead half of the cells: GRU-D leads 24-hour mortality and XGBoost leads AKI in every regime, while fine-tuned SSL leads hospital mortality narrowly and LOS. Among the SSL objectives, MAE has the highest mean AUPRC in all nine frozen cells under both probes and in 28 of 30 reduced-label fine-tuning classification cells, while LOS differences are small once the probe loss matches the metric. Generalization to unseen databases remains untested. Future work will add an out-of-domain protocol, make converged and loss-matched probes framework protocols, and test the objectives on recurrent encoders such as GRU-D, on other input representations, and under matched compute.
Acknowledgments
We thank the anonymous reviewers and the area chair for comments that led to the converged-probe, loss-matched LOS, per-source, compute, and reproduction analyses. LAC is funded by the National Institutes of Health through DS-I Africa U54 TW012043-01 and Bridge2AI OT2OD032701, the National Science Foundation through ITEST #2148451, a grant of the Boston-Korea Innovative Research Project (RS-2024-00403047), and a grant of the Korea Health Technology R&D Project (RS-2024-00439677) through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea.
Additional Benchmark Details
Benchmark and Data Flow
Figure 3 details the preprocessing path introduced in Figure 1. Combined pools the two source databases. It is not an external cohort. The varying unit includes objective-specific masks, auxiliary modules, and compute.
Figure 3. Data pipeline. RICU supplies concept-specific harmonization and hourly aggregation. SLICES applies cohort and feature filters and forms values, masks, and labels. Patient assignment precedes task-specific null-label filtering. Normalization uses observed training cells, followed by zero filling with the observation mask retained. Event rows are illustrative.
Downstream Protocols
Figure 5 separates frozen probing from full fine-tuning. Both use the same final pretraining encoder but differ in trainable scope, head capacity, learning rate, and epoch cap. These are two evaluation protocols, not a freeze-only ablation. The separate checkpoint diagnostic changes the SSL encoder selection policy only within its stated scope.
Figure 5. Reference downstream protocols. The displayed 50 and 100 epochs are maximum budgets. Validation patience is ten epochs. Protocol A (frozen probing) trains a linear head. Protocol B (full fine-tuning) adapts the encoder and a width-64 MLP head. Validation selects each downstream checkpoint before one test evaluation. Both start from final-epoch SSL weights.
Task Definitions and Test Support
Table 3 and Table 4 give the target rules and exported test support. Twenty-four-hour mortality and remaining LOS are fixed-window analogues of the decompensation and length-of-stay tasks of Harutyunyan et al. [8], and mortality, AKI, and LOS appear in YAIB van de Water et al., [21]. The four endpoints span 1.6%–20.3% prevalence, two horizons, a laboratory-defined label with structured missing labels, and a continuous target. The SSL ordering is uniform for frozen classification but not for fine-tuning or against supervised and classical baselines, so these differences matter for the comparison.
| Task | Horizon after input | Target | Null-label rule | Primary metric |
| 24h mortality | hours 24–48 | death in forward day | death in hours 0–24 | AUPRC |
| Hosp. mortality | after hour 24 | death before discharge | death in hours 0–24 | AUPRC |
| AKI | hours 24–48 | creatinine-defined KDIGO stage | missing baseline/forward evidence | AUPRC |
| Remaining LOS | after hour 24 | ICU LOS minus one day, clipped at 0 | none | Absolute error (days) |
Table 3. Downstream task contract. All tasks use hours 0–24 as model input.
Objective and Downstream Configuration
Development and test-set use. Every downstream run reports test metrics automatically, and the pipeline was run end to end a few times before the final rerun. Those earlier passes exposed defects that were fixed: an AKI label that overlapped the input window (detected through test AUROC above .97 for every method), mislabeled boundary cases in the mortality labels, and input features derived from future treatment durations. The test metrics we saw therefore came largely from earlier benchmark versions with different labels or inputs. Pretraining does not read the test partition, checkpoints are selected on validation metrics, and each run scores the test partition once after selection. The recorded objective-specific changes responded to label-free pretraining diagnostics: a rising JEPA loss with low representation spread, and a contrastive pretext task that was solved almost perfectly. Settings shaped by downstream comparisons are shared by all three objectives: input representation, encoder, one pretraining learning rate, one mask ratio, and one downstream recipe that the supervised Transformer also uses. Across the tested learning rates and mask ratios, mean test AUPRC moves by at most .018 within a sweep and .020 within an objective (Table 16).
Table 16: Targeted pretraining hyperparameter diagnostics. Downstream AUPRC mean and sample SD, per level.
| Sweep | Objective | Level | AUPRC | SD |
| lr sensitivity | Contrastive | 0.0002 | 0.462 | 0.035 |
| lr sensitivity | Contrastive | 0.0005 | 0.465 | 0.033 |
| lr sensitivity | Contrastive | 0.002 | 0.459 | 0.042 |
| lr sensitivity | JEPA | 0.0002 | 0.467 | 0.028 |
| lr sensitivity | JEPA | 0.0005 | 0.461 | 0.049 |
| lr sensitivity | JEPA | 0.002 | 0.449 | 0.053 |
| lr sensitivity | MAE | 0.0002 | 0.462 | 0.047 |
| lr sensitivity | MAE | 0.0005 | 0.451 | 0.051 |
| lr sensitivity | MAE | 0.002 | 0.457 | 0.050 |
| mask ratio sensitivity | JEPA | 0.3 | 0.454 | 0.057 |
| mask ratio sensitivity | JEPA | 0.75 | 0.447 | 0.029 |
| mask ratio sensitivity | MAE | 0.3 | 0.464 | 0.033 |
| mask ratio sensitivity | MAE | 0.75 | 0.447 | 0.035 |
| view mask sensitivity | Contrastive | 0.3 | 0.460 | 0.035 |
| view mask sensitivity | Contrastive | 0.75 | 0.447 | 0.061 |
Table 16.
The JEPA target encoder adds a non-trainable 159,808-parameter copy. Its EMA coefficient increases linearly from .999 to 1.0. The predictor has width 32, two layers, four heads, and feed-forward width 128. The contrastive projection has hidden width 256 and output width 64. All auxiliary modules are discarded before downstream evaluation.
Retained Features and Preprocessing
The ordered 84 RICU concept identifiers below define the model schema, not a claim that each is observed in both sources. Source-specific missingness remains explicit. Extraction uses RICU 0.6.3 concept definitions and unit conversions, requested at one-hour intervals. Concept-specific aggregation is inherited from RICU. Python tensorization neither quantizes values nor forward-fills across hours. It uses the observed training-partition mean and sample SD per feature. SD is floored at . Features absent in training retain a zero mean and unit SD. Missing values become zero after normalization. Low-label subsets retain the full training-partition normalizer. The code repository includes extraction scripts and the complete input blocklist, including direct LOS, future vasopressor-duration, and derived score features.
abx, adh_rate, alb, alp, alt, ast, avpu, basos, be, bicar, bili, bilir, bnd, bun, ca, cai, ck, ckmb, c1, cort, crea, crp, dbp, dex, dobu_rate, dopa_rate, egcs, eos, epi_rate, esr, etco2, etc-gcs, fgn, fio2, glu, hba1c, hbco, hct, hgb, hr, inr-pt, ins, k, lact, lymph, map, mch, mchc, mcv, mech-vent, methb, mg, mgcs, na, neut, norepi_rate, o2sat, pco2, ph, phn_rate, phos, plt, po2, pt, ptt, rass, rbc, rdw, resp, samp, sao2, sbp, spo2, supp_o2, tco2, temp, tgcs, tnt, tri, urine, vent_end, vent_start, vgcs, wbc.
Exact Losses and Baseline Configuration
MAE minimizes squared reconstruction error averaged over observed feature cells at masked non-empty
hours. JEPA uses elementwise squared error between predictions and layer-normalized stop-gradient target embeddings, averaged across valid target dimensions/hours. Its EMA coefficient rises from .999 to 1.0. EMA and target normalization are not evidence of collapse prevention. Core contrastive learning uses cosine similarity between normalized projected stay vectors. Each of the view embeddings has its paired view as positive and the other non-self batch views in an NT-Xent denominator, at temperature .07. Losses differ in target space and reduction unit.
Binary mortality and AKI use two-logit cross entropy with label smoothing .1 and class weights from the labeled training subset. They are classification tasks, not time-to-event regression. LOS uses untransformed remaining days, squared-error training loss, and absolute-error validation/test evaluation. The constant median is the empirical training-label median evaluated on the test labels. It is not fitted on the test set.
GRU-D has hidden width 64, one recurrent layer, dropout .1, learned input/hidden decay, and the full-fine-tuning head and schedule. XGBoost receives eight summaries per variable: observed mean, SD, minimum, maximum, first, last, count, and fraction, for 672 inputs. It uses at most 1,000 trees, depth 6, learning rate .1, row/column subsampling .8, minimum child weight 1, and validation patience 20. Classification uses the native negative/positive class ratio and validation aucpr. Regression uses the squared-error objective and validation mae. These differ from neural weighting and average-precision implementation. All test AUPRC values use the shared evaluation definition. Exported per-run scientific configurations, rather than current defaults, are the reproduction reference.
Full Baseline Matrix
Table 7 reports all full-label contextual baseline comparisons.
Secondary Analyses
Classification Results with Seed Variation
Table 8 gives sample SDs for the capped probe and fine-tuning, and Table 17 (Appendix C.1) gives them for the converged probe. Figure 6 shows matched-seed paired differences, MAE minus each comparator.
Label Efficiency
Only the labeled training subset is reduced. Validation and test partitions remain unchanged. Each seed draws its subsets as prefixes of one random permutation of the labeled training stays, keeping of stays at fraction . Subsets are therefore nested across fractions within a seed and are not class-stratified. Hospital mortality uses 1%, 5%, 10%, 25%, 50%, and the full-label reference. Twenty-four-hour mortality starts at 5%. AKI and LOS use 10% and full-label references. Each cell contains five seeds.
At the lowest reported fraction for Combined hospital mortality (1%), fine-tuned AUPRC means are .419 for MAE, .341 for JEPA, .377 for contrastive learning, and .386 for supervised training. At 5% labels for 24-hour mortality, the corresponding means are .277, .241, .245, and .267. When full-label references are included, MAE leads 39/45 fine-tuning cells among SSL methods, 35/45 with supervised included, and 37/39 classification probe cells among SSL methods (LOS probe cells use the superseded capped probe). Section 5 instead counts only fractions below 100%. Comparator sets and denominators must be stated together.
Figure 7 shows hospital mortality. Table 10 extends the comparison to every label fraction. For hospital mortality, fine-tuned MAE exceeds supervised training at every fraction in every regime. The difference is largest at low fractions on Combined and eICU-CRD and does not vanish at full labels. For 24-hour mortality it is also positive at every fraction on eICU-CRD and Combined. On MIMIC-IV, the smallest cohort, the 24-hour-mortality differences above 5% labels change sign. For AKI, supervised training leads at full labels in all three regimes. The
| Data | Endpoint | GRU-D | XGBoost | Supervised | Best SSL | SSL method |
| Combined | 24h mortality | 0.422 | 0.374 | 0.389 | 0.409 | MAE |
| Combined | Hospital mortality | 0.527 | 0.525 | 0.518 | 0.531 | MAE |
| Combined | AKI | 0.630 | 0.668 | 0.652 | 0.626 | JEPA |
| Combined | Remaining LOS | 2.174 | 2.340 | 2.190 | 2.118 | MAE |
| eICU-CRD | 24h mortality | 0.397 | 0.315 | 0.351 | 0.377 | MAE |
| eICU-CRD | Hospital mortality | 0.484 | 0.482 | 0.474 | 0.495 | MAE |
| eICU-CRD | AKI | 0.510 | 0.569 | 0.554 | 0.519 | MAE |
| eICU-CRD | Remaining LOS | 2.003 | 2.163 | 1.997 | 1.964 | MAE |
| MIMIC-IV | 24h mortality | 0.480 | 0.402 | 0.449 | 0.456 | Contrastive |
| MIMIC-IV | Hospital mortality | 0.577 | 0.577 | 0.565 | 0.579 | MAE |
| MIMIC-IV | AKI | 0.661 | 0.729 | 0.693 | 0.667 | MAE |
| MIMIC-IV | Remaining LOS | 2.394 | 2.613 | 2.397 | 2.359 | Contrastive |
Table 7. Full-data contextual baselines and the best SSL mean. Classification rows report AUPRC (higher is better). LOS rows report absolute error in days (lower is better). Bold marks the best unrounded mean per row. All entries average five seeds.
| Data | Protocol | Endpoint | MAE | JEPA | Contrastive | Supervised |
| Combined | Fine-tune | 24h mortality | 0.409 (0.024) | 0.391 (0.031) | 0.399 (0.029) | 0.389 (0.021) |
| Combined | Fine-tune | Hospital mortality | 0.531 (0.010) | 0.521 (0.009) | 0.514 (0.016) | 0.518 (0.008) |
| Combined | Fine-tune | AKI | 0.626 (0.004) | 0.626 (0.005) | 0.614 (0.004) | 0.652 (0.003) |
| Combined | Probe | 24h mortality | 0.284 (0.031) | 0.188 (0.018) | 0.205 (0.016) | – |
| Combined | Probe | Hospital mortality | 0.470 (0.013) | 0.357 (0.006) | 0.392 (0.010) | – |
| Combined | Probe | AKI | 0.447 (0.006) | 0.327 (0.005) | 0.396 (0.005) | – |
| eICU-CRD | Fine-tune | 24h mortality | 0.377 (0.013) | 0.363 (0.008) | 0.360 (0.016) | 0.351 (0.022) |
| eICU-CRD | Fine-tune | Hospital mortality | 0.495 (0.005) | 0.471 (0.007) | 0.471 (0.003) | 0.474 (0.004) |
| eICU-CRD | Fine-tune | AKI | 0.519 (0.017) | 0.515 (0.018) | 0.494 (0.012) | 0.554 (0.014) |
| eICU-CRD | Probe | 24h mortality | 0.276 (0.015) | 0.199 (0.017) | 0.195 (0.009) | – |
| eICU-CRD | Probe | Hospital mortality | 0.439 (0.006) | 0.355 (0.004) | 0.350 (0.010) | – |
| eICU-CRD | Probe | AKI | 0.361 (0.009) | 0.268 (0.008) | 0.286 (0.004) | – |
| MIMIC-IV | Fine-tune | 24h mortality | 0.451 (0.045) | 0.451 (0.050) | 0.456 (0.042) | 0.449 (0.032) |
| MIMIC-IV | Fine-tune | Hospital mortality | 0.579 (0.011) | 0.564 (0.009) | 0.568 (0.011) | 0.565 (0.008) |
| MIMIC-IV | Fine-tune | AKI | 0.667 (0.009) | 0.642 (0.009) | 0.645 (0.009) | 0.693 (0.013) |
| MIMIC-IV | Probe | 24h mortality | 0.312 (0.042) | 0.236 (0.028) | 0.254 (0.035) | – |
| MIMIC-IV | Probe | Hospital mortality | 0.510 (0.014) | 0.415 (0.020) | 0.458 (0.023) | – |
| MIMIC-IV | Probe | AKI | 0.504 (0.009) | 0.410 (0.017) | 0.478 (0.009) | – |
Table 8. Core classification AUPRC, mean (sample SD) over five matched seeds. Probe rows use the capped linear probe. Bold marks the highest unrounded mean in each row. The supervised Transformer is defined only for full fine-tuning.
Figure 6. Matched-seed classification AUPRC differences, MAE minus comparator. Positive values favor MAE. Points are mean paired differences. Bars span the five observed differences, not a confidence interval. Supervised appears only for full fine-tuning.
answer to when pretraining stops paying off therefore depends on endpoint and cohort more than on the label fraction.
Because subsets are drawn without class stratification, their positive counts vary by seed (Table 11). The counts were recomputed after review from the same split and subset rules. The same rules reproduce the exported test support exactly. MIMIC-IV 24-hour mortality at 5% labels has 31–43 training positives and also the largest AUPRC paired-difference SD in Table 9.
| Data | Task | Labels | MAE | Supervised | mean | SD |
| Combined | 24h mortality | 5% | 0.277 | 0.267 | 0.010 | 0.027 |
| Combined | Hospital mortality | 1% | 0.419 | 0.386 | 0.033 | 0.012 |
| Combined | AKI | 10% | 0.514 | 0.507 | 0.006 | 0.021 |
| Combined | Remaining LOS | 10% | 2.229 | 2.252 | -0.023 | 0.024 |
| eICU-CRD | 24h mortality | 5% | 0.260 | 0.248 | 0.011 | 0.015 |
| eICU-CRD | Hospital mortality | 1% | 0.385 | 0.355 | 0.029 | 0.019 |
| eICU-CRD | AKI | 10% | 0.405 | 0.411 | -0.006 | 0.009 |
| eICU-CRD | Remaining LOS | 10% | 2.067 | 2.081 | -0.014 | 0.034 |
| MIMIC-IV | 24h mortality | 5% | 0.294 | 0.268 | 0.026 | 0.041 |
| MIMIC-IV | Hospital mortality | 1% | 0.446 | 0.439 | 0.008 | 0.021 |
| MIMIC-IV | AKI | 10% | 0.565 | 0.549 | 0.016 | 0.012 |
| MIMIC-IV | Remaining LOS | 10% | 2.494 | 2.490 | 0.004 | 0.071 |
Table 9. Lowest exported label fractions, full fine-tuning. MAE and supervised means use AUPRC for classification and absolute error in days for LOS. is MAE minus supervised. Positive favors MAE only for classification. SD is computed from five paired differences, not from independent error propagation.
| Data | Task | Labels | 42 | 123 | 456 | 789 | 1011 | Mean | Labeled stays |
| Combined | 24h mortality | 5% | 144 | 117 | 110 | 118 | 116 | 121.0 | 7,214.0 |
| Combined | 24h mortality | 10% | 263 | 241 | 219 | 218 | 236 | 235.4 | 14,428.6 |
| Combined | 24h mortality | 25% | 594 | 605 | 580 | 560 | 586 | 585.0 | 36,072.2 |
| Combined | 24h mortality | 50% | 1,124 | 1,197 | 1,143 | 1,131 | 1,139 | 1,146.8 | 72,144.4 |
| Combined | Hospital mortality | 1% | 144 | 164 | 150 | 159 | 150 | 153.4 | 1,442.6 |
| Combined | Hospital mortality | 5% | 704 | 765 | 756 | 737 | 701 | 732.6 | 7,215.6 |
| Combined | Hospital mortality | 10% | 1,427 | 1,457 | 1,489 | 1,511 | 1,444 | 1,465.6 | 14,431.8 |
| Combined | Hospital mortality | 25% | 3,635 | 3,687 | 3,641 | 3,766 | 3,622 | 3,670.2 | 36,080.0 |
| Combined | Hospital mortality | 50% | 7,236 | 7,366 | 7,363 | 7,508 | 7,304 | 7,355.4 | 72,160.4 |
| Combined | AKI | 10% | 1,625 | 1,656 | 1,600 | 1,685 | 1,724 | 1,658.0 | 10,126.6 |
| eICU-CRD | 24h mortality | 5% | 69 | 57 | 61 | 68 | 71 | 65.2 | 4,600.6 |
| eICU-CRD | 24h mortality | 10% | 139 | 140 | 143 | 125 | 137 | 136.8 | 9,201.8 |
| eICU-CRD | 24h mortality | 25% | 387 | 367 | 380 | 357 | 365 | 371.2 | 23,005.2 |
| eICU-CRD | 24h mortality | 50% | 768 | 746 | 722 | 751 | 700 | 737.4 | 46,011.2 |
| eICU-CRD | Hospital mortality | 1% | 80 | 85 | 95 | 78 | 74 | 82.4 | 919.8 |
| eICU-CRD | Hospital mortality | 5% | 434 | 433 | 424 | 388 | 412 | 418.2 | 4,600.6 |
| eICU-CRD | Hospital mortality | 10% | 840 | 888 | 892 | 797 | 827 | 848.8 | 9,201.8 |
| eICU-CRD | Hospital mortality | 25% | 1,987 | 2,145 | 2,153 | 2,069 | 2,081 | 2,087.0 | 23,005.2 |
| eICU-CRD | Hospital mortality | 50% | 4,137 | 4,234 | 4,233 | 4,217 | 4,143 | 4,192.8 | 46,011.2 |
| eICU-CRD | AKI | 10% | 849 | 818 | 837 | 852 | 837 | 838.6 | 6,171.4 |
| MIMIC-IV | 24h mortality | 5% | 31 | 34 | 43 | 40 | 37 | 37.0 | 2,608.6 |
| MIMIC-IV | 24h mortality | 10% | 76 | 79 | 72 | 86 | 85 | 79.6 | 5,217.7 |
| MIMIC-IV | 24h mortality | 25% | 216 | 199 | 190 | 207 | 202 | 202.8 | 13,044.2 |
| MIMIC-IV | 24h mortality | 50% | 433 | 386 | 396 | 417 | 425 | 411.4 | 26,088.2 |
| MIMIC-IV | Hospital mortality | 1% | 61 | 56 | 69 | 68 | 74 | 65.6 | 521.8 |
| MIMIC-IV | Hospital mortality | 5% | 292 | 297 | 307 | 311 | 293 | 300.0 | 2,609.8 |
| MIMIC-IV | Hospital mortality | 10% | 633 | 608 | 624 | 610 | 608 | 616.6 | 5,220.2 |
| MIMIC-IV | Hospital mortality | 25% | 1,582 | 1,532 | 1,548 | 1,522 | 1,463 | 1,529.4 | 13,052.0 |
| MIMIC-IV | Hospital mortality | 50% | 3,102 | 3,116 | 3,090 | 3,029 | 3,020 | 3,071.4 | 26,104.4 |
| MIMIC-IV | AKI | 10% | 787 | 804 | 785 | 819 | 761 | 791.2 | 3,951.6 |
Table 11. Positive training labels in each reduced-label subset, by seed, with the mean number of labeled training stays. Only the labeled training subset is reduced. Subsets are drawn per seed without class stratification. Counts were recomputed after review.
Cross-Database Adaptation
Cross-database rows pretrain on one database and then fully fine-tune using labels from the other. They test whether a representation pretrained under one site’s measurement practice can be adapted with target labels. They are not out-of-domain validation: no model is evaluated on a database without training on its labels. MAE has the best mean primary metric on three of four MIMIC-IV-to-eICU-CRD tasks and ties contrastive learning on the fourth, AKI (.532 each). It leads three of four reverse-direction tasks, and contrastive learning leads reverse-direction 24-hour mortality (Table 12).
| Pretrain | Target | Task | Metric | MAE | JEPA | Contrastive |
| eICU-CRD | MIMIC-IV | 24h mortality | AUPRC | 0.447 | 0.453 | 0.454 |
| eICU-CRD | MIMIC-IV | Hospital mortality | AUPRC | 0.572 | 0.559 | 0.564 |
| eICU-CRD | MIMIC-IV | AKI | AUPRC | 0.646 | 0.643 | 0.627 |
| eICU-CRD | MIMIC-IV | Remaining LOS | Abs. error | 2.389 | 2.390 | 2.403 |
| MIMIC-IV | eICU-CRD | 24h mortality | AUPRC | 0.372 | 0.367 | 0.354 |
| MIMIC-IV | eICU-CRD | Hospital mortality | AUPRC | 0.485 | 0.477 | 0.476 |
| MIMIC-IV | eICU-CRD | AKI | AUPRC | 0.532 | 0.520 | 0.532 |
| MIMIC-IV | eICU-CRD | Remaining LOS | Abs. error | 1.966 | 1.987 | 1.998 |
Table 12. Cross-database full-fine-tuning means over five seeds. Classification uses AUPRC. LOS uses absolute error in days. Bold marks the best mean within each row, and every mean that ties it at three decimals.
Table 13 places cross-database MAE beside MAE pretrained on the target database and the supervised Transformer trained on the target. These comparators come from the core comparison with matched seeds. For mortality, cross-database encoders stay within .004–.010 AUPRC of target-database pretraining. For AKI, supervised training leads regardless of pretraining source. These are five-seed means, not tests of equivalence, and do not establish a general transfer advantage.
| Target | Task | Metric | Other-DB pretraining | Target-DB pretraining | Supervised |
| eICU-CRD | 24h mortality | AUPRC | 0.372 | 0.377 | 0.351 |
| eICU-CRD | Hospital mortality | AUPRC | 0.485 | 0.495 | 0.474 |
| eICU-CRD | AKI | AUPRC | 0.532 | 0.519 | 0.554 |
| eICU-CRD | Remaining LOS | Abs. error | 1.966 | 1.964 | 1.997 |
| MIMIC-IV | 24h mortality | AUPRC | 0.447 | 0.451 | 0.449 |
| MIMIC-IV | Hospital mortality | AUPRC | 0.572 | 0.579 | 0.565 |
| MIMIC-IV | AKI | AUPRC | 0.646 | 0.667 | 0.693 |
| MIMIC-IV | Remaining LOS | Abs. error | 2.389 | 2.368 | 2.397 |
Table 13. Fine-tuned MAE on each target database, by pretraining source, beside the supervised Transformer trained from scratch on the target. Means over five seeds. Classification uses AUPRC. LOS uses absolute error in days. Bold marks the best mean in each row.
Encoder Capacity and Subgroup Auditing
The capacity analysis compares MAE and supervised Transformers on MIMIC-IV 24-hour mortality at 5%, 10%, and 50% labels. Under full fine-tuning, medium and large MAE models exceed their corresponding supervised models by .013–.028 mean AUPRC across the six size–fraction cells, each with five seeds. This one-task, one-database slice does not test larger JEPA or contrastive encoders and cannot establish a general scaling rule.
Subgroup auditing records maximum–minimum performance gaps across eligible gender, age, and race groups, requiring at least 50 unique patients and a defined metric per group. In the core full-label classification cells, across-seed mean AUPRC gaps range from .004 to .095 for gender, .033 to .172 for age, and .080 to .513 for race. These ranges cover 63 gender cells, 63 age cells, and 42 race cells. Race metrics are unavailable for eICU-CRD, and Combined race analyses use MIMIC-IV-origin records. A 50-patient threshold does not ensure enough positive outcomes for stable AUPRC. These summaries describe gaps without ranking methods by fairness or attributing their causes. They do not assess intersectional groups or subgroup calibration.
TS2Vec Context
Our TS2Vec-style extension retains the SLICES Transformer and value–mask interface, rather than the original dilated convolutional encoder. It adds Gaussian noise of SD .01 to observed values, independently masks half the hours in each view, and uses no temporal crop in the reference configuration. A width-256/output-64 projection precedes temperature-.05 NT-Xent over overlapping timestep pairs. Negatives combine other times and other instances in one denominator. This differs from original TS2Vec’s separate temporal and instance losses. Four max-pooling scales (1, 2, 4, 8 hours) use mask-aware validity and an average over active scales. Thus these results characterize this extension, not original TS2Vec’s performance [26].
Table 14 counts the cells each implementation wins. The full-fine-tuning split is task dependent. For example, TS2Vec is higher on MIMIC-IV 24-hour mortality (.465 versus .456), while core contrastive is lower on MIMIC-IV LOS (2.359 versus 2.370). Under the capped probe, the core contrastive mean is better in every cell, including MIMIC-IV AKI (.478 versus .441) and LOS (2.592 versus 2.722). The TS2Vec-style encoders were not refit with converged or loss-matched probes. These five-seed differences do not establish equivalence or general dominance.
Demonstrated Extension Workflow
The code repository’s TS2VEC_CASE.md gives a runnable path through the existing extension. Objective code, its configuration dataclass, registry entries, encoder-compatibility entry, and SSL YAML define the replaceable components. Clinical preprocessing, feature order, the shared encoder, final-epoch transfer, task heads, protocol schedules, primary metrics, and result identities define the retained scientific contract. The case adds execution and reporting code. It does not alter the objective or the archived results.
The command extension.py --out extension-output uses 80 artificial patients, the shared 70/15/15 split function, train-only normalization, and the full 159,808-parameter encoder. It runs two SSL epochs and two epochs for each synthetic binary/regression task under each protocol. These reduced budgets, artificial targets, and single seed are recorded as departures from the scientific contract. Checks cover exact v3 encoder transfer, discarded projection weights, unchanged frozen parameters, updated fine-tuned parameters, validation selection, and exact test-checkpoint reload. Synthetic metrics remain in a separate output file. The report on archived runs instead joins 120 existing TS2Vec runs to their matched core-contrastive runs, retaining means, sample SDs, paired differences, and configuration IDs.
| Protocol | TS2Vec-style | Core contrastive |
| Fine-tune | 8 | 4 |
| Probe | 0 | 12 |
Table 14. TS2Vec-style extension versus the core contrastive implementation. Entries count dataset–task cells with the better unrounded mean among 12 cells per protocol.
Validation records distinguish implementation assistance and failed attempts from successful execution of the extracted archive’s instructions. This is a development-agent check on a fresh environment, with no independent human reuse study. It does not test credentialed extraction, clinical label construction, or historical numerical retraining equivalence, which Appendix C.§ 4 examines for a MIMIC-IV slice.
Post-Hoc Random-Encoder Diagnostic
The random diagnostic contains 40 task–dataset–seed rows over MIMIC-IV and eICU-CRD. It freezes an untrained instance of the same default Transformer and trains only the linear task head from cached embeddings. Pairing those rows with the three core SSL probes gives 120 comparisons. The random probe is worse in 118. The exceptions are both against JEPA on eICU-CRD, by at most .001 AUPRC, for AKI and 24-hour mortality in one run each. This analysis was produced after the archived results and was not part of the core family. The raw/mask control below is another separate diagnostic.
JEPA, Checkpoint, and Hyperparameter Diagnostics
The separate JEPA diagnostic covers all 15 core pre-training runs. Across logged steps, the minimum target representation SD is .37 and the maximum mean pairwise cosine is .82. The mean final SD is 1.17. These statistics cannot rule out partial collapse or establish downstream quality. The historical checkpoint audit covers 45 core pretrained encoders. Core downstream runs use the final 100-epoch encoder. Retained JEPA validation-checkpoint files are from epochs 3–5. These are not final encoders and do not mean training ended early.
The checkpoint diagnostic completes 60 frozen probes, or 30 final-versus-best-SSL-validation pairs, over the three core objectives, five seeds, and MIMIC-IV hospital mortality and LOS. Both arms use the same cached CPU implementation, patient splits, train-only normalization, head initialization and batch RNG. Linear heads use the reference AdamW/cosine settings, validation patience and epoch cap. LOS trains with squared error and selects/reports absolute error. Validation alone selects the head, with the earliest epoch retained on a tie. The test set is evaluated once afterward. Producer checks matched all 30 encoder hashes to the archived encoder audit. The public audit reconstructs paired statistics and all 60 selections from saved validation histories.
JEPA worsens in all five seeds on both tasks (Table 15). For MAE and contrastive learning, the choice between final and best-validation encoders changes mean AUPRC and mean LOS error by at most .001 (AUPRC or days). MAE leads mortality and contrastive leads LOS under both policies in every seed, while JEPA’s LOS mean rank drops from second to third. These LOS ranks use the capped squared-error probe, whose ordering did not survive loss matching (§ 5). The per-seed selections, validation histories, and paired differences are in results/checkpoint-sensitivity/, and CHECKPOINT_CASE.md documents how to recompute them. Rerun-final cell means differ from historical means by at most .001 AUPRC and .01 LOS days. The largest absolute per-seed LOS deviation is .09 days. Cached CPU execution and dedicated paired RNG differ from historical GPU/Lightning execution. This diagnostic took about 20 wall minutes on CPU, with no failed or timed-out workers. It supports scoped checkpoint sensitivity, not equivalence, general rank stability, or conclusions about other databases or full fine-tuning.
Table 15: Separate MIMIC-IV cached CPU probe diagnostic. Mean sample SD over five seeds. Mortality uses AUPRC. LOS uses absolute error in days. is best-SSL-validation minus rerun-final. Brackets give the observed paired range. These final-arm values are reruns, not substituted historical scores.
| Task | Encoder | Rerun final | Best SSL val. | mean [range] | |
| Hospital mortality | MAE | 5 | 0.5093 0.0135 | 0.5096 0.0139 | +0.0003 [-0.0009, +0.0019] |
| Hospital mortality | JEPA | 5 | 0.4139 0.0196 | 0.3733 0.0094 | -0.0406 [-0.0563, -0.0166] |
| Hospital mortality | Contrastive | 5 | 0.4574 0.0236 | 0.4573 0.0236 | -0.0001 [-0.0003, +0.0000] |
| Remaining LOS | MAE | 5 | 2.7655 0.0651 | 2.7644 0.0628 | -0.0011 [-0.0047, +0.0019] |
| Remaining LOS | JEPA | 5 | 2.6758 0.0903 | 2.9396 0.0916 | +0.2638 [+0.1909, +0.3689] |
| Remaining LOS | Contrastive | 5 | 2.5946 0.0662 | 2.5949 0.0655 | +0.0003 [-0.0009, +0.0018] |
Table 15.
Table 16 contains only MIMIC-IV 24-hour mortality, full fine-tuning, full labels, and five seeds per level. Learning-rate values are upstream pretraining rates. Mask sweeps change their respective objective’s mask ratio. The largest within-sweep range of mean AUPRC is .018. This targeted slice establishes neither hyperparameter insensitivity nor objective-specific optimal tuning. Its test-set rankings are not used to select a new headline configuration.
Raw-Input and Mask-Only CPU Control
This post-hoc diagnostic uses only MIMIC-IV hospital mortality and the five canonical patient splits. A linear logistic classifier receives either 2,016 flattened mask entries or those entries plus 2,016 normalized values. Normalization uses the same full-cohort training partition. Task labels are filtered after patient assignment. Fixed SGD uses logistic loss, coefficient , the library’s optimal learning-rate schedule, square-root-balanced sample weights, and 20 epochs. The epoch with highest validation average precision is evaluated once on the test set. No label smoothing or hyperparameter search is used. The implementation and input-file SHA256 values are recorded separately.
Test AUPRC is .301 (.025) for masks and .404 (.013) for values plus masks, mean (sample SD). Mean test support is 11,157 stays and 1,310 deaths. Total wall time is 136 seconds on a CPU limited to two threads, including both representations and all five seeds, under a 900-second hard timeout. Every mask-only run selects epoch 20, so further optimization could improve this control. The result does not establish the performance of an optimized raw-input model, nor does it isolate SSL gains from differences in optimization, regularization, input dimension, or head training.
Post-Review Diagnostics
The analyses in this section were run after review, to answer reviewer questions. They are stored separately from the archived results, with their scripts and checksums, in results/ml4h-rebuttal-v1/. render_rebuttal.py regenerates their tables from those aggregates.
Converged and Loss-Matched Probes
The classification probe is an L2-regularized logistic regression fitted with L-BFGS to convergence, with the reference square-root-balanced class weights and 17 log-spaced regularization strengths selected on validation AUPRC. The absolute-error LOS probe uses a Huber loss with a .01-day threshold and is selected on validation absolute error. The ridge probe is selected on validation squared error. Each selected probe scores the test set once. Embeddings come from the archived final-epoch encoders, whose SHA256 digests match the encoder audit in 45/45 cases. Training rows, positives, and test rows match the archived results in every cell. In three checked cells, cached embeddings match the benchmark forward pass to within . Embeddings are standardized with training statistics. The converged classification probe differs from the capped probe in optimizer (L-BFGS to convergence instead of AdamW for at most 50 epochs), regularization (tuned L2 instead of fixed weight decay), input scaling, and label smoothing (none instead of .1). Class weights are the same. All logistic, ridge, and Huber fits met their convergence criterion. No selected strength lies at the strongest end of its grid, and the selections at the weakest end (6/135 logistic, 15/45 ridge, 21/45 Huber) lie on a flat part of the regularization path.
Table 17 gives the converged-probe results with seed variation and paired advantages. AUROC means and objective rankings for both probes are in p1_objective_ranking.csv.
| Data | Endpoint | MAE | JEPA | Contrastive | conv. | cap. |
| Combined | 24h mortality | 0.309 (0.033) | 0.208 (0.012) | 0.239 (0.023) | 0.070 | 0.079 |
| Combined | Hospital mortality | 0.470 (0.012) | 0.363 (0.006) | 0.395 (0.010) | 0.075 | 0.078 |
| Combined | AKI | 0.451 (0.005) | 0.332 (0.004) | 0.402 (0.006) | 0.049 | 0.051 |
| eICU-CRD | 24h mortality | 0.287 (0.017) | 0.219 (0.019) | 0.215 (0.003) | 0.068 | 0.076 |
| eICU-CRD | Hospital mortality | 0.439 (0.007) | 0.359 (0.005) | 0.356 (0.009) | 0.081 | 0.084 |
| eICU-CRD | AKI | 0.366 (0.010) | 0.275 (0.004) | 0.293 (0.006) | 0.074 | 0.075 |
| MIMIC-IV | 24h mortality | 0.357 (0.035) | 0.298 (0.018) | 0.322 (0.037) | 0.035 | 0.059 |
| MIMIC-IV | Hospital mortality | 0.520 (0.008) | 0.440 (0.016) | 0.471 (0.018) | 0.049 | 0.052 |
| MIMIC-IV | AKI | 0.510 (0.008) | 0.433 (0.017) | 0.492 (0.011) | 0.018 | 0.026 |
Table 17. Converged frozen probe. Test AUPRC, mean (sample SD) over five seeds, on the same frozen embeddings as the capped probe. is the mean paired difference between MAE and the stronger alternative, for the converged (conv.) and capped (cap.) probe. Bold marks the highest mean.
Rerunning the capped recipe on fixed embeddings with five head-initialization seeds shows that its result is not a stable function of the embedding for 24-hour mortality and LOS, where early stopping on a noisy validation metric decides the outcome. Table 18 gives the capped-probe LOS results that Table 2 replaces. The Huber probe uses a fixed .01-day threshold rather than an exact absolute-error loss.
| Data | Protocol | MAE | JEPA | Contrastive | Supervised | Constant median |
| Combined | Fine-tune | 2.118 (0.029) | 2.135 (0.013) | 2.152 (0.024) | 2.190 (0.031) | 2.403 (0.025) |
| Combined | Probe | 2.425 (0.027) | 2.475 (0.023) | 2.309 (0.029) | – | 2.403 (0.025) |
| eICU-CRD | Fine-tune | 1.964 (0.017) | 1.977 (0.034) | 2.008 (0.031) | 1.997 (0.030) | 2.176 (0.006) |
| eICU-CRD | Probe | 2.191 (0.011) | 2.249 (0.022) | 2.139 (0.011) | – | 2.176 (0.006) |
| MIMIC-IV | Fine-tune | 2.368 (0.051) | 2.371 (0.069) | 2.359 (0.069) | 2.397 (0.066) | 2.733 (0.074) |
| MIMIC-IV | Probe | 2.755 (0.058) | 2.679 (0.070) | 2.592 (0.076) | – | 2.733 (0.074) |
Table 18. Capped-probe LOS results from the archived results, superseded for probing by Table 2. Absolute error in days, mean (sample SD), . Probes train with squared error for at most 50 epochs and are selected on validation absolute error. The constant predicts each seed’s training-label median. Duplicate logs are collapsed by dataset and seed. Bold marks the lowest model mean within a protocol.
Combined Results by Source Database
Table 19 scores the Combined-regime models separately on the MIMIC-IV and eICU-CRD portions of the Combined test split. Probe columns use the converged probe. Fine-tune and supervised columns rerun test inference on the archived reference checkpoints, after verifying their SHA256 digests. Pooled metrics from this inference reproduce the archived results to within . Prevalence differs between portions, so AUPRC should be compared within a portion. AUROC, which is higher on the MIMIC-IV portion in all 30 objective–protocol–task cells, is in the two p3_*_per_source_aggregated.csv files. For LOS, the converged absolute-error probe puts MAE first on the eICU-CRD portion and last on the MIMIC-IV portion, within .021 days there. Fine-tuned MAE has the lowest LOS error on both portions.
| Task | Portion | Prev. | Probe (converged) | Fine-tune | |||||
| MAE | JEPA | Con. | MAE | JEPA | Con. | Sup. | |||
| 24h mortality | eICU-CRD | 0.016 | 0.302 | 0.195 | 0.225 | 0.392 | 0.377 | 0.384 | 0.373 |
| 24h mortality | MIMIC-IV | 0.016 | 0.334 | 0.250 | 0.276 | 0.448 | 0.423 | 0.429 | 0.424 |
| Hospital mortality | eICU-CRD | 0.093 | 0.443 | 0.337 | 0.362 | 0.498 | 0.485 | 0.479 | 0.484 |
| Hospital mortality | MIMIC-IV | 0.118 | 0.507 | 0.401 | 0.442 | 0.576 | 0.570 | 0.563 | 0.566 |
| AKI | eICU-CRD | 0.139 | 0.373 | 0.261 | 0.309 | 0.553 | 0.556 | 0.541 | 0.582 |
| AKI | MIMIC-IV | 0.204 | 0.525 | 0.396 | 0.486 | 0.697 | 0.695 | 0.685 | 0.720 |
| Remaining LOS | eICU-CRD | – | 2.013 | 2.049 | 2.036 | 1.967 | 1.982 | 2.001 | 2.034 |
| Remaining LOS | MIMIC-IV | – | 2.513 | 2.509 | 2.493 | 2.387 | 2.408 | 2.422 | 2.468 |
Table 19. Combined-regime models scored separately on the two source portions of the Combined test split. Means over five seeds. Classification uses AUPRC. LOS uses absolute error in days, with the Huber probe in the probe columns. Prev. is the mean test prevalence of the portion. Bold marks the best mean within each protocol.
Compute
All 45 core pretraining runs completed the 100-epoch schedule at batch size 256 in fp32 on one NVIDIA L4. The reference runs shared the GPU, about four at a time, so their wall-clock times in Table 20 are not exclusive GPU time. One MAE MIMIC-IV run that ran nearly alone took 0.92 hours, against about 2.25 hours for each of the four that ran concurrently. We therefore also timed each objective alone on an idle L4 of the same machine type, with the reference configuration on real MIMIC-IV batches, 20 warm-up steps, and 300 timed training steps. Projected hours cover training steps only, without validation or checkpointing. Step time did not depend on the database and changed by at most 3% with eight data-loader workers. In a monitored repeat, GPU utilization peaked at 13–21% (typically 10–18%) while one host CPU core was saturated, so wall time does not follow FLOPs. The cost order depends on the measure. FLOPs and shared-GPU wall-clock rank MAE contrastive JEPA, while the idle GPU ranks contrastive MAE JEPA. Dividing each run’s shared wall-clock time by its mean number of concurrent runs gives about 55 L4-hours for the 45 runs. Projecting the idle-GPU step times onto their optimizer steps gives about 63 hours of training steps alone. Under an equal-FLOPs budget, JEPA would receive about 45 MAE-equivalent epochs and contrastive learning about 76. In the checkpoint diagnostic, early JEPA checkpoints of the 100-epoch sched-
| Objective | Fwd. MFLOPs per sample | Trainable params | Idle L4 ms/step | MIMIC-IV (h) Shared | MIMIC-IV (h) Idle | eICU-CRD (h) Shared | eICU-CRD (h) Idle | Combined (h) Shared | Combined (h) Idle |
| MAE | 4.78 | 269,460 | 119.6 | 1.99 (0.60) | 0.68 | 3.94 (0.04) | 1.19 | 6.16 (0.04) | 1.89 |
| JEPA | 10.60 | 189,440 + 159,808 EMA | 179.7 | 3.00 (0.02) | 1.02 | 5.34 (0.03) | 1.81 | 8.32 (0.06) | 2.86 |
| Contrastive | 6.32 | 193,408 | 97.6 | 2.67 (0.01) | 0.56 | 4.76 (0.05) | 0.99 | 7.41 (0.03) | 1.56 |
Table 20. Pretraining cost per objective. FLOPs are forward FLOPs per sample. Wall-clock hours are W&B runtimes on a shared L4, mean (sample SD) over five seeds. Idle-GPU columns time training steps alone on an idle L4 and project them to the 100-epoch schedule. The JEPA EMA target copy is not trainable.
ule probed worse, but a shorter full schedule was not tested, nor was longer contrastive training.
Reproduction with the Maintained Framework
The reference runs used the historical implementation. We reran a MIMIC-IV slice with the maintained repository code, starting from the processed tensors (Table 21). Training configurations equal the reference apart from data path, data-loader workers, and device. Downstream runs from the archived reference encoders (seed 42) isolate the downstream code. Full-pipeline runs (seeds 42 and 123) also retrain the encoders. All six pretraining runs completed the 100-epoch schedule with the reference number of optimizer steps, and their final validation losses lie within 0.63 across-seed SDs of the reference. The reruns used a laptop CPU, while the reference used an L4. CPU and GPU draw different random streams for masks and dropout, so a rerun with the same seed is a different realization of the same configuration. The archived-encoder probes are deterministic and match to within . In the full pipeline, the objective order differs from the reference in three fine-tuning cells whose five-seed reference means are near-ties. LOS fine-tuning error is higher than the reference in all nine comparisons. Twenty of the 24 seed-123 reference downstream runs, and 37 of the 420 core downstream runs overall, had been resumed from their last checkpoint after an interruption, while the reruns ran uninterrupted. This slice cannot separate chance, resumption, and a small CPU-versus-GPU effect.
| Starting point | Protocol | Runs | Mean AUPRC (max) | Mean LOS days (max) | 1 SD | 2 SD |
| Archived encoders | Probe | 12 | 0 (0) | 12 | 12 | |
| Archived encoders | Fine-tune | 12 | 0.010 (0.040) | 0.029 (0.045) | 12 | 12 |
| Full pipeline | Probe | 24 | 0.008 (0.031) | 0.007 (0.016) | 23 | 24 |
| Full pipeline | Fine-tune | 24 | 0.010 (0.048) | 0.044 (0.074) | 19 | 24 |
Table 21. Maintained-framework reruns against the same-seed reference runs on MIMIC-IV. is the absolute difference in the test metric. SD columns count runs whose primary-metric difference lies within one or two reference across-seed SDs of the same objective, task, and protocol.
Evidence and Reproducibility Boundary
The archived evidence bundle used for the paper contains:
an aggregate result table with one row per scientific cell and seed summaries.
a per-seed table used for matched differences and coverage checks.
long-form metrics and retained diagnostic test outputs.
label-efficiency, classical-context, and TS2Vec comparison tables.
an evidence manifest recording row counts, schemas, analysis revision, and source hashes.
The core audit filter is: core SSL analysis, default model size, full label fraction, the three data regimes, four tasks, and the two adaptation protocols. It yields 84 aggregate rows with five seeds each and 420 per-seed rows. The random-encoder diagnostic is stored separately and is not merged into the archived core ranking. No patient-level row is included in the paper source or code repository.
The code repository also includes thesis-coverage-evidence.json, with source hashes and aggregate cells for the capacity, subgroup, and LOS secondary-metric summaries. Post-review diagnostics (Appendix C) and the recomputed reduced-label positive counts (results/camera-ready-v1/) are separate packages with their own provenance notes. They do not modify the archived results. The paper source includes LaTeX, generated table fragments and vector figures, a bibliography, and the official class/style files. It builds with PDFLaTeX and BibTeX through latexmk.
References
- [1]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619–15629, 2023. doi: 10.1109/CVPR52729.2023.01499.DOI
- [2]Nicolas Bennett, Drago Plecko, Ida-Fong Ukor, Nicolai Meinshausen, and Peter Bühlmann. ricu: R’s interface to intensive care data. GigaScience, 12:giad041, 2023. doi: 10.1093/gigascience/giad041.DOI
- [3]Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 8:6085, 2018. doi: 10.1038/s41598-018-24271-9.DOI
- [4]Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794, 2016. doi: 10.1145/2939672.2939785.DOI
- [5]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597–1607, 2020. URL https://proceedings.mlr.press/v119/chen20j.html.arxiv.org/abs/2002.05709
- [6]Sofiane Ennadir, Siavash Golkar, and Leopoldo Sarra. Joint embeddings go temporal. In NeurIPS Workshop on Time Series in the Age of Large Models, 2024. URL https://openreview.net/forum?id=FIdbozebmy.arxiv.org/abs/2509.25449
- [7]Lin Lawrence Guo, Santiago Eduardo Arciniegas, Adam Paul Yan, Jason Fries, George A. Tomlinson, and Lillian Sung. Systematic review of foundation models for structured electronic health records. Journal of the American Medical Informatics Association, 33(6):1190–1198, 2026. doi: 10.1093/jamia/ocag033.DOI
- [8]Hayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6:96, 2019. doi: 10.1038/s41597-019-0103-9.DOI
- [9]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022. doi: 10.1109/CVPR52688.2022.01553.DOI
- [10]Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Brian Gow, Benjamin Moody, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV (version 3.1), 2024. URL https://doi.org/10.13026/kpb9-mt58.
- [11]Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo Anthony Celi, and Roger G. Mark. MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data, 10(1), 2023. doi: 10.1038/s41597-022-01899-x.DOI
- [13]Alex Labach, Alesha Pokhrel, Xiao Shi Huang, Saba Zuberi, Seung Eun Yi, Maksims Volkovs, Tomi Poutanen, and Rahul G. Krishnan. DuETT: Dual event time transformer for electronic health records. In Proceedings of the Machine Learning for Healthcare Conference, volume 219 of Proceedings of Machine Learning Research, pages 403–422, 2023. URL https://proceedings.mlr.press/v219/labach23a.html.
- [15]Tom Pollard, Alistair Johnson, Jesse Raffa, Leo Anthony Celi, Omar Badawi, and Roger Mark. eICU collaborative research database (version 2.0), 2019. URL https://doi.org/10.3026/C2WM1R.
- [16]Tom J. Pollard, Alistair E. W. Johnson, Jesse D. Raffa, Leo Anthony Celi, Roger G. Mark, and Omar Badawi. The eICU collaborative research database, a freely available multi-center database for critical care research. Scientific Data, 5:180178, 2018. doi: 10.1038/sdata.2018.178.DOI
- [17]Aniruddh Raghu, Payal Chandak, Ridwan Alam, John Guttag, and Collin Stultz. Sequential multi-dimensional self-supervised learning for clinical time series. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28531–28548, 2023. URL https://proceedings.mlr.press/v202/raghu23a.html.arxiv.org/abs/2307.10923
- [18]Takaya Saito and Marc Rehmsmeier. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3):e0118432, 2015. doi: 10.1371/journal.pone.0118432.DOI
- [19]Amelia L. M. Tan, Emily J. Getzen, Meghan R. Hutch, Zachary H. Strasser, et al. Informative missingness: What can we learn from patterns in missing laboratory data in the electronic health record? Journal of Biomedical Informatics, 139:104306, 2023. doi: 10.1016/j.jbi.2023.104306.DOI
- [20]Sindhu Tipirneni and Chandan K. Reddy. Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series. ACM Transactions on Knowledge Discovery from Data, 16(6):105, 2022. doi: 10.1145/3516367.DOI
- [21]Robin van de Water, Hendrik Schmidt, Paul Elbers, Patrick Thoral, Bert Anrrich, and Patrick Rocken-schaub. Yet another ICU benchmark: A flexible multi-center framework for clinical ML. In International Conference on Learning Representations, 2024.arxiv.org/abs/2306.05109
- [22]Xiao Xiang, David Restrepo, Hyewon Jeong, Yugang Jia, and Leo Anthony Celi. Learning representations from incomplete EHR data with dual-masked autoencoding. arXiv preprint arXiv:2602.15159, 2026. doi: 10.48550/arXiv.2602.15159.DOI
- [23]Yixuan Yang, Mehak Arora, Ryan Zhang, Baraa Abed, Junseob Kim, Tilendra Choudhary, Md Hassanuzzaman, Kevin Zhu, Ayman Ali, Chengkun Yang, Alasdair Edward Gent, Victor Moas, and Rishikesh Kamaleswaran. Clin-JEPA: A multi-phase co-training framework for joint-embedding predictive pretraining on EHR patient trajectories. arXiv preprint arXiv:2605.10840, 2026. doi: 10.48550/arXiv.2605.10840.DOI
- [24]Hugo Yèche, Gideon Dresdner, Francesco Locatello, Matthias Hüser, and Gunnar Rätsch. Neighborhood contrastive learning applied to online patient monitoring. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11964–11974, 2021. URL https://proceedings.mlr.press/v139/yeche21a.html.arxiv.org/abs/2106.05142
- [25]Zhihao Yu, Xu Chu, Yujie Jin, Yasha Wang, and Junfeng Zhao. SMART: Towards pre-trained missing-aware model for patient health status prediction. In Advances in Neural Information Processing Systems, volume 37, pages 63986–64009, 2024. doi: 10.52202/079017-2043.DOI
- [26]Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. TS2Vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8980–8987, 2022. doi: 10.1609/aaai.v36i8.20881.DOI







