betaFeedback

WorksComputer Science

Cracks in Embeddings: Robustness and Stability of JEPA-Based Chest X-Ray Representations

Abstract

Self-supervised Joint Embedding Predictive Architecture (JEPA) models are increasingly used for medical imaging, however there is a limited understanding about robustness of these models toward real-world image changes. We test four JEPA-based chest X-ray encoders, trained on 78,468 NIH ChestX-ray14 images, under six tests: two black-box adversarial attacks (SPSA, Boundary Attack), three image corruptions (Gaussian noise, brightness shift, contrast shift), and a label-scarcity test, measuring embedding change and classification AUROC. These perturbations cause severe, model-dependent damage: encoder-level Cosine Similarity drops as low as 0.33, and AUROC falls by as much as 0.12 in the worst case (I-JEPA under the Boundary Test). We test two ways of training the small predictor on top of each frozen encoder to address this. Adversarial fine-tuning, trained on clean and perturbed images together, raises embedding consistency above 0.97 in nearly every case and improves AUROC, relative to the same perturbed baseline, for most models, with the largest gain of +0.10+0.10 on I-JEPA. A Vector-Quantized codebook method, trained only on clean images and evaluated on perturbed test images for the first time, instead degrades under distortion for all four models, most noticeably for I-JEPA (−0.15-0.15 AUROC on average), showing that its discrete bottleneck alone does not provide the robustness that comes from training on perturbed data directly.

Introduction

Deep learning has become the standard tool for analysis of medical images, however training these models usually needs large amounts of labeled data. In the medical domain, labeling is slow and expensive: it has to be done by clinical experts, and privacy rules limit how freely data can be shared, which makes fully supervised training hard to scale. Self-supervised learning (SSL) offers a way around for this problem. An SSL model first learns useful image features from a large pool of unlabeled images, and only a small labeled set is then needed to train the final task head. Among recent SSL approaches, the Joint Embedding Predictive Architecture (JEPA) [3, 17] has drawn a lot of attention. JEPA relies on latent space predictions. It predicts the representation of masked image regions from the visible part instead of predicting raw pixels. This pushes the model to capture high-level structure rather than low-level texture. This idea has now been carried into chest X-ray analysis, where several JEPA-style world models have been proposed, including CheXWorld [29], X-WIN [28], and RadJEPA [1].

Strong accuracy on a clean test set, however, is not enough for a model that may be used in the clinic. Chest X-rays vary widely between scanners, hospitals, and acquisition settings, so the same anatomy can appear at different brightness, contrast, or noise levels. A deployed system may also be probed by an attacker who can query its outputs but not its internals. In all of these cases the image seen at test time differs from the clean training images, and a safety-critical model should degrade gracefully rather than fail silently. Whether today’s JEPA chest X-ray encoders stay reliable under such changes, and what can be done when they do not, is still largely an open question.

Prior work has started to evaluate self-supervised medical encoders from different angles. A closely related effort is REMEDIS [4], which showed that self-supervised pretraining can improve the robustness and data efficiency of medical imaging models, but studied a single representation-learning recipe rather than comparing paradigms. Bunde­le et al. [6] went broader, running a systematic study of SSL in medical imaging across many datasets and methods (standardised through the MedMNIST benchmark [26]) and covering label-efficient learning, cross-dataset generalisation, and out-of-distribution detection; they showed that choices such as the SSL method, the backbone, and the initialisation strategy strongly shape how well these models transfer, echoing earlier findings that transformer backbones benefit especially from pretraining in the medical domain [20]. These studies, however, do not measure how the learned representations behave when the input image itself is corrupted or attacked, and they offer no way to recover performance once it drops. Robustness to input perturbations has instead been studied mostly for natural images: through white-box adversarial attacks such as FGSM [10] and the optimization attack of Carlini and Wagner [7], through query-only black-box attacks such as SPSA [23] and the Boundary Attack [5], and through common, non-adversarial corruptions [16]; the underlying fragility of neural networks to small input changes has been known since Szegedy et al. [21]. This line of work has not yet been connected to JEPA-based chest X-ray models.

Two gaps therefore remain. First, the input-perturbation robustness of JEPA chest X-ray encoders, under both realistic corruptions and black-box attacks, has not been measured, so robustness tends to be assumed rather than tested. Second, even where damage is found, as measured by a drop in classification AUROC value, there is no established way to train the downstream head to handle it. We keep each encoder frozen, attach a small predictor and classifier on top, measure how four existing JEPA chest X-ray encoders degrade under six test conditions, and then study two training methods that touch only this lightweight head: adversarial fine-tuning and a Vector-Quantised codebook [24]. We make the following contributions:

  • We test four JEPA-based chest X-ray encoders under six conditions: two adversarial perturbations, three image corruptions, and a low-label test, using the full NIH ChestX-ray14 dataset, and measuring both embedding change and classification performance.

  • We tested a structurally different training method based on Vector Quantisation (a discrete codebook) and compared it with the predictor that does not include a codebook.

  • We design a fine-tuning method using the baseline predictor, which trains the predictor on clean and perturbed images together to improve the AUROC value.

Related Work

JEPA models. LeCun [17] proposed the JEPA idea: instead of rebuilding the raw pixels of a hidden part of an image, the model predicts that part’s representation in embedding space, using a small predictor network that looks at the visible part. This avoids wasting effort on small pixel-level details like exact texture or noise, and instead pushes the model to learn higher-level structure. I-JEPA [3] built this idea using a Vision Transformer: it hides a few large chunks of the image, then trains a predictor to guess what those chunks’ representations should look like, based on the rest of the image, without needing any hand-designed data augmentations. CheXWorld [29], X-WIN [28], and RadJEPA [1] apply this same idea to chest X-rays, training their models to predict how a chest X-ray’s representation changes across different views of the same anatomy, so the model learns the general structure of chest X-rays rather than just one narrow task. In these models, they are using a predictor that will predict the embeddings of missing patches. In our experiments, we use a predictor that follows the residual design, uses the same skip-connection idea as ResNet [14], so it only has to learn a small adjustment on top of the frozen encoder’s embedding. I-JEPA’s own backbone is a Vision Transformer [9], the same type of architecture used by all four encoders in our study.

Self-supervised representation learning. Beyond JEPA, contrastive and distillation methods such as SimCLR [8], MoCo [13], and BYOL [11] learn representations without labels by comparing augmented views of the same image. Masked Autoencoders [12] instead rebuild masked pixel patches directly, a design JEPA avoids on purpose in favor of predicting in representation space.

Evaluating self-supervised models. Bundele et al. Bundele et al. [6] tested self-supervised medical encoders for label-efficient learning and out-of-distribution detection, but not for robustness to corrupted or perturbed images, and gave no correction method. Our few-sample test (100%, 10%, and 1% of training labels) follows the standard low-shot protocol used for self-supervised vision models [8]. The encoders we study were pretrained on different datasets; for our own experiments, we use the NIH ChestX-ray14 dataset [25].

Black-box adversarial attacks. We use two attacks that do not need access to model gradients, matching a realistic threat model where an attacker can only query a deployed model. SPSA finds a useful attack direction by random probing and reading the model’s outputs [23]. The Boundary Attack starts from a large random change and slowly shrinks it while staying adversarial, using only the model’s final decision [5]. Both differ from the earlier line of gradient-based, white-box attacks and defenses FGSM [10], the Carlini and Wagner attack [7], and PGD-based adversarial training [18], the white-box counterpart to the fine-tuning approach we propose which all assume direct access to the model’s internals rather than just its outputs. Beyond adversarial noise, Hendrycks and Dietterich [16] showed that common, non-adversarial corruptions also hurt neural networks, which motivates our simple corruptions (noise, brightness, contrast) alongside the attacks. The general fragility of neural networks to small input changes was first pointed out by Szegedy et al. Szegedy et al. [21].

Method

Overview

Every encoder we test is frozen throughout the study. On top of it we attach two small trainable pieces: a predictor, which computes an adjusted output on top of it, and a classifier, which is used for disease prediction. Figure 1 summarises the study in two parts. Part (a) is training: all 78,468 training images pass through each of the four frozen encoders, and the resulting features train one predictor and classifier per encoder for 300 epochs on clean images only; this pair is the “Original” baseline used everywhere else in the paper. Part (b) is the perturbation-and-test loop used for every result in § 5: held-out test images go through the five perturbation tests, through the same frozen encoders, and then through two separate predictor variants built on the Original predictor: a fine-tuned predictor/classifier (§ 3.5.1) and a codebook predictor/classifier (§ 3.5.2). Both variants are evaluated on the same distorted test images, so they can be compared fairly. In all cases the encoder itself is never trained or changed; only the small predictor and classifier are trained and tested.

Training and testing procedure for robustness evaluation

Figure 1. (a) Training: each frozen encoder trains its own predictor/classifier pair on clean images, not shared across models. (b) Test: the held-out set is perturbed, passed through the same frozen encoders, and evaluated by two independently trained predictors, codebook (clean images only) and fine-tuned (clean and perturbed images).

Predictor Architecture

The predictor is a small residual network:

zp=z+W3σ(W2σ(W1z))(1)z_p = z + W_3 \sigma(W_2 \sigma(W_1 z)) \tag*{(1)}

Where, zz is the raw embedding produced by the frozen encoder and never changes; W1W_1, W2W_2, W3W_3 are the three trainable weight matrices of the predictor (sized D×512D \times512, 512×512512 \times512, and 512×D512 \times D, where DD is the encoder’s embedding size); σ\sigma is the GELU activation; zpz_p is the predictor’s output, passed on to the classifier. As Equation (1) shows, zpz_p is computed by adding the original embedding zz directly to the output of the three trainable layers a residual (skip) connection [14]. Because of this, the predictor only has to learn a small adjustment on top of zz, rather than reconstructing the whole embedding from scratch.

The classifier is a single linear layer, y^=Wczp+bc\hat{y} = W_c z_p + b_c, where WcW_c and bcb_c are trainable weights and y^\hat{y} is the predicted probability for each of the 14 chest X-ray disease labels.

The predictor and classifier are trained together, using only clean (unperturbed) images. After this training, both are frozen and used as the “Original” baseline for every robustness test in this paper.

Measuring Robustness

To test robustness, we take a clean image xx, apply a perturbation function AA (an attack or corruption) at some severity level to get a perturbed version of it, and compare the results on the clean image versus the perturbed image, using two measures: how much the predictor’s and classifier’s predictions change.

CosSim⁡(zp,z~p)=zp⋅z~p∥zp∥∥z~p∥(2)\operatorname{CosSim}(z_p,\tilde{z}_p)=\frac{z_p\cdot\tilde{z}_p}{\lVert z_p\rVert\lVert\tilde{z}_p\rVert} \tag*{(2)}

Where, zpz_p is the predictor’s output embedding for the clean image; z~p\tilde{z}_p is its output for the perturbed version of the same image; CosSim ranges from −1-1 to 11, where 11 means the two embeddings point in exactly the same direction (the perturbation caused no change) and lower values mean it pushed the embedding further from its original position. We also measure Macro AUROC of the classifier’s predictions on both the clean and the perturbed embedding, using the true disease labels.

The Six Robustness Tests

We test five perturbations at four severity levels each (the Boundary Attack has one fixed setting), plus one label-scarcity test. Table 1 lists every severity level used.

TestSeverity levels tested
SPSAϵ=0.02, 0.05, 0.10, 0.20\epsilon= 0.02,\ 0.05,\ 0.10,\ 0.20
Boundary Attackfixed (80 queries)
Gaussian Noiseσ=0.05, 0.15, 0.30, 0.50\sigma= 0.05,\ 0.15,\ 0.30,\ 0.50
Brightness Shiftfactor =0.3, 0.6, 1.6, 2.2= 0.3,\ 0.6,\ 1.6,\ 2.2
Contrast Shiftfactor =0.2, 0.4, 1.8, 2.5= 0.2,\ 0.4,\ 1.8,\ 2.5
Few-sample100%, 10%, 1% of sample images

Table 1. Severity levels used for each perturbation test.

  • SPSA: SPSA is a gradient-free attack [23]. It does not have access to the model’s internal gradients, only its outputs. At each step, it tries a small random change to the image and checks whether that change pushed the model’s output embedding further from the original. It uses this information to estimate which direction is most damaging. Repeating this process over many steps builds up a full, effective perturbation.

  • Boundary Attack: The Boundary Attack [5] begins with a large random change to the image, large enough to flip the model’s prediction. It then repeatedly shrinks the change by a small step, checking each time whether the prediction is still flipped. If it is, it keeps the smaller change and continues; if not, it steps back. It stops once no further shrinking is possible without the prediction flipping back.

  • Gaussian Noise: This corruption adds random noise, drawn from a Gaussian (normal) distribution, to every pixel of the image. It simulates the kind of sensor noise that real imaging hardware can introduce.

  • Brightness Shift: This corruption multiplies every pixel value by a fixed factor, making the whole image uniformly darker or brighter. It simulates exposure differences between scanners.

  • Contrast Shift: This corruption stretches or shrinks pixel values around the image’s average brightness. It imitates contrast differences between scanners. Together, these three corruptions follow the spirit of common-corruption robustness benchmarks [16], adapted here to chest X-ray conditions.

  • Few-Sample Robustness Test: This test does not perturb any image at all. Instead, it trains a fresh linear classifier, with no predictor involved, using only 100%, 10%, or 1% of the training labels. It measures how good the encoder’s raw embeddings are when very little labeled data is available.

We use only black-box attacks (SPSA, Boundary), which do not need access to model gradients, since a realistic attacker on a deployed medical system usually has query access only, not internal access, unlike white-box attacks such as FGSM [10].

Two Training Methods for the Predictor

Since the encoder is frozen, the only way to check how the model reacts to a distorted image is by how the predictor and classifier on top of it are trained.

Adversarial Fine-Tuning

The goal of this method is simple: teach the predictor to give the same answer on a clean image and its perturbed version. We start from the clean image to train the predictor and classifier for the baseline (§ 3.2), but now, for this adversarial fine-tuning, we use a mix of clean and perturbed images instead of clean images only. For each image in the training set, we generate one perturbed copy for every perturbation type listed in Table 1, always using that perturbation’s strongest (most damaging) severity level. This is similar in spirit to adversarial training [18], except we only train the small predictor head on top of a frozen encoder, not the whole network.

The training loss has three parts, one for each thing we want the model to do:

L=12BCE⁡(y^,y)+12BCE⁡(y~,y^)+λ(1−CosSim⁡(zp,z~p))(3)\mathcal{L} = \frac{1}{2}\operatorname{BCE}(\hat{y},y) + \frac{1}{2}\operatorname{BCE}(\tilde{y},\hat{y}) + \lambda(1-\operatorname{CosSim}(z_p,\tilde{z}_p)) \tag*{(3)}

Where, y^\hat{y} and y~\tilde{y} are the classifier’s predictions on the clean and perturbed image; yy is the true label; BCE is binary cross-entropy (the standard loss for multi-label classification); λ\lambda is a weight (we use λ=1\lambda=1) controlling how strongly we enforce embedding similarity. The first two terms push the model to classify correctly whether or not the image was perturbed. The third term penalises the model whenever the perturbed embedding drifts away from the clean embedding, so the predictor learns to treat both versions of the same image as the same thing.

For efficiency, the SPSA and Boundary Attack caches used during fine-tuning are built on a large subset of the training split rather than the full set, since these two attacks require repeated forward passes per image and are far more expensive to generate than the simple corruption-based attacks. The simple corruptions (Gaussian noise, brightness, contrast) are instead computed over the entire training split, since a single extra forward pass per image keeps their cost low. This reduced budget is used only to build training data. Every evaluation in this paper still uses the full, harder settings from Table 1.

Vector-Quantized Codebook

We were also motivated by prior work showing that vector quantization can improve robustness. Mao et al. [19] showed that adding discrete, VQ-based representations to a Vision Transformer makes it more robust to corruptions and adversarial attacks. Yang et al. [27] used a VQ codebook with self-attention to learn feature representations that stay useful across different corruption types, though it does so by training jointly on clean and corrupted images rather than relying on the codebook alone, and He and Singhal [15] showed that a VQ-based U-Net can filter adversarial noise from images before classification, using only clean training data. These results suggest that snapping an input to one of a small set of learned codebook vectors can remove small perturbations by construction, raising the question of whether this holds without ever needing to see a perturbed example during training. We test whether this same idea, applied to our predictor instead of to raw images or a large trainable backbone, is enough to produce robustness on its own.

This method changes the predictor’s architecture instead of its training data. Rather than pooling the encoder’s output into a single vector, we work directly on its un-pooled patch tokens Z∈RN×DZ \in\mathbb{R}^{N \times D}, where NN is the number of spatial patches (N=196N=196 for CheXWorld and X-WIN, N=256N=256 for RadJEPA and I-JEPA) and DD is the encoder’s embedding width. We reshape ZZ into a spatial grid X∈RD×g×gX \in\mathbb{R}^{D \times g \times g}, where g=Ng=\sqrt{N}, and pass it through one strided convolution:

H=σ(BN⁡(Conv⁡3×3,s=2(X)))∈R128×g′×g′(4)H = \sigma(\operatorname{BN}(\operatorname{Conv}_{3 \times3,s=2}(X))) \in\mathbb{R}^{128 \times g' \times g'} \tag*{(4)}

This 3×33 \times3, stride-2 convolution maps the DD input channels down to 128 and halves the spatial size: a 14×1414 \times14 grid (N=196N=196) becomes 7×77 \times7, and a 16×1616 \times16 grid (N=256N=256) becomes 8×88 \times8. BN is batch normalization and σ\sigma is GELU.

Each of the g′×g′g' \times g' spatial positions is quantized separately. For position ii, its 128-dimensional feature hih_i is normalized to ze,i=LayerNorm⁡(hi)z_{e,i} = \operatorname{LayerNorm}(h_i), then matched to the nearest of the codebook’s 128 learned vectors:

qi=eki,ki=argmin⁡j∥ze,i−ej∥(5)q_i = e_{k_i}, \quad k_i = \underset{j}{\operatorname{argmin}}\lVert z_{e,i} - e_j \rVert \tag*{(5)}

This gives 49 lookups per image for the 14×1414 \times14-grid encoders and 64 for the 16×1616 \times16-grid encoders.

Picking the nearest codebook vector is a hard, discrete choice, so it has no gradient. We use a straight-through estimator, qiST=ze,i+sg⁡[qi−ze,i]q_i^{\mathrm{ST}} = z_{e,i} + \operatorname{sg}[q_i-z_{e,i}], so the forward pass uses the real codebook vector while the backward pass lets gradients skip over this step untouched. Figure 2 shows the full predictor with this step included.

The codebook predictor: tokens are reshaped into a spatial grid, convolved, quantized at each position, and combined with the pre-quantization features via a residual connection before the classifier.

Figure 2. The codebook predictor: tokens are reshaped into a spatial grid, convolved, quantized at each position, and combined with the pre-quantization features via a residual connection before the classifier.

The codebook starts from kk-means clusters, not random values: we run the convolution and normalisation step on a 1,000-image subset of the clean training set and cluster the resulting vectors into 128 groups, whose centers become the codebook. During training, each codebook entry is nudged toward whatever vectors get assigned to it, using an exponential moving average (decay 0.99), and entries that are rarely used get periodically reset so the whole codebook stays useful.

The quantized feature map is then added back to the pre-quantization convolution output and flattened straight into one linear classifier layer:

F=H+Q,y^=Wcflatten⁡(F)+bc(6)F = H + Q,\qquad\hat{y} = W_c\operatorname{flatten}(F) + b_c \tag*{(6)}

QQ collects all the qiSTq_i^{\mathrm{ST}} vectors back into their grid positions. This single linear layer maps the flattened features (6272 or 8192 numbers, depending on the encoder) directly to the 14 disease outputs, with no hidden layer.

This predictor never sees a perturbed image during training, so its loss is just a classification term plus a commitment term that keeps the pre-quantized features close to their assigned codebook entries:

Lcodebook=BCE⁡(y,y^)+β128⋅g′2∑i=1g′2∥ze,i−sg⁡[qi]∥2(7)\mathcal{L}_{\mathrm{codebook}} = \operatorname{BCE}(y,\hat{y}) + \frac{\beta}{128 \cdot g^{\prime2}} \sum_{i=1}^{g^{\prime2}} \lVert z_{e,i} - \operatorname{sg}[q_i] \rVert^2 \tag*{(7)}

Here β=0.25\beta= 0.25 is the commitment weight, and sg⁡[⋅]\operatorname{sg}[\cdot] blocks gradients so this term only shapes the encoder side, not the codebook itself; the codebook is updated separately, by the EMA step described above.

We train this predictor for 300 epochs with Adam (learning rate 10−310^{-3}, weight decay 10−410^{-4}, cosine-annealed to 10−510^{-5}, batch size 128) on the full 78,468-image clean training set, then evaluate it for the first time on the NIH ChestX-ray14 test split (3,000 images), using the same perturbation tests as fine-tuning (Table 1). This tests one question: can the codebook’s discrete structure make the model resistant to perturbations through architecture alone, without ever training on a perturbed example? Results are given in § 5.2, alongside fine-tuning’s results in § 5.3.

Experimental Setup

Dataset. We use the official NIH ChestX-ray14 dataset [25], using its official train/test split throughout: 78,468 images for training, and a separate held-out test set for evaluation. Testing uses distorted versions of the held-out test images created by applying the perturbation tests in Table 1 to them so that evaluation reflects how the models behave on damaged, real-world-like inputs rather than clean ones.

Models. We test four frozen JEPA-style encoders. Three of them, CheXWorld [29], RadJEPA [1], and I-JEPA [3], are used exactly as publicly released. X-WIN [28] is the exception: its original pretrained checkpoint is not publicly available, and its original training recipe uses the NLST (National Lung Screening Trial) dataset [22], which is too large for us to pretrain on given our compute constraints. We therefore pretrained our own X-WIN encoder on the smaller LIDC-IDRI CT dataset [2] instead, for 100 epochs. Once pretrained, this encoder is frozen exactly like the other three and used for all further testing. I-JEPA is the only encoder pretrained on general natural images (ImageNet) rather than medical images, which lets us ask whether a strong general-purpose encoder transfers well to chest X-rays. None of the four encoders’ weights are ever changed after pretraining; only the small predictor and classifier on top of them are trained.

Training. The predictor and classifier use the Adam optimiser (weight decay 10−410^{-4}). The clean predictor and classifier are trained for 300 epochs at learning rate 10−210^{-2} on all 78,468 clean training images; this pair is the “Original” baseline. For adversarial fine-tuning, the predictor and classifier are warm-started from these clean-trained weights rather than trained from scratch, then trained for 200 epochs at learning rate 10−310^{-3} on the full training split augmented with its corrupted versions, plus a large subset used for the more expensive SPSA/Boundary attacks, so that classification and embedding consistency improve together (Equation (3)). The codebook variant instead uses a separate convolutional classifier with a 128-entry codebook (§ 3.5.2), trained with the Adam optimiser (learning rate 10−310^{-3}, weight decay 10−410^{-4}, cosine-annealed to 10−510^{-5}) for 300 epochs in mini-batches of 128, on the same 78,468 clean training images only; no perturbed images are used during codebook training (Equation (7)).

Evaluation metrics. Every result in § 5 uses two metrics: Macro AUROC (the per-class area under the ROC curve, averaged over the 14 labels) and Embedding Cosine Similarity between each image’s clean and perturbed predictor output (Equation (2)). These are evaluation-only metrics; they are never optimised directly. The training objectives are Equation (3) for the clean and fine-tuned predictors and Equation (7) for the codebook predictor.

Hardware. All experiments run on two NVIDIA RTX A6000 GPUs (48 GB each) in parallel, with two encoders assigned to each GPU.

Results

How Much Damage Does Each Test Cause?

Figures 3 and 4 show how Macro AUROC and encoder-level CosSim move as each perturbation gets stronger, across all four severity levels. Table 2 summarises this at the strongest level for the predictor without a codebook, on average, CheXWorld degrades by 6.90%, X-WIN by 3.17%, RadJEPA by 4.76%, and I-JEPA by 10.77%. Under SPSA and Gaussian Noise, CheXWorld, X-WIN, and RadJEPA stay almost flat, while I-JEPA degrades steadily, its Gaussian Noise CosSim collapsing to 0.33. Brightness and Contrast do not get steadily worse, embedding similarity is often lowest at the mildest setting (CheXWorld and X-WIN’s CosSim drops to 0.39 and 0.41 at Level 1), even though AUROC keeps falling through Level 4. The Boundary Test behaves like SPSA and Noise, with I-JEPA showing its sharpest single-test drop (18.46%, CosSim to 0.39). For RadJEPA under Gaussian Noise, the table shows −1.59%, meaning AUROC did not drop at all, it was actually a tiny bit higher on the noisy images (0.64) than on the clean ones (0.63). This does not mean noise helped the model, with only 1,000 test images, a difference this small can happen just by chance. Each test damages a different model the most, so no single test tells the whole story.

TestCheXWorldX-WINRadJEPAI-JEPA
Clean0.580.630.630.65
SPSA (0.20)0.58 (0.00%)0.63 (0.00%)0.63 (0.00%)0.60 (7.69%)
Boundary (80 q)0.57 (1.72%)0.63 (0.00%)0.63 (0.00%)0.53 (18.46%)
Gaussian Noise (0.50)0.57 (1.72%)0.62 (1.59%)0.64 (−1.59%)0.54 (16.92%)
Brightness (2.2)0.50 (13.79%)0.57 (9.52%)0.53 (15.87%)0.63 (3.08%)
Contrast (2.5)0.47 (18.97%)0.59 (6.35%)0.59 (6.35%)0.62 (4.62%)
Average0.54 (6.90%)0.61 (3.17%)0.60 (4.76%)0.58 (10.77%)

Table 2. Macro AUROC at the strongest severity level of each perturbation test, each model, on the 1,000-image held-out test subset.

Four panels showing Macro AUROC by Severity Level for (a) SPSA, (b) Gaussian Noise, (c) Brightness, and (d) Contrast

Figure 3. (a) SPSA (b) Gaussian Noise (c) Brightness (d) Contrast. Macro AUROC across all four severity levels of each perturbation test, each model, on the 1,000-image held-out test subset.

Codebook Test: Results

Table 3 reports Macro AUROC for the 128-entry codebook at the strongest severity level. On average, CheXWorld degrades by 10.00%, X-WIN by 13.56%, RadJEPA by 9.38%, and I-JEPA by 21.43%. RadJEPA and CheXWorld hold up best; I-JEPA is hit hardest, dropping by as much as 24.29% under Gaussian Noise.

TestCheXWorldX-WINRadJEPAI-JEPA
Clean0.700.590.640.70
SPSA (0.20)0.63 (10.00%)0.50 (15.25%)0.58 (9.38%)0.56 (20.00%)
Boundary (80 q)0.65 (7.14%)0.51 (13.56%)0.59 (7.81%)0.57 (18.57%)
Gaussian Noise (0.50)0.61 (12.86%)0.50 (15.25%)0.57 (10.94%)0.53 (24.29%)
Brightness (2.2)0.62 (11.43%)0.51 (13.56%)0.58 (9.38%)0.55 (21.43%)
Contrast (2.5)0.63 (10.00%)0.51 (13.56%)0.58 (9.38%)0.54 (22.86%)
Average0.63 (10.00%)0.51 (13.56%)0.58 (9.38%)0.55 (21.43%)

Table 3. Codebook Macro AUROC at the strongest severity level of each perturbation test, each model, on the 3,000-image held-out test subset.

This falls short of the gains reported for vector quantization in prior work [15, 27], likely because our setting is more constrained: our codebook is trained on clean images only, sits inside a small predictor rather than a large trainable backbone, has only 128 entries, and operates on top of an encoder that is never updated. With so little capacity to work with, a discrete bottleneck alone is not enough to resist perturbations it was never trained on.

Fine-Tuning: Results

Table 4 shows fine-tuned Macro AUROC at the strongest level of each test. On average across the five tests, fine-tuning reaches 0.62 for CheXWorld, 0.63 for X-WIN, 0.62 for RadJEPA, and 0.68 for I-JEPA, gains that hold across three of the four models, with I-JEPA (the most damaged model, per § 5.1) improving the most and RadJEPA (the least damaged) essentially unchanged, consistent with its already-high robustness.

TestCheXWorldX-WINRadJEPAI-JEPA
SPSA (0.20)0.650.660.650.68
Boundary (80 q)0.650.650.650.70
Gaussian Noise (0.50)0.640.650.650.68
Brightness (2.2)0.550.590.530.66
Contrast (2.5)0.600.630.600.67
Average0.620.630.620.68

Table 4. Fine-Tuned Macro AUROC at the strongest severity level of each perturbation test, each model, on the held-out test subset.

Few Sample Robustness: How Label-Efficient Are the Raw Embeddings?

Few Sample Robustness: How Label-Efficient Are the Raw Embeddings?

The tests above measure robustness to input perturbations. Here we ask a simpler question, how good are the encoders’ raw embeddings when very little labeled data is available? For each model we train a fresh linear classifier on clean images only, using 100%, 10%, or 1% of the training labels, and evaluate on the held-out set. Figure 5 reports the Macro AUROC of these Original-predictor embeddings.

Few-sample robustness of the raw (Original-predictor) embeddings

Figure 5. Few-sample robustness of the raw (Original-predictor) embeddings: Macro AUROC on the held-out test set when a fresh linear classifier is trained on 100%, 10%, or 1% of the training labels. Darker means higher AUROC.

Two things stand out. First, I-JEPA has the strongest embeddings at every budget (0.75, 0.73, 0.65), even though it was pretrained on natural images rather than chest X-rays, so a strong general-purpose encoder transfers surprisingly well. Second, the models differ in how gracefully they degrade as labels shrink. X-WIN starts highest among the medical encoders at full labels (0.67) but falls the most at 1% (0.55), while CheXWorld and RadJEPA are lower at full labels yet hold up better at 1% (both 0.60). We also passed clean images through the fine-tuned and codebook predictors and found that this training does not reliably improve low-label AUROC; it changes what an embedding is good at rather than uniformly helping. Training aimed at perturbation robustness should therefore not be assumed to also help label-efficient learning.

Fine-Tuning Across Random Seeds

The fine-tuning results in § 5.3 use a single random seed (42). To check whether they depend on that initialisation, we repeat fine-tuning under three more seeds (123, 456, 789), keeping every other setting identical, including the 3,000-image test subset (Table 1, § 3.5.1), and compare against the seed-42 baseline on the same subset. Figure 6 reports the mean and standard deviation of fine-tuned Macro AUROC across all five tests, per model, over the three repeat seeds.

Bar chart of average fine-tuned Macro AUROC across four models

Figure 6. Average fine-tuned Macro AUROC across all five tests (strongest level), per model, under three random seeds.

X-WIN and RadJEPA are essentially seed-invariant (std ≤ 0.001), matching their seed-42 baselines closely (0.633 and 0.616). I-JEPA varies slightly more (std = 0.006, mean 0.671 vs. seed-42’s 0.677). CheXWorld is the most seed-sensitive (std = 0.014), with its repeat-seed mean (0.579) noticeably below its seed-42 result (0.616). Overall, this supports that the fine-tuning gains in § 5.3 are not an artifact of one initialisation, though CheXWorld’s result should be read with its higher seed-sensitivity in mind.

Conclusion

Across four JEPA-based chest X-ray encoders and six robustness tests, we found sharp, model-dependent degradation, with embedding similarity falling to 0.33 in the worst case (Figure 4), none of these encoders is robust to realistic corruptions or black-box attacks by default. Fine-tuning the predictor on clean and perturbed images together improved AUROC for most models and tests (Table 4), and this held across three random seeds (Figure 6). The Vector-Quantised codebook, trained on clean data only, told a different story: it fell below its own clean-image performance for every encoder, and degraded more than the plain Original predictor (Table 3), a discrete bottleneck alone was not enough without training on perturbed examples. Robustness should not be assumed from pretraining or architecture; it has to be measured directly.

Encoder-level embedding cosine similarity across all four severity levels of each perturbation test for each model

Figure 4. Encoder-level Embedding Cosine Similarity across all four severity levels of each perturbation test, each model, on the same held-out test subset.

References

  1. [1]Anas Anwarul Haq Khan, Mariam Husain, and Khitiji Jadhav. 2026. RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture. arXiv e-prints (2026), arXiv–2601.
  2. [2]Samuel G Armato III, Geoffrey McLennan, Luc Bidaut, Michael F McNitt-Gray, Charles R Meyer, Anthony P Reeves, Binsheng Zhao, Denise R Aberle, Claudia I Henschke, Eric A Hoffman, et al. 2011. The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans. Medical physics 38, 2 (2011), 915–931.DOI
  3. [3]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. 2023. Self-supervised learning from images with a joint-embedding predictive architecture. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 15619–15629.arxiv.org/abs/2301.08243
  4. [4]Sekofoeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Kornblith, Ting Chen, Nenad Tomasev, Jovana Mitrović, Patricia Strachan, et al. 2023. Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging. Nature Biomedical Engineering 7, 6 (2023), 756–779.DOI
  5. [5]Wieland Brendl, Jonas Rauber, and Matthias Behgler. 2017. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint arXiv:1712.04248 (2017).
  6. [6]Valay Bundele, Karahan Saritas, Bora Kargi, Oğuz Ata Çal, Kıvanç Tezoren, Zohreh Ghaderi, and Hendrik Lensch. 2024. Valuating self-supervised learning in medical imaging: A benchmark for robustness, generalizability, and multi-domain impact. arXiv preprint arXiv:2412.19124 (2024).
  7. [7]Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). IEEE, 39–57.arxiv.org/abs/1608.04644
  8. [8]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597–1607.arxiv.org/abs/2002.05709
  9. [9]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mustafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11292 (2020).
  10. [10]Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014).DOI
  11. [11]Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. 2020. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems 33 (2020), 21271–21284.arxiv.org/abs/2006.07733
  12. [12]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 15979–15988.arxiv.org/abs/2111.06377
  13. [13]Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 9726–9735.arxiv.org/abs/1911.05722
  14. [14]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.arxiv.org/abs/1512.03385
  15. [15]Zhixun He and Mukesh Singhal. 2024. VQUNet: Vector quantization U-Net for defending adversarial attacks by regularizing unwanted noise. In Proceedings of the 2024 7th International Conference on Machine Vision and Applications. 69–76.DOI
  16. [16]Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 (2019).
  17. [17]Yann LeCun et al. 2022. A path towards autonomous machine intelligence version 0.9. 2, 2022–06-27. Open Review 62, 1 (2022), 1–62.
  18. [18]Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Representations. https://openreview.net/forum?id=rJzIbZAbarxiv.org/abs/1706.06083
  19. [19]Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, and Irfan Essa. 2021. Discrete representations strengthen vision transformer robustness. arXiv preprint arXiv:2111.10493 (2021).arxiv.org/abs/2111.10493
  20. [20]Christos Matsoukas, Johan Fredin Haslum, Magnus Söderberg, and Kevin Smith. 2021. Is it time to replace cnns with transformers for medical images? arXiv preprint arXiv:2108.09038 (2021).arxiv.org/abs/2108.09038
  21. [21]Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013).arxiv.org/abs/1312.6199
  22. [22]National Lung Screening Trial Research Team. 2011. The national lung screening trial: overview and study design. Radiology 258, 1 (2011), 243–253.DOI
  23. [23]Jonathan Uesato, Brendan O’Donoghue, Pushmeet Kohli, and Aaron Oord. 2018. Adversarial risk and the dangers of evaluating against weak attacks. In International conference on machine learning. PMLR, 5025–5034.arxiv.org/abs/1802.05666
  24. [24]Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. Advances in neural information processing systems 30 (2017).arxiv.org/abs/1711.00937
  25. [25]Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammad H. Bagherian, and Ronald M Summers. 2017. ChestX-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2097–2106.DOI
  26. [26]Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. 2023. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific data 10, 1 (2023), 41.
  27. [27]Zhou Yang, Weisheng Dong, Xin Li, Mengluan Huang, Yulin Sun, and Guangming Shi. 2023. Vector quantization with self-attention for quality-independent representation learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 24438–24448.DOI
  28. [28]Zefan Yang, Ge Wang, James Hendler, Mannudeep K Kalra, and Pingkun Yan. 2026. X-win: Building chest radiograph world model via predictive sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6920–6930.arxiv.org/abs/2511.14918
  29. [29]Yang Yue, Yulin Wang, Chenxin Tao, Pan Liu, Shiji Song, and Gao Huang. 2025. Chexworld: Exploring image world modeling for radiograph representation learning. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 20778–20788.arxiv.org/abs/2504.13820

Paper details

vlnc:W_mxfrcxw6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents