From Natural Images to Medical Scans: Transferability of Diffusion Models for Medical Image Generation

Introduction

Pedagogical drawing textbooks decompose image creation into a small set of basic primitives, e.g., edges, contours, proportions, and light-and-shade, from which complex scenes can be composed [9] [46]. A related principle appears in visual neuroscience, where neurons in early visual cortex respond to low-level features such as oriented edges [22, 23], which are present in any visual input, whether a natural photograph or a medical scan.

Domain expertise is built on top of this shared perceptual basis. Radiologists, for example, acquire domain-specific pattern-recognition skills through extensive exposure to medical images [1, 65], but this advantage does not generally extend to non-radiological visual search tasks [47]. This suggests a distinction between broadly reusable visual primitives and learned domain-specific expertise.

A similar pattern motivates transfer learning in computer vision. Initialising from ImageNet (IN) [6] is the standard approach in supervised medical imaging. Early works on convolutional models find that transfer benefits are modest, mostly faster convergence and reuse of low-level (generic) features [25, 53]. Mustafa et al. [44] identify scale as a crucial factor for transferability. Matsuoka et al. [39] extend their analysis to convolutional [13, 32] and transformer [8, 64] models and find feature reuse to be the main source when transfer helps, especially for smaller datasets and closer domains. Interestingly, transformers rely on feature reuse more than convolutional networks. In self-supervised (SSL) medical imaging, natural-image encoders are increasingly used as initialisation or off-the-shelf, at times matching domain-specific models [24, 67]. For generative modelling, however, and diffusion denoisers in particular, the role of transfer remains underexplored.

State-of-the-art diffusion generative models fall into two broad paradigms: pixel diffusion and latent diffusion. Pixel diffusion applies the denoiser directly to pixels. Latent diffusion pairs a tokeniser with a denoiser that operates in its latent space. In latent diffusion, natural-image priors can be transferred through the tokeniser, the denoiser, or both. In the medical domain, natural-image tokenisers have been reused, and a few works fine-tune pretrained text-to-image denoisers [3, 70]. Yet denoiser transfer remains uncommon [16, 21, 36, 52, 66] and is treated as an implementation detail rather than a studied phenomenon. Whether naturalimage denoiser transfer is beneficial for medical image generation, and which aspects of the pretrained model account for any observed benefit, remain open questions.

In this work, we study natural-image denoiser transfer for medical image generation. Our analysis revolves around two broad questions: do natural denoisers transfer to the medical domain, and if so, what is it that transfers and how? We address these across latent and pixel-space, model sizes (131M–952M), and two radiology datasets (RADIN; BRATS). Our results show a positive transfer, and suggest that the benefit is tied to natural-image initialisation and manifests as better spatial organisation of the internal denoiser representations. We centre our analysis on pixel denoisers, with parallel results for latent ones.

Our main findings are:

1. Natural-image initialisation improves generation in both paradigms. Pixel-space models benefit most, improving by 21–34% gFID against 14% in latent space, where the tokeniser carries much of the prior. FID gains are mirrored in radiomic Fréchet distance (FD-RAD) and pathology classification MACROF1, both proxies for clinical fidelity.

2. Transfer benefits reflect spatial structure. Semantic probing shows that linear separability does not predict generation quality, while a modified spatial structure metric for diffusion representations, LDSC, tracks gFID at $|r|=0.85$, positioning spatial organisation as a channel through which pretraining manifests.

3. Mixed-domain experiments show that the denoiser computes similar layer-wise representations for natural and medical inputs through most of the network, and mixing natural data into medical training improves medical fidelity.

4. The benefit is tied to natural-image initialisation. It cannot be recovered by medical pretraining or several reinitialisation schemes. Illustrative extrapolation suggests far more medical data would be needed to close the gap.

Related Work

Diffusion and Flow Models

Diffusion [18, 59, 60] and flow models [35, 37] generate data from noise. To do so, they are trained to reverse a forward noising process of data towards noise. The networks that approximate this denoising process are denoisers.

Latent-space denoising. Latent denoisers operate in the latent space of a visual tokeniser, typically a Variational Autoencoder (VAE) [27], which reduces spatial dimensionality and computational cost relative to pixel-space diffusion. Esser et al. [10] propose a VQ-GAN to compress the image space into a discrete codebook, trained with additional perceptual and patch discriminator [14] loss. Rombach et al. [54] use a continuous, KL-regularised VAE latent space with a U-Net denoiser [55]. VA-VAE [69] improves the latent space structure while achieving larger spatial compression (16×), by aligning it with DiTNV2 features [48]. On the denoiser side, Peebles et al. [50] replace the U-Net backbone with a Diffusion Transformer (DiT), and SiT [38] extends it to flow objectives. LightningDiT (LDiT) [69] proposes training and architectural advancements on top of the vanilla DiT architecture. Here, we couple LDiT with different VAEs, and study whether the denoiser benefits from natural-image transfer beyond the prior already provided by the tokeniser.

Pixel-space denoising. Pixel diffusion applies the denoiser directly to pixels, avoiding the VAE reconstruction (decoder) bottleneck. Early diffusion models operate in pixel space [18], but the high dimensionality led to the popularisation of the latent paradigm [10, 54]. Pixel-based approaches have also been used as multi-stage cascades for increasing resolution [19, 56]. Only recently, JiT [33] revived single-stage pixel diffusion with a plain transformer that denoises large pixel patches (16–32), carrying over advances from DiT and an x-prediction objective. We use JiT as our pixel-space denoiser, and, to the best of our knowledge, are the first to transfer it to the medical domain.

Representation Alignment & Joint denoising. REPA [72] aligns diffusion features to a pretrained encoder, and iREPA [58] finds that spatial structure, rather than semantics, drives alignment gains. We observe an analogous pattern for natural-image initialisation. Joint-denoising methods model images and features together. ReDi [28] couples VAE latents with PCA DINOv2 features, and REGLEU [51] denoises compressed DINOv2 features concatenated with VAE latents. Latent Forcing [2] brings joint denoising to pixel space, and V-Co [34] jointly denoises pixels and features with an MMDiT [11] architecture. We employ V-Co as a joint image-feature denoising baseline and, to the best of our knowledge, are the first to study this paradigm for medical image generation.

Medical-domain Generation and Transfer

Medical generative modelling. Diffusion and flow models are widely used for medical image generation. In 3D, a U-Net [55] denoiser is typically trained in 3D VQ-GAN [10] (discrete) or VAE [27, 54] (continuous) latents [26, 43], directly in voxel space [7, 68], or in wavelet space [12]. At scale, MAISI [16] generates 3D CT with a latent U-Net, and MAISI v2 [74] reuses MAISI’s VAE and U-Net, and replaces the objective with rectified flow [37]. 3D MedDiffusion [66] pairs a custom VAE with a latent denoiser trained from scratch on medical data. 2D slices can also be stacked into 3D volumes [31]. In 2D, most works train the denoiser from scratch [2, 30, 36, 42, 49, 52], yet a few fine-tune a pretrained text-to-image model, e.g., RoentGen [3], MediSyn [4], and Med-Art [15], fine-tune pretrained text-to-image models. In histopathology, PathLDM [70] initialises its denoiser from IN weights, improving generation, and ZoomLDM [71] follows a similar recipe. Most works treat transfer as an implementation detail. We instead isolate the denoiser and study it systematically across size and paradigms.

Transfer learning in medical imaging. Natural-image transfer is well studied in medical image analysis, where gains are often linked to faster convergence, scale, and feature reuse [25, 39, 44, 53]. More recently, SSL natural-image encoders have also been transferred as initialisation or off-the-shelf for medical tasks, with competitive results compared to domain-specific models [24, 67]. In contrast, we study transfer for generative modelling, focusing on whether natural-image denoisers transfer to medical generation and what transfers.

Experimental Setup

In this section, we describe our experimental setup, denoising schemes, datasets, evaluation protocol, initialisation schemes, and implementation details.

Denoising models

A denoiser $F(\cdot; W)$ with parameters $W$ differs across paradigms by what it operates on. Pixel-space models denoise noisy images directly, $F(x_t; W)$. Latent-space models first encode the image via a tokeniser, $z = E(x)$, and then denoise the noisy latent, $F(z_t; W)$. The joint modelling approach employed in this study also operates in pixel space and denoises the noisy image together with an auxiliary semantic-feature stream, $s = \Phi(x)$; we denote this as $F(x_t, s; W)$; $t$ denotes the diffusion timestep/noise level.

Latent denoising. We instantiate $F$ with LDiT-XL [69], a 28-layer (675M) latent diffusion transformer trained with a flow objective. We couple it with three tokenisers $E$ that produce latents $z$: i) SDVAE [54], the standard Stable Diffusion autoencoder trained on natural images; ii) VAVAE [69], a natural-image VAE aligned with DINOv2 [48]; and iii) MedVAE [63], our medical-domain VAE baseline.

Pixel denoising. We instantiate $F$ with JiT [33], a diffusion transformer that operates directly on noisy pixels $x_t$ and utilises x-prediction, and is trained via flow matching. We study three sizes, B/16, L/16, and H/16 (12, 24, 32 layers; 131M, 459M, 952M parameters), which allows us to examine phenomena across models scales.

Joint denoising. We instantiate $F$ with V-Co-B/16 [34], a joint denoising variant that operates on noisy pixels $x_t$ together with an auxiliary DINOv2 feature stream $s = \Phi(x)$.

Datasets

RadImageNet [40]. is a 2D radiology dataset covering CT and MR. Its CT and MR sub-collections comprise 530,534 slice PNGs spanning 5 (modality, body part) regions: CT abdomen, CT lung, MR hip, MR knee, and MR shoulder. Each region is further subdivided into per-pathology sub-folders, with 19 pathology classes remaining after discarding the rare ones (fewer than 100 samples) in our validation set.

BraTS [5, 41]. We use the 2024 BraTS post-treatment adult glioma cohort, where each subject has four co-registered, skull-stripped MRI modalities (T1, T1c, T2, T2FLAIR). From 1,621 training and 188 validation patients, we extract 226,940 training and 26,320 validation 4-channel samples. In the single-channel setup, channel flattening yields 907,760 training and 105,280 validation images. We detail more in the Supp.

Spatial Structure Metrics

Spatial (structure) similarity metrics (SSMs) describe a visual entity by its internal geometric layout, e.g., how each patch resembles other patches of the same image, rather than by appearance such as colour or texture [57]. Singh et al. [58] show, that this property, of the target representation (VFM), predicts generation quality under representation alignment [72]. We instead measure it on the internal representations of the denoiser itself.

LDS metric. Let $F_l(x_t) \in\mathbb{R}^{N \times D}$ be the denoiser representation at layer $l$ and noise level $t$, with $N$ patch tokens of dimension $D$. We measure self-similarity between tokens $i,j$ with the cosine kernel $K_{ij}$, and their spatial separation with the Manhattan distance $d_{ij}$. LDS [20] contrasts nearby ($R_n$) against distant ($R_f$) pairs,

where $i \ne j$, and $R_n$, $R_f$ are near and far thresholds.

LDSC metric. We modify LDS calculation by 1) mean-centering the tokens before computing the kernel and 2) excluding first and last blocks, whose representations remain close to the generation (pixel/latent) space, giving LDSC (a centered version of LDS).

Evaluation metrics

Generation metrics. We report FID [17] as our primary generation-quality metric. We additionally report density (Dens.) and coverage (Cov.) [45], more robust alternatives to precision and recall [29] that disentangle fidelity and diversity. We detail both in the Supp.

Spectral profile of generated images. As a fine-grained diagnostic, we compute the radial power spectrum of generated images and report the per-frequency ratio $P_{\mathrm{gen}}(f)/P_{\mathrm{real}}(f)$. A ratio of 1 matches the real power at that frequency band. This is a diagnostic, not a standalone metric. It is (more) informative for similar, within-family models, e.g., the same JiT architecture under different learning rates, initialisations or sizes.

Initialisation schemes

Whole-model initialisation. We compare three schemes: 1) IN loads the full IN-init denoiser weights; 2) RANDOM uses each model’s default random initialisation; and 3) Statistically-matched (ST) [53] resamples each tensor from a Gaussian matching the per-tensor mean and variance of the IN weights, preserving the weight statistics but not the learned structure. ST thus isolates how much of the benefit comes from statistics alone [39, 53].

Block reversion. Following [73], we take a fully fine-tuned denoiser, revert each transformer block to its original IN weights and measure the gFID change. A large change marks a block which the model relies on; a small one marks a block whose original IN weights remain useful.

Implementation details

Models and training. We use the pretrained VAVAE, SDVAE, and MedVAE tokenisers off the shelf. As VAVAE applies higher spatial compression, we pair it with LDiT-XL/1 (latent patch size 1), and the lower-compression SDVAE and MedVAE with LDiT-XL/2 (patch size 2). The denoisers are our object of study: we follow each repository’s recipe and hyperparameters for LDiT¹, JiT², and V-Co³, with pixel models JiT-{B,L,H}/16 and V-Co-B/16 using a 16×16 patch size. All models are trained at 256×256 resolution, with RADIN models conditioned on 5 body-part classes and BRATS models on 2 classes (tumour/no tumour). We train for 80 epochs on RADIN, 40 epochs on single-channel BRATS, and 55 epochs on multi-channel BRATS, unless otherwise stated. Additional details are provided in Supp.

Sampling. We follow each model’s recommended sampler and step budget: LDiT uses Euler ODE sampler with 250 steps, while JiT and V-Co use Heun with 50 steps. We use cfg=1.0 throughout to measure transfer effects without confounding from guidance tuning.

Evaluation. We compute all metrics with the dgm-eval⁴ suite [61], reporting gFID, DENS, and COV. We evaluate against each dataset’s held-out test split: 15k images for RADIN and 50k for BRATS. This split is not used for training or per-model hyperparameter tuning.

Natural-image transfer improves medical image generation

The case of RadImageNet

Table 1 reports generation metrics (gFID, DENS, COV) on RADIN across the three denoising paradigms.

Table 1. RadImageNet (RadIN) 256×256 gFID across the three denoising paradigms. rFID is the off-the-shelf tokenizer reconstruction FID; EP. is the number of RadIN training epochs; PARAMS reports trainable denoiser parameters only; $\Delta_{\mathrm{init}}$ and $\Delta_{\mathrm{init}}\%$ is the absolute and relative gFID improvement of IN (✓) vs RANDOM (✗) Denoiser, as indicated in INIT column. Bold = best per category.

DENOISER	TOKENIZER	INIT.	EP.	PARAMS	rFID↓	gFID↓	DENS.↑	COV.↑	$\Delta_{\mathrm{init}}/\Delta_{\mathrm{init}}(\%)$
LATENT DENOISING
LDiT-XL/2	MedVAE	✗	80	675M	6.35	21.33	0.64	0.72	—
LDiT-XL/2	SDVAE	✗	80	675M	2.63	19.15	0.69	0.80	—
LDiT-XL/1	VAVAE	✗	80	675M	1.88	19.05	0.79	0.85	—
LDiT-XL/1	VAVAE	✓	80	675M	1.88	16.43	0.77	0.86	2.6 / 13.8
PIXEL DENOISING
JiT-B/16	—	✗	80	131M	—	55.50	0.27	0.31	—
JiT-B/16	—	✓	80	131M	—	36.48	0.39	0.51	19.0 / 34.3
JiT-L/16	—	✗	80	459M	—	41.27	0.44	0.50	—
JiT-L/16	—	✓	80	459M	—	32.34	0.48	0.60	8.9 / 21.6
JiT-H/16	—	✗	40	952M	—	47.89	0.30	0.34	—
JiT-H/16	—	✓	40	952M	—	32.83	0.46	0.60	15.1 / 31.4
JOINT DENOISING
V-Co-B/16	DINOv2	✓	80	260M	—	28.86	0.44	0.56	—

Transfer helps, and most in pixel space. IN transfer consistently improves generation, but its magnitude differs across paradigms. In latent space, LDiT-XL/1 improves by 13.8%, whereas in pixel space JiT improves by 34.3% (B), 21.6% (L), and 31.4% (H), with density and coverage following suit. Latent thus attains the best absolute performance (16.43), while pixel gains most from transfer (21-34%). The tokeniser likely explains this gap: natural-image VAEs compress images spatially 8-16× and reconstruct radiology well (1.88 rFID VAVAE) despite no medical training, giving the denoiser 1) much of the transferable prior already, and 2) an easier, lower-dimensional problem. Together, these explain the latent denoiser’s strong performance (∼12 gFID over pixel/joint) and why IN-init adds a smaller yet significant, gain (2.62 gFID). Pixel diffusion has no tokeniser: the denoiser benefits from the prior directly, so transfer helps more. The joint V-Co (28.86 gFID) falls in the same performance regime. We therefore focus on pixel-space JiT, where transferability lies in the denoiser.

Larger capacity does not uniformly improve RadIN gFID. Within the JiT family, scaling from B/16 to L/16 improves gFID by over 4 points, and density and coverage by ∼0.1. Scaling to H/16 (952M), however, does not improve gFID. Thus, in our setup and data regime (∼0.5M images), increasing capacity beyond L/16 does not translate into a clear gain in generation metrics.

Spectral diagnostics show differences in generated image structure. In Figure 1 we use the spectral profile of generated images as a within-family diagnostic of image structure, across three comparisons. First, across JiT sizes: all models reproduce low-to-mid frequencies more closely than high ones, but while B/16 and L/16 undershoot high-frequency power, H/16 better preserves it. L/16 is in turn slightly better aligned in the low-to-mid bands, which may explain its better gFID; so although H/16 does not improve gFID on RADIN, it does change the frequency profile. Second, RANDOM vs IN-initialised JiT-L/16 shows that natural-image initialisation alters the generated spectrum. Finally, in latent space the difference between RAN-

DOM and IN LDiT-XL/1 is smaller, consistent with the tokeniser already providing a structured representation.

The case of BraTS

We next study transfer on BraTS, in single and multi-channel scenarios.

Table 2. BraTS-GLI 256×256 gFID across single- and multi-channel diffusion. (a) Single-channel: the denoiser randomly generates one of the four MRI modalities; (b) Multi-channel: the denoiser generates all four MRI modalities; EP. the number of training epochs; Bold = best per panel.

DENOISER	INIT.	EP.	gFID↓	DENS.↑	COV.↑	$\Delta_{\mathrm{init}}$
(a) Single-channel
LDiT-XL/1	✗	40	9.22	0.67	0.76	—
LDiT-XL/1	✓	40	7.66	0.68	0.78	1.56
JiT-L/16	✗	40	37.93	0.19	0.26	—
JiT-L/16	✓	40	29.34	0.25	0.36	8.59
JiT-H/16	✓	10	29.13	0.23	0.35	—
(b) Multi-channel
JiT-L/16	✗	55	43.97	0.09	0.14
JiT-L/16	✓	55	35.17	0.15	0.23
JiT-H/16	✓	55	32.75	0.18	0.26

Single-channel MRI reproduces the transfer pattern. IN initialisation improves both latent and pixel-space models: LDiT-XL/1 by 1.56 gFID and JiT-L/16 by 8.59, again a larger absolute gain in pixel space. JiT-H/16 reaches comparable gFID to JiT-L/16 in fewer epochs (10 vs. 40). Since H/16 has roughly twice the FLOPs of L/16,⁵ this corresponds to about half the compute.

Multi-channel generation benefits from capacity. Generating all four modalities jointly is more challenging than the single-channel task: JiT-L/16 gFID rises from 29.34 to 35.17. IN initialisation nonetheless remains beneficial, improving pretrained JiT-L/16 by 8.80 gFID over RANDOM. Unlike RadIN, additional capacity here improves all metrics, with JiT-H/16 outperforming JiT-L/16, suggesting capacity may matter more when jointly representing multiple co-registered MRI contrasts.

Do FID gains carry clinical signal?

To assess whether FID improvement translates to clinically relevant information we perform the following experiment. We first sample 100k samples from our training set. We then train two JiT-B/16 (scratch; IN) models on the whole RadIN, conditioned on all (67) classes, and generate 100k samples over the 19 (validation) classes. This way we get three 100k pools with identical pathology distributions.

FD-RAD. In Table 3, we use 93 radiomic features [62], a radiology feature space capturing low-level image properties and measure the Fréchet Distance from the real (100k) radiomic distribution. We find that lower gFID translates to lower FD-RAD, i.e., more clinically meaningful images.

Table 3. Clinical Proxies on synthetic data. We measure FD-RAD and MACROF1 on two synthetic 100k pools.

DENOISER	INIT.	gFID↓	FD-RAD↓	MACROF1↑
JiT-B/16	✓	40.81	37.99	0.47
JiT-B/16	✗	72.67	87.46	0.43

Pathology Classification We train ResNet-50 models on real and synthetic data as a proxy to clinical (19-class pathology) classification. The classifier trained on the real data pool achieves 0.72. For the synthetic pools (Table 3) we find that lower gFID translates to better MACROF1, pointing towards more meaningful clinical generation.

What transfers: spatial structure, not semantics

Since the pixel denoiser carries the entire prior, we focus on pixel-space JiT and ask whether the benefit reflects high-level (semantics) or low-level (spatial) structure.

Semantics do not predict generation quality

Following [72], a hypothesis is that pretraining helps via semantics, either by transferring semantic features or by providing an initialisation from which it learns stronger semantics. We probe frozen denoiser representations to classify the 19 pathology classes of RADIN. Since the generative models are conditioned on 5 broad anatomy classes, probes capture semantic information beyond generation labels.

Higher probe accuracy does not translate to better generation. JiT-B/16 from scratch probes similar to its IN counterpart (0.56) yet generates far worse samples (55.50 vs 36.48 gFID, Figure 2). The same holds across model sizes (L/H see Supp.), with all models performing comparably. JiT-B/16 probes slightly higher than JiT-H/16 (0.56 vs 0.54) but has worse gFID (36.48 vs 32.83). In latent space the scratch model also probes higher than the IN-init (0.557 vs 0.510) while generating worse (Supp.). Semantic separability cannot explain the transfer benefit.

Spatial structure tracks generation quality

An alternative hypothesis is that the benefit comes from spatial (low-level) structure [39, 58]. To test this, we collect all JiT models trained in this paper, i.e., pretrained, scratch, mixed (subsection 5.3), and limited-data (subsection 6.4). Applying LDS (subsection 3.3) to frozen denoiser representations gives inconclusive results. We modify LDS for denoiser representations, giving LDSC, and track gFID at $|r|=0.846$ over 12 models (Figure 3). The fitted law is $\Delta\mathrm{gFID}=-296.6\times\Delta\mathrm{LDSC}$. We ask how much of the pretrained-scratch gFID gap this rule predicts from the LDSC gap alone. We multiply the observed LDSC gap by the fitted slope to obtain a predicted gFID gap, and divide by the actual gFID gap to get the fraction that the LDSC signal captures. For B / L / H it captures 92 / 58 / 52%, or 67% on average. Two checks confirm this: 1) Monte-Carlo perturbation of every measured gFID by $\sigma=1$ (more than the seed noise), refitting over 5k runs gives $69\pm6\%$; 2) Removing both IN and scratch models of a size before refitting and predicting that size’s gap gives 95/62/42%, again 2/3 on average. LDSC predicts about two-thirds of the pretraining benefit. Better spatial organisation is one channel through which pretraining manifests.

Mixed-domain training

Early-to-mid representations are shared across domains We further test the structural hypothesis and train JiT-L/16 jointly on RADIN and 10% IN (mixed-JiT-L/16), so a single model represents both domains internally, and measure the per-block cosine similarity of its representations (concatenated over the token sequence) under natural and medical inputs. Figure 4 shows that early-to-mid blocks (0–16) stay highly similar across domains ($\cos\geq0.7$), so the model computes similar representations for both domains there. Deeper blocks (17–22) behave differently: cross-domain similarity drops ($\cos\approx0.35$–0.50), suggesting domain-specific behaviour.

Natural-image mixing helps medical fidelity Since both domains yield similar representations, we sweep the ratio of natural images added to the mix. gFID improves across all ratios (Table 4), with the largest gain (1.4 gFID) at 15%. Even at 1:1 we observe a small improvement, though its magnitude sits near gFID noise.

IN mix (%)	0	5	15	25	50
RADIN:IN	1:0	19:1	5.7:1	3:1	1:1
gFID$\downarrow$	32.34	31.46	30.94	31.01	31.80

The benefit is tied to natural-image initialisation

We ask whether the benefit of IN initialisation can be reproduced without it, or decomposed into simpler parts. To reproduce it, we test in-domain medical pretraining, statistically matched reinitialisation, and medical data scaling; none matches IN transfer in our setup. To decompose it, we compare parameter drift with block reversion, and find that drift does not predict functional importance.

Natural-image transfer outperforms medical pretraining

A natural alternative to IN-init is to pretrain on medical data. To our knowledge, no 2D radiology denoiser weights exist at a compatible architecture, so we train our own.

Single-channel target A JiT-B/16 pretrained from scratch on RADIN for 160 epochs and fine-tuned on single-channel BRATS for 20 epochs is 5 gFID worse than direct IN fine-tuning (Table 5). A RADIN stage on top of IN helps slightly, but within seed noise ($\sigma_{\mathrm{IN}}=0.41$, Supp.).

DENOISER (INIT)	PRETRAIN	EP.	gFID↓	DENS.↑	COV.↑
JiT-B/16 (RAND)	RADIN	160	36.73	0.209	0.277
JiT-B/16 (IN)	RADIN	80	31.13	0.221	0.320
JiT-B/16 (IN)	—	—	31.73	0.218	0.322

Multi-channel target. We test whether an intermediate single-channel medical stage helps before transferring to multi-channel BRATS (Table 6). Direct IN transfer is the best (40.17). Inserting a medical stage only hurts, giving 50.90 after single-channel BRATS and 75.40 after RADIN.

DENOISER	PRETRAIN	EP.	gFID↓
JiT-L/16	RADIN	30	75.40
JiT-L/16	BRATS	30	50.90
JiT-L/16	—	30	40.17

Per-tensor weight statistics are not sufficient

If IN only set a suitable per-tensor scale, statistically matched (ST) initialisation would recover its benefit. Instead, ST is worse than RANDOM in both paradigms. Table 7 shows gFID rises from 41.27 to 62.64 for JiT-L and from 19.05 to 51.45 for LDiT-XL. The benefit comes from the specific weight values, not their distribution.

DENOISER	RANDOM	ST	IN
LDiT-XL/1	19.05	51.45	16.43
JiT-L/16	41.27	62.64	32.34

Parameter drift and functional importance

We ask whether a layer’s change during training reflects its importance. We measure change with parameter drift and importance with block reversion.

Larger models drift less. Figure 5 (LEFT) shows that drift increases with depth. Early blocks remain closer to their IN-init, while deeper blocks move further. Averaged over blocks, drift decreases with model size: 12.10 (B), 8.80 (L), and 6.98 (H) ($\times10^{-3}$), so larger models adapt less.

Functional importance is not captured by drift alone. Block reversion (Figure 5 RIGHT) gives a different picture from parameter drift. Reverting the first transformer block to its IN weights is by far the most damaging, raising gFID by $\sim170$, even though this block drifts less than the deeper ones. The second and four final blocks are also important. Several intermediate blocks can be reverted with smaller changes in gFID. To account for any activation mismatch we also perform activation renormalisation after block reversion in the Supp. Results follow similar trend, thus drift cannot capture functional importance.

The gap is not closed by data scaling

A final hypothesis is that more medical data lets the model learn from scratch what transfer provides.

Limited data analysis We train JiT-B/16 from both scratch and IN init on 25%, 50%, and 100% of RADIN (Figure 6). The pretrained model leads at every scale, the gap narrows (from 30.0 to 19.0) but stays large. IN on 25% outperforms scratch on 100% (42.5 vs 55.5), while scratch fails to converge at 25%.

Data requirements extrapolation Fitting $\mathrm{FID}\propto N^{-\alpha}$ to each model, gives $\alpha_{\mathrm{Rand}}=0.29$ and $\alpha_{\mathrm{IN}}=0.11$. If scratch improves at $\alpha_{\mathrm{Rand}}$ until it reaches the fidelity of IN at 25%, and at $\alpha_{\mathrm{IN}}$ thereafter, it needs about 10$\times$ more data to close the gap (an illustrative estimate).

Conclusion & Discussion

We present a systematic study of natural-image denoiser transfer for medical image generation. IN transfer consistently improves generation performance across latent

and pixel space, model scales, and two radiology datasets. Gains are also mirrored in radiomic distance and pathology classification. Models trained from scratch reach similar semantic probe accuracy yet generate worse samples, whereas spatial organisation, measured by LDSC, correlates with gFID at $|r|=0.85$, positioning spatial structure as a channel through which pretraining manifests. Although natural and medical images differ in high-level content, a mixed-domain denoiser computes similar representations for both through most of the network, and mixing natural images into medical training improves fidelity. The benefit is tied to natural-image initialisation, and no alternative we test reproduces it, including in-domain medical pretraining. From a generative modelling lens, natural images are very useful for medical image generation.

Limitations and future work. LDSC predicts about two-thirds of the pretraining benefit, so spatial organisation is one channel through which it manifests, but might not be the full account. What exactly is inherited (edges, textures or contours), and how architecture, data, training budget and optimisation contribute, remains open. Moreover, whether the benefits of natural images can be learned from medical data alone remains an open question. Our extrapolation scales data quantity, not diversity. Furthermore, our clinical evidence relies on proxies, and expert clinical evaluation is needed before any claim of clinical utility. Finally, we study 2D generation, the native format for X-ray and histopathology, in which even volumetric scans are read slice by slice. Extending to 3D generation is a natural next step.

Acknowledgements

This work was supported by the Swiss National Science Foundation (TMSGI3_225913, CRSK-3_229103) and the Chad Tough Defeat DIPG Foundation. We acknowledge EuroHPC Joint Undertaking for awarding us access to Discoverer+ (project EHPC-DEV-2026D06-289), hosted by PetaSC at Sofia Tech Park, Bulgaria, and co-funded by EuroHPC JU and the Bulgarian government. E. Georgiou would like to thank T. Kouzelis for connecting him with I. Kakogeorgiou.

Additional Results

Tokenizer	BraTS	RadIN
Natural-image
SDVAE	2.27	2.63
SD3.5VAE	0.48	0.92
VAVAE	1.34	1.88
RAEv2 (DINOv3-L-k7)	1.60	2.42
Medical-domain
MedVAE	6.43	6.35

Theoretically, an optimal compressor approaches the Kolmogorov complexity of its training data: it learns patterns rather than memorising, and reuses them to compress OOD data. We find natural VAEs compress radiology images very well with no medical training. We treat this as another positive transfer result, and hypothesise MedVAE’s performance stems from limited data and optimisation, e.g., training epochs, regularisation. We do not claim that medical data and pretraining do not help in general. Our illustrative extrapolation suggests that closing the gap would need far more medical data, whereas natural images are abundant and transfer to radiology.

Table 9. Seed variance. gFID on RadIN after 80 epochs. $\delta$ is the range (max−min) and $\sigma$ the standard deviation across seeds. IN seeds share a fixed initialisation and vary only in data order, noise, etc., which likely explains why $\sigma_{\mathrm{Rand}} > \sigma_{\mathrm{IN}}$.

Seed	JiT-B/16 Rand.	JiT-B/16 IN	LDiT-XL/1 Rand.	LDiT-XL/1 IN
0	55.50	36.48	19.05	16.43
1	56.30	35.71	18.62	16.28
2	54.27	36.33	—	—
Mean	55.36	36.17	18.84	16.36
$\delta$	2.03	0.77	0.43	0.15
$\sigma$	1.02	0.41	-	-

METHOD	$L_0$	$L_1$	$L_{20}$	$L_{23}$
REINIT	+167.4	+60.7	+33.8	+48.5
+ RENORM	+101.0	+63.1	+14.3	+19.0

Datasets

RadImageNet

We use the RADIMAGENET as our primary diverse and large 2D radiology dataset. We were given access to its CT and MR sub-collections. Our subset covers five anatomical regions: CT abdomen, CT thorax (lung), MR hip, MR knee, and MR shoulder. Each region is further subdivided into per-pathology sub-folders, for a total of 71 pathology classes, with 67 of them having more than 100 samples.

The provided set consists of 530,534 2D slice PNGs. We randomly hold out 15,000 images as validation and train on the remaining 515,534. Preprocessing involves center-cropping and bicubic upsampling to $256 \times256$. The class-conditional generator is conditioned on the five (modality, body part) region labels. The finer pathology taxonomy enters only at evaluation, where the linear-probe analysis restricts to the 19 pathology classes by filtering out rare (long-tail) pathology classes with fewer than 100 samples in our validation set.

REGION	CLASS
CT ABD	renal_lesion
CT ABD	liver_lesion
CT ABD	bowel_abnormality
CT ABD	bowel_inflammation
CT ABD	post-op
CT LUNG	Airspace_opacity
CT LUNG	Nodule
CT LUNG	interstitial_lung_disease
MR KNEE	chondral_abnormality
MR KNEE	meniscal_abnormality
MR KNEE	soft_tissue_fluid_collection
MR KNEE	bone_inflammation
MR KNEE	acl_pathology
MR KNEE	mcl_pathology
MR KNEE	intra
MR SHOULDER	labral_pathology
MR SHOULDER	soft_tissue_fluid
MR SHOULDER	soft_tissue_edema
MR SHOULDER	supraspinatus_pathology

BraTS. We use the 2024 BraTS post-treatment adult glioma cohort, contributed by seven academic medical centers. Each subject is acquired with four co-registered, skull-stripped 1 mm isotropic multi-parametric MRI volumes covering the standard clinical contrasts: native T1 (T1), post-contrast T1-weighted (T1c), T2-weighted (T2), and T2 fluid-attenuated inversion recovery (T2FLAIR). BraTS consists of 1,621 training patients (6,484 3D mpMRI volumes). The official validation split contributes a held-out cohort of 188 patients (752 volumes).

We flatten each volume to 2D by keeping the axial slices 15–154 (140 per volume) and bicubic-upsampling from $192 \times192$ to $256 \times256$. Every multi-channel (patient, slice) pair is stored as a single $4 \times256 \times256$ tensor in which the four co-registered modalities occupy the channel dimension, yielding 226,940 training and 26,320 validation multi-channel samples.

We use the data in two ways. In the 4-channel setup, each slice is one 4-channel image. In the single-channel setup, each modality is a separate grayscale image (replicated to three channels for IN initialisation), which gives 907, 760 training and 105, 280 validation single-channel images.

Spatial structure metrics

LDS and LDSC: implementation

We follow the definitions of subsection 3.3 and use the reference implementation of [58]. JiT tokenises a $256 \times256$ image into a $16 \times16$ grid of $N=256$ patch tokens. In-context (class) tokens are removed before computing the metric. Near pairs are adjacent tokens ($d_{ij}=1$, i.e., $R_n=2$; 4 per interior token) and far pairs have $d_{ij} \geq R_f=6$ (195 per central token), following [58] (Figure 10 a,Figure 10b). A high LDSC means that nearby patches, e.g., the fur of a bird or a tissue in an MRI scan, have similar representations, while distant patches differ. For LDSC, we mean-centre the tokens of each image before computing the cosine kernel, so that LDSC is the near term minus the far term of the centred tokens (Figure 10c). We compute LDSC for every block at $t=0.5$ and report the peak over blocks, excluding the I/O blocks (Figure 10 d).

Robustness to the I/O exclusion

The main paper excludes only the first and last block. However, in deeper models the near-input region spans more than one block. For example, the $L/16$ and $H/16$ scratch denoisers reach their highest LDSC in block 1, directly after the input, where representations remain close to the pixel space (Figure 10d, Figure 11 top). This reduces their IN vs. scratch gap (2.8% and 4.3% for $L$ and $H$, vs. 10.3% for $B$). An alternative is to exclude a fixed fraction of depth, e.g., 5% or 10% at each end, i.e., removing these blocks and focusing on the mid-layers. The scratch peaks shift into the trunk ($L/16$ block 15, $H/16$ block 19 at 10%), the gap becomes similar across sizes (10.3/14.3/13.9% at 10%), and the correlation with gFID increases from $\lvert r\rvert=0.846$ to 0.903 and 0.945 (Figure 11 bottom). The IN peaks do not change. Therefore, our conclusions hold under all three rules, and the first/last-block rule is the conservative choice.

Evaluation Metrics

Evaluation setup. All quantitative metrics are computed with the dgm-eval suite, using the INCEPTION-V3 feature extractor. Evaluation features are extracted in fp32.

FID Measures the feature distance between the distributions of real and generated images. It uses the INCEPTION-V3 network and computes the Fréchet distance under the assumption that both feature distributions are multivariate Gaussian. Lower is better.

Density & Coverage Refine precision / recall to be more robust to outliers and large feature variance. DENS. counts how many real-sample $k$-NN neighbourhoods contain each generated sample (fidelity); Cov. is the fraction of real samples whose neighbourhood contains at least one generated sample (diversity). Higher is better.

Spectral profile. Diagnostic, not a headline metric. We take the 2D-FFT of each image, radially average the power to obtain $P(f)$ as a function of spatial frequency, and plot the ratio $P_{\mathrm{gen}}(f)/P_{\mathrm{real}}(f)$. 1 matches the real spectrum at that frequency, $<1$ is under-production, $>1$ is over-shoot. We only use it for within-family comparisons. We do not compare across architectures, because their spectra differ for reasons unrelated to transfer.

	LDiT-XL/2	LDiT-XL/2	LDiT-XL/1
TOKENISER	MedVAE	SDVAE	VAVAE
ARCHITECTURE
Tokeniser downsample	$8\times$	$8\times$	$16\times$
Tokeniser latent channels	4	4	32
Latent grid ($256\times256$ in)	$32\times32$	$32\times32$	$16\times16$
Latent patch size	2	2	1
Token count	256	256	256
Transformer blocks	28	28	28
Hidden dim	1152	1152	1152
Attention heads	16	16	16
MLP ratio	4	4	4
QK-norm	✓	✓	✓
Activation	SwiGLU	SwiGLU	SwiGLU
Positional emb.	RoPE	RoPE	RoPE
Normalisation	RMSNorm	RMSNorm	RMSNorm
Trainable params	675M	675M	675M
OPTIMISATION & TRAINING
Transport	rect. flow	rect. flow	rect. flow
Prediction parameterisation	velocity (v)	velocity (v)	velocity (v)
Loss	MSE on v	MSE on v	MSE on v
Velocity Directional Loss (aux)	✓	✓	✓
t-sampling	logit-normal ($\mu=0,\sigma=1$)	logit-normal ($\mu=0,\sigma=1$)	logit-normal ($\mu=0,\sigma=1$)
Optimizer	AdamW	AdamW	AdamW
$\beta_1,\beta_2$	0.9, 0.95	0.9, 0.95	0.9, 0.95
Grad-norm clip	1.0	1.0	1.0
Learning rate (LR)	$2\times10^{-4}$	$2\times10^{-4}$	$2\times10^{-4}$
LR schedule	constant	constant	constant
Warm-up	none	none	none
Effective batch size	384	384	384
Precision	bf16	bf16	bf16
SAMPLING
Sampler	Euler ODE	Euler ODE	Euler ODE
Sampling steps	250	250	250
CFG scale	1.0	1.0	1.0

Medical-image tokenizer. Natural-image tokenizer.

	JiT-B/16	JiT-L/16	JiT-H/16
ARCHITECTURE
Patch size	$16 \times16$	$16 \times16$	$16 \times16$
Token count ($256 \times256$ in)	256	256	256
Transformer blocks	12	24	32
Hidden dim	768	1024	1280
Attention heads	12	16	16
MLP ratio	4	4	4
QK-norm	✓	✓	✓
Activation	SwiGLU	SwiGLU	SwiGLU
Positional emb.	sin-cos + RoPE	sin-cos + RoPE	sin-cos + RoPE
Normalisation	RMSNORM	RMSNORM	RMSNORM
Trainable params	131M	459M	952M
OPTIMISATION & TRAINING
Transport	rect. flow	rect. flow	rect. flow
Prediction parameterisation	$x$-prediction	$x$-prediction	$x$-prediction
Loss	MSE on $v$	MSE on $v$	MSE on $v$
$t$-sampling	logit-normal ($\mu=-0.8$, $\sigma=0.8$)	logit-normal ($\mu=-0.8$, $\sigma=0.8$)	logit-normal ($\mu=-0.8$, $\sigma=0.8$)
Optimizer	AdamW	AdamW	AdamW
$\beta_1$, $\beta_2$	0.9, 0.95	0.9, 0.95	0.9, 0.95
Learning Rate (LR)	$5 \times10^{-5}$	$5 \times10^{-5}$	$5 \times10^{-5}$
LR schedule	constant	constant	constant
Warm-up	5 epochs	5 epochs	5 epochs
Effective batch size	288	288	144
Precision	bf16	bf16	bf16
SAMPLING
Sampler	Heun	Heun	Heun
Sampling steps	50	50	50
CFG scale	1.0	1.0	1.0