MedVTok: A General-Purpose Medical Visual Tokenizer

Introduction

Multimodal medical AI aims to integrate heterogeneous patient evidence, including laboratory measurements, longitudinal records, and medical images, to form a holistic understanding of a patient’s condition. As the field shifts from isolated task-specific models toward generalist multimodal systems [1, 2, 3], a central requirement is a shared interface through which diverse clinical evidence can be encoded, transferred, and reused across tasks. Among these evidence sources, medical images are particularly rich and high-dimensional, capturing anatomical structures, pathological findings, and disease progression. Visual tokenization therefore serves as a critical front end that converts raw imaging data into compact visual tokens compatible with multimodal models.

Preprint.

However, medical imaging data are highly heterogeneous across both modality and dimension, including 2D images such as dermoscopic images, color fundus photographs (CFP) and histopathology patches, as well as 3D volumes such as computed tomography (CT) and magnetic resonance imaging (MRI). These data differ in acquisition protocol, anatomical coverage, and diagnostic content. Existing medical visual tokenizers are often tied to a specific modality and dimension, producing fragmented representations that are difficult to reuse in a multimodal system. Consequently, downstream models must absorb much of the burden of modality adaptation and cross-modality representation learning [4]. A general-purpose medical visual tokenizer should instead provide a single token interface across modalities while natively supporting both 2D images and 3D volumes.

Building such a tokenizer poses two key challenges. (i) Preserving structural fidelity across dimensions. Unlike 2D images, 3D scans present volumetric continuity, where organs, vessels, lesions, and tissue boundaries extend across adjacent slices. Existing tokenizers often specialize in either 2D or 3D settings [5, 6], or are optimized for local appearance fidelity or per-pixel/per-voxel reconstruction [7, 8], failing to encode inter-slice structure. (ii) Capturing multimodal semantics across modalities. Different modalities may describe related clinical concepts while exposing distinct evidence, e.g., histopathology reveals cellular and tissue-level morphology, whereas CT and MRI capture organ-level anatomy. Existing semantic alignment strategies typically learn representations of different imaging modalities from a single pretrained semantic encoder [6, 9, 10, 11], which may underrepresent modality-specific semantics outside that encoder’s representation space.

In this paper, we introduce MedVTok, a general-purpose medical visual tokenizer that maps heterogeneous 2D images and 3D volumes from diverse imaging modalities into a unified token space. To preserve structural fidelity across dimensions, MedVTok introduces Slice Differential Regularization (SliDR) in addition to per-pixel/per-voxel supervision, which explicitly models inter-slice volumetric continuity. To capture clinical semantics across modalities, MedVTok proposes Multi-Expert Representation Alignment (MERA), which builds a shared representation space integrating knowledge from multiple modality-specialized experts instead of a single one, while retaining complementary semantics in each expert’s representation space. Together, they enable MedVTok to produce visual tokens that preserve structural fidelity and encode multimodal clinical semantics.

To train and evaluate MedVTok at scale, we curate a large-scale medical imaging corpus spanning 9 modalities, with over 28M 2D images and 70K 3D volumes. We assess MedVTok across 6 task types: classification, segmentation, retrieval, synthesis, visual question answering, and medical report generation, covering broad utility across 24 benchmarks. Our results show that MedVTok achieves state-of-the-art performance, demonstrating the value of a scalable token interface that unifies medical image dimension, modality, and downstream task usage.

Contributions. Our main contributions are as follows. (i) We introduce MedVTok, a general-purpose medical visual tokenizer that provides a shared token interface across diverse imaging modalities and natively supports both 2D images and 3D volumes. (ii) We propose a multi-level supervision strategy that combines slice differential regularization for inter-slice continuity with multi-expert representation alignment for cross-modality clinical semantics. (iii) We curate a large-scale medical imaging corpus and conduct comprehensive evaluations across 24 benchmarks, demonstrating the broad utility of MedVTok across classification, segmentation, retrieval, synthesis, visual question answering, and medical report generation.

Related Work

Tokenization across 2D and 3D medical imaging. Visual tokenizers convert images into compact tokens while preserving structural fidelity [7, 12]. Recent tokenizers [9, 13] further explore how a single tokenizer can handle heterogeneous visual formats such as images and videos. In medical imaging, however, tokenizers are commonly tied to specific input dimensionalities [5, 6]. A few works have explored adaptation to different dimensionalities by pooling 3D weights into 2D [8] or by duplicating 2D images to fit a 3D model [14]. While these approaches can handle 2D and 3D inputs simultaneously, they do not consider the respective properties of each dimensionality, e.g., through-plane anatomical continuity in volumes, and thus cannot fully preserve structural fidelity across input forms. In contrast, MedVTok introduces Slice Differential Regularization to explicitly preserve anatomical continuity in volumetric data, and provides a unified 2D–3D token interface via dimension-agnostic patchification.

Semantic representation across medical imaging modalities. Medical image representation learning has produced both broad biomedical encoders and modality-specialized expert models. General medical encoders [15, 16, 17, 18, 19] learn transferable representations from large-scale biomedical data, offering broad coverage across medical domains but potentially missing specialty-specific cues. Meanwhile, specialized encoders capture finer clinical semantics within individual imaging modalities or specialties [14, 20, 21, 22, 23, 24, 25, 26], yet remain isolated in different representation spaces. Existing tokenizers [5, 6] have only partially captured these different semantics, e.g., MedITok [6] aligns its representations with those of a single semantic encoder [15], thereby inheriting the single-expert bias and limiting the learning of complementary knowledge across modalities. MedVTok instead uses Multi-Expert Representation Alignment to integrate knowledge from multiple medical experts into a single token space, while retaining modality-specific semantics.

Method

Overview of MedVTok

MedVTok learns a unified latent space that captures both structural fidelity and clinical semantics for 2D and 3D data via two complementary supervision mechanisms: (1) Slice Differential Regularization, which preserves volumetric anatomical continuity beyond what per-voxel reconstruction can capture (§3.2), and (2) Multi-Expert Representation Alignment (MERA), which integrates clinical knowledge from multiple modality-specific expert encoders into the shared token space (§3.3).

Figure 2 illustrates the training framework. Specifically, given an input image or volume $x \in\mathbb{R}^{C \times T \times H \times W}$, with $T=1$ for images and $T>1$ for volumes, MedVTok first partitions it into $N=\frac{THW}{P_tP_hP_w}$ non-overlapping patches of size $C \times P_t \times P_h \times P_w$, where $N=\frac{THW}{P_tP_hP_w}$. Patches from different samples in a batch are joined into a sequence, where each patch is associated with a coordinate that records its position within the original input. This enables variable-resolution and mixed-dimension batching.

Then, the transformer encoder $E$, with rotary position embeddings (RoPE) [27] derived from the coordinates, maps the sequence to per-patch tokens $h=E(x)\in\mathbb{R}^{N\times d}$. These tokens are, on the one hand, regularized by a KL divergence term $\mathcal{L}_{\mathrm{KL}}$ to impose a standard Gaussian constraint after projection and reparameterization; on the other hand, they are processed by the decoder $D$ to reconstruct the original input $\hat{x}$. The reconstruction is supervised by the reconstruction loss $\mathcal{L}_{R}$ and the proposed slice differential regularization $\mathcal{L}_{\mathrm{SliDR}}$. In parallel, an aggregation head $f$ aggregates the tokens $h$ into a holistic representation $r=f(h)\in\mathbb{R}^{d}$, ³ which is used for semantic alignment via MERA.

Slice Differential Regularization

In tomographic volumes such as CT and MRI scans, adjacent slices share strong anatomical context: organs span continuously across the depth axis, and their boundaries shift gradually from slice to slice. Per-voxel reconstruction losses like $\ell_1$ can produce outputs that are individually accurate yet exhibit through-plane discontinuities, compromising diagnostic utility. We address this with the Slice Differential Regularization, which operates on inter-slice continuity rather than individual voxels.

For a 3D volume with $T>2$ slices along the depth axis, let $x_{(k)}$ and $\hat{x}_{(k)}$ denote the $k$-th axial slice of the input and the reconstruction, respectively. We define the first- and second-order slice differentials $\Delta_k(x)=x_{(k+1)}-x_{(k)}$ and $\Delta_k^2(x)=x_{(k+1)}-2x_{(k)}+x_{(k-1)}$, and penalize discrepancies between input and reconstructed transitions:

where the first term matches inter-slice transitions and the second preserves through-plane curvature.

Multi-Expert Representation Alignment

Apart from structural fidelity, a unified medical visual tokenizer should also encode high-level clinical semantics. Achieving this across the full breadth of medical imaging is challenging as different modalities embed distinct knowledge. Aligning to a single expert [6, 15] constrains the tokenizer to inherit a single semantic geometry, which may underrepresent modalities outside the expert’s training distribution. We therefore propose Multi-Expert Representation Alignment (MERA), which uses a panel of pretrained modality-specific encoders as semantic experts. MERA involves two objectives: (i) shared-space contrastive alignment, which jointly aligns representations from multiple modality experts into a shared space, and (ii) expert-space complementary alignment, which preserves complementary information in each modality expert’s representation space.

Shared-space contrastive alignment. An input $x_i$ of imaging modality $m(i)$ is routed to the corresponding expert $E_{m(i)}$ to produce a feature $e_i=E_{m(i)}(x_i)$, which will be linearly projected to a shared space, giving $\tilde{e}_i=p_{m(i)}(e_i)\in\mathbb{R}^{d}$. We then apply a contrastive loss $\mathcal{L}_{\mathrm{shared}}$ between MedVTok’s representations $r_i$ and expert features $\tilde{e}_i$ within a batch with a learnable temperature $\tau$:

Expert-space complementary alignment. Note that $\mathcal{L}_{\mathrm{shared}}$ projects heterogeneous expert features into a single space, which may discard modality-specific nuances present in each expert’s native space. To address this issue, we introduce expert-space complementary learning. For each modality $m(i)$, a backprojector $q_{m(i)}$ maps $\mathbf{r}_i$ back to the expert’s space, yielding $\widetilde{\mathbf{r}}_i = q_{m(i)}(\mathbf{r}_i) \in\mathbb{R}^{d_{m(i)}}$. We then enforce cosine similarity between backprojected representations and original expert features:

Together, $\mathcal{L}_{\mathrm{shared}}$ and $\mathcal{L}_{\mathrm{expert}}$ provide bidirectional representation alignment: the former anchors MedVTok within a single representation space, while the latter preserves modality-specific expert knowledge that might otherwise be compressed away. The total loss for MERA is given by:

Training and Inference

MedVTok is trained with the combined objective:

where the standard [7, 10, 29] reconstruction loss $\mathcal{L}_{\mathrm{R}}$ consists of pixel-level, LPIPS [30], and adversarial losses. Training proceeds in two stages: Stage 1 uses only 2D images to build foundational visual structure and semantic capabilities, with $\lambda_{\mathrm{SliDR}} = 0$; Stage 2 continues on both 2D images and 3D volumes with $\lambda_{\mathrm{SliDR}} = 0.1$. At inference, MedVTok serves as a general-purpose tokenizer: given an image or volume, it encodes tokens that are directly consumed by downstream models and transferable across various tasks without further fine-tuning of the tokenizer.

Experiments

Pretraining

Training data. We prepare the training data by aggregating existing large-scale publicly available training-ready datasets to ensure image quality and diversity. In total, we obtain 28,769,502 images and 70,144 volumes for training MedVTok. Please refer to § B.1 for data statistics.

Implementation details. We use the AdamW [31] optimizer with $(\beta_1, \beta_2) = (0.9, 0.95)$ and a cosine annealing schedule for 100K steps per stage. The learning rate decays from $10^{-4}$ to $10^{-5}$ in Stage 1 and from $2 \times10^{-5}$ to $2 \times10^{-6}$ in Stage 2, with $\lambda_{\mathrm{KL}} = 10^{-6}$, $\lambda_{\mathrm{MERA}} = 1$, and $\lambda_{\mathrm{SliDR}} = 0.1$. During Stage 2 training, 2D and 3D data are fed into MedVTok in alternating iterations, with an effective batch size of 600 for 2D images and 48 for 3D volumes. Please see § B.2 for details.

Evaluation on Downstream Tasks

Evaluation protocol. We compare MedVTok with both medical-domain tokenizers [5, 6, 8] and general-domain ones [4, 9, 10, 29, 32]. We first evaluate token fidelity of different tokenizers via autoencoding reconstruction, and then evaluate token representation quality on 6 downstream tasks: image classification, image-to-image retrieval, image segmentation, image synthesis, visual question answering (VQA), and medical report generation. All evaluations use a frozen tokenizer and focus on the quality of the encoded tokens rather than building task-specific models (§ C.2).

Evaluation data. The token fidelity, classification, and retrieval evaluations share 9 benchmarks: JSIEC (fundus, retinal conditions) [33], ChestCTScan (CT, chest cancers) [34], HAM10K (dermoscopy, skin diseases) [35], BRISC (MRI, brain tumor) [36], PolypDiag (endoscopy, polyp detection) [37], OCTDL (OCT, retinal diseases) [38], LungHist700 (pathology, lung cancers) [39], BUSBRA (ultrasound, tumor staging) [40], and Kermany (X-ray, pneumonia) [41]. Segmentation datasets include MaplesDR (fundus, optic disc/cup) [42], MosMedDataPlus (CT, COVID-19 lesion) [43], HAM10K (dermoscopy, lesion) [35], PolypDB (endoscopy, polyp) [44], BRISC (MRI, brain tumor) [36], PanNuke (pathology, different cells) [45], BUSBRA (ultrasound, tumor) [40], and VinDr-RibCXR (X-ray, rib) [46]. Image synthesis uses images from the GMAI-VL-5.5M [1] dataset. VQA benchmarks include GMAI-MMBench [47], Fundus-MMBench [48], PathVQA [49], MM-Skin [50], Kvasir-VQA-x1 [51], and SLAKE [52]. For 3D tasks, we use AMOS-MM [53] and MRISegmenter [54] to evaluate 3D token fidelity. AMOS-MM CT–report pairs are used for report generation evaluation (§ C.1).

Token fidelity evaluation. We assess token fidelity by how well the tokenizer reconstructs input data from its encoded tokens, reporting PSNR and SSIM between the input and the reconstructed data. Table 1 reports 2D reconstruction quality. MedVTok ranks first on all nine modalities in both PSNR and SSIM. The improvement is most notable in challenging domains such as pathology, where other tokenizers suffer from limited training diversity. Medical-domain tokenizers (MedITok, MedVAE) and general-domain ones (TokenFlow, UniTok, MingTok) lag significantly behind, confirming that medical images benefit from specialized architectures and large-scale multimodal training. Figure 3 visualizes the reconstruction error maps of the competing methods and the reconstructed images produced by MedVTok. The last two columns of Table 1 report native 3D reconstruction. MedVTok achieves the best PSNR and SSIM on both MRI and CT volumes by a large margin over PUMIT and MedVAE. The strong 3D performance suggests that our unified 2D–3D training paradigm effectively transfers learned representations to volumetric data. Please see § C.2 for qualitative comparisons.

Metrics	Method	CFP	CT	Derm.	Endo.	MRI	OCT	Path.	US	X-ray	Avg.	3D-MRI	3D-CT
PSNR (↑)	SDVAE [29]	37.62	36.38	34.68	33.57	34.31	29.17	24.43	32.13	35.29	33.06	–	–
	TokenFlow [4]	30.57	25.45	29.54	25.75	24.51	24.79	20.17	22.82	26.48	25.56	–	–
	MingTok [32]	35.58	31.23	32.43	29.85	29.37	26.92	21.87	26.42	31.25	29.43	–	–
	UniTok [10]	34.02	31.32	31.89	30.34	30.82	28.00	24.40	29.48	31.57	30.21	–	–
	AToken [9]	41.08	41.60	39.00	39.69	41.64	32.94	28.57	39.41	40.02	38.22	<u>31.46</u>	<u>35.68</u>
	PUMIT [8]	32.03	32.86	27.05	28.73	32.88	27.06	19.43	23.26	26.69	27.78	29.96	31.87
	MedVAE [5]	17.11	37.97	22.42	16.83	35.72	30.90	20.13	33.47	37.04	27.95	25.89	31.30
	MedITok [6]	35.92	35.61	33.81	32.42	34.82	30.26	26.03	32.62	34.70	32.91	–	–
	MedVTok	42.22	44.75	40.39	41.42	43.75	35.04	30.16	42.08	41.83	40.18	31.77	36.50
SSIM (↑)	SDVAE [29]	89.99	90.47	85.73	90.87	91.53	64.05	53.85	86.91	90.24	82.63	–	–
	TokenFlow [4]	83.75	73.40	78.03	77.23	74.13	59.89	31.75	55.34	76.99	67.84	–	–
	MingTok [32]	88.63	83.66	79.14	83.83	83.49	55.11	38.00	66.76	83.88	73.61	–	–
	UniTok [10]	88.54	88.70	82.48	89.62	90.12	65.24	55.36	87.21	88.20	81.72	–	–
	AToken [9]	94.74	<u>96.24</u>	<u>92.51</u>	<u>96.98</u>	<u>97.61</u>	<u>80.60</u>	<u>74.24</u>	<u>97.13</u>	<u>95.42</u>	<u>91.72</u>	85.19	95.50
	PUMIT [8]	88.09	90.82	84.84	90.17	90.92	65.34	57.92	81.99	89.30	82.16	<u>85.20</u>	<u>90.55</u>
	MedVAE [5]	77.26	92.67	86.63	83.86	93.87	72.44	54.18	89.97	92.90	82.64	75.14	90.44
	MedITok [6]	91.50	92.74	88.22	93.47	95.13	73.81	64.52	92.95	92.46	87.20	–	–
	MedVTok	95.95	97.86	94.51	97.90	98.66	86.54	80.52	98.28	97.21	94.16	87.79	95.65

Image classification and retrieval evaluation. To directly measure the representation quality for image classification, we adopt a $k$-nearest neighbor protocol. Each test sample is classified based on its $k$ nearest training neighbors with $k \in\{10, 20, 100, 200\}$, and we report the highest accuracy among $k$ values for each tokenizer. For retrieval, the training split serves as the gallery, and the test split provides the queries. Gallery images are ranked by cosine similarity, and we report Recall@1 performance.

Results are presented in Table 2. MedVTok achieves the highest average on both tasks, surpassing not only other tokenizers but also dedicated semantic encoders (BiomedCLIP, MedImageInsight, MedSigLIP) that are explicitly trained for representation learning. This indicates that the features produced by MedVTok are highly discriminative. The consistent ranking further confirms that the learned representations generalize across different downstream protocols.

Metrics	Method	CFP	CT	Derm.	Endo.	MRI	OCT	Path.	US	X-ray	Avg.
Acc. (↑)	SDVAE [29]	18.79	45.68	60.23	47.59	77.47	62.94	47.56	43.97	78.69	53.66
	TokenFlow [4]	35.73	46.10	61.95	42.83	79.73	68.53	45.38	46.10	86.91	57.03
	MingTok [32]	50.71	48.10	67.86	44.93	77.70	64.39	64.87	45.04	83.65	60.81
	UniTok [10]	66.95	74.79	70.63	44.82	80.53	72.88	69.23	46.28	91.03	68.57
	AToken [9]	64.12	65.82	72.31	46.63	88.13	66.46	66.54	46.19	91.29	67.50
	BiomedCLIP* [15]	60.59	66.03	68.78	46.03	88.43	73.91	79.10	48.32	93.80	69.45
	MedSigLIP* [17]	81.07	73.73	72.13	46.11	91.73	75.88	84.10	46.99	95.30	74.12
	MedImageInsight* [16]	75.64	75.32	85.87	44.65	95.47	90.79	71.79	50.18	96.21	76.21
	PUMIT [8]	13.84	43.78	60.34	45.85	78.00	59.52	47.44	43.44	79.49	52.41
	MedVAE [5]	13.28	45.25	60.12	45.34	75.57	57.56	49.62	41.76	73.40	51.32
	MedITok [6]	74.29	78.80	72.82	46.80	91.23	78.67	74.23	50.09	95.03	73.55
	MedVTok	90.25	89.77	74.16	48.71	94.23	94.00	82.69	51.33	96.90	80.23
Recall@1 (↑)	SDVAE [29]	22.75	43.81	57.41	56.14	90.10	57.14	32.30	39.10	82.53	53.47
	TokenFlow [4]	45.06	50.79	61.38	45.08	91.00	68.63	35.02	44.41	87.50	58.76
	MingTok [32]	54.94	52.38	63.16	44.82	89.50	68.01	53.70	42.29	83.01	61.31
	UniTok [10]	69.10	88.25	64.22	50.48	89.80	73.60	67.70	44.87	87.98	70.67
	AToken [9]	66.09	78.10	68.72	44.04	95.10	74.53	63.42	43.62	88.46	69.12
	BiomedCLIP* [15]	62.66	83.81	66.27	44.56	96.00	76.40	70.43	45.48	89.10	70.52
	MedSigLIP* [17]	82.40	92.06	67.79	44.82	96.90	77.02	83.27	39.63	94.55	75.38
	MedImageInsight* [16]	78.54	86.67	84.13	44.30	97.80	92.86	70.04	46.54	95.19	77.34
	PUMIT [8]	15.88	49.52	58.66	48.96	89.80	51.24	38.52	36.70	83.49	52.53
	MedVAE [5]	12.45	47.30	56.15	53.69	88.90	52.48	38.52	35.90	76.28	51.30
	MedITok [6]	73.82	87.62	68.65	52.52	95.60	80.75	72.76	46.01	92.15	74.43
	MedVTok	90.99	90.16	70.70	52.59	96.10	93.79	75.49	47.61	95.19	79.18

Image segmentation evaluation. We follow Siméoni et al. [55] and adopt a linear segmentation protocol to evaluate the representation quality for image segmentation. Specifically, a single linear

segmentation head is trained with a cross-entropy loss on top of the frozen patch tokens produced by each tokenizer, with learning rates searched over $\{10^{-2}, 10^{-3}, 10^{-4}\}$. For each tokenizer, the best-performing head on the validation set is selected for evaluation. Results are presented in Table 3. MedVTok achieves the best average Dice score, outperforming all baselines including semantic encoders. This task reveals a clear hierarchy: although general autoencoding tokenizers (e.g., SDVAE, PUMIT) can embed rich low-level details into the latent space, their high-level semantics are not activated, resulting in low segmentation performance. Semantic encoders provide explicit visual semantics and significantly outperform the autoencoding tokenizers. Unified tokenizers (MedITok, MedVTok) encode both fine-grained spatial information and high-level semantics, enjoying the benefits of both and improving dense prediction. Please see § C.2 for qualitative comparisons.

Metrics	Method	CFP	CT	Derm.	Endo.	MRI	Path.	US	X-ray	Avg.
Dice (↑)	SDVAE [29]	73.03	14.88	62.37	24.67	12.69	49.11	84.19	35.56	44.56
	TokenFlow [4]	70.39	19.05	72.62	40.53	30.49	52.34	91.33	49.52	53.28
	MingTok [32]	85.36	27.03	83.91	57.32	52.54	67.68	92.91	54.71	65.18
	UniTok [10]	60.93	33.13	86.25	67.25	54.90	72.51	93.35	57.60	65.74
	AToken [9]	97.09	37.33	87.45	69.88	63.57	74.30	94.90	59.69	73.03
	BiomedCLIP* [15]	50.22	31.83	87.48	60.64	57.95	73.65	95.41	59.83	64.63
	MedSigLIP* [17]	97.28	40.79	87.76	69.81	67.32	76.20	95.33	62.48	74.62
	MedImageInsight* [16]	92.34	25.60	85.45	66.96	55.81	42.93	95.80	50.16	64.38
	PUMIT [8]	40.05	25.53	77.94	37.32	36.69	66.00	90.73	54.82	53.63
	MedVAE [5]	61.74	12.57	57.59	19.73	10.13	36.40	81.90	30.81	38.86
	MedITok [6]	96.70	40.52	80.28	66.07	62.42	75.80	94.65	59.47	71.99
	MedVTok	97.44	42.91	90.25	72.52	69.62	77.74	96.91	61.13	76.07

Image synthesis evaluation. We evaluate how the tokenizer influences text-conditioned medical image synthesis within a latent diffusion [29] framework. A UNet denoising backbone is trained from scratch in each tokenizer’s latent space while keeping both the tokenizer and the text encoder frozen. This design isolates the effect of the tokenizer: the architecture, noise scheduler, optimizer, and training data are held constant across experiments. Training uses 70K image–text pairs spanning seven imaging modalities. We report the Fréchet Inception Distance (FID) [56, 57] in Table 4; lower FID indicates higher fidelity. MedVTok achieves the highest image synthesis quality with the best average FID, outperforming both specialized tokenizers (PUMIT, MedVAE) and the previous unified tokenizer (MedITok), as shown in Figure 4. PUMIT and MedVAE consistently underperform, confirming that synthesis quality depends not only on reconstruction fidelity but also on a well-structured, semantically meaningful latent space that facilitates the diffusion model’s learning.

Method	CFP	CT	Derm.	Endo.	MRI	Path.	X-ray	Avg.
PUMIT [8]	140.40	183.49	272.26	191.49	227.64	377.12	200.14	227.51
MedVAE [5]	118.13	154.42	202.09	170.29	145.25	325.87	88.55	172.08
MedITok [6]	90.65	142.46	176.94	58.38	122.96	99.66	77.24	109.76
MedVTok	59.85	146.39	142.87	82.39	116.78	94.65	69.17	101.73

Visual question answering evaluation. We evaluate each tokenizer as a frozen visual backbone within the LLaVA-Med [58] framework, training only the LLaVA-Med connector and the LLM, and evaluating the resulting models on VQA benchmarks [6]. We report accuracy assessed via LLM-as-judge grading. As shown in Table 5, MedVTok achieves the best average accuracy among the competing tokenizers. Notably, MedVTok does not use image–text pairs for training, yet still outperforms MedITok, which relies on millions of textual captions for representation learning.

Method	FundusMM	MM-Skin	Kvasir	PathVQA	SLAKE	GMAI-MM	Avg.
PUMIT [8]	$29.84\pm0.42$	$34.74\pm0.38$	$69.50\pm0.51$	$52.97\pm0.63$	$36.44\pm0.55$	$28.16\pm0.47$	$41.94\pm0.33$
MedVAE [5]	$28.39\pm0.56$	$34.21\pm0.44$	$66.64\pm0.72$	$51.15\pm0.58$	$31.57\pm0.67$	$27.61\pm0.53$	$39.93\pm0.41$
MedITok [6]	$31.29\pm0.35$	$36.18\pm0.41$	$70.63\pm0.29$	$58.46\pm0.52$	$45.61\pm0.38$	$30.15\pm0.26$	$45.39\pm0.28$
MedVTok	$33.39\pm0.31$	$37.49\pm0.28$	$72.98\pm0.34$	$57.24\pm0.45$	$45.94\pm0.37$	$31.81\pm0.29$	$46.48\pm0.22$

Medical report generation evaluation. Evaluation on generating free-text radiology reports from 3D volumes is similar to VQA, incorporating each frozen tokenizer into the LaMed framework [59] and training the connector and LLM to generate region-specific findings for three anatomical regions (abdomen, chest, and pelvis). We report BLEU-2, RaTEScore [60] (entity-level NER-based evaluation), and GREEN [61] (clinically aware error detection via an LLM judge). Table 6 presents the report generation results. MedVTok achieves the best scores on all metrics. The improvement is consistent across anatomical regions, with particularly strong gains in the pelvic region, which is the most challenging due to its complex anatomy and longer reference reports. Compared with MedITok, MedVTok shows meaningful improvements on clinically aware metrics (GREEN: 0.413 vs. 0.386), suggesting that our tokenizer encodes diagnostically relevant information that translates into more accurate clinical language generation. Additional results are provided in § C.2.

Metrics	Method	Abdomen	Chest	Pelvis	Avg.
BLEU-2 ($\uparrow$)	PUMIT [8]	0.289$\pm$0.008	0.193$\pm$0.003	0.164$\pm$0.007	0.215$\pm$0.005
BLEU-2 ($\uparrow$)	MedVAE-3D [5]	0.249$\pm$0.038	0.185$\pm$0.020	0.152$\pm$0.010	0.195$\pm$0.013
BLEU-2 ($\uparrow$)	MedITok [6]	0.256$\pm$0.013	0.219$\pm$0.020	0.294$\pm$0.017	0.257$\pm$0.012
BLEU-2 ($\uparrow$)	MedVTok	0.272$\pm$0.005	0.263$\pm$0.045	0.288$\pm$0.017	0.274$\pm$0.009
RaTEScore ($\uparrow$)	PUMIT [8]	0.604$\pm$0.005	0.511$\pm$0.035	0.201$\pm$0.015	0.439$\pm$0.006
RaTEScore ($\uparrow$)	MedVAE-3D [5]	0.613$\pm$0.022	0.497$\pm$0.053	0.187$\pm$0.003	0.432$\pm$0.011
RaTEScore ($\uparrow$)	MedITok [6]	0.610$\pm$0.006	0.501$\pm$0.008	0.594$\pm$0.031	0.568$\pm$0.015
RaTEScore ($\uparrow$)	MedVTok	0.619$\pm$0.005	0.570$\pm$0.074	0.612$\pm$0.006	0.600$\pm$0.006
GREEN ($\uparrow$)	PUMIT [8]	0.421$\pm$0.017	0.204$\pm$0.010	0.124$\pm$0.019	0.250$\pm$0.009
GREEN ($\uparrow$)	MedVAE-3D [5]	0.401$\pm$0.024	0.186$\pm$0.003	0.105$\pm$0.064	0.230$\pm$0.179
GREEN ($\uparrow$)	MedITok [6]	0.412$\pm$0.025	0.256$\pm$0.010	0.492$\pm$0.026	0.386$\pm$0.013
GREEN ($\uparrow$)	MedVTok	0.425$\pm$0.005	0.300$\pm$0.078	0.515$\pm$0.044	0.413$\pm$0.020

Ablation Studies

We ablate the key components of MedVTok in Figure 5. For efficiency, most components are evaluated on a training subset (3M images) in Stage 1 prior to data scaling.

Effect of MERA components. Starting from a baseline without any semantic alignment (E1 in Figure 5), shared-space contrastive learning with a single-expert $\mathcal{L}_{\mathrm{shared}}$ (E2, using MedImageInsight [16]) yields an initial improvement in classification and segmentation, confirming that semantic supervision is critical for downstream understanding. Extending $\mathcal{L}_{\mathrm{shared}}$ to multiple experts (E3) further improves both metrics, demonstrating that a panel of modality-specific experts provides more comprehensive semantics. Adding expert-space complementary learning $\mathcal{L}_{\mathrm{expert}}$ (E4) brings an additional +1.53% classification gain by preserving modality-specific nuances that shared-space contrastive learning alone may compress away during the shared-space projection.

Aggregation head design. Replacing attention pooling [9, 28] with mean pooling (E4$\rightarrow$E5) yields substantial gains in segmentation. Attention pooling tends to collapse patch-level details into a global summary, whereas mean pooling retains spatially distributed information critical for dense prediction tasks while still achieving competitive classification performance.

Data scaling. Scaling the training set from 3M to 28M images (E5$\rightarrow$E6) yields consistent improvements, confirming that our framework effectively leverages more diverse training data.

Effect of SliIDR. Stage 2 (E7) adds 3D data to enable volumetric tokenization. Adding $\mathcal{L}_{\mathrm{SliIDR}}$ (E8) improves 3D reconstruction in terms of both PSNR and SSIM, validating that SliIDR specifically strengthens through-plane continuity. Please see § C.3 for additional analyses.

Conclusion

We present MedVTok, a general-purpose medical visual tokenizer that encodes both 2D images and 3D volumes into a shared representational space through a single transformer architecture. Two complementary supervision mechanisms shape this space at different levels: Slice Differential Regularization preserves inter-slice fidelity that per-voxel supervision misses, and Multi-Expert Representation Alignment injects modality-specific clinical semantics from a panel of experts without requiring paired annotations. MedVTok achieves state-of-the-art results across reconstruction and six downstream tasks, ranging from pixel-level fidelity to report generation, demonstrating that structural fidelity and semantic richness can be jointly optimized within a single tokenizer.

We acknowledge several limitations. First, the training data focus on common imaging modalities at present. Other imaging modalities, clinical photos, or illustrations in medical publications remain unexplored. Incorporating these would enhance generalization to non-standard visual inputs encountered in real-world clinical workflows. Second, while MERA uses frozen expert encoders, jointly fine-tuning selected experts or incorporating new ones could further improve domain coverage when the computational budget allows. We hope MedVTok will serve as a useful building block for scalable multimodal medical AI systems that enable patient-level modeling across heterogeneous imaging evidence and clinical tasks.

Appendix

This appendix provides full experimental details and supplementary results. §A presents the tokenizer architecture, including input patchification, attention masking, positional embeddings, and expert encoders. §B describes the training settings: §B.1 details the training data construction and lists the 2D and 3D datasets; §B.2 specifies optimization, adversarial training, and the two-stage training procedure. §C covers evaluation: §C.1 summarizes all evaluation benchmarks grouped by downstream task, §C.2 details the evaluation protocols for each task along with additional quantitative and qualitative results, and §C.3 provides additional ablation studies and analyses. §D discusses broader societal impacts.

Tokenizer Overview

Relation to existing tokenizers. MedITok [6] combines reconstruction and semantic alignment for 2D medical images. AToken [9] provides a unified tokenizer for general-domain images, videos, and 3D assets. SAMTok [62] and HiMTok [63] encode segmentation masks as compact token sequences for multimodal language models. MedVTok focuses on a shared representation for medical images and volumes, combining depth-wise reconstruction regularization with semantic supervision from modality-specific experts.

Architecture hyperparameters. The encoder and decoder of MedVTok each consist of 27 transformer blocks with a hidden dimension of $d = 1152$. The encoder is initialized from SigLIP2-SO400M-patch16-NaFlex [28], while other components are trained from scratch. The patch size is $P_t \times P_h \times P_w$, where $P_t = 4$ is the depth patch size for 3D volumes and $P_h = 16$, $P_w = 16$ are the spatial patch sizes for both 2D images and 3D volumes. Rotary position embeddings (RoPE) [27] are derived from 3D patch coordinates $(z, x, y)$. Before the decoder, each encoded $d$-dimensional token embedding is projected to a 96-dimensional output, which is split into mean and log-variance for a diagonal Gaussian posterior, yielding latent codes of dimension $d_z = 48$.

Dimension-agnostic input patchification. All inputs, regardless of spatial resolution, aspect ratio, or dimensionality, are converted into a common concatenated patch format before entering the transformer encoder. Each patch spans $P_t \times P_h \times P_w = 4 \times16 \times16$ voxels across the depth, height, and width axes, yielding a per-patch feature vector of dimension $P_t \cdot P_h \cdot P_w \cdot C = 3072$ for $C = 3$ channels. For a 2D image of spatial size $H \times W$, the input is first zero-padded along the temporal axis to $P_t$ images. Patchification then yields $(H/P_h) \times(W/P_w) = \frac{HW}{256}$ patches (e.g., a 256 × 256 image produces 256 patches). For a 3D volume of depth $T$ and spatial size $H \times W$, patchification directly operates along all three axes, producing $(T/P_t) \times(H/P_h) \times(W/P_w)$ patches (e.g., a 32-slice 256 × 256 CT volume yields $8 \times16 \times16 = 2048$ patches).

All patches from a batch are concatenated into a single flat sequence of shape $[N, 3072]$, where $N$ is the total number of patches across the batch. This eliminates dense padding entirely and achieves $O(N)$ memory regardless of resolution heterogeneity. Each patch is associated with a coordinate vector $(b, z, x, y)$ encoding the batch index, depth position, and spatial position, together with a list of per-sample sequence lengths that enables $O(1)$ batch-wise slicing. Because the representation has no global spatial grid, a single batch can freely mix different spatial resolutions (e.g., 256 × 256 and 512 × 512), different depths (e.g., 16 and 64 slices), and 2D images with 3D volumes simultaneously.

Attention masking for concatenated sequences. Self-attention in the transformer blocks operates over the full concatenated sequence; however, cross-sample information leakage is prevented via block-diagonal attention masks, restricting each token to attend only within the boundaries of its own sample.

Expert encoders for MERA. Table 7 lists the expert encoders used in MERA. Features from all experts are precomputed offline and stored alongside the training data, so the expert encoders themselves are not loaded during training. Indices 9–10 (3D experts) are included only in Stage 2.

Index	Expert Encoder	Domains	$d_j$
0	DermLIP [21]	Dermoscopy	768
1	RetCLIP [22]	CFP	768
2	Conch [20]	Pathology	1024
3	PeskaVLP [24]	Endoscopy	2048
4–8	MedImageInsight [16]	CT, MR, US, X-ray, Other	2048
9	Spectre [25]	CT 3D	2160
10	HLIP [26]	MR 3D	2048

Each input uses the expert assigned to its modality, with a dedicated projector and backprojector. Shared-space alignment brings these representations into one token space, while expert-space alignment retains modality-specific information. Complementary knowledge thus refers to the distinct semantics contributed by experts across modalities.

Training Settings

Training Datasets

Figure 14 and Figure 15 provide a visual overview of the training data composition; Table 20 and Table 21 list the 2D and 3D training datasets, respectively. In total, we collect 28,769,502 images and 70,144 volumes spanning 9 imaging modalities.

Dataset	Modality	Count
RSNA STR Pulmonary Embolism Detection [69]	CT	2,854,455
MELA [70]	CT	1,198,641
RibFrac2020 [71]	CT	675,992
ATM2022 [72]	CT	419,489
LIDC-IDRI-ALL-CT [73]	CT	403,462
FLARE22 [74]	CT	236,577
QIN-HEADNECK [75]	CT	171,846
Pediatric-CT-SEG [76]	CT	156,111
AbdomenCT1K [77]	CT	145,254
TotalSegmentator v2 [78]	CT	123,420
ACRIN6668 [79]	CT	121,154
COVID-19-NY-SBU [80]	CT	116,764
Totalsegmentator [78]	CT	107,256
Parse22 [81]	CT	104,622
CTPelvic1k [82]	CT	96,949
LITS [83]	CT	90,524
AutoPET [84]	CT	81,135
NLST [85]	CT	76,000
CAD-PE [86]	CT	66,956
MSD Liver [87]	CT	65,104
CTORG [88]	CT	64,904
LNDb [89]	CT	62,969
HNSCC [90]	CT	61,077
ANODE09 [91]	CT	58,647
CTSpine1K Full [92]	CT	57,145
CPTAC-HNSCC [93]	CT	52,300
Lung-PET-CT-Dx [94]	CT	49,055
ACRIN-NSCLC-FDG-PET [95]	CT	42,772
Head-Neck Cetuximab [96]	CT	42,294
ACRIN-FLT-Breast [97]	CT	41,063
CT-Covid-19-August2020 [98]	CT	39,901
Anti-PD-1 MELANOMA [99]	CT	39,852
MSD Lung [87]	CT	36,684
OSIC Pul Fib Pro [100]	CT	35,231
SPIE AAPM [101]	CT	34,776
MIDRC-RICORD-1A [102]	CT	33,928
SPIE-AAPM-Lung-CT-Challenge [101]	CT	33,578
TCGA-LUAD [103]	CT	33,221
MIDRC-RICORD-1b [102]	CT	32,953
Anti-PD-1 Lung [104]	CT	32,388
CPTAC-LUAD [105]	CT	30,882
COVID-19-AR [106]	CT	30,304
TCIA-Covid-19 [107]	CT	30,304
TCGA-LUSC [108]	CT	29,954
SegRap2023 [109]	CT	27,932
VerSe20 [110]	CT	25,860
C4KC-KiTS [111]	CT	25,598
LOLA11 [112]	CT	24,156
EXACT09 [113]	CT	22,747
KiTS2021 [114]	CT	22,176
NSCLC-Cetuximab [115]	CT	21,911
TCGA-KIRC [116]	CT	20,630
CPTAC-CM [117]	CT	18,513
VESSEL2012 [118]	CT	18,289
TCGA-UCEC [119]	CT	17,924
TCGA-OV [120]	CT	17,917
OPC-Radiomics [121]	CT	17,853
CPTAC-SAR [122]	CT	17,502
Pancreas-CT [123]	CT	14,485
FastPET LD [124]	CT	14,047
ACRIN-HNSCC-FDG-PET-CT [125]	CT	13,989
COVID-19-20 [126]	CT	12,827
RadImageNet [127]	CT, MRI, Ultrasound	2,639,575

Dataset	Modality	Count
CT-RATE [178]	CT	15,000
RSNA STR Pulmonary Embolism Detection [69]	CT	7,148
FLARE22 [74]	CT	1,874
LIDC-IDRI-ALL-CT [73]	CT	967
AbdomenCT1K [77]	CT	882
QIN-HEADNECK [75]	CT	816
ACRIN6668 [79]	CT	764
MELA [70]	CT	688
OPC-Radiomics [121]	CT	606
CTPelvic1k [82]	CT	525
HNSCC [90]	CT	507
TCGA-KIRC [116]	CT	474
NSCLC-Cetuximab [115]	CT	465
Head-Neck-PET-CT [179]	CT	443
RibFrac2020 [71]	CT	417
Head-Neck Cetuximab [96]	CT	399
Totalsegmentator dataset [78]	CT	346
C4KC-KiTS [111]	CT	335
Pediatric-CT-SEG [76]	CT	296
MSD HepaticVessel [87]	CT	286
MSD Pancreas [87]	CT	280
ACRIN-NSCLC-FDG-PET [95]	CT	278
ACRIN-FLT-Breast [97]	CT	238
CTSpine1K Full [92]	CT	229
TCGA-UCEC [119]	CT	205
COVID-19-20 [126]	CT	199
TCGA-LUAD [103]	CT	188
CPTAC-HNSCC [93]	CT	186
GLIS-RT [154]	CT, MRI	644
MR-RATE [180]	MRI	20,000
BraTS2023 [147]	MRI	8,956
BrainTumour [149]	MRI	1,936
LLD-MMRI2023 [181]	MRI	1,580
Learn2Reg [150]	MRI	828
Vestibular-Schwannoma-SEG [151]	MRI	484
M&Ms-2 [182]	MRI	448
CrossMoDA2023 [155]	MRI	227
Total		70,144

We curate the training set from publicly available medical imaging repositories. Before quality filtering, we impose minimum dataset size thresholds: only 2D datasets containing at least 9,000 images and 3D datasets containing at least 180 volumes are included. This criterion serves two purposes. First, it prevents commonly used small-scale benchmark datasets from entering the training pool, thereby preserving their integrity for downstream evaluation. Second, it avoids highly fragmented data sources that contribute minimal training signal relative to the overhead of integration.

Following MedITok [6], we apply a series of automated quality filters to remove low-quality or clinically irrelevant 2D images, including checks on dynamic range, minimum resolution, pixel-value diversity, and information content, as well as manual screening to exclude non-clinical illustrations such as tables and plots.

Implementation Details

As stated in the main text (Eq. (5)), the total training objective is $\mathcal{L} = \mathcal{L}_{R} + \lambda_{\mathrm{SliDR}}\mathcal{L}_{\mathrm{SliDR}} + \lambda_{\mathrm{KL}}\mathcal{L}_{\mathrm{KL}} + \lambda_{\mathrm{MERA}}\mathcal{L}_{\mathrm{MERA}}$. Below we detail the loss components, the two-stage training procedure, and the optimization settings.

Loss functions. The reconstruction loss is defined as:

where $\ell_{1}$ and $\ell_{2}$ denote mean absolute error and mean squared error, respectively; $\mathcal{L}_{\mathrm{LPIPS}}$ [30] is the perceptual loss; and $\mathcal{L}_{\mathrm{adv}}$ is the adversarial loss with an adaptive weight $w_{\mathrm{adv}}$ that balances reconstruction and adversarial gradients on the decoder’s last layer, following VQGAN [7].

For the KL regularization, patch tokens $h \in\mathbb{R}^{N \times d}$ are linearly projected to a mean $\mu\in\mathbb{R}^{N \times d_z}$ and a log-variance $\log\sigma^{2} \in\mathbb{R}^{N \times d_z}$, where $d_z = 48$. The latent code is sampled via reparameterization as $z = \mu+ \sigma\odot\epsilon$, $\epsilon\sim\mathcal{N}(0, I)$. The KL term penalizes the deviation of the diagonal Gaussian posterior from the standard normal prior:

weighted by $\lambda_{\mathrm{KL}} = 10^{-6}$ following Rombach et al. [29]. We set $\lambda_{\mathrm{MERA}} = 1$ and $\lambda_{\mathrm{SliDR}} = 0.1$ by default; ablation studies on these coefficients can be found in Sec. C.3.

Two-stage training. In Stage 1, MedVTok is trained exclusively on 2D images with $\mathcal{L}_{R}$, $\mathcal{L}_{\mathrm{KL}}$, and $\mathcal{L}_{\mathrm{MERA}}$ ($\lambda_{\mathrm{SliDR}} = 0$). Only 2D expert projectors are active. A diversity sampler ensures that each mini-batch contains at least two samples from every available modality group, with the remaining slots filled from the global pool.

In Stage 2, we initialize from the Stage 1 checkpoint, introduce 3D expert projectors (indices 9–10), and activate $\mathcal{L}_{\mathrm{SliDR}}$ with $\lambda_{\mathrm{SliDR}} = 0.1$. Training alternates between 2D and 3D batches at every iteration: even iterations process a 2D batch (effective batch size 600) and odd iterations process a 3D batch (effective batch size 48). The 3D dataloader cycles indefinitely to maintain continuous 3D supervision regardless of the smaller 3D dataset size. Within each 3D batch, a balanced sampler maintains an approximately equal ratio of CT and MR volumes.

Optimization. We use AdamW [31] for both the tokenizer and discriminator. The tokenizer uses $(\beta_1, \beta_2) = (0.9, 0.95)$ with a cosine annealing schedule: in Stage 1, the learning rate decays from $10^{-4}$ to $10^{-5}$ over $\sim100\mathrm{K}$ steps; in Stage 2, from $2 \times10^{-5}$ to $2 \times10^{-6}$ over $\sim100\mathrm{K}$ steps. The discriminator uses $(\beta_1, \beta_2) = (0.5, 0.9)$. All training uses BFloat16 mixed precision. During training, images are of mixed resolutions: 256, 384, and 512.

Compute resources. Full-scale data training is conducted on 16× NVIDIA H200 GPUs (141 GB); peak GPU memory usage is 132.54 GB per device. Each compute node is an x86_64 machine with dual Intel Xeon Platinum 8558 processors (2 sockets, 48 physical cores per socket, 2 threads per core). Stage 1 takes approximately 15.75 hours and Stage 2 takes approximately 43.5 hours. The full research project required more compute than reported here due to preliminary and failed experiments during hyperparameter exploration and architecture design.

License. The code and model weights will be released under the Apache 2.0 license.

Evaluation

Evaluation Datasets

Table 8 summarizes all evaluation datasets used in our experiments, grouped by downstream task. We verified that all evaluation datasets listed are excluded from the training corpus.

Dataset	Modality	#Samples	#Cls	Task(s)
2D Perception – Reconstruction / Classification / Retrieval
ChestCTScan [34]	CT	1,000	4	Rec, Cls, Ret
JSIEC [33]	CFP	1,000	39	Rec, Cls, Ret
HAM10K [35]	Dermoscopy	10,015	7	Rec, Cls, Ret
PolypDiag [37]	Endoscopy	2,202	2	Rec, Cls, Ret
BRISC [36]	MRI	5,023	4	Rec, Cls, Ret
OCTDL [38]	OCT	3,236	7	Rec, Cls, Ret
LungHist700 [39]	Pathology	691	3	Rec, Cls, Ret
BUSBRA [40]	Ultrasound	1,875	4	Rec, Cls, Ret
Kermany [41]	X-ray	5,856	2	Rec, Cls, Ret
2D Perception – Segmentation
MaplesDR [42]	CFP	192	3	Seg
MosMedDataPlus [43]	CT	2,729	2	Seg
HAM10K [35]	Dermoscopy	10,015	2	Seg
PolypDB [44]	Endoscopy	3,843	2	Seg
BRISC [36]	MRI	4,746	2	Seg
PanNuke [45]	Pathology	7,901	6	Seg
BUSBRA [40]	Ultrasound	1,875	2	Seg
VinDr-RibCXR [46]	X-ray	245	2	Seg
3D Volumetric Reconstruction
AMOS-MM [53]	CT	100	–	3D Rec
MRISegmenter [54]	MRI	240	–	3D Rec
Image Synthesis
GMAI-VL-5.5M [1]	Multi-modality	70,000	–	Syn
Visual Question Answering (test benchmarks)
Fundus-MMBench [48]	CFP	620	–	VQA
GMAI-MMBench [47]	Multi-modality	4,550	–	VQA
Kvasir-VQA-x1 [51]	Endoscopy	15,955	–	VQA
MM-Skin [50]	Dermoscopy	5,481	–	VQA
PathVQA [49]	Pathology	6,719	–	VQA
SLAKE [52]	Radiography	2,094	–	VQA
Medical Report Generation
AMOS-MM [53]	CT	1,687	–	MRG

Evaluation Setup and Results

Token fidelity. We use autoencoding reconstruction as a proxy task to evaluate token fidelity: each input is encoded and decoded by the frozen tokenizer, and we measure how faithfully the tokens preserve the original signal. For 2D images, evaluation is performed at $512 \times512$ resolution. We report PSNR and SSIM between the input and reconstructed images. For 3D volumes, we evaluate on CT and MRI using two datasets. CT intensities are clipped to $[-1000, 1000]$ HU; MRI intensities are clipped at the 0.5th and 99.5th percentiles per volume. PSNR and SSIM are computed at the volume level. Only models with native 3D support (PUMIT, MedVAE, AToken, MedVTok) are included in the 3D evaluation.

Additional visualizations of reconstructions on 2D and 3D modalities are presented in Figure 6 and Figure 7, respectively.

Classification. We adopt a $k$-nearest neighbor ($k$-NN) protocol on frozen features extracted from the CLS token (or equivalent global representation) of each tokenizer. For tokenizers that do not output such a representation (e.g., SDVAE), we concatenate all features along the channel dimension. The feature vectors are $L_2$-normalized, and classification is performed via softmax-weighted voting over the $k$ nearest neighbors using cosine similarity with a temperature of $T = 0.07$. We sweep $k \in\{10, 20, 100, 200\}$ on the validation set and report the best accuracy for each model. Image resolution is controlled at $256 \times256$ when possible.

Retrieval. We evaluate image-to-image retrieval using the same frozen features and benchmarks as classification. The training split of each dataset serves as the gallery, and the test split provides the queries. For each query, we compute the cosine similarity between its $L_2$-normalized feature vector and all gallery features, rank the gallery by descending similarity, and report Recall@1 (the fraction of queries whose top-retrieved image shares the same label). Image resolution is controlled at $256 \times256$ when possible, consistent with the classification setup.

Segmentation. We evaluate the quality of frozen patch tokens for dense prediction by training a lightweight linear segmentation head on top of each tokenizer’s output, following the protocol of Siméoni et al. [55]. The segmentation head consists of a batch normalization layer followed by a $1 \times1$ convolution. For each tokenizer, we train separate heads under identical settings and select the best-performing one by validation Dice score. Training runs for 50 epochs with AdamW (no weight decay), a batch size of 32, cross-entropy loss, and learning rates searched over $\{10^{-2}, 10^{-3}, 10^{-4}\}$.

Image resolution is controlled at $256 \times256$ when possible. We report the overall Dice score over segmentation classes. Table 9 further reports Hausdorff Distance at the 95th percentile (HD95), where MedVTok still leads in terms of average performance across modalities.

Table 9: Additional results on image segmentation evaluation. Method type: $^{*}$ semantic encoder; $^{\ddagger}$ unified tokenizer.

Metrics	Method	CFP	CT	Derm.	Endo.	MRI	Path.	US	X-ray	Avg.
Dice ($\uparrow$)	SDVAE [29]	73.03	14.88	62.37	24.67	12.69	49.11	84.19	35.56	44.56
	TokenFlow$^{\ddagger}$ [4]	70.39	19.05	72.62	40.53	30.49	52.34	91.33	49.52	53.28
	MingTok$^{\ddagger}$ [32]	85.36	27.03	83.91	57.32	52.54	67.68	92.91	54.71	65.18
	UniTok$^{\ddagger}$ [10]	60.93	33.13	86.25	67.25	54.90	72.51	93.35	57.60	65.74
	AToken$^{\ddagger}$ [9]	97.09	37.33	87.45	69.88	63.57	74.30	94.90	59.69	73.03
	BiomedCLIP$^{*}$ [15]	50.22	31.83	87.48	60.64	57.95	73.65	95.41	59.83	64.63
	MedSigLIP$^{*}$ [17]	97.28	40.79	87.76	69.81	67.32	76.20	95.33	62.48	74.62
	MedImageInsight$^{*}$ [16]	92.34	25.60	85.45	66.96	55.81	42.93	95.80	50.16	64.38
	PUMIT [8]	40.05	25.53	77.94	37.32	36.69	66.00	90.73	54.82	53.63
	MedVAE [5]	61.74	12.57	57.59	19.73	10.13	36.40	81.90	30.81	38.86
	MedITok$^{\ddagger}$ [6]	96.70	40.52	80.28	66.07	62.42	75.80	94.65	59.47	71.99
	MedVTok$^{\ddagger}$	97.44	42.91	90.25	72.52	69.62	77.74	96.91	61.13	76.07
HD95 ($\downarrow$)	SDVAE [29]	109.69	218.98	56.30	124.25	343.33	241.99	172.77	46.39	164.21
	TokenFlow$^{\ddagger}$ [4]	57.43	211.68	56.90	73.33	151.20	278.48	45.72	23.82	112.32
	MingTok$^{\ddagger}$ [32]	115.80	180.54	40.70	109.14	76.67	230.27	51.49	34.18	104.85
	UniTok$^{\ddagger}$ [10]	104.54	123.44	18.10	51.08	66.77	162.47	22.59	13.85	70.35
	AToken$^{\ddagger}$ [9]	22.99	119.80	16.05	47.32	29.93	160.72	16.64	12.63	53.26
	BiomedCLIP$^{*}$ [15]	84.43	178.88	12.62	50.33	31.22	145.42	13.75	12.97	66.20
	MedSigLIP$^{*}$ [17]	38.47	112.54	13.94	35.48	21.55	138.56	16.82	21.45	49.85
	MedImageInsight$^{*}$ [16]	31.83	248.33	17.16	58.84	54.94	174.84	12.50	27.52	78.24
	PUMIT [8]	63.10	151.79	30.78	76.98	78.13	214.75	35.29	19.50	83.79
	MedVAE [5]	326.48	241.19	69.08	102.19	164.70	343.04	481.75	47.13	221.94
	MedITok$^{\ddagger}$ [6]	44.53	138.61	37.06	46.60	32.97	140.30	18.31	13.85	59.03
	MedVTok	19.12	105.06	11.67	46.31	24.76	121.30	12.43	15.21	44.48

Comparison with modality experts. Table 10 and Table 11 compare MedVTok with the experts used in MERA under the classification and segmentation protocols above. MedVTok exceeds MedImageInsight on seven of nine classification benchmarks and all eight segmentation benchmarks. Among the four specialized experts, it exceeds PeskaVLP in classification and RetCLIP, DermLIP, and PeskaVLP in segmentation. Conch retains higher scores on pathology.

Table 10: Classification accuracy (%) compared with MERA experts. Specialized experts are evaluated on their respective modalities. Values are means $\pm$ 95% confidence intervals.

Model	CFP	CT	Derm.	Endo.	MRI	OCT	Path.	US	X-ray
RetCLIP	91.95$\pm$1.05	–	–	–	–	–	–	–	–
DermLIP	–	–	79.72$\pm$0.66	–	–	–	–	–	–
PeskaVLP	–	–	–	48.64$\pm$2.29	–	–	–	–	–
Conch	–	–	–	–	–	–	93.08$\pm$2.44	–	–
MedImageInsight	75.64$\pm$2.12	75.32$\pm$2.83	85.87$\pm$0.38	44.65$\pm$0.74	95.47$\pm$0.29	90.79$\pm$1.18	71.79$\pm$2.92	50.18$\pm$2.67	96.21$\pm$0.83
MedVTok	90.54$\pm$1.05	87.03$\pm$1.64	75.27$\pm$0.99	51.00$\pm$1.85	95.03$\pm$0.52	94.00$\pm$1.18	78.97$\pm$1.65	52.92$\pm$1.68	97.22$\pm$0.61

Table 11: Segmentation Dice (%) compared with MERA experts. Specialized experts are evaluated on their respective modalities. Values are means $\pm$ 95% confidence intervals.

Model	CFP	CT	Derm.	Endo.	MRI	Path.	US	X-ray
RetCLIP	92.46$\pm$1.69	–	–	–	–	–	–	–
DermLIP	–	–	89.05$\pm$1.86	–	–	–	–	–
PeskaVLP	–	–	–	65.06$\pm$1.51	–	–	–	–
Conch	–	–	–	–	–	83.77$\pm$1.39	–	–
MedImageInsight	92.34$\pm$1.39	25.60$\pm$1.35	85.45$\pm$2.10	66.96$\pm$1.32	55.81$\pm$1.31	42.93$\pm$1.86	95.80$\pm$2.07	50.16$\pm$1.48
MedVTok	97.44$\pm$1.59	42.91$\pm$1.80	90.25$\pm$0.77	72.52$\pm$1.15	69.62$\pm$1.07	77.74$\pm$1.56	96.91$\pm$1.80	61.13$\pm$1.17

3D segmentation. We evaluate frozen tokenizers on AMOS-MM CT [53] and MRISegmenter MRI [54], both excluded from tokenizer pretraining. For each tokenizer, we train a lightweight 3D convolutional head on volumetric features to predict segmentation masks. Table 12 reports overall and per-organ Dice. MedVTok achieves the highest overall Dice on both datasets.

Organ	AMOS-MM CT: PUMIT	AMOS-MM CT: MedVAE 3D	AMOS-MM CT: AToken	AMOS-MM CT: MedVTok	MRISegmenter MRI: PUMIT	MRISegmenter MRI: MedVAE 3D	MRISegmenter MRI: AToken	MRISegmenter MRI: MedVTok
Overall	28.67	29.71	36.89	50.13	44.57 ± 0.76	43.70 ± 0.80	54.63 ± 0.68	57.20 ± 0.76
Spleen	63.68 ± 1.58	42.94 ± 2.64	55.76 ± 2.46	70.61 ± 2.30	40.83 ± 2.12	40.68 ± 2.14	63.28 ± 2.71	70.60 ± 2.85
Kidney	48.94 ± 1.96	45.95 ± 2.36	59.56 ± 2.44	74.93 ± 1.74	81.60 ± 1.29	77.46 ± 1.77	83.96 ± 0.67	86.48 ± 0.95
Liver	76.63 ± 2.68	73.73 ± 2.23	80.50 ± 1.72	86.31 ± 1.00	58.11 ± 1.32	59.11 ± 1.23	85.71 ± 0.94	85.83 ± 0.70
Pancreas	13.42 ± 1.99	23.51 ± 2.60	31.64 ± 2.72	41.90 ± 2.36	49.43 ± 2.22	43.98 ± 2.08	49.23 ± 1.92	49.49 ± 2.71
Adrenal	16.50 ± 1.30	12.74 ± 1.10	20.44 ± 1.47	23.64 ± 1.76	24.00 ± 1.43	21.69 ± 1.87	25.63 ± 1.49	23.16 ± 1.64
Gallbladder	18.78 ± 2.21	16.33 ± 2.99	14.93 ± 1.82	39.69 ± 4.57	24.20 ± 3.46	23.31 ± 3.27	32.85 ± 3.68	34.94 ± 3.76
Esophagus	16.29 ± 1.02	18.59 ± 2.98	15.14 ± 1.80	19.04 ± 2.68	48.05 ± 1.82	39.28 ± 2.26	44.92 ± 1.79	52.78 ± 1.61
Stomach	4.96 ± 0.76	3.71 ± 0.76	34.63 ± 3.53	62.15 ± 3.87	40.66 ± 2.03	46.67 ± 2.05	52.08 ± 1.30	59.89 ± 2.30
Duodenum	3.37 ± 1.73	10.44 ± 2.00	15.04 ± 1.46	30.48 ± 2.71	14.82 ± 1.64	16.41 ± 1.62	40.11 ± 1.75	39.23 ± 1.80
Major vessels	24.14 ± 1.82	49.19 ± 2.05	41.22 ± 1.65	52.53 ± 1.37	64.00 ± 1.28	68.46 ± 1.18	68.52 ± 0.91	69.56 ± 0.93

Cross-dataset 3D segmentation. We apply the models trained on MRISegmenter directly to AMOS-MM MRI, without training or tuning on the target dataset. Evaluation covers the ten shared anatomical labels. MedVTok achieves the highest transfer Dice (Table 13).

Model	MRISegmenter	AMOS-MM MRI
PUMIT	44.57±0.76	31.44±1.52
MedVAE-3D	43.70±0.80	33.52±3.82
AToken	54.63±0.68	44.39±1.75
MedVTok	57.20±0.76	47.24±1.47

Image synthesis. We evaluate how the quality of a tokenizer’s latent space influences text-conditioned medical image synthesis. Each frozen tokenizer is integrated into a latent diffusion framework [29] as the only variable: a UNet denoising backbone following the Stable Diffusion v1.4 architecture is randomly initialized for each tokenizer (with input/output channels adapted to match the latent dimensionality), and a frozen CLIP-based text encoder provides conditioning. We train separate diffusion models per tokenizer using identical data, architecture, and hyperparameters, ensuring that differences in generation quality are attributable solely to the tokenizer.

Training uses 70,000 images from GMAI-VL-5.5M [1] spanning seven imaging modalities (10,000 images each). The noise scheduler is DDPM with 1,000 timesteps, a scaled-linear beta schedule, and $\epsilon$-prediction parameterization. The model is trained with an effective batch size of 256, the AdamW optimizer, a constant learning rate with 5% linear warmup, and classifier-free guidance dropout of 10%. Note that the purpose of this experiment is to compare the generative suitability of different tokenizer latent spaces, rather than to build state-of-the-art text-to-image generative models or to benchmark fine-grained synthesis. The original paired descriptions in GMAI-VL-5.5M are often long and fine-grained; using them directly may introduce confounding factors from text complexity. We therefore simplify the setup, replacing each caption with a controlled template, e.g., “A medical {modality} image,” and keep this protocol identical for all tokenizers. Under this setting, differences in generation quality more directly reflect how well each frozen tokenizer organizes medical images into a latent space that is compact, learnable, and amenable to latent diffusion.

At inference, we use DDIM sampling with 100 denoising steps. We generate 500 images per modality and report the Kernel Inception Distance (KID) [64], Fréchet Inception Distance (FID) [56], and Inception Score (IS) [65] for evaluation in Table 14. MedVTok consistently achieves the best average performance across these metrics.

Metrics	Method	CFP	CT	Derm.	Endo.	MRI	Path.	X-ray	Avg.
KID (↓)	PUMIT [8]	0.098	0.110	0.096	0.110	0.067	0.296	0.080	0.123
	MedVAE [5]	0.116	0.084	0.135	0.154	0.077	0.288	0.068	0.132
	MedITok [6]	0.256	0.099	0.078	0.024	0.054	0.055	0.055	0.089
	MedVTok	0.027	0.076	0.077	0.035	0.048	0.139	0.019	0.060
FID (↓)	PUMIT [8]	140.40	183.49	272.26	191.49	227.64	377.12	200.14	227.51
	MedVAE [5]	118.13	154.42	202.09	170.29	145.25	325.87	88.55	172.08
	MedITok [6]	90.65	142.46	176.94	58.38	122.96	99.66	77.24	109.76
	MedVTok	59.85	146.39	142.87	82.39	116.78	94.65	69.17	101.73
IS (↑)	PUMIT [8]	1.024	1.031	1.077	1.048	1.049	1.066	1.057	1.050
	MedVAE [5]	1.035	1.060	1.069	1.035	1.071	1.061	1.073	1.058
	MedITok [6]	1.039	1.048	1.104	1.038	1.079	1.154	1.076	1.077
	MedVTok	1.044	1.060	1.127	1.054	1.075	1.159	1.074	1.085

Visual question answering. We evaluate each frozen tokenizer as a visual backbone within the LLaVA-Med [58] framework. Given a medical image, the tokenizer encodes it into patch-level tokens, which are then fed through a two-layer MLP connector into Mistral-7B-Instruct as the language model backbone. We train separate LLaVA-Med variants using identical data and hyperparameters while keeping the tokenizer frozen.

Training follows a two-stage protocol. In the first stage, only the MLP connector is trained on approximately 509,369 image–text pairs (learning rate $10^{-3}$, one epoch), with 500,000 pairs from PubMedVision [66] and 9,369 pairs from MM-Skin [50]. In the second stage, LoRA adapters (rank=128, $\alpha=256$, dropout=0.05) are applied to all linear layers of the language model and finetuned jointly with the connector (learning rate $10^{-4}$ for LoRA, $2 \times10^{-5}$ for the connector, one epoch), using 140,151 VQA items for training (70,000 from PubMedVision; 21,778 from the MM-Skin training split; 28,719 from the Kvasir-VQA-x1 training split; 19,654 from the PathVQA training split).

At inference, we use sampling with a temperature of 0.2. Model outputs are evaluated via LLM-as-judge binary grading using Qwen3-235B-A22B [67]. We train each LLaVA-Med variant with three random seeds and report accuracy as mean $\pm$ 95% confidence interval in Table 5.

Two representative cases of LLaVA-Med output (integrating MedVTok as the visual encoder) are presented in Figure 12, where the model identifies the pathological features in the given medical image and answers the question correctly.

VQA controls. We hold the connector, language model, data, LoRA settings, decoding, and judge fixed across tokenizers. We also evaluate a frozen language model with only the connector trained, and re-score the standard model outputs with GPT-5. MedVTok leads under connector-only training, and replacing the Qwen3-235B-A22B judge with GPT-5 preserves the tokenizer ranking (Table 15).

Model	Connector only Qwen3 judge	Standard Qwen3 judge	Standard GPT-5 judge
PUMIT	23.05±0.44	41.94±0.33	41.83±0.33
MedVAE	17.88±0.48	39.93±0.41	39.58±0.42
MedITok	24.76±0.39	45.39±0.28	44.82±0.26
MedVTok	29.86±0.36	46.48±0.22	46.02±0.24

Medical report generation. We use the AMOS-MM [53] dataset, which provides CT volumes paired with radiology reports covering three anatomical regions (abdomen, chest, and pelvis), with 1,287 pairs for training and 400 pairs for evaluation. CT intensities are windowed to $[-160, 240]$ HU. Each frozen visual tokenizer is integrated into the LaMed framework [59] as the visual backbone: given a 3D volume, the tokenizer first encodes it into latent tokens, which are then fed through a connector (a spatial pooling layer that applies 3D average pooling with $2\times$ downsampling per dimension) and into Llama-3.1-8B-Instruct adapted with LoRA (rank $=8$, $\alpha=16$, dropout$=0.1$).

We train separate LLaMed variants, each with a different visual tokenizer as the frozen backbone, using identical data and hyperparameters, and evaluate all variants on the same test set under the same inference settings. Training runs for 15 epochs with AdamW (learning rate $5\times10^{-5}$, cosine decay, 3% warmup), an effective batch size of 4, and BFloat16 precision.

At inference, the model generates region-specific findings autoregressively with a temperature of $T=0.2$ and top-$p=0.9$. We also include MedITok [6], a 2D-only tokenizer, for comparison. To adapt it to 3D inputs, we treat the depth dimension as a batch of 2D slices, encode them independently in parallel with the MedITok encoder, and concatenate the resulting tokens along the depth axis before passing them to the connector and LLM.

In Table 6 and Table 16, we report BLEU-2, ROUGE-L, METEOR, RaTEScore, and GREEN per anatomical region and globally, as mean ± 95% confidence interval over three random seeds. The LaMed model integrating MedVTok as the visual encoder achieves superior performance in terms of the average score on each metric. Representative outputs are shown in Figure 13, where MedVTok directly encodes the whole CT volume and documents the lesions and abnormalities present.

Metrics	Method	Abdomen	Chest	Pelvis	Avg.
BLEU ($\uparrow$)	PUMIT [8]	$0.289 \pm0.008$	$0.193 \pm0.003$	$0.164 \pm0.007$	$0.215 \pm0.005$
BLEU ($\uparrow$)	MedVAE-3D [5]	$0.249 \pm0.038$	$0.185 \pm0.020$	$0.152 \pm0.010$	$0.195 \pm0.013$
BLEU ($\uparrow$)	MedITok [6]	$0.256 \pm0.013$	$0.219 \pm0.020$	$0.294 \pm0.017$	$0.257 \pm0.012$
BLEU ($\uparrow$)	MedVTok	<u>$0.272 \pm0.005$</u>	$0.263 \pm0.045$	<u>$0.288 \pm0.017$</u>	$0.274 \pm0.009$
ROUGE-L ($\uparrow$)	PUMIT [8]	$0.415 \pm0.005$	$0.277 \pm0.005$	$0.207 \pm0.007$	$0.300 \pm0.003$
ROUGE-L ($\uparrow$)	MedVAE-3D [5]	$0.398 \pm0.004$	$0.266 \pm0.003$	$0.183 \pm0.004$	$0.282 \pm0.000$
ROUGE-L ($\uparrow$)	MedITok [6]	$0.389 \pm0.007$	<u>$0.299 \pm0.025$</u>	<u>$0.407 \pm0.033$</u>	<u>$0.365 \pm0.016$</u>
ROUGE-L ($\uparrow$)	MedVTok	<u>$0.410 \pm0.006$</u>	$0.353 \pm0.023$	$0.442 \pm0.011$	$0.402 \pm0.005$
METEOR ($\uparrow$)	PUMIT [8]	$0.353 \pm0.005$	$0.213 \pm0.003$	$0.189 \pm0.007$	$0.252 \pm0.004$
METEOR ($\uparrow$)	MedVAE-3D [5]	$0.318 \pm0.025$	$0.221 \pm0.010$	$0.172 \pm0.008$	$0.237 \pm0.014$
METEOR ($\uparrow$)	MedITok [6]	$0.318 \pm0.014$	<u>$0.279 \pm0.004$</u>	<u>$0.388 \pm0.021$</u>	<u>$0.328 \pm0.014$</u>
METEOR ($\uparrow$)	MedVTok	<u>$0.336 \pm0.005$</u>	$0.327 \pm0.045$	$0.399 \pm0.019$	$0.354 \pm0.007$
RaTEScore ($\uparrow$)	PUMIT [8]	$0.604 \pm0.005$	<u>$0.511 \pm0.035$</u>	$0.201 \pm0.015$	$0.439 \pm0.006$
RaTEScore ($\uparrow$)	MedVAE-3D [5]	<u>$0.613 \pm0.022$</u>	$0.497 \pm0.053$	$0.187 \pm0.003$	$0.432 \pm0.011$
RaTEScore ($\uparrow$)	MedITok [6]	$0.610 \pm0.006$	$0.501 \pm0.008$	<u>$0.594 \pm0.031$</u>	<u>$0.568 \pm0.015$</u>
RaTEScore ($\uparrow$)	MedVTok	$0.619 \pm0.005$	$0.570 \pm0.074$	$0.612 \pm0.006$	$0.600 \pm0.006$
GREEN ($\uparrow$)	PUMIT [8]	<u>$0.421 \pm0.017$</u>	$0.204 \pm0.010$	$0.124 \pm0.019$	$0.250 \pm0.009$
GREEN ($\uparrow$)	MedVAE-3D [5]	$0.401 \pm0.024$	$0.186 \pm0.003$	$0.105 \pm0.064$	$0.230 \pm0.179$
GREEN ($\uparrow$)	MedITok [6]	$0.412 \pm0.025$	<u>$0.256 \pm0.010$</u>	<u>$0.492 \pm0.026$</u>	<u>$0.386 \pm0.013$</u>
GREEN ($\uparrow$)	MedVTok	$0.425 \pm0.005$	$0.300 \pm0.078$	$0.515 \pm0.044$	$0.413 \pm0.020$

Additional Analyses

Effect of $\lambda_{\mathrm{MERA}}$. Figure 9 shows the effect of the MERA loss weight for values in $\{0, 0.1, 1, 2\}$. Without semantic alignment ($\lambda_{\mathrm{MERA}} = 0$), classification and segmentation scores remain significantly low despite strong reconstruction, as the tokenizer only learns the low-level structure without discriminating the high-level semantics. Increasing $\lambda_{\mathrm{MERA}}$ monotonically improves semantic metrics, but at the cost of reconstruction fidelity: at $\lambda_{\mathrm{MERA}} = 2$, PSNR drops by nearly 3 dB and SSIM falls to 0.868. We set $\lambda_{\mathrm{MERA}} = 1$ as the default, which achieves strong semantic performance while maintaining reconstruction quality.

Effect of $\lambda_{\mathrm{SliDR}}$. Figure 10 shows the effect of the SliDR loss weight. As expected, 2D reconstruction is unaffected by $\lambda_{\mathrm{SliDR}}$ since SliDR operates only on volumetric data. For 3D reconstruction, both PSNR and SSIM improve substantially when SliDR is enabled. Further increasing the weight to $\lambda_{\mathrm{SliDR}} = 0.5$ yields marginal 3D gains but degrades classification accuracy by 2.6%, indicating that excessive volumetric regularization interferes with semantic learning. We therefore adopt $\lambda_{\mathrm{SliDR}} = 0.1$, which captures most of the 3D benefit without compromising other objectives.

Comparison with alternative depth regularizers. We compare SliDR with direct gradient matching, total variation (TV), and Laplacian regularization. All variants are initialized from the same 2D-trained MedVTok checkpoint and trained on a controlled subset of 10K volumes. We evaluate volumetric reconstruction on AMOS-MM CT and MRISegmenter MRI (Table 17). SliDR achieves the highest mean PSNR and SSIM on both datasets under this setting.

Objective	PSNR (dB)	SSIM (%)	PSNR (dB)	SSIM (%)
Direct gradient	$33.24\pm0.22$	$93.01\pm0.25$	$26.11\pm0.14$	$78.62\pm0.44$
TV	$33.05\pm0.21$	$93.08\pm0.24$	$26.04\pm0.14$	$78.00\pm0.45$
Laplacian	$32.99\pm0.21$	$93.08\pm0.24$	$26.20\pm0.15$	$77.86\pm0.45$
SliDR	$33.46\pm0.22$	$93.21\pm0.25$	$26.23\pm0.15$	$78.70\pm0.44$

TV and Laplacian regularization penalize first- and second-order depth differences of the reconstructed volume, respectively, without reference to the ground truth. These penalties encourage smoothness even where the input contains genuine inter-slice changes. SliDR instead matches both orders of slice differentials to the ground-truth volume (Eq. (1)), providing a reconstruction target for anatomical transitions.

Transfer to an unseen modality. We evaluate frozen tokenizer features on PBC [68], a blood-cell microscopy dataset. This dataset and its hematology modality are absent from tokenizer pretraining.

We use $k$-NN classification and train a linear classifier on frozen features. MedVTok achieves the highest accuracy under both protocols: 94.78% and 96.84%, respectively (Table 18).

Model	$k$-NN	Linear probe
MedVAE	$40.47\pm0.15$	$79.47\pm1.35$
PUMIT	$36.98\pm0.55$	$83.25\pm0.84$
MedITok	$90.73\pm0.62$	$94.62\pm0.76$
AToken	$92.00\pm2.07$	$92.84\pm0.87$
MedVTok	$94.78\pm1.81$	$96.84\pm0.59$

Comparison on the same training data. For controlled comparison with state-of-the-art tokenizers, we retrain AToken, MedITok, and MedVTok on MedITok’s 2M image–text dataset using the same training settings. AToken and MedITok use text supervision; MedVTok uses the images alone. MedVTok achieves the highest reconstruction PSNR, classification accuracy, and segmentation Dice in this comparison (Table 19).

Model	PSNR (dB)	Accuracy (%)	Dice (%)
AToken	$39.35$	$63.65$	$69.57$
MedITok	$36.83$	$66.59$	$67.01$
MedVTok	$39.71$	$71.70$	$73.21$

$t$-SNE analysis. To qualitatively assess the discriminative structure of the learned representations, we visualize $t$-SNE embeddings of frozen tokenizer features in Figure 11. We compare PUMIT, MedVAE, MedITok, and MedVTok on two datasets: (a) a microscopy blood cell dataset PBC [68] (covering multiple cell types: basophil, eosinophil, erythroblast, immature granulocytes, lymphocyte, monocyte, neutrophil, platelet), which is an out-of-distribution modality not seen during training, and (b) the JSIEC [33] fundus dataset. On the PBC dataset, MedVTok produces clearly separated clusters for each cell type despite never seeing microscopy images during training, indicating strong generalization of the learned semantic structure. On JSIEC, MedVTok forms tighter, more coherent clusters for fine-grained retinal conditions compared to the baselines, which exhibit more inter-class overlap. These visualizations corroborate the quantitative classification and retrieval results, confirming that MedVTok encodes semantically meaningful representations that generalize across both in-distribution and out-of-distribution modalities.

Broader Impact

A general-purpose medical visual tokenizer can accelerate multimodal medical AI research and patient-level modeling by providing a shared, reusable visual interface across imaging modalities. This may lower the barrier for building clinical decision-support tools, particularly for under-resourced specialties or institutions that lack the data and compute. By open-sourcing the model and code, we aim to promote reproducibility and equitable access to medical AI infrastructure.

As with any medical AI component, there is a risk that downstream systems built on MedVTok could produce incorrect predictions that, if acted upon without appropriate clinical oversight, may lead to patient harm. We will release MedVTok under the Apache 2.0 license with clear documentation stating that it is intended for research use. We encourage downstream users to rigorously validate performance on their target populations and to comply with applicable regulations before any clinical deployment.

Can you identify any other features in the image that might indicate the presence of melanoma?

Yes, there are several features that could suggest the presence of melanoma. For example, the lesion has irregular borders, which is another common characteristic of melanoma. Additionally, the lesion appears to be asymmetrical, another hallmark of melanoma.

What is the grading of diabetic retinopathy in the fundus color photo?

A: proliferative diabetic retinopathy.

B: no diabetic retinopathy.

C: moderate nonproliferative diabetic retinopathy.

D: mild nonproliferative diabetic retinopathy.

E: severe nonproliferative diabetic retinopathy.

D: Mild nonproliferative diabetic retinopathy.

Are there any structures in the image that suggest inflammation or infection?

Yes, there are structures in the image that suggest inflammation or infection. For example, there is a dense cellular infiltrate with some small dark cells that may represent lymphocytes. This indicates that the body's immune system is responding to some kind of threat, which can be seen in inflammation.

Dataset	Modality	Count
Isic2024 [128]	Dermoscopy	618,536
Derm1M [21]	Dermoscopy	412,886
Isic2020 [129]	Dermoscopy	44,108
Severance [130]	Dermoscopy	35,170
BCN20000 [131]	Dermoscopy	18,946
Fitzpatrick17k [132]	Dermoscopy	16,577
DERM12345 [133]	Dermoscopy	12,345
MILK10k [134]	Dermoscopy	10,480
SurgeNetYoutube [135]	Endoscopy	2,061,065
EndoVis2023-Surgtoolloc [136]	Endoscopy	745,751
SARAS-MESAD [137]	Endoscopy	35,132
EndoVis2023-PitVis [138]	Endoscopy	19,967
In-house Fundus Dataset	CFP	2,811,937
Airogs [139]	CFP	101,442
KaggleDR [140]	CFP	88,712
Eyepacs [141]	CFP	35,126
BIOMEDICA [142]	CFP, Dermoscopy, X-ray	253,443
HISTAI [143]	Histopathology	4,000,000
PathGen [144]	Histopathology	1,606,060
Bone Marrow Cytomorphology [145]	Histopathology	27,783
Lysto [146]	Histopathology	14,920
BraTS2023 [147]	MRI	973,066
IXI [148]	MRI	924,426
BrainTumour [149]	MRI	246,671
Learn2Reg [150]	MRI	89,572
Vestibular-Schwannoma-SEG [151]	MRI	51,574
OASIS2 [152]	MRI	39,741
ATLAS 2 [153]	MRI	36,029
GLIS-RT [154]	MRI	24,979
CrossMoDA2023 [155]	MRI	21,981
CAS2023 [156]	MRI	19,200
ASC18 [157]	MRI	13,552
OLIVES [158]	OCT	152,346
OCT2017 [159]	OCT	84,484
APTOS-2021 [160]	OCT	56,972
Retina-OCT-C8 [161]	OCT	24,000
ACOUSLIC [162]	Ultrasound	210,929
SYSU-CEUS-FLL-intv1 [163]	Ultrasound	43,611
CACTUS [164]	Ultrasound	37,736
RSNA Mammography [165]	X-ray	54,710
DDSM [166]	X-ray	20,800
VinDr-Mammo [167]	X-ray	19,982
MIMIC-CXR-JPG [168]	X-ray	372,693
ReXGradient-160K [169]	X-ray	273,004
CheXpertPlus [170]	X-ray	223,315
PadChest [171]	X-ray	160,839
NIH Chest X-rays [172]	X-ray	112,120
COVIDx CXR-4 [173]	X-ray	84,817
BRAX [174]	X-ray	40,944
Ranzcr clip [175]	X-ray	33,665
RSNA PDC [176]	X-ray	29,684
VinBigData CXR [177]	X-ray	9,850
Total		28,769,502