WorksComputer Science
Dual-VaDE: Efficient and Robust Prototype Learning via Variational Dual-Domain Mixture Priors
Abstract
In this paper, we propose Dual-VaDE, an unsupervised generative framework for learning interpretable anatomical prototypes from unlabeled medical images. In contrast to previous variational methods that rely on a single spatial latent code, which often entangles global geometry with local textures and fails to capture their cross-domain dependencies, Dual-VaDE introduces a dual-domain Gaussian mixture prior that explicitly couples spatial anatomy with phase-preserving Fourier spectral representations. We derive an exact, closed-form variational lower bound that fuses this joint prior into the decoder and optimizes the entire network end-to-end via an alternating EM formulation, avoiding heuristic two-stage clustering while ensuring superior training stability and robustness. Evaluated on OASIS-3 brain MRI, Dual-VaDE delivers substantially sharper and more plausible prototypes, achieves superior cluster separation with improvements of up to 94% over spatial-only baselines, exhibits markedly reduced variance across random seeds, and incurs negligible inference overhead (0.28 ms per image).
Introduction
High-dimensional medical imaging exhibits complex morphological variability that conventional unsupervised representation learning often fails to disentangle. Unlike supervised pipelines built on predefined categories or diagnostic labels, unsupervised distribution modeling can capture the continuous spectrum of anatomical heterogeneity directly from raw cohorts [1]. Deep generative models enable density estimation and unsupervised representation learning without manual annotations [2, 3, 4, 5]. Compared with discriminative clustering, generative methods can improve interpretability by decoding latent representations into anatomical prototypes—inspectable templates that support subsequent morphological analysis [6].
Variational autoencoders (VAEs) with Gaussian mixture priors [7, 8, 9] link each mixture component to a discrete morphological centroid, where decoding the mean yields a prototype. However, typical formulations use a single spatial latent vector, entangling global geometry (e.g., compartment volume) with local appearance (e.g., contrast and boundary texture). With unstructured diagonal or unimodal covariances, such priors fail to capture cross-scale geometry–texture dependencies, often producing blurred or degenerate prototypes. The Fourier domain offers a complementary view of the image manifold: low frequencies largely determine spectral energy, while phase preserves topology, boundaries, and edges [10]. Magnitude-only models drop this geometric phase information, whereas complex-valued representations [11, 12, 13] or real–imaginary decompositions [14, 15] retain full anatomical content. Although dual-domain fusion is popular [16, 17, 18], deterministic frequency features alone are insufficient and often redundant with spatial cues (Sec. 4). Learning coherent morphological prototypes requires a structured prior that explicitly couples spatial and spectral latents.
We present Dual-VaDE, an unsupervised prototype-learning framework that factorizes latents into coupled spatial and spectral subspaces. Our contributions are:
Dual-domain mixture prior: We propose a dual-domain Gaussian mixture prior, parameterized by a newly designed mean and covariance matrix whose off-diagonal cross-covariance terms explicitly capture within-prototype spatial–spectral co-variation.
End-to-End Variational EM Optimization: We derive a new closed-form variational lower bound, optimized end-to-end via EM, avoiding multi-stage error accumulation and improving training stability and clustering robustness.
Empirical evaluation and open-source implementation: On structural brain MRI, Dual-VaDE yields sharper prototypes, better cluster separability, and more stable optimization than eight generative baselines with minimal overhead.
Background
Let be a medical image, a continuous latent code, and a discrete prototype index. VaDE [7] assumes the generative process , , and , with an amortized posterior and . The evidence lower bound (ELBO) is
where are the posterior prototype responsibilities [7, 8, 19], and related unsupervised clustering methods are explored in [19]. Because a single code must capture both global anatomy and local texture, each is an unstructured matrix without factor blocks, and the model cannot represent cross-domain dependencies explicitly. Multi-view generative models address a related problem by assigning one latent code per view [20]; we follow this idea within the prior by giving it two codes whose covariance blocks follow this split.
Methodology: Dual-VaDE
Dual-domain mixture prior
We keep the generative process of Sec. 2 and change only the prior. Each image is embedded in a joint spatial and spectral latent space by concatenating a spatial code and a spectral code , both in , into , on which we impose a Gaussian mixture with block-structured parameters:
Here, and are the within-prototype spatial and spectral covariances, and the cross-domain block captures how anatomy and spectral content co-vary within prototype .
To obtain a single generative path, we define with , learnable , and scalars . A linear map of a Gaussian mixture is again a Gaussian mixture with unchanged weights [21], whose parameters are
\Sigma_k^\star=\beta^2W_sS_k^sW_s^{\mathsf{T}}+\alpha^2W_fS_k^fW_f^{\mathsf{T}}+\alpha\beta\left(W_sC_kW_f^{\mathsf{T}}+W_fC_k^{\mathsf{T}}W_s^{\mathsf{T}}\right).
The covariance thus decomposes exactly into a spatial, a spectral, and a cross-domain term, the last of which vanishes when . The resulting dual-domain mixture prior on the decoder input is
which reduces to the VaDE prior for , , and ; we use this setting as initialization.
Phase-preserving spectral encoder
The spectral code is computed from the centered spectrum . Since magnitude-only inputs are invariant to phase and thus discard spatial localization [10], we represent by its real and imaginary parts as two channels, as in recent spectral networks [14, 15], on which the complex-valued layers [11] are implemented2. The final complex feature map is converted to real features by the real-part readout
which is linear and injective on Hermitian-symmetric spectra and therefore preserves both the magnitude and the relative phase of .
Inference and objective
The two encoders share a single decoder and yield and . The push-forward through gives the fused posterior with and , where the learnable weights sum to one. Training minimizes the dual-domain negative ELBO
In contrast to Eq. (1), the KL term is evaluated against the structured components of Eq. (3), so that the spatial and spectral codes are coupled through . The mixture parameters are updated by expectation-maximization [22] between gradient steps. The E-step computes responsibilities in the fused space,
whereas the M-step operates in the joint space, so that every block of , including , is estimated. With and ,
and Eq. (3) maps the updated parameters to the decoder input.
Experiments
Data. We use 253 OASIS-3 brain MR images [23] from Alzheimer’s patients aged 63–75 to reduce age/diagnosis confounding and capture within-disease anatomical heterogeneity. Images are registered, resampled to , and split into 202 train and 51 test.
Implementation, Baselines & Metrics
For a controlled comparison, all models share the same encoder/decoder and training schedule; they differ only in which latent code uses a mixture prior. We train with seeds 42/92/123 and . After training, we fit a -component GMM to posterior means on the training set and assign each test image to a prototype.
We compare VaDE [7] (mixture prior on the spatial code), Freq-VaDE (published spectral encoder), and Dual-VaDE (our spatial branch plus the dual-domain prior of Sec. 3 on the same spectral encoder). We test four Fourier inputs: full magnitude (FullFreq) [16], low-frequency magnitude (LowFreq) [24, 17], full complex spectrum with complex convolutions (PolarFreq) [11, 13], and real/imaginary channels with real convolutions (ReIm) [14, 12]. Unless noted, we use Dual-VaDE with ReIm. We report test Silhouette [25] and Davies-Bouldin (DB) [26] scores (mean SD over seeds) and compare to baselines with a one-sided Welch’s -test [27]. Inference time is averaged over ten runs on an NVIDIA RTX PRO 6000 Blackwell GPU.
Prototype separation. In Table 1, Dual-VaDE achieves the best Silhouette for all (0.648, 0.610, 0.447) and the lowest DB at and 7 (mean rank 1.08 vs. 4.08 for the runner-up); at its DB (0.449) ties Freq-VaDE+LowFreq. Versus VaDE, it raises Silhouette by 39.1%, 93.7%, and 13.7% and lowers DB by 46.4%, 61.9%, and 21.6% at , 5, and 7, with significant gains () in 37/48 comparisons. Fusion effects vary by phase: moving from Freq-VaDE to Dual-VaDE, magnitude-only FullFreq/LowFreq reduce Silhouette at and 5, while phase-preserving PolarFreq/ReIm improve for all , suggesting magnitude overlaps with the spatial code but phase provides complementary information. LowFreq is strong for but fails at (Silhouette 0.058, DB 2.003).
| Model | Spectral input | Phase | Silhouette | Silhouette | Silhouette | Davies-Bouldin | Davies-Bouldin | Davies-Bouldin | Avg. rank |
| VaDE [7] | None | 6.67 | |||||||
| Freq-VaDE | FullFreq [16] | ✗ | 4.33 | ||||||
| LowFreq [17] | ✗ | 4.25 | |||||||
| PolarFreq [13] | ✓ | 8.33 | |||||||
| ReIm [12] | ✓ | 6.17 | |||||||
| Dual-VaDE | FullFreq [16] | ✗ | 5.25 | ||||||
| LowFreq [17] | ✗ | 4.08 | |||||||
| PolarFreq [13] | ✓ | 4.83 | |||||||
| Dual-VaDE (ours) | ReIm | ✓ | <u></u> | 1.08 | |||||
| Relative change of ours vs. VaDE | +39.1% | +93.7% | +13.7% | −46.4% | −61.9% | −21.6% |
Table 1. Prototype separation on OASIS-3 (mean±SD, 3 seeds). Silhouette (↑, [−1, 1]): closer to own than nearest other prototype. Davies-Bouldin (↓): within-prototype scatter vs between-prototype distance. Phase: Fourier phase preserved. Dual-VaDE augments the cited spectral encoder with a spatial branch and our dual-domain prior. Bold/underline: best/second best. †, ‡: ours significantly better (one-sided Welch’s -test).
Prototypes and latent structure. Figure 2(a) compares learned prototypes and latent embeddings for our model, two single-domain variants, and the strongest baseline (Dual-VaDE with LowFreq), using one seed per and Hungarian-matched labels [28]. The t-SNE [29] plots align with Table 1: VaDE prototypes overlap (), while ours are more separated, especially at () and (). At (VaDE) and for both in Dual-VaDE with LowFreq, some prototypes show large variance deviations near the lateral ventricles and periventricular white matter (red arrows), consistent with enlargement markers [30]. Our deviations are smaller and more uniform, and the prototypes remain sharp.

Figure 2. (a) Prototypes with relative-variance overlays (red: above mean, blue: below; red arrows: large ventricular deviations) and t-SNE for and . Labels match ours, so colors indicate the same prototype across rows. : images per prototype; : latent separation. (b) Ours vs. single-branch variants. (c) Principal angles between our spatial and spectral latent subspaces (dots) and their mean (bars).
Efficiency, robustness, and complementarity. Figure 1 (right) shows that our model is the most accurate and among the fastest spectral models: mean Silhouette 0.568 vs. 0.493 next best, at 0.277 ms/image (Freq-VaDE with ReIm: 0.276 ms), and 20%/22% faster than PolarFreq/LowFreq. It is also the most robust, with the lowest across-seed SD (0.015 averaged over vs. 0.020–0.039) and the best worst-case over (0.447).

Figure 1. Dual-VaDE overview. Left: a spatial encoder and a phase-preserving spectral encoder (real/imaginary Fourier channels) produce two latent codes. A dual-domain mixture prior couples them via the cross-domain block . The fused code feeds a shared decoder to generate reconstructions and prototypes. Right: prototype separation (mean Silhouette over ) vs. throughput, and robustness measured as seed-to-seed variability under the worst case over .
Figure 2b shows that the full model beats the better single branch by 0.038, 0.114, and 0.054 Silhouette at , 5, and 7 (8/9 seeds). Figure 2c links this to complementarity: principal angles [31] between the top-10-PC subspaces of spatial and spectral latent means average 46.5°, 48.6°, and 55.3° for , 5, and 7, with some near-orthogonal directions. Thus, the branches capture distinct factors, making the cross-domain block informative.
Conclusion & Discussion
This work introduces Dual-VaDE, an unsupervised deep generative framework that jointly models latent mixture distributions across coupled spatial and Fourier domains. By explicitly encoding cross-domain covariance and phase-preserving spectral components, the formulation regularizes the latent manifold against high-frequency blur and prevents mode collapse. Model parameters are inferred through an analytically grounded expectation-maximization scheme tailored for the coupled dual-domain objective. Evaluated on real brain MRI cohorts, Dual-VaDE uncovers distinct neuroanatomical subtypes with sharp cortical and ventricular boundaries while substantially mitigating the random-seed sensitivity common to deep mixture clustering. These geometric and statistical gains are achieved without prohibitive parameter scaling or inference bottlenecks. Future work will extend this dual-domain mixture formulation to continuous spatiotemporal manifolds, establishing a unified generative prior for mapping longitudinal structural atrophy trajectories.
Acknowledgments & Ethics
This research was conducted using open-access data resources and received no external funding.
References
- [1]Mohamad Habes, Michel J. Grothe, Birkan Tunc, et al., “Disentangling heterogeneity in Alzheimer’s disease and related dementias using data-driven methods,” Biol. Psychiatry, vol. 88, no. 1, pp. 70–82, 2020.DOI
- [2]Diederik P. Kingma and Max Welling, “Auto-encoding variational Bayes,” in Proc. ICLR, 2014.arxiv.org/abs/1312.6114
- [3]Jonathan Ho, Ajay Jain, and Pieter Abbeel, “Denoising diffusion probabilistic models,” in Proc. NeurIPS, 2020, vol. 33, pp. 6840–6851.arxiv.org/abs/2006.11239
- [4]Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, et al., “Diffusion models in medical imaging: A comprehensive survey,” Med. Image Anal., vol. 88, pp. 102846, 2023.DOI
- [5]Evan M Yu and Mert R Sabuncu, “A convolutional autoencoder approach to learn volumetric shape representations for brain structures,” arXiv preprint arXiv:1810.07746, 2018.arxiv.org/abs/1810.07746
- [6]Zhijian Yang, Ilya M. Nasrallah, Haochang Shou, et al., “A deep learning framework identifies dimensional representations of Alzheimer’s disease from brain structure,” Nat. Commun., vol. 12, pp. 7065, 2021.DOI
- [7]Zhuxi Jiang, Yin Zheng, Huachun Tan, et al., “Variational deep embedding: An unsupervised and generative approach to clustering,” in Proc. IJCAI, 2017, pp. 1965–1972.arxiv.org/abs/1611.05148
- [8]Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garnelo, et al., “Deep unsupervised clustering with Gaussian mixture variational autoencoders,” arXiv preprint arXiv:1611.02648, 2016.arxiv.org/abs/1611.02648
- [9]Fabian Falck, Haoting Zhang, Matthew Willetts, George Nicholson, Christopher Yau, and Chris C Holmes, “Multi-facet clustering variational autoencoders,” Advances in Neural Information Processing Systems, vol. 34, pp. 8676–8690, 2021.arxiv.org/abs/2106.05241
- [10]Alan V. Oppenheim and Jae S. Lim, “The importance of phase in signals,” Proc. IEEE, vol. 69, no. 5, pp. 529–541, 1981.DOI
- [11]Chiheb Trabelsi, Olexa Bilanuk, Ying Zhang, et al., “Deep complex networks,” in Proc. ICLR, 2018.arxiv.org/abs/1705.09792
- [12]Aleksandar Stanić, Anand Gopalakrishnan, Kazuki Irie, et al., “Contrastive training of complex-valued autoencoders for object discovery,” in Proc. NeurIPS, 2023, vol. 36, pp. 11075–11101.arxiv.org/abs/2305.15001
- [13]Anand Gopalakrishnan, Aleksandar Stanić, Jürgen Schmidhuber, et al., “Recurrent complex-weighted autoencoders for unsupervised object discovery,” in Proc. NeurIPS, 2024, vol. 37, pp. 140787–140811.arxiv.org/abs/2405.17283
- [14]Lu Chi, Borui Jiang, and Yadong Mu, “Fast Fourier convolution,” in Proc. NeurIPS, 2020, vol. 33.
- [15]Yongming Rao, Wenliang Zhao, Zheng Zhu, et al., “Global filter networks for image classification,” in Proc. NeurIPS, 2021.arxiv.org/abs/2107.00645
- [16]Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz, “Leveraging frequency analysis for deep fake image recognition,” in Proc. ICML, 2020, vol. 119 of PMLR, pp. 3247–3258.arxiv.org/abs/2003.08685
- [17]Oren Rippel, Jasper Snoek, and Ryan P. Adams, “Spectral representations for convolutional neural networks,” in Proc. NeurIPS, 2015, vol. 28.arxiv.org/abs/1506.03767
- [18]Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren, “Learning in the frequency domain,” in Proc. CVPR, 2020, pp. 1740–1749.arxiv.org/abs/2002.12416
- [19]Junyuan Xie, Ross Girshick, and Ali Farhadi, “Unsupervised deep embedding for clustering analysis,” in Proc. ICML, 2016, pp. 478–487.arxiv.org/abs/1511.06335
- [20]Yuge Shi, N. Siddharth, Brooks Paige, et al., “Variational mixture-of-experts autoencoders for multi-modal deep generative models,” in Proc. NeurIPS, 2019, vol. 32.arxiv.org/abs/1911.03393
- [21]Christopher M. Bishop, Pattern Recognition and Machine Learning, Springer, New York, 2006.
- [22]Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. R. Stat. Soc. B, vol. 39, no. 1, pp. 1–38, 1977.DOI
- [23]Pamela J. LaMontagne, Tammie L. S. Benzinger, John C. Morris, et al., “OASIS-3: Longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and Alzheimer disease,” medRxiv, 2019.DOI
- [24]Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, et al., “Fourier neural operator for parametric partial differential equations,” in Proc. ICLR, 2021.
- [25]Peter J. Rousseeuw, “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,” J. Comput. Appl. Math., vol. 20, pp. 53–65, 1987.DOI
- [26]David L. Davies and Donald W. Bouldin, “A cluster separation measure,” IEEE Trans. Pattern Anal. Mach. Intell., vol. PAMI-1, no. 2, pp. 224–227, 1979.pubmed.ncbi.nlm.nih.gov/21868852
- [27]Bernard L. Welch, “The generalization of “Student’s” problem when several different population variances are involved,” Biometrika, vol. 34, no. 1–2, pp. 28–35, 1947.DOI
- [28]Harold W. Kuhn, “The Hungarian method for the assignment problem,” Nav. Res. Logist. Q., vol. 2, no. 1–2, pp. 83–97, 1955.
- [29]Laurens van der Maaten and Geoffrey Hinton, “Visualizing data using t-SNE,” J. Mach. Learn. Res., vol. 9, pp. 2579–2605, 2008.
- [30]Sean M. Nestor et al., “Ventricular enlargement as a possible measure of Alzheimer’s disease progression validated using the Alzheimer’s disease neuroimaging initiative database,” Brain, vol. 131, no. 9, pp. 2443–2454, 2008.DOI
- [31]Åke Björck and Gene H. Golub, “Numerical methods for computing angles between linear subspaces,” Math. Comput., vol. 27, pp. 579–594, 1973.DOI