WorksMathematics
Integrating Latent Block Models into Variational Autoencoders for Disentangled Representation Learning
Abstract
We introduce LBM-VAE, a novel framework that integrates Latent Block Models (LBMs) into Variational Autoencoders (VAEs) to jointly cluster observations and latent dimensions. Beyond its clustering capabilities, LBM-VAE provides a block-structured latent representation that naturally supports disentangled representation learning. In contrast to conventional approaches that associate each factor of variation with an individual latent dimension, our framework allows factors of variation to be encoded across groups of latent dimensions. Experimental results demonstrate the promising clustering performance of LBM-VAE, while achieving competitive results in disentangled representation learning. LBM-VAE highlights the potential of combining co-clustering and variational inference to learn structured and interpretable latent representations.
Introduction
Disentangled representation learning seeks to recover latent codes in which distinct factors of variation are encoded separately [1]. VAE-based methods such as -VAE [2], FactorVAE [3] and -TCVAE [4] promote the aggregate posterior towards a factorial distribution, assuming that each factor is encoded in a single latent dimension. However, this assumption may be overly restrictive, as certain factors, such as color, naturally exhibit a multi-dimensional structure.
Existing approaches, however, do not learn such block structures from data. In disentangled representation learning, subspace-based approaches such as ISA-VAE [5] partition latent coordinates into independent subspaces. However, both the number and dimensionality of these subspaces are specified a priori. In deep clustering, mixture-based priors such as VaDE [6] cluster the observations, with each component spanning the entire latent space. MFCVAE [7] extends this paradigm by introducing multiple facets, each associated with its own latent variables and mixture prior. Nevertheless, the latent dimensions assigned to each facet must be specified in advance by the user, which is not straightforward in practice. In contrast, we let the model discover this structure automatically, by replacing the conventional Gaussian prior with a Latent Block Model (LBM) [8]. The LBM performs a simultaneous co-clustering of the latent code matrix, clustering observations into row clusters and latent dimensions into column clusters, with a distinct Gaussian per block. Given only an upper bound on the number of blocks, the model learns which latent dimensions are grouped together and the size of each block, so that groups of dimensions capturing common factors of variation emerge directly from the data. Our contributions are summarised as follows: (1) we introduce LBM-VAE, a variational autoencoder whose prior jointly infers partitions of both observations and latent dimensions (co-clustering); (2) we derive coordinate-ascent updates for the variational approximations of both partition posteriors and develop a mini-batch training scheme that updates the column partition using memoized sufficient statistics; and (3) we evaluate the proposed approach through experiments on MNIST and Shapes3D, demonstrating its ability to learn structured latent representations.
Background
Let , , , be positive integers, , the probability simplex of , and the categorical distribution on with probabilities . The LBM [8] assumes that a data matrix is generated as follows:
Memberships and are drawn independently, and the entries of a block are i.i.d. given them. With , the likelihood is
This sum has terms and the posterior does not admit a factorized form, so exact EM is intractable. Variational EM [8] restricts the posterior to the mean-field family and alternates coordinate-ascent updates of the responsibilities and with a closed-form M-step.
The LBM-VAE framework
Generative Model
For observations and latent codes , the generative model is
where is a decoder distribution parametrised by a neural network with weights , with density .
Inference and Evidence Lower Bound
We use (mean field), with . The ELBO reads
As in [8], coordinate-ascent variational inference (CAVI) [9] yields explicit updates for the variational approximations of both label posteriors:
The row posterior only involves observation , whereas depends on the whole column of encoder outputs.
Training and Memoized Variational Inference
The row responsibilities are local: their update (6) only involves observation and is computed on the current mini-batch . In contrast, is a global variational parameter: its update (6) depends on through , which aggregates all observations, and since each training step changes , an exact update would require re-encoding the whole dataset after every step. We therefore adapt memoized online variational inference [10] to our case, which updates global parameters from cached per-batch sufficient statistics whose sums summarise the full dataset. For each mini-batch , we store
which are indexed by latent coordinate and row cluster , rather than by block. With the totals , , summed over mini-batches,
After the gradient step on , its statistics are recomputed with the updated encoder and swapped into the totals (old contribution subtracted, new one added), and is recomputed from (8); is updated at the end of each epoch.
Experiments
We conduct two experiments: clustering on MNIST [11], which evaluates the row partition against the digit classes (ARI, NMI, accuracy), and disentanglement on Shapes3D [12], which evaluates the block-structured representation against VAE-based baselines (MIG [4], SAP [13], DCI [14]).
Implementation settings. On MNIST, the encoder has three stride-2 convolutional layers (32, 32 and 64 channels) followed by two fully connected layers of 256 units, with ReLU activations and a mirrored decoder; we use , and . On Shapes3D, all models use the convolutional encoder and decoder of [15], so that differences in scores come from the prior and the objective. Baselines use as in [15], and, for the HFS variants [16], , among the values considered in [17]. The LBM-VAE uses , since the column partition becomes trivial when is close to , with at most blocks (number of ground-truth factors) and row clusters (number of object shapes). Models are trained with Adam (learning rate , batch size 256) for 200 epochs, the first 30 as a standard VAE before the LBM prior is activated, on one NVIDIA H100 GPU on Jean Zay supercomputer (IDRIS). Scores are reported as mean std over 10 seeds at the final epoch.
Clustering results. On the MNIST test set, the row partition reaches an ARI of , an NMI of and an accuracy of . These scores remain below those of dedicated deep clustering models such as VaDE [6], which is expected: co-clustering is a harder problem, since the blocks are unknown and each row cluster is described by a single Gaussian per block. Figure 1 shows, on a separate run with only 5 pretraining epochs, digit classes separating once the LBM prior is activated.

Figure 1. t-SNE projections of the MNIST test-set latent means at epochs 5, 30, 55 and 90, coloured by digit class (LBM prior activated at epoch 5).
Disentanglement results. Figure 2 shows the learned blocks. Clusters 1 and 6 illustrate the intended behaviour: the coordinates of Cluster 1 act only on the object orientation, and both coordinates of Cluster 6 control the wall hue. Cluster 5 encodes the object scale, and Cluster 4 gathers coordinates that are left unused by the decoder. Cluster 3 still mixes object hue, shape and floor hue.

Figure 2. Latent traversals of the LBM-VAE on Shapes3D. Each column varies one of the 15 latent coordinates and each row one traversal value. Columns are grouped by learned cluster.
Table 1 reports the scores. Without additional regularization, LBM-VAE is competitive with the best baselines: it obtains the highest mean SAP and DCI-D and comparable completeness and informativeness, while its MIG score remains below those of the HFS variants. This is consistent with a block-wise encoding: DCI-D only requires each coordinate to inform a single factor, which holds when a factor spans a block, whereas MIG rewards concentrating each factor in one coordinate.
| Model | MIG | SAP | D | C | I |
| -VAE | 0.268 ± 0.079 | 0.077 ± 0.037 | 0.603 ± 0.129 | 0.522 ± 0.118 | 0.926 ± 0.045 |
| -TCVAE | 0.312 ± 0.171 | <u>0.092 ± 0.030</u> | 0.741 ± 0.085 | 0.637 ± 0.084 | 0.977 ± 0.015 |
| -VAE + HFS | 0.616 ± 0.181 | 0.085 ± 0.055 | 0.765 ± 0.179 | 0.788 ± 0.109 | 0.920 ± 0.093 |
| -TCVAE + HFS | <u>0.534 ± 0.204</u> | <u>0.078 ± 0.029</u> | <u>0.848 ± 0.079</u> | 0.779 ± 0.088 | 0.991 ± 0.007 |
| LBM-VAE | 0.452 ± 0.124 | 0.129 ± 0.027 | 0.899 ± 0.107 | <u>0.780 ± 0.095</u> | <u>0.990 ± 0.018</u> |
Table 1. Disentanglement scores on Shapes3D (best mean in bold, second best underlined).
Conclusion
We introduced LBM-VAE, a variational autoencoder whose Latent Block Model prior jointly co-clusters observations and latent dimensions, using coordinate-ascent variational updates and a memoized update of the column partition. The learned blocks group latent coordinates that capture common factors of variation, yielding competitive disentanglement without any additional regularization term. A limitation is that all latent blocks share a single row partition, whereas distinct factors may induce distinct clusters of the observations. Future work will therefore allow each block to learn its own observation partition, thereby extending LBM-VAE towards a multiview clustering, unlike MFCVAE [7] where the views are predefined by the user.
Acknowledgments
This work was carried out within IRT VEDECOM, with the support of its industrial partners Stellantis and Renault Group, and in collaboration with the DAVID laboratory, UVSQ, Université Paris-Saclay. It was supported by the French government under the "France 2030" operated by the French National Research Agency. This work was granted access to the HPC resources of IDRIS under the allocation 2026-AD011017901R1 made by GENCI, on the Jean Zay supercomputer.
References
- [1]Xin Wang, Hong Chen, Si’ao Tang, Zihao Wu, and Wenwu Zhu. Disentangled representation learning. IEEE Trans. Pattern Anal. Mach. Intell., 46(12):9677–9696, December 2024.DOI
- [2]Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Andriy Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2016.
- [3]Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2649–2658. PMLR, 10–15 Jul 2018.arxiv.org/abs/1802.05983
- [4]Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
- [5]Jan Stühmer, Richard Turner, and Sebastian Nowozin. ISA-VAE: Independent subspace analysis with variational autoencoders, 2019.
- [6]Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: an unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI’17, page 1965–1972. AAAI Press, 2017.arxiv.org/abs/1611.05148
- [7]Fabian Falck, Haoting Zhang, Matthew Willetts, George Nicholson, Christopher Yau, and Chris C Holmes. Multi-facet clustering variational autoencoders. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8676–8690. Curran Associates, Inc., 2021.arxiv.org/abs/2106.05241
- [8]Gérard Govaert and Mohamed Nadif. Model-Based Co-Clustering, chapter 2, pages 55–77. John Wiley & Sons, Ltd, 2013.DOI
- [9]C.M. Bishop. Pattern recognition and machine learning, volume 4. Springer New York, 2006.
- [10]Michael C Hughes and Erik Sudderth. Memoized online variational inference for dirichlet process mixture models. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
- [11]Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010.
- [12]Chris Burgess and Hyunjik Kim. 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/, 2018.
- [13]Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. ArXiv, abs/1711.00848, 2017.arxiv.org/abs/1711.00848
- [14]Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
- [15]Francesco Locatello, Stefan Bauer, Mario Lucic̀, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Frederic Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, 2019. Best Paper Award.arxiv.org/abs/1811.12359
- [16]Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentanglement of correlated factors via hausdorff factorized support. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2210.07347
- [17]Théo Uscidda, Luca Eyring, Karsten Roth, Fabian J Theis, Zeynep Akata, and marco cuturi. Disentangled representation learning through geometry preservation with the gromov-monge gap. In ICML 2024 Workshop on Structured Probabilistic Inference & Generative Modeling, 2024.arxiv.org/abs/2407.07829