betaFeedback

WorksMathematics

Integrating Latent Block Models into Variational Autoencoders for Disentangled Representation Learning

Abstract

We introduce LBM-VAE, a novel framework that integrates Latent Block Models (LBMs) into Variational Autoencoders (VAEs) to jointly cluster observations and latent dimensions. Beyond its clustering capabilities, LBM-VAE provides a block-structured latent representation that naturally supports disentangled representation learning. In contrast to conventional approaches that associate each factor of variation with an individual latent dimension, our framework allows factors of variation to be encoded across groups of latent dimensions. Experimental results demonstrate the promising clustering performance of LBM-VAE, while achieving competitive results in disentangled representation learning. LBM-VAE highlights the potential of combining co-clustering and variational inference to learn structured and interpretable latent representations.

Introduction

Disentangled representation learning seeks to recover latent codes in which distinct factors of variation are encoded separately [1]. VAE-based methods such as β\beta-VAE [2], FactorVAE [3] and β\beta-TCVAE [4] promote the aggregate posterior towards a factorial distribution, assuming that each factor is encoded in a single latent dimension. However, this assumption may be overly restrictive, as certain factors, such as color, naturally exhibit a multi-dimensional structure.

Existing approaches, however, do not learn such block structures from data. In disentangled representation learning, subspace-based approaches such as ISA-VAE [5] partition latent coordinates into independent subspaces. However, both the number and dimensionality of these subspaces are specified a priori. In deep clustering, mixture-based priors such as VaDE [6] cluster the observations, with each component spanning the entire latent space. MFCVAE [7] extends this paradigm by introducing multiple facets, each associated with its own latent variables and mixture prior. Nevertheless, the latent dimensions assigned to each facet must be specified in advance by the user, which is not straightforward in practice. In contrast, we let the model discover this structure automatically, by replacing the conventional Gaussian prior with a Latent Block Model (LBM) [8]. The LBM performs a simultaneous co-clustering of the latent code matrix, clustering observations into row clusters and latent dimensions into column clusters, with a distinct Gaussian per block. Given only an upper bound LL on the number of blocks, the model learns which latent dimensions are grouped together and the size of each block, so that groups of dimensions capturing common factors of variation emerge directly from the data. Our contributions are summarised as follows: (1) we introduce LBM-VAE, a variational autoencoder whose prior jointly infers partitions of both observations and latent dimensions (co-clustering); (2) we derive coordinate-ascent updates for the variational approximations of both partition posteriors and develop a mini-batch training scheme that updates the column partition using memoized sufficient statistics; and (3) we evaluate the proposed approach through experiments on MNIST and Shapes3D, demonstrating its ability to learn structured latent representations.

Background

Let nn, pp, KK, LL be positive integers, [n]:={1,…,n}[n] := \{1,\ldots,n\}, ΔK−1\Delta^{K-1} the probability simplex of RK\mathbb{R}^{K}, and Cat⁡(π)\operatorname{Cat}(\boldsymbol{\pi}) the categorical distribution on [K][K] with probabilities π∈ΔK−1\boldsymbol{\pi} \in\Delta^{K-1}. The LBM [8] assumes that a data matrix Y=(yij)∈Rn×pY = (y_{ij}) \in\mathbb{R}^{n \times p} is generated as follows:

vi∼Cat⁡(π),wj∼Cat⁡(ρ),π∈ΔK−1, ρ∈ΔL−1,(1)v_i \sim\operatorname{Cat}(\boldsymbol{\pi}), \qquad w_j \sim\operatorname{Cat}(\boldsymbol{\rho}), \qquad\boldsymbol{\pi} \in\Delta^{K-1},\ \boldsymbol{\rho} \in\Delta^{L-1}, \tag*{(1)}
yij∣vi=k,wj=l∼N(μkl,σkl2),μkl∈R, σkl2>0.(2)y_{ij} \mid v_i = k, w_j = l \sim\mathcal{N}(\mu_{kl},\sigma_{kl}^{2}), \qquad\mu_{kl} \in\mathbb{R},\ \sigma_{kl}^{2} > 0. \tag*{(2)}

Memberships viv_i and wjw_j are drawn independently, and the entries of a block (k,l)(k,l) are i.i.d. given them. With η=(π,ρ,(μkl,σkl2)k,l)\boldsymbol{\eta} = (\boldsymbol{\pi},\boldsymbol{\rho},(\mu_{kl},\sigma_{kl}^{2})_{k,l}), the likelihood is

p(Y;η)=∑v∈[K]n∑w∈[L]p∏iπvi∏jρwj∏i,jN(yij;μviwj,σviwj2).(3)p(Y;\boldsymbol{\eta}) = \sum_{\boldsymbol{v}\in[K]^n}\sum_{\boldsymbol{w}\in[L]^p}\prod_i \pi_{v_i}\prod_j \rho_{w_j}\prod_{i,j}\mathcal{N}(y_{ij};\mu_{v_iw_j},\sigma_{v_iw_j}^{2}). \tag*{(3)}

This sum has KnLpK^nL^p terms and the posterior p(V,W∣Y)p(V,W\mid Y) does not admit a factorized form, so exact EM is intractable. Variational EM [8] restricts the posterior to the mean-field family q(V,W)=∏iq(vi)∏jq(wj)q(V,W)=\prod_i q(v_i)\prod_j q(w_j) and alternates coordinate-ascent updates of the responsibilities αik:=q(vi=k)\alpha_{ik}:=q(v_i=k) and βjl:=q(wj=l)\beta_{jl}:=q(w_j=l) with a closed-form M-step.

The LBM-VAE framework

Generative Model

For observations X=(x1,…,xn)X=(x_1,\ldots,x_n) and latent codes Z∈Rn×pZ\in\mathbb{R}^{n\times p}, the generative model is

vi∼Cat⁡(π),wj∼Cat⁡(ρ),i∈[n], j∈[p],(4)v_i \sim\operatorname{Cat}(\boldsymbol{\pi}), \qquad w_j \sim\operatorname{Cat}(\boldsymbol{\rho}), \qquad i\in[n],\ j\in[p], \tag*{(4)}
Zij∣vi=k,wj=l∼N(μkl,σkl2),i∈[n], j∈[p],(5)Z_{ij}\mid v_i=k,w_j=l\sim\mathcal{N}(\mu_{kl},\sigma_{kl}^{2}), \qquad i\in[n],\ j\in[p], \tag*{(5)}
xi∣Zi∼Dθ(Zi),i∈[n],(6)x_i\mid Z_i\sim D_{\theta}(Z_i), \qquad i\in[n], \tag*{(6)}

where DθD_{\theta} is a decoder distribution parametrised by a neural network with weights θ\theta, with density pθ(⋅∣Zi)p_{\theta}(\cdot\mid Z_i).

Inference and Evidence Lower Bound

We use q(Z,V,W∣X)=∏iqϕ(Zi∣xi)q(vi∣xi)∏jq(wj∣X)q(Z,V,W\mid X)=\prod_i q_{\phi}(Z_i\mid x_i)q(v_i\mid x_i)\prod_j q(w_j\mid X) (mean field), with qϕ(Zi∣xi)=N(mi,diag⁡si2)q_{\phi}(Z_i\mid x_i)=\mathcal{N}(\mathbf{m}_i,\operatorname{diag}\mathbf{s}_i^2). The ELBO reads

L=∑iEqϕ[log⁡pθ(xi∣Zi)]−∑i,jEqϕ[∑k,lαikβjllog⁡qϕ(Zij∣xi)N(Zij;μkl,σkl2)]\mathcal{L}=\sum_i\mathbb{E}_{q_{\phi}}\left[\log p_{\theta}(x_i\mid Z_i)\right]-\sum_{i,j}\mathbb{E}_{q_{\phi}}\left[\sum_{k,l}\alpha_{ik}\beta_{jl}\log\frac{q_{\phi}(Z_{ij}\mid x_i)}{\mathcal{N}(Z_{ij};\mu_{kl},\sigma_{kl}^{2})}\right]
−∑iKL⁡(αi∥π)−∑jKL⁡(βj∥ρ).(7)-\sum_i\operatorname{KL}(\alpha_i\Vert\boldsymbol{\pi})-\sum_j\operatorname{KL}(\beta_j\Vert\boldsymbol{\rho}). \tag*{(7)}

As in [8], coordinate-ascent variational inference (CAVI) [9] yields explicit updates for the variational approximations of both label posteriors:

eijkl:=Eqϕ[log⁡N(Zij;μkl,σkl2)]=−12(log⁡(2πσkl2)+(mij−μkl)2+sij2σkl2),e_{ij}^{kl} := \mathbb{E}_{q_\phi}\left[\log\mathcal{N}(Z_{ij};\mu_{kl},\sigma_{kl}^{2})\right] = -\frac{1}{2}\left(\log(2\pi\sigma_{kl}^{2})+\frac{(m_{ij}-\mu_{kl})^{2}+s_{ij}^{2}}{\sigma_{kl}^{2}}\right),
αik∝πkexp⁡(∑j,lβjleijkl),βjl∝ρlexp⁡(ℓjl),ℓjl:=∑i,kαikeijkl.(8)\alpha_{ik} \propto\pi_k \exp\left(\sum_{j,l}\beta_{jl}e_{ij}^{kl}\right),\qquad\beta_{jl} \propto\rho_l\exp(\ell_{jl}),\qquad\ell_{jl} := \sum_{i,k}\alpha_{ik}e_{ij}^{kl}. \tag*{(8)}

The row posterior αi\alpha_i only involves observation ii, whereas βj\beta_j depends on the whole column (mij,sij2)i∈[n](m_{ij},s_{ij}^{2})_{i\in[n]} of encoder outputs.

Training and Memoized Variational Inference

The row responsibilities αi\alpha_i are local: their update (6) only involves observation ii and is computed on the current mini-batch Bm\mathcal{B}_m. In contrast, β\beta is a global variational parameter: its update (6) depends on ϕ\phi through ℓjl\ell_{jl}, which aggregates all observations, and since each training step changes ϕ\phi, an exact update would require re-encoding the whole dataset after every step. We therefore adapt memoized online variational inference [10] to our case, which updates global parameters from cached per-batch sufficient statistics whose sums summarise the full dataset. For each mini-batch Bm\mathcal{B}_m, we store

Nk(m)=∑i∈Bmαik,Sjk(1,m)=∑i∈Bmαikmij,Sjk(2,m)=∑i∈Bmαik(mij2+sij2),(9)N_k^{(m)}=\sum_{i\in\mathcal{B}_m}\alpha_{ik},\qquad S_{jk}^{(1,m)}=\sum_{i\in\mathcal{B}_m}\alpha_{ik}m_{ij},\qquad S_{jk}^{(2,m)}=\sum_{i\in\mathcal{B}_m}\alpha_{ik}(m_{ij}^{2}+s_{ij}^{2}), \tag*{(9)}

which are indexed by latent coordinate jj and row cluster kk, rather than by block. With the totals NkN_k, Sjk(1)S_{jk}^{(1)}, Sjk(2)S_{jk}^{(2)} summed over mini-batches,

ℓjl=−12∑k(Nklog⁡(2πσkl2)+Sjk(2)−2μklSjk(1)+Nkμkl2σkl2),(10)\ell_{jl}=-\frac{1}{2}\sum_k\left(N_k\log(2\pi\sigma_{kl}^{2})+\frac{S_{jk}^{(2)}-2\mu_{kl}S_{jk}^{(1)}+N_k\mu_{kl}^{2}}{\sigma_{kl}^{2}}\right), \tag*{(10)}

After the gradient step on Bm\mathcal{B}_m, its statistics are recomputed with the updated encoder and swapped into the totals (old contribution subtracted, new one added), and β\beta is recomputed from (8); η\eta is updated at the end of each epoch.

Experiments

We conduct two experiments: clustering on MNIST [11], which evaluates the row partition against the digit classes (ARI, NMI, accuracy), and disentanglement on Shapes3D [12], which evaluates the block-structured representation against VAE-based baselines (MIG [4], SAP [13], DCI [14]).

Implementation settings. On MNIST, the encoder has three stride-2 convolutional layers (32, 32 and 64 channels) followed by two fully connected layers of 256 units, with ReLU activations and a mirrored decoder; we use p=20p=20, K=10K=10 and L=6L=6. On Shapes3D, all models use the convolutional encoder and decoder of [15], so that differences in scores come from the prior and the objective. Baselines use p=10p=10 as in [15], β=6\beta=6 and, for the HFS variants [16], γ=10\gamma=10, among the values considered in [17]. The LBM-VAE uses p=15p=15, since the column partition becomes trivial when pp is close to LL, with at most L=6L=6 blocks (number of ground-truth factors) and K=4K=4 row clusters (number of object shapes). Models are trained with Adam (learning rate 5×10−45\times10^{-4}, batch size 256) for 200 epochs, the first 30 as a standard VAE before the LBM prior is activated, on one NVIDIA H100 GPU on Jean Zay supercomputer (IDRIS). Scores are reported as mean ±\pm std over 10 seeds at the final epoch.

Clustering results. On the MNIST test set, the row partition reaches an ARI of 67.4±10.3%67.4\pm10.3\%, an NMI of 78.7±6.2%78.7\pm6.2\% and an accuracy of 75.3±8.3%75.3\pm8.3\%. These scores remain below those of dedicated deep clustering models such as VaDE [6], which is expected: co-clustering is a harder problem, since the blocks are unknown and each row cluster is described by a single Gaussian per block. Figure 1 shows, on a separate run with only 5 pretraining epochs, digit classes separating once the LBM prior is activated.

t-SNE projections of the MNIST test-set latent means at epochs 5, 30, 55 and 90

Figure 1. t-SNE projections of the MNIST test-set latent means at epochs 5, 30, 55 and 90, coloured by digit class (LBM prior activated at epoch 5).

Disentanglement results. Figure 2 shows the learned blocks. Clusters 1 and 6 illustrate the intended behaviour: the coordinates of Cluster 1 act only on the object orientation, and both coordinates of Cluster 6 control the wall hue. Cluster 5 encodes the object scale, and Cluster 4 gathers coordinates that are left unused by the decoder. Cluster 3 still mixes object hue, shape and floor hue.

Latent traversals of the LBM-VAE on Shapes3D

Figure 2. Latent traversals of the LBM-VAE on Shapes3D. Each column varies one of the 15 latent coordinates and each row one traversal value. Columns are grouped by learned cluster.

Table 1 reports the scores. Without additional regularization, LBM-VAE is competitive with the best baselines: it obtains the highest mean SAP and DCI-D and comparable completeness and informativeness, while its MIG score remains below those of the HFS variants. This is consistent with a block-wise encoding: DCI-D only requires each coordinate to inform a single factor, which holds when a factor spans a block, whereas MIG rewards concentrating each factor in one coordinate.

ModelMIGSAPDCI
β\beta-VAE0.268 ± 0.0790.077 ± 0.0370.603 ± 0.1290.522 ± 0.1180.926 ± 0.045
β\beta-TCVAE0.312 ± 0.171<u>0.092 ± 0.030</u>0.741 ± 0.0850.637 ± 0.0840.977 ± 0.015
β\beta-VAE + HFS0.616 ± 0.1810.085 ± 0.0550.765 ± 0.1790.788 ± 0.1090.920 ± 0.093
β\beta-TCVAE + HFS<u>0.534 ± 0.204</u><u>0.078 ± 0.029</u><u>0.848 ± 0.079</u>0.779 ± 0.0880.991 ± 0.007
LBM-VAE0.452 ± 0.1240.129 ± 0.0270.899 ± 0.107<u>0.780 ± 0.095</u><u>0.990 ± 0.018</u>

Table 1. Disentanglement scores on Shapes3D (best mean in bold, second best underlined).

Conclusion

We introduced LBM-VAE, a variational autoencoder whose Latent Block Model prior jointly co-clusters observations and latent dimensions, using coordinate-ascent variational updates and a memoized update of the column partition. The learned blocks group latent coordinates that capture common factors of variation, yielding competitive disentanglement without any additional regularization term. A limitation is that all latent blocks share a single row partition, whereas distinct factors may induce distinct clusters of the observations. Future work will therefore allow each block to learn its own observation partition, thereby extending LBM-VAE towards a multiview clustering, unlike MFCVAE [7] where the views are predefined by the user.

Acknowledgments

This work was carried out within IRT VEDECOM, with the support of its industrial partners Stellantis and Renault Group, and in collaboration with the DAVID laboratory, UVSQ, Université Paris-Saclay. It was supported by the French government under the "France 2030" operated by the French National Research Agency. This work was granted access to the HPC resources of IDRIS under the allocation 2026-AD011017901R1 made by GENCI, on the Jean Zay supercomputer.

References

  1. [1]Xin Wang, Hong Chen, Si’ao Tang, Zihao Wu, and Wenwu Zhu. Disentangled representation learning. IEEE Trans. Pattern Anal. Mach. Intell., 46(12):9677–9696, December 2024.DOI
  2. [2]Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Andriy Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2016.
  3. [3]Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2649–2658. PMLR, 10–15 Jul 2018.arxiv.org/abs/1802.05983
  4. [4]Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  5. [5]Jan Stühmer, Richard Turner, and Sebastian Nowozin. ISA-VAE: Independent subspace analysis with variational autoencoders, 2019.
  6. [6]Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: an unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI’17, page 1965–1972. AAAI Press, 2017.arxiv.org/abs/1611.05148
  7. [7]Fabian Falck, Haoting Zhang, Matthew Willetts, George Nicholson, Christopher Yau, and Chris C Holmes. Multi-facet clustering variational autoencoders. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8676–8690. Curran Associates, Inc., 2021.arxiv.org/abs/2106.05241
  8. [8]Gérard Govaert and Mohamed Nadif. Model-Based Co-Clustering, chapter 2, pages 55–77. John Wiley & Sons, Ltd, 2013.DOI
  9. [9]C.M. Bishop. Pattern recognition and machine learning, volume 4. Springer New York, 2006.
  10. [10]Michael C Hughes and Erik Sudderth. Memoized online variational inference for dirichlet process mixture models. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
  11. [11]Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010.
  12. [12]Chris Burgess and Hyunjik Kim. 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/, 2018.
  13. [13]Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. ArXiv, abs/1711.00848, 2017.arxiv.org/abs/1711.00848
  14. [14]Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
  15. [15]Francesco Locatello, Stefan Bauer, Mario Lucic̀, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Frederic Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, 2019. Best Paper Award.arxiv.org/abs/1811.12359
  16. [16]Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentanglement of correlated factors via hausdorff factorized support. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2210.07347
  17. [17]Théo Uscidda, Luca Eyring, Karsten Roth, Fabian J Theis, Zeynep Akata, and marco cuturi. Disentangled representation learning through geometry preservation with the gromov-monge gap. In ICML 2024 Workshop on Structured Probabilistic Inference & Generative Modeling, 2024.arxiv.org/abs/2407.07829

Paper details

vlnc:W_23wf4yv4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents