WorksComputer Science
Disentangled Embedding Optimization for Diffusion-Based Dataset Distillation
Abstract
Dataset distillation aims to compress large-scale datasets into compact synthetic subsets that enable efficient model training with comparable performance to the original data. Recent advances show that diffusion models are highly effective for this task due to their strong generative capability. Existing diffusion-based approaches typically perform class-wise clustering and use the obtained cluster prototypes as diffusion inputs to generate distilled samples. However, the uniformly distributed original samples in a category inevitably lead to entanglement near cluster boundaries, pulling the cluster prototypes closer and reducing the inter-cluster separation. Consequently, the diffusion model receives overly similar latent inputs, resulting in synthetic data with limited diversity. To address this issue, we propose Disentangled Embedding Optimization (DEO), a simple yet effective framework that replaces static clustering with trainable embeddings. These embeddings are optimized to match the Gaussian statistics (i.e., mean and covariance) of the original data distribution, while a diversity regularizer is designed to encourage separation among these embeddings to improve the intra-class variability. Extensive experiments demonstrate that our proposed DEO can generate more diversified and representative distilled data than existing cluster-based diffusion methods, achieving state-of-the-art performance and superior generalization across multiple datasets.
CCS Concepts
• Computing methodologies → Artificial intelligence.
Keywords
Dataset Distillation, Representative Learning, Semantic Diversity
Introduction
The success of deep learning in vision, language, and multimodal tasks largely stems from the availability of large-scale datasets [13, 17, 22, 25, 36]. However, the growing size of such datasets leads to substantial costs in storage, training time, and computation. To mitigate these issues, dataset distillation has emerged as a promising paradigm that synthesizes compact datasets capable of retaining the training efficacy of the original datasets [19, 34, 40]. Beyond efficiency, distilled datasets also benefit privacy-sensitive [8, 48] and resource-constrained scenarios [12, 37].
Early dataset distillation (DD) methods typically employ a bi-level optimization framework, where the outer loop updates model parameters and the inner loop refines synthetic data by aligning training trajectories [1, 9, 23] or gradients [18, 43, 44]. Despite their effectiveness, the interdependent optimization of models and distilled data incurs prohibitive computational costs, often restricting their application to low-resolution datasets like CIFAR-10/100. To mitigate this, subsequent research has decoupled the process by directly matching feature statistics [33, 39, 45, 46]. However, these methods still rely on pixel-space optimization, which makes scaling to high-resolution datasets inherently difficult [4, 24, 47].
To overcome the limitations of pixel-space optimization, several pioneering studies have incorporated generative priors into dataset distillation. For example, GLAD [2] improves distillation by projecting optimization losses onto the intermediate latent layers of a pre-trained GAN, thereby exploiting the structural and semantic information captured at different depths to regularize synthesized samples toward the natural image manifold. However, GAN-based frameworks still require backpropagation through complex generator architectures, resulting in substantial memory overhead. To alleviate this issue, more recent research has shifted toward diffusion-based distillation. Compared with GANs, diffusion models operate in a highly expressive and semantically linearized latent space induced by a pre-trained VAE. Within this paradigm, the VAE encoder first projects original images into a compact latent manifold, allowing distillation to be performed in a lower-dimensional yet semantically structured space. Specifically, D4M [31] facilitates distillation by using -means cluster prototypes as the latent foundation for diffusion. Building upon this paradigm, VLCP [49] integrates textual guidance by extracting LLaVA-generated descriptions for each cluster, leveraging these rich semantic prompts to further steer the synthesis of distilled samples.
However, a critical limitation is overlooked: in the latent space of a pretrained VAE encoder, features within each class tend to be uniformly distributed. Under such conditions, samples lying near the boundary of two clusters are inevitably and often arbitrarily assigned to one cluster or the other, which will reduce the distance between the corresponding cluster prototypes. As illustrated in Figure 1(a), we visualize the latent embeddings of a class, where samples belonging to two clusters are highlighted in orange and green, respectively. It is obvious that the entangled samples tend to pull their corresponding cluster prototypes closer in the feature space, thereby reducing inter-cluster separation. As shown in Figure 1(b), when these less separable cluster prototypes are used as latent inputs to the diffusion model, the generated distilled images appear visually similar and lack diversity. Consequently, the synthesized data fail to capture the full spectrum of intra-class semantic variations, limiting the performance of downstream tasks.
Figure 1. Illustration of the entanglement issue in distilled data generation. (a) t-SNE visualization shows that entangled sub-cluster prototypes in the original data reduce separability. (b) These less-separable latent inputs for the diffusion model result in indistinguishable synthesized images.
To address this issue, we propose Disentangled Embedding Optimization (DEO), a simple yet effective framework for diffusion-based dataset distillation. Instead of performing clustering analysis in the latent space, our method directly optimizes the latent embeddings, which serve as inputs of the diffusion model to generate distilled data. Specifically, we treat the latent embeddings as learnable parameters that are optimized to approximate the original data distribution. For each class, we estimate a Gaussian distribution in the latent space by computing the empirical mean and covariance matrix with the original data. Then, the mean and covariance of the trainable latent embeddings are aligned with those of the corresponding class distributions, ensuring semantic representativeness. To further enhance intra-class diversity of the distilled data, we introduce an explicit dispersion regularization that promotes mutual separation among the trainable latent embeddings. Finally, the optimized embeddings, together with their corresponding class textual prompts, are fed into the diffusion model to generate distilled data. Extensive experiments on CIFAR-10/100, ImageNet-1K, and several ImageNet subsets (ImageWoof, ImageNette, and ImageIDC) demonstrate that our DEO can generate diversified distilled data, which will consistently improve the performance of downstream tasks, all without introducing any external knowledge.
Our main contributions are summarized as follows:
We propose a simple yet effective framework, termed Disentangled Embedding Optimization (DEO) for diffusion-based dataset distillation, which makes the input latent space trainable rather than relying on conventional clustering.
We design a dispersion regularizer that explicitly encourages diversity among synthetic embeddings, effectively mitigating prototype collapse and promoting richer intra-class representations.
Extensive experiments on CIFAR-10/100, ImageNet-1K, and multiple ImageNet subsets show that DEO consistently outperforms prior diffusion-based baselines with strong cross-architecture generalization and lower computational cost.
Related Work
Matching-Based Methods
Early approaches to dataset distillation typically adopt a bi-level optimization framework, where the outer loop updates model parameters and the inner loop optimizes the distilled data to match either gradient signals or training trajectories. Specifically, gradient matching aligns the stepwise gradients computed on the original and distilled data [15, 18, 43, 44], whereas trajectory matching extends this idea by aligning the entire optimization paths [1, 5, 9, 23]. Despite demonstrating strong performance, these methods are computationally expensive due to the repeated alternation between model updates and data optimization.
To improve efficiency, recent methods leverage distribution matching, which aligns feature distributions without updating the model [45]. Early variants typically utilize randomly initialized networks to extract features. For example, CAFE [33] performs multi-layer prototype alignment; DataDAM [28] selects informative regions via attention mechanisms; and M3D [42] conducts feature alignment in a reproducing kernel Hilbert space. Building on this line of work, IDM [46] and DANCE [41] enhance representational capacity by employing multiple pretrained extractors, while IID [7] and DSDM [20] introduce feature moment matching to promote semantic diversity. However, these methods require both the dataset and the model to reside in memory, which limits scalability to large-scale settings.
To address the scalability challenge, decoupled distillation strategies have emerged [30, 32, 38]. A representative example is SRe2L [39], which first trains a reference model to absorb the knowledge from the full dataset, and then optimizes the distilled data by matching per-batch BatchNorm statistics to the model’s global statistics. Additionally, soft labels generated by the reference model provide supervisory signals for distillation. While this strategy substantially reduces computational cost, it may compromise diversity in synthetic samples, limiting generalization to unseen architectures. To mitigate this issue, DWA [10] introduces training perturbations and variance-aware weighting to encourage diverse sample generation. Complementarily, DEDA [21] incorporates a decorrelation regularization term that reduces redundancy among distilled features, thereby further enhancing semantic diversity. Although the decoupled distillation methods are more computationally efficient, they still follow the pixel-level optimization paradigm, where memory usage and computational cost grow greatly with image resolution, ultimately limiting their scalability to high-resolution datasets.
Generative-Based Methods
Recent research has shifted from pixel-space optimization to leveraging generative priors. As a pioneering work, GLAD [2] introduces a deep generative prior from GANs to regularize the distillation process, significantly enhancing cross-architecture generalization. However, GAN-based methods often struggle with high-resolution scaling or necessitate complex, memory-intensive optimization. Building on the success of generative modeling [26], diffusion models have emerged as robust backbones for dataset distillation due to their superior capacity for high-fidelity image synthesis. For instance, Gu et al. [11] enhance the representativeness of synthetic data by fine-tuning diffusion models to align predicted embeddings with those of original images. Although effective, this multi-stage fine-tuning pipeline is computationally expensive, especially for large-scale datasets. To mitigate the overhead, subsequent studies exploit a frozen VAE and perform distillation directly in its latent space. D⁴M [31] clusters latent codes using -means and conditions the generation process on cluster prototypes combined with label text prompts. MGD [3] further discovers intra-class patterns through clustering to guide the denoising trajectory of the diffusion model. Inspired by this idea, VLCP [49] enhances textual conditioning by leveraging LLaVA to generate per-image descriptions and selecting the most frequent sentence within each cluster to provide fine-grained textual descriptions for each cluster prototype. In these approaches, clustering is a crucial step for generating distilled data. However, clustering-based distillation often falls into a homogeneity trap. Because cluster prototypes frequently overlap in dense data regions, the resulting synthetic samples become repetitive, sacrificing dataset diversity for mere representative proximity.
Method
Preliminaries
Dataset distillation aims to construct a compact synthetic dataset , on which models trained can achieve comparable generalization performance to those trained on the full dataset , where . Typically, contains only IPC samples (IPC: images per class), satisfying . The goal is to train on such that it achieves comparable generalization to trained on .
Latent Diffusion Model for Dataset Distillation. Recently, latent diffusion models (LDM) have emerged as a promising paradigm for dataset distillation, due to their strong cross-architecture generalization and efficient sample generation [31, 49]. A typical LDM pipeline consists of: (1) a VAE image encoder for mapping input images into the latent space, (2) a conditioning text encoder that encodes class labels or descriptions, (3) a denoising U-Net trained to reverse noise corruption, and (4) a VAE decoder that reconstructs images from the denoised latents.
Latent Diffusion Objective. The latent diffusion model is trained to predict additive noise injected into latent embeddings:
where is a noised latent obtained from an original image through a VAE encoder and a forward diffusion process, is the ground-truth noise, and is the diffusion timestep. The term denotes the embedding of the class label produced by a text encoder , which provides semantic conditioning for the denoising network . During inference, a scheduler progressively denoises the latent from to , which is then decoded into an image using the VAE decoder. In the context of diffusion-based dataset distillation, the collection of such generated images constitutes the distilled dataset .
Disentangled Embedding Optimization
Recent diffusion-based dataset distillation methods typically perform clustering on the latent embeddings extracted by a VAE image encoder for each class and use the cluster prototypes as the inputs to the latent diffusion model to generate distilled images. However, since the samples of a class are often uniformly distributed in the latent space, the resulting cluster prototypes tend to lie close to one another. This will lead to the synthetic samples with limited intra-class diversity.
To mitigate this issue, instead of following the clustering-based pipeline, we propose to explicitly optimize the embeddings of distilled data . This design enables the generated samples to preserve the diversity of the original data distribution. As shown in Figure 2(a), we initialize the learnable latent embeddings independently and align them with the Gaussian distribution of the original data in the latent space of the pre-trained VAE encoder by minimizing a mean alignment loss and a covariance alignment loss . In addition, to promote sample-level diversity within each class, we introduce a diversity loss that encourages the optimized embeddings to be mutually dispersed in the latent space.
Figure 2. Overview of our Disentangled Embedding Optimization (DEO) for diffusion-based distillation. (a) Learnable latent embeddings are optimized to match the mean and covariance of the original data, with a diversity loss to encourage separation. (b) The optimized and text prompts are fed into the diffusion model to generate distilled samples .
Specifically, for each class , let denote the latent embeddings extracted from the original samples in class , where is the number of original samples. We define the mean alignment loss as follows:
where and are the mean latent embeddings of the synthetic and original samples in class , respectively. Similarly, we define the covariance alignment loss as follows:
To further encourage sample-level diversity within each class, we introduce a separation regularization term that penalizes latent embeddings with high similarity:
where denotes the -normalized version of . This term encourages the learnable embeddings to be well separated in the latent space, thereby promoting intra-class diversity in the distilled data.
Overall Objective and Generation Procedure
The training objective combines distribution alignment with diversity regularization:
| Algorithm 1: DEO |
| Input: : Original dataset; : Number of classes; IPC: Images per class; : Number of optimization steps; : Learning rate. |
| 1 (a): Disentangled Embedding Optimization |
| 2 for each class to do |
| 3 Initialize learnable latent embeddings |
| 4 Compute the class-wise feature mean and covariance from . |
| 5 for iteration to do |
| 6 Compute the mean and covariance of the latent embeddings . |
| 7 Compute the mean loss via Eq. (2). |
| 8 Compute the covariance loss via Eq. (3). |
| 9 Compute the diversity regularization loss via Eq. (4). |
| 10 Compute the overall objective via Eq. (5). |
| 11 Update latent embeddings: . |
| 12 (b): Diffusion-Based Data Generation |
| 13 for each class to do |
| 14 Combine the optimized latent embeddings with the class-specific textual prompt (i.e., “An image of [class]”). |
| 15 Generate synthetic images according to Eq. (6). |
| 16 Append generated images to the distilled dataset: . |
| Output: Distilled synthetic dataset . |
Table 1.
where and are tunable hyperparameters. After the latent embeddings are optimized, we feed them into the diffusion model following prior work [31, 49]. As illustrated in Figure 2(b), the per-class distilled latent set is paired with a class-specific textual prompt (e.g., “An image of [class”]). The prompt embedding conditions the U-Net, and each distilled image is synthesized as:
where is the noise-perturbed version of at diffusion timestep , denotes the timestep-dependent U-Net backbone, is the image decoder, and denotes channel-wise concatenation. This formulation enables the optimized latent codes and textual semantics to jointly guide the denoising process to generate the distilled dataset . The complete training and generation pipeline is summarized in Algorithm 1.
Discussion
The advantage of DEO over clustering-based latent distillation methods can be understood from the perspective of prototype bias caused by boundary ambiguity. In the latent space of a pretrained VAE, samples within the same class are often uniformly distributed without clear sub-cluster boundaries. As a result, samples near the overlapping region may be arbitrarily assigned to different clusters, which will pull the estimated cluster prototypes near each other.
Consider two ideal semantic modes with centers and . Due to boundary ambiguity, the estimated prototypes can be written as
where quantify cross-mode contamination. Their distance then becomes
This simple analysis shows that boundary ambiguity naturally shrinks the separation between cluster prototypes, causing the diffusion model to receive similar latent inputs. In contrast, DEO avoids hard cluster assignment and directly optimizes learnable embeddings under distribution matching and dispersion regularization, thereby preserving richer intra-class semantic variability.
Experiment
Experimental Setup
Datasets. We evaluate our proposed DEO method on a diverse suite of image benchmarks. For low-resolution settings, we utilize CIFAR-10 and CIFAR-100 [16], two widely used datasets containing 60,000 images at 32 × 32 resolution, with 10 and 100 classes respectively. For high-resolution evaluation, we employ ImageNet-1K [6], a large-scale dataset comprising 1.28 million images across 1,000 categories at 224 × 224 resolution, which offers a comprehensive test of generalization under diverse visual concepts. Additionally, we assess performance on several widely adopted subsets of ImageNet-1K, specifically ImageWoof, ImageNette [14], and ImageIDC [49]. ImageWoof focuses on fine-grained recognition among ten visually similar dog breeds, while ImageNette and ImageIDC serve as more general benchmarks with ten coarse-grained categories each.
Implementation Details. We follow the experimental protocols established in prior work [31, 49]. Unless otherwise specified, we fine-tune Stable Diffusion v1.5 on each dataset using paired image-label supervision. The trainable latent embeddings are optimized using the Adam optimizer with a learning rate of for 10,000 steps. To ensure both distribution alignment and sample diversity, we incorporate a covariance matching loss weighted by , alongside a diversity-promoting separation regularizer applied to the denoised latents with a coefficient of . All experiments are conducted on NVIDIA RTX A6000 or NVIDIA A100 (80GB) GPUs.
Evaluation Protocol. For a fair comparison, we follow the evaluation settings used in VLCP [49]. Two evaluation strategies are employed: (1) Direct Training: a classifier is trained from scratch using the generated images and their labels, with performance evaluated on the standard test split. This setting is used for ImageWoof, ImageNette, and ImageIDC. (2) Soft-Label Supervision: soft labels are obtained by applying the teacher model pre-trained by the original data to the distilled images, which are then used to train a new classifier. This setting is adopted for ImageNet-1K and CIFAR-10/100. All reported Top-1 accuracies are averaged over three random seeds.
Compared Methods. We compare DEO against a comprehensive set of state-of-the-art baselines: (1) Core-set selection methods, including K-Center [29] and Herding [35]; (2) Matching-based distillation methods, such as DM [45], IDC-1 [15], SRe2L [39], and RDED [32]; and (3) Diffusion-based distillation methods, including DiT [27], Minimax [11], D⁴M [31], and VLCP [49].
Main Results
ImageWoof, ImageNette, and ImageIDC. Table 1 reports the results on three benchmarks under a unified ResNetAP-10 evaluation protocol. DEO consistently achieves the best performance across all datasets and all IPC settings. On ImageWoof, DEO reaches 40.9%, 46.7%, and 59.9% at IPC=10, 20, and 50, respectively, outperforming the strongest baseline VLCP by 1.4%, 2.2%, and 2.6%. Similarly, on ImageNette, DEO improves over VLCP by 1.4%, 1.2%, and 1.0%, while on ImageIDC, the gains are 1.6%, 1.3%, and 0.7%. These results show that DEO produces more effective distilled samples across datasets with different visual characteristics and compression ratios.
ImageNet-1K. Table 2 further demonstrates the scalability of DEO on the challenging ImageNet-1K benchmark. DEO achieves 49.2% at IPC=10 and 61.4% at IPC=50, surpassing the strongest baseline VLCP by 2.5% and 0.9%, respectively. The consistent advantage on ImageNet-1K indicates that our embedding optimization strategy remains effective even when the dataset scale and semantic diversity increase substantially.
| IPC | SRe2L | RDED | DiT | Minimax | VLCP | DEO (Ours) |
| 10 | 21.3±0.6 | 42.0±0.1 | 39.6±0.4 | 44.3±0.5 | 46.7±0.4 | 49.2±0.8 |
| 50 | 46.8±0.2 | 56.5±0.1 | 52.9±0.6 | 58.6±0.3 | 60.5±0.2 | 61.4±0.3 |
Table 2. Performance comparison on ImageNet-1K with IPC=10/50.
CIFAR-10/100. We further evaluate DEO on low-resolution benchmarks in Table 3. On CIFAR-10, DEO achieves 41.8% and 64.1% at IPC=10 and 50, improving over VLCP by 2.8% and 0.9%, respectively. On CIFAR-100, DEO reaches 52.4% and 66.8%, exceeding VLCP by 1.8% and 0.7%. Compared with earlier baselines, the gains are even more pronounced; for example, on CIFAR-100 with IPC=10, DEO outperforms SRe2L and RDED by 25.4% and 9.8%, respectively. Overall, these results demonstrate that DEO consistently delivers superior distilled data quality on both large-scale high-resolution benchmarks and standard low-resolution datasets.
| Dataset | IPC | SRe2L | RDED | VLCP | DEO (Ours) |
| CIFAR-10 | 10 | 29.3±0.5 | 37.1±0.3 | 39.0±0.7 | 41.8±0.6 |
| CIFAR-10 | 50 | 45.0±0.7 | 62.1±0.1 | 63.2±0.3 | 64.1±0.7 |
| CIFAR-100 | 10 | 27.0±0.4 | 42.6±0.2 | 50.6±0.7 | 52.4±0.3 |
| CIFAR-100 | 50 | 50.2±0.4 | 62.6±0.1 | 66.1±0.3 | 66.8±0.5 |
Table 3. Performance comparison on CIFAR-10 and CIFAR-100 with IPC=10/50.
Cross-architecture Evaluation
Table 4 reports the results on ImageWoof using two target architectures, ConvNeXt and ResNet-18. DEO consistently achieves the best performance across all IPC settings on both backbones, demonstrating strong cross-architecture generalization of the distilled data.
| Backbone | IPC | Random | K-Center | Herding | DiT | DM | IDC-1 | Minimax | D⁴M | VLCP | DEO (Ours) |
| ConvNet | 10 | 24.3±1.1 | 19.4±0.9 | 26.7±0.5 | 34.2±1.1 | 26.9±1.2 | 33.3±1.1 | 33.3±1.7 | 29.4±0.9 | 34.8±2.4 | 36.2±0.8 |
| ConvNet | 20 | 29.1±0.7 | 21.5±0.8 | 29.5±0.3 | 36.1±0.8 | 29.9±1.0 | 35.5±0.8 | 37.3±0.1 | 34.0±2.3 | 37.9±1.9 | 39.4±0.3 |
| ConvNet | 50 | 41.3±0.6 | 36.5±1.0 | 40.3±0.7 | 46.5±0.8 | 44.4±1.0 | 43.9±1.2 | 50.9±0.8 | 47.4±0.9 | 54.5±0.6 | 55.6±0.3 |
| ResNet-18 | 10 | 27.7±0.9 | 21.1±0.4 | 30.2±1.2 | 34.7±0.4 | 33.4±0.7 | 37.3±0.2 | 35.7±1.6 | 32.3±1.2 | 39.9±2.6 | 41.5±1.1 |
| ResNet-18 | 20 | 29.7±0.5 | 23.6±0.3 | 32.2±0.6 | 40.5±0.5 | 29.8±1.7 | 38.6±0.2 | 41.8±1.9 | 38.4±1.1 | 44.5±2.0 | 45.0±0.8 |
| ResNet-18 | 50 | 47.9±1.8 | 39.6±1.0 | 48.3±1.2 | 50.1±0.5 | 46.2±0.6 | 48.3±0.8 | 53.7±0.6 | 53.7±2.2 | 58.9±1.5 | 60.1±0.9 |
Table 4. Performance comparison of dataset distillation methods on ImageWoof under varying IPC values. Evaluations are conducted using ConvNet and ResNet-18. The best results are shown in bold.
| Method | ImageWoof IPC=10 | ImageWoof IPC=20 | ImageWoof IPC=50 | ImageNette IPC=10 | ImageNette IPC=20 | ImageNette IPC=50 | ImageIDC IPC=10 | ImageIDC IPC=20 | ImageIDC IPC=50 |
| Random | 29.4±0.8 | 32.7±0.4 | 47.2±1.3 | 54.2±1.6 | 63.5±0.5 | 76.1±1.1 | 48.1±0.8 | 52.5±0.9 | 68.1±0.7 |
| K-Center | 22.1±0.1 | 25.1±0.7 | 40.6±0.4 | – | – | – | – | – | – |
| Herding | 32.0±0.3 | 34.9±0.1 | 49.1±0.7 | – | – | – | – | – | – |
| DiT | 34.7±0.5 | 41.1±0.8 | 49.3±0.2 | 59.1±0.7 | 64.8±1.2 | 73.3±0.9 | 54.1±0.4 | 58.9±0.2 | 64.3±0.6 |
| DM | 30.3±1.2 | 35.2±0.6 | 47.1±1.1 | 60.8±0.6 | 66.5±1.1 | 76.2±0.4 | 52.8±0.5 | 58.5±0.4 | 69.1±0.8 |
| IDC-1 | 39.1±0.5 | 43.4±0.3 | 48.3±1.0 | – | – | – | – | – | – |
| Minimax | 36.2±3.2 | 43.3±2.7 | 53.9±0.7 | 57.7±1.2 | 64.7±0.8 | 73.9±0.3 | 51.9±1.4 | 59.1±3.7 | 69.4±1.4 |
| D⁴M | 33.2±2.1 | 40.1±1.6 | 51.7±3.2 | 60.9±1.7 | 66.3±1.3 | 77.7±1.1 | 50.3±1.0 | 55.8±0.2 | 69.1±2.4 |
| VLCP | 39.5±1.5 | 44.5±2.2 | 57.3±0.5 | 64.8±3.6 | 71.4±0.5 | 81.2±0.8 | 57.0±1.4 | 63.3±1.2 | 71.9±0.4 |
| DEO (Ours) | 40.9±0.5 | 46.7±0.4 | 59.9±0.6 | 66.2±1.2 | 72.6±0.4 | 82.2±0.4 | 58.6±0.7 | 64.6±0.6 | 72.6±0.3 |
Table 1 (PDF p. 6). Performance comparison on ImageWoof, ImageNette, and ImageIDC with ResNetAP-10 under different IPC settings. The best and second-best results are highlighted in bold and underline, respectively; “–” denotes unavailable results.
Specifically, on ConvNet, DEO improves over the strongest baseline VLCP by 1.4%, 1.5%, and 1.1% at IPC=10, 20, and 50, respectively. On ResNet-18, the corresponding gains are 1.6%, 0.5%, and 1.2%. Moreover, compared with the most relevant diffusion-based baseline D⁴M, DEO achieves substantially larger gains, ranging from 5.4% to 9.2% across different settings. These results indicate that DEO learns more transferable and discriminative latent representations, leading to superior performance across different model families rather than on a single evaluation architecture only.
Analysis
Ablation Study. To evaluate the effectiveness of each component in our DEO, we conduct ablation studies on ImageWoof under three IPC settings: IPC = 10, 20, and 50. We analyze the impact of three core losses: mean alignment (), covariance alignment (), and diversity regularization (). As summarized in Table 5, we can draw four key conclusions. First, using only already outperforms D⁴M by 1.1% and 0.5%, demonstrating that aligning the class-wise means of latent embeddings effectively preserves the core semantics of the original data. Second, adding further captures intra-class variations around the class centers and brings the performance up to par with VLCP. This indicates that matching both the mean and covariance statistics is sufficient to reconstruct rich semantic information without relying on large pretrained models or long-horizon reasoning. Third, incorporating alongside yields an additional gain of 1.9%–2.5%, suggesting that enforcing separation among embeddings promotes greater semantic diversity and expressiveness. Finally, the combination of all three losses consistently achieves the best performance across all IPC settings, validating the effectiveness of jointly optimizing for semantic alignment and intra-class diversity in the latent embedding space.
| Method | IPC=10 | IPC=20 | IPC=50 | |||
| D⁴M | - | - | - | 33.2 | 40.1 | 51.7 |
| Ours | ✓ | - | - | 34.3 | 39.6 | 52.2 |
| Ours | ✓ | ✓ | - | 39.8 | 44.3 | 57.8 |
| Ours | ✓ | - | ✓ | 36.2 | 42.1 | 54.1 |
| Ours | ✓ | ✓ | ✓ | 40.9 | 46.7 | 59.9 |
Table 5. Ablation study on ImageWoof.
Effect of High IPC. We further analyze the effect of high IPC settings in Figure 3 by comparing D⁴M and DEO on ImageWoof. As IPC increases, both methods achieve better accuracy, showing
Figure 3. Performance of D⁴M and DEO under different high IPC settings on ImageWoof.
that a larger distilled budget generally improves downstream performance. DEO consistently surpasses D⁴M across all IPC settings, indicating its stronger ability to preserve informative class semantics. At the same time, the marginal gain of DEO gradually decreases as IPC grows, suggesting a clear diminishing-return effect. This implies that once sufficient semantic coverage is achieved, further increasing IPC brings only limited additional improvement.
Effect of . In our DEO framework, the hyperparameter controls the weight of the covariance alignment loss , which is designed to match the second-order statistics of the learnable embeddings to those of the original data. Covariance reflects both intra-class dispersion and inter-dimensional correlations, thereby capturing semantic variations around the class center. As shown in the left panel of Figure 4, on ImageWoof with IPC , increasing from 0 to 0.5 consistently improves classification performance, demonstrating the effectiveness of incorporating higher-order statistical alignment. However, when exceeds 0.5, the performance drops noticeably. We attribute this decline to the trade-off between mean alignment and covariance alignment: an overly large may overemphasize noisy covariance estimates, especially in low-IPC settings, and drive the embeddings to fit outlier samples. Unless otherwise specified, we set in all experiments.

Figure 4. Effect of and on ImageWoof with IPC . Red, blue, and green lines represent IPC = 10, 20, and 50, respectively, while dashed lines denote the corresponding results from D⁴M.
Effect of . The hyperparameter controls the strength of the diversity regularization, which encourages the learned embeddings to be more dispersed in the latent space. Such dispersion helps the diffusion model generate synthetic samples that are both more diverse and more representative. As shown in the right panel of Figure 4, we conduct a sensitivity analysis on ImageWoof with IPC . The results indicate that performance reaches its peak at . When becomes excessively large, however, the embeddings become over-dispersed, weakening intra-class compactness and leading to a gradual performance drop.
Diversity Analysis. The average cosine distance between samples and their class prototypes serves as a principled metric for intra-class diversity, which is fundamentally linked to the generalization capability of downstream models. To validate this, we report the average cosine distance for the first ten classes of CIFAR-100
and ImageNet-1K in Figure 5. As illustrated, our DEO consistently outperforms the D⁴M baseline across both datasets. Specifically, on CIFAR-100, DEO achieves an average cosine distance of 0.143, marking a 9.2% relative improvement over D⁴M (0.131). A similar trend is observed on ImageNet-1K, where DEO elevates the metric from 0.121 to 0.132 (a 9.1% gain). These quantitative results demonstrate that DEO more effectively captures intrinsic semantic
Figure 5. Comparison of intra-class diversity. Average cosine distance between distilled samples and class prototypes on CIFAR-100 and ImageNet-1K.
variability, thereby enriching the diversity of the distilled dataset and ultimately boosting downstream performance.
Computational Efficiency. We benchmark DEO against existing methods on a single NVIDIA A100 (80GB) GPU. As shown in Table 6, DEO substantially reduces the computational overhead of the distillation pipeline. On CIFAR-100, it decreases the per-class processing time from 9.5 to 7.4 minutes. The advantage is more pronounced on the large-scale ImageNet-1K dataset, where VLCP requires 25.2 minutes per class, mainly due to the heavy cost of LLaVA inference (21.7 minutes) and latent-space clustering (3.5 minutes). By removing these expensive stages and relying instead on efficient embedding-statistics computation with lightweight optimization, DEO reduces the total time to 13.4 minutes per class, yielding a 47% reduction in computational cost while also improving Top-1 accuracy. These results demonstrate that DEO achieves better performance with significantly lower resource demands.
| Stage (min) | CIFAR-100 VLCP | CIFAR-100 DEO (Ours) | ImageNet-1K VLCP | ImageNet-1K DEO (Ours) |
| Clustering | 1.3 | – | 3.5 | – |
| LLaVA Inf. | 8.2 | – | 21.7 | – |
| Emb. Stats | – | 3.6 | – | 9.6 |
| Optim. | – | 3.8 | – | 3.8 |
| Total Time | 9.5 | 7.4 | 25.2 | 13.4 |
Table 6. Per-class computational cost (minutes) at IPC=10 on CIFAR-100 and ImageNet-1K.
Visualization
Latent Space Visualization. To qualitatively demonstrate the effectiveness of our DEO in capturing the intrinsic semantic structure of original data, we visualize the distilled embeddings of the same class on ImageWoof using t-SNE under IPC = 10 and 50, comparing D⁴M with DEO as shown in Figure 6. In these plots, blue dots represent original samples, while red stars indicate distilled ones. For D⁴M, the clustering-based approach results in distilled points that are relatively close to each other, with two points nearly overlapping under IPC=10, and even more clustered under IPC=50, suggesting limited diversity. In contrast, DEO produces embeddings that better align with the original data distribution and maintain a higher degree of dispersion in the latent space. This suggests that optimizing learnable embeddings enables the diffusion model to capture and reproduce meaningful semantic structures from the dataset, rather than memorizing superficial patterns.

Figure 6. t-SNE visualization of latent distributions for original and distilled images. Results are shown for ImageWoof under IPC 10 and 50. Blue dots indicate original data; red stars denote distilled data from D⁴M and DEO.
Distilled Images. We further compare the visual quality and semantic fidelity of the distilled data generated by D⁴M and our DEO on CIFAR-100 and ImageWoof in Figure 7. Two major observations can be drawn: (1) On the low-resolution CIFAR-100, our DEO produces distilled samples that more faithfully retain the semantic characteristics of the original data. For example, in the bear and baby classes, DEO generates samples with diverse poses and expressions, whereas those from D⁴M tend to look very similar. (2) On the high-resolution ImageWoof, our DEO not only maintains clear class discriminability but also captures richer semantic diversity. For instance, in the Shih-Tzu class, D⁴M frequently generates seated samples with similar hair color and style, whereas DEO produces samples in both seated and lying poses with diverse hair colors (e.g., brown and black), better capturing the true intra-class variability. These results show that DEO more effectively captures semantic structure, yielding distilled samples with higher realism and diversity.
Figure 7. Visualization of distilled data. We compare original samples with distilled data generated by D⁴M and DEO on (a) CIFAR-100 (baby, bear) and (b) ImageWoof (Shih-Tzu, Beagle).
Conclusion
In this paper, we introduce Disentangled Embedding Optimization (DEO), a novel clustering-free framework for diffusion-based dataset distillation. Instead of relying on latent clustering, DEO directly optimizes the learnable embeddings, which are utilized to generate distilled data, by matching the Gaussian statistics (class-level means and covariances) of the original data distribution. To further enhance the diversity among embeddings, a separation regularization term is incorporated to encourage inter-embedding decorrelation. Extensive experiments across multiple benchmarks demonstrate that the proposed DEO consistently outperforms prior methods. It generates more diversified and informative distilled data, demonstrates strong generalization across different model architectures, and maintains lower computational overhead.
Acknowledgments
This work was supported by Strategic Priority Research Program of the Chinese Academy of Sciences under grant No.XDB0690302.
References
References
- [1]George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. 2022. Dataset Distillation by Matching Training Trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10718–10727.
- [2]George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. 2023. Generalizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3739–3748.arxiv.org/abs/2305.01649
- [3]Jeffrey A. Chan, Santiago, Praveen Tirumatur, Gaurav Kumar Nayak, Gaowen Liu, and Mubarak Shah. 2025. MGD³: Mode-Guided Dataset Distillation using Diffusion Models. In Proceedings of the 42nd International Conference on Machine Learning (ICML).arxiv.org/abs/2505.18963
- [4]Mingyang Chen, Jiawei Du, Bo Huang, Yi Wang, Xiaobo Zhang, and Wei Wang. 2025. Influence-Guided Diffusion for Dataset Distillation. In Proceedings of the International Conference on Learning Representations (ICLR).
- [5]Justin Cui, Ruchuan Wang, Si Si, and Cho-Jui Hsieh. 2023. Scaling up dataset distillation to imagenet-1k with constant memory. In International Conference on Machine Learning. PMLR, 6565–6590.arxiv.org/abs/2211.10586
- [6]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. IEEE, 248–255.
- [7]Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, and Yang Gao. 2024. Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 17057–17066.
- [8]Tian Dong, Bo Zhao, and Lingjuan Lyu. 2022. Privacy for Free: How does Dataset Condensation Help Privacy? In Proceedings of the International Conference on Machine Learning (ICML). 5378–5396.arxiv.org/abs/2206.00240
- [9]Jiawei Du, Yidi Jiang, Vincent YF Tan, Joey Tianyi Zhou, and Haizhou Li. 2023. Minimizing the accumulated trajectory error to improve dataset distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3749–3758.arxiv.org/abs/2211.11004
- [10]Jiawei Du, Xin Zhang, Juncheng Hu, Wenxin Huang, and Joey Tianyi Zhou. 2024. Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment. In Adv. Neural Inf. Process. Syst. (NeurIPS).arxiv.org/abs/2409.17612
- [11]Jianyong Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. 2024. Efficient dataset distillation via minimax diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15793–15803.arxiv.org/abs/2311.15529
- [12]Jianyong Gu, Kai Wang, Wei Jiang, and Yang You. 2024. Summarizing Stream Data for Memory-Restricted Online Continual Learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).arxiv.org/abs/2305.16645
- [14]Jeremy Howard. 2019. Imagenette: A smaller subset of 10 easily classified classes from ImageNet (and a little more French). https://github.com/fastai/imagenette. GitHub repository.
- [15]Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joon-Hyun Jeong, Jung-Woo Ha, and Hyun Oh Song. 2022. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning. PMLR, 11102–11118.arxiv.org/abs/2205.14959
- [16]Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).
- [18]Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. 2022. Dataset condensation with contrastive signals. In International Conference on Machine Learning. PMLR, 12352–12364.
- [19]Shiye Lei and Dacheng Tao. 2023. A Comprehensive Survey to Dataset Distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 1 (2023), 17–32.DOI
- [20]Hongcheng Li, Yucan Zhou, Xiaoyan Gu, Bo Li, and Weiping Wang. 2024. Diversified Semantic Distribution Matching for Dataset Distillation. In Proceedings of the ACM International Conference on Multimedia (MM).
- [21]Hongcheng Li, Yucan Zhou, Xiaoyan Gu, Bo Li, and Weiping Wang. 2025. Diversity-Enhanced Distribution Alignment for Dataset Distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).DOI
- [23]Dai Liu, Jindong Gu, Hu Cao, Carsten Trinitis, and Martin Schulz. 2024. Dataset Distillation by Automatic Training Trajectories. In Proceedings of the European Conference on Computer Vision (ECCV).arxiv.org/abs/2407.14245
- [24]Ping Liu and Jiawei Du. 2025. The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions. arXiv preprint arXiv:2502.05673 (2025).arxiv.org/abs/2502.05673
- [26]Yufan Liu, Wanqian Zhang, Huashan Chen, Lin Wang, Xiaojun Jia, Zheng Lin, and Weiping Wang. 2025. AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Prompts. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 17557–17566.
- [27]William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision. 4195–4205.arxiv.org/abs/2212.09748
- [28]Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z Liu, Yuri A Lawryshyn, and Konstantinos N Plataniotis. 2023. Datadream: Efficient dataset distillation with attention matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 17097–17107.arxiv.org/abs/2310.00093
- [29]Ozan Sener and Silvio Savarese. 2018. Active Learning for Convolutional Neural Networks: A Core-Set Approach. In International Conference on Learning Representations.
- [30]Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. 2024. Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 16709–16718.arxiv.org/abs/2311.17950
- [31]Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. 2024. D^4M: Dataset Distillation via Disentangled Diffusion Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5809–5818.arxiv.org/abs/2407.15138
- [32]Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. 2024. On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).arxiv.org/abs/2312.03526
- [33]Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. 2022. CAFE: Learning to Condense Dataset by Aligning Features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12196–12205.arxiv.org/abs/2203.01531
- [34]Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. 2018. Dataset distillation. arXiv preprint arXiv:1811.10959 (2018).arxiv.org/abs/1811.10959
- [35]Max Welling. 2009. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning (Montreal, Quebec, Canada) (ICML ’09). Association for Computing Machinery, New York, NY, USA, 1121–1128. doi:10.1145/1553374.1553517DOI
- [37]Enneng Yang, Li Shen, Zhenyi Wang, Tongliang Liu, and Guibing Guo. 2023. An Efficient Dataset Condensation Plugin and Its Application to Continual Learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS).DOI
- [38]Zeyuan Yin and Zhiqiang Shen. 2024. Dataset Distillation via Curriculum Data Synthesis in Large Data Era. Transactions on Machine Learning Research (2024).arxiv.org/abs/2311.18838
- [39]Zeyuan Yin, Eric Xing, and Zhiqiang Shen. 2023. Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. In Thirty-seventh Conference on Neural Information Processing Systems.arxiv.org/abs/2306.13092
- [40]Ruonan Yu, Songhua Liu, and Xinchao Wang. 2023. A Comprehensive Survey to Dataset Distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 1 (2023), 150–170.
- [41]Hansong Zhang, Shikun Li, Fanzhao Lin, Weiping Wang, Zhenxing Qian, and Shiming Ge. 2024. DANCE: Dual-View Distribution Alignment for Dataset Condensation. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI).arxiv.org/abs/2406.01063
- [42]Hansong Zhang, Shikun Li, Pengju Wang, Dan Zeng, and Shiming Ge. 2024. M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy. In The 38th Annual AAAI Conference on Artificial Intelligence (AAAI).arxiv.org/abs/2312.15927
- [43]Bo Zhao and Hakan Bilen. 2021. Dataset condensation with Differentiable Siamese Augmentation. In Proceedings of the International Conference on Machine Learning (ICML). 12674–12685.arxiv.org/abs/2102.08259
- [44]Bo Zhao and Hakan Bilen. 2021. Dataset Condensation with Gradient Matching. In Proceedings of the International Conference on Learning Representations (ICLR).
- [45]Bo Zhao and Hakan Bilen. 2023. Dataset Condensation with Distribution Matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 6514–6523.arxiv.org/abs/2110.04181
- [46]Ganlong Zhao, Guanbin Li, Lipeng Qin, and Yizhou Yu. 2023. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7856–7865.arxiv.org/abs/2307.09742
- [47]Lin Zhao, Yushu Wu, Xinru Jiang, Jianyong Gu, Yanzhi Wang, Xiaolin Xu, Pu Zhao, and Xue Lin. 2025. Taming Diffusion for Dataset Distillation with High Representativeness. In Proceedings of the International Conference on Machine Learning (ICML).arxiv.org/abs/2505.18399
- [48]Runkai Zheng, Vishnu Asutosh Dasu, Yining Olivier Wang, Haohan Wang, and Fernando De la Torre. 2025. Improving Noise Efficiency in Privacy-preserving Dataset Distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).arxiv.org/abs/2508.01749
- [49]Yawen Zou, Guang Li, Duo Su, Zi Wang, Jun Yu, and Chao Zhang. 2025. Dataset Distillation via Vision-Language Category Prototype. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).arxiv.org/abs/2506.23580




