WorksComputer Science
Shaping Useful Noise: Energy Distributions Predict Visual Pretraining Quality
Abstract
Synthetic data can scale visual pretraining, but current selection criteria only partially explain which datasets train strong representations. We propose using the energy profile of a dataset–the distribution of log-probability levels it occupies under a normalised real-image density model–as an audit and curation principle for synthetic data. To operationalise this view, we introduce Energy-Level Distribution Distance (ELD), which compares real and synthetic energy distributions. Real images span sparse high-likelihood scenes and dense low-likelihood textures, so useful synthetic data should cover this energy range rather than collapse to likely but low-variation samples. Across diverse synthetic datasets, ELD exposes mismatches missed by FID and predicts pretraining performance beyond FID, LPIPS variation, and feature-space recall, including across CNN and ViT self-supervised settings. We then use ELD to resample and mix synthetic sources, matching the real-image energy histogram while preserving diversity and improving downstream transfer.
Introduction
Synthetic visual pretraining turns data choice into a distributional design problem: models can be trained on simulations, procedural images, generator samples, or mixtures thereof, each inducing different visual statistics. Yet current selection criteria often reduce this choice to sample realism or feature-space similarity. These criteria are useful, but they compress datasets into distances, neighbourhoods or low-order feature summaries, and do not ask where samples lie in the probability landscape of real images. This missing axis matters because natural images do not occupy a single typical likelihood level. Normalised density models reveal a broad, content-dependent energy distribution, where sparse simple scenes and dense textured scenes occupy different likelihood regimes [10, 17]. This echoes classical natural-image statistics, which show that real images have structured spectral, wavelet and sparsity properties far from white noise [4, 9, 27, 28, 24]. A synthetic dataset may therefore match real feature moments while exposing a learner to the wrong likelihood regimes; selecting only the most likely samples may likewise remove complex variation that real data contains. We ask whether synthetic data should be selected and designed to match the distribution of likelihood levels occupied by real images.
Our Contributions
We make four contributions around a common claim: beyond visual realism or feature-space similarity, the energy shape of synthetic data is a useful predictor and design principle for visual pretraining.
First, we formalise this design axis through Energy-Level Distribution Distance (ELD), together with a percentile variant, which compares real and synthetic datasets through their normalised real-image log-probability distributions. Unlike mean likelihood, ELD measures the full energy profile of a dataset. It penalises synthetic data that drifts into low-likelihood, noise-like regions, as well as data that collapses towards high-likelihood but low-variation samples. This makes ELD complementary to feature-space metrics such as FID, LPIPS variation and recall, which measure moments, perceptual distances or coverage rather than probability-level structure.
Second, we use ELD to audit a broad suite of procedural, statistical, feature-visualisation and generator-based synthetic datasets. The audit reveals three recurring regimes: noise-like datasets that sit in low-likelihood regions, over-concentrated datasets that collapse around high-likelihood but low-variation samples, and broad-but-shifted datasets that span many energy levels while missing the real profile. These regimes expose why feature-space metrics can be incomplete: two datasets may have similar FID, LPIPS variation or recall while presenting very different likelihood structure to the learner. Third, we show that this structure is predictive, not merely descriptive: ELD explains pretraining utility beyond standard metrics, including across CNN and ViT self-supervised settings. Finally, we turn the audit into an intervention. By resampling within synthetic pools and mixing across generators, we match the real-image energy histogram without sacrificing feature diversity, yielding synthetic training sets with stronger downstream transfer.
Related Work
Synthetic data for visual pretraining. Synthetic visual data has been explored through several generation mechanisms: simulation and domain randomisation for transfer to real environments ([33]), fractal and procedural image datasets for pretraining without photographs ([19]), structured noise and statistical image models for contrastive representation learning ([1]), and text-to-image generators for large-scale synthetic representation learning ([32]). These works establish a rich design space for synthetic pretraining data. Our focus is not to introduce a new generator, but to identify a distribution-level property that predicts which generated datasets are useful across modern self-supervised objectives.
Distribution metrics for generated data. Most metrics compare real and generated distributions in a learned feature space. FID fits Gaussian approximations to Inception features ([13]); KID uses a kernel distance ([2]); precision and recall estimate fidelity and coverage ([21, 23]); and LPIPS measures perceptual similarity with deep features ([35]). These metrics capture fidelity and diversity, but recent work shows their sensitivity to feature choice and Gaussian assumptions ([16]). ELD is orthogonal to these criteria: it compares probability-level structure under a real-image density model rather than distances or moments in representation space.
Image likelihoods and energy structure. Energy-based models describe data by negative log density, but calibrated image likelihoods are hard to estimate [14, 22, 30]. Normalising flows give exact likelihoods through invertible transformations [25, 7, 8], while diffusion and score-based models estimate image distributions through denoising dynamics [29, 31] (Ho et al. 2020). Recent normalised density estimators enable direct analysis of natural-image likelihoods; Guth et al. [10] show that real-image log probabilities are broad, content-dependent, and approximately Gumbel-like. We use this likelihood-level view as a diagnostic and curation principle: it identifies which synthetic datasets are mismatched and how to reshape them.
Energy Structure of Natural and Synthetic Images
A useful synthetic pretraining distribution should reproduce more than the visual appearance of real images. We focus on one structural axis: the distribution of likelihood levels assigned to a dataset by a normalised real-image density model. This energy distribution is rich in natural images; it is not determined by FID or other moment-based feature-space metrics, and separates synthetic sources into three regimes with distinct measurable signatures.
The Energy Distribution of Natural Images
Let be a real-image distribution and a normalised density model fitted to . The real-image energy scores how typical is under the model: low for images assigns high likelihood, high for images assigns low likelihood. The pushforward records how often real images occur at each likelihood level. This energy distribution is broad: Guth et al. [10] show that on ImageNet it is approximately Gumbel-like, with sparse simple scenes at the low-energy/high-likelihood end and dense textured scenes forming a high-energy/low-likelihood tail, consistent with the statistical inhomogeneity of natural images [9, 27, 28, 24].
The implication for synthetic pretraining is concrete. A useful synthetic distribution should expose the learner to the same range of likelihood levels as real data, not merely maximise average likelihood. Pushing upward can collapse to the mode of ; ignoring likelihood can place mass arbitrarily far in the high-energy tail. Either failure appears as a mismatch in . The relevant comparison is therefore not a scalar summary such as mean likelihood, but the full energy profiles and , which § 3.1 measures.
Feature Moments Do Not Determine Energy Profiles
FID compares the means and covariances of a fixed feature map. These moments need not determine how samples occupy energy levels, as in Figure 1. For example, under the identity feature map, let
Figure 1. ELD distinguishes synthetic pretraining datasets FID cannot. Self-supervised pretraining benefits when synthetic data spans the full likelihood spectrum of real images; sources with identical FID can carry very different energy distributions, which ELD detects.
Both have mean zero and variance one, and therefore identical FID to any fixed reference. Under the normalised density , however, has energy almost surely, whereas has energies and with equal probability. Thus identical feature moments can conceal different energy profiles. Appendix § 7 gives the general statement.
A Diagnostic Taxonomy of Synthetic Energy Mismatch
If FID does not determine energy matching, the remaining question is empirical: do synthetic datasets differ along this axis? Figure 2 visualises representative sources scored under a shared real-image density model . Let , denote the mean and standard deviation of on real data, and , the corresponding quantities for a synthetic source. Three regimes recur. Shifted datasets have large and positive relative to , placing mass in the high-energy tail: the real model rates these images as implausible. Over-concentrated datasets have small , often with : samples may be individually plausible, but occupy too narrow a slice of natural-image structure. Broad/shifted datasets have substantial mismatch in either spread or location, with and/or far from the real reference. Mixed cases interpolate between these regimes. Figure 3 shows the same regimes at the level of full energy distributions, making the shift, compression, and tail mismatches visible beyond the two summary statistics.
Figure 2. Small-scale energy audit. Synthetic datasets scored under a grayscale ImageNet64 density model. Axes show mean energy shift and spread relative to ImageNet, while point size denotes PELD. The atlas provides a coarse diagnostic view of the regimes expanded in Figure 3: noise-like sources are shifted toward high-energy regions, over-concentrated sources occupy too narrow a likelihood band, and broad/shifted sources cover a wider range but remain displaced from the real energy profile.
Figure 3. Energy mismatch is structured. Synthetic sources differ in how they depart from ImageNet: some collapse into narrow high-likelihood bands, others extend into low-likelihood tails, and others remain broad but displaced. Effective synthetic pretraining may require matching the full probability-level profile of natural images, not simply maximising likelihood or feature realism.
These diagnostics depend on the calibration of ; § 3.1 introduces a percentile-rank variant that removes this dependence. As shown in § 2.2, datasets in different regimes can have similar FID despite different energy arrangements, and § 4.2 confirms this empirically. These regimes show why “realism” is incomplete for synthetic pretraining: a dataset can be too unlikely, too concentrated near high-likelihood regions, or mismatched across likelihood levels. The target is the full real-image energy distribution, not a scalar likelihood or feature-moment score.
ELD: Energy-Level Distribution Distance
Section 2.2 shows that feature-moment metrics do not determine a dataset’s energy profile. We now define a metric that compares datasets along this axis directly.
Definition and Estimation
Given a normalised real-image density model with energy , the Energy-Level Distribution Distance between real and synthetic distributions , is
the Wasserstein-1 distance between their one-dimensional energy distributions. In one dimension, equals the integral distance between empirical CDFs ([34]), making ELD simple to estimate from sorted energy scores. We use rather than KL because it remains well-defined under support mismatch and returns a distance in the units of the energy score.
Raw energies depend on the calibration of , so we also use a percentile version. Let denote the CDF of real-image energies for . We replace each energy by its real-energy rank . By construction, , so
For a continuous real-energy CDF , the probability integral transform yields uniform real-energy percentiles . In finite samples, we estimate from a real calibration set; independent real samples are therefore only approximately uniform. pELD is bounded in and invariant to strictly increasing recalibrations of the energy score applied consistently to real and synthetic samples. At the population level, if is strictly increasing on the combined real and synthetic energy support, it vanishes exactly when their energy distributions coincide. This removes score-scale dependence, but not changes in image ordering induced by a different scorer. Unless otherwise stated, we report pELD as ELD. Raw ELD is expressed in the units of its energy score; our plotted energies use dB per dimension.
We instantiate with the normalised ImageNet density estimator of Guth et al. [10], trained using a diffusion-inspired dual-score-matching objective. It directly provides tractable image log-density estimates, without requiring sampling from the scorer. Once energies are computed, ELD requires only empirical CDFs; § 4.2 shows that a few thousand samples per distribution suffice for stable rankings and evaluates robustness to the choice of density scorer. The main cost is energy evaluation, amortised across all synthetic datasets scored under the same .
Interpreting Eld
ELD turns the qualitative regimes of § 2.3 into a single distributional discrepancy. If a synthetic dataset is shifted into the high-energy tail, compressed around high-likelihood modes, or broad but displaced from the real profile, its empirical energy CDF departs from that of real images. The Wasserstein distance integrates these departures across the full likelihood axis, so the metric penalises datasets that are too unlikely, too concentrated, or likely at the wrong frequencies. This is the distinction missed by mean likelihood: ELD does not reward making every sample as likely as possible, but rather matching how real images populate likelihood levels.
This interpretation also clarifies the role of ELD. Rather than replacing semantic or feature-space coverage metrics, ELD isolates a complementary question: whether a dataset populates the same likelihood levels as real images under a shared density model. Two datasets may match in energy profile while differing in semantics or feature diversity; conversely, they may look similar to feature-moment metrics while occupying very different regions of the real-image likelihood axis. We therefore report ELD alongside FID, LPIPS variation, and recall in § 4.2. Figure 3 illustrates the measured object: when scored by the same ImageNet density model, synthetic families induce distinct energy profiles with different shifts, spreads, and tail structure relative to ImageNet.
Energy Shape Predicts Pretraining Utility
§ 2.2 and § 3.2 argue that energy structure is an axis FID does not see and that ELD measures it. We now ask whether this axis matters: across a broad suite and self-supervised recipes, ELD predicts downstream accuracy beyond FID, LPIPS variation, and recall.
Experimental Setup
We ask two questions: does ELD predict how well a synthetic dataset trains visual representations, and how does its held-out predictive accuracy compare with that of FID, LPIPS variation, and recall? We evaluate this in two complementary pretraining settings. The first is a self-supervised setting using DINO-style ViT pretraining (Caron et al., [5]). The second follows the setting of Baradad Jurjo et al. [1], where synthetic datasets are used directly as pretraining sources under the original CNN/contrastive evaluation protocol. In both settings, we score each synthetic dataset under the same metric suite and regress downstream accuracy on these metrics. Appendix § 8.§ 2 visualizes samples from the corresponding DINO/ViT-S and Learning-to-See synthetic sources.
Synthetic pretraining data. The suite spans the regimes of § 2.3: procedural and noise-like sources from Baradad Jurjo et al. [1], including dead leaves, statistical image models, feature visualizations, and untrained StyleGAN variants; classical synthetic pretraining sources such as FractalDB, Perlin noise, and Gaussian noise; generator-based sources including StyleGAN3, BigGAN, and diffusion/latent-diffusion samples at varied guidance or sampling settings; and curated mixtures such as StableRep and uniform mixtures (Baradad Jurjo et al., [1]; Kataoka et al., [20]; Karras et al., [18]; Brock et al., [3]; Rombach et al., [26]; Tian et al., [32]).
Pretraining recipes. Our first recipe is DINO with ViT-S under fixed hyperparameters across datasets (Caron et al., [5]). To connect with earlier synthetic-data pretraining work, we also run the “Learning to See by Looking at Noise” protocol using its corresponding CNN-based recipe and synthetic dataset families (Baradad Jurjo et al., [1]).
Evaluation. For the self-supervised setting, we evaluate ImageNet-1K representation quality using SimCLR-style linear probing and fine-tuning (Chen et al., [6]). For the Learning-to-See setting, we follow the original ImageNet100 evaluation protocol of Baradad Jurjo et al. [1]. In both settings, we additionally evaluate distribution shift on ImageNet-C and ImageNet-A (Hendrycks & Dietterich, [11]; Hendrycks et al., [12]); for Learning-to-See, these shift evaluations are restricted to the ImageNet100 class subset and resized to the encoder input resolution. For each pretraining dataset we compute ELD against ImageNet alongside FID, LPIPS, and Improved Recall (Heusel et al., [13]; Zhang et al., [35]; Kynkäänniemi et al., [21]). Table 1 reports DINO/ViT-S, while Table 3 reports the Learning-to-See setting. We use DINO/ViT-S as the modern self-supervised protocol and Learning-to-See as a complementary small-scale replication with classical synthetic pretraining.
| Dataset | ELD | FID | LPIPS | Recall | IN-1K Lin.% | IN-1K FT% | IN-C% | IN-A% |
| Noise-like | ||||||||
| FractalDB | .426 | 1254 | .260 | .018 | 23.91 | 56.66 | 18.70 | 10.30 |
| Perlin noise | .427 | 1922 | .208 | .000 | 21.53 | 55.38 | 16.79 | 9.11 |
| Gaussian noise | .500 | 2145 | .237 | .000 | 20.26 | 52.26 | 15.53 | 5.01 |
| Over-concentrated | ||||||||
| diffusion (high CFG) | .138 | 481 | .342 | .020 | 28.18 | 59.34 | 21.41 | 10.84 |
| Broad / shifted | ||||||||
| StyleGAN3 | .309 | 1148 | .305 | .001 | 24.71 | 57.49 | 18.34 | 10.59 |
| BigGAN | .112 | 566 | .303 | .101 | 28.36 | 58.97 | 18.10 | 10.35 |
| diffusion (mid CFG) | .088 | 459 | .338 | .030 | 29.17 | 59.57 | 21.44 | 12.35 |
| diffusion (low CFG) | .153 | 1129 | .297 | .022 | 28.67 | 59.60 | 21.38 | 11.99 |
| Curated mixtures | ||||||||
| uniform mixture | .112 | 616 | .294 | .023 | 27.45 | 57.61 | 20.80 | 11.07 |
Table 1. Self-supervised setting. Dataset metrics and DINO/ViT-S accuracy; metrics are computed against ImageNet, and evaluation reports ImageNet-1K linear/FT plus ImageNet-C/A robustness.
| Dataset | ELD | FID | LPIPS | Recall | IN Lin.% | IN FT% | IN-C% | IN-A% |
| Dead leaves | ||||||||
| dead-leaves squares | .393 | 1480 | .316 | .005 | 38.60 | 67.52 | 27.23 | 3.24 |
| dead-leaves oriented | .319 | 1390 | .299 | .035 | 40.90 | 68.48 | 25.12 | 3.97 |
| dead-leaves mixed | .310 | 1310 | .301 | .074 | 40.84 | 68.86 | 25.00 | 3.82 |
| dead-leaves textures | .261 | 1340 | .241 | .084 | 41.00 | 68.20 | 27.93 | 3.83 |
| Statistical image models | ||||||||
| spectrum | .314 | 1560 | .159 | .003 | 30.70 | 62.20 | 24.57 | 3.68 |
| WMM | .172 | 1520 | .152 | .003 | 34.08 | 63.30 | 16.90 | 3.53 |
| spectrum+color | .251 | 1290 | .258 | .001 | 37.88 | 65.18 | 27.82 | 4.12 |
| spectrum+color+WMM | .049 | 1210 | .228 | .010 | 38.84 | 65.62 | 30.28 | 4.56 |
| Untrained StyleGAN | ||||||||
| StyleGAN random | .479 | 1420 | .279 | .000 | 35.30 | 64.82 | 29.10 | 4.41 |
| StyleGAN high-freq. | .085 | 1320 | .268 | .002 | 39.62 | 66.21 | 29.84 | 4.38 |
| StyleGAN sparse | .278 | 1240 | .305 | .001 | 38.76 | 66.16 | 27.23 | 3.24 |
| StyleGAN oriented | .355 | 903 | .272 | .007 | 39.08 | 65.84 | 27.70 | 3.38 |
| Feature visualizations | ||||||||
| feature-vis random | .402 | 1590 | .252 | .000 | 35.26 | 66.22 | 22.89 | 2.65 |
| feature-vis dead leaves | .365 | 1400 | .223 | .001 | 38.70 | 68.00 | 32.04 | 4.71 |
Table 3. Table 3: Learning-to-See synthetic-pretraining setting. Dataset metrics and downstream accuracy for the small-scale families from Baradad Jurjo et al. [1]. Evaluation follows the original ImageNet protocol and adds ImageNet-C/A robustness with the same representations.
ELD Predicts Downstream Quality Beyond FID
We next test whether probability-level mismatch predicts pretraining utility in a modern self-supervised setting. Table 1 reports DINO/ViT-S models pretrained on each synthetic source, with ImageNet-1K linear probing, ImageNet-1K fine-tuning, and ImageNet-C/A robustness. Across this suite, ELD is consistently negatively associated with downstream performance: lower ELD corresponds to higher frozen ImageNet-1K linear accuracy (Spearman , ), higher ImageNet-1K fine-tuning accuracy (, ), and better robustness on ImageNet-C (, ) and ImageNet-A (, ). These results support the directional prediction of § 2.3: sources whose energy profiles depart from the real-image likelihood distribution yield weaker transferable representations. Figure 4 visualises the trend, with regimes organizing the scatter from high-ELD noise-like sources at the weak-transfer end to lower-ELD diffusion and mixture sources among the strongest models.
Figure 4. ELD predicts DINO/ViT-S downstream quality. Across the self-supervised synthetic pretraining suite, lower ELD is associated with stronger frozen ImageNet-1K linear-probe accuracy and better robustness on ImageNet-C and ImageNet-A. Points are colored by the energy-regime taxonomy, showing that high-ELD noise-like sources occupy the weak-transfer end of the trend, while lower-ELD diffusion and mixture sources cluster among the strongest representations.
Held-out prediction. FID and recall also correlate with transfer on the DINO suite. To compare predictive utility, we fit a separate univariate linear regression for each metric, omitting one dataset at a time and predicting its downstream accuracy. ELD gives the lowest prediction error for DINO linear accuracy among the four evaluated metrics (Table 2). This advantage persists after excluding Gaussian noise, Perlin noise, and FractalDB: ELD’s mean absolute error is 0.99 percentage points, compared with 1.66 for FID, 1.47 for LPIPS variation, and 3.27 for recall. Full comparisons and analyses appear in Appendix § 8.§ 3.
| DINO linear evaluation | ELD | FID | LPIPS | Recall |
| All sources () | 0.82 | 0.91 | 1.67 | 4.53 |
| Without noise-like sources () | 0.99 | 1.66 | 1.47 | 3.27 |
Table 2. Leave-one-dataset-out DINO prediction MAE (percentage points ).
We also evaluate ELD in the classical synthetic-pretraining setting of Baradad Jurjo et al. [1]. Table 3 reports the small-scale synthetic families under the original ImageNet100 transfer protocol, augmented with ImageNet-C/A evaluations. This benchmark is heterogeneous by construction: dead leaves, statistical models, untrained StyleGANs, and feature visualizations differ not only in energy profile but also in construction mechanism and semantics. Comparing across heterogeneous synthetic sources, the pooled correlations remain directionally consistent: lower ELD is associated with higher ImageNet-A accuracy (Spearman ) and weakly with higher ImageNet-C accuracy (). ImageNet-A accuracy is computed on the intersection with the trained ImageNet100 classes; examples outside this label set are excluded from the evaluation denominator.
The within-family comparisons are particularly informative. Among dead-leaves variants, lower ELD tracks stronger early ImageNet transfer ( for linear accuracy), with the lowest-ELD texture variant achieving the best linear accuracy and ImageNet-C accuracy in the group. Among statistical image models, ELD ranks both ImageNet linear and fine-tuned transfer strongly ( for each), and the best energy match, spectrum+color+WMM, is also the strongest source on ImageNet linear transfer, fine-tuning, ImageNet-C, and ImageNet-A. The untrained StyleGAN family shows the same pattern for ImageNet transfer: lower ELD is associated with stronger linear accuracy () and fine-tuned accuracy (), with the high-frequency variant attaining the best transfer scores. For the two feature-visualization sources, the lower-ELD dead-leaves variant is better on every downstream metric. ELD provides a consistent directional signal especially when comparing synthetic sources within a shared family.
Together, the two experimental settings support the regime picture of § 2.3. ELD does not subsume all factors governing synthetic pretraining: semantic coverage, augmentation compatibility, and feature diversity remain important. Instead, it isolates a complementary axis of mismatch, namely whether a synthetic source places probability mass at the likelihood levels occupied by real images. The DINO/ViT-S results show that this axis predicts transfer and robustness in a modern self-supervised protocol, while the Learning-to-See results show that the same signal remains interpretable in a classical small-scale synthetic-pretraining setting, especially within matched generator families. This is the role of ELD in our analysis: not a universal replacement for feature-space metrics, but a probability-level diagnostic that explains a structured component of downstream variation.
Estimation robustness. Figure 5 verifies that the reported percentile version is not an artefact of calibration or sample size. Raw energy-space distance and percentile ELD are strongly aligned across datasets, indicating that rank calibration preserves the relevant ordering while removing dependence on the absolute scale of the density model. Subsampling shows that ELD estimates stabilise after a few thousand scored examples, well below the sample sizes used in the main experiments.
Figure 5. ELD is robust to estimation choices. Raw and percentile distances agree with a Spearman coefficient of 0.7; subsampling shows stable estimates after a few thousand scored samples.
Robustness to the neural scorer. We recompute pELD for the nine DINO sources with two additional neural scoring constructions, calibrating each against its own ImageNet reference. Dataset rankings closely agree with those of the primary Guth scorer (Spearman –), and lower pELD remains associated with stronger transfer (Table 4). This supports the stability of the observed signal across the tested neural scorers.
| Scorer | Agreement with Guth | Linear | Fine-tuning |
| Guth dual-score model | 1.00 | -0.87 | -0.78 |
| DenseFlow | 0.89 | -0.85 | -0.78 |
| Diffusion VLB | 0.91 | -0.77 | -0.72 |
Table 4. Neural-scorer sensitivity on the DINO suite. The first column compares dataset rankings; the remaining columns correlate pELD with ImageNet-1K accuracy.
Energy-Aware Curation for Synthetic Data
ELD suggests a constructive use beyond evaluation: synthetic data can be shaped to match the probability-level profile of real images. Natural images occupy a broad energy spectrum containing both simple high-likelihood scenes and dense low-likelihood structure ([10, 9, 27, 28]). Filtering only for the highest-likelihood samples can therefore discard variation present in real data. The goal is to make synthetic pretraining data populate real-image likelihood levels in the right proportions, not only to retain the most probable samples.
Because real-energy percentiles are uniform under a continuous reference CDF, a simple curation rule is to resample synthetic data so that its percentile histogram is approximately uniform. In practice, we can score a candidate pool under , bin samples by their real-energy percentiles, and draw a balanced training stream across bins, with complementary diversity criteria guiding selection within each band. This changes the composition of the selected set without retraining a generator. Reweighting cannot recover energy bands absent from the candidate pool; such gaps require additional sources or generation settings. For mixtures, the same principle becomes a small convex design problem: choose source weights whose combined energy histogram best matches the real one under a convex histogram discrepancy, such as discretised Wasserstein-1 distance. Thus the audit becomes prescriptive: over-concentrated sources can be supplemented with underrepresented higher-energy structure; noise-like sources can be downweighted in favour of candidates in reference-supported energy bands; and broad but shifted sources can contribute where they fill missing likelihood bands. These interventions target energy occupancy, while semantic and feature-space diagnostics remain necessary to assess the resulting composition.
Controlled offline curation. Energy-profile matching is an alternative to selecting only the highest-likelihood images. We compare source-balanced uniform selection, top-likelihood filtering, and ELD-guided selection from a common pool spanning BigGAN, StyleGAN3, and three diffusion settings, using the same selected-set budget across strategies. Top-likelihood filtering retains the lowest-energy candidates; ELD-guided selection seeks to match the full ImageNet energy-percentile histogram. No generator retraining or downstream model fitting is required for this offline intervention.
We evaluate the selected sets using frozen ImageNet-pretrained ResNet features (Table 5). ELD-guided selection improves both reported nearest-neighbour distance diagnostics and increases effective class coverage and feature rank relative to uniform selection. Top-likelihood filtering performs worse on all four diagnostics. Thus matching energy occupancy can retain useful feature diversity that likelihood maximisation discards. These are offline proxy results; matched pretraining experiments are deferred to future work to further establish a transfer gain caused by the intervention.
| Selection | NN mean ↓ | NN p95 ↓ | Eff. classes ↑ | Eff. rank ↑ |
| Uniform | 0.634 | 0.784 | 116.445 | 59.175 |
| Top likelihood | 0.660 | 0.788 | 82.940 | 34.132 |
| ELD-guided | 0.621 | 0.780 | 128.001 | 71.828 |
Table 5. Controlled offline curation from a common source pool at equal selected-set budget. Frozen-feature diagnostics favour ELD-guided selection over uniform and top-likelihood selection. NN denotes the cosine-distance nearest-neighbour diagnostic.
More broadly, ELD reframes synthetic-data construction as distributional engineering. Given a synthetic pool with cached energies and features, we may select samples to match the real energy histogram while imposing complementary feature-coverage or diversity constraints within each band. The selected distribution should satisfy
while retaining useful variation along dimensions not determined by energy alone. This seeks to restore under-covered likelihood bands, rather than merely keep the most likely samples. It turns common design choices—which samples to retain, which generators to mix, and which prompts or guidance scales to use—into ways of redistributing mass along the real-image energy axis. The target along this axis is the full probability-level profile of natural images; energy matching complements, rather than replaces, semantic and task-specific selection criteria.
Conclusion
Discussion and Limitations. ELD makes synthetic data inspectable at the level of likelihood structure, revealing which parts of a reference probability landscape samples occupy. It complements FID, recall, and LPIPS variation: feature moments can miss energy-profile differences, while energy matching alone does not guarantee semantic coverage, diversity, label quality, or task relevance. Combining these criteria supports audits of both likelihood structure and feature-space realism and coverage. The reference data and density model are part of the experimental specification, so ELD should be reported with its scorer. Percentile calibration removes strictly increasing score recalibrations; our cross-scorer analyses support robustness across the tested neural scoring constructions. Our experiments focus on visual pretraining; supervised-label efficiency, aesthetic fidelity, and human preference may require additional task-aware criteria. CLIP-style multimodal pretraining further motivates joint audits of visual, linguistic, and image–text alignment structure.
Synthetic data is not characterised by visual realism or feature-space similarity alone; its position in the real-image probability landscape also matters. ELD measures this axis by comparing the distribution of likelihood levels induced by real and synthetic data under a shared density model. We show that synthetic sources occupy distinct energy regimes that feature moments do not determine, and that these regimes are predictive of downstream pretraining quality in the studied settings. Our controlled curation study demonstrates a practical intervention through improved frozen-feature diagnostics; matched pretraining runs can test how these gains translate into transfer. Since the mismatch is structured, ELD is also prescriptive: it identifies likelihood bands that are missing, overrepresented, or shifted, enabling resampling and mixing strategies that better match the reference energy profile. More broadly, this supports a design view of synthetic data: generators, filters, and mixtures can be selected for how they populate the real-image likelihood spectrum, alongside their realism, diversity, and task relevance.
Acknowledgments
This work was partially completed during C.L.’s visit to the Technical University of Munich, supported by the Munich Center for Machine Learning (MCML). S.J. was supported by an Alexander von Humboldt professorship. A.T. was supported by the ONR MURI grant N000142612255. This work was further supported by a Packard Fellowship to P.I., along with the ONR MURI grant N00014-22-1-2740.
Feature Moments and Energy Profiles
Proposition. Let be a measurable feature map, and let be a fixed reference energy. Let be a feature distribution with finite second moments. Suppose there exist measurable selections such that
Define . Then and have identical feature distributions, and hence identical FID to any fixed reference with finite feature second moments. Their energy profiles nevertheless differ whenever
Moreover, suppose the induced energies have finite first moments. If a measurable feature set has mass , the selections agree outside , and
then
Proof. By construction, . Thus both distributions have the same feature mean and covariance, yielding identical FID to any fixed reference. Their energy pushforwards are , which differ under the stated hypothesis.
For the quantitative claim, agreement outside gives
The Wasserstein-1 distance is at least the absolute difference of the means, proving the bound. □
The proposition establishes that feature moments do not determine energy occupancy. Conversely, matching an energy profile does not determine the image distribution or guarantee semantic coverage; the two diagnostics therefore provide complementary information.
Experiments
Compute details. All density scoring, metric computation, and downstream training jobs were run on NVIDIA V100 GPUs. Energy scores are computed once per dataset and reused across all ELD analyses; after scoring, ELD itself is computed from sorted one-dimensional energy values and is negligible compared with image scoring or representation training.
Diagnostics
Simple diagnostics give a coarse regime map, not a substitute for ELD. Mean energy shift and spread ratio provide a useful low-dimensional summary of how a source departs from ImageNet: they separate compression, displacement, and high-energy mismatch. Table 6 shows that these summaries recover the coarse regimes used in the main text. However, they do not determine the full energy distribution. Sources within the same regime can have different tail structure and different pELD, so the Wasserstein comparison remains necessary for quantifying residual distributional mismatch.
| Regime | # datasets | Mean shift | Mean spread | Mean PELD |
| Over-concentrated | 3 | -4.43 | 0.52 | .281 |
| Broad / shifted | 7 | -5.41 | 1.47 | .284 |
| Noise-like / high-energy | 4 | 2.92 | 0.22 | .300 |
Table 6. Coarse regime diagnostics. Mean shift and spread separate the main regimes; PELD captures distributional mismatch beyond these summaries.
Generic feature-density energies are not substitutes for ELD. We also test whether simpler surrogate energy models reproduce the ranking induced by the learned ImageNet density. Specifically, we fit three lightweight alternatives on ImageNet: a diagonal Gaussian over frozen ResNet-18 features, a diagonal Gaussian over low-frequency greyscale DCT coefficients, and a Gaussian mixture model over PCA-compressed DCT features. Table 7 reports their rank agreement with the primary scorer on the small-scale synthetic suite. These simple surrogates do not recover its dataset ordering, supporting the use of the learned real-image density scorer for this audit. Their rank correlations with primary pELD are negative or near zero: for ResNet-feature Gaussian energy, for diagonal DCT-Gaussian energy, and for DCT-GMM energy. This suggests that ELD is not merely measuring generic feature outlierness, spectrum mismatch, or distance from simple image statistics. The choice of matters: ELD is intended to compare datasets under a real-image density model, not under arbitrary feature-space novelty scores.
| Surrogate energy model | Description | Spearman with pELD |
| ResNet-feature Gaussian | Diagonal Gaussian on frozen features | |
| DCT Gaussian | Diagonal Gaussian on low-frequency DCT coefficients | |
| DCT-GMM | GMM on PCA-compressed DCT coefficients |
Table 7. Simple surrogate energies do not reproduce ELD. Feature- and DCT-based density surrogates fail to recover the ranking induced by the learned ImageNet density model, indicating that ELD is not reducible to generic feature novelty or low-level image-statistics mismatch.
Synthetic Source Visualizations
Figures 6 and 7 show sampled images from the synthetic sources used in the DINO/ViT-S and Learning-to-See experiments. The visual differences across sources are substantial, but visual appearance alone does not determine the energy profile: visually diverse generators can still be shifted or compressed along the real-image likelihood axis, while simpler sources can occupy distinct energy regimes. These grids provide qualitative context for the quantitative energy audit in the main text.
Held-Out Prediction and Source-Exclusion Analyses
We examine whether the DINO linear-accuracy prediction result depends on particular synthetic sources. Table 8 compares ELD, FID, LPIPS variation, and recall on the full suite and after excluding noise-like sources, GANs, diffusion sources, or the uniform mixture. Within each retained subset, we repeat leave-one-dataset-out prediction using a separate ordinary least-squares regression with an intercept for each metric, with no hyperparameter tuning. Each evaluated dataset is excluded from regression fitting. MAE is measured in accuracy percentage points. These comparisons assess sensitivity to source composition, not training-seed variability or generalisation to an omitted family.
| DINO subset | ELD | FID | LPIPS | Recall | |
| All sources | 9 | 0.82 | 0.91 | 1.67 | 4.53 |
| Without noise-like | 6 | 0.99 | 1.66 | 1.47 | 3.27 |
| Without GANs | 7 | 1.06 | 1.17 | 1.63 | 1.34 |
| Without diffusion | 6 | 1.07 | 0.46 | 2.26 | 4.08 |
| Without uniform mixture | 8 | 0.76 | 0.96 | 1.75 | 4.95 |
Table 8. Sensitivity of DINO linear prediction to source exclusion. Each row repeats leave-one-dataset-out fitting within the retained subset. It does not evaluate transfer to the omitted family.
ELD has the lowest MAE on the full suite and after excluding noise-like sources, GANs, or the uniform mixture. In particular, its advantage after removing noise-like sources supports a predictive signal beyond the separation of obvious low-quality datasets. The comparison remains family-dependent: FID performs better after diffusion sources are removed. These small-suite analyses support energy occupancy as a useful predictor of DINO linear transfer across several source compositions, alongside its diagnostic interpretation.

Figure 6. DINO/ViT-S synthetic pretraining sources. This grid highlights the range from noise and procedural structure to generator samples; ELD measures how these sources populate ImageNet likelihood levels.

Figure 7. Learning-to-See synthetic pretraining sources. This grid visualises the range from dead-leaves and statistical textures to untrained StyleGAN and feature-visualization samples; ELD measures how these families populate ImageNet likelihood levels.
References
- [1]Baradad Jurjo, M., Wulff, J., Wang, T., Isola, P., and Torralba, A. Learning to see by looking at noise. Advances in Neural Information Processing Systems, 34:2556–2569, 2021.arxiv.org/abs/2106.05963
- [2]Bíńkowski, M., Sutherland, D. J., Arbel, M., and Gretton, A. Demystifying mmd gans. arXiv preprint arXiv:1801.01401, 2018.arxiv.org/abs/1801.01401
- [3]Brock, A., Donahue, J., and Simonyan, K. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=B1xsqj09Fm.
- [4]Burton, G. J. and Moorhead, I. R. Color and spatial structure in natural scenes. Applied optics, 26(1):157–170, 1987.
- [5]Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650–9660, 2021.arxiv.org/abs/2104.14294
- [6]Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597–1607. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/chen20j.html.
- [7]Dinh, L., Krueger, D., and Bengio, Y. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.DOI
- [8]Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density estimation using real NVP. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=HkpbnH9lx.arxiv.org/abs/1605.08803
- [9]Field, D. J. Relations between the statistics of natural images and the response properties of cortical cells. Journal of the Optical Society of America A, 4(12):2379–2394, 1987.
- [10]Guth, F., Kadkhodaie, Z., and Simoncelli, E. P. Learning normalized image densities via dual score matching. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=wtYcS4kxpF.
- [11]Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HJz6tiCqYm.
- [12]Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262–15271, 2021.
- [13]Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local Nash equilibrium. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/8a1d694707eb0efec6587136907462d-Paper.pdf.
- [14]Hinton, G. E., Sejnowski, T. J., et al. Learning and relearning in boltzmann machines. Parallel distributed processing: Explorations in the microstructure of cognition, 1(282-317):2, 1986.
- [16]Jayasumana, S., Ramalingam, S., Veit, A., Glasner, D., Chakrabarti, A., and Kumar, S. Rethinking fid: Towards a better evaluation metric for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9307–9315, June 2024. 2arxiv.org/abs/2401.09603
- [17]Kadkhodaie, Z., Guth, F., Simoncelli, E. P., and Mallat, S. Generalization in diffusion models arises from geometry-adaptive harmonic representations. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=ANvmVS2YrO. 1arxiv.org/abs/2310.02557
- [18]Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., and Aila, T. Alias-free generative adversarial networks. Advances in neural information processing systems, 34:852–863, 2021. 6arxiv.org/abs/2106.12423
- [19]Kataoka, H., Okayasu, K., Matsumoto, A., Yamagata, E., Yamada, R., Inoue, N., Nakamura, A., and Satoh, Y. Pre-training without natural images. In Proceedings of the Asian Conference on Computer Vision, 2020. 2
- [20]Kataoka, H., Hayamizu, R., Yamada, R., Nakashima, K., Takashima, S., Zhang, X., Martinez-Noriega, E. J., Inoue, N., and Yokota, R. Replacing labeled real-image datasets with auto-generated contours. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 21232–21241, 2022. 6DOI
- [21]Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019. 2, 6
- [22]LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., Huang, F., et al. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006. 3
- [23]Naeem, M. F., Oh, S. J., Uh, Y., Choi, Y., and Yoo, J. Reliable fidelity and diversity metrics for generative models. In International conference on machine learning, pp. 7176–7185. PMLR, 2020. 2arxiv.org/abs/2002.09797
- [24]Portilla, J. and Simoncelli, E. P. A parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision, 40(1):49–70, 2000. 1, 3DOI
- [25]Rezende, D. and Mohamed, S. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530–1538. PMLR, 2015. 3arxiv.org/abs/1505.05770
- [26]Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684–10695, 2022. 6arxiv.org/abs/2112.10752
- [27]Ruderman, D. L. Origins of scaling in natural images. Vision research, 37(23):3385–3398, 1997. 1, 3, 9
- [28]Simoncelli, E. P. 4.7 statistical modeling of photographic images. Handbook of Video and Image Processing, 9, 2005. 1, 3, 9
- [29]Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256–2265. pmlr, 2015. 3
- [30]Song, Y. and Kingma, D. P. How to train your energy-based models. arXiv preprint arXiv:2101.03288, 2021. 3arxiv.org/abs/2101.03288
- [31]Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=PxTIG12RRHS. 3
- [32]Tian, Y., Fan, L., Isola, P., Chang, H., and Krishnan, D. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 48382–48402. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/971f1e59cd956cc094da4e2f78c6ea7c-Paper-Conference.pdf. 2, 6arxiv.org/abs/2306.00984
- [33]Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23–30. IEEE, 2017. 2arxiv.org/abs/1703.06907
- [34]Villani, C. Topics in optimal transportation, volume 58. American Mathematical Soc., 2021. 5
- [35]Zhang, R., Isola, P., Efros, A. A., Sしゃechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586–595, 2018. 2, 6arxiv.org/abs/1801.03924




