betaFeedback

WorksComputer Science

Scaling Transformer Energy-based Models For High Quality Image Synthesis

Abstract

Energy-Based Models (EBMs) are a flexible class of generative models that decouple probabilistic modeling from the generation process. This decoupling directly enables compositional generation and inference-time scaling, where sample quality can be refined simply by increasing sampling steps. However, severe challenges in high-dimensional sampling and training instabilities with deep architectures have restricted EBMs to low-dimensional regimes. To resolve these bottlenecks, we introduce the Transformer Energy-Based Model (TEM), a scalable transformer energy function operating entirely within a latent space. On top of this backbone, we extract energies at multiple layers and sum them, composing complementary constraints from local texture to global semantics. Furthermore, two components make this high-quality at scale: Positive Representation Alignment (PRA), which supervises TEM's intermediate representations with a vision foundation model, and advanced sampling strategies that improve sample quality at inference. TEM-XL achieves an FID-50K of 3.82 on class-conditional ImageNet 256×\times256 without classifier-free guidance, showing that a contrastive EBM can generate high-fidelity images. On ImageNet 512×\times512, TEM achieves an FID-50K of 8.24, demonstrating that the approach scales to higher resolution. We further validate the core promises of EBMs: compositional generation, zero-shot inpainting, and out-of-distribution detection.

Introduction

Energy-Based Models (EBMs) [9, 23] parameterize probability distributions via an energy function that assigns lower energies to more probable samples and higher energies to less probable samples. EBMs provide a flexible and mathematically elegant framework that fundamentally separates explicit generation from probabilistic modeling. Unlike Diffusion [18, 38] and Flow [24] models, which are mathematically constrained to pre-defined generative trajectories governed by carefully designed timestep schedules, EBMs define the data distribution solely through an energy function, entirely decoupling the model from the generative process. This decoupling of the model from the generative process yields profound advantages. First, EBMs decouple learning from inference, allowing the sampling procedure to be chosen independently of model training. This enables flexible test-time compute scaling by increasing the MCMC steps, progressively refining the samples through a deeper exploration of the learned energy landscape. Second, this flexibility enables the composition of different distributions together [9, 11], which can enable zero-shot generation and generalization capabilities.

However, despite their theoretical advantages and compositional properties, EBMs have struggled with widespread adoption due to several critical unresolved challenges. First, EBMs face severe vulnerability to the curse of dimensionality: MCMC sampling suffers from poor mixing in high-dimensional spaces, leading to slow convergence and mode collapse. Second, EBMs scale poorly with model capacity. As network depth increases, conventional architectures exhibit severe training instability, making robust optimization exceedingly difficult [9, 10]. Consequently, existing EBM research has been largely limited to smaller-scale benchmarks, unable to demonstrate competitive performance on challenging, high-resolution datasets.

Image not extracted

Figure 1. High-fidelity ImageNet 256×256, 512×512 generation and compositional synthesis by TEM-XL/2. Left: uncurated class-conditional samples from 80 Langevin steps. The two large images are 512×512 samples and the remaining tiles are 256×256. Right: compositional generation by summing the energy functions of two independent classes, producing a coherent scene that satisfies both concepts without retraining.

In this work, we address these fundamental limitations and demonstrate that EBMs trained contrastively, with negatives drawn from the model’s own MCMC sampler, can generate high-quality 256 × 256 ImageNet [6] samples through persistent sampling initialized solely from Gaussian noise, achieving performance competitive with strong generative models (Table 1). We further validate the core theoretical promises of EBMs by demonstrating compositional generalization as well as flexible inference-time control. Our work addresses the long-standing scalability bottleneck of contrastive EBMs, reestablishing them as a viable contender for high-fidelity image generation.

MethodParamsNFEFID↓\downarrowFID↓\downarrow (CFG)NFEFID↓\downarrowFID↓\downarrow (CFG)
GAN
BigGAN-deep112M16.95–18.43–
GigaGAN569M13.45––––
Autoregressive / Masked
MaskGIT227M86.18–127.32–
VAR-d302B10–1.92–––
Diffusion/Flow
ADM554M25010.94–25023.24–
LDM-4400M25010.563.60–––
DiT-XL/2675M2509.622.2725012.033.04
SiT-XL/2675M2508.32.06250–2.62
SiT-XL/2+REPA675M2505.91.42250–2.08
EBM
TEM-L415M604.74––––
TEM-XL609M603.82–608.24–

Table 1. Class-conditional generation on ImageNet-256×256 and 512×512 (FID-50K). Params refer to the 256×256 configuration unless a resolution-specific model is used. Baseline values are taken as reported in the original papers; “–” indicates that the corresponding setting is or was not reported. ImageNet 256×256 ImageNet 512×512.

Our key contributions are: (1) A scalable Vision Transformer-based EBM architecture with multiple energy heads that enables stable training at high resolution. (2) A comprehensive training framework encompassing latent space design, representation alignment, and advanced sampling strategies that together achieve state-of-the-art quality images for contrastive energy-based models. (3) Empirical validation of the theoretical advantages of energy-based modeling at high resolution: compositional generation, zero-shot inpainting, and out-of-distribution detection from a single scalar energy.

Background

Energy-Based Models

Energy-Based Models (EBMs) define a probability distribution implicitly over an input domain using a scalar energy function, expressed as pθ(x)=exp⁡(−Eθ(x))Z(θ)p_{\theta}(x) = \frac{\exp(-E_{\theta}(x))}{Z(\theta)}, where Eθ(x)E_{\theta}(x) is the parameterized energy function. In practice, Eθ(x)E_{\theta}(x) is parameterized by a neural network Eθ:X→RE_{\theta} : \mathcal{X} \to\mathbb{R}, yielding an unnormalized density model. Maximum likelihood training then shapes the energy landscape such that low-energy regions correspond to high-density regions of the data distribution pdata(x)p_{\mathrm{data}}(x), ensuring that pθ(x)p_{\theta}(x) approximates pdata(x)p_{\mathrm{data}}(x). However, Z(θ)=∫xexp⁡(−Eθ(x)) dxZ(\theta) = \int_x \exp(-E_{\theta}(x)) \, \mathrm{d}x is intractable, which makes exact maximum likelihood training of EBMs computationally infeasible. On the other hand, the gradient of log-likelihood remains tractable [9, 17]:

∇θEx∼pdata(x)[−log⁡pθ(x)]=Ex∼pdata(x)[∇θEθ(x)]−Ex∼pθ(x)[∇θEθ(x)](1)\nabla_{\theta}\mathbb{E}_{x\sim p_{\mathrm{data}}(x)}[-\log p_{\theta}(x)] = \mathbb{E}_{x\sim p_{\mathrm{data}}(x)}[\nabla_{\theta}E_{\theta}(x)] - \mathbb{E}_{x\sim p_{\theta}(x)}[\nabla_{\theta}E_{\theta}(x)] \tag*{(1)}

The first term is estimated directly from samples from the data. The second requires expectations under pθ(x)p_{\theta}(x), which is intractable to sample exactly. Running Markov Chain Monte Carlo (MCMC) to convergence is required, implying infinitely long chains with infinitesimal steps.

Short-Run Langevin MCMC

We therefore rely on short-run MCMC to obtain approximate samples. In particular, we start from a fixed initial distribution p0(z)p_0(z) and perform a finite number of TT Langevin updates with step size η\eta:

zt+1=zt+η∇zlog⁡π(zt)+2ηϵ,t=0,…,T−1,(2)z_{t+1} = z_t + \eta\nabla_z \log\pi(z_t) + \sqrt{2\eta}\epsilon,\qquad t = 0,\ldots,T-1, \tag*{(2)}
Image not extracted

Figure 2. Proposed Architecture and Training Strategy. Left: the Transformer Energy-Based Model processes VAE-encoded latent tokens through a stack of TEM blocks. Energy heads at multiple intermediate layers produce scalar energies whose sum defines the total energy. Right: contrastive training pushes down the energy of data samples x+x^{+} and pushes up the energy of MCMC negatives x−x^{-}, while Positive Representation Alignment (PRA) aligns the model’s internal features with a frozen VFM.

where ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0, I) [9, 29]. Let π~(z)\tilde{\pi}(z) denote the distribution of the final state zTz_{T}. Since initialization p0p_{0}, step size η\eta, and number of steps TT are fixed, π~\tilde{\pi} is well-defined. In practice, we use a finite number of steps and tune the step size and noise scale independently, yielding an efficient but biased sampler.

Scalable Transformer Energy-Based Models

Transformer Energy-Based Models Architecture

TEM Architecture. We parameterize the energy function Eθ(x,c)E_{\theta}(x, c) with a Vision Transformer (ViT) [8] backbone adapted for large-scale energy-based modeling. Given an input xx, we tokenize it into spatial patch embeddings, which are processed through a stack of transformer blocks. The final patch tokens are aggregated via global average pooling and passed through a small MLP head to produce a scalar energy Eθ(x,c)E_{\theta}(x, c).

Our architecture adopts several modern design choices for stability and expressivity. We encode spatial structure with rotary positional embeddings [39], and apply query-key normalization [15] to stabilize attention logits at scale. Feed-forward layers use SwiGLU activations [36], and the conditioning signal is projected into feature-wise affine parameters in an AdaLN-style formulation [33] built on the same RMSNorm variant.

Importantly, this design demonstrates stable and effective energy-based modeling at scale without requiring EBM specific components such as spectral normalization commonly used in prior energy-based models [9].

Multi-layer Energy Extraction. Energy-based models are naturally suited to composition, since combining distributions amounts to addition in energy space. Du et al. [10] demonstrate this by composing several separately trained models, each capturing a distinct concept. Composition of this kind has so far required multiple models.

We show that it can instead be obtained within one, extracting energies through multiple heads attached to representations at different depths of a single Transformer. Its layers encode progressively more abstract features, from low-level structure in early layers to high-level semantics in deeper ones [34]. For LL depth layer with kk heads, we attach them at the blocks nearest to (L/k,2L/k,…,L)(L/k, 2L/k, \ldots, L). The resulting heads form a hierarchy of energy terms, each enforcing the structure visible at its depth. We train on the sum of these energies, allowing each layer to contribute at its own level of abstraction. This not only yields a better energy function but also decomposes the generative process across layers, opening the way to understanding each layer’s role and further applications.

Training Transformer Energy-Based Models

Latent Space. Training EθE_\theta requires sampling negatives by MCMC at every step, which is expensive in high dimensions and mixes poorly. Therefore, we operate entirely in the latent space induced by the autoencoder of Stable Diffusion [35], which is common in the literature [33]. Given an input image x∈R256×256×3x \in\mathbb{R}^{256 \times256 \times3}, we encode to latent space z∈R32×32×4z \in\mathbb{R}^{32 \times32 \times4} where the encoder predicts the mean μenc(x)\mu_{\mathrm{enc}}(x) and variance σenc2(x)\sigma_{\mathrm{enc}}^2(x). We sample latent variables as

z∼N(μenc(x),σenc2(x))(3)z \sim\mathcal{N}\left(\mu_{\mathrm{enc}}(x), \sigma_{\mathrm{enc}}^2(x)\right) \tag*{(3)}

which corresponds to drawing from the learned posterior distribution of the autoencoder. In this latent space, we define an energy-based model Eθ(z,c)E_\theta(z,c) and perform learning and sampling directly on zz. The decoder D(⋅)D(\cdot) maps latent samples back to the image space via x≈D(z)x \approx D(z).

Contrastive Energy Objective. We train the energy-based model using a contrastive objective that encourages lower energy for data samples than for model samples.

Lcontrast=E(z+)−E(qθ(z0))+λ{E(z+)2+E(qθ(z0))2}(4)\mathcal{L}_{\mathrm{contrast}} = E(z^+) - E(q_\theta(z_0)) + \lambda\left\{E(z^+)^2 + E(q_\theta(z_0))^2\right\} \tag*{(4)}

Here, z+z^+ denotes an encoded positive sample drawn from the dataset, and qθ(z0)q_\theta(z_0) denotes a negative sample produced by sampler. z0z_0 is from pure noise or replay buffer. Since the likelihood gradient depends only on energy differences, the magnitudes of energies are relative and drift during training. We add an L2 regularization term penalizing the magnitude of the energy to resolve this gauge freedom and bound the energy magnitude following Du & Mordatch [9].

Sampler Matching Loss. Because the sampler runs for a finite number of steps, its output distribution qθq_\theta may not exactly match the model distribution pθp_\theta from which the negative phase should draw. Following Du et al. [10], we introduce a regularizer motivated by KL(qθ∥pθ)\mathrm{KL}(q_\theta\parallel p_\theta):

Lsampler-match=Eqθ(z^−)[Esg(θ)(z^−)]⏟Lopt+Eqθ(z^−)[log⁡qθ(z^−)]⏟Lent,(5)\mathcal{L}_{\mathrm{sampler\text{-}match}} = \underbrace{\mathbb{E}_{q_\theta(\hat{z}^-)}\left[E_{\mathrm{sg}(\theta)}(\hat{z}^-)\right]}_{\mathcal{L}_{\mathrm{opt}}} + \underbrace{\mathbb{E}_{q_\theta(\hat{z}^-)}\left[\log q_\theta(\hat{z}^-)\right]}_{\mathcal{L}_{\mathrm{ent}}}, \tag*{(5)}

where z^−\hat{z}^- is a sample generated by sampler and the stop-gradient sg(⋅)\mathrm{sg}(\cdot) freezes the energy parameters, so gradients reach θ\theta only through the sampler. The first term drives the sampler toward low-energy regions. The second is an entropy term, estimated with a nearest-neighbor statistic [2], that keeps its outputs diverse. Together, these terms encourage low-energy, diverse sampler outputs. Appendix A, B gives the details and reports training settings.

Representation Supervision. Contrastive EBM training provides only a scalar supervisory signal per sample, one energy value, which makes representation learning challenging at high resolution. A natural remedy is to supply a dense signal to the representation itself, as in diffusion models, where aligning generative features with those of a pretrained vision foundation model (VFM) substantially improves sample quality [43, 48]. We study how such alignment should be formulated for energy-based models and introduce Positive Representation Alignment (PRA), an objective designed for the contrastive setting.

Let hˉVFM(x+)∈RN×D\bar{h}_{\mathrm{VFM}}(x^+) \in\mathbb{R}^{N \times D} denote the spatially normalized patch-wise output of the VFM, where NN is the number of patches and DD is the embedding dimension. Let hθ(z+)h_\theta(z^+) denote the internal representation of the EBM and Pϕ(⋅)P_\phi(\cdot) a learnable projection network. PRA maximizes the patch-wise cosine similarity between the projected EBM features and the VFM targets:

LPRA(θ,Pϕ):=−1N∑n=1Nsim⁡(hˉVFM(x+)[n],Pϕ(hθ(z+))[n]).(6)\mathcal{L}_{\mathrm{PRA}}(\theta, P_\phi) := -\frac{1}{N}\sum_{n=1}^{N}\operatorname{sim}\left(\bar{h}_{\mathrm{VFM}}(x^+)_{[n]}, P_\phi\left(h_\theta(z^+)\right)_{[n]}\right). \tag*{(6)}

We apply this alignment loss only to positive (data) samples, and never to MCMC negatives. Two complementary intuitions motivate this choice. From a contrastive learning view, alignment acts as a shaping term on the low-energy side of the landscape, while the objective is simultaneously driving negatives toward high energy. Aligning negatives would therefore pull them back toward the data manifold in feature space, working directly against that push. From a manifold view, the VFM produces meaningful features only for inputs near the natural-data manifold. MCMC negatives are typically off-manifold, and this is most pronounced early in training and at intermediate sampling steps, so their embeddings provide no reliable semantic target to align against.

This positive-only design also clarifies how PRA is both similar and different to representation alignment in diffusion models, where the distinction lies in what each network receives as input. That line of work aligns the features of a noisy input xtx_t with the VFM features of its clean counterpart x0x_0; the clean target is dynamic and guides the denoising trajectory toward the data manifold. In our setting both the EBM and the VFM receive the same clean input x+x^+, so PRA does not guide any sampling trajectory. Instead it shapes the geometry of the low-energy region of the landscape directly, PRA encourages semantically structured representations of positive samples.

Overall Objective. Combining the contrastive energy objective Eq. (4), the sampler matching term Eq. (5), and the representation alignment loss Eq. (6), the full training objective is:

Ltotal=Lcontrast+λ1Lsampler-match+λ2LPRA,(7)\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{contrast}} + \lambda_{1}\mathcal{L}_{\mathrm{sampler\text{-}match}} + \lambda_{2}\mathcal{L}_{\mathrm{PRA}}, \tag*{(7)}

where λ1\lambda_{1} and λ2\lambda_{2} are coefficients for each loss. Each term plays a distinct role: Lcontrast\mathcal{L}_{\mathrm{contrast}} shapes the energy landscape by pushing data samples to low energy and sampled negatives to high energy, LPRA\mathcal{L}_{\mathrm{PRA}} injects semantic structure into the model’s internal representations, and Lsampler-match\mathcal{L}_{\mathrm{sampler\text{-}match}} encourages low-energy, diverse sampler outputs. The full training algorithm is provided in Appendix § 11.

Inference Time Scaling

At inference, we adjust the sampling step size while keeping the injected noise scale fixed:

xt+1=xt−η∇xEθ(xt)+σϵ,ϵ∼N(0,I).(8)x_{t+1} = x_t - \eta\nabla_x E_\theta(x_t) + \sigma\epsilon,\qquad\epsilon\sim\mathcal{N}(0,I). \tag*{(8)}

η\eta controls the balance between energy-driven refinement and stochastic exploration. It provides control over sample quality, without retraining or additional unconditional network evaluations.

Increasing the step size η\eta amplifies the deterministic energy-gradient update relative to the fixed noise. Similarly, extending sampling beyond the training horizon allows further refinement of intermediate samples. Empirically, both adjustments can sharpen individual samples and produce more class-typical images at the cost of mode coverage.

Experiments

Main Results

Unless otherwise noted, all TEM experiments use batch size 256, learning rate 2×10−42 \times10^{-4}, and 50K iterations, training with 40 steps sampling and inference with 60 steps sampling. Main benchmark numbers for TEM are obtained with inference-time scaling but without classifier-free guidance. All conditions within each ablation are controlled. Full training details are provided in Appendix § 9.

Our TEM-XL model achieves a guidance-free FID-50K of 3.82 on ImageNet 256×256 and 8.24 on 512×512 (Table 1). To our knowledge, these are the best reported large-scale results for a contrastively trained EBM using its own MCMC negatives. Evaluation uses persistent sampling initialized from Gaussian noise and outperforms unguided diffusion transformers of comparable size. This brings contrastive EBMs into the performance range of modern large-scale generative models. Although EBMs are often considered expensive, TEM’s training and inference costs are comparable to, or even lower than, those of diffusion transformers of similar size (Appendix C). As shown in Figure 3, TEM scales with model size in both training steps and training compute (left, middle), and scales more steeply than other baselines, overtaking them at the highest budget (right).

Image not extracted

Figure 3. Scaling behavior of TEM on class-conditional ImageNet 256×256. Left: FID-50K as a function of training steps across model sizes. Larger models consistently achieve lower FID at every checkpoint; 40 sampling steps are used for sampling during training. Middle: FID-50K as a function of cumulative training GFLOPs. All four model sizes follow a consistent compute–quality trend (dashed line). Right: FID-50K versus training GFLOPs for TEM-XL compared against DiT-XL, SiT-XL and SiT-XL+REPA showing that TEM tracks, and eventually achieves lower FID than unguided diffusion baselines with less training compute.

Ablations

Analysis of Multi-Head Energy Composition. We validate the use of multiple energy functions through three analyses (Figure 5). The three-head model reaches a lower FID than a single- or

Image not extracted

Figure 5. Ablations on Multi-energy Head and Loss Terms. Left: generated images from different energy heads, where (a), (b), (c) decompose the energy into early, middle, and final heads respectively, and (d) uses the full energy head. Middle: cosine similarity between energy head gradients across Langevin steps. Right: FID-50K at 50K steps for TEM-B. The upper block adds each loss term cumulatively with three heads; the lower block varies the number of energy heads under the full loss, against the three-head configuration above.

Algorithm 1 Sampling Algorithm

1: if ∣B∣=N|\mathcal{B}| = N then ▹B←∅\triangleright\mathcal{B} \leftarrow\varnothing at beginning 2: Sample x0∼Bx_0 \sim\mathcal{B} 3: else 4:   Sample x0∼N(0,I)x_0 \sim\mathcal{N}(0, I) 5: end if 6: for t=1t = 1 to TT do 7:   xt←xt−1−η∇xEθ(xt−1)+σϵx_t \leftarrow x_{t-1} - \eta\nabla_x E_\theta(x_{t-1}) + \sigma\epsilon 8:   if t=40t = 40 then 9:     Add x40x_{40} to replay buffer B\mathcal{B} 10:   end if 11: end for 12: return xTx_T

double-head baseline at 50K steps (right). Fitting three energy functions whose sum must jointly explain the data yields a better energy landscape.

Sampling from individual heads (left) suggests complementary roles. The early head recovers local texture without global structure (a), the middle head captures object layout without fine detail (b), and the final head preserves coarse semantics alone (c), while their sum (d) satisfies all three simultaneously. Gradient cosine similarity between heads (middle) is high early in sampling and decreases in later steps, suggesting a shared initial direction followed by more specialized contributions.

Ablation on Representation Supervision. As shown in Figure 4, PRA lowers FID-50K of TEM-B from 42.8 to 19.9 at 50K steps. Additionally aligning negative samples (Pos+Neg RA) still improves over no alignment but reaches only 36.6, erasing most of this gain. As discussed in § 3.2, negatives provide a wrong alignment target, and aligning them can even be harmful.

Image not extracted

Figure 4. Effect of PRA on training. FID-50K across training steps for TEM-B with no representation alignment, Pos+Neg RA, and PRA. PRA consistently outperforms both, reaching 19.9 FID at 50K steps versus 42.8 without alignment and 36.6 with Pos+Neg RA. This supports aligning only positive samples in our method.

Test-Time Scaling Analysis

We investigate the effect of scaling sampling dynamics at inference time by varying both the number of sampling steps and the step size. Figure 6 visualizes the progressive refinement: at T=40T = 40 (the training horizon), samples contain residual noise, but extended sampling produces increasingly sharp, class-typical images.

Image not extracted

Figure 6. Progressive sample refinement via extended sampling steps. Two generation trajectories (top: snail, bottom: lemon) visualized from T=0T = 0 (pure noise) to T=120T = 120. At the training horizon (T=40T = 40), coarse structure is established but residual noise remains. Extending beyond the training horizon produces increasingly sharp, class-typical images, demonstrating inference-time scaling without retraining.

Figure 7 quantifies this trade-off. Increasing the number of steps from 40 to 120 and scaling the step size from 500 to 1500 yields higher Inception Scores (indicating more recognizable samples) at the cost of slightly elevated FID (indicating reduced diversity). Notably, this FID-IS trade-off curve closely resembles that of classifier-free guidance, despite requiring no architectural modifications or unconditional network evaluations, but rather just inference-time scaling.

Image not extracted

Figure 7. Inference-time scaling pseudocode and quality–diversity trade-off. Left: MCMC sampling procedure of TEM. Chains can run beyond 40 steps, but only the step-40 state is inserted into the replay buffer, so the buffer stays matched to the training dynamics. Right: FID-50K vs. Inception Score for TEM-XL, increasing the number of sampling steps (blue, η=500\eta=500) or the step size (green, T=40T=40) from the training configuration (T=40T=40, η=500\eta=500). Moderately increasing the number of steps improves both metrics (FID 3.82 at 60 steps). Beyond that, IS keeps rising while FID degrades as samples concentrate on high-density modes. This mirrors the fidelity–diversity trade-off controlled by classifier-free guidance, but here it comes from a single network with no retraining.

This controllability lets users tune sampling to the application, with longer, larger-step trajectories for fidelity and shorter, smaller-step ones for diversity.

Applications of TEM

Compositional Generation. Compositional generation arises naturally in the energy-based formulation. Let p1(x)∝exp⁡(−E1(x))p_1(x) \propto\exp(-E_1(x)) and p2(x)∝exp⁡(−E2(x))p_2(x) \propto\exp(-E_2(x)) denote two energy-defined distributions. Their composition is defined as the product distribution pcomp(x)∝p1(x)p2(x)p_{\mathrm{comp}}(x) \propto p_1(x)p_2(x), which can be equivalently written as pcomp(x)∝exp⁡(−(E1(x)+E2(x)))p_{\mathrm{comp}}(x) \propto\exp(-(E_1(x) + E_2(x))). Consequently, compositional sampling is obtained by replacing the original energy with the summed energy E1(x)+E2(x)E_1(x) + E_2(x), enabling joint generation that simultaneously satisfies both energy functions.

We train TEM-B conditioned on object location and generate compositions of multiple objects at multiple specified positions with CLEVR dataset. Figure 9 shows qualitative samples alongside a comparison against Composable Diffusion [25], where we report object detection accuracy for 1-, 2-, and 3-object compositions. Following their evaluation protocol, we train the classifier ourselves using their training recipe. TEM outperforms Composable Diffusion at every composition count. Experimental details are provided in Appendix § 12.

Image not extracted

Figure 9. CLEVR 256×256 compositional generation. Top: conditioning inputs, yellow boxes indicate specified object positions; second row shows TEM-B samples, green boxes indicate detected objects. Bottom: compositional generation accuracy (↑), object detection success rate by a trained classifier over 1000 scenes for 1-, 2-, and 3-object compositions.

Models1 obj.2 obj.3 obj.
Comp. Diff.0.8630.5980.262
TEM (Ours)0.9560.7660.274

Table 2. Out-of-distribution detection on ImageNet benchmarks (AUROC↑). Comparison of DiT-XL/2, SiT-XL/2, SiT-XL/2+REPA and our TEM-XL.

Out-of-Distribution Detection. EBMs naturally provide a density-based score for OOD detection. Eθ(x)E_\theta(x) assigns lower energy to in-distribution data and higher energy to OOD inputs [9]. Thus, a single forward pass of Eθ(x)E_\theta(x) directly provides an OOD score without any task-specific training. In contrast, diffusion models do not directly provide such

OOD DatasetDiTSiTSiT-REPAOurs
iNaturalist0.7770.7330.7800.904
Textures0.4390.4260.4380.687
Places3650.6690.6210.4810.636
SUN0.5050.5130.5000.640
Average0.5980.5730.5490.717

Table 2 (PDF p. 8).

a scalar energy and require an additional scoring procedure based on their score function. For the diffusion baselines, we follow Wyatt et al. [44]. Using the ImageNet-1k validation set as the in-distribution reference, we report AUROC (↑) on four standard benchmarks [19]. Experiment details are in Appendix § 13.

Inpainting. Inpainting is particularly simple in energy-based models. Our EBM defines a single energy over image latents, allowing refinement from any intermediate completion. We mask the input image before VAE encoding, preventing access to ground-truth content in the missing region. We then apply sampling updates to the missing latent region, restoring the observed-region latents to their initial encoded values after each step. Figure 8 shows image composition and inpainting results.

Image not extracted

Figure 8. Application of TEM for Image Composition and Inpainting. Left: composed ImageNet images, car mirror + white shark and border collie + rapeseed (top), strawberry + petri dish and lemon + fountain (bottom). Right: inpainting of a meerkat (top) and a movie theater (bottom) across sampling steps. Both applications require no retraining or architectural modification.

Related Work

Foundations of Energy-Based Models. Energy-based models assign an unnormalized scalar energy to each input and define a Gibbs distribution over the data space, an idea with deep roots in statistical physics and undirected graphical models [17, 23]. Classical instantiations such as Restricted Boltzmann Machines relied on tractable architectures to make learning feasible, which in turn limited their representational capacity. Du & Mordatch [9] showed that short-run Langevin MCMC combined with a replay buffer yields a practical training loop, producing coherent samples on CIFAR-10. Follow-up work refined the training recipe through improved MCMC initialization, sampler matching terms, and regularization strategies [10, 29, 30], yet these methods operated at resolutions of 32×3232 \times32 or less diverse dataset, leaving the scalability of the paradigm unresolved.

Latent-Space Energy-Based Models. Pang et al. [32] proposed training an EBM as a prior over the latent codes of a variational autoencoder, while Hill et al. [16] explored energy-based priors for hierarchical latent variable models. Nie et al. [28] further demonstrated controllable generation by manipulating energies in a learned latent manifold. VAEBM [46] combines a VAE with an EBM refinement in data space, reparameterizing the MCMC chain through the VAE’s latent space to make sampling practical at higher resolutions. In contrast, we train a single scalar output Transformer energy function in the frozen latent space of a pretrained autoencoder [35], keeping the EBM as the sole generative component.

Scaling Energy-Based Models. Several recent efforts have pushed EBMs toward higher resolution and stronger performance, though each introduces auxiliary mechanisms that depart from the pure energy-based formulation. EGC [14] hybridizes energy-based learning with diffusion modeling and can produce 256×256256 \times256 ImageNet samples. However, its sampling procedure relies on the full diffusion denoising trajectory. On the adversarial side, Grathwohl et al. [13] reinterpret classifier logits as energies, and Yin et al. [47] scale this idea to ImageNet 256×256256 \times256. However, their sampler is initialized from an auxiliary image dataset at both training and inference, and is never seeded from noise at this scale.

Trained contrastively, TEM generates high-fidelity 256×256256 \times256 images, enabling inference-time scaling and all EBM native applications. Appendix § 14 provides an extended discussion of related work.

Limitations and Conclusion

Limitations. TEM’s primary limitation is the computational cost of iterative sampling during both training and inference. One promising direction is to train a feed-forward proposal model to generate candidates in low-energy regions, reducing the sampling needed to refine them. Extended sampling also introduces a fidelity–diversity trade-off: longer trajectories sharpen images but can reduce diversity and worsen FID. Adaptive noise schedules and proposals that enable movement between distinct modes could help preserve diversity while improving sample quality.

Conclusion. In this work, we introduce the Transformer Energy-Based Model (TEM), an approach that addresses long-standing scalability issues in contrastive training of energy-based models. With a modern transformer backbone, multi-layer energy extraction, and representation-aware training, we demonstrate that a contrastively trained EBM can synthesize high-fidelity ImageNet images at 256×256256 \times256 and 512×512512 \times512, competitive with other generative models. We also show that the properties that motivate energy-based modeling in the first place, compositional generation, inpainting, and out-of-distribution detection, remain available at this resolution. We hope this work serves as a starting point for further scaling of contrastively trained EBMs.

Acknowledgments

We thank William Moy for helpful discussion and feedback. The authors acknowledge computational support from the Kempner Institute. Jungsoo Lee is supported from Korea Foundation for Advance Studies Fellowship.

The Sampler-Matching Loss

Maximum likelihood training of an EBM minimizes LNLL=EpD[Eθ(x)]+log⁡Z(θ)\mathcal{L}_{\mathrm{NLL}} = \mathbb{E}_{p_D}[E_\theta(x)] + \log Z(\theta), whose gradient is given in Eq. (1): a positive phase over data and a negative phase over samples from pθp_\theta. The negative phase is the obstacle, since exact samples from pθp_\theta require MCMC run to convergence.

In practice we replace pθp_\theta by the distribution qθq_\theta of the final state of a short Langevin chain initialized at N(0,I)\mathcal{N}(0,I) or at a past sample. The resulting gradient,

Lcontrast−only=EpD(x)[Eθ(x)]−Esg(qθ(x′))[Eθ(x′)](9)\mathcal{L}_{\mathrm{contrast-only}} = \mathbb{E}_{p_D(x)}[E_\theta(x)] - \mathbb{E}_{\mathrm{sg}(q_\theta(x'))}[E_\theta(x')] \tag*{(9)}

is therefore biased by the gap between qθq_\theta and pθp_\theta, and the bias is largest exactly when the sampler fails to reach the model’s low-energy regions.

Following Du et al. [10], we introduce a loss motivated by this gap that encourages low-energy, diverse sampler outputs:

LKL=KL⁡(qθ∥psg(θ))+const=Eqθ(x)[Esg(θ)(x)]+Eqθ(x)[log⁡qθ(x)](10)\mathcal{L}_{\mathrm{KL}} = \operatorname{KL}(q_\theta\mathbin{\|} p_{\mathrm{sg}(\theta)}) + \mathrm{const} = \mathbb{E}_{q_\theta(x)}[E_{\mathrm{sg}(\theta)}(x)] + \mathbb{E}_{q_\theta(x)}[\log q_\theta(x)] \tag*{(10)}

where the stop-gradient sg⁡(⋅)\operatorname{sg}(\cdot) freezes the energy parameters so that gradients flow only through the sampler, and the constant −log⁡Z(sg⁡(θ))-\log Z(\operatorname{sg}(\theta)) does not depend on the differentiated parameters. Minimizing LKL\mathcal{L}_{\mathrm{KL}} makes the sampler’s outputs both low-energy and diverse, which is precisely what an accurate negative phase requires. It also counteracts the failure mode in which the energy function shapes a landscape that its own sampler cannot descend.

Estimating the sampler-energy term

Eq. (10) splits into two pieces, LKL=Lopt+Lent\mathcal{L}_{\mathrm{KL}} = \mathcal{L}_{\mathrm{opt}} + \mathcal{L}_{\mathrm{ent}}, with Lopt=Eqθ(x)[Esg(θ)(x)]\mathcal{L}_{\mathrm{opt}} = \mathbb{E}_{q_\theta(x)}[E_{\mathrm{sg}(\theta)}(x)] and Lent=Eqθ(x)[log⁡qθ(x)]=−H(qθ)\mathcal{L}_{\mathrm{ent}} = \mathbb{E}_{q_\theta(x)}[\log q_\theta(x)] = -H(q_\theta).

The first is directly tractable: each sampling step is a differentiable function of θ\theta, so

∂Lopt∂θ=Eqθ(x0,…,xT)[∂Esg(θ)(xT−1−η∇xEθ(xT−1)+ωT)∂θ](11)\frac{\partial\mathcal{L}_{\mathrm{opt}}}{\partial\theta} = \mathbb{E}_{q_\theta(x^0,\ldots,x^T)}\left[\frac{\partial E_{\mathrm{sg}(\theta)}\left(x^{T-1}-\eta\nabla_x E_\theta\left(x^{T-1}\right)+\omega^T\right)}{\partial\theta}\right] \tag*{(11)}

which is a second-order gradient term. Backpropagating through all TT steps is memory-prohibitive at our sequence lengths, so we differentiate only through the final sampling step, following [10].

Estimating the entropy term

The second piece is the obstacle. Evaluating log⁡qθ(x)\log q_\theta(x) exactly is intractable: qθq_\theta is the pushforward of the buffer distribution through TT sampling steps, and its density at a point requires marginalizing over all trajectories that could have produced it.

We instead use the non-parametric nearest-neighbor entropy estimator [2], which estimates H(p)H(p) from nn samples by their average log-distance to the nearest other sample. Up to factors independent of θ\theta, minimizing Lent\mathcal{L}_{\mathrm{ent}} therefore amounts to

Lent=Eqθ(x)[−log⁡NN⁡(x,B)],NN⁡(x,B)=min⁡x′∈B∥x−x′∥2(12)\mathcal{L}_{\mathrm{ent}} = \mathbb{E}_{q_\theta(x)}[-\log\operatorname{NN}(x,\mathcal{B})], \qquad\operatorname{NN}(x,\mathcal{B}) = \min_{x'\in\mathcal{B}} \lVert x-x'\rVert_2 \tag*{(12)}

where B\mathcal{B} is a set of past MCMC samples treated as constants (stop-gradient) and distances are L2L_2 in latent space. Repelling samples from one another increases their diversity. This class of estimators scale poorly with dimension, and the buffer samples are neither independent nor drawn from the current qθq_\theta. As in Du et al. [10], we use it only as a diversity regularizer rather than as an accurate estimate of H(qθ)H(q_\theta).

Training & Inference Details

Training Details.

The energy head consists of a two-layer MLP with SiLU activations. Details of the transformer architecture are provided in Table 3. All experiments operate in the latent space of a pre-trained Stable Diffusion VAE [35], following Peebles & Xie [33] and are trained with bfloat16 mixed precision. The latent space is clipped by norm 100 to stabilize sampling for 256×256256 \times256 and by norm 200 for 512×512512 \times512, which is chosen as the threshold containing over 99.99% of ImageNet training latents, which stabilizes sampling. A smaller threshold clips real data points, while a larger one expands the region the sampler has to search, which slightly degrades performance. For sampling, we use a step size of 500 and noise scale of 0.05. With a small step size, the network would have to produce very large input gradients to move samples meaningfully within a limited number of steps, and we found that this destabilizes training. With an excessively large step size, however, each update is coarse, and the model struggles to learn a fine-grained energy landscape. Following prior work [9, 10, 12], who likewise decouple the step size from the noise scale and use a step size much larger than the standard Langevin prescription, we select the step size empirically to balance these two effects. During training, 40-step sampling is employed.

ModelTEM-STEM-BTEM-LTEM-XL
Params33M122M415M609M
Depth (layers)12122428
Hidden size38476810241152
Attention heads6121616

Table 3. TEM architecture configurations. Model variants from TEM-S to TEM-XL. All variants use a patch size of 2×22 \times2 and operate in the latent space of a pretrained Stable Diffusion VAE. Scaling from S to XL increases both depth and hidden dimension, yielding monotonic FID improvements (see Figure 3).

The replay buffer’s replay probability is set to 0.99 and capacity is 16000. Class labels are drawn independently for every chain, including chains initialized from the buffer. Buffer states therefore serve as class independent initializations near the data manifold, which the conditional energy refines toward the requested class. Once full, each newly added sample replaces a uniformly randomly selected buffer entry. For the sampler matching term, the distance with the nearest neighbor term is calculated based on the distance from 100 randomly sampled seeds to the replay buffer following [10]. We use DINOv2-large [31] as the VFM and implement the projection network PϕP_{\phi} as a lightweight CNN with spatial normalization following Singh et al. [37] attached after the block nearest to one-third of the total network depth. We use λ=1\lambda= 1 for energy L2 regularization, λ1=1\lambda_1 = 1 and λ2=0.6\lambda_2 = 0.6 for overall objective’s coefficient at Equation (7) following Du et al. [10] and Singh et al. [37]. We optimize the model using Adam with a learning rate of 2×10−42 \times10^{-4}, momentum parameters (β1,β2)=(0,0.999)(\beta_1, \beta_2) = (0, 0.999), ϵ=10−8\epsilon= 10^{-8}, and no weight decay. We keep an exponential moving average (EMA) of the weights with decay 0.9999, updated after every optimizer step. All reported samples and metrics use the EMA weights. During training, we replace the class label with a null token with probability 0.15, enabling unconditional energy evaluation for OOD detection.

Inference details.

Evaluation uses persistent chains: the first 5,000 chains are initialized from N(0,I)\mathcal{N}(0, I), and subsequent chains continue from buffered 40-step states (Algorithm 1), so no external data enters sampling. Once full, each newly added 40 step state replaces a uniformly randomly selected buffer entry. Class labels are sampled independently for each chain and are randomly mixed throughout evaluation. We apply the same latent-norm clipping threshold used during training. Each output requires TT additional sampling steps, including for noise-initialized chains, giving an amortized cost of TT steps per sample under this persistent evaluation protocol.

To discuss more about details about fidelity-diversity trade-off at § 4.3, Table 4 directly quantifies this trade-off: increasing sampling steps from 40 to 80 raises precision from 0.65 to 0.81 while reducing recall from 0.65 to 0.51. This trend is similar to the fidelity-diversity trade-off observed with increasing classifier-free guidance in diffusion models.

Model# StepsFID ↓\downarrowIS ↑\uparrowPrecision ↑\uparrowRecall ↑\uparrow
TEM-XL407.63106.00.650.65
TEM-XL603.82176.20.760.57
TEM-XL804.72221.10.810.51
DiT-XL (g=1.0g=1.0)2509.62121.50.670.67
DiT-XL (g=1.25g=1.25)2503.22201.80.760.62
DiT-XL (g=1.5g=1.5)2502.27278.20.830.57

Table 4. Fidelity-diversity trade-off. Top: TEM-XL with varying numbers of sampling steps, precision/recall with k=3k = 3. Bottom: DiT-XL/2 with varying classifier-free guidance scale gg results from Peebles & Xie [33].

Computational Cost

Training the TEM-XL model for 50K steps on ImageNet 256×256256 \times256 dataset takes approximately 6 days using 4 H200 GPUs. For sampling, it takes about an hour to sample 50K images with 1 H200 GPU for 60 sampling steps. Higher resolution training on 512×512512 \times512 takes roughly 3–4 times longer than 256×256256 \times256 on both training and inference.

Training Cost

Training cost. We follow the FLOP accounting convention of Kaplan et al. [22], letting FF denote the cost of a single forward pass. A standard training step for a diffusion transformer costs 3F3F per image: one forward pass and a backward pass of roughly 2F2F, since gradients are taken with respect to both the activations and the parameters. A TEM step has four components:

  • Negative sampling. Each of the Ttrain=40T_{\mathrm{train}}=40 sampling steps requires ∇zEθ(z)\nabla_z E_\theta(z), a gradient with respect to the input only, which costs approximately FF in addition to the forward pass. The chain therefore costs about 2FTtrain=80F2F T_{\mathrm{train}} = 80F.

  • Sampler matching loss. This term calculates the second-order gradient for last sampling step which takes about 8F8F based on measurement.

  • Contrastive loss and parameter update. The energies of the positive and negative batches are each evaluated and backpropagated through, about 3F3F each.

  • Representation alignment. One forward pass of the frozen VFM, which is not backpropagated through. For DINOv2-L at our token count this is approximately 0.9F0.9F.

Despite this per-step overhead, TEM reaches a lower guidance-free FID with less total training compute, because it converges in far fewer steps. Table 5 compares the total training cost of TEM with that of representative models.

ModelTraining stepsTraining FLOPsFID ↓\downarrow
DiT-XL/27M638 E9.62
SiT-XL/27M638 E8.30
SiT-XL/2+REPA4M393 E5.90
TEM-XL50K146 E3.82

Table 5. Training cost on ImageNet 256×256256 \times256. All values use the same accounting convention. VFM forwards are included for TEM and SiT+REPA.

Inference Cost

Because each sampling requires the input gradient ∇zEθ(z)\nabla_z E_\theta(z) at every step, the number of steps is not directly comparable to that of diffusion models, we report FLOPs instead. Following the accounting convention of Kaplan et al. [22], let FF denote the cost of one forward pass. Sampling requires only the gradient with respect to the input, which costs approximately FF, as opposed to roughly 2F2F when parameter gradients are also required. Each sampling step therefore costs about 2F2F, and sampling with KK steps costs 2FK2FK. Table 6 compares the sampling cost of TEM with that of other models.

Although each sampling step costs roughly twice the cost of a single forward pass, TEM uses far fewer steps while reaching a lower guidance-free FID. Classifier-free guidance would improve the baselines’ quality but doubles their sampling cost. Although accelerated solvers can further reduce

ModelSteps / SampleFLOPs / SampleFID ↓
DiT-XL/225029.66 T9.62
SiT-XL/225029.66 T8.30
TEM-L609.87 T4.74
TEM-XL6014.48 T3.82

caption: Sampling cost on ImageNet 256×256. All results are without classifier-free guidance.

:::

Algorithm 2 Transformer Energy-Based Model (TEM) Training

  1. Input: Data distribution pD(x)p_D(\mathbf{x}), encoder E\mathcal{E}, frozen VFM hVFMh_{\mathrm{VFM}}, sampling step size η\eta, noise scale σ\sigma, number of steps TT, loss coefficients λ\lambda, λ1\lambda_1, λ2\lambda_2, replay probability preplayp_{\mathrm{replay}}

  2. Initialize: Replay buffer B←∅\mathcal{B} \leftarrow\emptyset

  3. while not converged do

  4.   # 1. Sample positive data and initialize negative chains

  5.   Sample real images x+∼pD(x)\mathbf{x}^{+} \sim p_D(\mathbf{x}); z+←E(x+)\mathbf{z}^{+} \leftarrow\mathcal{E}(\mathbf{x}^{+})

  6.   Sample z~0∼B\tilde{\mathbf{z}}^{0} \sim\mathcal{B} with probability preplayp_{\mathrm{replay}}, else z~0∼N(0,I)\tilde{\mathbf{z}}^{0} \sim\mathcal{N}(0,I)

  7.   # 2. Sampling

  8.   for t=1t = 1 to TT do

  9.     z~t←z~t−1−η∇zEθ(z~t−1)+ω,ω∼N(0,σ2)\tilde{\mathbf{z}}^{t} \leftarrow\tilde{\mathbf{z}}^{t-1} - \eta\nabla_{\mathbf{z}} E_\theta(\tilde{\mathbf{z}}^{t-1}) + \omega,\quad\omega\sim\mathcal{N}(0,\sigma^2)

  10.   end for

  11.   z^−←z~T;z−←sg⁡(z~T)\hat{\mathbf{z}}^{-} \leftarrow\tilde{\mathbf{z}}^{T};\quad\mathbf{z}^{-} \leftarrow\operatorname{sg}(\tilde{\mathbf{z}}^{T}) ▷ with and without gradient

  12.   # 3. Compute Training Objectives

  13.   L←1N∑(Eθ(z+)−Eθ(z−)+λ(Eθ(z+)2+Eθ(z−)2))\mathcal{L} \leftarrow\frac{1}{N}\sum\left(E_\theta(\mathbf{z}^{+}) - E_\theta(\mathbf{z}^{-}) + \lambda\left(E_\theta(\mathbf{z}^{+})^2 + E_\theta(\mathbf{z}^{-})^2\right)\right)

  14.     +λ1(Esg⁡(θ)(z^−)−log⁡NN⁡(z^−,B))+λ2LPRA(x+,z+)+\lambda_1\left(E_{\operatorname{sg}(\theta)}(\hat{\mathbf{z}}^{-}) - \log\operatorname{NN}(\hat{\mathbf{z}}^{-},\mathcal{B})\right) + \lambda_2\mathcal{L}_{\mathrm{PRA}}(\mathbf{x}^{+},\mathbf{z}^{+})

  15.   # 4. Backpropagation and Parameter Update

  16.   Δθ←∇θL\Delta\theta\leftarrow\nabla_\theta\mathcal{L}

  17.   # 5. Update persistent buffer

  18.   B←B∪{z−}\mathcal{B} \leftarrow\mathcal{B} \cup\{\mathbf{z}^{-}\}

  19. end while

the cost of diffusion sampling, the overhead of computing input gradients is modest in practice, and TEM attains a lower guidance-free FID at a comparable sampling cost.

Training Algorithm

The full training procedure for TEM is given in Algorithm 2.

Compositional Generation in 256x256 CLEVR Dataset

To quantitatively evaluate TEM’s compositional performance, we train TEM on a 256×256 location-conditioned CLEVR [20] dataset consisting of 70,000 images. Prior work on compositional generation evaluates at 128×128 with a classifier trained at that resolution. We extend this setting to 256×256 to test whether the compositional advantage of energy-based models persists at higher resolution, which is a harder generative setting. The following describes the training setup and the evaluation protocol used to compare against Composable Diffusion [25].

Training a location-conditioned TEM on CLEVR 256×256. We train our TEM-B model and evaluate it by providing object locations from real images and composing images according to the number of objects. We condition the energy network on a target object location p=(x,y)∈[0,1]2\mathbf{p}=(x,y)\in[0,1]^2 (normalized image coordinates) through two complementary pathways: a global coordinate code injected by adaptive layer normalization, and a spatial bias added directly to the patch-token grid. To condition TEM on location, we encode each coordinate with a NeRF-style Fourier feature encoding,

γ(c)=[c,sin⁡(20πc),cos⁡(20πc),…,sin⁡(2L−1πc),cos⁡(2L−1πc)],(13)\gamma(c) = [c,\sin(2^{0}\pi c),\cos(2^{0}\pi c),\ldots,\sin(2^{L-1}\pi c),\cos(2^{L-1}\pi c)], \tag*{(13)}

which gives a richer positional signal than raw coordinates [27]. With L=8L=8 frequency bands, (x,y)(x, y) maps to a 2(1+2L)=342(1+2L)=34-dimensional code, which is linearly projected, combined with the (optional) attribute embeddings, and injected into every transformer block via AdaLN. We train for 150k steps with batch size 64.

Evaluation protocol. Composable Diffusion [25] operates at 128×128128 \times128, and its evaluation classifier is trained at this resolution. We use their released 128×128128 \times128 CLEVR checkpoint directly, and train the classifier ourselves following their training recipe with 128×128128 \times128 CLEVR dataset. TEM generates at 256×256256 \times256, and we downsample its samples to 128×128128 \times128 only for scoring, so both methods are judged by the same classifier at the same resolution. The classifier reaches over 99.9% detection accuracy on real 128×128128 \times128 CLEVR images, so the reported accuracies reflect generation quality rather than detector error. We report detection success rate over 1,000 scenes for 1-, 2-, and 3-object compositions.

Out-of-Distribution Detection

TEM as an OOD Detector. TEM provides a scalar energy that we evaluate as an OOD score, using lower energy to indicate more ID-like inputs. We evaluate this property as a secondary capability of TEM. Specifically, we use a TEM-XL model trained with class-label dropout: during training, the class condition is randomly replaced with a null token, allowing the model to learn an unconditional energy function Eθ(x)E_\theta(x) alongside the conditional one. At test time, we query the unconditional branch and use the scalar energy S(x)=−Eθ(x)S(x)=-E_\theta(x) as the OOD score in a single forward pass, with no class labels, post-hoc calibration, or OOD exposure.

Diffusion Baselines. As baselines we use pretrained DiT-XL/2 [33], SiT-XL/2 [26], and SiT-XL/2+REPA [48] as ImageNet-trained generative baselines. We use the official released checkpoints: 7M training iterations for DiT-XL/2 and SiT-XL/2, and 4M for SiT-XL/2+REPA. Each test image xx is encoded to a latent z0=E(x)z_0=\mathcal{E}(x) by the Stable Diffusion VAE. For DiT, we construct zt=αˉtz0+1−αˉtϵz_t=\sqrt{\bar{\alpha}_t}z_0+\sqrt{1-\bar{\alpha}_t}\epsilon with ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I), and the network is queried under the null class token y=∅y=\varnothing (the learned embedding at index 1000, used for classifier-free guidance dropout during training). The OOD score is the denoising residual

s(x)=∥ϵ−ϵθ(zt,t,∅)∥2,(14)s(x)=\left\lVert\epsilon-\epsilon_\theta(z_t,t,\varnothing)\right\rVert^2, \tag*{(14)}

where lower s(x)s(x) indicates a more in-distribution input, following Wyatt et al. [44]. We score DiT using its noise-prediction residual and SiT-based models using their velocity-prediction residuals, with each model’s native interpolation and time convention. All baselines use a single network evaluation, matching the single-forward-pass cost of TEM. For DiT, we use timestep t=350t=350. For SiT+REPA, we select the timestep that matches the same noise level.

We include SiT-XL/2+REPA to separate two possible explanations for TEM’s performance. Since TEM is trained with PRA, its energy function inherits semantic structure from VFM, and one might attribute the gain to that supervision rather than to the energy score itself. SiT-XL/2+REPA receives the same DINOv2 alignment signal but is scored by its denoising residual, and its AUROC is close to that of SiT-XL/2 (Table 2). Alignment to a vision foundation model therefore does not by itself account for the gap. What distinguishes TEM is that the energy provides a single scalar that directly ranks inputs by fit to the learned distribution.

Evaluation Protocol. Following Huang & Li [19], we use the ImageNet-1k validation set (50,000 images) as the in-distribution reference and evaluate on four OOD benchmarks spanning a range of semantic distances from ImageNet: iNaturalist [41], SUN [45], Places365 [49], and Textures [5]. We report AUROC (↑).

Additional related work

Baselines in Table 1

All baseline numbers are taken from the original papers; a dash indicates that the corresponding setting was not reported. Among GANs we compare against BigGAN-deep [3] and GigaGAN [21]. For autoregressive and masked models we report MaskGIT [4] and VAR-d30 [40]. The diffusion and flow baselines are ADM [7], LDM-4 [35], DiT-XL/2 [33], SiT-XL/2 [26], and SiT-XL/2 with REPA [48]. Parameter counts refer to the 256×256256 \times256 configuration unless a resolution-specific model was released.

Scaling Energy-based models

A parallel line of work attains scale by learning energy landscapes from predictive targets rather than from negative samples drawn from the model itself. Du et al. [11] assign a scalar energy to each diffusion timestep via denoising score matching, recovering MCMC-based composition but forgoing a single global energy. Equilibrium Matching [42] fits a time-invariant equilibrium gradient through a regression objective, defining the energy only implicitly through its gradient field. Its explicit scalar-energy variants (EqM-E) substantially degrade generation quality (FID 73.40 and 75.53 in the B/4 setting). Energy-Based Transformers [12] learn an energy landscape by backpropagating a reconstruction loss through unrolled inner-loop optimization. Energy Matching [1] explicitly parameterizes a scalar potential, but achieves its high-fidelity generation results by combining an optimal-transport objective with contrastive energy learning. Its image generation experiments are limited to 32×3232 \times32 resolution (CIFAR-10 and unconditional ImageNet 32×3232 \times32), so it is not included in Table 1.

In contrast, TEM directly parameterizes a single scalar energy and trains it contrastively using only negatives produced by its own MCMC sampler, without a transport or flow-matching objective. Thus, while prior work demonstrates that energy parameterizations can scale when coupled with predictive, transport-based, or reconstruction-style objectives, our results show that an explicit, contrastively trained EBM can itself scale to ImageNet 256×256256 \times256.

References

  1. [1]Michal Balcerak, Tamaz Amiranashvili, Antonio Terpin, Suprosanna Shit, Lea Bogensperger, Sebastian Kaltenbach, Petros Koumoutsakos, and Bjoern Menze. Energy matching: Unifying flow matching and energy-based models for generative modeling. Advances in Neural Information Processing Systems, 38:8583–8609, 2025. 19arxiv.org/abs/2504.10612
  2. [2]Jan Beirlant, Edward J Dudewicz, László Györfi, Edward C Van der Meulen, et al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6(1):17–39, 1997. 4, 14
  3. [3]Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=B1xsqj09Fm. 18
  4. [4]Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11315–11325, 2022. 18arxiv.org/abs/2202.04200
  5. [5]Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606–3613, 2014. 18arxiv.org/abs/1311.3618
  6. [6]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009. 2
  7. [7]Prafullla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021. 19
  8. [8]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mustafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3arxiv.org/abs/2010.11929
  9. [9]Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in neural information processing systems, 32, 2019. 1, 2, 3, 4, 8, 9, 15arxiv.org/abs/1903.08689
  10. [10]Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316, 2020. 1, 3, 4, 9, 14, 15arxiv.org/abs/2012.01316
  11. [11]Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. In International conference on machine learning, pp. 8489–8510. PMLR, 2023. 1, 19arxiv.org/abs/2302.11552
  12. [12]Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, and Tariq Iqbal. Energy-based transformers are scalable learners and thinkers. arXiv preprint arXiv:2507.02092, 2025. 15, 19arxiv.org/abs/2507.02092
  13. [13]Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03263, 2019. 9arxiv.org/abs/1912.03263
  14. [14]Qiushan Guo, Chuofan Ma, Yi Jiang, Zehuan Yuan, Yizhou Yu, and Ping Luo. Egc: Image generation and classification via a diffusion energy-based model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22952–22962, 2023. 9arxiv.org/abs/2304.02012
  15. [15]Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4246–4253, 2020. 3arxiv.org/abs/2010.04245
  16. [16]Mitch Hill, Erik Nijkamp, Jonathan Mitchell, Bo Pang, and Song-Chun Zhu. Learning probabilistic models from generator latent spaces with hat ebm. Advances in Neural Information Processing Systems, 35:928–940, 2022. 9arxiv.org/abs/2210.16486
  17. [17]Geoffrey Hinton, Simon Osindero, Max Welling, and Yee-Whye Teh. Unsupervised discovery of nonlinear structure using contrastive backpropagation. Cognitive science, 30(4):725–731, 2006.DOI
  18. [18]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.arxiv.org/abs/2006.11239
  19. [19]Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8710–8719, 2021.arxiv.org/abs/2105.01879
  20. [20]Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901–2910, 2017.arxiv.org/abs/1612.06890
  21. [21]Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10124–10134, 2023.arxiv.org/abs/2303.05511
  22. [22]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.arxiv.org/abs/2001.08361
  23. [23]Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, et al. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
  24. [24]Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.arxiv.org/abs/2210.02747
  25. [25]Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European conference on computer vision, pp. 423–439. Springer, 2022.arxiv.org/abs/2206.01714
  26. [26]Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23–40. Springer, 2024.arxiv.org/abs/2401.08740
  27. [27]Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.arxiv.org/abs/2003.08934
  28. [28]Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent-space energy-based models. Advances in Neural Information Processing Systems, 34:13497–13510, 2021.
  29. [29]Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcmc toward energy-based model. Advances in Neural Information Processing Systems, 32, 2019.arxiv.org/abs/1904.09770
  30. [30]Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of mcmc-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5272–5280, 2020.arxiv.org/abs/1903.12370
  31. [31]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
  32. [32]Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. Advances in Neural Information Processing Systems, 33:21994–22008, 2020.arxiv.org/abs/2006.08205
  33. [33]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195–4205, 2023. 3, 4, 15, 16, 18, 19arxiv.org/abs/2212.09748
  34. [34]Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitsky. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:12116–12128, 2021. 3arxiv.org/abs/2108.08810
  35. [35]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684–10695, 2022. 4, 9, 15, 19arxiv.org/abs/2112.10752
  36. [36]Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. 3arxiv.org/abs/2002.05202
  37. [37]Jaskirat Singh, Xingjian Leng, Zongze Wu, Liang Zheng, Richard Zhang, Eli Shechtman, and Saining Xie. What matters for representation alignment: Global information or spatial structure? arXiv preprint arXiv:2512.10794, 2025. 15arxiv.org/abs/2512.10794
  38. [38]Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256–2265. pmlr, 2015. 1
  39. [39]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. 3
  40. [40]Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839–84865, 2024. 18
  41. [41]Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shephard, Hartwig Adam, Pietro Perona, and Serge Belongie. The naturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769–8778, 2018. 18
  42. [42]Runqian Wang and Yilun Du. Equilibrium matching: Generative modeling with implicit energy-based models. arXiv preprint arXiv:2510.02300, 2025. 19
  43. [43]Runqian Wang and Kaiming He. Diffuse and disperse: Image generation with representation regularization. arXiv preprint arXiv:2506.09027, 2025. 4
  44. [44]Julian Wyatt, Adam Leach, Sebastian M Schmon, and Chris G Willcocks. Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 649–655. IEEE, 2022. 9, 18
  45. [45]Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485–3492. IEEE, 2010. 18
  46. [46]Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy-based models. arXiv preprint arXiv:2010.00654, 2020. 9
  47. [47]Xuwang Yin, Claire Zhang, Julie Steele, Nir Shavit, and Tony Wang. Scalable energy-based models via adversarial training: Unifying discrimination and generation. In International Conference on Learning Representations, volume 2026, pp. 68385–68424, 2026. 9
  48. [48]Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940, 2024. 4, 18, 19arxiv.org/abs/2410.06940
  49. [49]Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017. 18

Paper details

vlnc:W_yzudtw2y
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents