WorksComputer Science
Self-Supervised Collapse Prevention with Score-Based Entropy Maximization
Abstract
Diffusion models have demonstrated the effectiveness of score-based objectives for generative modeling, yet their potential for self-supervised representation learning remains underexplored. We introduce Entropy-Based Regularization (EBReg), a framework that uses latent score estimation to regularize the latent distribution during self-supervised training. EBReg jointly learns an encoder and a latent score model, and utilizes the learned score to provide a diversity-preserving signal that prevents representational collapse. Motivated by a mutual-information lower bound, we formulate the objective as view alignment with smoothed marginal entropy maximization and optimize the marginal entropy without explicit density evaluation. The same score model can additionally be used at test time to refine corrupted representations toward the training latent distribution. Empirically, EBReg learns competitive representations on ImageNet-1K, achieving 75.57% linear-probe accuracy with ViT-L. Across diverse architectures, score-guided test-time adaptation further improves robustness on ImageNet-C, yielding an average relative accuracy gain of approximately 5%.
Introduction
Diffusion models ([35, 36, 25]) have established score estimation as a central component of modern generative modeling. However, representations learned through standard diffusion objectives often remain weaker than those produced by dedicated self-supervised learning (SSL) methods on recognition benchmarks ([44, 15]). Conversely, recent work shows that strong pretrained representations obtained from self-supervised learning can substantially improve diffusion training and generation, as demonstrated by REPA (Yu et al., [44] and Representation Autoencoders ([45]). Earlier work on denoising autoencoders also demonstrated that denoising objectives can improve representation learning ([39]). Together, these observations motivate the question: Can a score estimator and a self-supervised encoder be learned jointly in an end-to-end manner?
A central challenge in SSL is to enforce invariance across augmented views while maintaining sufficient diversity in the latent distribution to avoid collapse. Existing methods achieve this through different mechanisms, including negative-pair interactions ([14]), moment-based constraints ([8]), and regularization toward prescribed distributions ((Balistera & Le-Cun, 2025)). Despite their different forms, these mechanisms preserve representation diversity while enforcing invariance. We investigate whether an analogous anti-collapse signal can be obtained from a latent score estimated via score matching.
We propose Entropy-Based Regularization (EBReg), a self-supervised framework that jointly trains an encoder and a latent score model. EBReg uses the score of the evolving latent distribution as an adaptive entropy-promoting signal. Figure 1 illustrates this mechanism directly: samples concentrated along a low-dimensional structure are driven by the negative score toward a higher-entropy configuration that occupies both dimensions. Motivated by a variational mutual-information lower bound [7], the objective combines view alignment with maximizing the entropy of a Gaussian-smoothed latent distribution under a scale constraint. Since the latent density is unknown, EBReg estimates the score of the smoothed distribution via denoising score matching and uses it to optimize a gradient surrogate of marginal entropy without explicit density evaluation. The resulting encoder update promotes diversity in the latent distribution and counteracts representational collapse.
Figure 1. Illustration of the EBReg anti-collapse mechanism. Top: (a) Representations lie near a one-dimensional subspace of . (b) EBReg alone spreads them across both dimensions. (c) Alignment alone contracts them, resulting in dimensional collapse. (d) Alignment + EBReg yields compact, separated classes while preventing dimensional collapse. Bottom: Evolution of the latent distribution under EBReg. Samples are moved away from the diagonal and populate both dimensions, ultimately forming an isotropic Gaussian distribution.
We further connect the EBReg formulation to existing anti-collapse regularizers [4, 14] while replacing explicit negative pairs or fixed distribution matching with an adaptive score field. We characterize how latent scale constraints induce maximum-entropy solutions and relate these solutions to distributional targets and moment conditions underlying existing SSL regularizers. We further analyze the exact-score dynamics near dimensional collapse, including the behavior of low-variance directions. The learned score additionally enables test-time latent refinement, moving corrupted representations toward higher-density regions of the latent distribution while adapting only normalization parameters.
Empirically, EBReg learns competitive representations on ImageNet-1K [16] across convolutional and transformer backbones, performing comparably to established self-supervised methods including VICReg [8] and LeJEPA [4] under the same pretraining and evaluation setup. Score-guided test-time refinement further improves robustness on ImageNet-C [23] across diverse architectures.
Our main contributions are summarized as follows:
Method. We derive an alignment–entropy objective from a variational mutual-information lower bound and introduce EBReg, which uses a learned latent score as an anti-collapse regularizer by optimizing representation entropy without explicit density evaluation or negative pairs.
Theoretical analysis. We analyze the exact-score dynamics near dimensional collapse and characterize when EBReg amplifies low-variance directions. We further show how latent constraints determine maximum-entropy solutions that recover distributional targets and moment conditions underlying existing SSL regularizers.
Score-guided test-time adaptation. We reuse the learned score as a label-free adaptation signal, updating only encoder normalization parameters to guide corrupted representations toward higher-density regions of the source latent distribution and improve robustness on ImageNet-C.
Related Work
Latent Space Information Maximization. I-MAX [9], an early precursor to joint-embedding methods, maximizes mutual information between module outputs for related inputs. Early tractable formulations use stochastic binary outputs or Gaussian assumptions for scalar continuous outputs. Predictability Maximization [33] instead combines agreement between paired representations with a discrimination loss that discourages collapse but does not in general yield a mutual-information objective. EBReg revisits the information-maximization principle through an alignment–entropy objective derived from an MI lower bound (Section 3.1). Unlike these earlier formulations, EBReg uses a learned score to optimize marginal entropy without explicit density or mutual-information estimation.
Contrastive methods. Bromley et al. (1993) introduced Siamese networks, which use shared-weight encoders to learn representations by distinguishing matching from non-matching pairs. Building on this principle, modern contrastive methods [14, 21, 31] make use of negative examples to provide a non-parametric repulsive signal that prevents representational collapse. Random negative sampling, however, can yield many uninformative pairs in high-dimensional embedding spaces [42], resulting in dimensional collapse [26]. Prototype-based methods such as SwAV, DINO, and MSN [11, 12, 1] avoid instance-level negatives by balancing assignments over a finite set of learned prototypes, which imposes an explicit discretization of the latent structure. EBReg instead learns a parametric score field to provide entropy-promoting directions without explicit negative comparisons. This design draws on the empirical success of score-based generative models ([37, 32]) and theoretical evidence that score estimation can adapt to low-dimensional subspace structure, mitigating the curse of high dimensionality [13].
Non-contrastive methods. BYOL [19] helped revive non-contrastive learning through asymmetric branches and stop-gradient operations. Subsequent teacher–student distillation methods [2, 29, 30] learn strong representations, with collapse avoidance governed by interactions among architecture, distillation, and optimization. A complementary approach explicitly regularizes representation statistics: VICReg [8] enforces variance floors and penalizes cross-dimensional covariance, while LeJEPA [4] targets an isotropic Gaussian, with related distributional approaches in VISReg [43] and Rectified LpJEPA [27]. EBReg instead maximizes smoothed marginal entropy subject to a scale constraint, allowing the entropy objective and constraint to determine the preferred distribution. EBReg under different scale constraints yields maximum-entropy targets that satisfy moment conditions used by existing methods (Table 1).
| Latent constraint | Typical realization | Max.-entropy target | SSL connection |
| , | Batch normalization without affine parameters | Isotropic Gaussian | SIGReg: Gaussian target; VICReg: corresponding first- and second-order moments |
| , RMS, or layer normalization | Uniform on the sphere | Contrastive learning | |
| Discrete prototype / code assignment | Uniform code usage | SwAV and MSN: balanced prototype usage; | |
| , | -th moment scaling with support constraint | Generalized Gaussian (truncated to ) | Rectified LpJEPA |
Table 1. Maximum-entropy targets under different latent constraints. Each constraint induces a corresponding maximum-entropy target linked to existing SSL regularizers. First three targets maximize the smoothed entropy for any . The last holds as (Appendix § 14).
Reconstruction based methods. Another prominent representation-learning paradigm uses reconstruction objectives [24, 39], which encourage representations to retain information about the input. More recently, masked autoencoders have achieved strong performance by reconstructing missing input regions [22, 5]. Reconstruction naturally discourages collapse and can be interpreted as promoting high mutual information between inputs and their representations. However, input-space objectives may prioritize low-level reconstruction over downstream semantics [3]. Chen et al. (2025) connect denoising diffusion models to denoising autoencoders and propose -DAE, which corrupts low-dimensional latent features but reconstructs the original input in pixel space using a ViT-based decoder. In contrast, EBReg operates entirely in representation space, using a score function to regularize latent representations directly. This avoids pixel-level reconstruction and decouples collapse prevention from input-space fidelity.
Methodology
Our goal is to show that score matching itself can provide a viable objective for learning strong self-supervised representations. To this end, we first derive a mutual-information lower bound and introduce the score-based EBReg surrogate for optimizing marginal entropy (Section 3.1). We then describe alternating score fitting, encoder learning, and score-guided test-time adaptation (§ 3.2), with an overview in Figure 2 and training pseudocode in Appendix § 12. Finally, we theoretically characterize the exact-score dynamics in collapsed and low-variance directions (§ 3.4).
Figure 2. Overview of EBReg: training and test-time adaptation. (a) Two augmented views are encoded into normalized embeddings. The encoder minimizes a view-alignment loss and an entropy regularizer, , using a detached estimate of the score of the noise-smoothed embedding distribution. (b) The latent score is parameterized by a noise-prediction network fitted on detached embeddings, with updates per encoder step; it predicts the injected Gaussian noise, giving the score estimate . (c) At test time, the learned score is reused as a test-time adaptation signal. The score model and remaining encoder–projector parameters are frozen, while only normalization parameters are updated by minimizing , guiding corrupted embeddings toward higher-density regions of the learned source distribution.
From mutual information to score-based entropy regularization
EBReg combines alignment between augmented views with maximization of smoothed marginal entropy. We motivate this objective through the mutual information between representations of two augmented views. Maximizing mutual information favors representations that remain predictive across views while retaining variability in their marginal distribution. This leads naturally to an alignment–entropy formulation, where alignment promotes cross-view agreement and marginal entropy preserves representation diversity and prevents collapse. EBReg then uses a learned score to optimize the smoothed marginal entropy without explicit density evaluation.
An alignment–entropy lower bound. Given an input , we draw two transformations , independently from an augmentation distribution . A shared encoder and projector produce embeddings and in . Both views share the marginal distribution . Embeddings may occupy a lower-dimensional support and lack a density in the full embedding space, making their differential entropy ill-defined. Gaussian smoothing yields a full-dimensional density, allowing us to work with a well-defined smoothed entropy. For a fixed noise scale , we define
where is independent of both views. We write , where denotes differential entropy, and for the smoothed density.
To relate the smoothed representation to the mutual-information objective between the two clean views, consider . Since is obtained from by adding independent Gaussian noise, forms a Markov chain. The data processing inequality therefore gives . Combining this inequality with the variational bound of Barber & Agakov [6] applied to the smoothed pair yields
for any variational conditional density . We choose with fixed , yielding alignment loss. Choosing a Gaussian variational density does not require either the embedding distribution or the true conditional distribution to be Gaussian. Its cross-entropy is
where is the normalized alignment loss [41] and is independent of . See Appendix § 9 for the full derivation. Substituting this cross-entropy expression into Equation (2) yields the alignment–entropy lower bound
Here, denotes an embedding drawn from the shared marginal distribution of the two views. For fixed and , maximizing this lower bound is equivalent to minimizing , where . The objective therefore balances agreement between paired views with entropy of their shared marginal distribution. Increasing latent scale can increase entropy without enriching the representation. We therefore constrain latent scale to prevent entropy maximization through rescaling alone, with the choice of constraint discussed in § 3.1.
We now introduce a tractable gradient surrogate for optimizing the lower bound in Equation (4). The alignment term is computed directly from paired embeddings, but the smoothed entropy depends on the unknown density , which is generally intractable to evaluate. Rather than evaluating directly, EBReg estimates its score, which points toward locally higher-density regions. We therefore replace the negative entropy term with the EBReg gradient surrogate and train the encoder on
Here, and denotes the score estimate. The stop-gradient treats the score as fixed when differentiating the surrogate with respect to the encoder and projector. § 3.2 describes the alternating score-fitting and encoder updates.
We now show that, with an exact score, this surrogate implements entropy maximization at the gradient level. The entropy-gradient identity for implicit distributions [28, 34] gives, under standard regularity conditions,
where is the embedding Jacobian. Detaching the entire score vector in Equation 5 gives . Thus, with an exact score, the EBReg update implements gradient descent on without evaluating the density or entropy. For , the resulting representation-parameter gradient equals that of the negative MI lower bound up to the positive factor . With a learned score, this equivalence is approximate and the surrogate’s value is neither an entropy estimate nor an MI bound.
Intuitively, alignment pulls paired embeddings together, while EBReg encourages the embedding distribution to spread out. The learned score approximates the local direction of increasing density in the noise-smoothed embedding distribution. Minimizing the surrogate encourages embeddings to move against the score, toward locally lower-density regions, thereby producing an entropy-promoting encoder update.
Connections to existing regularizers. EBReg’s scale constraint determines its entropy-maximizing target. Under zero mean and unit marginal variances, the maximizer is the isotropic Gaussian targeted by SIGReg [4], whose moments coincide with the variance–covariance conditions encouraged by VICReg [8]. More generally, different scale constraints induce different maximum-entropy targets while leaving the EBReg formulation unchanged. Table 1 relates these targets to existing SSL regularizers. However, sharing the same target does not imply equivalence of the corresponding objectives. Appendix § 14 provides the derivations.
Alternating score fitting and encoder learning
Since the score in Equation (5) depends on the encoder’s evolving embedding distribution, EBReg alternates between fitting the score model to the current embeddings and updating the encoder with the total objective. Figure 2 summarizes these two phases. Algorithm 1 in Appendix § 12 provides the complete training procedure. Embedding normalization enforces the chosen scale constraint before Gaussian noise is added.
Fitting the latent score. We parameterize a noise-prediction network and recover the latent score from its output via Tweedie’s formula. For each minibatch, we pool embeddings from both augmented views and stop gradients through them, . With the encoder and projector held fixed, we take updates of on the denoising objective
using fresh for each update and the same noise level as in § 3.1. At the unrestricted population optimum, the predictor equals , whose negative, scaled by is the score of [38]. Finite network capacity and a finite number of updates yield an approximation to this score as the embedding distribution evolves.
Updating the encoder. We recompute embeddings for the same paired views with gradients enabled, draw fresh noise, and update using Equation (5). The alignment loss acts on clean paired embeddings while the entropy regularizer uses the score evaluated at their noisy counterparts. We detach the entire score output so that its value supplies the update direction while gradients flow through to the encoder and projector. The score is fixed during the encoder update, so score fitting updates only , and the encoder step updates only .
Score-guided test-time adaptation
Beyond its role in EBReg training, the learned score provides the direction of increasing log-density under the source latent distribution. This makes it directly useful for test-time adaptation, since corrupted inputs may produce representations that move away from high-density source regions and the score provides a direction back toward them. Let denote the encoder’s normalization parameters and the resulting test representation. We keep the score model, classifier, and remaining encoder parameters fixed, and adapt only , following Wang et al. [40].
We define the score-guided adaptation objective as
The normalization parameters are updated for steps using gradients of , while the score model remains fixed. This objective reverses the role of the score used during EBReg training. During pretraining, the negative-score contribution promotes entropy by spreading representations away from high-density regions. At test time, the positive-score direction instead encourages corrupted representations toward higher-density regions of the source latent distribution. Thus, the score learned for representation regularization also provides a label-free signal for adaptation under distribution shift. Section 4.3 evaluates this procedure under input corruptions.
Collapse Dynamics and Connection to SSL Regularization
We analyze EBReg near representational collapse and characterize how latent scale constraints determine maximum-entropy solutions related to existing SSL regularizers.
Instability under dimensional collapse. If the embeddings lie in a lower-dimensional subspace of , evaluating the smoothed score on the clean embeddings yields no component along collapsed directions and therefore cannot recover them. Evaluating the score at noise-perturbed embeddings introduces a nonzero component in these directions. The following theorem formalizes this behavior and shows that directions with small but nonzero variance are amplified.
Intuition. Part (a) explains why the score in Equation (5) is evaluated at rather than at . At a clean embedding, the score vanishes along any collapsed direction, regardless of the distribution within the occupied subspace. It also shows that exact collapse is a stationary point of the EBReg objective, with nonzero sample-wise updates along collapsed directions but zero mean drift. Part (b) shows that this stationary point is unstable, since the right-hand side of Equation (10) defines a variance-ascent direction and descending therefore amplifies any nonzero variance along . Because the update scales with the variance already present rather than introducing a fixed increment, the relative amplification in the low-variance regime is governed by (Appendix B). It is also not opposed by the alignment term, which penalizes disagreement between views while remaining invariant to variation shared across paired views. This theorem characterizes directional gradient contributions under an exact score and, in part (b), an independent Gaussian-coordinate assumption; it does not guarantee recovery from exact collapse or variance growth under the full training dynamics.
Theorem 1 (Instability of collapsed directions under EBReg). Let have finite second moment and fix a unit direction . Assume an exact score and the Gaussian perturbation above, with noise independent of the input and augmentation, and write for the parameter gradient of , with .
(a) The clean score is blind to dimensional collapse. If , then almost surely
so the per-sample gradient through , , is nonzero but has zero mean.
(b) Amplification. If with and independent of , then
Experiments
We compare EBReg with existing self-supervised methods, matching parameter count, data augmentations, and training epochs unless otherwise specified. We evaluate representation quality, collapse prevention, hyperparameter sensitivity, computational overhead, and test-time adaptation under input corruptions.
Linear evaluation
We evaluate the effectiveness of EBReg across multiple architectures and self-supervised objectives on ImageNet-1K (DeLong et al., 2009). We consider ResNet-50 [20], ViT-Large, and ViT-Huge [17] under three established pretraining protocols. For ResNet-50, we adopt the setup from Bardes et al. [8], including its data augmentations, optimization settings, and training schedule, and replace the original variance–covariance regularization with EBReg. For ViT-Large, we follow the setup from Balestriero & LeCun [4] and replace SIGReg with EBReg while keeping the remaining pretraining and evaluation protocol unchanged. Finally, we evaluate EBReg with ViT-Huge in the setup from Assran et al. [2], where EBReg is added to the existing objective rather than replacing it. This setting assesses whether score-based entropy regularization can provide a complementary signal to an existing self-supervised objective. Within each setting, EBReg and the corresponding baseline use the same backbone, augmentations, training schedule, and evaluation procedure. The results are summarized in Table 2.
| Backbone | Method / Regularizer | Epochs | Linear probe (%) ↑ | Eff. rank ↑ |
| ResNet-50 | Alignment only () | 100 | Collapse | – |
| ResNet-50 | VICReg | 100 | 68.02 | 1368 |
| ResNet-50 | SIGReg | 100 | 66.42 | 1256 |
| ResNet-50 | EBReg (ours) | 100 | 67.92 | 1431 |
| ViT-L | SIGReg | 100 | 75.32 | 648 |
| ViT-L | -DAE | 1600 | 75.00 | – |
| ViT-L | EBReg (ours) | 100 | 75.57 | 759 |
| ViT-H | I-JEPA | 100 | 79.03 | 540 |
| ViT-H | I-JEPA + StoP | 100 | 79.33 | 527 |
| ViT-H | I-JEPA + StoP + EBReg (ours) | 100 | 79.41 | 957 |
Table 2. Linear-probe performance and effective rank on ImageNet-1K. Results are grouped by pretraining setup. ResNet-50, ViT-L, and ViT-H use the setups from Bardes et al. [8], Balestriero & LeCun [4], and Assran et al. [2], respectively. Within each setup, EBReg and the corresponding baselines use the same backbone, data augmentations, optimization settings, training schedule, and evaluation procedure. Effective rank (Garrido et al. [18]) is computed on backbone representations after pretraining, with higher values indicating less dimensional collapse. Bold indicates higher accuracy within each setup, and underline indicates the second-best.
Table 2 shows that EBReg remains competitive in linear evaluation while consistently producing higher-rank representations across architectures. On ResNet-50, EBReg matches the performance of strong regularization-based baselines while yielding the highest effective rank. On ViT-L, it improves both linear-probe accuracy and representation rank relative to SIGReg.When added to StoP on ViT-H, EBReg yields a modest improvement in linear-probe accuracy while substantially increasing effective rank, suggesting that it complements the existing objective while preserving a richer representation space.
Hyperparameter Ablations and Computational Cost
Hyperparameter ablations. We ablate hyperparameter under the setup from Bardes et al. [8] and report ImageNet-1K linear-probe accuracy in Table 3. Among the tested values, the default configuration , , and yields the highest accuracy. Performance is most sensitive to the noise scale and the number of score updates, while projector dimension has comparatively little effect over the tested range. Increasing score-network width beyond the default also degrades performance, indicating that greater score-model capacity or more frequent score fitting does not necessarily improve representation quality.
| Factor | Values | Accuracy (%) |
| Regularization weight | 0.25 / 0.34 / 0.5 | 64.28 67.92 63.38 |
| Noise level | 4.25 / 4.5 / 5.0 | 62.32 67.92 65.43 |
| Score updates | 4 / 16 / 32 | 63.72 67.92 58.32 |
| Score-network width | 1024 / 2048 / 4096 | 67.65 67.92 61.35 |
| width | 0.1 / 0.5 / 0.7 | 67.51 67.92 67.43 |
| Projector dimension | 64 / 256 / 512 | 67.42 67.92 67.51 |
Table 3. ResNet-50 sensitivity: ImageNet-1K linear-probe accuracy, varying one factor at a time. Defaults are underlined.
Computational cost. Table 4 reports the computational cost of EBReg under the setup from Balestriero & LeCun [4] using identical hardware. For a backbone with approximately 300M parameters, the score network adds only 5.21M additional parameters, under 2% of the backbone parameter count. The primary overhead arises from repeatedly updating the score model to track the evolving latent distribution. Relative to SIGReg, this increases wall-clock training time by approximately 60 seconds per epoch, from 440 to 500 seconds, corresponding to a 13.6% increase.
| Metric | SIGReg | EBReg |
| Extra params (M) | 0 | 5.21 |
| Time / epoch (s) | 440 | 500 |
| Peak memory (GB) | 90.33 | 89.84 |
| GPU-hours | 195.83 | 224.01 |
Table 4. ViT-L training costs on identical hardware. Extra parameters are relative to the backbone and projector.
The additional computation remains moderate because score fitting is performed entirely in the projected latent space. Its cost is therefore determined primarily by the latent dimension, score-network architecture, and number of score updates, rather than by the image resolution or backbone size. The strong performance obtained with a compact score network may indicate that the latent distribution admits a sufficiently structured score field that can be approximated with limited capacity, although we do not directly characterize its intrinsic dimensionality.
Score-guided test-time adaptation
We evaluate score-guided test-time adaptation on ImageNet-C ([23]) using the EBReg-trained models and pretraining setups from § 4.1. In each case, EBReg training yields a corresponding latent score model, which we reuse at test time as the adaptation signal. The score model and classifier remain fixed, while only the encoder’s normalization parameters are updated using the objective from § 3.3. Table 5 reports accuracy before and after five adaptation steps.
| Corruption | ViT-L Setup from Balestriero & LeCun [4] Before | ViT-L Setup from Balestriero & LeCun [4] After | ViT-H Setup from Assran et al. [2] Before | ViT-H Setup from Assran et al. [2] After | ResNet-50 Setup from Bardes et al. [8] Before | ResNet-50 Setup from Bardes et al. [8] After |
| Fog | 44.18 | 46.34 | 58.41 | 61.42 | 24.00 | 38.60 |
| Impulse noise | 43.41 | 46.86 | 50.77 | 56.33 | 15.52 | 22.71 |
| Motion blur | 41.12 | 46.29 | 58.01 | 61.52 | 18.86 | 25.37 |
| Glass blur | 21.72 | 25.39 | 44.75 | 49.93 | 7.78 | 13.59 |
| Zoom blur | 39.26 | 43.00 | 50.92 | 54.70 | 20.19 | 27.27 |
| Gaussian noise | 44.56 | 48.21 | 49.17 | 55.97 | 25.78 | 25.99 |
| Contrast | 57.79 | 58.50 | 60.44 | 63.70 | 40.94 | 44.81 |
| Mean (15 corruptions) | 46.55 | 48.18 | 57.79 | 61.05 | 28.67 | 30.50 |
Table 5. ImageNet-C accuracy (%) before and after five adaptation steps. Seven corruptions with gains for every model are selected by mean gain across models; the final row averages all 15 corruptions. Bold indicates higher accuracy within each model.
Score-guided adaptation improves mean accuracy across all three EBReg-trained models and a broad range of corruptions. These gains indicate that the learned score remains informative under distribution shift, guiding corrupted representations toward higher-density regions of the corresponding source latent distribution. Improvements are particularly pronounced for severe noise and blur corruptions, though their magnitude varies across architectures and corruption types.
Conclusion
In this paper, we introduce EBReg, a representation learning framework derived from a lower bound on mutual information. EBReg jointly trains the encoder with a lightweight score network and uses the learned score to counter representation collapse. Our analysis shows that dimension-collapsed solutions are unstable under the proposed objective, which induces a persistent force toward activating collapsed dimensions. Under constraints preventing unbounded feature scaling, EBReg recovers target structures related to existing self-supervised regularizers. Empirically, EBReg consistently learns informative, non-collapsed representations across a range of settings. At test time, the learned score guides corrupted representations toward the learned manifold, improving robustness to noise.
Ethics Statement
This work addresses fundamental research questions in self-supervised representation learning and entropy-based regularization. We do not identify ethical concerns specific to the proposed method or the experiments reported in this paper. As with other representation-learning methods, downstream applications should be assessed for potential bias, privacy implications, and misuse in their intended context.
Reproducibility Statement
Section 3 specifies the training objective, alternating optimization procedure, and test-time adaptation rule, with pseudocode in Appendix C. Section 4 describes the datasets, backbone architectures, baseline protocols, and evaluation settings; Tables 3 and 4 report hyperparameter ablations and computational costs. The derivation of the mutual-information bound, analysis of collapsed directions, and maximum-entropy connections are provided in Appendices A, B, and E, respectively.
References
The smoothed variational bound
For completeness we record the two steps between the data processing inequality and Equation (2); both are the standard variational decomposition of mutual information [6] (Poole et al. 2019), applied to the smoothed variable . Writing the conditional entropy as a cross-entropy and then inserting ,
where is any conditional density. Both differential entropies are finite because has a density for every and the embeddings have finite second moments, so the decomposition is valid even when itself is supported on a set of measure zero.
Gaussian variational conditional. For , independence and the moments of the Gaussian noise give
where . The gap between the original MI and the lower bound is the sum of the smoothing loss and the conditional KL divergence in Equation (11). The variational step is tight when equals the true smoothed conditional density almost surely. Under finite second moments, the smoothed MI and the Gaussian conditional KL term are finite, whereas the smoothing loss can be infinite if the original MI is infinite.
Dimensional collapse
Setup and notation. Let be an embedding with mean , covariance , and finite second moment. Let be independent of and of the input and augmentation that generate it. The noisy embedding is ; when we want to stress that it is a random variable with a density we write for it. Conditionally on , is Gaussian with mean and covariance , so averaging over gives its density
This density is smooth and strictly positive for every , even when itself is discrete or lies on a lower-dimensional set. The score is .
Proof of Theorem 1
EBReg approximates the gradient of . The entropy objective itself disfavors dimensional collapse, since under a fixed variance budget its unique maximizer is full rank (Appendix § 14, part (i)). That objective-level fact does not describe what the EBReg update does to a collapsed encoder; Theorem 1 does, and we prove it here.
The clean score is blind
Goal. Assuming , show that and , and that the resulting gradient has zero mean. Idea. If the embeddings lie in a subspace, then in the directions orthogonal to it the noisy embedding is pure Gaussian noise, and the score of a Gaussian is known in closed form.
Step 1: the embeddings lie in a subspace. Let be the subspace the embeddings occupy, , the orthogonal projector onto , and the projector onto . The collapsed directions are exactly the vectors of . For any ,
Applying this to a basis of gives almost surely: every embedding equals the mean plus a vector in .
Step 2: split the distance into two orthogonal pieces. Fix a point . Insert and split into its components in and :
The two pieces are orthogonal, so their squared norms add:
Only the first term depends on .
Step 3: the density factorizes. Substitute Step 2 into Equation (13). The exponential of a sum is a product, and the factor that does not depend on comes out of the expectation:
The first factor depends on only through its component in , and carries everything about the embedding distribution. The second is a Gaussian density in with variance , and is the same whatever the embeddings do inside .
Step 4: the score in the collapsed directions. Take the logarithm and differentiate:
By the chain rule the first term equals , so it lies in . Applying removes it, because , and leaves
Step 5: evaluate at clean and noisy embeddings. At a clean embedding, Step 1 gives , so . At a noisy embedding,
For a unit collapsed direction we have , so taking the inner product with gives Equation (9).
Step 6: the gradient through has zero mean. The per-sample EBReg loss is . The score is detached and the noise does not depend on the parameters, so by the chain rule its parameter gradient is . The part of this gradient that passes through direction is
The vector depends only on the view, while is independent of it. Therefore
So each sample contributes a nonzero push along , but the pushes average to zero: exact collapse is a stationary point of the EBReg term.
PART (B): AMPLIFICATION
Goal. Assuming with , where is independent of the remaining coordinates , show that the expected EBReg gradient through is a negative multiple of . Idea. Along the noisy embedding is an independent one-dimensional Gaussian, whose score is linear; the rest is a covariance computation. Write for the parameters.
Step 1: the noisy coordinate along is an independent Gaussian. Along ,
This is a constant plus two independent Gaussians with variances and , so . It is also independent of the remaining coordinates : is independent of the first term by assumption, and for isotropic Gaussian noise the component is independent of the components orthogonal to .
Step 2: the score along . Independence means the density is a product of the one-dimensional Gaussian density of and a density of the remaining coordinates. Only the first depends on , so
At we have , hence
Setting recovers part (a).
Step 3: the expected gradient through . Multiply by and take expectations:
The noise term vanishes because is independent of the view and has zero mean, and because . The noise no longer cancels everything: the embedding’s own variation correlates with how the encoder responds.
Step 4: the gradient of the variance. Differentiate under the expectation, which and local domination of the parameter derivative permit. Using ,
where the last step uses .
Step 5: combine. Steps 3 and 4 contain the same covariance. Eliminating it,
which is (10). □
Consequence: exponential growth. Gradient descent subtracts times the gradient, so the contribution of this term to the update is , a step up the variance. To first order in the variance changes by
Insert from Step 4 and :
The variance therefore grows by a fixed fraction of itself per step, which is exponential growth at a rate proportional to . The drift is first order in , does not depend on the batch size, and vanishes as , consistent with part (a). Starting from a seed , reaching a fixed scale takes on the order of steps, so the size of the seed matters only logarithmically.
EBReg training procedure
Algorithms 1 and 2 detail the EBReg training procedure and score-guided test-time adaptation described in Sections 3.2 and 3.3, respectively.
Input: normalized encoder–projector , noise predictor , augmentation distribution ; .
Notation: sg = stopgrad. Score-fitting and regularization means pool the embeddings; alignment uses the matched pairs as in (5). Each noise draw is independent and standard Gaussian.
for do
Sample a minibatch and paired views using .
.
repeat times
Draw noise ; .
Update only on mean .
end repeat
Recompute on the same views with gradients; draw fresh .
; .
Update only on , using (5).
end for; return and .
Algorithm 1: EBReg training
Input: EBReg-trained encoder , fixed projector , score model , classifier ; and adaptation learning rate .
Notation: denotes the encoder’s normalization parameters and sg = stopgrad. The projector, score model, classifier, and all encoder parameters outside remain fixed.
for each test minibatch do
Reset to its pretrained source values.
repeat times
Compute encoder representation .
Compute score-space embedding .
.
.
Update only on with learning rate .
end repeat
Recompute and predict .
end for
Algorithm 2: Score-guided test-time adaptation
Implementation Details
In this section, we elaborate on various hyperparameter and design choices for our experiments. We use ImageNet-1k for all our experiments, using the standard train, validation sets comprising 1.28M and 50K images, respectively. We use ResNet-50, ViT-H, and ViT-L as the major architectural choices for our study. The augmentation pipeline matches those used in the corresponding protocols. We use a simple MLP as a denoiser in all the cases. We use a simple MLP projector for both VICReg and LeJEPA cases with projector dims of 256 and 64, respectively. Other hyperparameters for each of the protocols are listed below:
VICReg protocol. The projector has layer widths 8192–8192–256. The denoiser uses a hidden width of 2048 and four blocks. The alignment-loss coefficient is 1.
LeJEPA protocol. The augmentation pipeline produces two global crops of size 224 224 with scale range [0.3, 1.0] and eight local crops of size 98 98 with scale range [0.05, 0.3]. Images use ImageNet channel-wise normalization with mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225). The learning-rate warmup starts at 0.01 times the configured learning rate. Training uses bfloat16 precision.
I-JEPA/StoP protocol. The input crop size is , with crop scale range and patch size 14. We use one context mask with scale range and four target masks with scale range , mask aspect ratios in , and a minimum of 10 retained patches. The predictor has depth 12 and embedding dimension 384. The initial and final learning rates are and , respectively; the configuration specifies constant learning-rate and weight-decay strategies from epoch 250. StoP is enabled with variance parameter 0.2.
Maximum-entropy targets under each constraint
We compute the maximum-entropy distribution under each constraint of Table 1, labelled (i)–(iv) in the order of the table. All four follow from one standard computation.
The maximum-entropy recipe. Consider distributions on a support that satisfy moment constraints for . Suppose the exponential-family density
satisfies the constraints for some multipliers . Then is the unique maximum-entropy distribution. To see this, take any feasible . Because , its average under involves only the constrained moments:
Hence
with equality only when . With no moment constraints, , the recipe gives the constant density, that is, the uniform distribution on .
(i) Quadratic constraint: isotropic Gaussian. Let and ; unit marginal variances are a special case. The noisy embedding then satisfies
Apply the recipe to on with . It gives , a zero-mean isotropic Gaussian, and matching the second moment fixes its variance at per coordinate. Its entropy is the bound
with equality only when . This happens exactly when : adding the independent noise to gives that Gaussian, and conversely the characteristic function of is the product , so dividing by the noise factor leaves . The result holds for every . Since the maximizer has , a rank-deficient representation is never entropy-maximal under this constraint. The target is SIGReg’s reference (Balestiero & LeCun, 2025), its covariance is the second-moment condition that VICReg’s penalties drive toward ([8]), and batch normalization without affine parameters enforces the constraint up to finite-batch error.
(ii) Spherical constraint: uniform on the sphere. Let , so and there is no moment constraint. The recipe gives the constant density with respect to surface measure, the uniform distribution, with entropy . The same holds on a sphere of any radius, such as under RMS normalization. Layer normalization additionally centers each embedding, , so is the sphere inside the hyperplane orthogonal to and the target is uniform on that lower-dimensional sphere. Spherical uniformity is the target that contrastive objectives approach through negatives ([41]).
(iii) Categorical constraint: uniform code usage. Let , so is a set of points with counting measure and again no moment constraint. The recipe gives equal probabilities , with entropy . Uniform usage is SwAV’s equipartition constraint and the marginal-entropy regularizer of MSN [11, 1]; DINO’s centering and sharpening act as a related mechanism rather than a prescribed density [12]. Marginal balance across inputs differs from uninformative uniform predictions for every input.
(iv) Moment constraint: generalized Gaussian. Let on a support . The recipe with gives
with chosen to satisfy the constraint, whenever this density is normalizable. On it is a product generalized Gaussian, which reduces to the Gaussian of part (i) at , and restricting to the positive orthant gives its truncation. This is the principle behind Rectified LpJEPA’s target design [27], whose additional rectification introduces atoms at zero and requires mixed-measure entropy.
Effect of the smoothing. Part (i) is computed directly for the smoothed entropy and is exact at every noise level. Parts (ii)–(iv) are computed for the entropy of itself, which approaches from above as : adding independent noise can only add entropy, , and the excess vanishes with . For the sphere and the codes the targets remain maximizers of at every by symmetry. Rotating a distribution on the sphere, or permuting the codes, leaves unchanged because the noise is isotropic; the uniform distribution is the average of all such rotated or permuted copies; and the entropy of a mixture is at least the average entropy of its components. For the moment constraint with the maximizer of at fixed is a smoothed analogue of Equation (17) that we do not characterize.
Use of Large Language Models
LLMs were used to assist with writing, revising, organizing, and polishing the manuscript; coding tasks; table and bibliography formatting; and LaTeX troubleshooting. They also assisted in deriving proofs presented in the appendix, reviewing mathematical arguments, identifying gaps between stated assumptions and claimed conclusions, and drafting revisions to narrow the interpretation of the directional gradient results. All experimental values used in preparing tables and discussing results were supplied by the authors. The authors take full responsibility for the manuscript’s correctness, originality, and proper attribution, including all LLM-assisted text, code, proofs, and mathematical reasoning.
References
- [1]Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In ECCV, 2022.arxiv.org/abs/2204.07141
- [2]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, 2023.arxiv.org/abs/2301.08243
- [3]Randall Balestriero and Yann LeCun. How learning by reconstruction produces uninformative features for perception. In ICML, 2024.arxiv.org/abs/2402.11337
- [4]Randall Balestriero and Yann LeCun. LeJEPA: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025.arxiv.org/abs/2511.08544
- [5]Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT pre-training of image transformers. In ICLR, 2022.arxiv.org/abs/2106.08254
- [6]David Barber and Felix Agakov. The IM algorithm: A variational approach to information maximization. In NeurIPS, 2003a.
- [7]David Barber and Felix Agakov. The im algorithm: a variational approach to information maximization. In Proceedings of the 17th International Conference on Neural Information Processing Systems, 2003b.
- [8]Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In ICLR, 2022.arxiv.org/abs/2105.04906
- [9]Suzanna Becker and Geoffrey E. Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 1992.pubmed.ncbi.nlm.nih.gov/1729650
- [11]Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.arxiv.org/abs/2006.09882
- [12]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. Preprint. Under review.
- [13]Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In ICML, 2023.arxiv.org/abs/2302.07194
- [14]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.arxiv.org/abs/2002.05709
- [15]Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning. In ICLR, 2025.arxiv.org/abs/2401.14404
- [16]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009.
- [17]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
- [18]Quentin Garrido, Randall Balestiero, Laurent Najman, and Yann LeCun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. In ICML, 2023.arxiv.org/abs/2210.02885
- [19]Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS, 2020.arxiv.org/abs/2006.07733
- [20]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.arxiv.org/abs/1512.03385
- [21]Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.arxiv.org/abs/1911.05722
- [22]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.arxiv.org/abs/2111.06377
- [23]Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
- [24]Geoffrey E. Hinton and Ruslan R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 2006.pubmed.ncbi.nlm.nih.gov/16873662
- [25]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.arxiv.org/abs/2006.11239
- [26]Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.arxiv.org/abs/2110.09348
- [27]Yilun Kuang, Yash Dagade, Tim G. J. Rudner, Randall Balestiero, and Yann LeCun. Rectified LpJEPA: Joint-embedding predictive architectures with sparse and maximum-entropy representations. arXiv preprint arXiv:2602.01456, 2026.arxiv.org/abs/2602.01456
- [28]Yingzhen Li and Richard E. Turner. Gradient estimators for implicit models. In ICLR, 2018.arxiv.org/abs/1705.07107
- [29]Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-JEPA 2.1: Unlocking dense features in video self-supervised learning. arXiv preprint arXiv:2603.14482, 2026.
- [30]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeddin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. TMLR, 2024. Ben Poole, Sheril Ozair, Aaron van den Oord, Alexander A. Alemi, and George Tucker. On variational bounds of mutual information. In ICML, 2019.
- [31]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.arxiv.org/abs/2103.00020
- [32]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.arxiv.org/abs/2112.10752
- [33]Jürgen Schmidhuber and Daniel Prelinger. Discovering predictable classifications. Neural Computation, 1993.DOI
- [34]Jiaxin Shi, Shengyang Sun, and Jun Zhu. A spectral approach to gradient estimation for implicit distributions. In ICML, 2018.arxiv.org/abs/1806.02925
- [35]Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
- [36]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurIPS, 2019.arxiv.org/abs/1907.05600
- [37]Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
- [38]Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 2011.DOI
- [39]Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008.DOI
- [40]Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.arxiv.org/abs/2006.10726
- [41]Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.arxiv.org/abs/2005.10242
- [42]Chao-Yuan Wu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühl. Sampling matters in deep embedding learning. In ICCV, 2017.arxiv.org/abs/1706.07567
- [43]Haiyu Wu, Randall Balestriero, and Morgan Levine. VISReg: Variance-invariance-sketching regularization for JEPA training. arXiv preprint arXiv:2606.02572, 2026.arxiv.org/abs/2606.02572
- [44]Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In ICLR, 2025.arxiv.org/abs/2410.06940
- [45]Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders. In ICLR, 2026.arxiv.org/abs/2510.11690

