betaFeedback

WorksComputer Science

Self-Supervised Collapse Prevention with Score-Based Entropy Maximization

Abstract

Diffusion models have demonstrated the effectiveness of score-based objectives for generative modeling, yet their potential for self-supervised representation learning remains underexplored. We introduce Entropy-Based Regularization (EBReg), a framework that uses latent score estimation to regularize the latent distribution during self-supervised training. EBReg jointly learns an encoder and a latent score model, and utilizes the learned score to provide a diversity-preserving signal that prevents representational collapse. Motivated by a mutual-information lower bound, we formulate the objective as view alignment with smoothed marginal entropy maximization and optimize the marginal entropy without explicit density evaluation. The same score model can additionally be used at test time to refine corrupted representations toward the training latent distribution. Empirically, EBReg learns competitive representations on ImageNet-1K, achieving 75.57% linear-probe accuracy with ViT-L. Across diverse architectures, score-guided test-time adaptation further improves robustness on ImageNet-C, yielding an average relative accuracy gain of approximately 5%.

Introduction

Diffusion models ([35, 36, 25]) have established score estimation as a central component of modern generative modeling. However, representations learned through standard diffusion objectives often remain weaker than those produced by dedicated self-supervised learning (SSL) methods on recognition benchmarks ([44, 15]). Conversely, recent work shows that strong pretrained representations obtained from self-supervised learning can substantially improve diffusion training and generation, as demonstrated by REPA (Yu et al., [44] and Representation Autoencoders ([45]). Earlier work on denoising autoencoders also demonstrated that denoising objectives can improve representation learning ([39]). Together, these observations motivate the question: Can a score estimator and a self-supervised encoder be learned jointly in an end-to-end manner?

A central challenge in SSL is to enforce invariance across augmented views while maintaining sufficient diversity in the latent distribution to avoid collapse. Existing methods achieve this through different mechanisms, including negative-pair interactions ([14]), moment-based constraints ([8]), and regularization toward prescribed distributions ((Balistera & Le-Cun, 2025)). Despite their different forms, these mechanisms preserve representation diversity while enforcing invariance. We investigate whether an analogous anti-collapse signal can be obtained from a latent score estimated via score matching.

We propose Entropy-Based Regularization (EBReg), a self-supervised framework that jointly trains an encoder and a latent score model. EBReg uses the score of the evolving latent distribution as an adaptive entropy-promoting signal. Figure 1 illustrates this mechanism directly: samples concentrated along a low-dimensional structure are driven by the negative score toward a higher-entropy configuration that occupies both dimensions. Motivated by a variational mutual-information lower bound [7], the objective combines view alignment with maximizing the entropy of a Gaussian-smoothed latent distribution under a scale constraint. Since the latent density is unknown, EBReg estimates the score of the smoothed distribution via denoising score matching and uses it to optimize a gradient surrogate of marginal entropy without explicit density evaluation. The resulting encoder update promotes diversity in the latent distribution and counteracts representational collapse.

Figure 1. Illustration of the EBReg anti-collapse mechanism. Top: (a) Representations lie near a one-dimensional subspace of R2\mathbb{R}^{2}. (b) EBReg alone spreads them across both dimensions. (c) Alignment alone contracts them, resulting in dimensional collapse. (d) Alignment + EBReg yields compact, separated classes while preventing dimensional collapse. Bottom: Evolution of the latent distribution under EBReg. Samples are moved away from the diagonal and populate both dimensions, ultimately forming an isotropic Gaussian distribution.

We further connect the EBReg formulation to existing anti-collapse regularizers [4, 14] while replacing explicit negative pairs or fixed distribution matching with an adaptive score field. We characterize how latent scale constraints induce maximum-entropy solutions and relate these solutions to distributional targets and moment conditions underlying existing SSL regularizers. We further analyze the exact-score dynamics near dimensional collapse, including the behavior of low-variance directions. The learned score additionally enables test-time latent refinement, moving corrupted representations toward higher-density regions of the latent distribution while adapting only normalization parameters.

Empirically, EBReg learns competitive representations on ImageNet-1K [16] across convolutional and transformer backbones, performing comparably to established self-supervised methods including VICReg [8] and LeJEPA [4] under the same pretraining and evaluation setup. Score-guided test-time refinement further improves robustness on ImageNet-C [23] across diverse architectures.

Our main contributions are summarized as follows:

Method. We derive an alignment–entropy objective from a variational mutual-information lower bound and introduce EBReg, which uses a learned latent score as an anti-collapse regularizer by optimizing representation entropy without explicit density evaluation or negative pairs.

Theoretical analysis. We analyze the exact-score dynamics near dimensional collapse and characterize when EBReg amplifies low-variance directions. We further show how latent constraints determine maximum-entropy solutions that recover distributional targets and moment conditions underlying existing SSL regularizers.

Score-guided test-time adaptation. We reuse the learned score as a label-free adaptation signal, updating only encoder normalization parameters to guide corrupted representations toward higher-density regions of the source latent distribution and improve robustness on ImageNet-C.

Related Work

Latent Space Information Maximization. I-MAX [9], an early precursor to joint-embedding methods, maximizes mutual information between module outputs for related inputs. Early tractable formulations use stochastic binary outputs or Gaussian assumptions for scalar continuous outputs. Predictability Maximization [33] instead combines agreement between paired representations with a discrimination loss that discourages collapse but does not in general yield a mutual-information objective. EBReg revisits the information-maximization principle through an alignment–entropy objective derived from an MI lower bound (Section 3.1). Unlike these earlier formulations, EBReg uses a learned score to optimize marginal entropy without explicit density or mutual-information estimation.

Contrastive methods. Bromley et al. (1993) introduced Siamese networks, which use shared-weight encoders to learn representations by distinguishing matching from non-matching pairs. Building on this principle, modern contrastive methods [14, 21, 31] make use of negative examples to provide a non-parametric repulsive signal that prevents representational collapse. Random negative sampling, however, can yield many uninformative pairs in high-dimensional embedding spaces [42], resulting in dimensional collapse [26]. Prototype-based methods such as SwAV, DINO, and MSN [11, 12, 1] avoid instance-level negatives by balancing assignments over a finite set of learned prototypes, which imposes an explicit discretization of the latent structure. EBReg instead learns a parametric score field to provide entropy-promoting directions without explicit negative comparisons. This design draws on the empirical success of score-based generative models ([37, 32]) and theoretical evidence that score estimation can adapt to low-dimensional subspace structure, mitigating the curse of high dimensionality [13].

Non-contrastive methods. BYOL [19] helped revive non-contrastive learning through asymmetric branches and stop-gradient operations. Subsequent teacher–student distillation methods [2, 29, 30] learn strong representations, with collapse avoidance governed by interactions among architecture, distillation, and optimization. A complementary approach explicitly regularizes representation statistics: VICReg [8] enforces variance floors and penalizes cross-dimensional covariance, while LeJEPA [4] targets an isotropic Gaussian, with related distributional approaches in VISReg [43] and Rectified LpJEPA [27]. EBReg instead maximizes smoothed marginal entropy subject to a scale constraint, allowing the entropy objective and constraint to determine the preferred distribution. EBReg under different scale constraints yields maximum-entropy targets that satisfy moment conditions used by existing methods (Table 1).

Latent constraintTypical realizationMax.-entropy targetSSL connection
E[h]=0\mathbb{E}[h]=0, tr⁡Cov⁡(h)=d\operatorname{tr}\operatorname{Cov}(h)=dBatch normalization without affine parametersIsotropic GaussianSIGReg: Gaussian target; VICReg: corresponding first- and second-order moments
∥h∥2=r\lVert h\rVert_2=rℓ2\ell_2, RMS, or layer normalizationUniform on the sphereContrastive learning
h∈{e1,…,eK}h \in\{e_1,\ldots,e_K\}Discrete prototype / code assignmentUniform code usageSwAV and MSN: balanced prototype usage;
E∥h∥pp=M\mathbb{E}\lVert h\rVert_p^p=M, h∈Sh\in Spp-th moment scaling with support constraintGeneralized Gaussian (truncated to SS)Rectified LpJEPA

Table 1. Maximum-entropy targets under different latent constraints. Each constraint induces a corresponding maximum-entropy target linked to existing SSL regularizers. First three targets maximize the smoothed entropy for any σ>0\sigma>0. The last holds as σ→0\sigma\to0 (Appendix § 14).

Reconstruction based methods. Another prominent representation-learning paradigm uses reconstruction objectives [24, 39], which encourage representations to retain information about the input. More recently, masked autoencoders have achieved strong performance by reconstructing missing input regions [22, 5]. Reconstruction naturally discourages collapse and can be interpreted as promoting high mutual information between inputs and their representations. However, input-space objectives may prioritize low-level reconstruction over downstream semantics [3]. Chen et al. (2025) connect denoising diffusion models to denoising autoencoders and propose ℓ\ell-DAE, which corrupts low-dimensional latent features but reconstructs the original input in pixel space using a ViT-based decoder. In contrast, EBReg operates entirely in representation space, using a score function to regularize latent representations directly. This avoids pixel-level reconstruction and decouples collapse prevention from input-space fidelity.

Methodology

Our goal is to show that score matching itself can provide a viable objective for learning strong self-supervised representations. To this end, we first derive a mutual-information lower bound and introduce the score-based EBReg surrogate for optimizing marginal entropy (Section 3.1). We then describe alternating score fitting, encoder learning, and score-guided test-time adaptation (§ 3.2), with an overview in Figure 2 and training pseudocode in Appendix § 12. Finally, we theoretically characterize the exact-score dynamics in collapsed and low-variance directions (§ 3.4).

Figure 2. Overview of EBReg: training and test-time adaptation. (a) Two augmented views are encoded into normalized embeddings. The encoder minimizes a view-alignment loss and an entropy regularizer, LEBRegL_{\mathrm{EBReg}}, using a detached estimate of the score of the noise-smoothed embedding distribution. (b) The latent score is parameterized by a noise-prediction network fitted on detached embeddings, with KscoreK_{\mathrm{score}} updates per encoder step; it predicts the injected Gaussian noise, giving the score estimate sψ,σ(h~)=−ϵ^ψ(h~)/σs_{\psi,\sigma}(\tilde{h})=-\hat{\epsilon}_{\psi}(\tilde{h})/\sigma. (c) At test time, the learned score is reused as a test-time adaptation signal. The score model and remaining encoder–projector parameters are frozen, while only normalization parameters α\alpha are updated by minimizing LTTAL_{\mathrm{TTA}}, guiding corrupted embeddings toward higher-density regions of the learned source distribution.

From mutual information to score-based entropy regularization

EBReg combines alignment between augmented views with maximization of smoothed marginal entropy. We motivate this objective through the mutual information between representations of two augmented views. Maximizing mutual information favors representations that remain predictive across views while retaining variability in their marginal distribution. This leads naturally to an alignment–entropy formulation, where alignment promotes cross-view agreement and marginal entropy preserves representation diversity and prevents collapse. EBReg then uses a learned score to optimize the smoothed marginal entropy without explicit density evaluation.

An alignment–entropy lower bound. Given an input xx, we draw two transformations tt, t′t' independently from an augmentation distribution T\mathcal{T}. A shared encoder fθf_\theta and projector gϕg_\phi produce embeddings h(1)=gϕ(fθ(t(x)))\mathbf{h}^{(1)}=g_\phi(f_\theta(t(x))) and h(2)=gϕ(fθ(t′(x)))\mathbf{h}^{(2)}=g_\phi(f_\theta(t'(x))) in Rd\mathbb{R}^{d}. Both views share the marginal distribution pθ,ϕp_{\theta,\phi}. Embeddings may occupy a lower-dimensional support and lack a density in the full embedding space, making their differential entropy ill-defined. Gaussian smoothing yields a full-dimensional density, allowing us to work with a well-defined smoothed entropy. For a fixed noise scale σ>0\sigma>0, we define

h~(1)=h(1)+σϵ,ϵ∼N(0,Id),(1)\tilde{\mathbf{h}}^{(1)}=\mathbf{h}^{(1)}+\sigma\epsilon,\qquad\epsilon\sim\mathcal{N}(\mathbf{0},I_d), \tag*{(1)}

where ϵ\epsilon is independent of both views. We write Hσ(h)=H(h+σϵ)H_\sigma(\mathbf{h})=H(\mathbf{h}+\sigma\epsilon), where HH denotes differential entropy, and pσ=pθ,ϕ∗N(0,σ2Id)p_\sigma=p_{\theta,\phi}*\mathcal{N}(\mathbf{0},\sigma^2I_d) for the smoothed density.

To relate the smoothed representation to the mutual-information objective between the two clean views, consider I(h~(1);h(2))I(\tilde{\mathbf{h}}^{(1)};\mathbf{h}^{(2)}). Since h~(1)\tilde{\mathbf{h}}^{(1)} is obtained from h(1)\mathbf{h}^{(1)} by adding independent Gaussian noise, h(2)→h(1)→h~(1)\mathbf{h}^{(2)}\to\mathbf{h}^{(1)}\to\tilde{\mathbf{h}}^{(1)} forms a Markov chain. The data processing inequality therefore gives I(h(1);h(2))≥I(h~(1);h(2))I(\mathbf{h}^{(1)};\mathbf{h}^{(2)}) \ge I(\tilde{\mathbf{h}}^{(1)};\mathbf{h}^{(2)}). Combining this inequality with the variational bound of Barber & Agakov [6] applied to the smoothed pair yields

I(h(1);h(2))≥Hσ(h(1))+Elog⁡q(h~(1)∣h(2)),(2)I(\mathbf{h}^{(1)};\mathbf{h}^{(2)}) \ge H_{\sigma}(\mathbf{h}^{(1)}) + \mathbb{E}\log q(\tilde{\mathbf{h}}^{(1)} \mid\mathbf{h}^{(2)}), \tag*{(2)}

for any variational conditional density qq. We choose qτ(⋅∣h(2))=N(h(2),τ2Id)q_{\tau}(\cdot\mid\mathbf{h}^{(2)}) = \mathcal{N}(\mathbf{h}^{(2)},\tau^{2}I_d) with fixed τ>0\tau> 0, yielding alignment loss. Choosing a Gaussian variational density does not require either the embedding distribution or the true conditional distribution to be Gaussian. Its cross-entropy is

−Elog⁡qτ(h~(1)∣h(2))=d2τ2Lalign+Cσ,τ,(3)-\mathbb{E}\log q_{\tau}(\tilde{\mathbf{h}}^{(1)} \mid\mathbf{h}^{(2)}) = \frac{d}{2\tau^{2}}\mathcal{L}_{\mathrm{align}} + C_{\sigma,\tau}, \tag*{(3)}

where Lalign=d−1E∥h(1)−h(2)∥22\mathcal{L}_{\mathrm{align}} = d^{-1}\mathbb{E}\lVert\mathbf{h}^{(1)}-\mathbf{h}^{(2)}\rVert_{2}^{2} is the normalized alignment loss [41] and Cσ,τ=d2log⁡(2πτ2)+dσ22τ2C_{\sigma,\tau} = \frac{d}{2}\log(2\pi\tau^{2}) + \frac{d\sigma^{2}}{2\tau^{2}} is independent of (θ,ϕ)(\theta,\phi). See Appendix § 9 for the full derivation. Substituting this cross-entropy expression into Equation (2) yields the alignment–entropy lower bound

I(h(1);h(2))≥Hσ(h)−d2τ2Lalign−Cσ,τ.(4)I(\mathbf{h}^{(1)};\mathbf{h}^{(2)}) \ge H_{\sigma}(\mathbf{h}) - \frac{d}{2\tau^{2}}\mathcal{L}_{\mathrm{align}} - C_{\sigma,\tau}. \tag*{(4)}

Here, h\mathbf{h} denotes an embedding drawn from the shared marginal distribution of the two views. For fixed σ\sigma and τ\tau, maximizing this lower bound is equivalent to minimizing Lalign−γHσ(h)\mathcal{L}_{\mathrm{align}}-\gamma H_{\sigma}(\mathbf{h}), where γ=2τ2/d\gamma= 2\tau^{2}/d. The objective therefore balances agreement between paired views with entropy of their shared marginal distribution. Increasing latent scale can increase entropy without enriching the representation. We therefore constrain latent scale to prevent entropy maximization through rescaling alone, with the choice of constraint discussed in § 3.1.

We now introduce a tractable gradient surrogate for optimizing the lower bound in Equation (4). The alignment term is computed directly from paired embeddings, but the smoothed entropy depends on the unknown density pσp_{\sigma}, which is generally intractable to evaluate. Rather than evaluating pσp_{\sigma} directly, EBReg estimates its score, which points toward locally higher-density regions. We therefore replace the negative entropy term with the EBReg gradient surrogate and train the encoder on

Ltotal=Lalign+γE[stopgrad⁡(sψ,σ(h~))Th~]⏟LEBReg.(5)\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{align}} + \gamma\mathbb{E}\underbrace{\left[\operatorname{stopgrad}\left(s_{\psi,\sigma}(\tilde{\mathbf{h}})\right)^{\mathsf{T}}\tilde{\mathbf{h}}\right]}_{\mathcal{L}_{\mathrm{EBReg}}}. \tag*{(5)}

Here, h~=h+σϵ\tilde{\mathbf{h}}=\mathbf{h}+\sigma\boldsymbol{\epsilon} and sψ,σs_{\psi,\sigma} denotes the score estimate. The stop-gradient treats the score as fixed when differentiating the surrogate with respect to the encoder and projector. § 3.2 describes the alternating score-fitting and encoder updates.

We now show that, with an exact score, this surrogate implements entropy maximization at the gradient level. The entropy-gradient identity for implicit distributions [28, 34] gives, under standard regularity conditions,

∇θ,ϕHσ(h)=−E[Jθ,ϕ(h)Tsσ(h~)],(6)\nabla_{\theta,\phi}H_{\sigma}(\mathbf{h}) = -\mathbb{E}\left[J_{\theta,\phi}(\mathbf{h})^{\mathsf{T}}s_{\sigma}(\tilde{\mathbf{h}})\right], \tag*{(6)}

where Jθ,ϕ(h)=∂h/∂(θ,ϕ)J_{\theta,\phi}(\mathbf{h})=\partial\mathbf{h}/\partial(\theta,\phi) is the embedding Jacobian. Detaching the entire score vector in Equation 5 gives ∇θ,ϕLEBReg=E[Jθ,ϕ(h)Tsψ,σ(h~)]\nabla_{\theta,\phi}\mathcal{L}_{\mathrm{EBReg}} = \mathbb{E}[J_{\theta,\phi}(\mathbf{h})^{\mathsf{T}}s_{\psi,\sigma}(\tilde{\mathbf{h}})]. Thus, with an exact score, the EBReg update implements gradient descent on −Hσ-H_{\sigma} without evaluating the density or entropy. For γ=2τ2/d\gamma=2\tau^{2}/d, the resulting representation-parameter gradient equals that of the negative MI lower bound up to the positive factor γ\gamma. With a learned score, this equivalence is approximate and the surrogate’s value is neither an entropy estimate nor an MI bound.

Intuitively, alignment pulls paired embeddings together, while EBReg encourages the embedding distribution to spread out. The learned score sψ,σ(h~)s_{\psi,\sigma}(\tilde{\mathbf{h}}) approximates the local direction of increasing density in the noise-smoothed embedding distribution. Minimizing the surrogate encourages embeddings to move against the score, toward locally lower-density regions, thereby producing an entropy-promoting encoder update.

Connections to existing regularizers. EBReg’s scale constraint determines its entropy-maximizing target. Under zero mean and unit marginal variances, the maximizer is the isotropic Gaussian targeted by SIGReg [4], whose moments coincide with the variance–covariance conditions encouraged by VICReg [8]. More generally, different scale constraints induce different maximum-entropy targets while leaving the EBReg formulation unchanged. Table 1 relates these targets to existing SSL regularizers. However, sharing the same target does not imply equivalence of the corresponding objectives. Appendix § 14 provides the derivations.

Alternating score fitting and encoder learning

Since the score in Equation (5) depends on the encoder’s evolving embedding distribution, EBReg alternates between fitting the score model to the current embeddings and updating the encoder with the total objective. Figure 2 summarizes these two phases. Algorithm 1 in Appendix § 12 provides the complete training procedure. Embedding normalization enforces the chosen scale constraint before Gaussian noise is added.

Fitting the latent score. We parameterize a noise-prediction network ϵ^ψ\hat{\epsilon}_{\psi} and recover the latent score from its output via Tweedie’s formula. For each minibatch, we pool embeddings from both augmented views and stop gradients through them, hˉ=stopgrad⁡(h)\bar{h}=\operatorname{stopgrad}(h). With the encoder and projector held fixed, we take KscoreK_{\mathrm{score}} updates of ψ\psi on the denoising objective

LDSM(ψ)=Ehˉ,ϵ[∥ϵ^ψ(hˉ+σϵ)−ϵ∥22],sψ,σ(z)=−ϵ^ψ(z)σ,(7)\mathcal{L}_{\mathrm{DSM}}(\psi)=\mathbb{E}_{\bar{h},\epsilon}\left[\left\lVert\hat{\epsilon}_{\psi}(\bar{h}+\sigma\epsilon)-\epsilon\right\rVert_2^2\right],\qquad s_{\psi,\sigma}(z)=-\frac{\hat{\epsilon}_{\psi}(z)}{\sigma}, \tag*{(7)}

using fresh ϵ∼N(0,Id)\epsilon\sim\mathcal{N}(0,I_d) for each update and the same noise level σ\sigma as in § 3.1. At the unrestricted population optimum, the predictor equals E[ϵ∣h+σϵ=z]\mathbb{E}[\epsilon\mid h+\sigma\epsilon=z], whose negative, scaled by 1/σ1/\sigma is the score of pσp_{\sigma} [38]. Finite network capacity and a finite number of updates yield an approximation to this score as the embedding distribution evolves.

Updating the encoder. We recompute embeddings for the same paired views with gradients enabled, draw fresh noise, and update (θ,ϕ)(\theta,\phi) using Equation (5). The alignment loss acts on clean paired embeddings while the entropy regularizer uses the score evaluated at their noisy counterparts. We detach the entire score output so that its value supplies the update direction while gradients flow through hˉ\bar{h} to the encoder and projector. The score is fixed during the encoder update, so score fitting updates only ψ\psi, and the encoder step updates only (θ,ϕ)(\theta,\phi).

Score-guided test-time adaptation

Beyond its role in EBReg training, the learned score provides the direction of increasing log-density under the source latent distribution. This makes it directly useful for test-time adaptation, since corrupted inputs may produce representations that move away from high-density source regions and the score provides a direction back toward them. Let α\alpha denote the encoder’s normalization parameters and hαh_{\alpha} the resulting test representation. We keep the score model, classifier, and remaining encoder parameters fixed, and adapt only α\alpha, following Wang et al. [40].

We define the score-guided adaptation objective as

LTTA(α)=−E[stopgrad⁡(sψ,σ(hα))Thα].(8)\mathcal{L}_{\mathrm{TTA}}(\alpha)=-\mathbb{E}\left[\operatorname{stopgrad}\left(s_{\psi,\sigma}(h_{\alpha})\right)^{\mathsf{T}}h_{\alpha}\right]. \tag*{(8)}

The normalization parameters are updated for KttaK_{\mathrm{tta}} steps using gradients of LTTA\mathcal{L}_{\mathrm{TTA}}, while the score model remains fixed. This objective reverses the role of the score used during EBReg training. During pretraining, the negative-score contribution promotes entropy by spreading representations away from high-density regions. At test time, the positive-score direction instead encourages corrupted representations toward higher-density regions of the source latent distribution. Thus, the score learned for representation regularization also provides a label-free signal for adaptation under distribution shift. Section 4.3 evaluates this procedure under input corruptions.

Collapse Dynamics and Connection to SSL Regularization

We analyze EBReg near representational collapse and characterize how latent scale constraints determine maximum-entropy solutions related to existing SSL regularizers.

Instability under dimensional collapse. If the embeddings lie in a lower-dimensional subspace of Rd\mathbb{R}^{d}, evaluating the smoothed score on the clean embeddings yields no component along collapsed directions and therefore cannot recover them. Evaluating the score at noise-perturbed embeddings introduces a nonzero component in these directions. The following theorem formalizes this behavior and shows that directions with small but nonzero variance are amplified.

Intuition. Part (a) explains why the score in Equation (5) is evaluated at h~\tilde{h} rather than at hh. At a clean embedding, the score vanishes along any collapsed direction, regardless of the distribution within the occupied subspace. It also shows that exact collapse is a stationary point of the EBReg objective, with nonzero sample-wise updates along collapsed directions but zero mean drift. Part (b) shows that this stationary point is unstable, since the right-hand side of Equation (10) defines a variance-ascent direction and descending Ltotal\mathcal{L}_{\mathrm{total}} therefore amplifies any nonzero variance along qq. Because the update scales with the variance already present rather than introducing a fixed increment, the relative amplification in the low-variance regime is governed by γ/σ2\gamma/\sigma^{2} (Appendix B). It is also not opposed by the alignment term, which penalizes disagreement between views while remaining invariant to variation shared across paired views. This theorem characterizes directional gradient contributions under an exact score and, in part (b), an independent Gaussian-coordinate assumption; it does not guarantee recovery from exact collapse or variance growth under the full training dynamics.

Theorem 1 (Instability of collapsed directions under EBReg). Let h∈Rdh \in\mathbb{R}^{d} have finite second moment and fix a unit direction qq. Assume an exact score sσs_{\sigma} and the Gaussian perturbation above, with noise independent of the input and augmentation, and write aq=Jθ,ϕ(h)Tqa_q = J_{\theta,\phi}(h)^{\mathsf{T}}q for the parameter gradient of qThq^{\mathsf{T}}h, with E∥aq∥22<∞\mathbb{E}\lVert a_q\rVert_{2}^{2} < \infty.

(a) The clean score is blind to dimensional collapse. If Var⁡(qTh)=0\operatorname{Var}(q^{\mathsf{T}}h)=0, then almost surely

qTsσ(h~)=−qTϵσ,qTsσ(h)=0,(9)q^{\mathsf{T}}s_{\sigma}(\tilde{h})=-\frac{q^{\mathsf{T}}\epsilon}{\sigma}, \qquad q^{\mathsf{T}}s_{\sigma}(h)=0, \tag*{(9)}

so the per-sample gradient through qq, gq=−(γ/σ)(qTϵ)aqg_q=-(\gamma/\sigma)(q^{\mathsf{T}}\epsilon)a_q, is nonzero but has zero mean.

(b) Amplification. If qTh=c+δuq^{\mathsf{T}}h=c+\delta u with δ>0\delta>0 and u∼N(0,1)u\sim\mathcal{N}(0,1) independent of (Id−qqT)h(I_d-qq^{\mathsf{T}})h, then

γE[aqqTsσ(h~)]=−γ2(δ2+σ2)∇θ,ϕVar⁡(qTh).(10)\gamma\mathbb{E}\left[a_q q^{\mathsf{T}}s_{\sigma}(\tilde{h})\right] =-\frac{\gamma}{2(\delta^{2}+\sigma^{2})}\nabla_{\theta,\phi}\operatorname{Var}(q^{\mathsf{T}}h). \tag*{(10)}

Experiments

We compare EBReg with existing self-supervised methods, matching parameter count, data augmentations, and training epochs unless otherwise specified. We evaluate representation quality, collapse prevention, hyperparameter sensitivity, computational overhead, and test-time adaptation under input corruptions.

Linear evaluation

We evaluate the effectiveness of EBReg across multiple architectures and self-supervised objectives on ImageNet-1K (DeLong et al., 2009). We consider ResNet-50 [20], ViT-Large, and ViT-Huge [17] under three established pretraining protocols. For ResNet-50, we adopt the setup from Bardes et al. [8], including its data augmentations, optimization settings, and training schedule, and replace the original variance–covariance regularization with EBReg. For ViT-Large, we follow the setup from Balestriero & LeCun [4] and replace SIGReg with EBReg while keeping the remaining pretraining and evaluation protocol unchanged. Finally, we evaluate EBReg with ViT-Huge in the setup from Assran et al. [2], where EBReg is added to the existing objective rather than replacing it. This setting assesses whether score-based entropy regularization can provide a complementary signal to an existing self-supervised objective. Within each setting, EBReg and the corresponding baseline use the same backbone, augmentations, training schedule, and evaluation procedure. The results are summarized in Table 2.

BackboneMethod / RegularizerEpochsLinear probe (%) ↑Eff. rank ↑
ResNet-50Alignment only (γ=0\gamma= 0)100Collapse–
ResNet-50VICReg10068.021368
ResNet-50SIGReg10066.421256
ResNet-50EBReg (ours)10067.921431
ViT-LSIGReg10075.32648
ViT-Lℓ\ell-DAE160075.00–
ViT-LEBReg (ours)10075.57759
ViT-HI-JEPA10079.03540
ViT-HI-JEPA + StoP10079.33527
ViT-HI-JEPA + StoP + EBReg (ours)10079.41957

Table 2. Linear-probe performance and effective rank on ImageNet-1K. Results are grouped by pretraining setup. ResNet-50, ViT-L, and ViT-H use the setups from Bardes et al. [8], Balestriero & LeCun [4], and Assran et al. [2], respectively. Within each setup, EBReg and the corresponding baselines use the same backbone, data augmentations, optimization settings, training schedule, and evaluation procedure. Effective rank (Garrido et al. [18]) is computed on backbone representations after pretraining, with higher values indicating less dimensional collapse. Bold indicates higher accuracy within each setup, and underline indicates the second-best.

Table 2 shows that EBReg remains competitive in linear evaluation while consistently producing higher-rank representations across architectures. On ResNet-50, EBReg matches the performance of strong regularization-based baselines while yielding the highest effective rank. On ViT-L, it improves both linear-probe accuracy and representation rank relative to SIGReg.When added to StoP on ViT-H, EBReg yields a modest improvement in linear-probe accuracy while substantially increasing effective rank, suggesting that it complements the existing objective while preserving a richer representation space.

Hyperparameter Ablations and Computational Cost

Hyperparameter ablations. We ablate hyperparameter under the setup from Bardes et al. [8] and report ImageNet-1K linear-probe accuracy in Table 3. Among the tested values, the default configuration γ=0.34\gamma= 0.34, σ=4.5\sigma= 4.5, and Kscore=16K_{\mathrm{score}} = 16 yields the highest accuracy. Performance is most sensitive to the noise scale and the number of score updates, while projector dimension has comparatively little effect over the tested range. Increasing score-network width beyond the default also degrades performance, indicating that greater score-model capacity or more frequent score fitting does not necessarily improve representation quality.

FactorValuesAccuracy (%)
Regularization weight γ\gamma0.25 / 0.34 / 0.564.28 67.92 63.38
Noise level σ\sigma4.25 / 4.5 / 5.062.32 67.92 65.43
Score updates KscoreK_{\mathrm{score}}4 / 16 / 3263.72 67.92 58.32
Score-network width1024 / 2048 / 409667.65 67.92 61.35
σ\sigma width0.1 / 0.5 / 0.767.51 67.92 67.43
Projector dimension dd64 / 256 / 51267.42 67.92 67.51

Table 3. ResNet-50 sensitivity: ImageNet-1K linear-probe accuracy, varying one factor at a time. Defaults are underlined.

Computational cost. Table 4 reports the computational cost of EBReg under the setup from Balestriero & LeCun [4] using identical hardware. For a backbone with approximately 300M parameters, the score network adds only 5.21M additional parameters, under 2% of the backbone parameter count. The primary overhead arises from repeatedly updating the score model to track the evolving latent distribution. Relative to SIGReg, this increases wall-clock training time by approximately 60 seconds per epoch, from 440 to 500 seconds, corresponding to a 13.6% increase.

MetricSIGRegEBReg
Extra params (M)05.21
Time / epoch (s)440500
Peak memory (GB)90.3389.84
GPU-hours195.83224.01

Table 4. ViT-L training costs on identical hardware. Extra parameters are relative to the backbone and projector.

The additional computation remains moderate because score fitting is performed entirely in the projected latent space. Its cost is therefore determined primarily by the latent dimension, score-network architecture, and number of score updates, rather than by the image resolution or backbone size. The strong performance obtained with a compact score network may indicate that the latent distribution admits a sufficiently structured score field that can be approximated with limited capacity, although we do not directly characterize its intrinsic dimensionality.

Score-guided test-time adaptation

We evaluate score-guided test-time adaptation on ImageNet-C ([23]) using the EBReg-trained models and pretraining setups from § 4.1. In each case, EBReg training yields a corresponding latent score model, which we reuse at test time as the adaptation signal. The score model and classifier remain fixed, while only the encoder’s normalization parameters are updated using the objective from § 3.3. Table 5 reports accuracy before and after five adaptation steps.

CorruptionViT-L Setup from Balestriero & LeCun [4] BeforeViT-L Setup from Balestriero & LeCun [4] AfterViT-H Setup from Assran et al. [2] BeforeViT-H Setup from Assran et al. [2] AfterResNet-50 Setup from Bardes et al. [8] BeforeResNet-50 Setup from Bardes et al. [8] After
Fog44.1846.3458.4161.4224.0038.60
Impulse noise43.4146.8650.7756.3315.5222.71
Motion blur41.1246.2958.0161.5218.8625.37
Glass blur21.7225.3944.7549.937.7813.59
Zoom blur39.2643.0050.9254.7020.1927.27
Gaussian noise44.5648.2149.1755.9725.7825.99
Contrast57.7958.5060.4463.7040.9444.81
Mean (15 corruptions)46.5548.1857.7961.0528.6730.50

Table 5. ImageNet-C accuracy (%) before and after five adaptation steps. Seven corruptions with gains for every model are selected by mean gain across models; the final row averages all 15 corruptions. Bold indicates higher accuracy within each model.

Score-guided adaptation improves mean accuracy across all three EBReg-trained models and a broad range of corruptions. These gains indicate that the learned score remains informative under distribution shift, guiding corrupted representations toward higher-density regions of the corresponding source latent distribution. Improvements are particularly pronounced for severe noise and blur corruptions, though their magnitude varies across architectures and corruption types.

Conclusion

In this paper, we introduce EBReg, a representation learning framework derived from a lower bound on mutual information. EBReg jointly trains the encoder with a lightweight score network and uses the learned score to counter representation collapse. Our analysis shows that dimension-collapsed solutions are unstable under the proposed objective, which induces a persistent force toward activating collapsed dimensions. Under constraints preventing unbounded feature scaling, EBReg recovers target structures related to existing self-supervised regularizers. Empirically, EBReg consistently learns informative, non-collapsed representations across a range of settings. At test time, the learned score guides corrupted representations toward the learned manifold, improving robustness to noise.

Ethics Statement

This work addresses fundamental research questions in self-supervised representation learning and entropy-based regularization. We do not identify ethical concerns specific to the proposed method or the experiments reported in this paper. As with other representation-learning methods, downstream applications should be assessed for potential bias, privacy implications, and misuse in their intended context.

Reproducibility Statement

Section 3 specifies the training objective, alternating optimization procedure, and test-time adaptation rule, with pseudocode in Appendix C. Section 4 describes the datasets, backbone architectures, baseline protocols, and evaluation settings; Tables 3 and 4 report hyperparameter ablations and computational costs. The derivation of the mutual-information bound, analysis of collapsed directions, and maximum-entropy connections are provided in Appendices A, B, and E, respectively.

References

The smoothed variational bound

For completeness we record the two steps between the data processing inequality and Equation (2); both are the standard variational decomposition of mutual information [6] (Poole et al. 2019), applied to the smoothed variable h~(1)=h(1)+σϵ\tilde{\mathbf{h}}^{(1)} = \mathbf{h}^{(1)} + \sigma\boldsymbol{\epsilon}. Writing the conditional entropy as a cross-entropy and then inserting p=(p/q) qp = (p/q)\,q,

I(h~(1);h(2))=Hσ(h(1))+Eh~(1),h(2)log⁡p(h~(1)∣h(2))=Hσ(h(1))+Eh~(1),h(2)log⁡q(h~(1)∣h(2))+Eh(2)DKL(p(⋅∣h(2))∥q(⋅∣h(2)))≥Hσ(h(1))+Eh~(1),h(2)log⁡q(h~(1)∣h(2)),(11)\begin{aligned} I(\tilde{\mathbf{h}}^{(1)};\mathbf{h}^{(2)}) = H_{\sigma}(\mathbf{h}^{(1)}) + \mathbb{E}_{\tilde{\mathbf{h}}^{(1)},\mathbf{h}^{(2)}} \log p(\tilde{\mathbf{h}}^{(1)} \mid\mathbf{h}^{(2)}) = H_{\sigma}(\mathbf{h}^{(1)}) + \mathbb{E}_{\tilde{\mathbf{h}}^{(1)},\mathbf{h}^{(2)}} \log q(\tilde{\mathbf{h}}^{(1)} \mid\mathbf{h}^{(2)}) + \mathbb{E}_{\mathbf{h}^{(2)}} D_{\mathrm{KL}}\left(p(\cdot\mid\mathbf{h}^{(2)}) \mathbin{\|} q(\cdot\mid\mathbf{h}^{(2)})\right) \\ &\ge H_{\sigma}\left(\mathbf{h}^{(1)}\right)+\mathbb{E}_{\widetilde{\mathbf{h}}^{(1)},\mathbf{h}^{(2)}}\log q\left(\widetilde{\mathbf{h}}^{(1)}\mid\mathbf{h}^{(2)}\right), \tag*{(11)} \end{aligned}

where q(⋅∣h(2))q(\cdot\mid\mathbf{h}^{(2)}) is any conditional density. Both differential entropies are finite because h~(1)\widetilde{\mathbf{h}}^{(1)} has a density for every σ>0\sigma>0 and the embeddings have finite second moments, so the decomposition is valid even when h(1)\mathbf{h}^{(1)} itself is supported on a set of measure zero.

Gaussian variational conditional. For qτ(⋅∣h(2))=N(h(2),τ2Id)q_{\tau}(\cdot\mid\mathbf{h}^{(2)})=\mathcal{N}(\mathbf{h}^{(2)},\tau^{2}I_d), independence and the moments of the Gaussian noise give

−Eh~(1),h(2)log⁡qτ(h~(1)∣h(2))=d2log⁡(2πτ2)+12τ2E∥h(1)+σϵ−h(2)∥22=d2log⁡(2πτ2)+12τ2[E∥h(1)−h(2)∥22+2σE[(h(1)−h(2))⊤ϵ]+σ2E∥ϵ∥22]=12τ2E∥h(1)−h(2)∥22+Cσ,τ,(12)\begin{aligned} -\mathbb{E}_{\widetilde{\mathbf{h}}^{(1)},\mathbf{h}^{(2)}}\log q_{\tau}\left(\widetilde{\mathbf{h}}^{(1)}\mid\mathbf{h}^{(2)}\right)=\frac{d}{2}\log(2\pi\tau^{2})+\frac{1}{2\tau^{2}}\mathbb{E}\left\|\mathbf{h}^{(1)}+\sigma\boldsymbol{\epsilon}-\mathbf{h}^{(2)}\right\|_{2}^{2} \\ &=\frac{d}{2}\log(2\pi\tau^{2})+\frac{1}{2\tau^{2}}\left[\mathbb{E}\left\|\mathbf{h}^{(1)}-\mathbf{h}^{(2)}\right\|_{2}^{2}+2\sigma\mathbb{E}\left[(\mathbf{h}^{(1)}-\mathbf{h}^{(2)})^{\top}\boldsymbol{\epsilon}\right]+\sigma^{2}\mathbb{E}\left\|\boldsymbol{\epsilon}\right\|_{2}^{2}\right] \\ &=\frac{1}{2\tau^{2}}\mathbb{E}\left\|\mathbf{h}^{(1)}-\mathbf{h}^{(2)}\right\|_{2}^{2}+C_{\sigma,\tau}, \tag*{(12)} \end{aligned}

where Cσ,τ=d2log⁡(2πτ2)+dσ22τ2C_{\sigma,\tau}=\frac{d}{2}\log(2\pi\tau^{2})+\frac{d\sigma^{2}}{2\tau^{2}}. The gap between the original MI and the lower bound is the sum of the smoothing loss I(h(1);h(2))−I(h~(1);h(2))I(\mathbf{h}^{(1)};\mathbf{h}^{(2)})-I(\widetilde{\mathbf{h}}^{(1)};\mathbf{h}^{(2)}) and the conditional KL divergence in Equation (11). The variational step is tight when qq equals the true smoothed conditional density almost surely. Under finite second moments, the smoothed MI and the Gaussian conditional KL term are finite, whereas the smoothing loss can be infinite if the original MI is infinite.

Dimensional collapse

Setup and notation. Let h∈Rd\mathbf{h}\in\mathbb{R}^{d} be an embedding with mean μ\boldsymbol{\mu}, covariance Σ=Cov⁡(h)\boldsymbol{\Sigma}=\operatorname{Cov}(\mathbf{h}), and finite second moment. Let ϵ∼N(0,Id)\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},I_d) be independent of h\mathbf{h} and of the input and augmentation that generate it. The noisy embedding is h~=h+σϵ\widetilde{\mathbf{h}}=\mathbf{h}+\sigma\boldsymbol{\epsilon}; when we want to stress that it is a random variable with a density we write z\mathbf{z} for it. Conditionally on h\mathbf{h}, z\mathbf{z} is Gaussian with mean h\mathbf{h} and covariance σ2Id\sigma^{2}I_d, so averaging over h\mathbf{h} gives its density

pσ(z)=cσEh[exp⁡(−∥z−h∥222σ2)],cσ=(2πσ2)−d/2.(13)p_{\sigma}(\mathbf{z})=c_{\sigma}\mathbb{E}_{\mathbf{h}}\left[\exp\left(-\frac{\left\|\mathbf{z}-\mathbf{h}\right\|_{2}^{2}}{2\sigma^{2}}\right)\right],\qquad c_{\sigma}=(2\pi\sigma^{2})^{-d/2}. \tag*{(13)}

This density is smooth and strictly positive for every σ>0\sigma>0, even when h\mathbf{h} itself is discrete or lies on a lower-dimensional set. The score is sσ(z)=∇zlog⁡pσ(z)\mathbf{s}_{\sigma}(\mathbf{z})=\nabla_{\mathbf{z}}\log p_{\sigma}(\mathbf{z}).

Proof of Theorem 1

EBReg approximates the gradient of −Hσ-H_{\sigma}. The entropy objective itself disfavors dimensional collapse, since under a fixed variance budget its unique maximizer is full rank (Appendix § 14, part (i)). That objective-level fact does not describe what the EBReg update does to a collapsed encoder; Theorem 1 does, and we prove it here.

The clean score is blind

Goal. Assuming Var⁡(q⊤h)=0\operatorname{Var}(\mathbf{q}^{\top}\mathbf{h})=0, show that q⊤sσ(h)=0\mathbf{q}^{\top}\mathbf{s}_{\sigma}(\mathbf{h})=0 and q⊤sσ(h~)=−q⊤ϵ/σ\mathbf{q}^{\top}\mathbf{s}_{\sigma}(\widetilde{\mathbf{h}})=-\mathbf{q}^{\top}\boldsymbol{\epsilon}/\sigma, and that the resulting gradient has zero mean. Idea. If the embeddings lie in a subspace, then in the directions orthogonal to it the noisy embedding is pure Gaussian noise, and the score of a Gaussian is known in closed form.

Step 1: the embeddings lie in a subspace. Let S=range⁡(Σ)S=\operatorname{range}(\boldsymbol{\Sigma}) be the subspace the embeddings occupy, r=dim⁡Sr=\dim S, PP the orthogonal projector onto SS, and P⊥=Id−PP_{\perp}=I_d-P the projector onto S⊥S^{\perp}. The collapsed directions are exactly the vectors of S⊥S^{\perp}. For any v∈S⊥\mathbf{v}\in S^{\perp},

Var⁡(v⊤h)=v⊤Σv=0⟹v⊤(h−μ)=0 almost surely.\operatorname{Var}(\mathbf{v}^{\top}\mathbf{h})=\mathbf{v}^{\top}\boldsymbol{\Sigma}\mathbf{v}=0\quad\Longrightarrow\quad\mathbf{v}^{\top}(\mathbf{h}-\boldsymbol{\mu})=0\ \text{almost surely}.

Applying this to a basis of S⊥S^{\perp} gives P⊥(h−μ)=0P_{\perp}(\mathbf{h}-\boldsymbol{\mu})=\mathbf{0} almost surely: every embedding equals the mean plus a vector in SS.

Step 2: split the distance into two orthogonal pieces. Fix a point zz. Insert μ\mu and split z−μz-\mu into its components in SS and S⊥S^{\perp}:

z−h=[P(z−μ)−(h−μ)]⏟∈S by Step 1+P⊥(z−μ)⏟∈S⊥.z-h=\underbrace{\left[P(z-\mu)-(h-\mu)\right]}_{\in S\text{ by Step 1}}+\underbrace{P_{\perp}(z-\mu)}_{\in S^{\perp}}.

The two pieces are orthogonal, so their squared norms add:

∥z−h∥22=∥P(z−μ)−(h−μ)∥22+∥P⊥(z−μ)∥22.\lVert z-h\rVert_{2}^{2}=\left\lVert P(z-\mu)-(h-\mu)\right\rVert_{2}^{2}+\left\lVert P_{\perp}(z-\mu)\right\rVert_{2}^{2}.

Only the first term depends on hh.

Step 3: the density factorizes. Substitute Step 2 into Equation (13). The exponential of a sum is a product, and the factor that does not depend on hh comes out of the expectation:

pσ(z)=pS(P(z−μ))×(2πσ2)−(d−r)/2exp⁡(−∥P⊥(z−μ)∥222σ2).p_{\sigma}(z)=p_{S}\left(P(z-\mu)\right)\times(2\pi\sigma^{2})^{-(d-r)/2}\exp\left(-\frac{\left\lVert P_{\perp}(z-\mu)\right\rVert_{2}^{2}}{2\sigma^{2}}\right).
pS(y):=(2πσ2)−r/2Eh[exp⁡(−∥y−(h−μ)∥222σ2)].p_{S}(y):=(2\pi\sigma^{2})^{-r/2}\mathbb{E}_{h}\left[\exp\left(-\frac{\left\lVert y-(h-\mu)\right\rVert_{2}^{2}}{2\sigma^{2}}\right)\right].

The first factor depends on zz only through its component in SS, and carries everything about the embedding distribution. The second is a Gaussian density in S⊥S^{\perp} with variance σ2\sigma^{2}, and is the same whatever the embeddings do inside SS.

Step 4: the score in the collapsed directions. Take the logarithm and differentiate:

sσ(z)=∇zlog⁡pS(P(z−μ))−P⊥(z−μ)σ2.s_{\sigma}(z)=\nabla_{z}\log p_{S}\left(P(z-\mu)\right)-\frac{P_{\perp}(z-\mu)}{\sigma^{2}}.

By the chain rule the first term equals P∇log⁡pSP\nabla\log p_{S}, so it lies in SS. Applying P⊥P_{\perp} removes it, because P⊥P=0P_{\perp}P=0, and leaves

P⊥sσ(z)=−P⊥(z−μ)σ2for every z∈Rd.(14)P_{\perp}s_{\sigma}(z)=-\frac{P_{\perp}(z-\mu)}{\sigma^{2}}\qquad\text{for every }z\in\mathbb{R}^{d}. \tag*{(14)}

Step 5: evaluate at clean and noisy embeddings. At a clean embedding, Step 1 gives P⊥(h−μ)=0P_{\perp}(h-\mu)=0, so P⊥sσ(h)=0P_{\perp}s_{\sigma}(h)=0. At a noisy embedding,

P⊥(h~−μ)=P⊥(h−μ)⏟=0+σP⊥ϵ⟹P⊥sσ(h~)=−P⊥ϵσ.P_{\perp}(\widetilde{h}-\mu)=\underbrace{P_{\perp}(h-\mu)}_{=0}+\sigma P_{\perp}\epsilon\quad\Longrightarrow\quad P_{\perp}s_{\sigma}(\widetilde{h})=-\frac{P_{\perp}\epsilon}{\sigma}.

For a unit collapsed direction q∈S⊥q\in S^{\perp} we have qTP⊥=qTq^{\mathsf{T}}P_{\perp}=q^{\mathsf{T}}, so taking the inner product with qq gives Equation (9).

Step 6: the gradient through qq has zero mean. The per-sample EBReg loss is γstopgrad⁡(sσ(h~))Th~\gamma\operatorname{stopgrad}(s_{\sigma}(\widetilde{h}))^{\mathsf{T}}\widetilde{h}. The score is detached and the noise does not depend on the parameters, so by the chain rule its parameter gradient is γJθ,ϕ(h)Tsσ(h~)\gamma J_{\theta,\phi}(h)^{\mathsf{T}}s_{\sigma}(\widetilde{h}). The part of this gradient that passes through direction qq is

gq=γJθ,ϕ(h)Tq⏟= aqqTsσ(h~)⏟=−qTϵ/σ=−γσ(qTϵ)aq.g_{q}=\gamma\underbrace{J_{\theta,\phi}(h)^{\mathsf{T}}q}_{=\,a_{q}}\underbrace{q^{\mathsf{T}}s_{\sigma}(\widetilde{h})}_{=-q^{\mathsf{T}}\epsilon/\sigma}=-\frac{\gamma}{\sigma}(q^{\mathsf{T}}\epsilon)a_{q}.

The vector aqa_{q} depends only on the view, while qTϵ∼N(0,1)q^{\mathsf{T}}\epsilon\sim\mathcal{N}(0,1) is independent of it. Therefore

E[gq]=−γσE[qTϵ]E[aq]=0,E∥gq∥22=γ2σ2E[(qTϵ)2]E∥aq∥22=γ2σ2E∥aq∥22.\mathbb{E}[g_{q}]=-\frac{\gamma}{\sigma}\mathbb{E}[q^{\mathsf{T}}\epsilon]\mathbb{E}[a_{q}]=0,\qquad\mathbb{E}\lVert g_{q}\rVert_{2}^{2}=\frac{\gamma^{2}}{\sigma^{2}}\mathbb{E}\left[(q^{\mathsf{T}}\epsilon)^{2}\right]\mathbb{E}\lVert a_{q}\rVert_{2}^{2}=\frac{\gamma^{2}}{\sigma^{2}}\mathbb{E}\lVert a_{q}\rVert_{2}^{2}.

So each sample contributes a nonzero push along qq, but the pushes average to zero: exact collapse is a stationary point of the EBReg term.

PART (B): AMPLIFICATION

Goal. Assuming q⊤h=c+δu\mathbf{q}^{\top}\mathbf{h}=c+\delta u with δ>0\delta>0, where u∼N(0,1)u\sim\mathcal{N}(0,1) is independent of the remaining coordinates (Id−qq⊤)h(I_d-\mathbf{q}\mathbf{q}^{\top})\mathbf{h}, show that the expected EBReg gradient through q\mathbf{q} is a negative multiple of ∇Var⁡(q⊤h)\nabla\operatorname{Var}(\mathbf{q}^{\top}\mathbf{h}). Idea. Along q\mathbf{q} the noisy embedding is an independent one-dimensional Gaussian, whose score is linear; the rest is a covariance computation. Write Θ=(θ,ϕ)\Theta=(\theta,\phi) for the parameters.

Step 1: the noisy coordinate along q\mathbf{q} is an independent Gaussian. Along q\mathbf{q},

q⊤z=c+δu+σq⊤ϵ.\mathbf{q}^{\top}\mathbf{z}=c+\delta u+\sigma\mathbf{q}^{\top}\boldsymbol{\epsilon}.

This is a constant plus two independent Gaussians with variances δ2\delta^2 and σ2\sigma^2, so q⊤z∼N(c,δ2+σ2)\mathbf{q}^{\top}\mathbf{z}\sim\mathcal{N}(c,\delta^2+\sigma^2). It is also independent of the remaining coordinates (Id−qq⊤)z=(Id−qq⊤)h+σ(Id−qq⊤)ϵ(I_d-\mathbf{q}\mathbf{q}^{\top})\mathbf{z}=(I_d-\mathbf{q}\mathbf{q}^{\top})\mathbf{h}+\sigma(I_d-\mathbf{q}\mathbf{q}^{\top})\boldsymbol{\epsilon}: uu is independent of the first term by assumption, and for isotropic Gaussian noise the component q⊤ϵ\mathbf{q}^{\top}\boldsymbol{\epsilon} is independent of the components orthogonal to q\mathbf{q}.

Step 2: the score along q\mathbf{q}. Independence means the density is a product of the one-dimensional Gaussian density of q⊤z\mathbf{q}^{\top}\mathbf{z} and a density of the remaining coordinates. Only the first depends on q⊤z\mathbf{q}^{\top}\mathbf{z}, so

q⊤sσ(z)=∂∂(q⊤z)[−(q⊤z−c)22(δ2+σ2)]=−q⊤z−cδ2+σ2.\mathbf{q}^{\top}s_{\sigma}(\mathbf{z})=\frac{\partial}{\partial(\mathbf{q}^{\top}\mathbf{z})}\left[-\frac{(\mathbf{q}^{\top}\mathbf{z}-c)^2}{2(\delta^2+\sigma^2)}\right]=-\frac{\mathbf{q}^{\top}\mathbf{z}-c}{\delta^2+\sigma^2}.

At z=h~\mathbf{z}=\widetilde{\mathbf{h}} we have q⊤h~−c=δu+σq⊤ϵ\mathbf{q}^{\top}\widetilde{\mathbf{h}}-c=\delta u+\sigma\mathbf{q}^{\top}\boldsymbol{\epsilon}, hence

q⊤sσ(h~)=−δu+σq⊤ϵδ2+σ2.\mathbf{q}^{\top}s_{\sigma}(\widetilde{\mathbf{h}})=-\frac{\delta u+\sigma\mathbf{q}^{\top}\boldsymbol{\epsilon}}{\delta^2+\sigma^2}.

Setting δ=0\delta=0 recovers part (a).

Step 3: the expected gradient through q\mathbf{q}. Multiply by aq\mathbf{a}_{q} and take expectations:

E[aqq⊤sσ(h~)]=−1δ2+σ2(δE[aqu]+σE[aq]E[q⊤ϵ]⏟=0)=−δCov⁡(aq,u)δ2+σ2.\mathbb{E}\left[\mathbf{a}_{q}\mathbf{q}^{\top}s_{\sigma}(\widetilde{\mathbf{h}})\right]=-\frac{1}{\delta^2+\sigma^2}\left(\delta\mathbb{E}[\mathbf{a}_{q}u]+\sigma\mathbb{E}[\mathbf{a}_{q}]\underbrace{\mathbb{E}[\mathbf{q}^{\top}\boldsymbol{\epsilon}]}_{=0}\right)=-\frac{\delta\operatorname{Cov}(\mathbf{a}_{q},u)}{\delta^2+\sigma^2}.

The noise term vanishes because ϵ\boldsymbol{\epsilon} is independent of the view and has zero mean, and E[aqu]=Cov⁡(aq,u)\mathbb{E}[\mathbf{a}_{q}u]=\operatorname{Cov}(\mathbf{a}_{q},u) because E[u]=0\mathbb{E}[u]=0. The noise no longer cancels everything: the embedding’s own variation δu\delta u correlates with how the encoder responds.

Step 4: the gradient of the variance. Differentiate Var⁡(q⊤hΘ)=E[(q⊤hΘ−E[q⊤hΘ])2]\operatorname{Var}(\mathbf{q}^{\top}\mathbf{h}_{\Theta})=\mathbb{E}[(\mathbf{q}^{\top}\mathbf{h}_{\Theta}-\mathbb{E}[\mathbf{q}^{\top}\mathbf{h}_{\Theta}])^2] under the expectation, which E∥aq∥22<∞\mathbb{E}\|\mathbf{a}_{q}\|_2^2<\infty and local domination of the parameter derivative permit. Using ∇Θ(q⊤hΘ)=aq\nabla_{\Theta}(\mathbf{q}^{\top}\mathbf{h}_{\Theta})=\mathbf{a}_{q},

∇ΘVar⁡(q⊤hΘ)=2E[(q⊤h−E[q⊤h])(aq−E[aq])]=2Cov⁡(aq,q⊤h)=2δCov⁡(aq,u),\nabla_{\Theta}\operatorname{Var}(\mathbf{q}^{\top}\mathbf{h}_{\Theta})=2\mathbb{E}\left[(\mathbf{q}^{\top}\mathbf{h}-\mathbb{E}[\mathbf{q}^{\top}\mathbf{h}])(\mathbf{a}_{q}-\mathbb{E}[\mathbf{a}_{q}])\right]=2\operatorname{Cov}(\mathbf{a}_{q},\mathbf{q}^{\top}\mathbf{h})=2\delta\operatorname{Cov}(\mathbf{a}_{q},u),

where the last step uses q⊤h=c+δu\mathbf{q}^{\top}\mathbf{h}=c+\delta u.

Step 5: combine. Steps 3 and 4 contain the same covariance. Eliminating it,

γE[aqq⊤sσ(h~)]=−γδCov⁡(aq,u)δ2+σ2=−γ2(δ2+σ2)∇ΘVar⁡(q⊤h),\gamma\mathbb{E}\left[\mathbf{a}_{q}\mathbf{q}^{\top}s_{\sigma}(\widetilde{\mathbf{h}})\right]=-\frac{\gamma\delta\operatorname{Cov}(\mathbf{a}_{q},u)}{\delta^2+\sigma^2}=-\frac{\gamma}{2(\delta^2+\sigma^2)}\nabla_{\Theta}\operatorname{Var}(\mathbf{q}^{\top}\mathbf{h}),

which is (10). □

Consequence: exponential growth. Gradient descent subtracts η\eta times the gradient, so the contribution of this term to the update is ΔΘ=ηγ2(δ2+σ2)∇ΘVar⁡(q⊤h)\Delta\Theta=\frac{\eta\gamma}{2(\delta^2+\sigma^2)}\nabla_{\Theta}\operatorname{Var}(\mathbf{q}^{\top}\mathbf{h}), a step up the variance. To first order in η\eta the variance changes by

ΔVar⁡(q⊤h)=∇ΘVar⁡⊤ΔΘ+O(η2)=ηγ2(δ2+σ2)∥∇ΘVar⁡(q⊤h)∥22+O(η2).\Delta\operatorname{Var}(\mathbf{q}^{\top}\mathbf{h})=\nabla_{\Theta}\operatorname{Var}^{\top}\Delta\Theta+O(\eta^2)=\frac{\eta\gamma}{2(\delta^2+\sigma^2)}\left\|\nabla_{\Theta}\operatorname{Var}(\mathbf{q}^{\top}\mathbf{h})\right\|_2^2+O(\eta^2).

Insert ∥∇ΘVar⁡∥22=4δ2∥Cov⁡(aq,u)∥22\|\nabla_{\Theta}\operatorname{Var}\|_2^2=4\delta^2\|\operatorname{Cov}(\mathbf{a}_{q},u)\|_2^2 from Step 4 and Var⁡(q⊤h)=δ2\operatorname{Var}(\mathbf{q}^{\top}\mathbf{h})=\delta^2:

ΔVar⁡(q⊤h)=κVar⁡(q⊤h)+O(η2),κ=2ηγ∥Cov⁡(aq,u)∥22δ2+σ2→δ→02ηγ∥Cov⁡(aq,u)∥22σ2.\Delta\operatorname{Var}(\mathbf{q}^{\top}\mathbf{h})=\kappa\operatorname{Var}(\mathbf{q}^{\top}\mathbf{h})+O(\eta^2),\qquad\kappa=\frac{2\eta\gamma\|\operatorname{Cov}(\mathbf{a}_{q},u)\|_2^2}{\delta^2+\sigma^2}\xrightarrow{\delta\to0}\frac{2\eta\gamma\|\operatorname{Cov}(\mathbf{a}_{q},u)\|_2^2}{\sigma^2}.

The variance therefore grows by a fixed fraction of itself per step, which is exponential growth at a rate proportional to γ/σ2\gamma/\sigma^2. The drift is first order in η\eta, does not depend on the batch size, and vanishes as δ→0\delta\to0, consistent with part (a). Starting from a seed δ0\delta_0, reaching a fixed scale takes on the order of log⁡(1/δ0)/κ\log(1/\delta_0)/\kappa steps, so the size of the seed matters only logarithmically.

EBReg training procedure

Algorithms 1 and 2 detail the EBReg training procedure and score-guided test-time adaptation described in Sections 3.2 and 3.3, respectively.

Input: normalized encoder–projector gϕ∘fθg_{\phi} \circ f_{\theta}, noise predictor ϵ^ψ\hat{\epsilon}_{\psi}, augmentation distribution T\mathcal{T}; B,T,Kscore,σ,γB, T, K_{\mathrm{score}}, \sigma, \gamma.

Notation: sg = stopgrad. Score-fitting and regularization means pool the 2B2B embeddings; alignment uses the BB matched pairs as in (5). Each noise draw is independent and standard Gaussian.

  1. for r=1,…,Tr = 1,\ldots,T do

  2. Sample a minibatch and paired views x(1),x(2)x^{(1)}, x^{(2)} using T\mathcal{T}.

  3. hˉ←sg⁡(gϕ(fθ(x(1:2))))\bar{h} \leftarrow\operatorname{sg}(g_{\phi}(f_{\theta}(x^{(1:2)}))).

  4. repeat KscoreK_{\mathrm{score}} times

  5. Draw noise ϵ\epsilon; h~←hˉ+σϵ\tilde{h} \leftarrow\bar{h} + \sigma\epsilon.

  6. Update only ψ\psi on mean ∥ϵ^ψ(h~)−ϵ∥22\lVert\hat{\epsilon}_{\psi}(\tilde{h})-\epsilon\rVert_{2}^{2}.

  7. end repeat

  8. Recompute h←gϕ(fθ(x(1:2)))h \leftarrow g_{\phi}(f_{\theta}(x^{(1:2)})) on the same views with gradients; draw fresh ϵ\epsilon.

  9. h~←h+σϵ\tilde{h} \leftarrow h + \sigma\epsilon; s←sg⁡(−ϵ^ψ(h~)/σ)s \leftarrow\operatorname{sg}(-\hat{\epsilon}_{\psi}(\tilde{h})/\sigma).

  10. Update only θ,ϕ\theta,\phi on Lalign+γmean⁡(sTh~)\mathcal{L}_{\mathrm{align}}+\gamma\operatorname{mean}(s^{\mathsf{T}}\tilde{h}), using (5).

  11. end for; return gϕ∘fθg_{\phi} \circ f_{\theta} and sψ,σs_{\psi,\sigma}.

Algorithm 1: EBReg training

Input: EBReg-trained encoder fθf_{\theta}, fixed projector gϕg_{\phi}, score model sψ,σs_{\psi,\sigma}, classifier cc; KttaK_{\mathrm{tta}} and adaptation learning rate η\eta.

Notation: α⊂θ\alpha\subset\theta denotes the encoder’s normalization parameters and sg = stopgrad. The projector, score model, classifier, and all encoder parameters outside α\alpha remain fixed.

  1. for each test minibatch xx do

  2. Reset α\alpha to its pretrained source values.

  3. repeat KttaK_{\mathrm{tta}} times

  4. Compute encoder representation rα←fθ,α(x)r_{\alpha} \leftarrow f_{\theta,\alpha}(x).

  5. Compute score-space embedding hα←gϕ(rα)h_{\alpha} \leftarrow g_{\phi}(r_{\alpha}).

  6. s←sg⁡(sψ,σ(hα))s \leftarrow\operatorname{sg}(s_{\psi,\sigma}(h_{\alpha})).

  7. LTTA←−mean⁡(sThα)\mathcal{L}_{\mathrm{TTA}} \leftarrow-\operatorname{mean}(s^{\mathsf{T}}h_{\alpha}).

  8. Update only α\alpha on LTTA\mathcal{L}_{\mathrm{TTA}} with learning rate η\eta.

  9. end repeat

  10. Recompute rα←fθ,α(x)r_{\alpha} \leftarrow f_{\theta,\alpha}(x) and predict y^←c(rα)\hat{y} \leftarrow c(r_{\alpha}).

  11. end for

Algorithm 2: Score-guided test-time adaptation

Implementation Details

In this section, we elaborate on various hyperparameter and design choices for our experiments. We use ImageNet-1k for all our experiments, using the standard train, validation sets comprising 1.28M and 50K images, respectively. We use ResNet-50, ViT-H, and ViT-L as the major architectural choices for our study. The augmentation pipeline matches those used in the corresponding protocols. We use a simple MLP as a denoiser in all the cases. We use a simple MLP projector for both VICReg and LeJEPA cases with projector dims of 256 and 64, respectively. Other hyperparameters for each of the protocols are listed below:

VICReg protocol. The projector has layer widths 8192–8192–256. The denoiser uses a hidden width of 2048 and four blocks. The alignment-loss coefficient is 1.

LeJEPA protocol. The augmentation pipeline produces two global crops of size 224 ×\times 224 with scale range [0.3, 1.0] and eight local crops of size 98 ×\times 98 with scale range [0.05, 0.3]. Images use ImageNet channel-wise normalization with mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225). The learning-rate warmup starts at 0.01 times the configured learning rate. Training uses bfloat16 precision.

I-JEPA/StoP protocol. The input crop size is 224×224224 \times224, with crop scale range [0.3,1.0][0.3, 1.0] and patch size 14. We use one context mask with scale range [0.85,1.0][0.85, 1.0] and four target masks with scale range [0.15,0.2][0.15, 0.2], mask aspect ratios in [0.75,1.5][0.75, 1.5], and a minimum of 10 retained patches. The predictor has depth 12 and embedding dimension 384. The initial and final learning rates are 2×10−42 \times10^{-4} and 10−610^{-6}, respectively; the configuration specifies constant learning-rate and weight-decay strategies from epoch 250. StoP is enabled with variance parameter 0.2.

Maximum-entropy targets under each constraint

We compute the maximum-entropy distribution under each constraint of Table 1, labelled (i)–(iv) in the order of the table. All four follow from one standard computation.

The maximum-entropy recipe. Consider distributions on a support SS that satisfy moment constraints E[fk(h)]=ck\mathbb{E}[f_k(\mathbf{h})] = c_k for k=1,…,mk = 1, \ldots, m. Suppose the exponential-family density

p∗(h)=1Zexp⁡(−∑k=1mλkfk(h)) on S(15)p^*(\mathbf{h}) = \frac{1}{Z} \exp\left(-\sum_{k=1}^{m} \lambda_k f_k(\mathbf{h})\right) \text{ on } S \tag*{(15)}

satisfies the constraints for some multipliers λk\lambda_k. Then p∗p^* is the unique maximum-entropy distribution. To see this, take any feasible pp. Because −log⁡p∗=log⁡Z+∑kλkfk-\log p^* = \log Z + \sum_k \lambda_k f_k, its average under pp involves only the constrained moments:

−∫plog⁡p∗=log⁡Z+∑kλkEp[fk(h)]=log⁡Z+∑kλkck=H(p∗).-\int p \log p^* = \log Z + \sum_k \lambda_k \mathbb{E}_p[f_k(\mathbf{h})] = \log Z + \sum_k \lambda_k c_k = H(p^*).

Hence

H(p)=−∫plog⁡p∗−DKL(p∥p∗)=H(p∗)−DKL(p∥p∗)≤H(p∗).H(p) = -\int p \log p^* - D_{\mathrm{KL}}(p \mathbin\Vert p^*) = H(p^*) - D_{\mathrm{KL}}(p \mathbin\Vert p^*) \le H(p^*).

with equality only when p=p∗p = p^*. With no moment constraints, m=0m = 0, the recipe gives the constant density, that is, the uniform distribution on SS.

(i) Quadratic constraint: isotropic Gaussian. Let E[h]=0\mathbb{E}[\mathbf{h}] = 0 and tr⁡Cov⁡(h)=d\operatorname{tr}\operatorname{Cov}(\mathbf{h}) = d; unit marginal variances are a special case. The noisy embedding z=h+σϵ\mathbf{z} = \mathbf{h} + \sigma\boldsymbol{\epsilon} then satisfies

E[z]=0,E∥z∥22=E∥h∥22+σ2E∥ϵ∥22=d(1+σ2).\mathbb{E}[\mathbf{z}] = 0,\qquad\mathbb{E}\lVert\mathbf{z}\rVert_2^2 = \mathbb{E}\lVert\mathbf{h}\rVert_2^2 + \sigma^2\mathbb{E}\lVert\boldsymbol{\epsilon}\rVert_2^2 = d(1+\sigma^2).

Apply the recipe to z\mathbf{z} on S=RdS = \mathbb{R}^d with f(z)=∥z∥22f(\mathbf{z}) = \lVert\mathbf{z}\rVert_2^2. It gives p∗(z)∝exp⁡(−λ∥z∥22)p^*(\mathbf{z}) \propto\exp(-\lambda\lVert\mathbf{z}\rVert_2^2), a zero-mean isotropic Gaussian, and matching the second moment fixes its variance at 1+σ21+\sigma^2 per coordinate. Its entropy is the bound

Hσ(h)=H(z)≤H(N(0,(1+σ2)Id))=d2log⁡(2πe(1+σ2)),(16)H_\sigma(\mathbf{h}) = H(\mathbf{z}) \le H\left(\mathcal{N}\left(0,(1+\sigma^2)I_d\right)\right) = \frac{d}{2}\log\left(2\pi e(1+\sigma^2)\right), \tag*{(16)}

with equality only when z∼N(0,(1+σ2)Id)\mathbf{z} \sim\mathcal{N}\left(0,(1+\sigma^2)I_d\right). This happens exactly when h∼N(0,Id)\mathbf{h} \sim\mathcal{N}(0,I_d): adding the independent noise σϵ\sigma\boldsymbol{\epsilon} to N(0,Id)\mathcal{N}(0,I_d) gives that Gaussian, and conversely the characteristic function of z\mathbf{z} is the product φh(ω)exp⁡(−12σ2∥ω∥22)\varphi_{\mathbf{h}}(\boldsymbol{\omega})\exp\left(-\frac{1}{2}\sigma^2\lVert\boldsymbol{\omega}\rVert_2^2\right), so dividing by the noise factor leaves φh(ω)=exp⁡(−12∥ω∥22)\varphi_{\mathbf{h}}(\boldsymbol{\omega}) = \exp\left(-\frac{1}{2}\lVert\boldsymbol{\omega}\rVert_2^2\right). The result holds for every σ>0\sigma> 0. Since the maximizer has Cov⁡(h)=Id\operatorname{Cov}(\mathbf{h}) = I_d, a rank-deficient representation is never entropy-maximal under this constraint. The target N(0,Id)\mathcal{N}(0,I_d) is SIGReg’s reference (Balestiero & LeCun, 2025), its covariance is the second-moment condition that VICReg’s penalties drive toward ([8]), and batch normalization without affine parameters enforces the constraint up to finite-batch error.

(ii) Spherical constraint: uniform on the sphere. Let ∥h∥2=1\lVert\mathbf{h}\rVert_2 = 1, so S=Sd−1S = \mathbb{S}^{d-1} and there is no moment constraint. The recipe gives the constant density with respect to surface measure, the uniform distribution, with entropy log⁡∣Sd−1∣\log|\mathbb{S}^{d-1}|. The same holds on a sphere of any radius, such as ∥h∥2=d\lVert\mathbf{h}\rVert_2 = \sqrt{d} under RMS normalization. Layer normalization additionally centers each embedding, 1Th=0\mathbf{1}^{\mathsf{T}}\mathbf{h} = 0, so SS is the sphere inside the hyperplane orthogonal to 1\mathbf{1} and the target is uniform on that lower-dimensional sphere. Spherical uniformity is the target that contrastive objectives approach through negatives ([41]).

(iii) Categorical constraint: uniform code usage. Let h∈{e1,…,eK}\mathbf{h} \in\{e_1,\ldots,e_K\}, so SS is a set of KK points with counting measure and again no moment constraint. The recipe gives equal probabilities 1/K1/K, with entropy log⁡K\log K. Uniform usage is SwAV’s equipartition constraint and the marginal-entropy regularizer of MSN [11, 1]; DINO’s centering and sharpening act as a related mechanism rather than a prescribed density [12]. Marginal balance across inputs differs from uninformative uniform predictions for every input.

(iv) Moment constraint: generalized Gaussian. Let E∥h∥p=M\mathbb{E}\lVert\mathbf{h}\rVert_p = M on a support SS. The recipe with f(h)=∥h∥ppf(\mathbf{h})=\lVert\mathbf{h}\rVert_p^p gives

p∗(h)=Z−1exp⁡(−α∥h∥pp)1S(h),(17)p^*(\mathbf{h}) = Z^{-1}\exp(-\alpha\lVert\mathbf{h}\rVert_p^p)\mathbb{1}_S(\mathbf{h}), \tag*{(17)}

with α\alpha chosen to satisfy the constraint, whenever this density is normalizable. On Rd\mathbb{R}^d it is a product generalized Gaussian, which reduces to the Gaussian of part (i) at p=2p=2, and restricting to the positive orthant gives its truncation. This is the principle behind Rectified LpJEPA’s target design [27], whose additional rectification introduces atoms at zero and requires mixed-measure entropy.

Effect of the smoothing. Part (i) is computed directly for the smoothed entropy HσH_\sigma and is exact at every noise level. Parts (ii)–(iv) are computed for the entropy of h\mathbf{h} itself, which HσH_\sigma approaches from above as σ→0\sigma\to0: adding independent noise can only add entropy, Hσ(h)≥H(h)H_\sigma(\mathbf{h}) \ge H(\mathbf{h}), and the excess vanishes with σ\sigma. For the sphere and the codes the targets remain maximizers of HσH_\sigma at every σ>0\sigma> 0 by symmetry. Rotating a distribution on the sphere, or permuting the codes, leaves HσH_\sigma unchanged because the noise is isotropic; the uniform distribution is the average of all such rotated or permuted copies; and the entropy of a mixture is at least the average entropy of its components. For the moment constraint with p≠2p \ne2 the maximizer of HσH_\sigma at fixed σ>0\sigma> 0 is a smoothed analogue of Equation (17) that we do not characterize.

Use of Large Language Models

LLMs were used to assist with writing, revising, organizing, and polishing the manuscript; coding tasks; table and bibliography formatting; and LaTeX troubleshooting. They also assisted in deriving proofs presented in the appendix, reviewing mathematical arguments, identifying gaps between stated assumptions and claimed conclusions, and drafting revisions to narrow the interpretation of the directional gradient results. All experimental values used in preparing tables and discussing results were supplied by the authors. The authors take full responsibility for the manuscript’s correctness, originality, and proper attribution, including all LLM-assisted text, code, proofs, and mathematical reasoning.

References

  1. [1]Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In ECCV, 2022.arxiv.org/abs/2204.07141
  2. [2]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, 2023.arxiv.org/abs/2301.08243
  3. [3]Randall Balestriero and Yann LeCun. How learning by reconstruction produces uninformative features for perception. In ICML, 2024.arxiv.org/abs/2402.11337
  4. [4]Randall Balestriero and Yann LeCun. LeJEPA: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025.arxiv.org/abs/2511.08544
  5. [5]Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT pre-training of image transformers. In ICLR, 2022.arxiv.org/abs/2106.08254
  6. [6]David Barber and Felix Agakov. The IM algorithm: A variational approach to information maximization. In NeurIPS, 2003a.
  7. [7]David Barber and Felix Agakov. The im algorithm: a variational approach to information maximization. In Proceedings of the 17th International Conference on Neural Information Processing Systems, 2003b.
  8. [8]Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In ICLR, 2022.arxiv.org/abs/2105.04906
  9. [9]Suzanna Becker and Geoffrey E. Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 1992.pubmed.ncbi.nlm.nih.gov/1729650
  10. [11]Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.arxiv.org/abs/2006.09882
  11. [12]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. Preprint. Under review.
  12. [13]Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In ICML, 2023.arxiv.org/abs/2302.07194
  13. [14]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.arxiv.org/abs/2002.05709
  14. [15]Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning. In ICLR, 2025.arxiv.org/abs/2401.14404
  15. [16]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009.
  16. [17]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
  17. [18]Quentin Garrido, Randall Balestiero, Laurent Najman, and Yann LeCun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. In ICML, 2023.arxiv.org/abs/2210.02885
  18. [19]Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS, 2020.arxiv.org/abs/2006.07733
  19. [20]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.arxiv.org/abs/1512.03385
  20. [21]Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.arxiv.org/abs/1911.05722
  21. [22]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.arxiv.org/abs/2111.06377
  22. [23]Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
  23. [24]Geoffrey E. Hinton and Ruslan R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 2006.pubmed.ncbi.nlm.nih.gov/16873662
  24. [25]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.arxiv.org/abs/2006.11239
  25. [26]Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.arxiv.org/abs/2110.09348
  26. [27]Yilun Kuang, Yash Dagade, Tim G. J. Rudner, Randall Balestiero, and Yann LeCun. Rectified LpJEPA: Joint-embedding predictive architectures with sparse and maximum-entropy representations. arXiv preprint arXiv:2602.01456, 2026.arxiv.org/abs/2602.01456
  27. [28]Yingzhen Li and Richard E. Turner. Gradient estimators for implicit models. In ICLR, 2018.arxiv.org/abs/1705.07107
  28. [29]Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-JEPA 2.1: Unlocking dense features in video self-supervised learning. arXiv preprint arXiv:2603.14482, 2026.
  29. [30]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeddin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. TMLR, 2024. Ben Poole, Sheril Ozair, Aaron van den Oord, Alexander A. Alemi, and George Tucker. On variational bounds of mutual information. In ICML, 2019.
  30. [31]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.arxiv.org/abs/2103.00020
  31. [32]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.arxiv.org/abs/2112.10752
  32. [33]Jürgen Schmidhuber and Daniel Prelinger. Discovering predictable classifications. Neural Computation, 1993.DOI
  33. [34]Jiaxin Shi, Shengyang Sun, and Jun Zhu. A spectral approach to gradient estimation for implicit distributions. In ICML, 2018.arxiv.org/abs/1806.02925
  34. [35]Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
  35. [36]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurIPS, 2019.arxiv.org/abs/1907.05600
  36. [37]Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
  37. [38]Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 2011.DOI
  38. [39]Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008.DOI
  39. [40]Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.arxiv.org/abs/2006.10726
  40. [41]Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.arxiv.org/abs/2005.10242
  41. [42]Chao-Yuan Wu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühl. Sampling matters in deep embedding learning. In ICCV, 2017.arxiv.org/abs/1706.07567
  42. [43]Haiyu Wu, Randall Balestriero, and Morgan Levine. VISReg: Variance-invariance-sketching regularization for JEPA training. arXiv preprint arXiv:2606.02572, 2026.arxiv.org/abs/2606.02572
  43. [44]Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In ICLR, 2025.arxiv.org/abs/2410.06940
  44. [45]Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders. In ICLR, 2026.arxiv.org/abs/2510.11690

Paper details

vlnc:W_4bjtrrbh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

LLMs

View statement
Looking for related papers…
Contents