betaFeedback

WorksMathematics

Contrastive latent representation learning with denoising score matching

Abstract

Contrastive representation learning methods operate by pulling different views of the same data sample together while pushing them away from views of different samples. A fundamental challenge is the estimation of the repulsive force, which typically requires large batch sizes, can exhibit high variance, and is susceptible to the curse of dimensionality. We introduce ScoreCLR, a contrastive representation learning method that offers a novel way to estimate this repulsion term. We consider the framework of representation learning by maximizing the mutual information between two latent views of a sample. By deriving the gradient of the Shannon entropy with respect to the encoder parameters, we obtain an equivalent objective that involves the score of the latent representation. We then use denoising score matching techniques to estimate this score with a lightweight network that is trained jointly with the encoder. We further propose regularization techniques using different noise levels and a Gaussian prior on the latent representation. ScoreCLR replaces the batch of explicit negative samples in contrastive methods with a learned repulsion whose cost is linear in the batch size rather than quadratic. On CIFAR-10 and ImageNet-100 it does so with minimal loss of linear-probe accuracy compared to SimCLR under a shared recipe, while producing projector outputs of higher effective rank, closer to the Gaussian prior.

Introduction

The goal of representation learning is to learn features from data that make downstream tasks easier to solve [22]. It is now widely used across several machine learning domains, including generative modeling [48], reinforcement learning [26], and causality [49]. Representation learning methods can be broadly divided into two categories: reconstructive approaches [56, 33, 28], which train by reconstructing samples from their latent encodings, and non-reconstructive ones [9, 7, 1], which optimize a training criterion in the latent space.

For the latter, two main approaches are typically considered. Contrastive methods [54, 9] bring representations of different views of the same sample closer while separating those of different samples. Estimating the repulsion force is generally challenging: it relies on in-batch negatives whose cost grows quadratically with the batch size, suffers from high variance and bias [45], and requires careful hyperparameter tuning (Chen et al., 2021). This has led to the development of non-contrastive methods that focus solely on aligning latent representations and prevent collapse through architectural design and training asymmetry [25, 7, 1] (Chen & He 2021) or explicit regularization in the training objective [4, 3]. Although these techniques do prevent collapse [44], it is unclear which structure they impose on the latent space, as well as the efficiency of the estimator they rely on (see § 3.4).

We introduce ScoreCLR, a novel method for contrastive representation learning that estimates the repulsion term using denoising score matching techniques [55, 52]. Our training criterion derives from maximizing the mutual information between two views of a sample [53]. By deriving the gradient of the Shannon entropy with respect to the encoder parameters, we construct an equivalent loss, sharing the same gradient, that involves the score of the latent representation ∇zlog⁡pZ(z)\nabla_z \log p_Z(z). We then learn this score across different noise levels using a lightweight score network trained jointly with the encoder following the methodology of Yin et al. [59]. We call this score-based estimate of the entropy term of the mutual information ScoreReg, by analogy with VICReg [4] and SIGReg [3].

Unlike previous contrastive methods [9, 27] whose loss estimation is quadratic in the batch size, ours is linear: the repulsion is amortized in a score network instead of being computed against a batch (Figure 1). Empirically, ScoreCLR reaches competitive linear-probe accuracy on CIFAR-10 and ImageNet-100, while yielding projector outputs of higher effective rank than SimCLR and LeJEPA, which span only a fraction of the dimensions. The code is available at https://github.com/chris-mrn/ScoreCLR_code.

Figure 1. Repulsion force acting on a latent code zz (negative gradient of the loss). SimCLR (dashed) estimates it against the BB codes of the batch, with a cost O(B2)O(B^2) per batch, whereas ScoreCLR (solid) uses the score −∇zlog⁡pZ(z)-\nabla_z \log p_Z(z) of the latent distribution, learned by a small network, with a cost O(B)O(B).

Representation learning via mutual information maximization

Let XX denote the random variable representing the data. We consider learning representations by learning to distinguish the joint distribution of two different latent views of XX from the product of their marginals. More formally, let TT denote a stochastic transformation of the data (e.g., cropping, masking, blurring, flipping, etc.) and T1T_1, T2T_2 two i.i.d. copies of TT. We will consider the latent representation to be Z=G(T(X))Z = G(T(X)), where GG is an encoding function and denote Z1=G(T1(X))Z_1 = G(T_1(X)) and Z2=G(T2(X))Z_2 = G(T_2(X)) two latent views of XX. The objective is to maximize

D(pZ1,Z2∥pZ1pZ2)D(p_{Z_1,Z_2}\mathbin\Vert p_{Z_1}p_{Z_2})

where DD is a divergence criterion. Several divergences have been explored in the literature for learning representations such as Wasserstein [42], Maximum Mean Discrepancy (MMD) [23, 38], and Jensen–Shannon [29]. In our case we will consider the Kullback–Leibler (KL) divergence that inherits from desirable statistical properties [14]. We are then in the InfoMax principle (Linkser, 1988) framework and our objective is to maximize

I(Z1,Z2)=KL⁡(p(Z1,Z2)∥pZ1pZ2)=H(Z1)−H(Z1∣Z2).I(Z_1,Z_2) = \operatorname{KL}(p_{(Z_1,Z_2)} \mathbin{\|} p_{Z_1}p_{Z_2}) = H(Z_1) - H(Z_1 \mid Z_2).

Using this decomposition, we can interpret the objective as simultaneously encouraging the representation to have high entropy and making it predictable from the other representation view, through minimizing the conditional entropy H(Z1∣Z2)H(Z_1 \mid Z_2). We will parametrize the encoding function with a deep neural network GφG^\varphi, denote Z1φ=Gφ(T1(X))Z_1^\varphi= G^\varphi(T_1(X)), Z2φ=Gφ(T2(X))Z_2^\varphi= G^\varphi(T_2(X)), and aim at minimizing

LMI(φ)=H(Z1φ∣Z2φ)−H(Z2φ).\mathcal{L}_{\mathrm{MI}}(\varphi) = H(Z_1^\varphi\mid Z_2^\varphi) - H(Z_2^\varphi).

We will now present different ways to estimate the mutual information loss that can be found in the literature.

Variational bound and kernel density estimator

A common way to construct an estimator is to consider a lower bound using a dual representation of the KL. A standard choice is the Donsker–Varadhan representation, where the supremum is taken over measurable functions ff such that both expectations below are finite, which gives

KL⁡(p(Z1,Z2)∥pZ1pZ2)=sup⁡f{E[f(Z1,Z2)]−log⁡E[ef(Z1,Z~2)]}.\operatorname{KL}(p_{(Z_1,Z_2)} \mathbin{\|} p_{Z_1}p_{Z_2}) = \sup_f \left\{ \mathbb{E}[f(Z_1,Z_2)] - \log\mathbb{E}\left[e^{f(Z_1,\widetilde{Z}_2)}\right] \right\}.

where Z~2=Gφ(T2(X~))\widetilde{Z}_2 = G^\varphi(T_2(\widetilde{X})) with X~\widetilde{X} independent of XX, so that (Z1,Z~2)∼pZ1pZ2(Z_1,\widetilde{Z}_2) \sim p_{Z_1}p_{Z_2}. Any ff thus gives a lower bound on the mutual information. Let fτ(x,y)=1τxTy∥x∥∥y∥f_\tau(x,y) = \frac{1}{\tau}\frac{x^\mathsf{T}y}{\lVert x\rVert\lVert y\rVert}. We consider the critic

f(z1,z2)=fτ(z1,z2)−log⁡EZ~2[efτ(z1,Z~2)].f(z_1,z_2) = f_\tau(z_1,z_2) - \log\mathbb{E}_{\widetilde{Z}_2}\left[e^{f_\tau(z_1,\widetilde{Z}_2)}\right].

Since Z~2\widetilde{Z}_2 is independent of Z1Z_1, EZ1,Z~2[ef(Z1,Z~2)]=1\mathbb{E}_{Z_1,\widetilde{Z}_2}\left[e^{f(Z_1,\widetilde{Z}_2)}\right] = 1, and the logarithm term of the Donsker–Varadhan bound vanishes. Using Monte Carlo estimators of the expectations with i.i.d. samples (Z1i,Z2i)i∈{1,…,B}(Z_1^i,Z_2^i)_{i\in\{1,\ldots,B\}} of (Z1,Z2)(Z_1,Z_2) and (Z~2j)j∈{1,…,B}(\widetilde{Z}_2^j)_{j\in\{1,\ldots,B\}} of Z~2\widetilde{Z}_2, we obtain the loss

L=−1B∑i=1Bfτ(Z1i,Z2i)+1B∑i=1Blog⁡(1B∑j=1Bexp⁡(fτ(Z1i,Z~2j))).\mathcal{L} = -\frac{1}{B}\sum_{i=1}^{B} f_\tau(Z_1^i,Z_2^i) + \frac{1}{B}\sum_{i=1}^{B}\log\left(\frac{1}{B}\sum_{j=1}^{B}\exp\left(f_\tau(Z_1^i,\widetilde{Z}_2^j)\right)\right).

This loss is closely related to the InfoNCE loss of van den Oord et al. [54], the difference being that the positive pair (Z1i,Z2i)(Z_1^i,Z_2^i) does not appear in the denominator, as in decoupled contrastive learning [58].

Up to an additive constant, the repulsion term is minus a plug-in kernel density estimator of the entropy of the normalized latent on the sphere, with the kernel kτ(x,y)=exp⁡(fτ(x,y))k_\tau(x,y) = \exp(f_\tau(x,y)). Minimizing it encourages the latent representation to have high entropy, which favors uniformity on the sphere [57].

The SimCLR loss [9] is a close variant, which symmetrizes the loss with respect to the two views, uses the other samples of the batch from both views as negatives, and includes the positive pair in the denominator. Other works modify the samples used in this estimator. NNCLR [19] replaces Z1iZ_1^i by its nearest neighbour in a queue of embeddings stored from previous batches, and MoCo [27] draws negatives from a dynamic memory queue, which decouples the number of negatives from the batch size and enables the use of smaller batch sizes.

These estimators all suffer from the curse of dimensionality under standard regularity conditions on the density of interest [50, 37]. The relevant dimension is in general the intrinsic dimension of the data rather than the ambient one, but it has been recently suggested that this intrinsic dimension is still too large for standard statistical estimators on real-world data [46]. Contrary to classic kernel density estimators, which only leverage a weak prior on the distribution of interest, the ScoreCLR repulsion estimator includes a neural prior that may benefit from a better sample complexity [5].

Neural methods

Several approaches have been proposed to estimate or maximize mutual information in representation learning relying on neural critics. For example, MINE [6] used the dual representation of the KL with a critic learned by a neural network fψf_\psi, which was subsequently used for unsupervised representation learning in Deep InfoMax [29]. The mutual-information-based loss can be rewritten as

L(φ)=sup⁡ψ{E[fψ(Z1φ,Z2φ)]−log⁡E[efψ(Z1φ,Z~2φ)]}.\mathcal{L}(\varphi)=\sup_{\psi}\left\{\mathbb{E}\left[f_\psi\left(Z_1^\varphi,Z_2^\varphi\right)\right]-\log\mathbb{E}\left[e^{f_\psi\left(Z_1^\varphi,\widetilde{Z}_2^\varphi\right)}\right]\right\}.

where Z~2φ=Gφ(T2(X~))\widetilde{Z}_2^\varphi=G^\varphi(T_2(\widetilde{X})) with X~\widetilde{X} independent from XX.

Training then proceeds by alternating maximization over the critic parameters ψ\psi and maximization of the resulting bound over the encoder parameters φ\varphi. An estimator of the mutual information during training is given by

I^DV(φ,ψ)=1B∑i=1Bfψ(Z1,iφ,Z2,iφ)−log⁡(1B∑i=1Befψ(Z1,iφ,Z~2,iφ)).\widehat{I}_{\mathrm{DV}}(\varphi,\psi)=\frac{1}{B}\sum_{i=1}^{B}f_\psi\left(Z_{1,i}^\varphi,Z_{2,i}^\varphi\right)-\log\left(\frac{1}{B}\sum_{i=1}^{B}e^{f_\psi\left(Z_{1,i}^\varphi,\widetilde{Z}_{2,i}^\varphi\right)}\right).

In practice, these variational estimators can suffer from optimization instability and can lead to collapsed representations [8]. Moreover, these neural estimators do not necessarily preserve desirable properties of the underlying KL divergence, such as invariance to reparameterization [51].

ScoreCLR

More recently, diffusion-based approaches have been proposed for mutual information estimation [20]. These methods achieve improved performance and pass the invariance to reparameterization test [32]. To the best of our knowledge, these diffusion-based approaches to mutual information estimation have not yet been explored in a self-supervised representation learning framework.

Equivalent loss

Since we optimize our loss with gradient descent, replacing it by another loss with the same gradient does not change the training dynamic. The following proposition gives a way to make such a replacement for the entropy term.

Proposition 3.1. For any random variable YY, denoting Zφ=Fφ(Y)Z^\varphi=F^\varphi(Y), under Assumption A.1 (Appendix § 6) we have

∇φH(Zφ)=12∇φE[∥Zφ−sg⁡(Zφ+∇zlog⁡pZφ(Zφ))∥2].\nabla_\varphi H\left(Z^\varphi\right)=\frac{1}{2}\nabla_\varphi\mathbb{E}\left[\left\|Z^\varphi-\operatorname{sg}\left(Z^\varphi+\nabla_z\log p_{Z^\varphi}\left(Z^\varphi\right)\right)\right\|^2\right].

where sg denotes the stop grad operator.

For the conditional entropy, we will consider the Gaussian conditional model Z1∣Z2∼N(Z2,σ2I)Z_1\mid Z_2\sim\mathcal{N}\left(Z_2,\sigma^2I\right) which will give a lower bound on the mutual information,

LMI(φ)=12σ2E[∥Gφ(T1(X))−Gφ(T2(X))∥2]−H(Zφ),\mathcal{L}_{\mathrm{MI}}(\varphi)=\frac{1}{2\sigma^2}\mathbb{E}\left[\left\|G^\varphi\left(T_1(X)\right)-G^\varphi\left(T_2(X)\right)\right\|^2\right]-H\left(Z^\varphi\right),

that gives the same training dynamic as

LMI-score(φ)=12σ2E[∥Gφ(T1(X))−Gφ(T2(X))∥2]+12E[∥Zφ−sg⁡(Zφ−∇zlog⁡pZφ(Zφ))∥2].\mathcal{L}_{\mathrm{MI\text{-}score}}(\varphi)=\frac{1}{2\sigma^2}\mathbb{E}\left[\left\|G^\varphi\left(T_1(X)\right)-G^\varphi\left(T_2(X)\right)\right\|^2\right]+\frac{1}{2}\mathbb{E}\left[\left\|Z^\varphi-\operatorname{sg}\left(Z^\varphi-\nabla_z\log p_{Z^\varphi}\left(Z^\varphi\right)\right)\right\|^2\right].

We call its second term ScoreReg. This loss can be seen as enforcing invariance of the latent representation to the given transformation while encouraging it to remain of high entropy.

Regularization

Gaussian prior Without any extra regularization the problem is only constrained by the architecture of the network and the entropy of the latent could grow arbitrarily high. To alleviate this issue we consider a Gaussian prior on our latent variable by adding an L2L^2 penalty term to the loss. Noticing that −H(Z)+12E[∥Z∥2]=C+KL⁡(pZ∥N(0,I))-H(Z) + \frac{1}{2}\mathbb{E}[\lVert Z\rVert^2] = C + \operatorname{KL}(p_Z \mathbin\Vert\mathcal{N}(0,I)), we recover the same regularization term as in Balestriero & LeCun [3], the maximum-entropy prior with mean 0 and variance 1. In practice the penalty is applied to the noised latents at which the score is queried (Algorithm 1).

Noise regularization To estimate the score of the latent we use denoising score matching (Vincent, [55]; Song & Ermon [52]): we approximate the score of the latent distribution convolved with Gaussian noise of standard deviation tt. We denote Ztφ=Zφ+tϵZ_t^\varphi= Z^\varphi+ t\epsilon, where ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I), and train a lightweight noise-prediction network ϵθ\epsilon^\theta with the denoising loss

LDSM(θ)=E[∥ϵθ(Ztφ,t)−ϵ∥2],t∼p(t),\mathcal{L}_{\mathrm{DSM}}(\theta) = \mathbb{E}\left[\left\lVert\epsilon^\theta(Z_t^\varphi,t)-\epsilon\right\rVert^2\right], \qquad t \sim p(t),

where p(t)p(t) is the noise level distribution, specified in § 4.1. The score is recovered by Tweedie’s formula, stθ(z)=−ϵθ(z,t)/ts_t^\theta(z)=-\epsilon^\theta(z,t)/t. Further details can be found in Appendix § 8.

For simplicity, we replace the 1/(2σ2)1/(2\sigma^2) factor by a tunable weight λ\lambda. The regularized loss is

LMIreg(φ)=λE[∥Z1φ−Z2φ∥2]+Et∼p(t)[KL⁡(pZtφ∥N(0,I))].\mathcal{L}_{\mathrm{MI}}^{\mathrm{reg}}(\varphi)=\lambda\mathbb{E}\left[\left\lVert Z_1^\varphi-Z_2^\varphi\right\rVert^2\right]+\mathbb{E}_{t\sim p(t)}\left[\operatorname{KL}\left(p_{Z_t^\varphi}\mathbin\Vert\mathcal{N}(0,I)\right)\right].

Using Proposition 3.1, LMIreg\mathcal{L}_{\mathrm{MI}}^{\mathrm{reg}} will give the same training dynamic as

LMI-scorereg(φ)=λE[∥Z1φ−Z2φ∥2]+12Et∼p(t)E[∥Zφ−sg⁡(Zφ−∇zlog⁡pZtφ(Ztφ)−Ztφ)∥2].\mathcal{L}_{\mathrm{MI\text{-}score}}^{\mathrm{reg}}(\varphi)=\lambda\mathbb{E}\left[\left\lVert Z_1^\varphi-Z_2^\varphi\right\rVert^2\right]+\frac{1}{2}\mathbb{E}_{t\sim p(t)}\mathbb{E}\left[\left\lVert Z^\varphi-\operatorname{sg}\left(Z^\varphi-\nabla_z\log p_{Z_t^\varphi}(Z_t^\varphi)-Z_t^\varphi\right)\right\rVert^2\right].

To have a low variance estimate of the repulsive term, we may average it over KK noise levels, using independent Gaussian noise for each level.

Training

We want, for every update of φ\varphi, a statistical estimator of the score stθ∗(φ)s_t^{\theta^*(\varphi)} that minimizes well LDSM(θ)\mathcal{L}_{\mathrm{DSM}}(\theta). To do so, we follow the methodology of distribution matching distillation (Yin et al., [59]), which alternates training between updates of φ\varphi and θ\theta with a 1-to-kk proportion rule that has proven to be efficient in large-scale training. The score network is unconditional: it takes only the noised code and the noise level.

The final loss can be made of three components, an invariance term Linv\mathcal{L}_{\mathrm{inv}}, a repulsion term Lrep\mathcal{L}_{\mathrm{rep}} that estimates ScoreReg, and a regularization term Lreg\mathcal{L}_{\mathrm{reg}}, the last two forming the prior term Lprior\mathcal{L}_{\mathrm{prior}}. On a batch, their expressions are

Linv=1Bd∑i=1B∥Zi1−Zi2∥2,\mathcal{L}_{\mathrm{inv}}=\frac{1}{Bd}\sum_{i=1}^{B}\left\lVert Z_i^1-Z_i^2\right\rVert^2,
Lrep=12Bd∑i=1B∑v∈{1,2}12∥Zvi−sg⁡(Zvi−sˉvi)∥2.(1)\mathcal{L}_{\mathrm{rep}}=\frac{1}{2Bd}\sum_{i=1}^{B}\sum_{v\in\{1,2\}}\frac{1}{2}\left\lVert Z_v^i-\operatorname{sg}\left(Z_v^i-\bar{s}_v^i\right)\right\rVert^2. \tag*{(1)}
Lreg=12Bd∑i=1B∑v∈{1,2}1K∑ℓ=1K12∥Zv,ℓi∥2,\mathcal{L}_{\mathrm{reg}}=\frac{1}{2Bd}\sum_{i=1}^{B}\sum_{v\in\{1,2\}}\frac{1}{K}\sum_{\ell=1}^{K}\frac{1}{2}\left\lVert Z_{v,\ell}^i\right\rVert^2,
LScoreCLR=λLinv+Lprior,Lprior=Lrep+Lreg,\mathcal{L}_{\mathrm{ScoreCLR}}=\lambda\mathcal{L}_{\mathrm{inv}}+\mathcal{L}_{\mathrm{prior}}, \qquad\mathcal{L}_{\mathrm{prior}}=\mathcal{L}_{\mathrm{rep}}+\mathcal{L}_{\mathrm{reg}},

where BB is the batch size, dd is the dimension of the codes, Zvi=Gφ(Tv(Xi))Z_v^i=G^\varphi(T_v(X^i)), Zv,ℓi=Zvi+tv,ℓiϵv,ℓiZ_{v,\ell}^i=Z_v^i+t_{v,\ell}^i\epsilon_{v,\ell}^i with tv,ℓi∼p(t)t_{v,\ell}^i\sim p(t) and ϵv,ℓi∼N(0,I)\epsilon_{v,\ell}^i\sim\mathcal{N}(0,I) drawn independently for each sample, view and query, and sˉvi=1K∑ℓ=1Kstv,ℓiθ(Zv,ℓi)\bar{s}_v^i=\frac{1}{K}\sum_{\ell=1}^{K}s_{t_{v,\ell}^i}^{\theta}(Z_{v,\ell}^i). The training procedure is summarized in Algorithm 1. Connections to the SimCLR loss and to drifting generative models (Deng et al., [17]) are given in Appendices § 7.§ 1 and § 7.§ 2, respectively.

Algorithm 1 ScoreCLR training

Require: Encoder GφG^{\varphi}, noise-prediction network εθ\varepsilon^{\theta}, augmentations T1T_1, T2T_2, noise distribution p(t)p(t), score-network steps SS, noise queries KK, weight λ\lambda

1: for each batch (Xi)i=1B(X^i)_{i=1}^{B} do 2: Zvi=Gφ(Tv(Xi))Z_v^i = G^{\varphi}(T_v(X^i)) for v∈{1,2}v \in\{1,2\} 3: for j=1,…,Sj = 1,\ldots,S do ▷ score-network updates 4: Sample new tvi,(j)∼p(t)t_v^{i,(j)} \sim p(t) and εvi,(j)∼N(0,I)\varepsilon_v^{i,(j)} \sim\mathcal{N}(0,I) for every ii and vv 5: Z~vi,(j)=sg⁡(Zvi)+tvi,(j)εvi,(j)\widetilde{Z}_v^{i,(j)} = \operatorname{sg}(Z_v^i) + t_v^{i,(j)}\varepsilon_v^{i,(j)} 6: θ←1\theta\leftarrow1 update of 12B∑i=1B∑v∈{1,2}∥εθ(Z~vi,(j),tvi,(j))−εvi,(j)∥2\frac{1}{2B}\sum_{i=1}^{B}\sum_{v\in\{1,2\}}\left\|\varepsilon^{\theta}\left(\widetilde{Z}_v^{i,(j)},t_v^{i,(j)}\right)-\varepsilon_v^{i,(j)}\right\|^2 7: end for 8: Sample new tv,ℓi∼p(t)t_{v,\ell}^i \sim p(t), εv,ℓi∼N(0,I)\varepsilon_{v,\ell}^i \sim\mathcal{N}(0,I), ℓ=1,…,K\ell=1,\ldots,K, and compute Zv,ℓiZ_{v,\ell}^i and sˉvi\bar{s}_v^i with the updated θ\theta 9: Compute LScoreCLR\mathcal{L}_{\mathrm{ScoreCLR}} with Eq. (1) 10: φ←1\varphi\leftarrow1 update of LScoreCLR\mathcal{L}_{\mathrm{ScoreCLR}} 11: end for

Comparison with other regularization methods

Several explicit regularization terms have been proposed to prevent collapse in non-contrastive self-supervised learning. VICReg [4] regularizes the variance and the covariance of the latent and thus implicitly assumes that the latent is Gaussian. SIGReg [3] pushes the latent toward an isotropic Gaussian with an Epps–Pulley characteristic-function test on random one-dimensional projections, relying on the Cramér–Wold theorem. As for every sliced divergence, the sample complexity of this test does not depend on the dimension [41], but its statistical power does. Under concentration assumptions on the latent, most one-dimensional projections of a high-dimensional distribution are close to Gaussian [18], and the fraction of directions in which a projection deviates from Gaussian by more than ϵ\epsilon decays as exp⁡(−cdϵ2)\exp(-c d \epsilon^2) [40], so the number of random slices needed to detect such a deviation grows exponentially with the dimension, which recovers the curse of dimensionality in the statistical power of the test. A latent supported on a low-dimensional subspace can thus pass the test, which is the geometric collapse we observe for LeJEPA in § 4.3.

ScoreCLR instead derives its repulsion from the contrastive criterion itself, as the gradient of the entropy term of the mutual information estimated with the full score of the latent, and does not suffer from this geometric collapse (§ 4.3).

Concurrent work. We recently became aware of the concurrent and independent work of Kappiyath et al. [31], which also trains a latent score network jointly with the encoder and uses the same identity between the gradient of the entropy and the score (Proposition 3.1) to prevent collapse. They report strong results on ImageNet-1K with ResNet and ViT encoders, and also use the learned score at test time to adapt the representations of corrupted images. The two methods differ in how the latent is constrained, how the score is learned and how the entropy term is written. Kappiyath et al. [31] regularize the embeddings, learn the score at a single noise level with 16 score-network updates per encoder step, and use the linear surrogate sg⁡(s)T~z\operatorname{sg}(s)\widetilde{T}z. ScoreCLR constrains the latent with a soft Gaussian prior, which gives the KL divergence to N(0,I)\mathcal{N}(0,I), learns the score over a distribution of noise levels with one to five score-network updates per step, and writes the entropy term together with the prior as a regression towards a target moved along the drift field −∇zlog⁡p−z-\nabla_z\log p-z, the velocity of the Wasserstein gradient flow of KL⁡(⋅∥N(0,I))\operatorname{KL}(\cdot\|\mathcal{N}(0,I)) (Appendix B.2); both forms give the same encoder gradient. We also study the geometry of the codes and the cost of the objective as the batch size grows (§ 4.3 and § 4.7).

Experiments

We compare ScoreCLR with SimCLR, and on CIFAR-10 with LeJEPA [3], BYOL [25] and Barlow Twins (Zbontar et al., 2021), on linear evaluation, representation geometry, out-of-domain transfer, batch size, numerical precision and training cost. The main results of SimCLR, LeJEPA, BYOL, Barlow Twins and ScoreCLR under AdamW are means ±\pm standard deviations over three seeds (1, 2, 3); the SimCLR (LARS) rows and the batch-size study are single runs (seed 5). Unless stated otherwise, runs are in fp32; SimCLR (LARS) is trained in 16-bit mixed precision as in solo-learn (da Costa et al., [15]). ScoreCLR uses λ=5\lambda= 5, one score-network step (S=1S = 1) and one noise query per code (K=1K = 1) on CIFAR-10, and λ=2\lambda= 2, S=5S = 5 and K=3K = 3 on ImageNet-100 (Appendix C); the AdamW methods share the same augmentations and gradient clipping (norm 1.0).

Setup

Data. We pre-train on CIFAR-10 (Krizhevsky, [36]) and ImageNet-100 (Deng et al., [16]). Transfer targets are CIFAR-10, STL-10 (Coates et al., [13]), and DTD (Cimpoi et al., [12]).

Models. ResNet-18 with a two-layer MLP projector (2048 hidden units, 256 outputs, no BatchNorm: a BatchNorm layer improves the results but partly prevents collapse on its own (Richemond et al., [47]), so we remove it). SimCLR, LeJEPA, BYOL, Barlow Twins and ScoreCLR share the encoder, the projector, the optimizer (AdamW), the augmentations and the gradient clipping (Table 8, Appendix C). We also report SimCLR with LARS under solo-learn’s recipe (da Costa et al., [15]), with the shared 2048/256 head and its published 4096/512 head.

Shared AdamWSimCLR (LARS)ScoreCLR
Backbone; epochsResNet-18; 1000 / 400ResNet-18; 1000 / 400ResNet-18; 1000 / 400
Batch size128 / 2×1282 \times128256 / 2×1282 \times128128 / 2×1282 \times128
OptimizerAdamWLARSAdamW (encoder and score net)
Encoder LR2×10−42 \times10^{-4}0.4 / 0.310−410^{-4} / 2×10−42 \times10^{-4}
Weight decay10−610^{-6}10−410^{-4}10−610^{-6} (score net 10−210^{-2})
LR scheduleconstant, cosine over last 100 ep10-ep warm-up + cosineconstant, cosine over last 100 ep (encoder and score net)
Gradient clip1.0none1.0 (encoder and score net)
Precisionfp3216-bit mixedfp32
Augmentationssymmetric pipeline (Appendix C.1)solo-learn’s ownsame as shared AdamW
ProjectorLinear–ReLU–Linear, no BatchNorm, 2048 / 256(LARS control: 4096 / 512)
Classifier LR0.010.10.01
Objectivemethod-specificτ=0.2\tau= 0.2λ=5/2\lambda= 5 / 2; repulsion and prior weight 1
Score network––unconditional ϵ\epsilon-MLP-FiLM, 3×2563 \times256, 64-d emb., 0.28M
Score-net LR; steps SS––10−310^{-3} (10× / 5× encoder); 1 / 5
Queries KK––1 / 3
DSM noise ln⁡t\ln t––N(−1.2,1.22)\mathcal{N}(-1.2, 1.2^2) / N(−1.2,1.02)\mathcal{N}(-1.2, 1.0^2)

Table 8. Pre-training hyperparameters (CIFAR-10 / ImageNet-100 where two values are given). “Shared AdamW” is the recipe of SimCLR, LeJEPA, BYOL and Barlow Twins. ImageNet-100 uses 2×1282 \times128 images per step with DDP-averaged gradients and no learning-rate rescaling.

Noise levels. The noise levels of ScoreCLR follow ln⁡t∼N(−1.2,1.22)\ln t \sim\mathcal{N}(-1.2, 1.2^{2}) on CIFAR-10 and ln⁡t∼N(−1.2,1.02)\ln t \sim\mathcal{N}(-1.2, 1.0^{2}) on ImageNet-100.

Evaluation. (i) Offline linear probe and weighted kk-NN (k=20k = 20) on frozen features; (ii) alignment and uniformity (Wang & Isola, [57]) of the projector output; (iii) effective rank (Garrido et al., [21]) of the backbone features and of the projector output, and an estimate of the Poincaré constant of the projector output; (iv) out-of-domain linear transfer following Kornblith et al. [35]; (v) measured time and peak memory of the objective, of a training step and of a training epoch.

Linear and k-NN evaluation

Discussion. With the same optimizer and projector, ScoreCLR is competitive with the contrastive and joint-embedding baselines. On CIFAR-10 its linear accuracy is 0.4 points below SimCLR (within one standard deviation) and 0.8 points below LeJEPA, within 0.1 of BYOL and 0.8 above Barlow Twins, and its kk-NN accuracy is within 0.5 points of SimCLR, LeJEPA and BYOL. On ImageNet-100 the linear probe is 1.2 points below SimCLR and the kk-NN accuracy 0.4 points below, with a smaller seed spread (0.06 against 0.62). SimCLR with LARS is stronger than all AdamW methods; these rows come from solo-learn’s own pipeline and are reference points, not a controlled comparison of objectives (Appendix C.1). The takeaway is that replacing the O(B2)O(B^{2}) batch-based repulsion by a learned score costs 0.4 points of linear accuracy on CIFAR-10 and 1.2 on ImageNet-100: what the method offers lies in the geometry of zz and in the cost of the objective at large batch sizes (§ 4.3 and § 4.7), not in the probe accuracy.

Representation Geometry

Table 2 reports alignment and uniformity of the projector output and Table 3 the effective rank (Garrido et al., [21]) of the backbone features hh and of the projector output zz. In zz, SimCLR spans about half of the 256 dimensions on ImageNet-100 and a third on CIFAR-10, the dimensional collapse of InfoNCE described by Jing et al. [30]. LeJEPA has the lowest rank in both hh (253/512) and zz (58/256) although it is trained with the SIGReg criterion (§ 3.4). ScoreCLR’s zz has a much higher effective rank (229/256 on CIFAR-10 and 244/256 on ImageNet-100), and its uniformity equals SimCLR’s on CIFAR-10 and is better on ImageNet-100 (−3.93 against −3.86). It pays for this with a worse alignment (0.51 against 0.37 for SimCLR on CIFAR-10, and 0.38 against 0.24 on ImageNet-100): the positive pairs are held less tightly together. The takeaway is that the score-based repulsion spreads the codes over almost all the available dimensions, which neither InfoNCE nor the SIGReg regularizer does. The ranks of the backbone features hh are similar for ScoreCLR and SimCLR on CIFAR-10 (332 and 336/512) and slightly higher for ScoreCLR on ImageNet-100 (346 against 329/512).

Table 3 also reports an estimate C^P\widehat{C}_P of the Poincaré constant of the codes. The Poincaré constant CPC_P of a distribution pp is the smallest constant such that Var⁡p(f)≤CPEp∥∇f∥2\operatorname{Var}_p(f) \le C_P \mathbb{E}_p \lVert\nabla f \rVert^2 for every smooth function ff (Bakry et al., [2]). It equals 1 for the standard Gaussian, and it is large when the distribution has separated modes or strongly anisotropic directions: it is the inverse of the spectral gap of the Langevin diffusion that targets pp, so it controls how fast that diffusion mixes (Bakry et al., [2]) and the statistical efficiency of learning the score of pp by score matching (Koehler et al., [34]). It is therefore a natural measure of how close the geometry of zz is to the Gaussian prior of ScoreCLR. We estimate it with the random-feature estimator of Pillaud-Vivien et al. [43] (Appendix § 8.8). With this estimator, on whitened codes, the value for ScoreCLR is the smallest of the three methods on CIFAR-10 (it was not computed on ImageNet-100).

MethodOptim.CIFAR-10ImageNet-100
AlignUnifAlignUnif
SimCLRLARS0.349−3.8180.240−3.856
SimCLRAdamW0.367±0.004−3.804±0.0020.235±0.001−3.855±0.001
LeJEPAAdamW0.316±0.003−3.540±0.003––
ScoreCLR (ours)AdamW0.514±0.006−3.802±0.0010.378±0.005−3.931±0.000

Table 2. Alignment (↓) and uniformity (↓) (Wang & Isola, [57]) of the projector output, ResNet-18 (AdamW rows: mean ± std over three seeds; LARS rows: single run). “–”: not measured.

Transfer

Table 4 probes frozen features on unseen datasets, STL-10 and DTD. ScoreCLR transfers slightly worse than SimCLR and is close to LeJEPA. After CIFAR-10 pre-training, it is 0.9 points below SimCLR on STL-10 and 2.5 below on DTD, and close to LeJEPA (75.2 against 76.0 on STL-10, 28.0 against 27.0 on DTD). After ImageNet-100 pre-training the gap to SimCLR is larger (1.5 points on STL-10, 3.1 on DTD, 4.4 on CIFAR-10), in line with the lower linear accuracy of Table 1. The takeaway is that the higher effective rank of zz does not by itself make the backbone features transfer better: transfer follows the in-domain accuracy of the representation.

MethodOptim.ProjectorCIFAR-10 Linear Top-1CIFAR-10 Linear Top-5CIFAR-10 kk-NN Top-1CIFAR-10 kk-NN Top-5ImageNet-100 Linear Top-1ImageNet-100 Linear Top-5ImageNet-100 kk-NN Top-1ImageNet-100 kk-NN Top-5
SimCLRAdamW2048/25689.26±\pm0.3899.7088.73±\pm0.1898.9072.39±\pm0.6292.1866.41±\pm0.4487.32
LeJEPAAdamW2048/25689.67±\pm0.1399.6588.40±\pm0.1398.65––––
BYOLAdamW2048/25688.96±\pm0.3099.7088.44±\pm0.2299.01––––
Barlow TwinsAdamW2048/25688.05±\pm0.1599.5787.17±\pm0.0998.57––––
ScoreCLR (ours)AdamW2048/25688.87±\pm0.4499.6688.27±\pm0.1398.8571.23±\pm0.0691.0366.05±\pm0.3986.09
SimCLRLARS2048/25690.7999.7489.9999.0275.4093.3069.9888.64
SimCLRLARS4096/512––––77.0293.8471.6089.10

Table 1. Linear-probe and weighted kk-NN (k=20k = 20) accuracy (%) on frozen features, ResNet-18. Top: shared optimizer and projector (AdamW, gradient clip 1.0, 2048/256 projector without BatchNorm); mean ±\pm std over three seeds (top-5 averaged); BYOL and Barlow Twins use a projector and predictor without BatchNorm (Appendix C). Bottom: SimCLR trained with solo-learn’s LARS recipe, with the shared 2048/256 projector and with the published 4096/512 projector. With the latter, our offline linear probe gives 77.02 on ImageNet-100, against 77.64 for the online probe of the published checkpoint.

Table 4: Out-of-domain linear transfer, top-1 (%): logistic regression on frozen L2-normalized features at the pretraining resolution (Appendix § 8.§ 5). “·” is the pretraining dataset (Table 1). All rows: mean ±\pm std over three seeds.

MethodOptim.CIFAR-10STL-10DTD
Pretrained on CIFAR-10 (ResNet-18, 32 px)
SimCLRAdamW·76.12±0.8176.12\pm0.8130.46±0.4930.46\pm0.49
LeJEPAAdamW·76.01±0.5476.01\pm0.5427.02±1.9927.02\pm1.99
BYOLAdamW·77.07±0.5477.07\pm0.5431.35±0.5831.35\pm0.58
Barlow TwinsAdamW·74.94±0.4274.94\pm0.4228.94±0.6728.94\pm0.67
ScoreCLR (ours)AdamW·75.21±0.1375.21\pm0.1328.01±0.5328.01\pm0.53
Pretrained on ImageNet-100 (ResNet-18, 224 px)
SimCLRAdamW75.64±0.3075.64\pm0.3084.35±0.4284.35\pm0.4259.38±1.1859.38\pm1.18
ScoreCLR (ours)AdamW71.23±0.3671.23\pm0.3682.87±0.4082.87\pm0.4056.28±0.6956.28\pm0.69

Table 4.

Batch size

We train SimCLR and ScoreCLR at larger batches (Table 5). The encoder learning rate follows the square-root rule (lr⁡∝B/128\operatorname{lr} \propto\sqrt{B/128}) for both methods; for ScoreCLR the score-network learning rate is kept fixed. ScoreCLR stays within 0.9 points of SimCLR up to B=2048B = 2048 and is 1.4 points below at B=4096B = 4096. With a larger batch there are fewer iterations per epoch, and so fewer score-network updates. In our runs, the number SS of score-network updates per step had to grow with the batch size to keep the accuracy (Appendix § 8.6). In these experiments, ScoreCLR does not reduce the batch size needed for a given accuracy; at large batches, its main benefit is the lower cost of the objective (§ 4.7).

Table 3: Effective rank (RankMe, Garrido et al., [21]) of the backbone features hh and of the projector output zz on raw (un-normalised) validation features, out of the space dimension, and an estimate C^P\widehat{C}_P of the Poincaré constant of the whitened zz (1 for a Gaussian; lower is closer to Gaussian), estimated with the random-feature estimator of Pillaud-Vivien et al. [43] (Appendix § 8.8). Both datasets use a ResNet-18. AdamW rows: mean ±\pm std over three seeds; LARS rows: single run. The low rank of zz under InfoNCE is the dimensional collapse analysed by Jing et al. [30].

Pre-trainingMethodOptim.RankMe hh (/512)RankMe zz (/256)C^P\widehat{C}_P of zz
CIFAR-10SimCLRLARS37288–
ResNet-18SimCLRAdamW336±2336\pm296±196\pm125.0±1.425.0\pm1.4
LeJEPAAdamW253±1253\pm158±058\pm013.5±1.513.5\pm1.5
BYOLAdamW292±8292\pm898±598\pm5–
Barlow TwinsAdamW243±3243\pm373±073\pm0–
ScoreCLR (ours)AdamW332±2332\pm2229±0229\pm07.8±0.87.8\pm0.8
ImageNet-100SimCLRLARS368118–
ResNet-18SimCLRAdamW329±1329\pm1143±0143\pm0–
ScoreCLR (ours)AdamW346±1346\pm1244±0244\pm0–

Table 3.

Numerical precision

SimCLR (LARS) is trained in 16-bit mixed precision (solo-learn), while the other runs use fp32. ScoreCLR works equally well in fp16: over three seeds, fp16 training reaches 88.82±0.04%88.82\pm0.04\% linear accuracy against 88.87±0.44%88.87\pm0.44\% in fp32, with the same effective rank of zz (Table 6). The comparison with the mixed-precision SimCLR (LARS) baseline is therefore not biased by the precision.

PrecisionLinearkk-NNRankMe zz
fp3288.87±\pm0.4488.27±\pm0.13229±\pm0
fp1688.82±\pm0.0488.39±\pm0.05229±\pm1

Table 6. ScoreCLR on CIFAR-10 in fp32 and fp16 mixed precision (encoder and objective under autocast, score-network updates in fp32); mean ±\pm std over seeds 1–3.

Computational Cost

InfoNCE compares every pair of codes in the batch, so the cost of its objective grows quadratically with the batch size BB. ScoreCLR instead passes each code through a small score network (0.28M parameters, 2.2% of the encoder and projector) a fixed number of times, SS updates and KK queries, so the cost of its objective grows linearly with BB. We measured both objectives on one H100 GPU in fp32 (Figure 2, Table 11). For the objective alone, InfoNCE is faster at small batches (0.6 ms against 2.3 ms at B=128B = 128), ScoreCLR uses less memory from B≈1B \mathrel{\approx} 1k, and it is faster from B≈2B \mathrel{\approx} 2k–4k with S=1S = 1 and above B≈8B \mathrel{\approx} 8k with S=10S = 10, the number of score-network updates the large-batch runs need. At B=32768B = 32768, InfoNCE takes 228 ms and 48.3 GiB; ScoreCLR takes 7.6 ms with S=1S = 1, K=1K = 1, 30 ms with the ImageNet-100 recipe (S=5S = 5, K=3K = 3) and 48.5 ms with S=10S = 10, K=1K = 1, and 1.1 GiB in each case. In a full training step the encoder dominates, so the gain is smaller but measured: one step of one device at B=32768B = 32768 over 8 devices (emulated on one GPU) takes 508 ms and 43.1 GiB for ScoreCLR (S=5S = 5, K=3K = 3) against 531 ms and 48.7 GiB for SimCLR (the encoder alone takes 500 ms; Table 7), and a full epoch takes the same time within 8% (Table 12).

Figure 2. Measured cost of the objective alone vs. the batch size BB (images), on one H100 GPU (fp32). Codes are given; we time one step of the loss (forward and backward to the codes; for ScoreCLR, SS score-network updates and KK queries, as indicated; S=5S=5, K=3K=3 is the ImageNet-100 recipe), median over 30 iterations, and report the extra peak memory, which does not depend on SS or KK.

Time (ms)Peak memory (GiB)
Encoder only (no objective)499.843.1
ScoreCLR, S=5S=5508.243.1
SimCLR530.948.7

Table 7. Cost of one full training step per device (ResNet-18, CIFAR-10 resolution, fp32, one H100): B=4096B=4096 images per device, i.e. 32768 images over 8 devices, emulated on one GPU. ScoreCLR: K=3K=3.

BBTime per step (ms): InfoNCETime per step (ms): ScoreCLR, S=1S=1, K=1K=1Time per step (ms): ScoreCLR, S=5S=5, K=3K=3Time per step (ms): ScoreCLR, S=10S=10, K=1K=1Extra memory: InfoNCEExtra memory: ScoreCLR
1280.622.38.215.12.0 MiB5.4 MiB
5120.782.48.516.016 MiB19 MiB
20481.42.58.614.9208 MiB71 MiB
40964.42.58.915.2800 MiB141 MiB
819214.72.49.114.83.1 GiB281 MiB
327682287.630.348.548.3 GiB1.1 GiB

Table 11. Measured cost of the objective alone (one H100, fp32, median over 30 iterations; same job as Figure 2): time per step and extra peak memory. The memory of ScoreCLR depends on neither SS nor KK.

SettingScoreCLREpoch time (s)Peak memory (GiB)
CIFAR-10, B=128B=128, fp32S=1S=1, K=3K=38.96 / 8.28–
ImageNet-100, B=2×128B=2\times128, fp32S=5S=5, K=3K=371.0 / 70.35.63 / 5.62
CIFAR-10, B=2048B=2048, fp16S=10S=10, K=1K=16.81 / 6.6411.0 / 11.1
CIFAR-10, B=4096B=4096, fp16S=10S=10, K=1K=18.27 / 8.2821.7 / 22.2
CIFAR-10, B=8192B=8192, fp16S=10S=10, K=1K=110.00 / 9.9843.1 / 45.8

Table 12. Measured time of one training epoch and peak memory, ScoreCLR / SimCLR (CIFAR-10: one H100; ImageNet-100: two H100s, memory per GPU). “–”: not recorded. The B=128B=128 row was timed with K=3K=3 queries, more work per step than the K=1K=1 recipe of Table 1.

Conclusion

We introduced ScoreCLR, which replaces the batch-based repulsion of contrastive learning by a score-based estimate of the entropy gradient, learned jointly with the encoder. A natural extension is a fully score-based critic, where the conditional entropy is also derived from the learned score. Since the score network predicts clean information from a noisy input, it would also be interesting to test whether the predictor of JEPA methods (Assran et al., [1]), which predicts the clean latent of a target from a masked context, could by itself be a proxy to enforce high entropy. Finally, since the score-based estimator keeps the projector output at a higher effective rank than existing regularizers (§ 4.3), it would be interesting to study priors with finer geometric properties than the isotropic Gaussian and to compare with those methods in that setting. Since ScoreCLR learns the score of the latent distribution at all noise levels, we also plan on investigating whether its score network can serve as a generative model of the representation.

Code and reproducibility

All training recipes, evaluation protocols and the cost model are given in Appendix § 8, and the proof of Proposition 3.1 in Appendix § 6. The ScoreCLR implementation is a self-contained PyTorch code base, and the SimCLR (LARS) baselines use the public solo-learn library (da Costa et al., [15]); the code is available at https://github.com/chris-mrn/ScoreCLR_code.

AI use statement

We used generative AI tools (Claude Code) to implement the training, evaluation and cost-measurement code of ScoreCLR and of the baselines, to launch and monitor experiments, to suggest ablations, to draft the analysis of the experimental results and parts of the experimental section, to assist in writing proofs, to create figures, to edit the text for grammar, to identify relevant literature and to format references. We did not use them to generate data, to develop theoretical models, to formulate mathematical claims, to provide critical ingredients of proofs, or to propose hypotheses. The authors verified the code against the equations of the paper, checked every number in the tables against the training logs and read all suggested references. We take full responsibility for the content of this paper.

Proofs

Assumption A.1.

(i) For every xx, the map φ↦Fφ(x)\varphi\mapsto F^{\varphi}(x) is continuously differentiable.

(ii) ZφZ^{\varphi} has a density pZφp_{Z^{\varphi}} that is positive everywhere, continuously differentiable in (z,φ)(z,\varphi), with finite entropy, finite second moment and finite Fisher information E[∥∇zlog⁡pZφ(Zφ)∥2]\mathbb{E}\left[\left\lVert\nabla_z \log p_{Z^{\varphi}}(Z^{\varphi})\right\rVert^2\right].

(iii) Derivatives with respect to φ\varphi can be exchanged with expectations and integrals (e.g. by dominated convergence).

Proof of Proposition 3.1. Since Zφ=Fφ(Y)Z^{\varphi}=F^{\varphi}(Y), the law of the unconscious statistician gives

H(Zφ)=−EZφ[log⁡pZφ(Zφ)]=−EY[log⁡pZφ(Fφ(Y))].H(Z^{\varphi})=-\mathbb{E}_{Z^{\varphi}}\left[\log p_{Z^{\varphi}}(Z^{\varphi})\right]=-\mathbb{E}_{Y}\left[\log p_{Z^{\varphi}}(F^{\varphi}(Y))\right].

The encoder parameters enter in two places: through the point Fφ(Y)F^{\varphi}(Y) at which the density is evaluated, and through the density pZφp_{Z^{\varphi}} itself. Differentiating with respect to φ\varphi,

∇φH(Zφ)=−EY[(∇φFφ(Y))T(∇zlog⁡pZφ)(Fφ(Y))]−EY[(∇φlog⁡pZφ)(Fφ(Y))],\nabla_{\varphi}H(Z^{\varphi})=-\mathbb{E}_{Y}\left[\left(\nabla_{\varphi}F^{\varphi}(Y)\right)^{\mathsf{T}}\left(\nabla_z\log p_{Z^{\varphi}}\right)(F^{\varphi}(Y))\right]-\mathbb{E}_{Y}\left[\left(\nabla_{\varphi}\log p_{Z^{\varphi}}\right)(F^{\varphi}(Y))\right],

where in the second term ∇φ\nabla_{\varphi} acts on the parameter index of the density only, the evaluation point being held fixed.

The second term vanishes. Indeed,

EY[(∇φlog⁡pZφ)(Fφ(Y))]=EZφ[∇φlog⁡pZφ(Zφ)]=∫pZφ(z)∇φpZφ(z)pZφ(z) dz=∇φ∫pZφ(z) dz=∇φ(1)=0,\begin{aligned} \mathbb{E}_{Y}\left[\left(\nabla_{\varphi}\log p_{Z^{\varphi}}\right)(F^{\varphi}(Y))\right]=\mathbb{E}_{Z^{\varphi}}\left[\nabla_{\varphi}\log p_{Z^{\varphi}}(Z^{\varphi})\right]=\int p_{Z^{\varphi}}(z)\frac{\nabla_{\varphi}p_{Z^{\varphi}}(z)}{p_{Z^{\varphi}}(z)}\,\mathrm{d}z \\ &=\nabla_{\varphi}\int p_{Z^{\varphi}}(z)\,\mathrm{d}z=\nabla_{\varphi}(1)=0, \end{aligned}

the interchange of ∇φ\nabla_{\varphi} and the integral being justified by dominated convergence. Hence

∇φH(Zφ)=−EY[(∇φFφ(Y))T(∇zlog⁡pZφ)(Fφ(Y))].(2)\nabla_{\varphi}H(Z^{\varphi})=-\mathbb{E}_{Y}\left[\left(\nabla_{\varphi}F^{\varphi}(Y)\right)^{\mathsf{T}}\left(\nabla_z\log p_{Z^{\varphi}}\right)(F^{\varphi}(Y))\right]. \tag*{(2)}

It remains to identify the right-hand side as the gradient of the stated surrogate. Write

L(φ)=12EY[∥Fφ(Y)−cφ(Y)∥2],cφ(Y):=sg⁡(Fφ(Y)+(∇zlog⁡pZφ)(Fφ(Y))).\mathcal{L}(\varphi)=\frac{1}{2}\mathbb{E}_{Y}\left[\left\lVert F^{\varphi}(Y)-c^{\varphi}(Y)\right\rVert^2\right],\qquad c^{\varphi}(Y):=\operatorname{sg}\left(F^{\varphi}(Y)+\left(\nabla_z\log p_{Z^{\varphi}}\right)(F^{\varphi}(Y))\right).

The stop-gradient has zero derivative, so ∇φcφ(Y)=0\nabla_{\varphi}c^{\varphi}(Y)=0 and only the first occurrence of FφF^{\varphi} is differentiated:

∇φL(φ)=EY[(∇φFφ(Y))T(Fφ(Y)−cφ(Y))].\nabla_{\varphi}\mathcal{L}(\varphi)=\mathbb{E}_{Y}\left[\left(\nabla_{\varphi}F^{\varphi}(Y)\right)^{\mathsf{T}}\left(F^{\varphi}(Y)-c^{\varphi}(Y)\right)\right].

At the point of evaluation the stop-gradient is the identity, so Fφ(Y)−cφ(Y)=−(∇zlog⁡pZφ)(Fφ(Y))F^{\varphi}(Y)-c^{\varphi}(Y)=-(\nabla_z\log p_{Z^{\varphi}})(F^{\varphi}(Y)), and therefore

∇φL(φ)=−EY[(∇φFφ(Y))T(∇zlog⁡pZφ)(Fφ(Y))]=∇φH(Zφ)\nabla_{\varphi}\mathcal{L}(\varphi)=-\mathbb{E}_{Y}\left[\left(\nabla_{\varphi}F^{\varphi}(Y)\right)^{\mathsf{T}}\left(\nabla_z\log p_{Z^{\varphi}}\right)(F^{\varphi}(Y))\right]=\nabla_{\varphi}H(Z^{\varphi})

by equation 2.

Connection to other methods

ScoreCLR was first motivated by a comparison between distribution matching distillation [59] and generative modeling by drifting [17]. We present below its links with SimCLR and with drifting models.

SimCLR

SimCLR normalizes the codes, so ∥Zvi∥=1\lVert Z_v^i\rVert= 1. Then fτ(z,y)=1τ−12τ∥z−y∥2f_\tau(z,y) = \frac{1}{\tau} - \frac{1}{2\tau}\lVert z-y\rVert^2 and kτ(z,y)∝exp⁡(−∥z−y∥2/(2τ))k_\tau(z,y) \propto\exp(-\lVert z-y\rVert^2/(2\tau)) is a Gaussian kernel of variance τ\tau. Up to an additive constant, the SimCLR loss [9] writes

LSimCLR=12τLinv+LrepSimCLR,LrepSimCLR=12B∑i=1B∑v∈{1,2}log⁡(∑(j,u)≠(i,v)e−∥Zvi−Zuj∥2/(2τ)).\mathcal{L}_{\mathrm{SimCLR}} = \frac{1}{2\tau}\mathcal{L}_{\mathrm{inv}} + \mathcal{L}_{\mathrm{rep}}^{\mathrm{SimCLR}}, \qquad\mathcal{L}_{\mathrm{rep}}^{\mathrm{SimCLR}} = \frac{1}{2B}\sum_{i=1}^{B}\sum_{v\in\{1,2\}}\log\left(\sum_{(j,u)\ne(i,v)}e^{-\lVert Z_v^i-Z_u^j\rVert^2/(2\tau)}\right).

The invariance term is the same as in ScoreCLR. Since Zvi=Gφ(Tv(Xi))Z_v^i = G^\varphi(T_v(X^i)), the gradient of the repulsion with respect to φ\varphi is

∇φLrepSimCLR=12Bτ∑i=1B∑v(∇φZvi)T[∑(j,u)≠(i,v)e−∥Zvi−Zuj∥2/(2τ)∑(l,w)≠(i,v)e−∥Zvi−Zwl∥2/(2τ)(Zuj−Zvi)+∑(j,u)≠(i,v)e−∥Zuj−Zvi∥2/(2τ)∑(l,w)≠(j,u)e−∥Zuj−Zwl∥2/(2τ)(Zuj−Zvi)].\nabla_\varphi\mathcal{L}_{\mathrm{rep}}^{\mathrm{SimCLR}} = \frac{1}{2B\tau}\sum_{i=1}^{B}\sum_v(\nabla_\varphi Z_v^i)^\mathsf{T}\left[\sum_{(j,u)\ne(i,v)}\frac{e^{-\lVert Z_v^i-Z_u^j\rVert^2/(2\tau)}}{\sum_{(l,w)\ne(i,v)}e^{-\lVert Z_v^i-Z_w^l\rVert^2/(2\tau)}}(Z_u^j-Z_v^i) + \sum_{(j,u)\ne(i,v)}\frac{e^{-\lVert Z_u^j-Z_v^i\rVert^2/(2\tau)}}{\sum_{(l,w)\ne(j,u)}e^{-\lVert Z_u^j-Z_w^l\rVert^2/(2\tau)}}(Z_u^j-Z_v^i)\right].

whereas for ScoreCLR, with t∼p(t)t\sim p(t) and ε∼N(0,I)\varepsilon\sim\mathcal{N}(0,I),

∇φLrepScoreCLR=12B∑i=1B∑v(∇φZvi)TEt,ε[stθ(Zvi+tε)].\nabla_\varphi\mathcal{L}_{\mathrm{rep}}^{\mathrm{ScoreCLR}} = \frac{1}{2B}\sum_{i=1}^{B}\sum_v(\nabla_\varphi Z_v^i)^\mathsf{T}\mathbb{E}_{t,\varepsilon}\left[s_t^\theta(Z_v^i+t\varepsilon)\right].

The first term of the bracket is the score of a Gaussian kernel density estimator (KDE) of bandwidth τ\sqrt{\tau} built on the 2B−12B-1 other codes, i.e. a batch estimate of the score of pZτp_{Z_\tau} at t=τt=\sqrt{\tau}. It is the counterpart of the ScoreCLR term, with two differences: (i) it costs O(B2)O(B^2) and uses a fixed t=τt=\sqrt{\tau}, whereas ScoreCLR uses a learned score at a random t∼p(t)t\sim p(t) at cost O(B)O(B); (ii) it is evaluated at the clean code ZviZ_v^i, whereas ScoreCLR evaluates it at Zvi+tεZ_v^i+t\varepsilon.

The second term has no counterpart in ScoreCLR. It appears because ZviZ_v^i is also a kernel centre in the sums of the other anchors, so moving ZviZ_v^i changes the estimated density at the other codes. For the true density this effect is zero, as in the proof of Proposition 3.1:

E[∇φlog⁡pZφ(Z)∣Z=Zφ]=∫∇φpZφ(z) dz=∇φ1=0.\mathbb{E}\left[\left.\nabla_\varphi\log p_{Z_\varphi}(Z)\right|_{Z=Z_\varphi}\right] = \int\nabla_\varphi p_{Z_\varphi}(z)\,\mathrm{d}z = \nabla_\varphi1 = 0.

ScoreCLR never computes it, since the score network is a stop-gradient. In SimCLR, the KDE is built from the codes being differentiated, so the term is not exactly zero on a finite batch.

Finally, SimCLR has no regularization term Lreg\mathcal{L}_{\mathrm{reg}}: the constraint ∥Zvi∥=1\lVert Z_v^i\rVert= 1 plays the role of the Gaussian prior.

Drifting models

Considering the terminology of Deng et al. [17], we denote as drifting field

drift⁡iφ=−∇zlog⁡pZφ(Ziφ)−Ziφ,i∈{1,2},\operatorname{drift}_i^\varphi= -\nabla_z\log p_{Z_\varphi}(Z_i^\varphi)-Z_i^\varphi,\qquad i\in\{1,2\},

the velocity field of the Wasserstein gradient flow of KL(⋅∥N(0,I))KL(\cdot\mathbin\Vert\mathcal{N}(0,I)) [24]. Symmetrizing the repulsion force with respect to the views and using the stop-gradient trick of DMD [59], we get the mutual information loss with Gaussian prior, which writes as

LMI-scoreGaussian prior(φ)=λE[∥Z1φ−Z2φ∥2]+12∑i=12E[∥Ziφ−sg⁡(Ziφ+drift⁡iφ)∥2].\mathcal{L}_{\mathrm{MI\text{-}score}}^{\mathrm{Gaussian\ prior}}(\varphi) = \lambda\mathbb{E}\left[\lVert Z_1^\varphi-Z_2^\varphi\rVert^2\right] + \frac{1}{2}\sum_{i=1}^{2}\mathbb{E}\left[\left\lVert Z_i^\varphi-\operatorname{sg}\left(Z_i^\varphi+\operatorname{drift}_i^\varphi\right)\right\rVert^2\right].

with the same convention as Eq. (1): λ\lambda weights the invariance term and the prior term has weight one.

Experimental Details

Pre-training protocol

Shared recipe. SimCLR, LeJEPA, BYOL, Barlow Twins and ScoreCLR share the ResNet-18 encoder, the two-layer MLP projector (Linear–ReLU–Linear, 2048 hidden units, 256 outputs, no BatchNorm), the optimizer (AdamW, weight decay 10−610^{-6}), the gradient-norm clip 1.0, the schedule (constant learning rate, then a cosine decay to zero over the last 100 epochs), the batch size (128 for 1000 epochs on CIFAR-10, 2×1282 \times128 for 400 epochs on ImageNet-100), the augmentations and fp32 precision (Table 8). The clip applies to the encoder, the projector and the online classifier and, for ScoreCLR, also to the score network. Only the encoder learning rate is tuned per method: 2×10−42 \times10^{-4} for SimCLR, LeJEPA, BYOL and Barlow Twins, and 10−410^{-4} (CIFAR-10) or 2×10−42 \times10^{-4} (ImageNet-100) for ScoreCLR.

Augmentations. SimCLR, LeJEPA, BYOL, Barlow Twins and ScoreCLR use the same symmetric pipeline on both datasets: random resized crop (scale [0.08, 1]), horizontal flip (probability 0.5), colour jitter (0.8, 0.8, 0.8, 0.2) applied with probability 0.8, grayscale with probability 0.2 and Gaussian blur with probability 0.5, without solarization. The SimCLR (LARS) rows come from solo-learn’s own pipeline (data loading and augmentations) and are reference points, not controlled comparisons.

Baselines. SimCLR uses the symmetrized InfoNCE loss with temperature τ=0.2\tau= 0.2 and negatives gathered across devices. LeJEPA replaces the repulsion by SIGReg (λ=0.02\lambda= 0.02, 17 knots, tmax⁡=3t_{\max} = 3, 256 slices). BYOL uses a 4096-unit predictor without BatchNorm and an EMA coefficient that follows a cosine from 0.99 to 1; Barlow Twins uses λBT=0.0051\lambda_{\mathrm{BT}} = 0.0051 and a loss scale of 0.1, both solo-learn’s CIFAR-10 values. BYOL and Barlow Twins are run in this harness with seeds 1–3; removing BatchNorm from their predictor and projector disadvantages them (BYOL reaches 89.96% linear accuracy with a BatchNorm predictor, seed 1). The SimCLR (LARS) rows follow solo-learn’s recipe: LARS (momentum 0.9, η=0.02\eta= 0.02, biases and normalization layers excluded), weight decay 10−410^{-4}, a 10-epoch warm-up from 3×10−53 \times10^{-5} followed by a cosine decay, τ=0.2\tau= 0.2, 16-bit mixed precision, learning rate 0.4 with batch 256 on CIFAR-10 (1000 epochs) and learning rate 0.3 with batch 2×1282 \times128 on ImageNet-100 (400 epochs). The projector is narrowed from the published 4096/512 to the shared 2048/256 head, except in the control row, which keeps the published head and every other published default.

ScoreCLR implementation

Loss. We minimize Eq. (1), L=λLinv+Lrep+Lreg\mathcal{L} = \lambda\mathcal{L}_{\mathrm{inv}} + \mathcal{L}_{\mathrm{rep}} + \mathcal{L}_{\mathrm{reg}}, with λ=5\lambda= 5 on CIFAR-10 and λ=2\lambda= 2 on ImageNet-100. In the Gaussian conditional model used to derive the loss, λ=1/(2σ2)\lambda= 1/(2\sigma^2), i.e. σ2=0.1\sigma^2 = 0.1 and 0.25. The repulsion and the prior carry weight one: they are two parts of one estimate of KL⁡(pZ∥N(0,I))\operatorname{KL}(p_Z \mathbin\Vert\mathcal{N}(0,I)) and are not tuned separately. Each of the three terms is divided by the code dimension d=256d = 256; this common factor leaves the relative weights unchanged and only matters through the gradient clip. The repulsion and the prior are averaged over KK independent noise queries per code, with K=1K = 1 on CIFAR-10 and K=3K = 3 on ImageNet-100. The noise levels follow ln⁡t∼N(−1.2,pstd)\ln t \sim\mathcal{N}(-1.2, p_{\mathrm{std}}) with pstd=1.2p_{\mathrm{std}} = 1.2 on CIFAR-10 and 1.0 on ImageNet-100.

Score network. The score network is an unconditional noise-prediction MLP εθ(xt,t)\varepsilon^\theta(x_t,t): three linear layers of width 256 with SiLU activations on the concatenation of ztz_t and a 64-dimensional sinusoidal embedding of ln⁡t\ln t, the two hidden layers being FiLM-modulated by the same embedding (modulation initialized to the identity). It has 0.28M parameters, 2.2% of the encoder and projector. It is trained with the unweighted loss LDSM\mathcal{L}_{\mathrm{DSM}} on the detached codes of both views, with fresh noise for each of the SS updates per encoder step (S=1S = 1 on CIFAR-10, S=5S = 5 on ImageNet-100), and queried with stθ=−εθ/ts_t^\theta= -\varepsilon^\theta/t. Its optimizer is AdamW with learning rate 10−310^{-3}, weight decay 10−210^{-2}, gradient clip 1.0. Like the encoder’s, its learning rate is constant and then follows a cosine decay to zero over the last 100 epochs.

Linear evaluation

The offline probe trains a linear classifier for 100 epochs on frozen backbone features in fp32, with the standard crop-and-flip augmentation, using the same protocol for every method; the only exception is the kk-NN accuracy of the SimCLR (LARS) rows, computed with a memory bank under crop-and-flip augmentation. On CIFAR-10 the features are used raw and the classifier is trained with SGD (momentum 0.9, learning rate 0.1, divided by 10 at epochs 60 and 80). On ImageNet-100 the features are L2-normalized and the classifier is trained with SGD (momentum 0.9, learning rate 10, warm-up then cosine). The kk-NN accuracy is a weighted cosine kk-NN (k=20k = 20, temperature 0.07) of the validation features against the clean training set. Alignment is computed on L2-normalized projector outputs, uniformity on 4000 validation codes, and RankMe on the 10,000 validation codes.

Effective rank

RankMe ([21]) is the exponential of the entropy of the normalized singular values of the feature matrix, computed on the validation set. We report it on both the backbone features and the projector output because Jing et al. [30] (DirectCLR) showed that contrastive learning can collapse the projector output (dimensional collapse) while the backbone stays full-rank.

Out-of-domain transfer

Following Kornblith et al. [35], a multinomial logistic-regression classifier is fit on frozen, L2-normalized backbone features of each downstream training set. The L2 penalty is selected on a validation split (the dataset’s own where one exists, else a stratified 20% hold-out) and the classifier is refit on train + val. Downstream images are resized and center-cropped to the pretraining resolution: 32×3232 \times32 with the CIFAR stem for CIFAR-10 backbones and 224×224224 \times224 for ImageNet-100 backbones.

Ablations of the recipe

Table 9 varies ingredients of the CIFAR-10 recipe (batch 128, 1000 epochs, λ=5\lambda= 5). The reference is K=1K=1, S=1S=1, pstd=1.2p_{\mathrm{std}}=1.2 with clip 1.0 (mean ±\pm standard deviation over seeds 1–3); a paired row is compared with the reference run of the same seed.

Conditioning. A score network that also receives a second latent as input, namely the other view of the same image when it is trained to denoise and the code of another image of the batch when it is queried, loses 3.3 to 5.6 points of linear accuracy with respect to the unconditional network

VariantSeedsLinearkk-NNRankMe zzRef.
Reference (K=1K=1, S=1S=1, pstd=1.2p_{\mathrm{std}}=1.2, clip)1–388.87±0.4488.87\pm0.4488.27±0.1388.27\pm0.13229–
pstd=1.0p_{\mathrm{std}}=1.01–388.57±0.1988.57\pm0.1988.20±0.0888.20\pm0.08233–
Conditional score network1 / 283.83 / 85.2783.54 / 84.3425 / 5189.38 / 88.58
K=1K=1, no clipping{sup}†`5, 4288.87 / 88.8588.44 / 88.43229–
K=3K=3, no clipping1–378.62±4.9778.62\pm4.9778.94±4.5178.94\pm4.51105±32105\pm32–

Table 9. Ablations on CIFAR-10: linear accuracy (%), kk-NN accuracy (%) and RankMe of zz. “Ref.” is the linear accuracy of the same-seed run with the same noise width and clip. †: seeds 5 and 42, run with an earlier code snapshot that implements the same loss. The K=3K=3 row uses seeds 1–3 without gradient clipping.

(Table 9). The recipe was not re-tuned for the conditional network, so this ablation measures the conditioning inside the present recipe only.

Queries and clipping. Averaging K=3K=3 queries without gradient clipping is unstable across seeds (79.45, 83.16 and 73.26 for seeds 1–3), whereas K=1K=1 without clipping gives 88.87 and 88.85. We use the clip for fairness with the baselines, for which it is neutral (SimCLR: 89.41 without it, 89.45 with it, seed 5). On ImageNet-100, K=3K=3 with the clip is stable (71.23±0.0671.23\pm0.06 over three seeds).

Score-network steps. Extra score-network steps matter when iterations are scarce. On ImageNet-100 (494 iterations per epoch), K=1K=1 and S=1S=1 reach 57.14 against 71.23±0.0671.23\pm0.06 for the recipe of Table 8. At batch 2048 on CIFAR-10 (24 iterations per epoch), S=1S=1 is not sufficient, S=5S=5 gives 87.84 and S=10S=10 88.29.

Larger batches. At B=4096B=4096, S=10S=10 gives 87.50 with pstd=1.2p_{\mathrm{std}}=1.2 (Table 5) and 87.56 with pstd=1.0p_{\mathrm{std}}=1.0; with pstd=1.0p_{\mathrm{std}}=1.0, S=5S=5 gives 85.76.

Table 5: Linear accuracy (%), kk-NN accuracy (%) and effective rank of zz against the batch size, CIFAR-10, 1000 epochs, seed 5. SimCLR: AdamW, gradient clipping, encoder learning rate scaled by B/128\sqrt{B/128}. ScoreCLR: K=1K=1, gradient clipping, same encoder scaling, unscaled score-network learning rate, S=1S=1 at B=128B=128 (main recipe), S=5S=5 at B=512B=512 and S=10S=10 at B=2048B=2048 and 4096.

ScoreCLRSimCLR
BBLinearkk-NNRankMe zzLinearkk-NNRankMe zz
12888.8088.5122989.4588.5696
51288.5188.3923189.3289.07108
204888.2987.7623288.6388.92107
409687.5086.8823088.8888.78107

Table 5.

Cost model

With N=2BN=2B codes of dimension d=256d=256, InfoNCE’s similarity matrix costs 6N2d6N^{2}d FLOPs and about 16N216N^{2} bytes. ScoreCLR queries its score network (P=0.28MP=0.28\mathrm{M} parameters) separately for each code with S=5S=5 score-matching updates and K=3K=3 repulsion queries, about 2P(6S+2K)=202P(6S+2K)=20 MFLOPs per image (6.76.7 MFLOPs with S=1S=1). The CIFAR-10 recipe (S=1S=1, K=1K=1) costs 2P(6+2)=4.52P(6+2)=4.5 MFLOPs per image; Table 7 and Table 10 use the heavier ImageNet-100 setting; Figure 2 and Table 11 report three settings. In fp32, one score-matching step saves 30.2 KB per image for backward; the SS steps are sequential and the repulsion queries run without gradient, so this does not depend on SS and KK. The loss saves 9.2 KB more, which gives 39 KB per image. Architecture constants are counted on the implemented modules. For the full-iteration memory ratio, activations saved for backward are counted in 16-bit mixed precision; under data parallelism over WW devices each device holds a 2B/W×2B2B/W \times2B block of the similarity matrix, so the ratio is independent of WW. At B=32768B=32768 ScoreCLR’s objective needs 10× fewer FLOPs and 53× less memory than InfoNCE, and the per-device saving is 27% for ResNet-18 at 32 px and 8% at 224 px; SimCLR’s objective alone exceeds one 80 GB GPU at B=65536B=65536.

Objective onlyFull iteration, memory ratio
BBTFLOPs SimCLRTFLOPs ScoreCLRMemory (GB) SimCLRMemory (GB) ScoreCLRR18 32 pxR18 224 px
40960.100.081.10.160.960.99
81920.410.174.30.320.920.98
163841.650.3317.20.650.850.96
327686.600.6668.71.290.730.92
6553626.41.322752.580.570.85

Table 10. Analytical cost vs. global batch size BB (ScoreCLR: S=5S=5, K=3K=3). Left: the objective alone. Right: ScoreCLR / SimCLR per-device memory for a full data-parallel iteration.

Measured cost. Table 11 gives the measured time and extra peak memory of the objective alone, in the protocol of Figure 2, and Table 12 the time of a full training epoch, measured for both methods in the same job on the same GPU. In a full epoch the encoder dominates: ScoreCLR takes 1.08 times the time of SimCLR on CIFAR-10 at B=128B = 128, 1.01 times on ImageNet-100, and the same time within 3% from B=2048B = 2048 to 81928192, where it uses 2% less memory at B=4096B = 4096 and 6% less at B=8192B = 8192 in 16-bit precision (the model of Table 10 gives 4% and 8%). At B=32768B = 32768 the measured objective uses 44 times less memory than InfoNCE (53 in the model), and the per-device saving of one step measured in fp32 is 11% (Table 7), against 27% in the model, which counts 16-bit activations.

Overhead at small batch. The time of the ScoreCLR objective is nearly flat from B=128B = 128 to 81928192 (2.3–2.9 ms with S=1S = 1, K=1K = 1; Figure 2, Table 11) while the work grows by a factor 64, so at small batch the time is a fixed cost, set by the number of small operations in the score-network step (three linear layers, the modulation, the gradient clipping and the optimizer step) rather than by the amount of computation. We did not optimize this. A fused kernel for the score network and its update, or capturing the step in a CUDA graph, should reduce this fixed cost; we have not tested it.

Estimator of the Poincaré constant

The Poincaré constant is CP=sup⁡fVar⁡p(f)/Ep∥∇f∥2C_P = \sup_f \operatorname{Var}_p(f)/\mathbb{E}_p\lVert\nabla f\rVert^2, and we estimate it by maximizing this ratio over a class of test functions. Following Pillaud-Vivien et al. [43], we take ff in the span of the coordinates of zz and of M=1024M = 1024 random Fourier features cos⁡(wkTz+bk)\cos(w_k^\mathsf{T}z + b_k), with wk∼N(0,(ℓm)−2I)w_k \sim\mathcal{N}(0,(\ell m)^{-2}I), bkb_k uniform on [0,2π][0,2\pi] and mm the median distance between codes. The ratio is then a generalized Rayleigh quotient between the covariance of the features and the Gram matrix of their gradients, and it is maximized by a generalized eigenvector. We fit it on half of the validation codes, select the bandwidth factor ℓ∈{0.25,0.5,1,2}\ell\in\{0.25,0.5,1,2\}, a ridge term and the eigenvector on a quarter, and report the ratio on the last quarter, because the ratio measured on the fitting codes overestimates the constant. The constant scales as the square of the codes, so we first center and whiten them (PCA, keeping the directions whose variance exceeds 10−610^{-6} times the largest; 174 of 256 directions for SimCLR, whose output is collapsed). The estimate then reflects the shape of the distribution rather than its scale or anisotropy, and it equals 1 for a Gaussian. Since the estimate uses a fixed class of test functions, a smaller value is an indication, not a proof, that the codes are closer to a Gaussian.

References

  1. [1]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15619–15629, 2023.arxiv.org/abs/2301.08243
  2. [2]Dominique Bakry, Ivan Gentil, and Michel Ledoux. Analysis and Geometry of Markov Diffusion Operators. Springer, 2014.DOI
  3. [3]Randall Balestriero and Yann LeCun. Lejepa: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025. doi: 10.48550/arXiv.2511.08544. URL https://arxiv.org/abs/2511.08544.
  4. [4]Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, 2022.arxiv.org/abs/2105.04906
  5. [5]Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, 39(3):930–945, 1993.DOI
  6. [6]Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R. Devon Hjelm. MINE: Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 531–540. PMLR, 2018.arxiv.org/abs/1801.04062
  7. [7]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650–9660, 2021.arxiv.org/abs/2104.14294
  8. [8]Jiseok Chae, Kyuwon Kim, and Donghwan Kim. Open problem: Is there a first-order method that only converges to local minimax optima? In Proceedings of the Thirty-Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pp. 5957–5964. PMLR, 2023.
  9. [9]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, pp. 1597–1607. PMLR, 2020.arxiv.org/abs/2002.05709
  10. [12]Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.arxiv.org/abs/1311.3618
  11. [13]Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
  12. [14]Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley-Interscience, 2 edition, 2006.
  13. [15]Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23(56):1–6, 2022.
  14. [16]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE, 2009.
  15. [17]Mingyang Deng, He Li, Tianhong Li, Yilun Du, and Kaiming He. Generative modeling via drifting, 2026. URL https://arxiv.org/abs/2602.04770. arXiv preprint.
  16. [18]Persi Diaconis and David Freedman. Asymptotics of graphical projection pursuit. The Annals of Statistics, 12(3):793–815, 1984.DOI
  17. [19]Debidatta Dwibedi, Yusuf Aytar, Jonathan Thompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9588–9597, 2021.arxiv.org/abs/2104.14548
  18. [20]Giulio Franzese, Mustapha Bounoua, and Pietro Michiardi. Minde: Mutual information neural diffusion estimation. arXiv preprint arXiv:2310.09031, 2024. doi: 10.48550/arXiv.2310.09031.DOI
  19. [21]Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. In International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2210.02885
  20. [22]Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
  21. [23]Arthur Gretton, Olivier Bousquet, Alexander J. Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International Conference on Algorithmic Learning Theory, pp. 63–77. Springer, 2005.
  22. [24]Arthur Gretton, Li Kevin Wenliang, Alexandre Galashov, James Thornton, Valentin De Bortoli, and Arnaud Doucet. On the wasserstein gradient flow interpretation of drifting models. arXiv preprint arXiv:2605.05118, 2026.arxiv.org/abs/2605.05118
  23. [25]Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, pp. 21271–21284, 2020.arxiv.org/abs/2006.07733
  24. [26]Danijar Hafner, Juris Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2024. doi: 10.48550/arXiv.2301.04104.DOI
  25. [27]Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729–9738, 2020. doi: 10.1109/CVPR42600.2020.00975.DOI
  26. [28]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.arxiv.org/abs/2111.06377
  27. [29]R. Devon Hjelm, Alex Federov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.arxiv.org/abs/1808.06670
  28. [30]Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022.arxiv.org/abs/2110.09348
  29. [31]Adarsh Kappiyath, Silpa Vadakkeveetil Sreelatha, Xiatian Zhu, Yann LeCun, Lu Yin, and Amir Bar. Self-supervised collapse prevention with score-based entropy maximization. arXiv preprint arXiv:2611.11379, 2026.arxiv.org/abs/2611.11379
  30. [32]Sergei Kholkin, Ivan Butakov, Evgeny Burnaev, Nikita Gushchin, and Alexander Korotin. Info-Bridge: Mutual information estimation via bridge matching. arXiv preprint arXiv:2502.01383, 2025. doi: 10.48550/arXiv.2502.01383.DOI
  31. [33]Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.arxiv.org/abs/1312.6114
  32. [34]Frederic Koehler, Alexander Heckett, and Andrej Risteski. Statistical efficiency of score matching: The view from isoperimetry. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.00726
  33. [35]Simon Kornblith, Jonathan Shlens, and Quoc V. Le. Do better ImageNet models transfer better? In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.arxiv.org/abs/1805.08974
  34. [36]Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
  35. [37]Yann LeCun. A path towards autonomous machine intelligence, version 0.9.2, 2022-06-27. Open Review, 62(1):1–62, 2022.
  36. [38]Yazhe Li, Roman Pogodin, Danica J. Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. In Advances in Neural Information Processing Systems, volume 34, pp. 15543–15556, 2021.arxiv.org/abs/2106.08320
  37. [40]Elizabeth Meckes. Quantitative asymptotics of graphical projection pursuit. Electronic Communications in Probability, 14:176–185, 2009.arxiv.org/abs/0811.2769
  38. [41]Kimia Nadjahi, Alain Durmus, Lénaïc Chizat, Soheil Koulouri, Shahin Shahrampour, and Umut Simsekli. Statistical and topological properties of sliced probability divergences. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2003.05783
  39. [42]Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. arXiv preprint arXiv:1903.11780, 2019. doi: 10.48550/arXiv.1903.11780. URL https://arxiv.org/abs/1903.11780.
  40. [43]Loucas Pillaud-Vivien, Francis Bach, Tony Lelièvre, Alessandro Rudi, and Gabriel Stoltz. Statistical estimation of the poincaré constant and application to sampling multimodal distributions. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), volume 108 of PMLR, pp. 2753–2763, 2020.
  41. [44]Jean Ponce, Basile Tervier, Martial Hebert, and Michael Arbel. Dual perspectives on non-contrastive self-supervised learning. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/349a45f211fb1b3850da1ccd829e869e-Abstract-Conference.html.arxiv.org/abs/2507.01028
  42. [45]Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A. Alemi, and George Tucker. On variational bounds of mutual information. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 5171–5180. PMLR, 2019.arxiv.org/abs/1905.06922
  43. [46]Phillip Pope, Chen Zhu, Ahmed Abd el kader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations (ICLR), 2021.arxiv.org/abs/2104.08894
  44. [47]Pierre H. Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, and Michal Valko. Byol works even without batch statistics. arXiv preprint arXiv:2010.10241, 2020.arxiv.org/abs/2010.10241
  45. [48]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. arXiv preprint arXiv:2112.10752, 2022.arxiv.org/abs/2112.10752
  46. [49]Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. arXiv preprint arXiv:2102.11107, 2021. doi: 10.48550/arXiv.2102.11107.DOI
  47. [50]David W. Scott and M. P. Wand. Feasibility of multivariate density estimates. Biometrika, 78(1): 197–205, 1991.DOI
  48. [51]Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. arXiv preprint arXiv:1910.06222, 2020. doi: 10.48550/arXiv.1910.06222. URL https://arxiv.org/abs/1910.06222.
  49. [52]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11895–11907, 2019.arxiv.org/abs/1907.05600
  50. [53]Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations (ICLR), 2020. URL https://arxiv.org/abs/1907.13625.
  51. [54]Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.arxiv.org/abs/1807.03748
  52. [55]Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661–1674, 2011.DOI
  53. [56]Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11:3371–3408, 2010. URL https://jmlr.csail.mit.edu/papers/volume11/vincent10a/vincent10a.pdf. Editor: Léon Bottou.DOI
  54. [57]Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 9929–9939. PMLR, 2020. URL https://arxiv.org/abs/2005.10242.
  55. [58]Chun-Hsiao Yeh, Cheng-Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In European Conference on Computer Vision (ECCV), 2022.arxiv.org/abs/2110.06848
  56. [59]Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T. Freeman. Improved distribution matching distillation for fast image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2024a. doi: 10.48550/arXiv.2405.14867.arxiv.org/abs/2405.14867

Paper details

vlnc:W_sgx23tr7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC0-1.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Claude Code

View statement
Looking for related papers…
Contents