WorksMathematics
Contrastive latent representation learning with denoising score matching
Abstract
Contrastive representation learning methods operate by pulling different views of the same data sample together while pushing them away from views of different samples. A fundamental challenge is the estimation of the repulsive force, which typically requires large batch sizes, can exhibit high variance, and is susceptible to the curse of dimensionality. We introduce ScoreCLR, a contrastive representation learning method that offers a novel way to estimate this repulsion term. We consider the framework of representation learning by maximizing the mutual information between two latent views of a sample. By deriving the gradient of the Shannon entropy with respect to the encoder parameters, we obtain an equivalent objective that involves the score of the latent representation. We then use denoising score matching techniques to estimate this score with a lightweight network that is trained jointly with the encoder. We further propose regularization techniques using different noise levels and a Gaussian prior on the latent representation. ScoreCLR replaces the batch of explicit negative samples in contrastive methods with a learned repulsion whose cost is linear in the batch size rather than quadratic. On CIFAR-10 and ImageNet-100 it does so with minimal loss of linear-probe accuracy compared to SimCLR under a shared recipe, while producing projector outputs of higher effective rank, closer to the Gaussian prior.
Introduction
The goal of representation learning is to learn features from data that make downstream tasks easier to solve [22]. It is now widely used across several machine learning domains, including generative modeling [48], reinforcement learning [26], and causality [49]. Representation learning methods can be broadly divided into two categories: reconstructive approaches [56, 33, 28], which train by reconstructing samples from their latent encodings, and non-reconstructive ones [9, 7, 1], which optimize a training criterion in the latent space.
For the latter, two main approaches are typically considered. Contrastive methods [54, 9] bring representations of different views of the same sample closer while separating those of different samples. Estimating the repulsion force is generally challenging: it relies on in-batch negatives whose cost grows quadratically with the batch size, suffers from high variance and bias [45], and requires careful hyperparameter tuning (Chen et al., 2021). This has led to the development of non-contrastive methods that focus solely on aligning latent representations and prevent collapse through architectural design and training asymmetry [25, 7, 1] (Chen & He 2021) or explicit regularization in the training objective [4, 3]. Although these techniques do prevent collapse [44], it is unclear which structure they impose on the latent space, as well as the efficiency of the estimator they rely on (see § 3.4).
We introduce ScoreCLR, a novel method for contrastive representation learning that estimates the repulsion term using denoising score matching techniques [55, 52]. Our training criterion derives from maximizing the mutual information between two views of a sample [53]. By deriving the gradient of the Shannon entropy with respect to the encoder parameters, we construct an equivalent loss, sharing the same gradient, that involves the score of the latent representation . We then learn this score across different noise levels using a lightweight score network trained jointly with the encoder following the methodology of Yin et al. [59]. We call this score-based estimate of the entropy term of the mutual information ScoreReg, by analogy with VICReg [4] and SIGReg [3].
Unlike previous contrastive methods [9, 27] whose loss estimation is quadratic in the batch size, ours is linear: the repulsion is amortized in a score network instead of being computed against a batch (Figure 1). Empirically, ScoreCLR reaches competitive linear-probe accuracy on CIFAR-10 and ImageNet-100, while yielding projector outputs of higher effective rank than SimCLR and LeJEPA, which span only a fraction of the dimensions. The code is available at https://github.com/chris-mrn/ScoreCLR_code.
Figure 1. Repulsion force acting on a latent code (negative gradient of the loss). SimCLR (dashed) estimates it against the codes of the batch, with a cost per batch, whereas ScoreCLR (solid) uses the score of the latent distribution, learned by a small network, with a cost .
Representation learning via mutual information maximization
Let denote the random variable representing the data. We consider learning representations by learning to distinguish the joint distribution of two different latent views of from the product of their marginals. More formally, let denote a stochastic transformation of the data (e.g., cropping, masking, blurring, flipping, etc.) and , two i.i.d. copies of . We will consider the latent representation to be , where is an encoding function and denote and two latent views of . The objective is to maximize
where is a divergence criterion. Several divergences have been explored in the literature for learning representations such as Wasserstein [42], Maximum Mean Discrepancy (MMD) [23, 38], and Jensen–Shannon [29]. In our case we will consider the Kullback–Leibler (KL) divergence that inherits from desirable statistical properties [14]. We are then in the InfoMax principle (Linkser, 1988) framework and our objective is to maximize
Using this decomposition, we can interpret the objective as simultaneously encouraging the representation to have high entropy and making it predictable from the other representation view, through minimizing the conditional entropy . We will parametrize the encoding function with a deep neural network , denote , , and aim at minimizing
We will now present different ways to estimate the mutual information loss that can be found in the literature.
Variational bound and kernel density estimator
A common way to construct an estimator is to consider a lower bound using a dual representation of the KL. A standard choice is the Donsker–Varadhan representation, where the supremum is taken over measurable functions such that both expectations below are finite, which gives
where with independent of , so that . Any thus gives a lower bound on the mutual information. Let . We consider the critic
Since is independent of , , and the logarithm term of the Donsker–Varadhan bound vanishes. Using Monte Carlo estimators of the expectations with i.i.d. samples of and of , we obtain the loss
This loss is closely related to the InfoNCE loss of van den Oord et al. [54], the difference being that the positive pair does not appear in the denominator, as in decoupled contrastive learning [58].
Up to an additive constant, the repulsion term is minus a plug-in kernel density estimator of the entropy of the normalized latent on the sphere, with the kernel . Minimizing it encourages the latent representation to have high entropy, which favors uniformity on the sphere [57].
The SimCLR loss [9] is a close variant, which symmetrizes the loss with respect to the two views, uses the other samples of the batch from both views as negatives, and includes the positive pair in the denominator. Other works modify the samples used in this estimator. NNCLR [19] replaces by its nearest neighbour in a queue of embeddings stored from previous batches, and MoCo [27] draws negatives from a dynamic memory queue, which decouples the number of negatives from the batch size and enables the use of smaller batch sizes.
These estimators all suffer from the curse of dimensionality under standard regularity conditions on the density of interest [50, 37]. The relevant dimension is in general the intrinsic dimension of the data rather than the ambient one, but it has been recently suggested that this intrinsic dimension is still too large for standard statistical estimators on real-world data [46]. Contrary to classic kernel density estimators, which only leverage a weak prior on the distribution of interest, the ScoreCLR repulsion estimator includes a neural prior that may benefit from a better sample complexity [5].
Neural methods
Several approaches have been proposed to estimate or maximize mutual information in representation learning relying on neural critics. For example, MINE [6] used the dual representation of the KL with a critic learned by a neural network , which was subsequently used for unsupervised representation learning in Deep InfoMax [29]. The mutual-information-based loss can be rewritten as
where with independent from .
Training then proceeds by alternating maximization over the critic parameters and maximization of the resulting bound over the encoder parameters . An estimator of the mutual information during training is given by
In practice, these variational estimators can suffer from optimization instability and can lead to collapsed representations [8]. Moreover, these neural estimators do not necessarily preserve desirable properties of the underlying KL divergence, such as invariance to reparameterization [51].
ScoreCLR
More recently, diffusion-based approaches have been proposed for mutual information estimation [20]. These methods achieve improved performance and pass the invariance to reparameterization test [32]. To the best of our knowledge, these diffusion-based approaches to mutual information estimation have not yet been explored in a self-supervised representation learning framework.
Equivalent loss
Since we optimize our loss with gradient descent, replacing it by another loss with the same gradient does not change the training dynamic. The following proposition gives a way to make such a replacement for the entropy term.
Proposition 3.1. For any random variable , denoting , under Assumption A.1 (Appendix § 6) we have
where sg denotes the stop grad operator.
For the conditional entropy, we will consider the Gaussian conditional model which will give a lower bound on the mutual information,
that gives the same training dynamic as
We call its second term ScoreReg. This loss can be seen as enforcing invariance of the latent representation to the given transformation while encouraging it to remain of high entropy.
Regularization
Gaussian prior Without any extra regularization the problem is only constrained by the architecture of the network and the entropy of the latent could grow arbitrarily high. To alleviate this issue we consider a Gaussian prior on our latent variable by adding an penalty term to the loss. Noticing that , we recover the same regularization term as in Balestriero & LeCun [3], the maximum-entropy prior with mean 0 and variance 1. In practice the penalty is applied to the noised latents at which the score is queried (Algorithm 1).
Noise regularization To estimate the score of the latent we use denoising score matching (Vincent, [55]; Song & Ermon [52]): we approximate the score of the latent distribution convolved with Gaussian noise of standard deviation . We denote , where , and train a lightweight noise-prediction network with the denoising loss
where is the noise level distribution, specified in § 4.1. The score is recovered by Tweedie’s formula, . Further details can be found in Appendix § 8.
For simplicity, we replace the factor by a tunable weight . The regularized loss is
Using Proposition 3.1, will give the same training dynamic as
To have a low variance estimate of the repulsive term, we may average it over noise levels, using independent Gaussian noise for each level.
Training
We want, for every update of , a statistical estimator of the score that minimizes well . To do so, we follow the methodology of distribution matching distillation (Yin et al., [59]), which alternates training between updates of and with a 1-to- proportion rule that has proven to be efficient in large-scale training. The score network is unconditional: it takes only the noised code and the noise level.
The final loss can be made of three components, an invariance term , a repulsion term that estimates ScoreReg, and a regularization term , the last two forming the prior term . On a batch, their expressions are
where is the batch size, is the dimension of the codes, , with and drawn independently for each sample, view and query, and . The training procedure is summarized in Algorithm 1. Connections to the SimCLR loss and to drifting generative models (Deng et al., [17]) are given in Appendices § 7.§ 1 and § 7.§ 2, respectively.
Algorithm 1 ScoreCLR training
Require: Encoder , noise-prediction network , augmentations , , noise distribution , score-network steps , noise queries , weight
1: for each batch do 2: for 3: for do ▷ score-network updates 4: Sample new and for every and 5: 6: update of 7: end for 8: Sample new , , , and compute and with the updated 9: Compute with Eq. (1) 10: update of 11: end for
Comparison with other regularization methods
Several explicit regularization terms have been proposed to prevent collapse in non-contrastive self-supervised learning. VICReg [4] regularizes the variance and the covariance of the latent and thus implicitly assumes that the latent is Gaussian. SIGReg [3] pushes the latent toward an isotropic Gaussian with an Epps–Pulley characteristic-function test on random one-dimensional projections, relying on the Cramér–Wold theorem. As for every sliced divergence, the sample complexity of this test does not depend on the dimension [41], but its statistical power does. Under concentration assumptions on the latent, most one-dimensional projections of a high-dimensional distribution are close to Gaussian [18], and the fraction of directions in which a projection deviates from Gaussian by more than decays as [40], so the number of random slices needed to detect such a deviation grows exponentially with the dimension, which recovers the curse of dimensionality in the statistical power of the test. A latent supported on a low-dimensional subspace can thus pass the test, which is the geometric collapse we observe for LeJEPA in § 4.3.
ScoreCLR instead derives its repulsion from the contrastive criterion itself, as the gradient of the entropy term of the mutual information estimated with the full score of the latent, and does not suffer from this geometric collapse (§ 4.3).
Concurrent work. We recently became aware of the concurrent and independent work of Kappiyath et al. [31], which also trains a latent score network jointly with the encoder and uses the same identity between the gradient of the entropy and the score (Proposition 3.1) to prevent collapse. They report strong results on ImageNet-1K with ResNet and ViT encoders, and also use the learned score at test time to adapt the representations of corrupted images. The two methods differ in how the latent is constrained, how the score is learned and how the entropy term is written. Kappiyath et al. [31] regularize the embeddings, learn the score at a single noise level with 16 score-network updates per encoder step, and use the linear surrogate . ScoreCLR constrains the latent with a soft Gaussian prior, which gives the KL divergence to , learns the score over a distribution of noise levels with one to five score-network updates per step, and writes the entropy term together with the prior as a regression towards a target moved along the drift field , the velocity of the Wasserstein gradient flow of (Appendix B.2); both forms give the same encoder gradient. We also study the geometry of the codes and the cost of the objective as the batch size grows (§ 4.3 and § 4.7).
Experiments
We compare ScoreCLR with SimCLR, and on CIFAR-10 with LeJEPA [3], BYOL [25] and Barlow Twins (Zbontar et al., 2021), on linear evaluation, representation geometry, out-of-domain transfer, batch size, numerical precision and training cost. The main results of SimCLR, LeJEPA, BYOL, Barlow Twins and ScoreCLR under AdamW are means standard deviations over three seeds (1, 2, 3); the SimCLR (LARS) rows and the batch-size study are single runs (seed 5). Unless stated otherwise, runs are in fp32; SimCLR (LARS) is trained in 16-bit mixed precision as in solo-learn (da Costa et al., [15]). ScoreCLR uses , one score-network step () and one noise query per code () on CIFAR-10, and , and on ImageNet-100 (Appendix C); the AdamW methods share the same augmentations and gradient clipping (norm 1.0).
Setup
Data. We pre-train on CIFAR-10 (Krizhevsky, [36]) and ImageNet-100 (Deng et al., [16]). Transfer targets are CIFAR-10, STL-10 (Coates et al., [13]), and DTD (Cimpoi et al., [12]).
Models. ResNet-18 with a two-layer MLP projector (2048 hidden units, 256 outputs, no BatchNorm: a BatchNorm layer improves the results but partly prevents collapse on its own (Richemond et al., [47]), so we remove it). SimCLR, LeJEPA, BYOL, Barlow Twins and ScoreCLR share the encoder, the projector, the optimizer (AdamW), the augmentations and the gradient clipping (Table 8, Appendix C). We also report SimCLR with LARS under solo-learn’s recipe (da Costa et al., [15]), with the shared 2048/256 head and its published 4096/512 head.
| Shared AdamW | SimCLR (LARS) | ScoreCLR | |
| Backbone; epochs | ResNet-18; 1000 / 400 | ResNet-18; 1000 / 400 | ResNet-18; 1000 / 400 |
| Batch size | 128 / | 256 / | 128 / |
| Optimizer | AdamW | LARS | AdamW (encoder and score net) |
| Encoder LR | 0.4 / 0.3 | / | |
| Weight decay | (score net ) | ||
| LR schedule | constant, cosine over last 100 ep | 10-ep warm-up + cosine | constant, cosine over last 100 ep (encoder and score net) |
| Gradient clip | 1.0 | none | 1.0 (encoder and score net) |
| Precision | fp32 | 16-bit mixed | fp32 |
| Augmentations | symmetric pipeline (Appendix C.1) | solo-learn’s own | same as shared AdamW |
| Projector | Linear–ReLU–Linear, no BatchNorm, 2048 / 256 | (LARS control: 4096 / 512) | |
| Classifier LR | 0.01 | 0.1 | 0.01 |
| Objective | method-specific | ; repulsion and prior weight 1 | |
| Score network | – | – | unconditional -MLP-FiLM, , 64-d emb., 0.28M |
| Score-net LR; steps | – | – | (10× / 5× encoder); 1 / 5 |
| Queries | – | – | 1 / 3 |
| DSM noise | – | – | / |
Table 8. Pre-training hyperparameters (CIFAR-10 / ImageNet-100 where two values are given). “Shared AdamW” is the recipe of SimCLR, LeJEPA, BYOL and Barlow Twins. ImageNet-100 uses images per step with DDP-averaged gradients and no learning-rate rescaling.
Noise levels. The noise levels of ScoreCLR follow on CIFAR-10 and on ImageNet-100.
Evaluation. (i) Offline linear probe and weighted -NN () on frozen features; (ii) alignment and uniformity (Wang & Isola, [57]) of the projector output; (iii) effective rank (Garrido et al., [21]) of the backbone features and of the projector output, and an estimate of the Poincaré constant of the projector output; (iv) out-of-domain linear transfer following Kornblith et al. [35]; (v) measured time and peak memory of the objective, of a training step and of a training epoch.
Linear and k-NN evaluation
Discussion. With the same optimizer and projector, ScoreCLR is competitive with the contrastive and joint-embedding baselines. On CIFAR-10 its linear accuracy is 0.4 points below SimCLR (within one standard deviation) and 0.8 points below LeJEPA, within 0.1 of BYOL and 0.8 above Barlow Twins, and its -NN accuracy is within 0.5 points of SimCLR, LeJEPA and BYOL. On ImageNet-100 the linear probe is 1.2 points below SimCLR and the -NN accuracy 0.4 points below, with a smaller seed spread (0.06 against 0.62). SimCLR with LARS is stronger than all AdamW methods; these rows come from solo-learn’s own pipeline and are reference points, not a controlled comparison of objectives (Appendix C.1). The takeaway is that replacing the batch-based repulsion by a learned score costs 0.4 points of linear accuracy on CIFAR-10 and 1.2 on ImageNet-100: what the method offers lies in the geometry of and in the cost of the objective at large batch sizes (§ 4.3 and § 4.7), not in the probe accuracy.
Representation Geometry
Table 2 reports alignment and uniformity of the projector output and Table 3 the effective rank (Garrido et al., [21]) of the backbone features and of the projector output . In , SimCLR spans about half of the 256 dimensions on ImageNet-100 and a third on CIFAR-10, the dimensional collapse of InfoNCE described by Jing et al. [30]. LeJEPA has the lowest rank in both (253/512) and (58/256) although it is trained with the SIGReg criterion (§ 3.4). ScoreCLR’s has a much higher effective rank (229/256 on CIFAR-10 and 244/256 on ImageNet-100), and its uniformity equals SimCLR’s on CIFAR-10 and is better on ImageNet-100 (−3.93 against −3.86). It pays for this with a worse alignment (0.51 against 0.37 for SimCLR on CIFAR-10, and 0.38 against 0.24 on ImageNet-100): the positive pairs are held less tightly together. The takeaway is that the score-based repulsion spreads the codes over almost all the available dimensions, which neither InfoNCE nor the SIGReg regularizer does. The ranks of the backbone features are similar for ScoreCLR and SimCLR on CIFAR-10 (332 and 336/512) and slightly higher for ScoreCLR on ImageNet-100 (346 against 329/512).
Table 3 also reports an estimate of the Poincaré constant of the codes. The Poincaré constant of a distribution is the smallest constant such that for every smooth function (Bakry et al., [2]). It equals 1 for the standard Gaussian, and it is large when the distribution has separated modes or strongly anisotropic directions: it is the inverse of the spectral gap of the Langevin diffusion that targets , so it controls how fast that diffusion mixes (Bakry et al., [2]) and the statistical efficiency of learning the score of by score matching (Koehler et al., [34]). It is therefore a natural measure of how close the geometry of is to the Gaussian prior of ScoreCLR. We estimate it with the random-feature estimator of Pillaud-Vivien et al. [43] (Appendix § 8.8). With this estimator, on whitened codes, the value for ScoreCLR is the smallest of the three methods on CIFAR-10 (it was not computed on ImageNet-100).
| Method | Optim. | CIFAR-10 | ImageNet-100 | ||
| Align | Unif | Align | Unif | ||
| SimCLR | LARS | 0.349 | −3.818 | 0.240 | −3.856 |
| SimCLR | AdamW | 0.367±0.004 | −3.804±0.002 | 0.235±0.001 | −3.855±0.001 |
| LeJEPA | AdamW | 0.316±0.003 | −3.540±0.003 | – | – |
| ScoreCLR (ours) | AdamW | 0.514±0.006 | −3.802±0.001 | 0.378±0.005 | −3.931±0.000 |
Table 2. Alignment (↓) and uniformity (↓) (Wang & Isola, [57]) of the projector output, ResNet-18 (AdamW rows: mean ± std over three seeds; LARS rows: single run). “–”: not measured.
Transfer
Table 4 probes frozen features on unseen datasets, STL-10 and DTD. ScoreCLR transfers slightly worse than SimCLR and is close to LeJEPA. After CIFAR-10 pre-training, it is 0.9 points below SimCLR on STL-10 and 2.5 below on DTD, and close to LeJEPA (75.2 against 76.0 on STL-10, 28.0 against 27.0 on DTD). After ImageNet-100 pre-training the gap to SimCLR is larger (1.5 points on STL-10, 3.1 on DTD, 4.4 on CIFAR-10), in line with the lower linear accuracy of Table 1. The takeaway is that the higher effective rank of does not by itself make the backbone features transfer better: transfer follows the in-domain accuracy of the representation.
| Method | Optim. | Projector | CIFAR-10 Linear Top-1 | CIFAR-10 Linear Top-5 | CIFAR-10 -NN Top-1 | CIFAR-10 -NN Top-5 | ImageNet-100 Linear Top-1 | ImageNet-100 Linear Top-5 | ImageNet-100 -NN Top-1 | ImageNet-100 -NN Top-5 |
| SimCLR | AdamW | 2048/256 | 89.260.38 | 99.70 | 88.730.18 | 98.90 | 72.390.62 | 92.18 | 66.410.44 | 87.32 |
| LeJEPA | AdamW | 2048/256 | 89.670.13 | 99.65 | 88.400.13 | 98.65 | – | – | – | – |
| BYOL | AdamW | 2048/256 | 88.960.30 | 99.70 | 88.440.22 | 99.01 | – | – | – | – |
| Barlow Twins | AdamW | 2048/256 | 88.050.15 | 99.57 | 87.170.09 | 98.57 | – | – | – | – |
| ScoreCLR (ours) | AdamW | 2048/256 | 88.870.44 | 99.66 | 88.270.13 | 98.85 | 71.230.06 | 91.03 | 66.050.39 | 86.09 |
| SimCLR | LARS | 2048/256 | 90.79 | 99.74 | 89.99 | 99.02 | 75.40 | 93.30 | 69.98 | 88.64 |
| SimCLR | LARS | 4096/512 | – | – | – | – | 77.02 | 93.84 | 71.60 | 89.10 |
Table 1. Linear-probe and weighted -NN () accuracy (%) on frozen features, ResNet-18. Top: shared optimizer and projector (AdamW, gradient clip 1.0, 2048/256 projector without BatchNorm); mean std over three seeds (top-5 averaged); BYOL and Barlow Twins use a projector and predictor without BatchNorm (Appendix C). Bottom: SimCLR trained with solo-learn’s LARS recipe, with the shared 2048/256 projector and with the published 4096/512 projector. With the latter, our offline linear probe gives 77.02 on ImageNet-100, against 77.64 for the online probe of the published checkpoint.
Table 4: Out-of-domain linear transfer, top-1 (%): logistic regression on frozen L2-normalized features at the pretraining resolution (Appendix § 8.§ 5). “·” is the pretraining dataset (Table 1). All rows: mean std over three seeds.
| Method | Optim. | CIFAR-10 | STL-10 | DTD |
| Pretrained on CIFAR-10 (ResNet-18, 32 px) | ||||
| SimCLR | AdamW | · | ||
| LeJEPA | AdamW | · | ||
| BYOL | AdamW | · | ||
| Barlow Twins | AdamW | · | ||
| ScoreCLR (ours) | AdamW | · | ||
| Pretrained on ImageNet-100 (ResNet-18, 224 px) | ||||
| SimCLR | AdamW | |||
| ScoreCLR (ours) | AdamW |
Table 4.
Batch size
We train SimCLR and ScoreCLR at larger batches (Table 5). The encoder learning rate follows the square-root rule () for both methods; for ScoreCLR the score-network learning rate is kept fixed. ScoreCLR stays within 0.9 points of SimCLR up to and is 1.4 points below at . With a larger batch there are fewer iterations per epoch, and so fewer score-network updates. In our runs, the number of score-network updates per step had to grow with the batch size to keep the accuracy (Appendix § 8.6). In these experiments, ScoreCLR does not reduce the batch size needed for a given accuracy; at large batches, its main benefit is the lower cost of the objective (§ 4.7).
Table 3: Effective rank (RankMe, Garrido et al., [21]) of the backbone features and of the projector output on raw (un-normalised) validation features, out of the space dimension, and an estimate of the Poincaré constant of the whitened (1 for a Gaussian; lower is closer to Gaussian), estimated with the random-feature estimator of Pillaud-Vivien et al. [43] (Appendix § 8.8). Both datasets use a ResNet-18. AdamW rows: mean std over three seeds; LARS rows: single run. The low rank of under InfoNCE is the dimensional collapse analysed by Jing et al. [30].
| Pre-training | Method | Optim. | RankMe (/512) | RankMe (/256) | of |
| CIFAR-10 | SimCLR | LARS | 372 | 88 | – |
| ResNet-18 | SimCLR | AdamW | |||
| LeJEPA | AdamW | ||||
| BYOL | AdamW | – | |||
| Barlow Twins | AdamW | – | |||
| ScoreCLR (ours) | AdamW | ||||
| ImageNet-100 | SimCLR | LARS | 368 | 118 | – |
| ResNet-18 | SimCLR | AdamW | – | ||
| ScoreCLR (ours) | AdamW | – |
Table 3.
Numerical precision
SimCLR (LARS) is trained in 16-bit mixed precision (solo-learn), while the other runs use fp32. ScoreCLR works equally well in fp16: over three seeds, fp16 training reaches linear accuracy against in fp32, with the same effective rank of (Table 6). The comparison with the mixed-precision SimCLR (LARS) baseline is therefore not biased by the precision.
| Precision | Linear | -NN | RankMe |
| fp32 | 88.870.44 | 88.270.13 | 2290 |
| fp16 | 88.820.04 | 88.390.05 | 2291 |
Table 6. ScoreCLR on CIFAR-10 in fp32 and fp16 mixed precision (encoder and objective under autocast, score-network updates in fp32); mean std over seeds 1–3.
Computational Cost
InfoNCE compares every pair of codes in the batch, so the cost of its objective grows quadratically with the batch size . ScoreCLR instead passes each code through a small score network (0.28M parameters, 2.2% of the encoder and projector) a fixed number of times, updates and queries, so the cost of its objective grows linearly with . We measured both objectives on one H100 GPU in fp32 (Figure 2, Table 11). For the objective alone, InfoNCE is faster at small batches (0.6 ms against 2.3 ms at ), ScoreCLR uses less memory from k, and it is faster from k–4k with and above k with , the number of score-network updates the large-batch runs need. At , InfoNCE takes 228 ms and 48.3 GiB; ScoreCLR takes 7.6 ms with , , 30 ms with the ImageNet-100 recipe (, ) and 48.5 ms with , , and 1.1 GiB in each case. In a full training step the encoder dominates, so the gain is smaller but measured: one step of one device at over 8 devices (emulated on one GPU) takes 508 ms and 43.1 GiB for ScoreCLR (, ) against 531 ms and 48.7 GiB for SimCLR (the encoder alone takes 500 ms; Table 7), and a full epoch takes the same time within 8% (Table 12).
Figure 2. Measured cost of the objective alone vs. the batch size (images), on one H100 GPU (fp32). Codes are given; we time one step of the loss (forward and backward to the codes; for ScoreCLR, score-network updates and queries, as indicated; , is the ImageNet-100 recipe), median over 30 iterations, and report the extra peak memory, which does not depend on or .
| Time (ms) | Peak memory (GiB) | |
| Encoder only (no objective) | 499.8 | 43.1 |
| ScoreCLR, | 508.2 | 43.1 |
| SimCLR | 530.9 | 48.7 |
Table 7. Cost of one full training step per device (ResNet-18, CIFAR-10 resolution, fp32, one H100): images per device, i.e. 32768 images over 8 devices, emulated on one GPU. ScoreCLR: .
| Time per step (ms): InfoNCE | Time per step (ms): ScoreCLR, , | Time per step (ms): ScoreCLR, , | Time per step (ms): ScoreCLR, , | Extra memory: InfoNCE | Extra memory: ScoreCLR | |
| 128 | 0.62 | 2.3 | 8.2 | 15.1 | 2.0 MiB | 5.4 MiB |
| 512 | 0.78 | 2.4 | 8.5 | 16.0 | 16 MiB | 19 MiB |
| 2048 | 1.4 | 2.5 | 8.6 | 14.9 | 208 MiB | 71 MiB |
| 4096 | 4.4 | 2.5 | 8.9 | 15.2 | 800 MiB | 141 MiB |
| 8192 | 14.7 | 2.4 | 9.1 | 14.8 | 3.1 GiB | 281 MiB |
| 32768 | 228 | 7.6 | 30.3 | 48.5 | 48.3 GiB | 1.1 GiB |
Table 11. Measured cost of the objective alone (one H100, fp32, median over 30 iterations; same job as Figure 2): time per step and extra peak memory. The memory of ScoreCLR depends on neither nor .
| Setting | ScoreCLR | Epoch time (s) | Peak memory (GiB) |
| CIFAR-10, , fp32 | , | 8.96 / 8.28 | – |
| ImageNet-100, , fp32 | , | 71.0 / 70.3 | 5.63 / 5.62 |
| CIFAR-10, , fp16 | , | 6.81 / 6.64 | 11.0 / 11.1 |
| CIFAR-10, , fp16 | , | 8.27 / 8.28 | 21.7 / 22.2 |
| CIFAR-10, , fp16 | , | 10.00 / 9.98 | 43.1 / 45.8 |
Table 12. Measured time of one training epoch and peak memory, ScoreCLR / SimCLR (CIFAR-10: one H100; ImageNet-100: two H100s, memory per GPU). “–”: not recorded. The row was timed with queries, more work per step than the recipe of Table 1.
Conclusion
We introduced ScoreCLR, which replaces the batch-based repulsion of contrastive learning by a score-based estimate of the entropy gradient, learned jointly with the encoder. A natural extension is a fully score-based critic, where the conditional entropy is also derived from the learned score. Since the score network predicts clean information from a noisy input, it would also be interesting to test whether the predictor of JEPA methods (Assran et al., [1]), which predicts the clean latent of a target from a masked context, could by itself be a proxy to enforce high entropy. Finally, since the score-based estimator keeps the projector output at a higher effective rank than existing regularizers (§ 4.3), it would be interesting to study priors with finer geometric properties than the isotropic Gaussian and to compare with those methods in that setting. Since ScoreCLR learns the score of the latent distribution at all noise levels, we also plan on investigating whether its score network can serve as a generative model of the representation.
Code and reproducibility
All training recipes, evaluation protocols and the cost model are given in Appendix § 8, and the proof of Proposition 3.1 in Appendix § 6. The ScoreCLR implementation is a self-contained PyTorch code base, and the SimCLR (LARS) baselines use the public solo-learn library (da Costa et al., [15]); the code is available at https://github.com/chris-mrn/ScoreCLR_code.
AI use statement
We used generative AI tools (Claude Code) to implement the training, evaluation and cost-measurement code of ScoreCLR and of the baselines, to launch and monitor experiments, to suggest ablations, to draft the analysis of the experimental results and parts of the experimental section, to assist in writing proofs, to create figures, to edit the text for grammar, to identify relevant literature and to format references. We did not use them to generate data, to develop theoretical models, to formulate mathematical claims, to provide critical ingredients of proofs, or to propose hypotheses. The authors verified the code against the equations of the paper, checked every number in the tables against the training logs and read all suggested references. We take full responsibility for the content of this paper.
Proofs
Assumption A.1.
(i) For every , the map is continuously differentiable.
(ii) has a density that is positive everywhere, continuously differentiable in , with finite entropy, finite second moment and finite Fisher information .
(iii) Derivatives with respect to can be exchanged with expectations and integrals (e.g. by dominated convergence).
Proof of Proposition 3.1. Since , the law of the unconscious statistician gives
The encoder parameters enter in two places: through the point at which the density is evaluated, and through the density itself. Differentiating with respect to ,
where in the second term acts on the parameter index of the density only, the evaluation point being held fixed.
The second term vanishes. Indeed,
the interchange of and the integral being justified by dominated convergence. Hence
It remains to identify the right-hand side as the gradient of the stated surrogate. Write
The stop-gradient has zero derivative, so and only the first occurrence of is differentiated:
At the point of evaluation the stop-gradient is the identity, so , and therefore
by equation 2.
Connection to other methods
ScoreCLR was first motivated by a comparison between distribution matching distillation [59] and generative modeling by drifting [17]. We present below its links with SimCLR and with drifting models.
SimCLR
SimCLR normalizes the codes, so . Then and is a Gaussian kernel of variance . Up to an additive constant, the SimCLR loss [9] writes
The invariance term is the same as in ScoreCLR. Since , the gradient of the repulsion with respect to is
whereas for ScoreCLR, with and ,
The first term of the bracket is the score of a Gaussian kernel density estimator (KDE) of bandwidth built on the other codes, i.e. a batch estimate of the score of at . It is the counterpart of the ScoreCLR term, with two differences: (i) it costs and uses a fixed , whereas ScoreCLR uses a learned score at a random at cost ; (ii) it is evaluated at the clean code , whereas ScoreCLR evaluates it at .
The second term has no counterpart in ScoreCLR. It appears because is also a kernel centre in the sums of the other anchors, so moving changes the estimated density at the other codes. For the true density this effect is zero, as in the proof of Proposition 3.1:
ScoreCLR never computes it, since the score network is a stop-gradient. In SimCLR, the KDE is built from the codes being differentiated, so the term is not exactly zero on a finite batch.
Finally, SimCLR has no regularization term : the constraint plays the role of the Gaussian prior.
Drifting models
Considering the terminology of Deng et al. [17], we denote as drifting field
the velocity field of the Wasserstein gradient flow of [24]. Symmetrizing the repulsion force with respect to the views and using the stop-gradient trick of DMD [59], we get the mutual information loss with Gaussian prior, which writes as
with the same convention as Eq. (1): weights the invariance term and the prior term has weight one.
Experimental Details
Pre-training protocol
Shared recipe. SimCLR, LeJEPA, BYOL, Barlow Twins and ScoreCLR share the ResNet-18 encoder, the two-layer MLP projector (Linear–ReLU–Linear, 2048 hidden units, 256 outputs, no BatchNorm), the optimizer (AdamW, weight decay ), the gradient-norm clip 1.0, the schedule (constant learning rate, then a cosine decay to zero over the last 100 epochs), the batch size (128 for 1000 epochs on CIFAR-10, for 400 epochs on ImageNet-100), the augmentations and fp32 precision (Table 8). The clip applies to the encoder, the projector and the online classifier and, for ScoreCLR, also to the score network. Only the encoder learning rate is tuned per method: for SimCLR, LeJEPA, BYOL and Barlow Twins, and (CIFAR-10) or (ImageNet-100) for ScoreCLR.
Augmentations. SimCLR, LeJEPA, BYOL, Barlow Twins and ScoreCLR use the same symmetric pipeline on both datasets: random resized crop (scale [0.08, 1]), horizontal flip (probability 0.5), colour jitter (0.8, 0.8, 0.8, 0.2) applied with probability 0.8, grayscale with probability 0.2 and Gaussian blur with probability 0.5, without solarization. The SimCLR (LARS) rows come from solo-learn’s own pipeline (data loading and augmentations) and are reference points, not controlled comparisons.
Baselines. SimCLR uses the symmetrized InfoNCE loss with temperature and negatives gathered across devices. LeJEPA replaces the repulsion by SIGReg (, 17 knots, , 256 slices). BYOL uses a 4096-unit predictor without BatchNorm and an EMA coefficient that follows a cosine from 0.99 to 1; Barlow Twins uses and a loss scale of 0.1, both solo-learn’s CIFAR-10 values. BYOL and Barlow Twins are run in this harness with seeds 1–3; removing BatchNorm from their predictor and projector disadvantages them (BYOL reaches 89.96% linear accuracy with a BatchNorm predictor, seed 1). The SimCLR (LARS) rows follow solo-learn’s recipe: LARS (momentum 0.9, , biases and normalization layers excluded), weight decay , a 10-epoch warm-up from followed by a cosine decay, , 16-bit mixed precision, learning rate 0.4 with batch 256 on CIFAR-10 (1000 epochs) and learning rate 0.3 with batch on ImageNet-100 (400 epochs). The projector is narrowed from the published 4096/512 to the shared 2048/256 head, except in the control row, which keeps the published head and every other published default.
ScoreCLR implementation
Loss. We minimize Eq. (1), , with on CIFAR-10 and on ImageNet-100. In the Gaussian conditional model used to derive the loss, , i.e. and 0.25. The repulsion and the prior carry weight one: they are two parts of one estimate of and are not tuned separately. Each of the three terms is divided by the code dimension ; this common factor leaves the relative weights unchanged and only matters through the gradient clip. The repulsion and the prior are averaged over independent noise queries per code, with on CIFAR-10 and on ImageNet-100. The noise levels follow with on CIFAR-10 and 1.0 on ImageNet-100.
Score network. The score network is an unconditional noise-prediction MLP : three linear layers of width 256 with SiLU activations on the concatenation of and a 64-dimensional sinusoidal embedding of , the two hidden layers being FiLM-modulated by the same embedding (modulation initialized to the identity). It has 0.28M parameters, 2.2% of the encoder and projector. It is trained with the unweighted loss on the detached codes of both views, with fresh noise for each of the updates per encoder step ( on CIFAR-10, on ImageNet-100), and queried with . Its optimizer is AdamW with learning rate , weight decay , gradient clip 1.0. Like the encoder’s, its learning rate is constant and then follows a cosine decay to zero over the last 100 epochs.
Linear evaluation
The offline probe trains a linear classifier for 100 epochs on frozen backbone features in fp32, with the standard crop-and-flip augmentation, using the same protocol for every method; the only exception is the -NN accuracy of the SimCLR (LARS) rows, computed with a memory bank under crop-and-flip augmentation. On CIFAR-10 the features are used raw and the classifier is trained with SGD (momentum 0.9, learning rate 0.1, divided by 10 at epochs 60 and 80). On ImageNet-100 the features are L2-normalized and the classifier is trained with SGD (momentum 0.9, learning rate 10, warm-up then cosine). The -NN accuracy is a weighted cosine -NN (, temperature 0.07) of the validation features against the clean training set. Alignment is computed on L2-normalized projector outputs, uniformity on 4000 validation codes, and RankMe on the 10,000 validation codes.
Effective rank
RankMe ([21]) is the exponential of the entropy of the normalized singular values of the feature matrix, computed on the validation set. We report it on both the backbone features and the projector output because Jing et al. [30] (DirectCLR) showed that contrastive learning can collapse the projector output (dimensional collapse) while the backbone stays full-rank.
Out-of-domain transfer
Following Kornblith et al. [35], a multinomial logistic-regression classifier is fit on frozen, L2-normalized backbone features of each downstream training set. The L2 penalty is selected on a validation split (the dataset’s own where one exists, else a stratified 20% hold-out) and the classifier is refit on train + val. Downstream images are resized and center-cropped to the pretraining resolution: with the CIFAR stem for CIFAR-10 backbones and for ImageNet-100 backbones.
Ablations of the recipe
Table 9 varies ingredients of the CIFAR-10 recipe (batch 128, 1000 epochs, ). The reference is , , with clip 1.0 (mean standard deviation over seeds 1–3); a paired row is compared with the reference run of the same seed.
Conditioning. A score network that also receives a second latent as input, namely the other view of the same image when it is trained to denoise and the code of another image of the batch when it is queried, loses 3.3 to 5.6 points of linear accuracy with respect to the unconditional network
| Variant | Seeds | Linear | -NN | RankMe | Ref. |
| Reference (, , , clip) | 1–3 | 229 | – | ||
| 1–3 | 233 | – | |||
| Conditional score network | 1 / 2 | 83.83 / 85.27 | 83.54 / 84.34 | 25 / 51 | 89.38 / 88.58 |
| , no clipping{sup}†` | 5, 42 | 88.87 / 88.85 | 88.44 / 88.43 | 229 | – |
| , no clipping | 1–3 | – |
Table 9. Ablations on CIFAR-10: linear accuracy (%), -NN accuracy (%) and RankMe of . “Ref.” is the linear accuracy of the same-seed run with the same noise width and clip. †: seeds 5 and 42, run with an earlier code snapshot that implements the same loss. The row uses seeds 1–3 without gradient clipping.
(Table 9). The recipe was not re-tuned for the conditional network, so this ablation measures the conditioning inside the present recipe only.
Queries and clipping. Averaging queries without gradient clipping is unstable across seeds (79.45, 83.16 and 73.26 for seeds 1–3), whereas without clipping gives 88.87 and 88.85. We use the clip for fairness with the baselines, for which it is neutral (SimCLR: 89.41 without it, 89.45 with it, seed 5). On ImageNet-100, with the clip is stable ( over three seeds).
Score-network steps. Extra score-network steps matter when iterations are scarce. On ImageNet-100 (494 iterations per epoch), and reach 57.14 against for the recipe of Table 8. At batch 2048 on CIFAR-10 (24 iterations per epoch), is not sufficient, gives 87.84 and 88.29.
Larger batches. At , gives 87.50 with (Table 5) and 87.56 with ; with , gives 85.76.
Table 5: Linear accuracy (%), -NN accuracy (%) and effective rank of against the batch size, CIFAR-10, 1000 epochs, seed 5. SimCLR: AdamW, gradient clipping, encoder learning rate scaled by . ScoreCLR: , gradient clipping, same encoder scaling, unscaled score-network learning rate, at (main recipe), at and at and 4096.
| ScoreCLR | SimCLR | |||||
| Linear | -NN | RankMe | Linear | -NN | RankMe | |
| 128 | 88.80 | 88.51 | 229 | 89.45 | 88.56 | 96 |
| 512 | 88.51 | 88.39 | 231 | 89.32 | 89.07 | 108 |
| 2048 | 88.29 | 87.76 | 232 | 88.63 | 88.92 | 107 |
| 4096 | 87.50 | 86.88 | 230 | 88.88 | 88.78 | 107 |
Table 5.
Cost model
With codes of dimension , InfoNCE’s similarity matrix costs FLOPs and about bytes. ScoreCLR queries its score network ( parameters) separately for each code with score-matching updates and repulsion queries, about MFLOPs per image ( MFLOPs with ). The CIFAR-10 recipe (, ) costs MFLOPs per image; Table 7 and Table 10 use the heavier ImageNet-100 setting; Figure 2 and Table 11 report three settings. In fp32, one score-matching step saves 30.2 KB per image for backward; the steps are sequential and the repulsion queries run without gradient, so this does not depend on and . The loss saves 9.2 KB more, which gives 39 KB per image. Architecture constants are counted on the implemented modules. For the full-iteration memory ratio, activations saved for backward are counted in 16-bit mixed precision; under data parallelism over devices each device holds a block of the similarity matrix, so the ratio is independent of . At ScoreCLR’s objective needs 10× fewer FLOPs and 53× less memory than InfoNCE, and the per-device saving is 27% for ResNet-18 at 32 px and 8% at 224 px; SimCLR’s objective alone exceeds one 80 GB GPU at .
| Objective only | Full iteration, memory ratio | |||||
| TFLOPs SimCLR | TFLOPs ScoreCLR | Memory (GB) SimCLR | Memory (GB) ScoreCLR | R18 32 px | R18 224 px | |
| 4096 | 0.10 | 0.08 | 1.1 | 0.16 | 0.96 | 0.99 |
| 8192 | 0.41 | 0.17 | 4.3 | 0.32 | 0.92 | 0.98 |
| 16384 | 1.65 | 0.33 | 17.2 | 0.65 | 0.85 | 0.96 |
| 32768 | 6.60 | 0.66 | 68.7 | 1.29 | 0.73 | 0.92 |
| 65536 | 26.4 | 1.32 | 275 | 2.58 | 0.57 | 0.85 |
Table 10. Analytical cost vs. global batch size (ScoreCLR: , ). Left: the objective alone. Right: ScoreCLR / SimCLR per-device memory for a full data-parallel iteration.
Measured cost. Table 11 gives the measured time and extra peak memory of the objective alone, in the protocol of Figure 2, and Table 12 the time of a full training epoch, measured for both methods in the same job on the same GPU. In a full epoch the encoder dominates: ScoreCLR takes 1.08 times the time of SimCLR on CIFAR-10 at , 1.01 times on ImageNet-100, and the same time within 3% from to , where it uses 2% less memory at and 6% less at in 16-bit precision (the model of Table 10 gives 4% and 8%). At the measured objective uses 44 times less memory than InfoNCE (53 in the model), and the per-device saving of one step measured in fp32 is 11% (Table 7), against 27% in the model, which counts 16-bit activations.
Overhead at small batch. The time of the ScoreCLR objective is nearly flat from to (2.3–2.9 ms with , ; Figure 2, Table 11) while the work grows by a factor 64, so at small batch the time is a fixed cost, set by the number of small operations in the score-network step (three linear layers, the modulation, the gradient clipping and the optimizer step) rather than by the amount of computation. We did not optimize this. A fused kernel for the score network and its update, or capturing the step in a CUDA graph, should reduce this fixed cost; we have not tested it.
Estimator of the Poincaré constant
The Poincaré constant is , and we estimate it by maximizing this ratio over a class of test functions. Following Pillaud-Vivien et al. [43], we take in the span of the coordinates of and of random Fourier features , with , uniform on and the median distance between codes. The ratio is then a generalized Rayleigh quotient between the covariance of the features and the Gram matrix of their gradients, and it is maximized by a generalized eigenvector. We fit it on half of the validation codes, select the bandwidth factor , a ridge term and the eigenvector on a quarter, and report the ratio on the last quarter, because the ratio measured on the fitting codes overestimates the constant. The constant scales as the square of the codes, so we first center and whiten them (PCA, keeping the directions whose variance exceeds times the largest; 174 of 256 directions for SimCLR, whose output is collapsed). The estimate then reflects the shape of the distribution rather than its scale or anisotropy, and it equals 1 for a Gaussian. Since the estimate uses a fixed class of test functions, a smaller value is an indication, not a proof, that the codes are closer to a Gaussian.
References
- [1]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15619–15629, 2023.arxiv.org/abs/2301.08243
- [2]Dominique Bakry, Ivan Gentil, and Michel Ledoux. Analysis and Geometry of Markov Diffusion Operators. Springer, 2014.DOI
- [3]Randall Balestriero and Yann LeCun. Lejepa: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025. doi: 10.48550/arXiv.2511.08544. URL https://arxiv.org/abs/2511.08544.
- [4]Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, 2022.arxiv.org/abs/2105.04906
- [5]Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, 39(3):930–945, 1993.DOI
- [6]Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R. Devon Hjelm. MINE: Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 531–540. PMLR, 2018.arxiv.org/abs/1801.04062
- [7]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650–9660, 2021.arxiv.org/abs/2104.14294
- [8]Jiseok Chae, Kyuwon Kim, and Donghwan Kim. Open problem: Is there a first-order method that only converges to local minimax optima? In Proceedings of the Thirty-Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pp. 5957–5964. PMLR, 2023.
- [9]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, pp. 1597–1607. PMLR, 2020.arxiv.org/abs/2002.05709
- [12]Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.arxiv.org/abs/1311.3618
- [13]Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
- [14]Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley-Interscience, 2 edition, 2006.
- [15]Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23(56):1–6, 2022.
- [16]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE, 2009.
- [17]Mingyang Deng, He Li, Tianhong Li, Yilun Du, and Kaiming He. Generative modeling via drifting, 2026. URL https://arxiv.org/abs/2602.04770. arXiv preprint.
- [18]Persi Diaconis and David Freedman. Asymptotics of graphical projection pursuit. The Annals of Statistics, 12(3):793–815, 1984.DOI
- [19]Debidatta Dwibedi, Yusuf Aytar, Jonathan Thompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9588–9597, 2021.arxiv.org/abs/2104.14548
- [20]Giulio Franzese, Mustapha Bounoua, and Pietro Michiardi. Minde: Mutual information neural diffusion estimation. arXiv preprint arXiv:2310.09031, 2024. doi: 10.48550/arXiv.2310.09031.DOI
- [21]Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. In International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2210.02885
- [22]Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
- [23]Arthur Gretton, Olivier Bousquet, Alexander J. Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International Conference on Algorithmic Learning Theory, pp. 63–77. Springer, 2005.
- [24]Arthur Gretton, Li Kevin Wenliang, Alexandre Galashov, James Thornton, Valentin De Bortoli, and Arnaud Doucet. On the wasserstein gradient flow interpretation of drifting models. arXiv preprint arXiv:2605.05118, 2026.arxiv.org/abs/2605.05118
- [25]Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, pp. 21271–21284, 2020.arxiv.org/abs/2006.07733
- [26]Danijar Hafner, Juris Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2024. doi: 10.48550/arXiv.2301.04104.DOI
- [27]Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729–9738, 2020. doi: 10.1109/CVPR42600.2020.00975.DOI
- [28]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.arxiv.org/abs/2111.06377
- [29]R. Devon Hjelm, Alex Federov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.arxiv.org/abs/1808.06670
- [30]Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022.arxiv.org/abs/2110.09348
- [31]Adarsh Kappiyath, Silpa Vadakkeveetil Sreelatha, Xiatian Zhu, Yann LeCun, Lu Yin, and Amir Bar. Self-supervised collapse prevention with score-based entropy maximization. arXiv preprint arXiv:2611.11379, 2026.arxiv.org/abs/2611.11379
- [32]Sergei Kholkin, Ivan Butakov, Evgeny Burnaev, Nikita Gushchin, and Alexander Korotin. Info-Bridge: Mutual information estimation via bridge matching. arXiv preprint arXiv:2502.01383, 2025. doi: 10.48550/arXiv.2502.01383.DOI
- [33]Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.arxiv.org/abs/1312.6114
- [34]Frederic Koehler, Alexander Heckett, and Andrej Risteski. Statistical efficiency of score matching: The view from isoperimetry. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.00726
- [35]Simon Kornblith, Jonathan Shlens, and Quoc V. Le. Do better ImageNet models transfer better? In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.arxiv.org/abs/1805.08974
- [36]Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
- [37]Yann LeCun. A path towards autonomous machine intelligence, version 0.9.2, 2022-06-27. Open Review, 62(1):1–62, 2022.
- [38]Yazhe Li, Roman Pogodin, Danica J. Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. In Advances in Neural Information Processing Systems, volume 34, pp. 15543–15556, 2021.arxiv.org/abs/2106.08320
- [40]Elizabeth Meckes. Quantitative asymptotics of graphical projection pursuit. Electronic Communications in Probability, 14:176–185, 2009.arxiv.org/abs/0811.2769
- [41]Kimia Nadjahi, Alain Durmus, Lénaïc Chizat, Soheil Koulouri, Shahin Shahrampour, and Umut Simsekli. Statistical and topological properties of sliced probability divergences. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2003.05783
- [42]Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. arXiv preprint arXiv:1903.11780, 2019. doi: 10.48550/arXiv.1903.11780. URL https://arxiv.org/abs/1903.11780.
- [43]Loucas Pillaud-Vivien, Francis Bach, Tony Lelièvre, Alessandro Rudi, and Gabriel Stoltz. Statistical estimation of the poincaré constant and application to sampling multimodal distributions. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), volume 108 of PMLR, pp. 2753–2763, 2020.
- [44]Jean Ponce, Basile Tervier, Martial Hebert, and Michael Arbel. Dual perspectives on non-contrastive self-supervised learning. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/349a45f211fb1b3850da1ccd829e869e-Abstract-Conference.html.arxiv.org/abs/2507.01028
- [45]Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A. Alemi, and George Tucker. On variational bounds of mutual information. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 5171–5180. PMLR, 2019.arxiv.org/abs/1905.06922
- [46]Phillip Pope, Chen Zhu, Ahmed Abd el kader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations (ICLR), 2021.arxiv.org/abs/2104.08894
- [47]Pierre H. Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, and Michal Valko. Byol works even without batch statistics. arXiv preprint arXiv:2010.10241, 2020.arxiv.org/abs/2010.10241
- [48]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. arXiv preprint arXiv:2112.10752, 2022.arxiv.org/abs/2112.10752
- [49]Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. arXiv preprint arXiv:2102.11107, 2021. doi: 10.48550/arXiv.2102.11107.DOI
- [50]David W. Scott and M. P. Wand. Feasibility of multivariate density estimates. Biometrika, 78(1): 197–205, 1991.DOI
- [51]Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. arXiv preprint arXiv:1910.06222, 2020. doi: 10.48550/arXiv.1910.06222. URL https://arxiv.org/abs/1910.06222.
- [52]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11895–11907, 2019.arxiv.org/abs/1907.05600
- [53]Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations (ICLR), 2020. URL https://arxiv.org/abs/1907.13625.
- [54]Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.arxiv.org/abs/1807.03748
- [55]Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661–1674, 2011.DOI
- [56]Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11:3371–3408, 2010. URL https://jmlr.csail.mit.edu/papers/volume11/vincent10a/vincent10a.pdf. Editor: Léon Bottou.DOI
- [57]Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 9929–9939. PMLR, 2020. URL https://arxiv.org/abs/2005.10242.
- [58]Chun-Hsiao Yeh, Cheng-Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In European Conference on Computer Vision (ECCV), 2022.arxiv.org/abs/2110.06848
- [59]Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T. Freeman. Improved distribution matching distillation for fast image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2024a. doi: 10.48550/arXiv.2405.14867.arxiv.org/abs/2405.14867

