WorksComputer Science
Proving and Improving the Platonic Representation Hypothesis with JEPAs
Abstract
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a Platonic Representation Theorem: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene–good views–and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with bad views, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64% against at most 56% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Introduction
Whether different observers of one world end up with the same picture of it is a question with roots dating back millennia, from the shadows of Plato’s cave to perception as unconscious inference [48, 63] and the reduction of redundancy in sensory messages [7]. Deep networks revived that question. Word embeddings trained separately on different languages turned out to be related by a linear map [39], even by a rotation that can be found without a single translated pair, enough to translate words and then sentences with no parallel data [11, 29]. Across modalities, Huh et al. [22] observed that the representations of vision and language models become increasingly similar as models scale, and hypothesized that they converge to a shared statistical model of reality—coined the Platonic Representation Hypothesis (PRH).
Alas, the PRH remains an observation. Its supporting argument is an idealized contrastive learner whose kernel converges to the pointwise mutual information of co-occurring world events [22], the closest proof we know of covers embedded deep linear networks trained by SGD [75], and a recent theory explains why translations share representations inside multilingual language models [14]. For the self-supervised objectives used in practice, and in particular for Joint-Embedding Predictive Architectures (JEPAs) [4, 30], there is no proof, no prediction of when the hypothesis holds, and hence no recipe to improve alignment. As a result, scaling models and data remains the main lever used so far.
Our study proposes to turn that observation into a theorem by asking the following question: when do models, trained separately on different modalities, provably learn the same representation? We answer it for JEPAs with a short argument. A JEPA with whitened second-order anti-collapse learns the spectral embedding of its positive pairs, the pairs of inputs it declares to be views of each other [3, 19]. If the modalities render the same world and their positive pairs relate the same world states, their training problems are one problem written in different coordinate systems, and their solutions coincide up to a rotation—which we coin the Platonic Representation Theorem (Theorem 1). Nothing in this argument is specific to images and text: it holds for any modalities that observe one world, e.g., images, video, text, audio or actions, and for any number of them at once. We therefore obtain the following key insight. The PRH is a statement about positive views: a JEPA whose positives are two observations of the same scene learns a representation that belongs to the world, while a JEPA whose positives are corruptions of one observation learns one that belongs to its modality (Figure 1). We coin the former good-view and the latter bad-view JEPAs, and summarize our contributions below.
Figure 1. Good views make the PRH a theorem, and alignment 42 times more sample-efficient. Left: the PRH picture [22], one world observed as images and as texts. When a JEPA’s positive pairs are two observations of one scene (good views: two photographs, two captions), the encoders of every modality learn the same embedding up to one rotation (Theorem 1); bad views, two corruptions of one observation, and next-token prediction, a prefix and its continuation, align only partly (Propositions 3 and 4). Right: alignment with DINOv2 on the PRH paper’s benchmark (WIT-1024, mutual 10-NN) of text models trained on texts of web images, against the samples they saw (means of three seeds, one for the 109M bad-view JEPA; lines: log-linear trends). With 21.6M samples a good-view JEPA matches next-token prediction on the same corpus at 900M samples (.109 and .111; 14 times fewer text tokens), and with 900M it reaches .184, above Qwen2.5-72B and Llama-3-70B, trained on 15–18T tokens (dashed). This demonstrates that with good views, alignment needs far less scale.
Contribution 1: We turn the PRH into a theorem. In an idealized model, any number of JEPAs trained separately on renderings of one world learn the same embedding up to one rotation each when their positive pairs relate the same world states and their minimizers are unique up to rotation, as we prove for whitened JEPAs under a spectral gap (Theorem 1 and Proposition 1); otherwise, their misalignment is bounded by the mismatch of their positive pairs (Theorem 2). Positives that corrupt a single observation make an encoder spend dimensions on its modality’s nuisance whenever they keep more of it than of the world’s -th direction, which no rotation can align, while two observations of one scene spend none (Theorem 3 and Proposition 3), and views and data together set what a JEPA learns (Theorem 4).
Contribution 2: We show that next-token prediction is a bad-view JEPA. Its positives are a prefix and its continuation within one document, and its anti-collapse term is the softmax log-partition (Proposition 4). This suggests why language models align with vision only partly, and mostly through scale.
Contribution 3: We validate good views and uncover a scaling law of alignment. With identical texts and encoders, good views align best in all but a few of the comparisons we ran (Section C): from 145k Flickr captions to the texts of 940M web images, with image encoders trained from scratch or frozen, and on video (§ 3). Alignment follows a scaling law in the fraction of good views and the samples seen, which predicts 85 new runs, before they are trained, as accurately as rerunning them with another seed (§ 4.4).
Contribution 4: We improve alignment beyond 70B language models, which scale poorly in comparison. Good views help in three ways (Section 4). They scale with data: trained on the texts of 50M web images, a 33M-parameter text JEPA reaches .125 on the PRH paper’s benchmark, against .034 on Flickr captions, level with Qwen2.5-3B and -7B. They need fewer samples: on the same corpus, a good-view JEPA reaches with 21.6M training samples the alignment that next-token prediction reaches with 900M, 42 times more (Figure 1). They beat larger models: trained on the texts of 940M web images, a 109M-parameter text JEPA reaches .184, above every language model we evaluated, up to Qwen2.5-72B (.154), and above CLIP’s text encoder (.124), on two benchmarks and two metrics; it also leads the language models with MAE, I-JEPA and CLIP’s image encoders, e.g., .230 against .210 with CLIP ViT-H. For alignment, next-token prediction is thus a remarkably inefficient use of scale: our 109M encoder, which read 116B text tokens, beats models 660 times larger trained on over 100 times more.
Contribution 5: We caption images without training a network on image–caption pairs. One linear map with orthonormal rows, the only part fitted on pairs, joins frozen DINOv2 and DINOv3 image encoders to a language model fine-tuned on text alone, reaching CIDEr .838 on COCO (§ 5.1). With no pair at all, the rotation can be found from the geometry of unpaired images and captions: on the benchmark of Schnaus et al. [54], which aligns a vision and a language encoder from unpaired images and captions and scores the result by zero-shot classification, our 109M text JEPA beats every zero-shot accuracy they report, over 21 pairs of vision and language models and five settings, and it beats their strongest language model, Qwen3-8B, rerun under identical conditions, by 17 points of zero-shot ImageNet-100 accuracy (§ 5.2). This demonstrates that the PRH–with JEPAs–is no longer a hypothesis but a theorem and a tool.
Shared Positives Provably Make Representations Platonic
We start by formalizing a world, the modalities that observe it and their positive pairs, then prove that JEPAs with shared positives agree up to one rotation (Theorem 1), bound their misalignment otherwise (Theorem 2), and show which positives make an encoder spend its embedding on its modality’s nuisance (Theorems 3 and 4 and Proposition 3), of which next-token prediction is an instance (Proposition 4).
The Platonic Representation Theorem
World, modalities, and positive pairs. Let a world be a finite set of states with probabilities , and let modalities observe it, modality through an injective rendering , e.g., the pixels of a photograph or the tokens of a caption. A JEPA of modality is trained on positive pairs whose world states are drawn from a symmetric positive-pair distribution with both marginals equal to . Its encoder minimizes
a prediction term plus an anti-collapse term that depends on only through the distribution of its embeddings (possibly a hard constraint, ). As in our experiments, the predictor is the identity: compares the two embeddings directly, whereas JEPAs used as world models predict one embedding from the other through a learned predictor, often conditioned on an action, which our analysis does not cover. Both and are invariant to rotations : , and rotating every embedding leaves unchanged. Spectral contrastive learning and whitened MSE fit this form [16, 19], VICReg and Barlow Twins in a whitened limit [3, 6, 72], and LeJEPA’s SIGReg, which pushes the embeddings toward an isotropic Gaussian, in expectation over its directions [4] (Section A). The key observation is that an injective rendering only changes coordinates: , the same objective written on world states, for (Lemma 1). We are now able to state when modalities learn the same representation.
Theorem 1: Platonic Representation Theorem
Let modalities share one positive-pair distribution, , and let the minimizers of be unique up to rotation. Then there is one embedding of the world such that every minimizer of , for every modality , satisfies
so any two modalities agree up to the single rotation . (Proof in Section A.)
In words, encoders that never see each other’s modality, and whose renderings need not be related beyond describing the same world, learn the same embedding up to one rotation each: in this idealized JEPA setting–one finite world rendered injectively, shared positives, population optima, and minimizers unique up to rotation–the PRH becomes a theorem, for images and texts as for any other set of modalities. The proof is direct: every minimizes . The key assumption is uniqueness up to rotation, which holds for JEPAs with second-order anti-collapse under a spectral gap, as formalized below. Write for the matrix of (), , and for the normalized adjacency of the graph that positives draw on the world, with eigenvalues and orthonormal eigenvectors .
Proposition 1: Spectral JEPAs Are Unique Up to Rotation
Let , let whiten the embeddings (, ), and let . Then , and if (with ) the minimizers are exactly , , with the spectral embedding . (Proof in Section A.)
In words, a JEPA with second-order anti-collapse learns the top eigenvectors of the graph its positives draw, the population form of Laplacian eigenmaps [3, 8, 19], unique up to rotation when the spectrum has a gap. LeJEPA’s SIGReg is rotation invariant and has the same second moments; Proposition 1 does not cover uniqueness under its full-distribution constraint, which we leave as an assumption.
When Positives Differ
Alas, in practice the modalities’ positives are not the same pairs of world states.
Theorem 2: Kernel Mismatch Bounds Misalignment
Let share the marginal , let be their spectral embeddings (any orthonormal top- eigen-
:::`
vectors if has ties), and let be the eigengap of . Then
(Proof in Section A, empirical validation in Figure 2.)
Figure 2. Kernel mismatch bounds misalignment (Theorem 2). Each point is one randomly generated finite world (8–39 states, random marginal, embedding dimension from 1 to 5) with a pair of kernels on it: the world’s kernel, and a second kernel of the same marginal that mixes in an unrelated kernel with weight to 1, as if a second modality had partly different positives. For each pair we compute the two spectral embeddings (Proposition 1) and plot their distance after the best rotation (vertical) against the bound of Theorem 2 (horizontal); of the 300 worlds, half are clustered, with small spectral gaps. We clearly observe that the error lies below the bound in every world (dashed: equality), at 18% of it at most.
In words, for the objective of Proposition 1 the left side is the root-mean-square distance between the embeddings of paired observations after the best single rotation, and only two quantities matter: how different the graphs drawn by the two modalities’ positives are, and how separated the top of their spectrum is. Crucially, the same argument applied to a block of eigenvectors, with gaps on both sides of the block [71], suggests where alignment lives: directions separated from the rest of the spectrum by large gaps are protected, while directions near the end of the spectrum need not align at all. Figure 2 validates the bound on randomly generated finite worlds, each point being one world with a pair of kernels that differ by a controlled amount: in all 300 worlds the error lies below the bound, at 18% of it at most.
What pairs are for. Theorem 1 says that the embeddings agree up to a rotation, but not which one. As formalized below, the first two moments of unpaired data cannot find it, and pairs suffice.
Proposition 2: Pairs Identify the Rotation
In the setting of Theorem 1 and Proposition 1, (i) every rotation of a whitened embedding is whitened, so no statistic of the first two moments of unpaired samples from two modalities identifies the rotation between them; (ii) if the embeddings of world states span , the rotation with is the unique minimizer over of (orthogonal Procrustes). (Proof in Section A.)
In words, pairs enter only through one orthogonal matrix, which pairs in general position determine; when the positives differ, more pairs average out the residual misalignment of Theorem 2. Higher-order structure of a finite world can reveal the rotation without any pair, as recent work shows by aligning encoders from their shared geometry alone [54]; with good views, whose geometry they share ties to the world’s, this works best (§ 5.2). When the two embedding dimensions differ (768 for DINOv2, 384 for our text JEPA) we fit a linear map with orthonormal rows, a case the theory does not cover; such a map between dimensions has free parameters, 0.22M for DINOv2 ViT-B and 1.9M for the 5,120-dimensional features of our best captioner, all fitted on pairs.
Good-View and Bad-View JEPAs
Theorem 2 singles out what a JEPA must get right for the PRH: its positive pairs. Call a JEPA good-view when its positives are two observations of world states that the world itself relates, e.g., two photographs or two descriptions of one scene, and bad-view when they are corruptions of a single observation, e.g., two crops of one photograph or two maskings of one sentence (Figure 1). Injective renderings cannot express this distinction: there, a state has a single observation. We therefore let each observation carry a nuisance of its own modality—which words a caption happens to use, from which viewpoint a photograph was taken—drawn independently of the state. It is a nuisance only for alignment: a world model that acts through a camera must track its viewpoint, but no caption determines it, so no rotation can match it across modalities. We render injectively, so that a state has many observations. A good-view JEPA’s positives are with and independent nuisances; a bad-view JEPA’s are two corruptions of one observation, which share its nuisance. For an encoder centered and whitened on its observations, let be the covariance of the part of its embedding that the world state explains, and its scene-explained fraction. The cost of encoding nuisance is formalized below.
Theorem 3: Nuisance Costs Alignment. Let and be centered, whitened encoders of two modalities into . Over paired observations of one world state with independent nuisances,
and if and only if is a function of the world state alone. (Proof in Section A.)
In words, every direction an encoder spends on its own modality’s nuisance is lost for alignment, whatever whitened embedding the other modality learns. Which JEPAs spend directions on nuisance is formalized below.
Proposition 3: Good Views Ignore Nuisance, Bad Views Capture It. Take the objective of Proposition 1. (i) If , a good-view JEPA learns a function of the world state alone (): good-view JEPAs of any number of modalities with the same world kernel agree up to one rotation each on every observation, although every state has many. (ii) Let a bad-view JEPA corrupt state and nuisance independently, let be the correlation that two corruptions of one observation keep along the -th direction of the state, and the largest they keep along a direction of the nuisance. If its kernel has a gap after its -th eigenvalue, it spends of its directions on nuisance, so that its alignment error with any centered, whitened encoder of another modality is at least , and if and only if . (Formal statement and proof in Section A.)
In words, a corruption that preserves the nuisance more than the world’s -th direction—spans of one caption keep its word choice, crops of one photograph its viewpoint and lighting—makes the JEPA spend embedding directions on that nuisance, which no rotation can align with another modality; two observations of one scene share only the scene, so a good-view JEPA spends none. The model still assumes that every observation determines its state, while a caption sees only part of a photograph; Theorem 2 covers that departure only through the kernel mismatch. By the same definition our image encoders, and DINOv2, are bad-view JEPAs: their positives are crops of one photograph. Our experiments mostly vary the text side against a frozen DINOv2, the vision model with which Huh et al. [22] measured the PRH, so that our text encoders compare directly with the language models measured there; DINOv3 [56], released after that benchmark, serves in our caption clips is far weaker than DINOv2, and scaling good image views is left to future work.
Views and data together. Proposition 3 lets a corruption act on state and nuisance separately. In general, whether the positives of two different observations coincide depends on the data: masking two captions of one scene can leave the same words. In the augmentation-graph analyses of contrastive learning, such overlaps are what connect samples of one class [19, 66], with the function class deciding what overlaps [9, 26, 52]. We now state an analogue with world states in place of classes, and its cross-modal consequence (Figure 3). Given any law of positive pairs over modality ’s finitely many observations—good views, corruptions, or a mix—call its induced world kernel the probability that a positive pair joins an observation of to one of .

Figure 3. What counts as a positive, and which observations it connects. Eight captions of two scenes (dots), joined as each objective’s positives join them. (a) Good views join every description of a scene and never another, so the world kernel they induce is the world’s own (Theorem 4). (b) Weak corruptions mostly pair a caption with itself, keeping its wording, and join other captions only where their corruptions overlap, which takes dense data. (c) Strong corruptions also coincide across scenes, so the induced kernel mixes scenes whatever the amount of data. (d) Next-token prediction pairs a prefix with its own continuation, and implicitly contexts followed by the same token (Proposition 4): narrow links within scenes, plus spill through frequent tokens. That is, good views fix the world kernel by design, while bad views and next-token prediction get it from the data.
Theorem 4: Views and Data Set the World Kernel. Take the objective of Proposition 1 on the observations of each modality , and let the -th eigenvalue of exceed both the next one and the next shared noise—nuisance its positives keep. (i) Every spectral solution is within of a rotation of the spectral embedding of , where grows with how much positive pairs mix world state with nuisance and vanishes when they do not. (ii) Over paired observations of two modalities,
with the misalignment of the spectral embeddings of and , which Theorem 2 bounds. (Formal statement and proof in Section A.)
In words, a spectral JEPA learns, up to , the spectral solution of the world kernel that its views and its data induce, and two such JEPAs of different modalities agree, up to , exactly as well as those kernels’ top eigenvectors do. Good views induce the world’s own kernel with , which gives Theorem 1 with many observations per state. Crucially, a bad-view JEPA does not choose its world kernel: records which states its positives connect, which depends on how the data cover each state and its neighbors. Bad-view JEPAs of two modalities therefore agree only insofar as their data make the positives connect the same states, while good views fix the kernel by design.
How much data, and which. Read on finitely many samples, with positives that join two samples when their corruptions overlap—the empirical augmentation graph—weak views join few samples, so each level set of the world splits into small pieces that a spectral JEPA separates, and more samples chain the overlaps through each level set. Good views join all observations of related states in one step and need no such density. The kind of data, in contrast, sets the kernel itself: chains of overlapping positives stay within a scene for literal descriptions, but cross into states that share only a nuisance for alt-texts sharing a brand name. Figure 4 maps this trade-off on a synthetic world: the reach that weak bad views need falls as the samples per state grow, along , the scale of the largest gap between a state’s samples; reach large enough to cross scenes caps alignment at any sample size; and good views align exactly from one sample per state. Mixed data, whose classes overlap, cap every bad-view JEPA lower still (Figure 12). Our runs show the same trade-off: a good-view JEPA reaches the alignment of next-token prediction on the same corpus with 42 times fewer samples (Figure 1), and alignment follows a scaling law in the fraction of good views and the samples seen (§ 4.4). Real data agree with this reading as well: with the same bad views, a text JEPA trained on LLaVA recaptions, literal descriptions of what is visible, aligns with DINOv2 better than one trained on alt-texts (Table 7); changing the data changes how positives mix state and nuisance as well as the kernel, so this comparison cannot single out the kernel.
Figure 4. Positives that join less of a scene need more data (Theorem 4). A synthetic world of five classes of six states (literal data, as three samplings; a state has width ). White: the contour, and (dashed, one fitted constant), the scale of the largest gap between a state’s samples. Reach large enough to cross classes caps alignment (.95), and good views (top strip) align exactly at any . We observe that weaker positives need more data for the same alignment.
The views and the data both set the alignment. BERT-S text JEPAs, one recipe per scale, three seeds (mean and range), scored on WIT-1024 with DINOv2 ViT-B and on Flickr through one rotation fitted on 30k pairs; scene-explained: on Flickr captions (, seed 0, Table 6). The paraphrase is written from the alt-text alone by a text-only language model, so that row involves no image captioner. We observe that with the same bad views, LLaVA recaptions align far better than alt-texts, and with the same texts, good views double alignment (Theorem 4).
| Texts of | texts per image | views | WIT (DINOv2-B) | Flickr R@1 | scene-explained |
| 2.24M images | alt-text, two LLaVA recaptions | good | .107 (.095–.115) | .339 (.336–.344) | .793 |
| 2.24M images | alt-text, text-only paraphrase | good | .060 (.058–.061) | .233 (.230–.235) | .654 |
| 2.24M images | two LLaVA recaptions | good | .080 (.076–.082) | .348 (.346–.351) | .778 |
| 2.24M images | alt-text | bad | .055 (.054–.056) | .180 (.179–.180) | .606 |
| 50M images | alt-text, two LLaVA recaptions | good | .125 (.118–.129) | .376 (.370–.381) | .802 |
| 50M images | two LLaVA recaptions | good | .096 (.093–.100) | .386 (.373–.396) | .789 |
| 50M images | alt-text, two LLaVA recaptions | bad | .063 (.062–.066) | .290 (.283–.301) | .673 |
| 50M images | two LLaVA recaptions | bad | .092 (.090–.093) | .325 (.323–.329) | .734 |
| 50M images | alt-text | bad | .056 (.056–.057) | .199 (.193–.203) | .618 |
Table 7.
Figure 12. Weak views need many data, the kind of data sets the ceiling, good views need neither (Theorem 4). A synthetic world of five classes of six states; the image side is a good-view JEPA whose world kernel links states of one class. The text side observes each state through samples on a line, one block per class, with gaps between blocks (left, literal data) or without (right, mixed data); its bad views join two samples with weight (weak , medium .08, strong 1; a state has width 1/6). Alignment is the mean canonical correlation of the two spectral solutions over paired observations (; 1: equal up to a rotation); lines are means over five samplings, bands their range.
Next-Token Prediction is a Bad-View JEPA
Language models are the usual text side of the PRH [22]. We now show that their training objective is a JEPA in disguise, as formalized below.
Proposition 4: Next-Token Prediction Has the Form of a JEPA Let a context be embedded by and tokens by a table . The next-token loss with a linear
softmax head is
(i) This has the structure of a JEPA, generalizing (1) to two encoders and an anti-collapse term that depends
::: on the target encoder: context encoder , target encoder , inner-product prediction, positives drawn from within one sequence. (i) does not depend on the target token , and as , so the temperature-scaled loss approaches the margin , which vanishes iff the target attains the maximal logit. (iii) With full-support conditionals and large enough, minimizers satisfy , a set invariant under (, ), so is identified at best up to an invertible linear map, not a rotation. (Proof in Section A.)
In words, next-token prediction is a JEPA whose positives are a prefix and its continuation, and whose anti-collapse term is the log-partition. Part (iii) is the matrix-factorization view of word embeddings [31], and (ii) plays the role of the uniformity term of contrastive learning [65], much as masked autoencoders implicitly align the two parts that a mask creates [73]. Crucially, (i) tells us what next-token prediction’s positives are: a prefix and its continuation inside one document. Its kernel therefore relates texts through shared wording, like masked-span JEPAs, and not through shared world states. Heuristically, Proposition 3 then applies to it: a prefix and its continuation share their document’s nuisance (its author’s wording and style), so next-token prediction should spend directions on that nuisance and align with vision worse than a good-view JEPA. Proposition 3 assumes one encoder and a kernel on one space, so this is a prediction to test rather than a corollary—which we do next.
Read through Theorem 4, again heuristically since next-token prediction is not a spectral JEPA and is identified only up to , it has a second kind of positive. At the optimum (iii), , and contexts followed by the same token gather around that token’s output vector, the neural collapse that language models approach with scale [46, 67]. Contexts thus become implicit positives of each other when they share a continuation: very weak views, since one token rarely spans a scene, so joining the descriptions of one scene takes long chains of shared continuations, i.e., very dense data, while frequent continuations (function words, a document’s style) also join contexts across scenes. Good-view JEPA views instead join all descriptions of a scene at once and never leave it. This is our reading of why language models align with vision mostly through scale, and why good views need far less of it.
Good Views Improve Alignment
We now empirically validate the predictions of § 2 and Proposition 4: good-view JEPAs should align across modalities better than bad-view JEPAs and next-token prediction trained on identical data, whatever the data, the encoders and the modality.
Setup. We strive for simplicity and thus adopt a unified recipe: image and text encoders are trained separately, each with the same LeJEPA objective [4] (invariance to the mean of two global views, plus SIGReg with ), projector, optimizer and schedule, using stable-pretraining. Data is Flickr30k [70] with the Karpathy split [25]: 29k training images with five captions each. The image encoder is a ViT-S/16 [15] trained on 2 global () and 6 local () crops. The text encoder is a 12-layer BERT of the same width (384, 33M parameters; BERT-S below, with BERT-T, -B and -L of 11M, 109M and 335M parameters for scaling) [13], trained from scratch in three variants that differ only in their positives: masked spans (bad-view JEPA): 8 views of one caption, each trimmed to a random contiguous span, with 15% of tokens masked; caption views (good-view JEPA): each of the 8 views is drawn from a different caption of the same image, then trimmed and masked the same way; next-token prediction: the same BERT made causal, with a tied softmax head, on one caption per image per epoch; as a control, we also train it on pairs of captions of the same image. No encoder ever sees the other modality. The runs of Table 1 use 200 epochs, batch 256 and AdamW (lr , weight decay 0.05); longer and larger runs use a lower learning rate with gradient clipping. We train three seeds of every comparison and report their mean unless stated otherwise. Details are in Section B.
| Text encoder | mutual 10-NN | R@1 | R@5 |
| Good-view JEPA | .062±.003 | .044±.002 | .133±.004 |
| Bad-view JEPA | .047±.001 | .027±.002 | .095±.003 |
| Next-token | .042±.001 | .021±.002 | .077±.005 |
| Next-token, pairs | .043±.001 | .023±.003 | .083±.008 |
| Random init | .010 | .001 | .011 |
| Chance | .010 | .001 | .005 |
Table 1. Good views align across modalities. Image JEPA vs. each text encoder (epoch 199) on 1,000 Flickr test pairs (backbone features; mean ± half-range over three seeds): good-view JEPA (caption views), bad-view JEPA (masked spans), next-token prediction, and next-token prediction on pairs of captions of one image; R@1 and R@5 use a single rotation fitted on validation pairs. We clearly observe that caption views beat masked spans and next-token prediction on every metric, even when next-token prediction is trained on pairs of captions of the same image.
Metrics. We embed the 1,014 validation and 1,000 test image–caption pairs (first caption). Mutual 10-NN [22] measures the overlap of the 10-nearest-neighbor sets in the two spaces without fitting anything; chance is . To test equality up to rotation directly, we fit one orthogonal map on the validation pairs [55] and report image→caption retrieval R@1 and R@5 among the 1,000 test captions (chance 0.1%).
Good Views Align Across Modalities
Figure 5 and Table 1 report the alignment of a separately trained image JEPA with each text encoder, and we obtain a few key observations. First, cross-modal alignment rises from chance throughout training for every text encoder, although none ever sees the other modality. Second, with caption views the true caption is retrieved first for 4.4% of test images through one rotation, 44 times chance, while masked spans and next-token prediction, trained on the same captions with the same encoder, nearly coincide and stay well below—a ranking that holds in every seed, with gaps several times the seed spread. Third, the gain comes from the objective and not from the information that two captions describe the same image: next-token prediction trained on two captions of the same image, [CLS] a [SEP] b [SEP], barely moves (mutual 10-NN .042 to .043, R@1 .021 to .023), i.e., the same positives help next-token prediction far less than they help as JEPA views, as per Proposition 4. Fourth, alignment concentrates in the leading spectral directions, as the block version of Theorem 2 suggests: after the best rotation, the correlation between image and text embeddings is 0.70 on the first principal direction of the text space and decays to about 0.1 by the 60th, while it is near zero everywhere at initialization (Figure 6). Finally, the nuisance mechanism of Theorem 3 can be measured directly, since every Flickr image (a world state) has five captions (five nuisances): from all captions of the 2,014 validation and test images, we estimate for each text encoder, restricted to its top- principal directions and whitened, with the mean of five captions recorded for its within-image spread. For , the scene explains a fraction of .804 of the good-view JEPA’s embedding, against .655 for the bad-view JEPA and .608 for next-token prediction (each within .005 across seeds), so Theorem 3 bounds their best-rotation error against DINOv2 below by 12.6, 22.1 and 25.1 (from the unrounded fractions), which the measured errors (33.5, 39.4 and 40.3) respect and follow in order; gives the same ordering (Table 6). That is, *bad views and next-token pre-
Figure 5. Good views align, bad views and next-token prediction do not. Alignment between a separately trained image JEPA and three text encoders on Flickr, at matched epochs (mean of three seeds, band: range; dashed: chance). All start at chance. We clearly observe that the good-view JEPA (caption views) aligns markedly better than the bad-view JEPA (masked spans) and next-token prediction, which nearly coincide, as predicted by Theorem 2 and Proposition 4.
Figure 6. Alignment lives in the leading spectral directions. Left: after the best rotation, correlation between image and text embeddings along the text embedding’s principal directions (good-view JEPA, at initialization and at epoch 199). Right: variance of the text embedding along the same directions. This suggests that alignment concentrates where the spectrum is well separated, consistent with Theorem 2, although noise in low-variance directions would produce the same trend.
Bad views spend their embedding on nuisance. Scene-explained fraction of each text encoder’s top- whitened embedding (mean half-range over three seeds), the lower bound of Theorem 3 on the best-rotation error against DINOv2 ViT-B, and that error (means over seeds; bounds from the unrounded fractions; uncorrelated embeddings give , i.e., 64 and 256). We clearly observe that the bound holds for every encoder, and the scene-explained fraction ranks the encoders as their alignment does.
| Text encoder (Flickr) | ||||||
| scene-explained | bound | error | scene-explained | bound | error | |
| Good-view JEPA | .804±.003 | 12.6 | 33.5 | .654±.005 | 88.6 | 157.0 |
| Bad-view JEPA | .655±.005 | 22.1 | 39.4 | .507±.003 | 126.1 | 171.5 |
| Next-token | .608±.001 | 25.1 | 40.3 | .440±.002 | 143.4 | 176.6 |
| trained on Recap texts only (seed 0), measured on the same Flickr captions | ||||||
| Good-view JEPA, 50M images | .802 | 12.7 | 32.4 | .642 | 91.7 | 156.8 |
| Bad-view JEPA, 50M images | .673 | 20.9 | 37.6 | .506 | 126.4 | 168.7 |
| Next-token, 50M images | .618 | 24.4 | 40.1 | .483 | 132.3 | 171.0 |
| Good-view JEPA, 100M images | .806 | 12.4 | 32.0 | .661 | 86.9 | 153.7 |
| Good-view JEPA, BERT-B, 940M images | .830 | 10.9 | 30.0 | .722 | 71.1 | 145.3 |
Table 6.
diction spend more of their embedding on the wording of a particular caption, as Proposition 3 predicts, and align less.
Trimming Makes Good Views, the Rest Barely Matters
Recall that a good view should keep only what two observations share. To identify which knob of the good-view JEPA matters, Figure 7 changes one setting at a time and scores the text JEPA against frozen DINOv2 (R@1 through a rotation fitted on validation pairs, last epoch, no selection on test pairs). With three seeds for every setting except the SIGReg weights, the larger learning rate and the 1,000-epoch run, every run that does not collapse lands between .18 and .28, far above the bad-view JEPA (.13) and next-token prediction (.12), which use the same recipe and were not tuned. Trimming matters most: whole captions as views lose a quarter of the alignment (.185 against .246), as trimming is what makes two descriptions of a scene share only their common content. Masking, the number of views, batch size, width and schedule move the mean by at most .016, about the spread of the default’s own seeds (.230–.260). Stability is another matter. Without gradient clipping, 3 of the 36 runs with the default SIGReg weight collapse abruptly between epochs 20 and 60 of 200 (two of three seeds with 2 + 2 views, one with mask 30%), and so does every other SIGReg weight we tried. With clipping, 2 + 2 views trains for all three seeds (.262) and so do and 0.1, while a larger learning rate and still collapse; none of our longer runs, all trained with clipping and a lower learning rate, collapses (Table 8). We thus recommend views drawn from different observations of the same world state, each trimmed (cropped, for images) and lightly masked, trained with gradient clipping, as the starting point for the modalities we tested.
Figure 7. Trimming is the knob that matters. One change at a time to the good-view text JEPA on Flickr: R@1 with frozen DINOv2 through one rotation, last epoch, three seeds where they exist (faint dots; large dot: mean of the seeds that do not collapse; black: default). Top: the two changes to trimming, with the change of their mean from the default’s; whole captions as views lose a quarter of the alignment. Dashed: the bad-view JEPA and next-token prediction with the default recipe. That is, trimming matters most, and every variant of the good-view JEPA that does not collapse stays far above both baselines; gradient clipping prevents most collapses.
| Good-view text JEPA, one change from the default | last-epoch R@1, seeds 0 / 1 / 2 | mean | best* (epoch), seed 0 |
| Default: 2 + 6 views, mask 15%, lr , | .248 / .260 / .230 | .246 | .276 (99) |
| BERT-B (768 wide; lr , clipping 1) | .273 / .252 / .255 | .260 | .275 (189) |
| BERT-T (192 wide) | .244 / .229 / .228 | .234 | .254 (149) |
| no masking | .236 / .246 / .244 | .242 | .259 (109) |
| mask 30% | .243 / × / .264 | .254 | .266 (129) |
| no trimming (whole captions) | .193 / .182 / .180 | .185 | .242 (79) |
| milder trimming (70–100% / 40–80%) | .211 / .204 / .191 | .202 | .268 (59) |
| 2 + 2 views | .247 / × / × | .247 | .257 (129) |
| 2 + 2 views, clipping 1 | .260 / .258 / .267 | .262 | .272 (159) |
| 2 + 10 views | .254 / .247 / .243 | .248 | .262 (119) |
| 4 + 4 views | .242 / .249 / .248 | .246 | .258 (109) |
| lr | .265 / .246 / .253 | .255 | .270 (169) |
| lr (clipping 1) | collapses | – | – |
| batch 512 | .239 / .243 / .240 | .241 | .245 (129) |
| 100 epochs | .233 / .261 / .248 | .247 | .239 (59) |
| collapses | – | .196 (29) | |
| , clipping 1 | .238 | .238 | .242 (119) |
| collapses | – | .242 (89) | |
| , clipping 1 | collapses | – | .162 (19) |
| collapses | – | .191 (79) | |
| , clipping 1 | .202 | .202 | .215 (149) |
| 1,000 epochs, stopped at 359 (lr , clipping 1) | .242 (359) | .242 | .278 (99) |
Table 8. Cropping matters, the rest barely moves the mean, and clipping prevents most collapses. R@1 against frozen DINOv2 on Flickr test pairs with one rotation fitted on validation pairs (chance .001), last epoch, for three seeds where shown (one otherwise); a run collapses (×) when its last-epoch R@1 falls below .05. Default views: 2 global and 6 local spans of 50–100% and 15–50% of a caption. Mean: over the seeds that do not collapse (–: every seed collapses). *Best over checkpoints on the test pairs, for reference only; the other columns involve no selection. That is, every run that does not collapse stays far above the bad-view JEPA and next-token prediction, and without gradient clipping 3 of the 36 runs with the default SIGReg weight collapse. In stable-pretraining as we ran it, SIGReg is scaled by the number of embeddings it sees, so the batch and view rows also change its effective weight: ×2 for batch 512, × for 2 + 2 views and ×1.5 for 2 + 10 views.
Good Views Help on Both Sides
To test the image half of the prediction we need a world state observed several times in both modalities, which video provides. On MSR-VTT [68] (7k training clips, 20
captions each) we train the same image and text JEPAs with views from different frames (images) or captions (text) of a clip (good), or from a single frame or caption (bad), and evaluate on the 1,000 test clips (middle frame and one caption; Section B). Good views raise mutual 10-NN on each side, in every seed, the text side much more than the image side: .067 when both sides have good views, .059 with good text views only, .043 with good image views only, and .039 with neither (same ordering in each seed; chance .01; Figure 8 and Table 13). Conversely, frozen DINOv2, whose positives are crops, agrees more with the image JEPA trained on crops than with the one trained on frames (.144 vs. .132, in every seed). That is, alignment follows agreement of positives, as per Theorem 2.
Figure 8. Good views help on both sides. MSR-VTT, 1,000 test clips. Top: image JEPAs trained on different frames of a clip (good) or crops of one frame (bad), against text JEPAs trained on different captions (good) or spans of one caption (bad); dots are the three seeds, dashed is chance. Bottom: alignment of each with frozen DINOv2, whose own positives are crops. This demonstrates that good views on both sides align best in mutual 10-NN, and that DINOv2 prefers the image JEPA whose positives match its own.
Table 13. Good views help on both sides (numbers behind Figure 8). MSR-VTT, 1,000 test clips, final checkpoints, mean ± half-range over three seeds (chance: mutual 10-NN .01, R@1 .001). R@1 and R@5 use one rotation fitted on the 1,990 validation clips. Every seed ranks the four combinations the same way in mutual 10-NN; the image-side effect is small and reverses on R@5.
| Image JEPA | Text JEPA | mutual 10-NN | R@1 | R@5 |
| good views (frames) | good views (captions) | .067±.001 | .032±.006 | .099±.009 |
| bad views (crops) | good views (captions) | .059±.002 | .024±.002 | .105±.008 |
| good views (frames) | bad views (spans) | .043±.002 | .020±.005 | .063±.002 |
| bad views (crops) | bad views (spans) | .039±.003 | .017±.002 | .072±.009 |
| frozen DINOv2 ViT-B | good views (captions) | .129±.002 | .085±.006 | .258±.011 |
| frozen DINOv2 ViT-B | bad views (spans) | .084±.002 | .052±.002 | .164±.006 |
| frozen DINOv2 ViT-B | image JEPA, good views | .132±.002 | .219±.010 | .462±.011 |
| frozen DINOv2 ViT-B | image JEPA, bad views | .144±.001 | .285±.005 | .559±.008 |
Table 13.
Good-View JEPAs Beat a Pretrained Language Model and Approach Pair-Trained Encoders
Our absolute numbers so far are limited by the image encoder, a ViT-S trained from scratch on 29k images: training it five times longer raises its mutual 10-NN with the good-view text JEPA from .065 to .077 (seed 0), and with DINOv2 from .101 to .121. To isolate the text side, we pair each text encoder with a frozen DINOv2 ViT-B/14 [44] under the same protocol (Table 2). With final checkpoints and no selection on test data, the good-view JEPA retrieves the true caption first for 24.6% of test images, about twice the bad-view JEPA (.133), our next-token model (.118, or .137 with caption pairs) and the pretrained Qwen2.5-0.5B language model [49] (.135, last layer). Text encoders trained on vastly more data reach .300 (all-mpnet-base-v2 [51, 57], a billion sentence pairs, Flickr and COCO captions among them) and .320 (CLIP’s text encoder, 400M image–text pairs, [50]), but our good-view JEPAs have the neighborhoods that agree most with DINOv2’s (mutual 10-NN .204–.257 vs. .189 and .167), and the one trained on the texts of 940M web images also retrieves better than both (.349, seed 0). Retrieval is limited by how well the rotation is estimated: fitting the rotation on all 30k Flickr train and validation pairs instead of 1k, still the only part fitted on pairs, raises R@1 to .346 for the good-view JEPA (.363 with BERT-B and .400 for a BERT-B trained on Flickr and COCO captions, one seed each), against .21–.25 for the bad-view JEPA and next-token prediction and .48–.49 for the two reference encoders. Crucially, a good-view JEPA trained on web text alone closes most of that last gap (.464, Section 4).
| Text encoder, vs. frozen DINOv2 ViT-B/14 | mutual 10-NN | R@1, rotation fit on 1k pairs | 30k pairs |
| Qwen2.5-0.5B | .098 | .135 | .240 |
| BERT-base | .093 | .149 | .273 |
| bad-view JEPA | .132±.001 | .133±.007 | .207±.008 |
| next-token | .122±.001 | .118±.002 | .213±.004 |
| early-stopped | .124±.002 | .134±.008 | – |
| next-token, pairs | .132±.002 | .137±.004 | .247±.016 |
| good-view JEPA | .232±.006 | .246±.015 | .346±.007 |
| good-view JEPA, BERT-B | .245 | .273 | .363 |
| good, BERT-T, +COCO | .228 | .238 | .335 |
| good, +COCO | .228 | .256 | .367 |
| good, BERT-B, +COCO | .233 | .285 | .400 |
| good, BERT-L, +COCO | .227 | .269 | .376 |
| good-view JEPAs on Recap texts, no Flickr or COCO caption | |||
| texts of 2.16M images | .204±.004 | .236±.019 | .335±.016 |
| BERT-B | .214±.003 | .272±.010 | .388±.014 |
| texts of 10M images | .205±.004 | .243±.004 | .348±.010 |
| BERT-B | .219±.003 | .280±.003 | .387±.007 |
| texts of 50M images | .219±.002 | .283±.002 | .376±.006 |
| BERT-B | .236±.002 | .300±.011 | .408±.003 |
| texts of 100M images | .229±.001 | .309±.007 | .402±.007 |
| texts of 940M, BERT-B | .257 | .349 | .464 |
| all-mpnet-base-v2 | .189 | .300 | .481 |
| CLIP B/32 text | .167 | .320 | .489 |
Table 2. Table 2. Good-view JEPAs agree with DINOv2 more than the pretrained models we tested. Same protocol with a frozen DINOv2 ViT-B/14. Top two rows and bottom two: pretrained models (last layer); the others are ours (BERT-S unless named). All entries use final checkpoints, except the early-stopped row (selected on a loss over the validation and test captions, which can only favor this baseline), with no selection on test data (mean ± half-range over three seeds where shown, seed 0 elsewhere). The last column fits the single map on all 30k Flickr train and validation pairs instead of 1k (–: we kept no epoch29 checkpoint to embed the training captions with). No network is trained on pairs in either case. For reference, our Flickr image JEPA vs. DINOv2 (image vs. image) reaches mutual 10-NN .101 and R@1 .217. The rows above the first rule are trained on Flickr (and COCO) captions, so this comparison is in-domain for them and out-of-domain for most pretrained models (all-mpnet-base-v2, trained on a billion sentence pairs, saw Flickr and COCO captions); the Recap rows never saw a Flickr or COCO caption. CLIP’s text encoder was trained with 400M images. Bold: best of ours. This demonstrates that good-view JEPAs have the neighborhoods closest to DINOv2’s and, trained on web text alone, match or exceed the retrieval of text encoders trained on pairs.
Next-Token Prediction Scales Poorly Compared to Good-View JEPAs
Language models align with vision through scale [22]. We now show that next-token prediction spends that
scale poorly: on the same corpus, good views reach its alignment with 42 times fewer samples, and a 109M good-view JEPA that read 116B text tokens beats language models with up to 660 times more parameters, trained on 15–18T tokens, over 100 times more.
Good Views Scale on Web Text
Trained on more text, a text JEPA leaves the caption domain. We train the same 33M-parameter encoder on the texts of Recap-DataComp web images [34]: each image’s alt-text and two recaptions, written by an image captioner (LLaVA built on LLaMA-3-8B), are its views. We obtain three key observations (Figure 9). First, on the WIT-1024 benchmark of Huh et al. [22], on which none of our models is trained, alignment with DINOv2 ViT-B rises from .034 when trained on Flickr captions to .108, .111 and .125 when trained on the texts of 2.16M, 10M and 50M images and stays at .125 with 100M (seeds differ by at most ±.009). At 50M and 100M images this 33M-parameter encoder, trained without next-token or masked-token prediction, is level with Qwen2.5-3B and -7B (.125 and .118; intervals in Section C). Second, views matter at both scales we tested: with the same encoder, texts and steps, the bad-view JEPA reaches .053 and next-token prediction .071, against .108 for the good-view JEPA, and with 23 times more images (1.9 times more steps) .063 and .079 against .125. Third, Flickr retrieval keeps rising with data, to .348, .376 and .402 with the texts of 10M, 50M and 100M images and .408 with BERT-B on 50M, level with our best in-domain model (.400) without a single Flickr or COCO caption: good views on web text carry most of the in-domain alignment. Width helps Flickr retrieval but not yet WIT at these scales: BERT-B reaches R@1 .388 and .408 with the texts of 2.16M and 50M images, but WIT .109 and .116, at or below BERT-S (.108 and .125), and with Flickr and COCO captions fixed, R@1 grows with the size of the text encoder up to BERT-B and then drops (.335, .367, .400 and .376 for 11M, 33M, 109M and 335M parameters; one seed).
Figure 9. Good views scale. Left and middle: text JEPAs trained on the texts of 2.16M to 940M web images (Recap-DataComp), never on a Flickr or COCO caption, against frozen DINOv2 ViT-B on WIT-1024 (PRH-paper protocol) and on Flickr (R@1 through one rotation fitted on 30k pairs). Solid lines and circles: BERT-S (33M); dashed lines and diamonds: BERT-B (109M), trained with next-token prediction and bad views on the texts of 940M images only, hence single diamonds there. Lines are means over seeds and faint dots the seeds (one seed for the bad-view JEPA at 940M). Shaded: the language models we evaluated (Qwen2.5 0.5–72B, Llama-3-8B and -70B, Mistral-7B; left); grey dashed: CLIP’s text encoder and Qwen2.5-0.5B under the same protocol (middle). Right: image JEPAs trained from scratch (ViT-S on 2.16M and ViT-B on 10M Recap images, ViT-S on ImageNet-1k) and frozen DINOv2, each against the good-view, next-token and bad-view text JEPAs trained on the same texts of 50M web images (WIT; means of three seeds, dots: seeds). We clearly observe that good views beat bad views and next-token prediction at every scale and with every image encoder we compared, and that a 109M good-view JEPA on the texts of 940M images surpasses every language model we evaluated.
Good Views Need 42 Times Fewer Samples
With the same 109M encoder and texts from the same corpus, the good-view JEPA reaches .109 on WIT after 21.6M training samples (the texts of 2.16M images), the alignment that next-token prediction reaches after 900M samples (.111, the texts of 940M images; we scored no earlier next-token checkpoint), and a higher Flickr R@1 (.388 against .335); each run uses the recipe of its scale (learning rate and batch 256 against and 2,048). That is 42 times fewer samples, 14 times fewer text tokens since a good-view sample reads three texts, and 435 times fewer unique images (Figure 1, right). At matched samples the gap is larger still: .108 against .071 at 21.6M samples for the 33M encoder.
A 109M Text JEPA Beats Language Models up to 72B Parameters
Scaling data and model together changes the picture. We train the 109M-parameter BERT-B text JEPA with 900M samples from the texts of all 940M Recap-DataComp images (8 or 16 GPUs), and as controls next-token prediction and the bad-view JEPA (two masked copies of one text) with the same encoder, texts and samples. Table 3 gathers the comparison, from which we draw four conclusions.
| Text encoder | WIT, DINOv2-B 10-NN | WIT CKA | WIT CLIP-H | COCO 10-NN | Flickr R@1 |
| good-view JEPA | .184 | .507 | .230 | .161 | .478 |
| next-token | .111 | – | .146 | – | .335 |
| bad-view JEPA | .088 | – | .116 | – | .318 |
| Qwen2.5-72B | .154 | .418 | .210 | .108 | – |
| Llama-3-70B | .151 | .380 | .206 | .113 | – |
| CLIP B/32 text | .124 | .339 | .159 | .132 | .489 |
| all-mpnet-base-v2 | .118 | .390 | .159 | .156 | .481 |
Table 3. Table 3. At scale, good views beat language models up to 72B parameters. Text encoders against frozen vision models under the PRH protocol: mutual 10-NN and unbiased linear CKA on WIT-1024 with DINOv2 ViT-B, mutual 10-NN with WIT-1024 with CLIP ViT-H/14 and on the 5,000 COCO test pairs with DINOv2 ViT-B, and Flickr R@1 through one rotation fitted on 30k pairs. Our three encoders are BERT-B models (109M) trained with 900M samples from the texts of 940M web images (one seed for the bad-view JEPA); –: not measured. More models in Table 16 to Table 18. We clearly observe that the good-view JEPA agrees with both vision models more than any other text encoder, while next-token prediction on the same texts falls far behind.
Good views beat every language model we evaluated. On the PRH paper’s benchmark, the good-view JEPA reaches .184 with DINOv2 ViT-B, above Qwen2.5 from 0.5B to 72B, Llama-3-8B and -70B and Mistral-7B, and it stays ahead with every DINOv2 size, with linear CKA in place of mutual 10-NN, and on the five times larger COCO benchmark (Table 16 and Table 17).
| WIT-1024 | COCO-5k | ||||||
| mutual 10-NN, DINOv2 S / B / L / g | CKA, B | 10-NN, B | CKA, B | ||||
| Qwen2.5-0.5B | .092 | .102 | .114 | .115 | .318 | .100 | .571 |
| Qwen2.5-1.5B | .101 | .108 | .123 | .124 | .324 | .089 | .538 |
| Qwen2.5-3B | .110 | .125 | .139 | .141 | .363 | .099 | .550 |
| Qwen2.5-7B | .110 | .118 | .137 | .133 | .353 | .101 | .535 |
| Qwen2.5-14B | .119 | .130 | .147 | .148 | .333 | .106 | .539 |
| Qwen2.5-32B | .116 | .128 | .145 | .144 | .336 | .104 | .533 |
| Qwen2.5-72B | .138 | .154 | .166 | .169 | .418 | .108 | .534 |
| Mistral-7B | .126 | .140 | .157 | .157 | .349 | .112 | .539 |
| Llama-3-8B | .130 | .145 | .165 | .164 | .377 | .109 | .599 |
| Llama-3-70B | .136 | .151 | .166 | .166 | .380 | .113 | .587 |
| ours: good-view text JEPA, 109M | .166 | .184 | .196 | .197 | .507 | .161 | .622 |
Table 16. No language model up to 72B parameters reaches the 109M good-view JEPA. PRH protocol (per-layer features, maximum over layer pairs) on WIT-1024 and on the 5,000 COCO test pairs, for base language models from 0.5B to 72B parameters and our 109M good-view text JEPA trained on the texts of 940M images (mean of three seeds). We clearly observe that scaling next-token prediction by 660 times in parameters does not reach the alignment that good views give.
| Text encoder (COCO-5k) | mutual 10-NN DINOv2-B | mutual 10-NN DINOv2-g | unbiased CKA DINOv2-B | unbiased CKA DINOv2-g |
| ours, good, BERT-B, 940M images | .161 | .169 | .622 | .667 |
| ours, good, 100M images | .134 | .140 | .587 | .598 |
| ours, good, 50M images | .126 | .135 | .578 | .576 |
| ours, bad, 50M images | .085 | .090 | .447 | .423 |
| ours, next-token, 50M images | .091 | .096 | .522 | .487 |
| Qwen2.5-0.5B | .100 | .105 | .571 | .529 |
| Qwen2.5-1.5B | .089 | .095 | .538 | .505 |
| Qwen2.5-3B | .099 | .104 | .550 | .525 |
| Qwen2.5-7B | .101 | .107 | .535 | .510 |
| Qwen2.5-14B | .106 | .112 | .539 | .526 |
| Qwen2.5-32B | .104 | .109 | .533 | .521 |
| Qwen2.5-72B | .108 | .113 | .534 | .521 |
| Mistral-7B | .112 | .118 | .539 | .526 |
| Llama-3-8B | .109 | .115 | .599 | .560 |
| Llama-3-70B | .113 | .118 | .587 | .529 |
| BERT-base | .104 | .109 | .485 | .475 |
| all-mpnet-base-v2 | .156 | .162 | .608 | .577 |
| CLIP text encoder | .132 | .138 | .576 | .565 |
Table 17. The ranking holds on COCO. PRH protocol on the 5,000 COCO Karpathy test pairs (first caption), seed 0 for our encoders except the 940M row, the mean of three seeds (each at or above all-mpnet-base-v2); chance mutual 10-NN is .002. We clearly observe that the 109M good-view JEPA aligns best on a second, five times larger benchmark, ahead of language models up to 72B parameters and of text encoders trained on pairs.
The views make the gain, not the texts. Next-token prediction and the bad-view JEPA trained on the same texts stay far below, and, as on Flickr captions, the scene explains more of the good-view embedding than of theirs (measured on the texts of 50M images, Table 6).
The lead holds with every vision family. Language models align best with CLIP’s image encoders [22], yet with CLIP ViT-H/14 the good-view JEPA is ahead of Qwen2.5-72B and Llama-3-70B, and it leads every language model with MAE and I-JEPA as well (Table 18).
| Vision encoder | ours (33M), texts of 50M images good | ours (33M), texts of 50M images bad | ours (33M), texts of 50M images next-token | ours (109M), 940M good | Qwen2.5 0.5B | Qwen2.5 7B | Mistral 7B |
| MAE ViT-B/16 (IN-1k) | .074 | .045 | .053 | .100 | .060 | .067 | .077 |
| MAE ViT-L/16 (IN-1k) | .093 | .050 | .059 | .126 | .074 | .084 | .097 |
| MAE ViT-H/14 (IN-1k) | .092 | .050 | .058 | .125 | .071 | .084 | .096 |
| I-JEPA ViT-H/14 (IN-22k) | .095 | .051 | .059 | .129 | .068 | .081 | .096 |
| DINOv2 ViT-B/14 | .129 | .066 | .079 | .191 | .102 | .118 | .140 |
| DINOv2 ViT-L/14 | .133 | .069 | .084 | .199 | .114 | .137 | .157 |
| CLIP ViT-B/16 (LAION-2B) | .147 | .076 | .093 | .221 | .135 | .156 | .180 |
| CLIP ViT-L/14 (LAION-2B) | .148 | .078 | .098 | .224 | .144 | .164 | .194 |
| CLIP ViT-H/14 (LAION-2B) | .156 | .080 | .099 | .233 | .148 | .169 | .196 |
Table 18. Good views win with every vision family. WIT-1024 (PRH protocol) for frozen vision encoders trained by pixel reconstruction (MAE), latent prediction (I-JEPA), crops (DINOv2) or image–text contrast (CLIP’s image encoders, the vision models with which language models align best in Huh et al. [22]), against our text JEPAs (good / bad / next-token on the texts of 50M images, seed 0; the 109M good-view JEPA on the texts of 940M images, seed 0) and pretrained language models. We clearly observe that good views beat next-token prediction and bad views with every vision encoder, and that the 109M good-view JEPA beats the three language models with each of them. MAE and I-JEPA align alike, both below DINOv2, which saw far more curated images; we draw no conclusion on the vision objective. With CLIP’s image encoders, the 109M good-view JEPA’s three seeds average .215, .219 and .230, against .193, .203 and .210 for the best of all our language models up to 72B (Qwen2.5-72B), and its next-token and bad-view controls on the same texts .137–.146 and .108–.116 (runs/results/wit_clip.txt).
It rivals text encoders trained on pairs. It agrees with DINOv2 more than CLIP’s text encoder (400M image–text pairs) on both benchmarks, more than all-mpnet-base-v2 (a billion sentence pairs, COCO captions among them) on WIT and as much on COCO, and it retrieves Flickr captions through one rotation almost as well as both. CLIP’s text encoder is trained to agree with CLIP’s own image encoder, not with DINOv2, so this compares agreement with an independently trained vision model.
Three caveats qualify these numbers. A fourth seed, trained with twice the effective SIGReg weight, reaches only .121 on WIT, after which we kept the first seed’s weight for the others; all four seeds average .168, still above every language model (Section C). The recaptioner that wrote most of these texts sees images through CLIP ViT-L/14 (§ 7). And at least 43 of the 1,024 WIT captions appear among the Recap-DataComp alt-texts, although removing them changes none of the nine text models we checked by more than .002. Still, our model has over 600 times fewer parameters than Qwen2.5-72B and never saw pixels: good views turn scale into alignment far more efficiently than next-token prediction.
The image side matters as well. Image JEPAs trained from scratch without texts, on more images (all bad-view JEPAs: their positives are crops of one image), rank the text side the same way: with a ViT-S on the 2.16M Recap images, a ViT-B on 10M and a ViT-S on ImageNet-1k, the good-view text JEPA aligns best, next-token prediction second and the bad-view JEPA last (Figure 9, right). The ImageNet ViT-S, which sees the most samples, agrees most with DINOv2 and with our text JEPA, and with it two JEPAs trained separately from scratch agree about as much as the image JEPA agrees with Mistral-7B and Qwen2.5-7B (Section C).
A Scaling Law for the PRH
Scaling laws turned the training of language models from trial and error into prediction; we now show that alignment admits one too. With one recipe (BERT-S on the
texts of Recap-DataComp images; Section C), each training item draws good views (two texts of its image) with probability and bad views (two corruptions of one text) otherwise, on the texts of images for samples. A bad-view floor rising with plus a good-view term,
fits the 45 cells of a grid up to 80M samples with RMSE .007 (Figure 11). We then predicted 85 runs at random on 10.2M images, with from 3M to 133M samples, before training them: it predicts their Flickr R@1 with RMSE .0093, while two seeds of one run differ by .0096 (pooled over six cells and three seeds each). That is, the law predicts a run it has never seen as well as rerunning that run with another seed. Along its iso-alignment lines, more samples substitute for good views sub-linearly (), and unique images enter weakly (): without good views, WIT stays within .054–.057 from the texts of 0.41M to 50.9M images, while with them it rises from .090 to .115. The law fails further out, for good views only: at 320M samples, four times its longest fitted run and 31 passes over the images, it overestimates the runs with good views (R@1 .418 against .455 predicted at , .348 against .389 at ), while the run without matches (.338 against .337); it has no term for repeated data. Its exponents are poorly identified, and it is an empirical summary of one recipe, not a consequence of Theorem 4.
Figure 11. Figure 11. A scaling law of alignment in good views and samples. Flickr text R@1 (DINOv2 ViT-B, 30k-pair rotation) of BERT-S text JEPAs trained on the texts of 10.2M web images, against the samples seen. Each point is a run, colored by its fraction of training items with good views (black: , bad views only). Circles: the grid the law is fitted on; diamonds: runs predicted before training. Lines: the law at five values of , dashed beyond the fitted range. We observe that R@1 grows linearly in the logarithm of the samples seen at every fraction of good views, faster with more of them, and that the law predicts unseen runs within seed noise up to twice its range.
Platonic Representations as a Tool
Alignment up to one rotation lets separately trained encoders be composed. We show two uses: a captioner whose only part fitted on pairs is one rotation, and alignment with no pair at all.
Captioning Without Training a Network on Image–Caption Pairs
Alignment up to one rotation makes a new kind of captioner possible. We train a caption decoder on text alone: a causal transformer reconstructs each training caption from the frozen good-view JEPA embedding of that caption, with Gaussian noise added to the embedding so that it tolerates alignment error [43]. To caption an image, we embed it with the frozen DINOv2, apply the rotation and decode. That is, the rotation is the only part fitted on image–caption pairs, and no network is trained on them. Crucially, the noise level is matched to the alignment error rather than tuned: after the rotation, an image and its caption have mean cosine (.29 on held-out validation pairs, .33 on test pairs), and Gaussian noise whose norm is times the embedding’s leaves an expected cosine of with it, so matching the two gives ; we use , and transfer peaks there (Figure 13). We obtain a few key observations. First, on the 1,000 Flickr test images the captioner reaches CIDEr .396 (on a 0–1 scale throughout; most prior work reports it times 100) against five human captions, nine times a random rotation and 2.5 times retrieving the nearest caption through the same rotation (Table 9), and .431 with the rotation fitted on all 30k Flickr pairs. Second, rotations fitted on fewer pairs than dimensions are underdetermined (Proposition 2 (ii)) but improve when refined on unpaired images and captions (Figure 13): from 30 pairs, the captioner (.169) matches caption retrieval with 1,014 pairs (.158). Third, the captioner transfers across datasets: on the 5,000 COCO Karpathy test images, with the rotation fitted on Flickr pairs only and the decoder trained on Flickr and COCO training captions, it reaches CIDEr .668 and BLEU-4 .199 with a beam of three (.630 greedy), against .038 for a random rotation and .169 for retrieving the nearest Flickr caption. Captioners built on CLIP’s jointly trained encoders, and hence on 400M image–text pairs, report .345 (ZeroCap, [60], no training), .493 (MAGIC, [59]) and .918 and .912 (CapDec and DeCap, [32, 43]); unlike these, ours uses no encoder trained on pairs but fits its rotation on in-domain Flickr pairs, which they do not use. The rotation is also what makes the captioner transfer: replacing it by maps trained on the same 30k Flickr pairs (Table 10), an unconstrained linear map gains a little on Flickr (CIDEr .531 against .504, web-pretrained decoder, beam 3) but loses a fifth on COCO (.635 against .802), and a two-layer MLP loses on both (.498 and .585); with consensus decoding, the same maps reach .561 and .511 on Flickr but .652 and .597 on COCO, against .538 and .832 for the rotation. That is, a map free to fit the pairs fits Flickr, while a rotation, which can only turn one space onto the other, carries over to new images, as expected if the two spaces agree up to a rotation.
| Captioner (Flickr test) | BLEU-4 | CIDEr |
| Oracle: decode the true caption embedding | .261 | .628 |
| Image → rotation (30k pairs) → decode | .185 | .431 |
| Image → rotation (1,014 pairs) → decode | .173 | .396 |
| Image → rotation (300 pairs) → decode | .146 | .317 |
| Image → rotation (100 pairs) → decode | .119 | .244 |
| + unpaired refinement (100 pairs) | .138 | .294 |
| + unpaired refinement (30 pairs) | .099 | .169 |
| Retrieve nearest caption (30k pairs) | .065 | .188 |
| Retrieve nearest caption (1,014 pairs) | .054 | .158 |
| Random rotation | .059 | .043 |
Table 9. Captioning from independently trained encoders. BLEU-4 [45], CIDEr [62] on Flickr test images. The decoder never sees an image; the image encoder never sees text; pairs only fit one linear map. BLEU-4 rewards generic n-grams, so the random map’s fluent but generic captions score above 1,014-pair retrieval on BLEU-4, not on CIDEr. We clearly observe that decoding through the rotation beats retrieving captions through it by a wide margin.
Figure 13. The captioner’s noise and pairs are predicted, not tuned. Flickr test images. Top: CIDEr against the noise the decoder is trained with; dashed, the noise matched to the residual of the rotation (rotation from 1,014 pairs). Bottom: CIDEr against the number of pairs that fit the one linear map, with and without unpaired refinement, against retrieving the nearest caption through the same map; dashed, decoding the true caption embedding. We clearly observe that transfer peaks at the matched noise, and that a few pairs refined on unpaired data match caption retrieval from 1,014 pairs.
| Captioner (CIDEr; one rotation from the 30k Flickr pairs) | Flickr | COCO |
| Small decoder, DINOv2 ViT-B, in-domain text JEPA (Table 9) | .431 | .668 |
| Qwen2.5-1.5B decoder, web text JEPA, DINOv2 ViT-g (CLS) | .466 | .720 |
| + mean patch token | .480 | .756 |
| + captions as written (case, punctuation) | .479 | .765 |
| + DINOv3 ViT-L, concatenated | .496 | .806 |
| + consensus of 32 samples | .519 | .838 |
| Variants of the DINOv3 ViT-L row | ||
| DINOv3 ViT-7B instead of ViT-L | .507 | .816 |
| decoder first trained on 4M web sentences | .504 | .802 |
| map trained on the same pairs instead: any linear map | .531 | .635 |
| map trained on the same pairs instead: two-layer MLP | .498 | .585 |
| + consensus of 32 samples (the demo) | .538 | .832 |
| map trained on the same pairs instead: any linear map | .561 | .652 |
| map trained on the same pairs instead: two-layer MLP | .511 | .597 |
| Qwen2.5-3B decoder (captions-as-written row) | .460 | .730 |
Table 10. Table 10. The captioner improves without pairs. Building the captioner up one change at a time: Flickr test (1,000 images) and COCO Karpathy test (5,000 images), beam 3 unless stated; consensus rows choose among the greedy caption and 32 samples. The text JEPA is BERT-S trained on the texts of 100M web images; the decoders are trained on its embeddings of Flickr and COCO training captions only, with noise matched to the alignment (); the image encoders are frozen. The first row is the captioner of Table 9: on Flickr, the Flickr text JEPA and decoder (); on COCO, the Flickr and COCO ones (); it decodes greedily on Flickr and with beam 3 on COCO. Decoding the true caption embedding gives .694 and 1.305 (captions-as-written row), and .743 and 1.369 with the web-pretrained decoder. Only the rotation is fitted on image–caption pairs. Without consensus decoding, the best captioner reaches .496 and .806. This demonstrates that stronger frozen encoders, a pretrained language model and consensus decoding each add to the captioner, from CIDEr .668 to .838 on COCO, while a larger decoder does not.
Lastly, the captioner improves further without pairs (Table 10): with a pretrained language model fine-tuned on text alone as the decoder (Qwen2.5-1.5B, [49]), the text JEPA trained on the texts of 100M web images, DINOv2 ViT-g and DINOv3 ViT-L [56] as frozen image encoders, and the consensus of 32 sampled captions [27], a single map fitted on the same 30k Flickr pairs gives CIDEr .519 on Flickr and .838 on COCO, and .538 and .832 with the decoder of our demo, first trained on web sentences (Figure 10). We thus recommend matching the decoder’s noise to the residual cosine of the rotation, , rather than tuning it.
Figure 10. Captions without training a network on image–caption pairs. Selected captions of COCO test images (first lines) with the decoder of our demo: frozen DINOv2 and DINOv3 image embeddings, one linear map fitted on the 30k Flickr pairs, and a language model fine-tuned on text only (consensus of 32 samples; Table 10), with one human caption (grey). That is, frozen encoders joined by one rotation describe scenes, objects and actions, although only the rotation is fitted on image–caption pairs.
From photos to videos. Neither the encoders nor the decoder ever saw a video, yet the same captioner describes one from the mean of its frames’ rotated embeddings. On the 2,990 MSR-VTT test clips [68], with the rotation fitted on the 30k Flickr pairs and the mean of eight frames, it reaches CIDEr .246 against the 20 human captions of each clip, above the zero-shot captioners whose decoder is trained on image captions or general text alone, all built on CLIP’s jointly trained encoders (.058 for ZeroCap, .074 for MAGIC and .150–.186 for DeCap, [32]), though below video–language systems such as VideoCoCa (.271, 144M video–text pairs, [69]) and TOPA (.329, CLIP features and LLM-written textual videos, [33]), and below decoders trained on MSR-VTT captions (.348). Averaging frames helps both captions and alignment, as Theorem 3 suggests: the frames of a clip are several observations of one scene, and their mean averages out what is particular to each frame. On the 1,000 clips of the 1k-A split, it raises the fit-free mutual 10-NN between clips and captions from .148 (middle frame) to .167, and the same rotation, fitted on photos, retrieves the right clip first for 16.5% of captions, against 31.2% for CLIP ViT-B/32 trained on 400M image–text pairs [35] (Table 11). That is, the rotation fitted on photos carries over to video.
| MSR-VTT test, 2,990 clips | CIDEr | BLEU-4 | METEOR | ROUGE-L |
| Ours: mean of 8 frames, rotation from 30k Flickr pairs, consensus of 32 | .246 | .166 | .206 | .444 |
| beam 3 | .222 | .154 | .204 | .432 |
| map from 30k Flickr and 200k web pairs (the demo’s) | .268 | .186 | .232 | .471 |
| middle frame instead of the mean of 8 | .242 | .172 | .219 | .457 |
| oracle: decode the text embedding of one human caption | .383 | .257 | .265 | .538 |
| CLIP ViT-B/32 encoders (400M image–text pairs), no MSR-VTT text; from Li et al. [32], Tab. 4 | ||||
| Zerocap [60] | .058 | – | – | – |
| MAGIC [59] | .074 | – | – | – |
| DeCap, decoder on CC3M captions | .150 | – | – | – |
| DeCap, decoder on COCO captions | .186 | .147 | .204 | – |
| DeCap, decoder on MSR-VTT captions | .348 | – | – | – |
| MSR-VTT 1k-A (1,000 clips) | map fitted on | mutual 10-NN | text→video R@1/5/10 | video→text R@1 |
| ours, mean of 8 frames | 30k Flickr pairs | .167 | 16.5 / 38.8 / 49.1 | 18.6 |
| ours, mean of 8 frames | 9k MSR-VTT clips | .167 | 19.3 / 45.6 / 57.9 | 20.5 |
| ours, middle frame | 30k Flickr, 200k web pairs | .148 | 14.5 / 33.1 / 44.9 | 17.1 |
| CLIP ViT-B/32 [35] | – (jointly trained) | – | 31.2 / 53.7 / 64.2 | 27.2 |
Table 11. Table 11. Captions and retrieval for video, from photos alone. Top: zero-shot captioning of the MSR-VTT test clips (20 references each; CIDEr on a 0–1 scale) by our captioner, whose networks never saw a video, a video–caption pair or MSR-VTT text, and by captioners built on CLIP’s encoders without MSR-VTT text, with one trained on MSR-VTT captions for reference (–: not reported). Bottom: alignment (mutual 10-NN fits nothing) and retrieval through one map on the 1k-A split, against zero-shot CLIP. We observe that a rotation fitted on photos captions and retrieves videos, above captioners built on CLIP that use no MSR-VTT text, though CLIP’s jointly trained encoders retrieve better.
Alignment Without Any Pair
Proposition 2 says that the first two moments of unpaired data cannot find the rotation, but the geometry of a finite world can: Schnaus et al. [54] recover it with no pair at all, by matching clusters of the two clouds and refining an orthogonal map by optimal transport. The theory predicts that this works best when the two encoders’ geometries agree, i.e., with good views. We test it with their method, code, hyperparameters and five seeds unchanged, on their protocol: DINOv2 ViT-B/14 against a text encoder, the map fitted on disjoint unpaired halves of COCO train2014 (images from one half, captions from the other), and scored on the 40,504 val2014 pairs by FOSCTTM (the fraction of samples closer than the true match; 0 is perfect, .5 chance) and by zero-shot classification (Table 4). Our setup reproduces their numbers for all-mpnet-base-v2. Our 109M good-view JEPA (three training seeds, five method seeds each) reaches FOSCTTM .018 and zero-shot accuracies of 86.0% (CIFAR-10 top-1), 67.2% (CIFAR-100 top-5) and 64.4% (ImageNet-100 top-5): above every zero-shot accuracy of their paper, over 21 pairs of seven vision and three language models, up to DINOv3-7B and Qwen3-8B, and five settings (at most 82%, 62% and 56%), and level with their best FOSCTTM on these data (.02; .01 with captions from another dataset). Rerun under identical conditions, their strongest text encoder, Qwen3-8B, reaches .018, 83.4%, 61.7% and 47.6%: our 70 times smaller JEPA matches it on FOSCTTM and CIFAR-10 and beats it by 5.5 points on CIFAR-100 and 16.8 on ImageNet-100 (Mann–Whitney and ). It even matches CLIP’s text encoder, trained on 400M image–text pairs, on both CIFARs, though not on FOSCTTM (.006) or ImageNet-100 (68.7%). Next-token prediction on the same texts fails in half of its runs (FOSCTTM .287, 25.5% on CIFAR-10), while our bad-view JEPA aligns well too (.026), so on COCO the contrast is between JEPAs and next-token prediction. On Flickr, whose unpaired pools are smaller (15,007 images and the captions of 15,007 others), only good-view JEPAs leave chance: with no pair, the 109M good-view JEPA retrieves the right caption first for 16.4% 10.4% of test images (median 16.1%; chance 0.1%), against at most 0.3% for CLIP’s text encoder, all-mpnet-base-v2, BERT, Qwen2.5-0.5B, and our bad-view JEPA and next-token models trained on the same texts (Section C). The method’s own objective on the unpaired pools, which uses no test data, picks the best of its five seeds in all six good-view JEPA cases (R@1 13.5–36.2%). Through such a map, a captioner fitted with zero pairs reaches CIDEr .158 .042 on Flickr (.208 for the map objective selects), against .430 with the 30k-pair rotation and .028 for a random map. That is, good views make two modalities alignable from their geometry alone, which next-token prediction does not.
| Text encoder (runs) | FOSCTTM | C-10 | C-100 | IN-100 |
| good-view JEPA, 109M (15) | .018 | 86.0 | 67.2 | 64.4 |
| good-view JEPA, 33M (15) | .036 | 83.2 | 60.9 | 57.9 |
| bad-view JEPA, 109M (5) | .026 | 78.1 | 55.3 | 46.7 |
| next-token, 109M (15) | .287 | 25.5 | 19.9 | 20.3 |
| Qwen3-8B, rerun (5) | .018 | 83.4 | 61.7 | 47.6 |
| MPNet, rerun (5) | .043 | 70.2 | 45.6 | 45.4 |
| their best, these data | .02 | 80 | 58 | 46 |
| their best, any setting | .01 | 82 | 62 | 56 |
| CLIP B/32 text (5) | .006 | 84.0 | 67.0 | 68.7 |
| 82,783 pairs | .002–.005 | 83–92 | 65–85 | 57–88 |
Table 4. A 109M good-view JEPA beats every pair-free zero-shot accuracy of Schnaus et al. [54]. The protocol and method of Schnaus et al. [54], unchanged: DINOv2 ViT-B/14 and a text encoder aligned with zero pairs on unpaired halves of COCO train2014, scored on the 40,504 val2014 pairs (FOSCTTM, lower is better) and by zero-shot top-1 on CIFAR-10 and top-5 on CIFAR-100 and ImageNet-100 (%). Means over training seeds times the method’s five seeds (std. in Section C). Their best: the best number of their paper for each metric over its 21 vision–language pairs, on these data and in any of its five settings (paper/data/schnaus2026_reported.csv); our reruns of their MP-Net pair (all-mpnet-base-v2) reproduce their reported .04, 70, 46 and 45. Last row: the ceiling of Procrustes on all training pairs, range over all encoders. Our good-view JEPAs are trained on the texts of 940M and 100M web images. We clearly observe that a 109M good-view JEPA beats every zero-shot accuracy of Schnaus et al. [54], matches their best FOSCTTM on these data, and matches or significantly beats their 8B language model under identical conditions, while next-token prediction on the same texts fails; only CLIP’s text encoder, trained on 400M image–text pairs, aligns better on FOSCTTM and ImageNet-100.
Related Work
The PRH was formulated by Huh et al. [22], whose contrastive argument assumes bijective observations and co-occurrence positives. They note that it fails for lossy observations, which remain open; Theorem 2 addresses a different departure, mismatched positives. Ziyin and Chuang [75] prove perfect alignment for deep linear networks trained by SGD, and Doshi et al. [14] explain why translations align inside multilingual language models by a hierarchical structure that languages share beneath surface features of their own, a within-modality analogue of our world states and nuisances. Identifiability results show that contrastive and augmentation-based learning recover latent variables up to simple transformations [64, 74], including across modalities when trained jointly [12]; we instead train modalities separately and characterize when they agree. The spectral view of self-supervised learning [3, 8, 19] underlies Proposition 1. How views and data shape what one encoder learns has been analyzed through augmentation graphs and their overlaps [19, 66], the inductive bias of the function class
[9, 26, 52], the eigenfunctions of the positive-pair kernel [24] and multi-view redundancy [61]; Theorem 4 is an analogue with world states in place of classes, and its part (ii) the cross-modal consequence. The density view of JEPA anti-collapse [4, 5] underlies our reading of next-token prediction. Aligning separately trained embedding spaces with few or no pairs goes back to word translation, through orthogonal Procrustes and unsupervised matching [1, 11], and extends to relative representations [41], training-free multimodal coupling from a few anchor pairs [42], translation between text-embedding spaces without pairs [23], vision–language matching without parallel data [53] and, most recently, cross-modal alignment with no pairs at all from shared geometry [54]; unimodal vision and language encoders can be matched with little or no paired data [36]. These works align given encoders; we ask which training objective makes encoders alignable, and our Procrustes map is the simplest of these tools. Text-only captioners decode the embeddings of jointly trained encoders [32, 43, 59, 60], a linear map trained on pairs can feed image features to a frozen language model [38], and unpaired captioning trains adversarially or through shared detector concepts [18, 28]; ours needs no joint training and uses pairs only to fit one map with orthonormal rows.
Conclusion
We have turned the Platonic Representation Hypothesis into a Platonic Representation Theorem for JEPAs which, for population optima on injective renderings of one world and objectives whose minimizers are unique up to rotation, reduces it to one requirement for any number of modalities: they must agree on which world states are positives. We further proved that positives which corrupt a single observation make an encoder spend dimensions on its modality’s nuisance, which no rotation can align, and showed that next-token prediction has the form of such a bad-view JEPA. Empirically, good views align best across data, encoders and modalities, follow a scaling law that predicts unseen runs within seed noise, need 42 times fewer samples than next-token prediction on the same corpus, and give a 109M text JEPA that beats every language model we evaluated, up to 72B parameters, on the PRH paper’s own benchmark, and a captioner whose only parameters fitted on image–caption pairs are those of one rotation, which unpaired data alone can find well enough to beat every pair-free zero-shot classification reported with language models up to 8B parameters.
Known limitations. Our theorems are population statements over a finite world with injective renderings and matched marginals; real modalities are lossy, and finite-sample training relies on the network’s inductive bias. Uniqueness up to rotation is proved for second-order anti-collapse, not for LeJEPA’s full-distribution constraint, which a finite world cannot meet exactly; Theorem 2 is proved for spectral solutions only, and no experiment meets the hypotheses of Theorem 1 exactly, since our runs use LeJEPA and the image side mostly uses crops as positives. The main caveat is where good views come from: they require knowing which texts describe the same scene, the grouping of human captions by image on Flickr and COCO, while on Recap-DataComp two of each image’s three texts are generated by an image captioner (LLaVA built on LLaMA-3-8B, whose vision encoder is CLIP ViT-L/14, [34]) and carry information distilled from CLIP text training. Without any image captioner, good views on the alt-text and a text-only paraphrase of it reach .060 on WIT against .107 with the recaptions (Table 7), so the comparisons with language models and CLIP’s text encoder are real only on these texts, while the comparisons between views, on identical texts, do not. Paired information thus enters through the definition of positives, one fitted linear map and, on Recap, captioner-generated texts, never through a cross-modal loss. The bad-view JEPA and next-token prediction see exactly the same texts as the good-view JEPA. Outside MSR-VTT our image encoders are bad-view JEPAs by our own definition, and hyperparameters were tuned only for the good-view JEPA.
Critically, our study provides what the PRH has long needed: a provable sufficient condition that directly informs how to train for it, and evidence that next-token prediction, the objective behind today’s largest models, is an inefficient way to get there. We hope that our study will establish positive views, and not scale alone, as the lever to build platonic representations, and open new avenues for modalities beyond images and text, e.g., audio, video and actions.
Acknowledgments
We thank Kerda Varaku, Lukas Kuhn, Hazel (Heejeong) Nam and Marcin Przewięźlikowski for their helpful feedback and discussions.
Proofs
We prove the results in the order of Section 2, one subsection per subsection of the main text. Proposition 3 and Theorem 4 are stated informally in the main text; their formal statements precede their proofs.
Notation and Objectives
Notation. On a finite world, the positive-pair distribution defines the Markov operator , i.e., the matrix acting on functions . It is self-adjoint in , with , and maps it to : the two share the eigenvalues , and the eigenfunctions of are orthonormal in , with , so that the spectral embedding of Proposition 1 is . The operator and Hilbert–Schmidt norms of on are the spectral and Frobenius norms of , with . On world states, the objective of (1) is , where depends on only through the distribution of , .
Objectives of the form (1). Spectral contrastive learning, whose minimizers are rescaled eigenfunctions, and whitened MSE fit this form [16, 19]; VICReg (with the mean fixed) and Barlow Twins (with ) do only in a whitened hard-constraint limit, where their minimizers are those of Proposition 1 and, both being translation invariant, agree only after centering [3, 6, 72]; LeJEPA’s SIGReg term is rotation invariant in expectation over its projection directions [4].
The Platonic Representation Theorem (Section 2.1)
Lemma 1: Renderings Only Change Coordinates If is injective, is a bijection between encoders on and functions , and .
Proof of Lemma 1. Since is injective it has a left inverse on , and inverts . For the objective, the pair with is mapped by to with , and the embeddings of the data , , have the distribution of . Hence .
Proof of Theorem 1. By Lemma 1, minimizes if and only if minimizes , for every modality . The minimizers of lie in one rotation orbit ; hence for some , for every , and for every and every two modalities .
Proof of Proposition 1. Write . The constraints say and . Expanding, and using that both marginals of equal ,
is self-adjoint on and leaves invariant, with eigenpairs there. By Ky Fan’s maximum principle [17], over orthonormal families in with equality if and only if is spanned by eigenfunctions with the largest eigenvalues. When that space is , and two orthonormal bases of the same -dimensional space differ by an orthogonal matrix, so . Conversely every is feasible and attains .
When Positives Differ (Section 2.2)
Proof of Theorem 2. Write . The map is an isometry from to under which becomes . Both matrices have the same top eigenvector with eigenvalue , because the marginals coincide, and both leave its orthogonal complement invariant. On , let be the matrix whose columns are orthonormal eigenvectors of for its largest eigenvalues; then the row of at is , and . The variant of the Davis–Kahan theorem of Yu et al. [71 Theorem 2], applied on with , , , (so the gap is of ), gives an orthogonal with
which is the bound with , since . The restriction to leaves both norms unchanged, because .
Proof of Proposition 2. Let , so that and . (i) for every , and the first moments are zero for every . Beyond the finite world: if the embedding distribution is rotation invariant, as SIGReg’s target in the limit of a continuum of world states, then for every and no statistic of unpaired samples identifies ; on a finite world, the distribution of is finitely supported and cannot be rotation invariant for , which is why unpaired data can refine a map in practice, and why the shared geometry of a finite world can reveal it [54]. (ii) Write and . The objective is , with equality at . If then , and since has rank , .
Good-View and Bad-View JEPAs
Proof of Theorem 3. Write and for a paired observation, with independent of each other and of . Since and are centered and whitened, for every , with , and the minimum over is (von Neumann’s trace inequality, attained by orthogonal Procrustes). Given , and are independent, so with and , both centered, with covariances and . By the law of total covariance , and likewise . Let and (pseudoinverses). Since for , lies in the range of almost surely, so , and likewise , with and the orthogonal projections onto the ranges of and ; then with , and by Cauchy–Schwarz. By Hölder’s inequality for Schatten norms, , which bounds the error below by ; exchanging the roles of and bounds it below by as well, hence by . Since , if and only if , i.e., , i.e., is almost surely a function of .
Proposition 3 (Formal): Good Views Ignore Nuisance, Bad Views Capture It
Modality has a finite nuisance set with distribution , , the state and the two nuisances being mutually independent, and renders injectively as ; a paired image and caption are with independent. A good-view JEPA’s positives are with and independent; a bad-view JEPA’s are two corruptions of one observation , which share its nuisance. Take the objective of Proposition 1. (i) A good-view JEPA’s kernel is ; if , its minimizers are , functions of the state alone, and good-view JEPAs of any number of modalities with the same world kernel agree up to one rotation each on every observation: Theorem 1 when every state has many observations. (ii) Let a bad-view JEPA corrupt state and nuisance independently, with and row-stochastic with stationary laws and , so that corrupted views are observable observations with the law of clean ones, and let and be the eigenvalues of the Markov operators on and on of two corruptions of one state and of one nuisance ( the adjoint; for ), with eigenfunctions and (). Its kernel then has eigenvalues with eigenfunctions ; let be those with , with multiplicity. If , for the top have , its minimizers are with zero eigenvalues, i.e. , and by Theorem 3 their alignment error with any centered, whitened encoder of the other modality is at least . Moreover if and only if .
Proof of Proposition 3. (i) The good kernel has marginal , and its Markov operator on is , where projects constants out of . Its eigenpairs are and the eigenvalue on . If , the largest eigenvalues on are , separated from the rest, and Proposition 1 gives the minimizers : . For two modalities with the same , for all . (ii) Two corruptions of one observation, drawn independently given it, have the joint law with , which factorizes as . By stationarity its marginal is , the law of clean observations, so the minimizer is centered and whitened on clean observations and Theorem 3 applies to it; its Markov operator is , with the adjoint of in , and and are positive semidefinite contractions fixing , so its eigenpairs are with and . Under the separation assumption, Proposition 1 gives the minimizers , where stacks the top eigenfunctions after . Since , which is for and for , with equal to on the eigenfunctions with and on the others, so , , and Theorem 3 gives the error bound . Finally, every eigenvalue with satisfies , so if and only if is among ; the eigenvalues with that exceed are the , , and if there are at most of them, so and ; conversely, if , then for and separation leaves . A corruption that keeps the nuisance exactly (, for every ) violates separation as soon as , but there every minimizer lies in the nuisance-only eigenspace of eigenvalue , and when .
Theorem 4 (Formal): Views and Data Set the World Kernel. Let modality have finitely many observations , each of a world state , with law () of state marginal , and let be any symmetric law on pairs of observations whose marginal is , the law on which encoders are whitened and evaluated: good views, ; corruptions of one observation that leave the law of observations invariant (, as in Proposition 3); or a mix. Its induced world kernel is the probability that a positive pair joins an observation of to one of . Let be the conditional expectation onto functions of the state in , let be the largest eigenvalue of the compression to nuisance functions (), which measures how much of its own nuisance a positive keeps ( if every state has one observation), and let and be the operator and Hilbert–Schmidt norms of , which measure how much state and nuisance mix. Take the objective of Proposition 1 with , let be the eigenvalues of on (), and suppose . (i) Every spectral solution of (orthonormal top- eigenfunctions of on ) is close to a spectral embedding of its induced world kernel,
and . (ii) For two modalities of one world, over paired observations from any coupling with and ,
where is at most by Theorem 2, with the gap of .
Special cases and scope. Good views have , and (or ), which gives Theorem 1 with many observations per state when . The product corruptions of Proposition 3 have , and , and reads : exactly the separation assumption of Proposition 3 together with (for ). The encoder class enters only through : restricting encoders to be constant on groups of observations of one state is the theorem applied to the merged observations, which leaves unchanged and can only lower and , while inductive biases that merge observations of different states, or act softly, are outside the model.
Proof of Theorem 4. (i) Drop the index . In , let be the functions of the state with zero mean and the nuisance functions, so that orthogonally (a nuisance function has zero mean because its conditional mean is zero). For and , and , because the state marginal of is : the compression of to is unitarily equivalent to on . On write with , which is block diagonal, and , whose two off-diagonal blocks are adjoint, so and . The eigenvalues of are the , with eigenfunctions , and those of the compression to , all at most . If , the largest are and the next is , so the top- eigenspace of is spanned by with gap . The variant of the Davis–Kahan theorem of Yu et al. [71 Theorem 2], applied as in the proof of Theorem 2 with , , , , gives the bound for any orthonormal top- eigenbasis of on , with or without ties in the spectrum of ; by the equality case of Ky Fan’s principle in the proof of Proposition 1, these are the minimizers of the objective on observations. Since and is a contraction, for the minimizing , and since is centered and whitened, . (ii) By (i), with and . For , , and the marginals of the paired law are and , so by Minkowski’s inequality the root-mean-square of the left side lies within of ; minimizing over , which makes range over , gives (ii). Both induced world kernels have marginal , so Theorem 2 bounds , and .
Next-Token Prediction is a Bad-View JEPA
Proof of Proposition 4. (i) and the identity (6) are immediate from the definition of the softmax. (ii) involves the whole table but not ; and . (iii) For a fixed context with conditional of full support, the expected loss , with the matrix whose rows are the , is minimized exactly when , i.e. , which is attainable when is at least the rank of the matrix modulo constants. The logits are unchanged by , for any invertible .
Numerical Checks
The released tests check the statements of § 2 numerically on random finite worlds: Lemma 1 (for the prediction term), Proposition 1 (the spectral solution attains , and 50 random whitened embeddings do no better), Theorem 1 (two modalities that render the world through different injective maps and solve their spectral JEPAs in closed form agree up to a rotation to machine precision), Theorem 2 on 300 pairs of kernels with a common marginal, half of them clustered worlds with small spectral gaps (the bound holds in every world; the largest ratio of error to bound, .18, is that of Figure 2), Proposition 2 (Procrustes on generic pairs recovers , and pairs admit another exact fit), the decomposition and GL() invariance of Proposition 4, Theorem 3 (the bound holds for 60 pairs of random whitened encoders of two modalities with nuisances), and Proposition 3 (good-view solutions of two modalities with different nuisances agree up to a rotation to machine precision; bad-view solutions lose exactly the predicted directions to nuisance and stay at least from a good encoder of the other modality; a corruption that forgets the nuisance loses none), and Theorem 4 (product corruptions give , and , and on 200 random ones the bound is finite exactly when Proposition 3 finds separation and ; positives that resample the nuisance align exactly; on random worlds with one to four observations per state and positives that mix good views, pairs of an observation with itself and random pairs, both bounds always hold, and where they are informative, below the trivial cases for (i), 307 for (ii)—the error is at most 32% of the bound in (i), median 11%, and 10% in (ii), median 0.8%).
Experimental details
Code. Everything is built on stable-pretraining: its LeJEPA class (loss, SIGReg with 1024 slices, projector with batch normalization), Module, Manager (training loop, checkpointing, SLURM requeue), and GPU augmentation ops. The experiment-specific code is a text view generator, a text encoder wrapper, a next-token loss, and the alignment metrics, a few hundred lines.
Image encoder. ViT-S/16 (timm vit_small_patch16_224, drop path 0.1), CLS embedding. Views: 2 global crops (, scale 0.3–1.0) and 6 local crops (, scale 0.05–0.3), each with horizontal flip, color jitter (0.4, 0.4, 0.2, 0.1; ), grayscale (), Gaussian blur () and solarization (). Crops are made on CPU workers and photometric operations on GPU.
Text encoders. BERT with hidden size 384, 12 layers, 6 heads, MLP 1536, random initialization, bert-base-uncased tokenizer, maximum length 64. The JEPAs use the CLS token. Views are trimmed: global views keep a contiguous span of 50–100% of the caption, local views 15–50%; 15% of kept tokens are replaced by [MASK]. The next-token model uses causal attention, a softmax head tied to the input embeddings, and mean-pooled final hidden states as its embedding; the last token’s hidden state, the other usual choice for a causal model, aligns worse with DINOv2 ViT-B (Flickr R@1 .061 against .119 for mean pooling, three seeds; evaluate.py embed --last-token).
Optimization. AdamW, learning rate with 10 epochs of linear warmup and cosine decay, weight decay 0.05, batch 256, bf16, 200 epochs, seeds 0, 1 and 2 (image and text reseeded together), for the Flickr runs. Longer and larger text runs (BERT-B on Flickr, the Flickr+COCO runs, every Recap run) use learning rate with gradient clipping at 1.0 and one seed unless stated; the MSR-VTT runs use 800 epochs. One GPU per run, except eight the ImageNet and ViT-B image JEPAs and for the 940M-image BERT-B runs (16 for its first seed), and two for the 200M-image BERT-B.
MSR-VTT. The 7,010 clips of the 7k split train, the other 1,990 clips of the 9k split (official test clips outside the JSFUSION split, as is usual with the 9k split) fit the means and the rotation, and the 1,000 clips of the JSFUSION split test. We keep 8 evenly spaced frames per clip (prh.data.extract_frames). Good image views draw each of the 2 global and 6 local crops from its own frame of the clip; bad image views crop one frame, drawn once per clip and epoch. Text views are as on Flickr, with 20 captions per clip. All runs use the Flickr recipe for 800 epochs (21.6k steps, about as many as on Flickr), seeds 0, 1 and 2; evaluation uses the middle frame and, for test clips, the JSFUSION caption.
PRH protocol. For every WIT-1024 score we follow the released code of Huh et al. [22]: WIT-1024 pairs, first caption, per-layer features (vision: CLS after each block; language: mask-weighted mean of each hidden state), elementwise clipping at the 0.95 quantile of absolute values, normalization, mutual 10-NN, maximum over layer pairs.
Additional results
Text encoders alone. Table 5 scores each text encoder without images, and the alignment of projector outputs; Figure 14 and Figure 15 show the alignment of the good-view JEPA qualitatively.
| caption→caption R@1 | mutual 10-NN (projector) | R@1 (projector) | whitened R@1 (projector) | |
| Good-view JEPA | .606 | .064 | .030 | .034 |
| Bad-view JEPA | .381 | .044 | .018 | .022 |
| Next-token prediction | .331 | – | – | – |
| TF-IDF | .442 | – | – | – |
| Random init | .178 | – | – | – |
Table 5. Good views also make better text encoders. Text encoders alone (caption→caption retrieval among the 5,000 Flickr test captions: is the nearest caption one of the four others describing the same image?) and alignment measured on projector outputs (seed 0; –: no trained projector). We clearly observe that the bad-view JEPA learns little beyond word overlap (below TF-IDF), while the good-view JEPA goes well beyond it. The image JEPA reaches 78.5% 20-NN accuracy on Imagenette [21] (29.1% at initialization).
Figure 14. One rotation superimposes two separately trained encoders. Top: joint t-SNE of test images after the best rotation (blue) and their captions (orange), at initialization and after training (good-view JEPA); grey lines join 60 random true pairs. Bottom: independent t-SNEs of the image and text embeddings, colored by caption keywords used only for coloring. We clearly observe that after training, images and their captions share one geometry.
Figure 15. Retrieval through one rotation. Image→caption retrieval (good-view JEPA, epoch 199) for five Flickr test images, with the rank of the true caption among the 1,000 above each list (✓ would mark it among the top three). We observe that misses are mostly semantically close.
Next-token overfitting. On 145k short captions the next-token model’s validation loss is lowest near epoch 29 (3.04) and rises to 4.49 by epoch 189. The objectives also see the captions differently: each next-token epoch reads one caption per image, while each JEPA sample draws its eight views from all five, so next-token prediction sees fewer tokens per step; more passes would not close the gap, since its validation loss is lowest at epoch 29 of 200, and the caption-pair control (Table 1) gives it two captions of the same image per sample. All our tables report the last checkpoint of each run (Table 14 also lists intermediate checkpoints of the 940M-image BERT-B); early stopping it on the validation loss instead (epoch 29 for each seed) gives R@1 .134 against DINOv2 (Table 2), and even selecting its best checkpoint on the test pairs would give at most .146, still far below the good-view JEPA (.246).
| Text JEPA | texts of | global batch | samples seen | WIT (DINOv2-B) | Flickr R@1 |
| BERT-S | 100M images | 256 | 80M | .125† | .402† |
| BERT-B | 200M images | 256 | 89M | .125 | .447 |
| BERT-B | 200M images | 256 | 117M | .126 | .447 |
| BERT-B | 200M images | 256 | 160M | .126 | .451 |
| BERT-B | 940M images | 2,048 | 430M | .182 | .410 |
| BERT-B | 940M images | 2,048 | 580M | .182 | .463 |
| BERT-B | 940M images | 2,048 | 850M | .191 | .468 |
| BERT-B | 940M images | 2,048 | 900M | .191 | .464 |
| BERT-B, seed 1 | 940M images | 2,048 | 900M | .181 | .484 |
| BERT-B, seed 2 | 940M images | 2,048 | 900M | .179 | .485 |
| BERT-B‡ | 940M images | 2,048 | 900M | .121 | .477 |
| BERT-B, next-token, seed 0 | 940M images | 2,048 | 900M | .109 | .346 |
| BERT-B, next-token, seed 1 | 940M images | 2,048 | 900M | .111 | .332 |
| BERT-B, next-token, seed 2 | 940M images | 2,048 | 900M | .112 | .328 |
| BERT-B, bad views | 940M images | 2,048 | 900M | .088 | .318 |
Table 14. Alignment keeps growing with the texts of 940M images. Checkpoints of our largest good-view JEPAs (seed 0 unless named; †: mean of three seeds at the end of training; ‡: another seed with twice the effective SIGReg weight, see text; next-token: the same encoder trained to predict the next token; bad views: two masked copies of one text, at the effective SIGReg weight of the first 940M run), scored on WIT-1024 with DINOv2 ViT-B and on Flickr through one rotation fitted on 30k pairs. We observe that WIT alignment jumps between the texts of 200M and 940M images and rises again by 850M samples, while Flickr retrieval rises more slowly. Bold: best per column.
Nuisance on real encoders. Table 6 gives the measurement of Theorem 3 behind § 3: each Flickr text encoder (epoch 199) embeds all five captions of the 2,014 validation and test images; we keep its top- principal directions and whiten them, estimate as the covariance of the per-image caption means minus a fifth of the mean within-image covariance (the variance the average of five captions keeps), and clip its eigenvalues to . The best-rotation error pairs each image’s first caption, whitened, with DINOv2 ViT-B restricted to its own top- principal directions, whitened, as (evaluate.py nuisance).
Which texts give the gain. Two of each Recap-DataComp image’s three texts are written by an image captioner, so its good views carry information distilled from image–text training. Table 7 separates the texts from the views with one recipe per scale (BERT-S, learning rate , gradient clipping at 1, 64 tokens; 20M samples on the texts of 2.24M images, 40M on 50M; three seeds each). Without any image captioner—the alt-text paired with a paraphrase that a text-only language model (Qwen2.5-7B-Instruct) writes from the alt-text alone—good views reach .060 on WIT, against .107 with the three Recap texts on the same images and .080 with the two LLaVA recaptions alone (.096 against .125 on the texts of 50M images), which retrieve Flickr captions best at both scales (R@1 .348 and .386), plausibly because
Appendix
Flickr captions are literal like LLaVA’s while WIT’s Wikipedia captions read more like alt-texts; on the same alt-texts, good views from the paraphrase still beat bad views of the alt-text (.060 against .055 on WIT, .233 against .180 R@1, no overlap between seeds), so the views help without any image captioner; with the same bad views, the two LLaVA recaptions reach .092 and the alt-text .056 (tables.py). Measured on Flickr captions, the scene-explained fraction of Theorem 3 ranks the nine encoders as WIT does but for one adjacent pair (Spearman .98). That is, the data matter even with bad views, and the views matter with any texts, as Theorem 4 predicts: literal descriptions make positives connect related scenes, and a paraphrase of an alt-text, a meaning-preserving corruption of one text, keeps the alt-text’s own nuisance.
Data kind in the synthetic world. Figure 12 follows bad views of three reaches along the number of samples per state, on literal data (classes apart in the text space) and mixed data (classes adjacent): weak views need many samples, strong ones blur the classes at any sample size, and mixed data cap every bad-view JEPA near .89, while good views align exactly (figures.py theory).
The scaling law in detail. Figure 11 (§ 4.4) varies the positives and the data with one recipe (BERT-S, learning rate $10^{-4}$, clipping 1, 64 tokens, 1M samples per epoch, seed 0): each training item draws good views (two texts of its image) with probability and bad views (two corruptions of one text) otherwise, on the texts of M images for M samples, and on 10.2M images for M at (tables.py). Bad views gain nothing from more images: at , WIT stays within .054–.057 from 0.41M to 50.9M images, while good views rise from .090 to .115. A fraction of good views helps Flickr retrieval before WIT (R@1 .26 at , .28 at q = .3; WIT only from $q = .3), and training longer raises both views (R@1 .221 to .310 from 5M to 80M samples with bad views, .292 to .409 with good views; WIT with good views .078, .099, .124 and .132 at 5M, 20M, 40M and 80M samples). The law
fitted on the 33 cells at 5M and 20M samples finished when its predictions were registered (RMSE .0067), predicted the eight 40M- and 80M-sample cells before they finished with RMSE .0115: within .007 at 40M samples and within .024 at 80M, where it underestimates bad views most (.024 at ), as the bad-view floor rises faster than the fitted on short runs. A simpler law in the good-view samples alone, registered first, underestimated the 80M-sample cells by .019 to .055: bad-view samples are not worthless. Refitted on all 37 cells at 5M and 20M samples (RMSE .0068), its iso-alignment lines run as (, ), so more training substitutes for good views, sub-linearly, and unique images enter weakly (). The exponents are poorly identified—refitted on all 45 grid cells up to 80M samples (v2, Figure 11), the same lines run as with an equally good fit (RMSE .0072)—and the law is an empirical summary of one recipe, not a consequence of Theorem 4. To test it off the grid, we then trained runs at random on 10.2M images— log-uniform on [.001, 1] or zero, drawn on a log scale, from 3M to 133M samples—and three runs at 320M samples, after registering the predictions of both fits for every one of them. On all 85 random runs, v2 predicts R@1 with RMSE .0093: .0065 for , .0071 for , and .012 below, where it underestimates the bad-view floor between 20M and 60M samples (by .013 on average, at most .024). Reruns of six grid cells with two other seeds each put the seed-to-seed standard deviation of one run at .0068 (pooled, 18 runs), so two seeds of one run differ by in RMSE: the law predicts a run it has never seen as well as a rerun of that run with another seed. It fails further out, for good views only: at 320M samples, four times the longest fitted run and 31 passes over the 10.2M images, the runs with good views reach R@1 .418 against .455 predicted at and .348 against .389 at (WIT still rises at , from .132 at 80M samples to .145), while the run without reaches .338 against .379 by the law, which has no term for repeated data, overestimates very long training with good views (Figure 11, diamonds).
Robustness of the good-view recipe. Table 8 gives the numbers behind Figure 7, three seeds per setting where shown, with the best checkpoint of seed 0 for reference (selected on test pairs, so not used in the text). Gains saturate near epoch 100 on Flickr’s 145k captions, and a wider encoder (BERT-B) matches BERT-A as its peak is reached while degrading less afterwards. Collapses come abruptly, after a stretch of normal training: the mask-30% seed that collapses reaches .243 at epoch 49 and chance by epoch 59, and the two 2 + 2-view seeds collapse between epochs 19 and 39 after tracking seed 0 until then. Scored on validation pairs alone instead (one rotation fitted on 507 of the 1,014 validation pairs, R@1 among the other 507; evaluate.py align --val-only --split 507), the same runs collapse, and the two trimming changes remain the two lowest settings with several seeds (.209 and .245, against .271–.316 for every other one), while the other settings, within seed noise of each other, change order (Spearman .40 between validation and test means). That is, no conclusion of Figure 7 rests on selection on test pairs. Outside this sweep, four runs collapsed and feed no table or figure: the default recipe trained for 1,000 epochs (to chance between epochs 139 and 159), our first good and bad runs on web text at learning rate (by epoch 7), and BERT-L on web text, twice, even with gradient clipping (by epochs 3 to 7); hence the lower learning rate and the clipping of every longer or larger run, and no BERT-L beyond Flickr and COCO captions. The run with clipping does not collapse but is partly degenerate (effective rank 33 of 384), although it scores .238.
Few pairs and no pairs. The unpaired refinement alternates optimal transport between the rotated image cloud and the text cloud (entropic, Sinkhorn) and Procrustes on the transport plan, on the 1,014 validation images and captions with their pairing hidden. From 100 pairs, CIDEr rises from .244 to .294; from 30 pairs, the refined captioner reaches .169 (Table 9). With no pairs at all, matching these two small clouds by their internal geometry (Gromov–Wasserstein
[37, 47], then the same refinement) stays at chance (test R@1 ≈ .001); so does the method of Schnaus et al. [54] with this Flickr-caption JEPA (R@1 .012), whose geometry is learned from 145k short captions. With the good-view JEPAs trained on web text, that method aligns the two clouds far above chance (§ 5.2).
Video. Table 11 details the video results of § 5.1: each MSR-VTT clip is embedded by the captioner’s frozen image encoders (DINOv2 ViT-g and DINOv3 ViT-L) on 8 evenly spaced frames, the rotated embeddings are averaged, and the decoder of Table 10 (web-pretrained Qwen2.5-1.5B) decodes the mean (evaluate.py video-embed, video-caption and video-align). Its captions are in the style of image captions and longer than MSR-VTT’s (11.4 against 9.3 words), and miss what the clips’ human captions often describe, actions and speech. The demo’s map, fitted on the 30k Flickr pairs and 200k web image–text pairs, reaches .268.
Alignment without any pair, in detail. Table 4 runs the code of Schnaus et al. [54] unmodified (Wasserstein Procrustes with clusters, restarts, points, refinement rounds, the MPOpt solver, their five seeds) on their stored DINOv2 ViT-B/14 mean-token features, with each image’s captions averaged into one text embedding, as they do; each side is centered by the mean of its own unpaired training half, and no pair, val2014 image or test label enters the fit (evaluate.py pairfree, prh/pairfree.py). Means ± std over training seeds times method seeds: FOSCTTM .018 ± .020 (median .010), CIFAR-10 86.0 ± 2.3, CIFAR-100 67.2 ± 4.4 and ImageNet-100 64.4 ± 3.6 for the 109M good-view JEPA (15 runs); .036 ± .034, 83.2 ± 7.4, 60.9 ± 8.2 and 57.9 ± 8.8 for the 33M one (15); .018 ± .002, 83.4 ± 2.9, 61.7 ± 1.4 and 47.6 ± 2.2 for Qwen3-8B with generative pooling (5); .006 ± .001, 84.0 ± 5.2, 67.0 ± 1.4 and 68.7 ± 1.8 for CLIP’s text encoder (5); and .287 ± .162 and 25.5 ± 16.0 on CIFAR-10 for next-token prediction (15, half of the runs above FOSCTTM .25). Against Qwen3-8B rerun, the 109M good-view JEPA is level on FOSCTTM () and CIFAR-10 () and ahead on CIFAR-100 () and ImageNet-100 (; Mann–Whitney); against CLIP’s text encoder it is level on both CIFARs and on FOSCTTM () and behind on ImageNet-100 (). These tests treat the 15 runs as independent; at the level of the three training seeds, the separation from Qwen3-8B on CIFAR-100 and ImageNet-100 is complete. Their third pair,
Qwen3-Embedding-8B, reproduces their FOSCTTM but not their CIFAR-10 (64 against 74), for reasons we could not trace; their all-mpnet-base-v2 numbers reproduce on all four metrics. Their best FOSCTTM on these data is .02, with DINOv2 ViT-g, and .01 in two of their other settings (COCO images with paragraphs from SPC, and SPC alone); their best zero-shot accuracies over all five settings are 82%, 62% and 56% (paper/data/schn aus2026_reported.csv). An independent audit recomputed 35 fitted maps from scratch, refitted one bit for bit, and checked that shuffling each text coordinate across the caption pool, which keeps every marginal and destroys the geometry, drops the method to chance (FOSCTTM .57, CIFAR-10 11–15%). Failures come from the initialization: the 30 cluster centers matched by CKA are mostly wrong (13–24% correct), and a seed whose averaged initialization points away from the paired map does not recover; the method’s own objective on the unpaired pools is anti-correlated with test FOSCTTM (Spearman −.78 over 85 COCO runs, −.80 over 80 Flickr runs) and flags such seeds. On Flickr (DINOv2 ViT-B CLS features, first captions, disjoint halves of the 30,014 non-test pairs), zero-pair R@1 is .164 ± .104 for the 109M good-view JEPA (15 runs; R@5 .349), .091 ± .052 for the 33M one (15), and at most .003 for every encoder that is not a good-view JEPA (.074 for the 50M-image good-view JEPA and .012 for the one trained on Flickr captions); given 10 known pairs, the same method reaches R@1 .32 with the good-view JEPA (one seed), against about .01 for Procrustes on those 10 pairs and .067 for CLIP’s text encoder. Two disclosures: the first training seed of our 109M good-view JEPA ran on twice as many GPUs as the other two, with the SIGReg weight scaled to keep its effective value; and its positives are texts of the same web image, two of three of which are LLaVA recaptions, so that, although the text encoder never sees a pixel or a COCO pair, its views are grouped by image and partly written by a vision–language model (Table 7).
Decoder noise. The captioner’s training noise is set by matching cosines (§ 5.1), with the mean cosine between a rotated image embedding and its caption’s embedding. We first measured it on test pairs; held-out validation pairs (a rotation fitted on one half of the 1,014 validation pairs, the cosine taken on the other half) give nearly the same noise (column cos of evaluate.py align, with --val-only --split 507 for held-out validation pairs). With the rotation fitted on 1,014 pairs ( held out, ; on test pairs, ), Flickr test CIDEr is .225, .304, .368, .396 and .377 for : lower noise decodes the true caption embedding better but transfers worse, and transfer peaks where predicted. The 30k-pair rotation leaves a smaller residual ( held out, 2.3 on test pairs) and gives .438 at against .431 at . Differences below about .01 CIDEr are within the variation of greedy decoding across hardware (floating-point ties), and we do not interpret them. For COCO, the decoder is trained with on the embeddings of a text JEPA trained on Flickr and COCO training captions (Karpathy train and restval splits, disjoint from its test split), and the rotation is fitted on the 30k Flickr pairs.
Scaling the captioner. The language-model decoder maps the centered, unit-norm text-JEPA embedding to 10 input embeddings of Qwen2.5 through a two-layer MLP (a soft prompt, as in ClipCap, [40]) and is fine-tuned entirely (AdamW, learning rate , batch 256, gradient clipping 1.0, bf16) on 1.4M draws of Flickr and COCO training captions (Karpathy train and restval), each embedded by the frozen text JEPA with noise , matched to the cosine between rotated DINOv2-g embeddings and their captions. The target is the caption as written, at most 64 tokens. Two image encoders are concatenated after rescaling each embedding to unit root-mean-square per coordinate (Table 12); otherwise the encoder with larger features dominates. Consensus decoding (minimum Bayes risk, [27]) samples 32 captions besides the greedy one (temperature 0.7, top- 0.95) and keeps, of these 33, the one most similar to the other 32 under CIDEr’s n-gram similarity without document frequencies, which one image’s candidates cannot estimate; it uses no reference caption and no other image. With 16 samples it gains little, and 64 samples gain no more than 32 on Flickr. Keeping instead the beam candidate whose text-JEPA embedding is closest to the rotated image embedding does not help. Pretraining the decoder on 4M first sentences of Recap recaptions before the captions lets it decode true caption embeddings better (.743 and 1.369) but transfers no better on Flickr and COCO; on photos of other kinds its captions are more often right (coats of arms, a cake shaped like a sheep), and the demo uses it. Two other changes do not help: a higher input resolution raises retrieval (R@1 .440 to .462 with DINOv2-g at 448 px) but not the cosine the decoder sees, and a per-direction gain after the rotation, fitted on the same pairs, at best raises Flickr CIDEr by .017 and lowers COCO’s by .053 (web-pretrained decoder, beam 3).
Table 12. Linear-evaluation features and concatenated encoders align best. Flickr test pairs, text JEPAs trained on Recap texts only (seed 0), each against a frozen self-supervised image encoder through one map with orthonormal rows fitted on the 30k Flickr train and validation pairs (chance: mutual 10-NN .01, R@1 .001). Bottom rows: the CLS + mean patch features of two encoders, each rescaled to unit root-mean-square per coordinate, concatenated. No network is trained on pairs. Notably, with DINOv3 ViT-7B, one rotation retrieves the true caption first for more than half of the test images.
| Frozen image encoder | features | BERT-B, texts of 50M images mutual 10-NN | BERT-B, texts of 50M images R@1 | BERT-B, texts of 50M images R@5 | BERT-S, 100M R@1 | BERT-S, 100M R@5 |
| DINOv2 ViT-B/14 | CLS | .234 | .412 | .733 | .400 | .711 |
| DINOv2 ViT-L/14 | CLS | .242 | .437 | .745 | .410 | .732 |
| DINOv2 ViT-g/14 | CLS | .245 | .437 | .769 | .425 | .741 |
| DINOv2 ViT-g/14 | CLS + mean patch | .250 | .460 | .784 | .440 | .763 |
| DINOv3 ViT-L/16 | CLS | .209 | .436 | .722 | .414 | .703 |
| DINOv3 ViT-L/16 | CLS + mean patch | .234 | .514 | .792 | .489 | .780 |
| DINOv3 ViT-H+/16 | CLS | .181 | .353 | .625 | .342 | .607 |
| DINOv3 ViT-H+/16 | CLS + mean patch | .228 | .479 | .779 | .486 | .775 |
| DINOv3 ViT-7B/16 | CLS | .207 | .455 | .752 | .452 | .730 |
| DINOv3 ViT-7B/16 | CLS + mean patch | .231 | .531 | .803 | .523 | .798 |
| I-JEPA ViT-H/14 (IN-22k) | mean patch | .109 | .188 | .451 | .183 | .436 |
| I-JEPA ViT-g/16 (IN-22k) | mean patch | .140 | .246 | .502 | .234 | .491 |
| DINOv2-g + DINOv3-L | both, concatenated | .255 | .505 | .806 | .488 | .791 |
| DINOv2-g + DINOv3-7B | both, concatenated | .256 | .535 | .824 | .511 | .801 |
Table 12.
Projector outputs. On projector outputs, where SIGReg acts, whitening each side before the rotation barely changes retrieval (R@1 .030 vs. .034), as expected once the embedding is nearly isotropic.
Transfer to COCO. The rotation fitted on the 30k Flickr pairs (text JEPA trained on Flickr and COCO captions, BERT-S) ranks the true caption first for 11.3% of the 5,000 COCO Karpathy test images (chance 0.02%; mutual 10-NN .185), with each side centered by its own unpaired COCO mean; a rotation fitted on 2,500 COCO pairs and tested on the other 2,500 reaches 15.8% (chance 0.04%).
Frozen image encoders. Table 12 pairs our two strongest text JEPAs, BERT-B on the texts of 50M images and BERT-S on those of 100M, with other frozen self-supervised image encoders through the same 30k-pair rotation. Within DINOv2, larger encoders align better, and so do linear-evaluation features, the CLS token concatenated with the mean patch token. With DINOv3 ViT-7B, the true caption is retrieved first for 53.1% of Flickr test images, through the rotation alone. Retrieval through the rotation and agreement of neighborhoods, which fits nothing, need not rank encoders alike: DINOv2 ViT-g has the most similar neighborhoods of single encoders and also gives their best captions (CIDEr .480 on Flickr and .756 on COCO with its CLS and mean patch token, against .468 and .746 for DINOv3 ViT-L and .448 and .652 for DINOv3 ViT-7B; Qwen2.5-1.5B decoder, beam 3). Concatenating two encoders helps both, and DINOv2 ViT-g with DINOv3 ViT-L gives the best captions for its cost (Table 10). I-JEPA, whose positives are blocks of one image and which saw only ImageNet-22k, aligns far less.
Text-only quality. Good views also make better text encoders on their own. On the STS-Benchmark test set [10] (Spearman correlation between the cosine similarity of two sentences’ embeddings and human similarity, no fine-tuning), the good-view JEPA, the bad-view JEPA and next-token prediction score .579 / .528 / .430 when trained on Flickr captions and .612 / .501 / .377 when trained on 2M Recap images, and the 50M-image good-view JEPA reaches .632, above BERT-base (.551) and Qwen2.5-0.5B (.352, mean-pooled), though below all-mpnet-base-v2 (.834), which is trained on a billion supervised sentence pairs. The ordering good > bad > next-token is the same as for alignment on Flickr (on WIT, next-token prediction is above the bad-view JEPA).
Uncertainty on WIT. WIT-1024 is small. We compute each score on 200 random 800-pair subsets drawn without replacement (the same subsets for every model; each model at its best layers on all pairs; DINOv2 ViT-B) and turn their spread into a 95% interval for the 1,024-pair score: for average-like statistics, the spread of 800-pair subsets around the full score is times the full score’s standard error, so we rescale it by (prh.metrics.subsample_interval). Against language models our text JEPAs are level: the 50M-image JEPA minus Qwen2.5-7B is +.010 ([-.005, +.025]) and minus Qwen2.5-3B +.003 ([-.011, +.018]), the 100M-image JEPA minus Mistral-7B −.011 ([−.026, +.004]), and the 2M-image JEPA minus Qwen2.5-0.5B +.005 ([−.008, +.018]). Between views the differences are clear: on 2M images, the good-view JEPA minus next-token prediction is +.037 ([+.025, +.049]) and minus the bad-view JEPA +.054 ([+.041, +.067]); on 50M images, +.050 ([+.037, +.062]) and +.063 ([+.050, +.075]). Training seeds matter as much as test sampling: the three seeds of the 50M-image JEPA score .129, .118 and .127 with DINOv2 ViT-B (seed 0 is used above), so the comparisons with language models are within noise, while the gaps between views, at least +.040 over next-token prediction and +.056 over the bad-view JEPA for every seed, are far outside it.
Image JEPAs trained from scratch. Behind Figure 9 (right): a ViT-S on the 2.16M Recap images (10 epochs), a ViT-B on 10M Recap images (8 epochs) and a ViT-S on ImageNet-1k (300 epochs), all trained without texts. The ImageNet ViT-S, which sees the most samples, agrees most with DINOv2 (.407, against .315 and .317) and with the 50M-image text JEPA (.095 over three text seeds, against .079 and .080), and its alignment with the 109M good-view JEPA reaches .130 (Table 15); two more ImageNet ViT-S trained from other seeds reproduce it (.414 and .405 with DINOv2; with the good-view, next-token and bad-view text JEPAs .098, .061 and .055, and .100, .062 and .053, against .098, .062 and .055). With it, two JEPAs trained separately from scratch agree as much as the image JEPA agrees with the strongest language models: for seed 0, our text JEPA minus Mistral-7B is −.003 ([−.016, +.011]) and minus Qwen2.5-7B +.011 ([−.001, +.024]), while it beats next-token prediction on the same texts by +.037 ([+.026, +.048]; intervals as in the previous paragraph).
| Vision | Qwen2.5 0.5B | Qwen2.5 1.5B | Qwen2.5 3B | Qwen2.5 7B | Mistral 7B | ours (33M), on the texts of Recap images 2.16M good | ours (33M), on the texts of Recap images 2.16M bad | ours (33M), on the texts of Recap images 2.16M nt | ours (33M), on the texts of Recap images 10M good | ours (33M), on the texts of Recap images 50M good / bad / nt | ours (33M), on the texts of Recap images 100M good | ours (109M) 940M good |
| DINOv2 ViT-S/14 | .092 | .101 | .110 | .110 | .126 | .099 / .052 / .065 | .102 | .114 | .062 / .052 / .074 | .114 | .166 | |
| DINOv2 ViT-B/14 | .102 | .108 | .125 | .118 | .140 | .108 / .053 / .071 | .111 | .125 | .063 / .079 | .125 | .184 | |
| DINOv2 ViT-L/14 | .114 | .123 | .139 | .137 | .157 | .111 / .057 / .075 | .115 | .130 / .068 / .085 | .131 | .196 | ||
| DINOv2 ViT-g/14 | .115 | .124 | .141 | .133 | .157 | .114 / .059 / .076 | .117 | .132 / .070 / .086 | .133 | .197 | ||
| ours, ViT-S, Recap 2M | .062 | .068 | .078 | .073 | .086 | .070 / .042 / .053 | .071 | .079 / .048 / .057 | .080 | .106 | ||
| ours, ViT-B, Recap 10M | .064 | .070 | .079 | .075 | .087 | .071 / .043 / .054 | .072 | .080 / .048 / .058 | .081 | .106 | ||
| ours, ViT-S, ImageNet | .074 | .078 | .091 | .087 | .101 | .082 / .046 / .057 | .085 | .095 / .053 / .062 | .094 | .130 | ||
| ours, ViT-S, Flickr | .050 | .051 | .057 | .055 | .063 | .055 / .034 / .043 | .055 | .060 / .038 / .047 | .060 | .078 |
Table 15. A 109M text JEPA beats language models on the PRH benchmark. PRH protocol of Huh et al. [22]: WIT-1024 pairs, per-layer features, clipping, normalization, maximum over layer pairs. For reference, DINOv2 ViT-B × BERT-base scores .111, all-mpnet-base-v2 .118 and CLIP’s text encoder .124 (.113–.137 across DINOv2 sizes for these two); our Flickr-trained text encoders (good / bad / next-token) score .029–.036 with every DINOv2 size and .023–.024 with our Flickr image JEPA; and our Flickr image JEPA × DINOv2 ViT … .net: next-token prediction. Our 33M text JEPAs’ entries are means of three seeds, which differ by at most ±.009 with DINOv2 ViT-B; the 109M column is the mean of three seeds (each above every language model). At least 43 of the 1,024 WIT captions appear among the Recap-DataComp alt-texts; on the other 981 pairs, none of the nine text models we checked with DINOv2 ViT-B (ours, Qwen2.5-3B/7B, Mistral-7B) changes by more than .002. This demonstrates that the 109M good-view JEPA surpasses every language model we evaluated with every vision encoder it was scored with, including our image JEPAs trained from scratch.
Scaling to the texts of 940M images. Table 14 follows our largest good-view JEPAs along training. With the same BERT-B encoder, the texts of 200M images and a global batch of 256 reach .125–.126 on WIT after 89M–160M samples (the end of training), while the texts of all 940M images and a global batch of 2,048 reach .182 after 430M samples and .191 after 850M and at the end of training (900M). These runs change unique texts, samples seen and batch size together, and we do not separate their effects. The SIGReg weight also differs: in stable-pretraining as we ran it, the Epps–Pulley statistic is computed on each GPU’s batch and scaled by its size, so its effective weight grows with the per-GPU batch, and the first 940M run (16 GPUs of 128) trains with the equivalent of on a single GPU, half the weight of our single-GPU runs, as do the 200M-image BERT-B (2 GPUs of 128) and the ImageNet and ViT-B image JEPAs (8 GPUs of 128). A second seed on 8 GPUs of 256, i.e., at , reaches .121 on WIT and .477 Flickr R@1: at this scale WIT is sensitive to the SIGReg weight, which our ablations on Flickr captions did not reveal. We chose the weight of the first run for every other 940M run after seeing this result, so that weight was selected on WIT: at on 8 GPUs of 256, the effective weight of the first run, two more seeds reach .181 and .179 on WIT and .484 and .485 Flickr R@1 (three seeds: .184 and .478), the bad-view control .088 and .318, and three next-token seeds .109, .111 and .112 on WIT and .346, .332 and .328 Flickr R@1. Across DINOv2 sizes, the three good-view seeds average .166–.197 (Table 15), the bad-view control .082–.097 and next-token prediction .101–.122. Averaged over all four good-view seeds, WIT is .168 with DINOv2 ViT-B, above every language model, and .198, .202 and .212 with CLIP ViT-B, -L and -H, level with the best language model (.193, .203 and .210).
WIT captions in the training texts. DataComp is crawled from the web, so some WIT captions may appear among the Recap-DataComp alt-texts. We lower-case each WIT caption and each of the 940M alt-texts, keep letters, digits and spaces, and match them exactly or by their first 40 characters: 26 WIT captions match exactly and 17 more by prefix, at least 43 of 1,024 in all (captions shorter than 20 characters are not checked; 16 within the texts of the first 100M images, 20 within the first 200M); 3.3M of the 940M images are hosted on Wikimedia. Removing the 43 pairs changes none of the nine text models we checked with DINOv2 ViT-B by more than .002 (Section 4), and the 109M model scores .193 on the 981 remaining pairs. The WIT-1024 release does not identify its images, so we cannot match them to DataComp images. Flickr30k images can be: their file names are Flickr photo ids, which Flickr-hosted DataComp URLs contain, and at least 178 of the 31,014 Flickr30k photos appear among the 940M Recap-DataComp images (166 train, 6 validation and 6 of the 1,000 test photos, each with its web alt-text and recaptions, never its Flickr captions), so these can move Flickr test retrieval by at most .006 (python -m prh.data overlap). COCO photos, also hosted on Flickr, match the same way: at least 1,207 COCO 2014 photos are among the 940M images, 393 of them from val2014, where Table 4 scores, and 50 of the 5,000 COCO test photos of Table 17, again with their web texts and never their COCO captions; removing these 50 changes the COCO score of the 109M JEPA, and of every baseline we checked, by at most .001 (evaluate.py wit -bench coco -drop).
WIT, PRH protocol. Table 15 applies the exact protocol of Huh et al. [22] on their WIT-1024 pairs, which none of our models is trained on (DataComp images may overlap with Wikipedia’s). With DINOv2, pretrained language models reach .09–.16; alignment mostly grows with the size of the vision model and, for Qwen2.5, with the language model up to 3B, the trend reported by Huh et al. [22]. Our Flickr-trained text encoders, trained on 145k short captions, do not transfer to Wikipedia captions (.029–.037 for every objective, three seeds each, including next-token prediction on caption pairs, and every DINOv2 size; best text layer among the first few, mostly the embeddings). Trained on the texts of 2.16M Recap-DataComp images instead, the same recipe reaches .099–.114 depending on the DINOv2 size (means of three seeds), with its last layer as the best, level with Qwen2.5-0.5B.
A second alignment metric. Mutual 10-NN only counts shared neighbors. We also score every layer pair with linear CKA and the unbiased HSIC estimator [58], as in the released code of Huh et al. [22], with the same features, preprocessing and maximum over layer pairs (prh.metrics.unbiased_cka). With DINOv2 ViT-B, the 109M good-view JEPA reaches .507 (mean of three seeds, .495–.518), against .390 for all-mpnet-base-v2, .363 for Qwen2.5-3B, .353 for Qwen2.5-7B, .349 for Mistral-7B and .339 for CLIP’s text encoder; on the texts of 50M images, the good, bad-view and next-token JEPAs score .385, .192 and .261 (seed 0), and on those of 2.16M images .301, .185 and .225. Across the 16 text models we scored, the two metrics rank them alike (Spearman .88 with DINOv2 ViT-B and .86 with ViT-L). That is, the ranking of views and the lead of the 109M good-view JEPA do not depend on the alignment metric.
Larger language models. Table 16 extends the comparison to Qwen2.5 up to 72B parameters and Llama-3 at 8B and 70B (base models, the same per-layer protocol; the 14B and larger ones split across two GPUs in bfloat16). Alignment grows little with size beyond a few billion parameters–Qwen2.5 reaches .118–.130 from 7B to 32B with DINOv2 ViT-B, and .154 at 72B–and no language model reaches the 109M good-view JEPA with any DINOv2 size, either metric or either benchmark. Llama-3-8B is the language model of the image captioner that wrote Recap-DataComp’s recaptions; it scores .145.
A second benchmark. WIT-1024 is small, so we repeat the protocol of Huh et al. [22] (per-layer features, clipping, normalization, maximum over layer pairs) on the 5,000 COCO Karpathy test images and their first captions (evaluate.py wit-embed –bench coco), whose captions no text encoder of ours is trained on (Table 17; 50 of its photos are among the 940M web images, with their web texts, and removing them moves every text encoder we checked by at most .001). The ranking is that of WIT: the 109M good-view JEPA leads every text encoder with both DINOv2 sizes and both metrics, including all-mpnet-base-v2, whose billion training pairs include COCO captions, and CLIP’s text encoder, and at fixed data the good-view JEPA is ahead of next-token prediction and the bad-view JEPA.
Other vision families. Table 18 repeats the WIT protocol with frozen vision encoders trained by pixel reconstruction (MAE, [20]) and by latent prediction from image blocks (I-JEPA, [2]; its per-layer feature is the mean token, as it has no CLS token). The ranking of the text side does not depend on the vision encoder: with each of the six, the good-view text JEPA beats next-token prediction and the bad-view JEPA trained on the same texts, and the 109M good-view JEPA beats every language model.
References
- [1]David Alvarez-Melis and Tommi Jaakkola. Gromov-Wasserstein alignment of word embedding spaces. In Conference on Empirical Methods in Natural Language Processing, 2018.
- [2]Mahmoud Assran, Quentin Duval, Ishhan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.arxiv.org/abs/2301.08243
- [3]Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2205.11508
- [4]Randall Balestriero and Yann LeCun. LeJEPA: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025.arxiv.org/abs/2511.08544
- [5]Randall Balestriero, Nicolas Ballas, Mike Rabbat, and Yann LeCun. Gaussian embeddings: How JEPAs secretly learn your data density. arXiv preprint arXiv:2510.05949, 2025.arxiv.org/abs/2510.05949
- [6]Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, 2022.arxiv.org/abs/2105.04906
- [7]Horace B. Barlow. Possible principles underlying the transformation of sensory messages. In Walter A. Rosenblith, editor, Sensory Communication, pages 217–234. MIT Press, 1961.
- [8]Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6):1373–1396, 2003.DOI
- [9]Vivien Cabannes, Bobak T. Kiani, Randall Balestriero, Yann LeCun, and Alberto Bietti. The SSL interplay: Augmentations, inductive bias, and generalization. In International Conference on Machine Learning, 2023.arxiv.org/abs/2302.02774
- [10]Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In International Workshop on Semantic Evaluation, 2017.DOI
- [11]Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference on Learning Representations, 2018.
- [12]Imant Daunhawer, Alice Bizuel, Emanuele Palumbo, Alexander Marx, and Julia E. Vogt. Identifiability results for multimodal contrastive learning. In International Conference on Learning Representations, 2023.arxiv.org/abs/2303.09166
- [13]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics, 2019.arxiv.org/abs/1810.04805
- [14]Darshil Doshi, Wenjie Zhou, Corinna Elena Wegner, Daniel J. Korchinski, Santiago Acevedo, and Matthieu Wyart. A theory of platonic representations in language models. arXiv preprint arXiv:2610.07168, 2026.arxiv.org/abs/2610.07168
- [15]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
- [16]Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In International Conference on Machine Learning, 2021.arxiv.org/abs/2007.06346
- [17]Ky Fan. On a theorem of Weyl concerning eigenvalues of linear transformations. Proceedings of the National Academy of Sciences, 35(11):652–655, 1949.
- [18]Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. Unsupervised image captioning. In Conference on Computer Vision and Pattern Recognition, 2019.arxiv.org/abs/1811.10787
- [19]Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2106.04156
- [20]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.arxiv.org/abs/2111.06377
- [21]Jeremy Howard. Imagenette. https://github.com/fastai/imagenette, 2019.
- [22]Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. In International Conference on Machine Learning, 2024.arxiv.org/abs/2405.07987
- [23]Rishi Jha, Collin Zhang, Vitaly Shmatikov, and John X. Morris. Harnessing the universal geometry of embeddings. arXiv preprint arXiv:2505.12540, 2025.arxiv.org/abs/2505.12540
- [24]Daniel D. Johnson, Ayoub El Hanchi, and Chris J. Maddison. Contrastive learning can find an optimal basis for approximately view-invariant functions. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.01883
- [25]Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
- [26]Bobak T. Kiani, Randall Balestriero, Yubei Chen, Seth Lloyd, and Yann LeCun. Joint embedding self-supervised learning in the kernel regime. arXiv preprint arXiv:2209.14884, 2022.arxiv.org/abs/2209.14884
- [27]Shankar Kumar and William Byrne. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, 2004.
- [28]Iro Laina, Christian Rupprecht, and Nassir Navab. Towards unsupervised image captioning with shared multimodal embeddings. In International Conference on Computer Vision, 2019.arxiv.org/abs/1908.09317
- [29]Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations, 2018.arxiv.org/abs/1711.00043
- [30]Yann LeCun. A path towards autonomous machine intelligence. OpenReview, 2022.
- [31]Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems, 2014.
- [32]Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. DeCap: Decoding CLIP latents for zero-shot captioning via text-only training. In International Conference on Learning Representations, 2023.arxiv.org/abs/2303.03032
- [33]Wei Li, Hehe Fan, Yongkang Wong, Mohan Kankanhalli, and Yi Yang. TOPA: Extending large language models for video understanding via text-only pre-alignment. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2405.13911
- [34]Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, Yuyin Zhou, and Cihang Xie. What if we recaption billions of web images with LLaMA-3? arXiv preprint arXiv:2406.08478, 2024.arxiv.org/abs/2406.08478
- [35]Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning. Neurocomputing, 508:293–304, 2022.DOI
- [36]Mayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Mohamed El Amine Seddik, Sanath Narayan, Karttikeya Mangalam, and Noel E. O’Connor. Do vision and language encoders represent the world similarly? In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.arxiv.org/abs/2401.05224
- [37]Facundo Mémoli. Gromov–Wasserstein distances and the metric approach to object matching. Foundations of Computational Mathematics, 11(4):417–487, 2011.DOI
- [38]Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. In International Conference on Learning Representations, 2023.arxiv.org/abs/2209.15162
- [39]Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168, 2013.arxiv.org/abs/1309.4168
- [40]Ron Mokady, Amir Hertz, and Amit H. Bermano. ClipCap: CLIP prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021.arxiv.org/abs/2111.09734
- [41]Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In International Conference on Learning Representations, 2023.arxiv.org/abs/2209.15430
- [42]Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, and Francesco Locatello. ASIF: Coupled data turns unimodal models to multimodal without training. In Advances in Neural Information Processing Systems, 2023.
- [43]David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise-injected CLIP. In Findings of the Association for Computational Linguistics: EMNLP, 2022.
- [44]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2304.07193
- [45]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics, 2002.
- [46]Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117, 2020.DOI
- [47]Gabriel Peyré, Marco Cuturi, and Justin Solomon. Gromov–Wasserstein averaging of kernel and distance matrices. In International Conference on Machine Learning, 2016.
- [48]Plato. The Republic. Translated by G. M. A. Grube, revised by C. D. C. Reeve, Hackett, 1992. Book VII, c. 375 BC.
- [49]Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
- [50]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.arxiv.org/abs/2103.00020
- [51]Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Conference on Empirical Methods in Natural Language Processing, 2019.arxiv.org/abs/1908.10084
- [52]Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. In International Conference on Machine Learning, 2022.arxiv.org/abs/2202.14037
- [53]Dominik Schnaus, Nikita Araslanov, and Daniel Cremers. It’s a (blind) match! Towards vision-language correspondence without parallel data. In Conference on Computer Vision and Pattern Recognition, 2025.arxiv.org/abs/2503.24129
- [54]Dominik Schnaus, Thomas Dagès, Daniel Cremers, Xi Wang, and Philipp Isola. Shared geometry as a rosetta stone: Cross-modal alignment without paired data. arXiv preprint arXiv:2610.09411, 2026.arxiv.org/abs/2610.09411
- [55]Peter H. Schönenmann. A generalized solution of the orthogonal Procrustes problem. Psychometrika, 31(1):1–10, 1966.DOI
- [56]Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. DINOv3. arXiv preprint arXiv:2508.10104, 2025.arxiv.org/abs/2508.10104
- [57]Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MPNet: Masked and permuted pre-training for language understanding. In Advances in Neural Information Processing Systems, 2020.arxiv.org/abs/2004.09297
- [58]Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, 13:1393–1434, 2012.
- [59]Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. Language models can see: Plugging visual controls in text generation. arXiv preprint arXiv:2205.02655, 2022.arxiv.org/abs/2205.02655
- [60]Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. ZeroCap: Zero-shot image-to-text generation for visual-semantic arithmetic. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
- [61]Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In International Conference on Algorithmic Learning Theory, 2021.arxiv.org/abs/2008.10150
- [62]Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.arxiv.org/abs/1411.5726
- [63]Hermann von Helmholtz. Handbuch der physiologischen Optik. Leopold Voss, Leipzig, 1867.
- [64]Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2106.04619
- [65]Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, 2020.arxiv.org/abs/2005.10242
- [66]Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference on Learning Representations, 2022.arxiv.org/abs/2203.13457
- [67]Robert Wu and Vardan Papyan. Linguistic collapse: Neural collapse in (large) language models. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2405.17767
- [68]Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
- [69]Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. VideoCoCa: Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04979, 2022.arxiv.org/abs/2212.04979
- [70]Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014.
- [71]Yi Yu, Tengyao Wang, and Richard J. Samworth. A useful variant of the Davis–Kahan theorem for statisticians. Biometrika, 102(2):315–323, 2015.arxiv.org/abs/1405.0680
- [72]Jure Zbontar, Li Jing, Ishaan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, 2021.arxiv.org/abs/2103.03230
- [73]Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2210.08344
- [74]Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, 2021.arxiv.org/abs/2102.08850
- [75]Liu Ziyin and Isaac Chuang. Proof of a perfect platonic representation hypothesis. arXiv preprint arXiv:2507.01098, 2025.arxiv.org/abs/2507.01098













