betaFeedback

WorksComputer Science

V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations

Abstract

World-model video encoders such as V-JEPA learn how scenes change (direction, speed, order, state) from video alone. To answer questions, they are joined to a language model trained on captions and question-answer pairs. But each caption or question names only a few attributes of a clip, so the model learns to retrieve those attributes rather than to read the clip, and the dynamics the encoder carries go unused: on the same items, a linear probe reads speed from frozen V-JEPA features at 87% while the VLM answers at chance. We show where dynamics lives in a frozen encoder: in a subspace shared across videos for predictive encoders such as V-JEPA, but not as a shared component of frame-wise, text-aligned encoders such as SigLIP. We then build counterfactual supervision that makes the language model read it: pixel transformations turn public videos into answer-balanced variants whose answers are set by the transformation and differ only in the change. Trained in a single stage on 4.53M public samples with 32 visual tokens per frame, where open video LLMs use several stages, 9M to 172M samples and 64 to 256 tokens per frame on average, the resulting V-JEPA-LM models (1B to 8B, Qwen3 and Llama) lead every size class in average score across eleven video benchmarks; our 4B model scores 72.1, against 61.1 for the best open 4B and 64.8 for the best open 8B model. A controlled encoder x supervision study shows that the encoder decides what can be read and the supervision what is read: at a matched budget, counterfactual supervision adds 4.5 points on average and 38 on the speed questions of natural benchmarks. Code and models: https://github.com/kkt94/Llama-V-JEPA-LM.

Introduction

Self-supervised video encoders trained to predict masked spatiotemporal regions in latent space, the V-JEPA family [3, 4, 6, 46], learn how the physical world moves from video alone, without a single caption: they are world models whose features support motion understanding, anticipation and planning. To answer in language, however, V-JEPA 2 and 2.1 are turned into video LLMs with the recipe of image–text VLMs [12, 27, 41, 49]: alignment, pre-training and instruction stages on captions and question–answer pairs, 88.5M and 72.5M samples, and 288 tokens per frame for V-JEPA 2 [4, 46]. The model built on a world-model encoder is thus aligned to text once more, and only the encoders are released, so how a language model can read what such an encoder knows remains open.

Scale does not make the model read dynamics. Each caption or question names a few attributes of its clip, and an answer that matches them lowers the loss, so the model learns to fetch what the text asks about rather than to read the clip, and most such items can be answered from one frame [33]. On held-out speed-ratio items, a linear probe reads the answer from frozen V-JEPA features at 87% and from the bridge of a VLM trained on the caption and QA mixture at 84%, yet that VLM answers at chance (§ 3.2). A stronger encoder does not fix this: in Meta’s controlled comparison, V-JEPA 2.1 lowers TemporalBench and TOMATO relative to V-JEPA 2 under the same recipe [46]. What is missing is supervision that requires the change.

We therefore design the supervision from where the dynamics lives in the encoder (§ 3). For a frame-wise, text-aligned encoder such as SigLIP, time reversal merely reorders the tokens, so the arrow of time is in no token’s content. In V-JEPA, reversing, mirroring or re-timing a clip moves its tokens along a change component that points the same way across videos, 20 to 100 times more consistently than in SigLIP; a text-aligned video encoder with attention across frames behaves like SigLIP. When a clip and a transformed copy are asked the same question and their answers differ by construction, only the change can tell them apart, so the content they share cannot separate the answers (Figure 1). Fourteen such constructions in six families are rendered from public videos by pixel transformations, without new annotation; adding a video collection takes a short script that names its clips and the constructions to apply (§ 4).

Figure 1. Top: what captions and QA teach (left), where the dynamics lives in the encoder (middle), and the supervision built from it (right). Bottom: the model, trained in one stage, and what the recipe changes (before → after: the V-JEPA 2 video LLM [4]; accuracy: the best open 4B model).

Far less data, better models. V-JEPA-LM trains in a single stage on 4.53M public samples with 32 visual tokens per frame, where open video LLMs use several stages, 9M to 172M samples and 64 to 256 tokens per frame on average; for every model we count the vision–language training that turns a pretrained vision encoder and a pretrained language model into the VLM (Figure 3). With 2 to 38 times less data it leads every size class on eleven video benchmarks, and its 8B models lead Meta’s V-JEPA video LLMs with a twentieth of the data. At a matched budget, counterfactual supervision adds 4.5 points to the 2B model’s average and 38 on speed questions, while annotated temporal questions or the arrow of time at the same weight do not.

Figure 3. Twelve-score average against training samples as reported (log scale). Marker area: visual tokens per clip. Sample counts: Sec. C.

Our contributions are as follows.

  • Where dynamics lives: a measured analysis separating frame-wise text-aligned from predictive encoders, which tells what supervision a predictive encoder can use.

  • Counterfactual supervision and an extensible generator: answers fixed by pixel transformations of public videos, so content cancels out; any public video collection becomes training data with a short script and no annotation.

  • Data-efficient video LLMs: V-JEPA-LM, 1B to 8B on Qwen3 and Llama, one stage, 4.53M samples, first in every size class; weights and inference code are released, and training code, data generator and manifests will follow.

  • A controlled encoder × supervision study with matched-weight controls, held-out generalization and internal measurements.

Supplementary material: proofs and Propositions 4 and 5 in Sec. A; the 33 counterfactual sources, examples, rendering, label audit and closest work in Sec. B; the mixture, settings, curves, protocol, reported against measured scores, train–evaluation audit, held-out sets, successive playback, single-frame control and inference cost in Sec. C; encoder measurements in Sec. D; every 2×22 \times2 checkpoint, pooled differences, ability groups and the AoT, temporal-QA and 4B controls in Sec. E; reversal and mirror consistency in Sec. F; answer-position measurements and subspace removal in Sec. G; explicit pairs in Sec. H; scope in § 19.

Related Work

Video LLMs and encoders. Open video LLMs [2, 12, 20, 27, 34, 40, 44, 49, 75, 78] pair contrastive image encoders [50, 59, 74] with a language model and train on large caption and VQA corpora. Encoder studies favor SigLIP, with video encoders helping mainly on temporal perception [18, 57, 79]. V-JEPA 2 and 2.1 attach their encoders to Llama 3.1 8B with this recipe [4, 46]; VL-JEPA predicts text embeddings rather than tokens [11]; we keep the language model as the answer head and change the supervision rather than the scale.

Dynamics present but unread; temporal training signals. Motion direction remains linearly decodable inside video LLMs while the answer ignores it, and a projector-level motion objective helps [32]. Closest to us, Tracing the Arrow of Time [23] finds arrow-of-time information in video-centric but not frame-centric encoders, shows that query-based connectors disrupt it, and trains with a time-preserving MLP and arrow-of-time supervision (TVBench +1.3). We share the encoder observation and go beyond one binary target: a theory of which transformations give a shared, content-free component, fourteen answer-balanced constructions in six families, and a controlled encoder × supervision study; an arrow-of-time-only variant of our data is compared in § 5.1. Reversed, re-timed and shuffled clips have long served as self-supervised pretext tasks for learning video features [7, 19, 45, 61, 65, 68, 71] and as temporal benchmarks and probes [5, 8, 9, 13, 24, 31, 37, 42, 55, 76]; we apply them to a frozen encoder, as answer-balanced questions that teach a language model to read features the encoder already has. Training signals include perturbed-clip negatives [38], temporal preferences [36], frame-order fragments [26], an arrow-of-time reward [69] and generated counterfactual videos [48], each using one or a few transformations; SynRL [28] renders synthetic shapes with exact labels for CoT and RL training. TIME [64] adds answer-balanced temporal instructions in five dimensions, with reversed videos for direction. We instead set each answer by a transformation of the same clip, so scene content cancels from it, choose transformations by structure measured in the frozen encoder, and add them to a mixture that already holds annotation-based temporal questions (§ 5.1).

Where the Dynamics Lives in a Video Encoder

We derive when an encoder carries a shared change component and measure it, because this decides how efficiently a language model learns to use the dynamics.

Setup. Let z(v)z(v) be the frozen encoder tokens of a video vv, and TT a pixel transformation with an answer map πT\pi_T and token permutation ΠT\Pi_T, so that y(Tv)=πT(y(v))y(Tv)=\pi_T(y(v)). With zT(v)=ΠT−1z(Tv)z_T(v)=\Pi_T^{-1}z(Tv), the tokens of TvTv at the positions of vv, we define the content c=12(z(v)+zT(v))c=\frac{1}{2}(z(v)+z_T(v)) and the change d=12(z(v)−zT(v))d=\frac{1}{2}(z(v)-z_T(v)). We describe dd by its relative size rd=E∥d∥2/E∥z−zˉ∥2r_d=\mathbb{E}\lVert d\rVert^2/\mathbb{E}\lVert z-\bar{z}\rVert^2, its overlap ρ\rho with the leading content directions, and its consistency across videos κ=∥E[yd]∥2/E∥d∥2\kappa=\lVert\mathbb{E}[y d]\rVert^2/\mathbb{E}\lVert d\rVert^2, with y=±1y=\pm1 the direction of the change: 1 if every video moves by the same vector, about 0.001 for random directions.

Proposition 1 (frame-wise, text-aligned encoders). (i) If tokens are functions of single frames, E(v)=[f(x1),…,f(xN)]E(v)=[f(x_1),\ldots,f(x_N)], time reversal only reorders them, so the in-token change is d=0d=0 exactly: the arrow of time is not in any token’s content and can only be computed from token positions; a speed change selects other frames, so dd is a scene-specific frame difference. (ii) If the caption distribution is invariant to mirroring, a contrastive loss is too, so for an unnormalized (convex) loss a minimizer can be mirror-invariant in the caption-aligned subspace; for normalized embeddings we measure it (κ\kappa, Table 2).

Encoderreversal rdr_dreversal κ\kappaspeed rdr_dspeed κ\kappamirror rdr_dmirror κ\kappaproto 8probe
SigLIP so400m00–.0016.001450.073.5
ViCLIP (text-aligned).052.0012.042.001960.170.3
VideoMAE (pixel pred.).502.0032.632.045100.076.8
V-JEPA ViT-L (1B, 2B).064.037.080.03166.381.2
V-JEPA ViT-g (3B–8B).108.039.167.07477.284.7

Table 2. Change component of frozen encoders (1,200 clips; 70% fit, 30% held out). rdr_d: relative size of the in-token change after aligning token positions; κ\kappa: cross-video consistency; proto 8: accuracy of a prototype read from 8 pairs (chance 50); probe: linear probe on all training clips. ViCLIP and VideoMAE: diagnostic encoders with joint space–time attention, not used in our models. All measures: Table 22.

f(x1),…,f(xN)f(x_1),\ldots,f(x_N), time reversal only reorders them, so the in-token change is d=0d=0 exactly: the arrow of time is not in any token’s content and can only be computed from token positions; a speed change selects other frames, so dd is a scene-specific frame difference. (ii) If the caption distribution is invariant to mirroring, a contrastive loss is too, so for an unnormalized (convex) loss a minimizer can be mirror-invariant in the caption-aligned subspace; for normalized embeddings we measure it (κ\kappa, Table 2).

Proposition 2 (predictive spatiotemporal encoders). Assume the training loss is invariant under TT (data, masks and the predictor’s positional coding are symmetric) and that minimizers differ only by an invertible linear map on token channels, the same at every position (identifiability [53], assumed here). Then ΠT⊤E(Tv)=ATE(v)\Pi_T^{\top}E(Tv)=A_TE(v) with ATA_T acting on channels and T↦ATT\mapsto A_T a group representation. For an involution the features split into fixed subspaces V+⊕V−V_+\oplus V_- with c∈V+c\in V_+ and d∈V−d\in V_- for every video. If a linear readout ww that weights all token positions alike (ΠT⊤w=w\Pi_T^\top w=w) answers a question whose answer flips under TT, its part w−=12(w−ATw)w_-=\frac{1}{2}(w-A_Tw) gives the same answer on every video and ignores content (w−⋅c=0w_-\cdot c=0). Here “content” means the TT-invariant component, not the scene’s semantics.

Both concern token content, not relations between tokens, the route left to a frame-wise encoder; we measure their consequences, κ\kappa and ρ\rho, directly (order, speed and proofs: Sec. A).

Testing the two propositions on video encoders

We test them on the frozen features of the encoders of our study, SigLIP so400m and V-JEPA 2.1 ViT-L and ViT-g, and of two more video encoders with joint space–time attention, ViCLIP [63] (text-aligned) and VideoMAE [58] (pixel prediction). We use 1,200 SSv2 training clips that appear in no evaluation set, decoded as in training, each paired with its reversal, a half-speed version and its mirror image, so dd holds only what token content says about TT. Four measures ask whether a change component exists (rdr_d), whether it is shared across clips (κ\kappa), whether 8 pairs read it on held-out clips (proto 8) and whether it is present at all (probe). In Table 2, SigLIP has rd=0r_d = 0 under reversal exactly, and its speed and mirror components have κ<0.002\kappa< 0.002, so 8 pairs give chance; a probe that sees token positions still reads time direction (65.3%) and speed (73.5%), the relational route of Proposition 1. Both V-JEPA encoders carry a change component with κ\kappa 20 to 100 times larger, readable from 8 pairs, and the larger encoder is better on every measure. Reversal and mirror changes are nearly orthogonal to content (ρ\rho 0.03–0.12, against 0.44–0.63 for a content direction); the speed change is the most consistent but overlaps content (ρ\rho 0.40 / 0.59), which answer balance must separate (§ 4). Attention across frames does not by itself give this structure: text-aligned ViCLIP has no shared reversal or mirror direction either (κ≤.002\kappa\leq.002), pixel-predicting VideoMAE shares speed and mirror changes but not reversal (κ=.003\kappa= .003), and only V-JEPA shares all three and reads time direction and speed best (probes 80.8 and 81.2–84.7; Table 22). V-JEPA’s sign-flip and speed changes thus need only a few counterfactual pairs to fix their direction (Proposition 5), whereas SigLIP must learn them from token relations; order moves token positions in both encoders and should help both alike (Table 23).

EncoderTransf.rdr_dκ\kappaρ\rhoproto 8proto all
SigLIP so400mreversal0––50.050.0
speed.016.0016.1850.050.3
mirror.066.0014.1849.849.0
V-JEPA ViT-Lreversal.064.037.1258.863.7
speed.055.080.4066.371.0
mirror.093.031.03158.770.4
V-JEPA ViT-greversal.108.039.07563.974.3
speed.080.167.5977.278.8
mirror.104.074.02969.281.2
Other video encoders (diagnostic)
ViCLIP ViT-L/14reversal.052.0012.2449.849.7
speed.057.042.1760.171.2
mirror.029.0019.1450.250.3
VideoMAE ViT-Lreversal.502.0032.4150.655.8
speed.370.632.17100.0100.0
mirror.172.045.09464.779.3

Table 22. Encoder decomposition on 1,200 clips (token-averaged dd; 70% fit, 30% held out). κ\kappa uses y=+1y=+1 for every reversal and speed pair (the direction of the transformation) and y=±1y=\pm1 for left-to-right / right-to-left motion under mirroring, on frozen features pooled to 4×44 \times4 cells per tubelet and standardised per dimension on the fitting clips; random directions give κ≈1/840≈0.001\kappa\approx1/840 \approx0.001. ρ\rho: overlap of the change direction with the 32 leading content components (a content direction scores 0.44–0.63). SigLIP’s reversal component is zero by construction. ViCLIP [63]: video–text contrastive training with joint attention over all frame tokens (16 frames, the 8-frame temporal embedding interpolated); VideoMAE [58]: masked pixel prediction with 2×16×162 \times16 \times16 tubelets (32 frames); both at 2242224^2 (ViCLIP: penultimate block; VideoMAE: last layer), on the same clips, variants and pooling.

FamilyTransformationTheoryPredictionκ\kappa (frozen) V-JEPAκ\kappa (frozen) SigLIPCF gain at 8k V-JEPACF gain at 8k SigLIPAbility
Speed, raterescaling τa\tau_aProposition 1(i); measured κ\kappalarge on V-JEPA, small on SigLIP.080.0016+32.8+11.4speed
Sign flipsreversal, camera movesProposition 2: d∈V−d \in V_-; Proposition 1(i): d≡0d \equiv0large on V-JEPA.037–+6.80.0camera
+4.1−0.4state change
Sign flipsmirrorProposition 2; Proposition 1(ii)large on V-JEPA.031.0014+3.7+0.1direction
Event ordersegment permutationlocal receptive field: A=IA=Iequal on both––+1.9+1.7order

Table 23. From encoder structure to gain. For each family: the transformation, the result that predicts how its change is carried, the prediction, the measured cross-video consistency κ\kappa of the frozen ViT-L and SigLIP (Table 22; SigLIP reversal has rd=0r_d=0, no in-token component), and the CF gain on the matching natural-benchmark ability at 8k steps (Table 26). Order changes token positions rather than token content, so both encoders gain alike. Speed is the one family the propositions do not cover for V-JEPA (a rate change gives no shared direction by Proposition 2, Sec. A); its prediction rests on the measured κ\kappa alone, with Proposition 1 giving SigLIP only scene-specific frame differences. Gains are those of the whole counterfactual mixture against captions and QA, read per ability; at the final step the direction and state-change gains of the two encoders draw closer, while speed stays far apart (Table 26).

Present in the encoder, unused by the VLM

Does a VLM built on this encoder with the usual caption and QA mixture use the change? Table 3 follows one item set through every stage: the 540 held-out speed-ratio items of § 5.2 (unseen ratios, three classes, chance 33.3), read by a linear probe on the frozen encoder, on the bridge output and on language-model layers at the answer position, and answered by the VLM itself. For V-JEPA, the frozen features give 86.7% and the bridge of the caption and QA model keeps 83.5%, yet at the answer position 47–54% remains and the answer is at chance (33.7%); the model gives the same answer to a clip and its re-timed counterpart 89% of the time. With the counterfactual rows of § 4, the same encoder and mixture give 91.2% at the bridge, 96% from layer 18 on, and a 92.8% answer. For SigLIP no stage exceeds 40%, with or without the rows, as Proposition 1 and κ≈0.002\kappa\approx0.002 predict. The information reaches the language model, but captions and questions do not require it, and the answer does not use it. Nor is the item format the obstacle: SigLIP trained on the same counterfactual rows reaches only 34.8 on these items at 8k steps and 55.7 at the end, against 98.7 for V-JEPA (§ 5.2), and on TOMATO’s natural speed questions, posed in another format, the rows add 62.9 points (§ 6.1).

EncoderTrainingfrozenlinear probe on the stage bridgeL4L14L18L28VLM answer
SigLIPcaps+QA36.438.736.835.935.734.733.1
SigLIP+CF36.339.137.537.135.935.834.4
V-JEPAcaps+QA86.783.553.847.249.446.933.7
V-JEPA+CF86.891.266.678.996.396.592.8

Table 3. One item set at every stage: the 540 held-out speed-ratio items (unseen ratios 1.6 / 2.5 / 4.0 and equal; SSv2 validation; 3 classes, chance 33.3), 2B models at 8k steps. Probe: linear (PCA 64, logistic) at the answer position for language-model layers and on a 16×2×2 token grid for the frozen encoder and the bridge, 5-fold cross-validation grouped by source clip, 4 probe seeds (s.d. ≤1.8). Answer: the VLM on the same inputs (one option order per clip), within 2 points of the evaluation in Table 6.

Counterfactual Supervision

§ 3.2 shows the change reaching the language model without being required. The caption and QA mixture stays as it is and keeps teaching language and knowledge; we add rows that require the change.

Constructions with answers set by the transformation. A generator takes a clip vv from a public dataset and a transformation TT, renders TvTv, and writes the question, options and answer πT(y)\pi_T(y), which follows from the transformation: no new manual annotation and no language model are involved (constructions on actions and events reuse the source datasets’ own labels). We use fourteen constructions in six families (Table 1): sign flips (reversal, mirroring, camera moves), event order, temporal position, speed and rate, counts, and stopped or impossible events. Within each source the variants of a clip are drawn so that every scene appears with each answer equally often in expectation; for 29% of the rows a counterpart from the same clip, source and question family with a different answer is also in the data, an explicit pair (Figure 2 and Sec. H). Rows are sampled independently, so counterparts are not forced into one batch; the speed sources have no explicit pairs and rely on answer balance alone. The generator also balances the correct option’s length rank and each option’s prior under the text-only language model, and decorrelates answers from clip length and timing.

Figure 2. One generated pair: the same clip forward and backward, with the same question and opposite answers (4 of 32 frames). Other constructions: Figure 6.

FamilyConstruction of TvT_vAnswer rule πT\pi_TSource videosRowsPair%
Sign flipState change played backwardsaction ↔\leftrightarrow inverse (open/close)Charades-Ego, EPIC, SSv249k3
Object direction, clip mirrored or reverseddirection remappedSSv237k0
Camera pan, tilt or zoom on static footage, or reversedmove remappedCLEVRER, Charades-Ego, SSv223k0
Two clips joined, each optionally reverseddirection A then BSSv27k0
OrderTwo events in sequence or stacked at oncefirst / second / simultaneousLLaVA-Video, SA-V, SSv265k0
Segment matching a two-action captionmatching segmentHoloAssist20k80
PositionJoined clips; one holds the queried actionbeginning / middle / endHoloAssist, EPIC, Kinetics-70048k68
MagnitudeOne clip stacked twice over unequal spansfaster / same / slowerSA-V, SSv281k0
Span split 1:3, 3:1 or 1:1 between halvesaccelerates / decelerates / steadySA-V, SSv248k0
Frames spread over a short or long windowslow motion / normal / time-lapseEPIC12k0
CountOne action segment repeated kk timeskkHoloAssist, EPIC24k86
Segments of 1–4 different activitiesnumber of activitiesKinetics-70012k100
Stop/cutClip cut before the action completescomplete / incompleteCharades-Ego16k5
Simulator physics edits (teleport, vanish)possible / impossiblesimulator66k100

Table 1. The fourteen counterfactual constructions (33 sources, 509k rows, 6.1% of sampled training rows). Each row is rendered at training time from a clip of a public dataset and a transformation; the answer follows from πT\pi_T. Pair%: rows whose counterpart from the same clip with a different answer is also in the same source. Per-source details, real questions and rendered examples: Sec. B.

Why the construction makes the model read the change. Let s(v)s(v) be the answer log-odds at the scoring position and g=∂s/∂zg = \partial s/\partial z the readout through bridge and language model; in the linear regime a pair splits into a common part m=g⋅cm = g \cdot c and a differential part ds=g⋅dd_s = g \cdot d.

Proposition 3 (pair loss and content). For L=ℓ(y s(v))+ℓ(−y s(Tv))L = \ell(y\,s(v)) + \ell(-y\,s(Tv)) with logistic ℓ\ell, (i) ∇gL=−y[(a−b)c+(a+b)d]\nabla_g L = -y[(a-b)c + (a+b)d] with a=−ℓ′(y s(v))a = -\ell'(y\,s(v)), b=−ℓ′(−y s(Tv))b = -\ell'(-y\,s(Tv)), so at equal margins the content coefficient is exactly zero; (ii) for fixed dsd_s, LL is even and convex in mm, hence minimized at m=0m = 0. If the readout can set g⋅cg \cdot c and $g \cdot d independently (content and change separately addressable, e.g. c∈V+c \in V_+, d∈V−d \in V_- as in Proposition 2), the optimum of the pair loss therefore has no content in the answer: g⋅c=0g \cdot c = 0.

Without separability m=0m = 0 is not implied (Sec. A gives an example), so we measure the content share at the answer (§ 6.2). For binary answers it extends to any answer-balanced set at equal error weights, in expectation for separate rows and within each scene for explicit pairs.

Without pairs, a scene–answer correlation gives the readout a content component (Proposition 4); from a zero readout the first step on κ\kappa pairs is the prototype ∑iyidi\sum_i y_i d_i, which reads a new pair’s change better as κ\kappa grows and a single clip better as ρ\rho shrinks (Proposition 5; both in Sec. A). With the measured κ\kappa, this anticipates large gains from sign-flip and speed constructions on V-JEPA, small ones on SigLIP and equal gains from order on both (Table 23), and a change in how the answer layers are organized more than in what a probe can read at the bridge input (§ 6.2).

Model and training. A frozen V-JEPA 2.1 encoder (ViT-L for 1B and 2B, ViT-g for 3B–8B) reads 32 frames; a spectral bridge maps each clip to 1,024 tokens: a fixed PCA rotation and rescaling computed once from frozen encoder statistics, followed by a trainable MLP projector. After a bridge-only warm-up of 0.6–0.8% of steps, the language model {[22, 70]} is trained in full in a single stage {[29]}, in place of the usual alignment, pre-training and instruction stages, on 4.53M rows drawn at fixed per-source weights from the training splits of 175 public sources, the caption and QA collections {[12, 77, 78]} plus the 33 counterfactual sources (about 6% of drawn rows), at global batch 150–300 (settings: Table 11; curves and metrics: Figure 7 and Table 12).

ModelLanguage modelEncoderBatch/GPU×GPUsGlobalStepsSamplesLR (LLM)EFLOPs
Ours-2B-QQwen3-1.7BViT-L12×1412 \times1416827,3824.60M1.05e−51.05\mathrm{e}{-5}121
Ours-1B-LLlama-3.2-1B-Instr.ViT-L6×286 \times2816827,3824.60M1.05e−51.05\mathrm{e}{-5}104
Ours-4B-QQwen3-4B-Instr.-2507ViT-g5×305 \times3015030,1874.53M1.2e−51.2\mathrm{e}{-5}314
Ours-3B-LLlama-3.2-3B-Instr.ViT-g10×3010 \times3030015,0944.53M1.7e−51.7\mathrm{e}{-5}277
Ours-8B-QQwen3-8BViT-g5×305 \times3015030,1874.53M8e−68\mathrm{e}{-6}437
Ours-8B-LLlama-3.1-8B-Instr.ViT-g5×305 \times3015030,1874.53M8e−68\mathrm{e}{-6}426

Table 11. Training settings. All: frozen V-JEPA 2.1 encoder at 32 frames, spectral bridge to 1,024 tokens (fixed PCA rotation and rescaling, then a trainable MLP), bridge learning rate 2×10−42 \times10^{-4} (3B: 2.83×10−42.83 \times10^{-4}), weight decay 0.01 (LLM) / 0.05 (bridge), linear warm-up over 0.2–0.3% of steps with the LLM frozen for the first 0.6–0.8%, cosine decay, maximum sequence 2,560 tokens, 4.53M rows drawn at fixed per-source weights from the 175-source mixture (4.60M for the ViT-L models). Compute: Table 12; EFLOPs (101810^{18}): trained parameters × trained tokens × 6 plus attention and the frozen encoder’s forward pass, measured per clip (activation recomputation excluded).

Figure 7. Training loss, gradient norm (bin median) and language-model learning rate of the released models against samples seen.

ModelH100sGlobal batchStepsSamplesSamples/sHoursH100-hoursPeak mem. (GB)Loss, first 1%Loss, last 5%
Ours-2B-Q1416827,3824.60M7816.4229461.530.41
Ours-1B-L2816827,3824.60M11810.8303181.310.43
Ours-4B-Q3015030,1874.53M7616.6499441.410.39
Ours-3B-L3030015,0944.53M8814.3429511.510.39
Ours-8B-Q3015030,1874.53M5821.6648331.360.38
Ours-8B-L3015030,1874.53M6319.9596321.370.40

Table 12. Training metrics of the released models (one stage, one run each). Samples/s: median throughput over the run; hours and H100-hours: samples divided by that throughput, so pauses between resumed segments are not counted. Peak memory: per GPU (activation checkpointing). Loss: mean next-token cross-entropy on answer tokens of the logged batches.

Experiments

Benchmarks and protocol. We start from the benchmarks of Meta’s V-JEPA video LLMs {[4, 46]}, MVBench {[35]}, PerceptionTest {[47]}, TVBench {[13]}, TempCompass {[42]} (multiple choice), TOMATO {[55]}, MVP {[31]} (pair accuracy) and TemporalBench {[9]} (binary), and extend them to twelve scores from eleven benchmarks with MotionBench [24], NExT-QA [67], SSv2 [21] (multiple choice over action labels), and Vinoground [76] text and video (pair accuracy). The additions cover motion (MotionBench), causal and temporal reasoning (NExT-QA), fine-grained actions (SSv2) and temporal order that a single frame cannot answer (Vinoground). We evaluate every open model ourselves with the same protocol: 32 frames, the same prompts and option-letter scoring; their own reported numbers, under their settings, are in Table 14. We report the plain average of the twelve.

ModelMVBenchPerceptionTestTVBenchTempCompassTOMATOMotionBenchNeXt-QA
2B class
Qwen3-VL-2B61.7 (61.0)– (65.1)– (49.8)– (67.4)– (30.3)– (57.5)– (77.1)
InternVL3.5-2B65.9 (66.8)– (64.2)– (50.9)– (65.7)– (31.0)– (56.2)– (76.1)
PLM-1B70.1 (68.0)72.7t (69.8)50.4 (48.2)64.6 (60.5)25.5 (24.5)52.2 (51.7)80.3 (79.8)
VideoLLaMA3-2B65.5 (57.9)68.0t (62.9)– (44.0)63.4 (64.1)– (24.0)– (56.3)81.1 (75.4)
Apollo-1.5B– (61.1)61.0 (63.9)– (47.9)60.8 (60.3)– (27.8)– (51.1)– (78.8)
Ours-2B-Q69.575.370.773.550.954.374.0
4B class
Qwen3-VL-4B68.9 (68.0)– (72.0)– (59.2)– (70.9)– (36.3)– (61.5)– (80.9)
InternVL3.5-4B71.2 (70.9)– (69.1)– (60.0)– (71.1)– (32.1)– (59.0)– (79.8)
PLM-3B74.7 (73.0)79.3t (76.0)58.9 (56.6)69.3 (67.0)30.9 (36.3)60.4 (58.9)83.4 (84.0)
Molmo2-4B75.1 (74.2)81.3t (81.4)– (65.5)72.8 (70.8)39.8 (36.5)61.6 (62.3)85.5 (85.1)
Apollo-3B– (62.0)65.0 (67.1)– (50.9)62.5 (63.0)– (29.0)– (55.0)– (80.9)
Ours-4B-Q72.178.274.879.953.559.179.3
8B class
Qwen3-VL-8B68.7 (68.8)– (73.0)– (58.2)– (73.6)– (36.5)– (62.9)– (82.1)
InternVL3.5-8B72.1 (71.0)– (72.7)– (62.7)– (71.0)– (33.6)– (58.1)– (81.5)
PLM-8B77.1 (76.0)82.7t (80.6)63.5 (62.7)72.7 (70.9)33.2 (42.6)61.4 (62.7)84.1 (85.7)
Molmo2-8B75.9 (75.6)82.1t (81.9)– (64.1)73.4 (72.1)39.6 (35.4)62.2 (61.4)86.2 (85.4)
VideoLLaMA3-7B69.7 (65.6)72.8t (69.7)– (49.2)68.1 (69.3)– (29.4)– (57.8)84.5 (81.5)
Ours-8B-Q72.877.875.379.456.559.181.2

Table 14. Scores reported by the models’ authors, with ours under the common protocol in parentheses (32 frames, option-letter scoring; Table 4). Reported settings differ from ours: PerceptionTest test split (superscript t; ours: validation), Qwen3-VL at 2 fps up to 2,048 frames, InternVL3.5 the best of 16–64 frames, Molmo2 at 2 fps up to 128 frames, VideoLLaMA3 at 1 fps up to 180 frames, PLM 32 frames; PLM’s TVBench from its repository. –: not reported. Gemma 4 reports no video benchmark; MVP, TemporalBench (binary), SSv2 and Vinoground are not reported by any of these models.

Main results. Table 4 lists every benchmark. With one training stage, a fraction of the data and 32 tokens per frame, our models have the highest average in each size class: the 4B model (72.14) is 11.1 points above the best open 4B model and 7.4 above the best open 8B model, the 2B model (66.02) is 13.6 above the best open 2B model, and the Llama models lead alike (68.97 on Llama-3.2-3B and 62.21 on Llama-3.2-1B). Without SSv2 and the two Vinoground scores every model still leads its class (4B: 70.1 against 66.0). The largest margins are on the dynamics benchmarks (4B class: TOMATO 53.5 against at most 36.5, Vinoground video 66.8 against at most 33.8), and our 4B model answers both the original and the reversed or mirrored version of natural items correctly on 47.7% and 50.4%, against at most 35.4% for open 4B models (Table 27). Against Meta’s unreleased video LLMs on the same encoder family (numbers quoted from their papers; Table 5), our 8B models lead on their six benchmarks by 6.7 (Qwen3-8B) and 4.9 (Llama-3.1-8B, Meta’s own).

ModelMVBPTTVBTCTOMMoBNQAMVPTBSSv2VG-TVG-VAvg
2B class
Qwen3-VL-2B61.065.149.867.430.357.577.132.365.959.236.227.652.45
InternVL3.5-2B66.864.250.965.731.056.276.128.871.357.040.021.052.42
PLM-1B68.069.848.260.524.551.779.830.958.161.733.229.651.33
Apollo-1.5B61.163.947.960.327.851.178.826.055.856.038.822.449.16
VideoLLaMA3-2B57.962.944.064.124.056.375.423.164.150.038.025.248.75
Ours-2B-Q (Qwen3-1.7B)69.575.370.773.550.954.374.054.663.991.663.250.866.02
Ours-1B-L (Llama-3.2-1B)66.371.967.971.648.249.570.248.557.889.651.253.862.21
4B class
Molmo2-4B74.281.465.570.836.562.385.145.472.857.048.233.861.08
PLM-3B73.076.056.667.036.358.984.046.266.268.355.031.459.91
InternVL3.5-4B70.969.160.071.132.159.079.841.473.264.153.628.058.52
Qwen3-VL-4B68.072.059.270.936.361.580.939.668.964.847.631.058.39
Apollo-3B62.067.150.963.029.055.080.927.460.164.737.425.851.94
Ours-4B-Q (Qwen3-4B)72.178.274.879.953.559.179.358.175.792.276.066.872.14
Ours-3B-L (Llama-3.2-3B)70.774.370.376.555.954.277.056.367.391.371.062.868.97
8B class
PLM-8B76.080.662.770.942.662.785.749.071.267.963.444.864.79
Molmo2-8B75.681.964.172.135.461.485.443.470.855.249.233.860.69
Qwen3-VL-8B68.873.058.273.636.562.982.141.870.865.951.035.059.97
InternVL3.5-8B71.072.762.771.033.658.181.538.872.166.747.228.458.65
VideoLLaMA3-7B65.669.749.269.329.457.881.529.970.559.144.828.654.62
Ours-8B-Q (Qwen3-8B)72.877.875.379.456.559.181.258.173.593.875.238.670.11
Ours-8B-L (Llama-3.1-8B)71.975.071.877.655.555.979.057.672.392.374.263.070.51

Table 4. Main results: twelve scores on eleven video benchmarks. All models evaluated by us with one protocol (32 frames, same prompts, option-letter scoring). Shaded: V-JEPA-LM (-Q: Qwen3, -L: Llama). Bold: best in class. MVB: MVBench, PT: PerceptionTest, TVB: TVBench, TC: TempCompass, TOM: TOMATO, MoB: MotionBench, NQA: NExT-QA, MVP and VG-T/VG-V (Vinoground text/video): pair accuracy, TB: TemporalBench (binary), SSv2: Something-Something v2 (multiple choice). 95% bootstrap intervals of the average: ±0.7 for every model (Sec. C). Gemma 4, last in both classes: Tab. 16.

ModelTok.SamplesMVBPTTVBTCTOMMVPAvg
V-JEPA 2 (L3.1-8B)9,21688.5M73.584.060.676.940.344.563.3
V-JEPA 2.1 (L3.1-8B)–72.5M75.483.162.874.038.043.262.8
Ours-8B-Q (Qwen3-8B)1,0244.53M72.877.875.379.456.558.170.0
Ours-8B-L (L3.1-8B)1,0244.53M71.975.071.877.655.557.668.2

Table 5. Against Meta’s V-JEPA video LLMs on the six benchmarks both report [4, 46] (L3.1-8B: Llama-3.1-8B, also the language model of Ours-8B-L). Meta values quoted from the papers (the video LLMs are not released, only the encoders): V-JEPA 2 with ViT-G384_{384}, V-JEPA 2.1 with ViT-G384_{384}; PerceptionTest is the test split after training on its train split for V-JEPA 2 and the validation split for V-JEPA 2.1. Ours measured.

Modelreversal (1,350 items, 5 benchmarks) origfollowsbothstuckmirror (811 items) origfollowsbothstuck
Molmo2-4B59.951.531.941.648.744.623.945.5
InternVL3.5-4B54.648.529.746.453.351.435.450.7
Qwen3-VL-4B55.047.925.346.142.438.117.550.9
PLM-3B47.446.321.047.339.639.820.346.9
V-JEPA caps+QA (2B, 8k)56.647.932.340.150.750.634.548.1
SigLIP caps+QA (2B, 8k)43.543.019.449.435.638.514.862.8
SigLIP +CF (2B, 8k)43.640.419.748.537.238.114.357.0
V-JEPA +CF (2B, 8k)59.852.236.832.752.352.838.041.4
Ours-2B-Q65.554.740.731.657.358.242.737.1
Ours-1B-L63.253.641.031.355.159.342.936.0
Ours-4B-Q70.057.947.727.359.461.250.433.0

Table 27. Counterfactual consistency in full. Items qualify automatically when the correct option, with the transformation’s word swaps applied (left↔right, towards↔away, open↔close, before↔after, . . .), equals another option exactly. Orig: accuracy on the original video; follows: the transformed answer is the swapped option; both: both correct; stuck: same answer for both videos.

Data and compute. For every model we count the samples of the vision–language training that turns a pretrained vision encoder and a pretrained language model into the VLM, all of its stages as each paper reports them (for Molmo2 only its SFT, a lower bound); the pre-training of the vision encoder (V-JEPA 2.1 here, SigLIP or InternViT in the open models) and of the language model is excluded alike (Sec. C). On this count our single stage uses 4.53M public samples (4.60M for ViT-L), 2 to 38 times fewer than the open models that report counts (Figure 3), with 2 to 8 times fewer visual tokens, in 499 H100-hours (4B) or 229 (2B). At inference (Table 21; H100) the bridge reduces the encoder’s 9,216 patches to 1,024 tokens, so the language model prefills with 2 to 18 times fewer FLOPs than open models of the same size, more than our whole encoder (14 or 38 TFLOPs) against models with 8,192 tokens; the first token takes 70–182 ms at any resolution. Each model is first in its class at every matched sample count (Table 13).

Samples (M)2.73.03.33.63.94.24.54.53
Ours-4B-Q69.9770.2469.9871.4371.4171.7372.1872.14
Ours-3B-L66.4067.1467.3968.3368.7368.5468.9268.97
Ours-8B-Q68.6668.4769.2869.8970.1269.9270.2370.11
Ours-8B-L68.4068.8268.7969.7870.4170.5970.6170.51

Table 13. Twelve-score average at matched training samples for the ViT-g models (3B: global batch 300; 4B, 8B: 150). Each model is first in its class at every point. The ViT-L models (1B, 2B; global batch 168) are checkpointed at other sample counts; the 2B trajectory is in Figure 4.

Modelvis. tokensvision encoder TFLOPsvision encoder msconnector msLLM prefill TFLOPsLLM prefill msfirst token msdecode ms/token720p clip (ms) LLM prefill720p clip (ms) first token
2B class
Qwen3-VL-2B1,920 / 11,5205.035in enc.7.0316618.2154308
InternVL3.5-2B8,19223.2550.440.510415918.4104159
PLM-1B8,19222.2871.825.2571469.357146
Ours-2B-Q1,02413.9600.83.3228320.02182
Ours-1B-L1,02413.9600.62.310708.21071
3B–4B class
Qwen3-VL-4B1,920 / 11,5205.035in enc.18.1589324.9319474
InternVL3.5-4B8,19223.2550.5104.421827325.3218273
Molmo2-4B2,59219.7132in enc.26.19322527.8103235
PLM-3B8,19222.2873.070.413822815.6138228
Ours-4B-Q1,02437.81370.98.62916723.929168
Ours-3B-L1,02437.81380.96.72015915.120159
8B class
Qwen3-VL-8B1,920 / 11,5206.843in enc.32.28112425.2450666
InternVL3.5-8B8,19223.2550.8160.731437024.5313369
Molmo2-8B2,59219.7133in enc.45.412726128.4137271
PLM-8B8,192123.63364.6151.927061018.3271610
Ours-8B-Q1,02437.81390.915.84218224.442182
Ours-8B-L1,02437.81380.916.23617517.136175

Table 21. Inference cost per 32-frame clip on one H100 (batch 1, bf16, video decoding excluded; times: median of 10 runs after 3 warm-ups; 480p MVBench clip unless noted). TFLOPs: matrix multiplications including attention, counted with PyTorch’s FLOP counter. Vision encoder: the vision tower; for our models the frozen, unmodified V-JEPA 2.1 forward (ViT-L for 1B and 2B, ViT-g otherwise), a cost set by the choice of encoder and identical for any recipe built on it: V-JEPA attends jointly over all frames, while the open models’ encoders attend within a frame or frame pair. Connector: projector or bridge (“in enc.”: part of the vision module; at most 1.5 TFLOPs for every model). LLM prefill: the language model over the whole prompt; first token: all three parts; decode: per generated token with KV cache (128 tokens). Visual tokens: 480p / 720p where they differ (Molmo2: from its processor). The outer count and cost do not depend on resolution; Qwen3-VL’s grow with it. –: not timed.

Encoder × supervision

Which part of the gain is the encoder’s and which the supervision’s? Four 2B models cross V-JEPA ViT-L or SigLIP so400m with or without the counterfactual rows, on the same mixture and schedule (Figure 4 and Table 6).

Figure 4. Twelve-score average during training (2B). Solid: with counterfactual rows; dashed: captions and QA only.

EncoderSupervision12-score avg 8k12-score avg finalspeeddirectioncameraknowledge6-set avgunseen ratiosrev. cons.
SigLIP so400mcaps+QA53.4658.5234.039.936.463.547.135.019.4
SigLIP so400m+CF56.7762.1345.440.036.462.659.334.819.7
V-JEPA ViT-Lcaps+QA56.6561.5332.459.740.564.158.633.532.3
V-JEPA ViT-LAoT only56.3761.1733.055.743.963.660.831.930.8
V-JEPA ViT-L+CF61.1866.0265.363.347.463.791.391.536.8

Table 6. Encoder × supervision (2B; same mixture and schedule). +CF: with the counterfactual rows; AoT only: an arrow-of-time question in their place. Final: step 27,382. Abilities: § 6.1; held-out chance 33.3; rev. cons.: original and reversed video both correct (Table 27). All steps and benchmarks: Table 24.

CF helps both encoders at 8k (4.5 and 3.3 points) and at the end (4.5 and 3.6), and the 4B model by 4.8 at 8k (Sec. E); V-JEPA with CF reaches the final SigLIP+CF score (62.13) by 10k steps; the abilities it unlocks differ (§ 6.1). Against the best open 2B model, all three parts of the recipe add: our mixture and bridge with SigLIP give +6.1 (58.52 against 52.45; +3.2 without SSv2 and Vinoground), the V-JEPA encoder +3.0 and the counterfactual rows +4.5. Matched-weight controls fall short. The arrow-of-time question of [23] is learned (95.9% on unseen clips) but gives 56.37 at 8k and 61.17 at the end, no better than captions and QA (56.65 and 61.53); the mixture’s 22 annotation-based temporal-QA sources at that weight give 58.21. Speed stays at the caps-only level (33.0 and 31.3 against 32.4) and the held-out sets at 60.8 and 62.1 (Sec. E): the gain comes from the counterfactual construction, not from more temporal questions.

Generalization beyond the training constructions

Six held-out sets (3,294 items; Table 7 and Table 17) each change one factor: speed ratios absent from training (1.6, 2.5, 4.0; training uses equal, 2:1 and 3:1), Kinetics-700, a corpus never used for these families, unseen combinations, and an unseen simulator. Clips come from validation splits or a simulator unused for camera rows, and wordings and option pools are the training generator’s own, so only the tested factor is new. Our models average 94.5–96.3 and open models 43.5–55.5 (Table 7); ratios between and beyond the trained ones are solved alike (97.2, 100 and 99.4 for 1.6, 2.5 and 4.0 on the 4B model; Table 19). Successive playback changes the construction itself: the clip is played twice in a row, a short central span and the full span, so the motion speeds up or slows down halfway, which no training row uses for speed; our models score 92.5–99.2 against 19.4–51.9 for open models. In the controlled study V-JEPA with CF reaches 91.3 at 8k steps and 95.0 at the end, against 64.3 without CF and 74.8 for SigLIP with the same rows, which reaches only 55.7 on unseen ratios against 98.7 (Table 18); at 4B, CF likewise lifts the held-out sets from 61.4 to 93.0 and successive playback from 33.1 to 90.0 at 8k steps (Sec. E). Without motion (middle frame repeated) our 2B and 4B models fall to 37.9 and 37.6, so single frames do not solve the sets; apparent motion follows the speed labels on 95–100% of rows (Sec. B).

Modelnew rationew corpus spdnew corpus dirnew comb. dirnew comb. spdnew cam.avgsucc. play
Qwen3-VL-2B37.045.535.256.544.643.943.833.1
Qwen3-VL-4B35.950.540.773.350.249.450.036.9
Qwen3-VL-8B36.750.841.978.468.057.155.551.9
V-JEPA, no CF33.344.369.897.149.192.464.339.4
Ours-2B-Q98.798.575.798.598.999.895.093.9
Ours-4B-Q98.999.080.999.0100.0100.096.396.7
Ours-8B-L99.398.780.098.899.6100.096.199.2

Table 7. Generalization (chance 33.3): the six held-out sets of § 5.2 and successive playback (succ. play). V-JEPA, no CF: the 2B captions and QA arm of Table 6; like the open models, it never saw these constructions. All models: Table 18 and Table 20.

SetItemsSourceWhat is new
Speed ratio540SSv2 validationrelative speed of a clip stacked over itself at span ratios 1.6, 2.5, 4.0 (training: equal, 2:1 and 3:1, and 1.0–1.4 in the fine-motion source; none of the three ratios occurs), plus the equal item of each clip
Domain, speed600Kinetics-700 validationrelative speed (2:1, 3:1, equal) and acceleration on a corpus never used by the speed family
Domain, direction540Kinetics-700 validationcamera direction labeled by optical flow, original / reversed / mirrored, on a corpus unused for this family
Combo, direction584SSv2 validationmirror and reversal together (the answer is the original direction), with the three trained variants of the same clip
Combo, speed540SSv2 validationacceleration played backwards (accelerating becomes decelerating), plus freeze
Camera, domain490NVIDIA PhysX simulationsynthetic camera moves on a simulator never used for camera rows

Table 17. Held-out generalization sets (chance 33.3 on average). Question wordings, option pools and option-length balancing are the training builders’ own.

Modelratiodom. spddom. dircomb. dircomb. spdcameraavg
Open models
Qwen3-VL-2B37.045.535.256.544.643.943.8
Qwen3-VL-4B35.950.540.773.350.249.450.0
InternVL3.5-4B33.946.730.761.153.045.145.1
Molmo2-4B38.350.529.159.451.132.743.5
PLM-3B35.949.835.764.958.544.948.3
Qwen3-VL-8B36.750.841.978.468.057.155.5
InternVL3.5-8B33.045.533.963.052.449.246.2
Molmo2-8B39.352.233.969.259.637.648.6
PLM-8B38.349.035.469.053.551.049.4
VideoLLaMA3-7B32.245.028.763.550.041.443.5
Controlled 2B arms (Tab. 6), at 8k steps / final
SigLIP caps+QA, 8k35.050.043.058.054.342.247.1
SigLIP caps+QA, final35.951.050.473.660.052.954.0
SigLIP +CF, 8k34.859.852.066.386.356.559.3
SigLIP +CF, final55.772.061.588.994.675.974.8
V-JEPA caps+QA, 8k33.544.855.094.048.775.958.6
V-JEPA caps+QA, final33.344.369.897.149.192.464.3
V-JEPA AoT only, 8k31.942.860.496.154.179.660.8
V-JEPA AoT only, final34.645.372.697.448.790.464.8
V-JEPA temporal QA in place of CF, 8k36.149.262.292.855.077.662.1
V-JEPA +CF, 8k91.593.068.597.997.899.291.3
V-JEPA +CF, final (= Ours-2B-Q)98.798.575.798.598.999.895.0
Ours (final checkpoints)
Ours-2B-Q98.798.575.798.598.999.895.0
Ours-1B-L97.899.271.598.699.6100.094.5
Ours-4B-Q98.999.080.999.0100.0100.096.3
Ours-3B-L99.199.278.999.0100.0100.096.0
Ours-8B-Q99.199.078.599.199.4100.095.9
Ours-8B-L99.398.780.098.899.6100.096.1

Table 18. Held-out construction sets (definitions in Tab. 17: unseen speed ratios, an unseen source for the speed and direction families, unseen combinations, a simulator unused for camera rows). Ratio: unseen playback-speed ratios (chance 33.3); dom.: speed / direction on a new source corpus; comb.: unseen mirror × reversal and acceleration × reversal combinations; camera: simulated camera motion. 4B-class open models average 43.5–50.0 and Qwen3-VL-2B 43.8.

Model1.62.54.0
Controlled 2B arms, 8k
SigLIP +CF38.335.630.6
V-JEPA caps+QA32.834.433.3
V-JEPA temporal QA in place of CF37.836.733.9
V-JEPA +CF88.991.793.9
Ours (final checkpoints)
Ours-2B-Q97.299.499.4
Ours-1B-L94.499.499.4
Ours-4B-Q97.2100.099.4
Ours-3B-L97.899.4100.0
Ours-8B-Q97.899.4100.0
Ours-8B-L97.8100.0100.0

Table 19. Unseen speed ratios separately (180 items each, chance 33.3). Training uses equal, 2:1 and 3:1 and fine ratios up to 1.4, so 1.6 and 2.5 lie between trained ratios and 4.0 beyond them.

Analysis: Why the Supervision Works

Abilities from the encoder and from the supervision

The encoder and the supervision contribute different abilities. We group the natural benchmarks’ own question categories into abilities (speed, direction, camera motion, state change, …) and compare models item by item with a paired bootstrap (Figure 5; all groups in Table 26; pooled over checkpoints in Table 25). Speed needs both at this budget: without CF the two encoders are level on speed (−1.5, interval includes zero); CF adds 32.8 points on V-JEPA (29.4 on the 4B model) and 11.4 on SigLIP (18.0 by 24k); at the end the V-JEPA gain holds on both source benchmarks (TOMATO +62.9, TempCompass +21.8; Sec. E). Direction, counting and fine actions come from the encoder: without CF SigLIP is already 19.7, 7.7 and 11.5 points behind, and CF barely moves direction and counting on SigLIP. Camera and state change need a consistent change direction: CF adds 6.8 and 4.1 on V-JEPA and nothing on SigLIP. General knowledge is kept across encoders (+0.8 at 24k, interval includes zero).

Figure 5. Gain from counterfactual supervision per ability (2B, 8k steps); whiskers: 95% paired bootstrap interval.

ComparisonStepsMVBPTTVBTCTOMMoBNQAMVPSSv2VG-Tall items
SigLIP+CF − V-JEPA+CF12–24k-1.7-2.4-3.9-5.2-13.0-3.3+1.7-1.6-6.5-5.7-2.37
SigLIP+CF − SigLIP caps12–24k-0.4+0.2+0.9+2.2+6.4+0.2+0.2+0.2+1.0+0.5+0.63
SigLIP caps − V-JEPA+CF12–24k-1.3-2.6-4.8-7.4-19.4-3.5+1.5-1.8-7.6-6.2-3.00
V-JEPA caps − V-JEPA+CF2–8k+2.4-0.5+0.7-4.3-12.6+1.0+1.2-1.6+1.9-4.7-0.22
SigLIP+CF − V-JEPA+CF2–8k-1.3-2.1-5.3-2.0-11.6-2.7+3.1-3.6-8.5-3.9-1.97
SigLIP caps − V-JEPA caps2–8k-2.4-1.7-7.0+0.6-3.4-1.5+1.7-2.6-11.1-0.5-1.89
SigLIP+CF − SigLIP caps2–8k-1.3+0.0+1.0+1.6+4.4-2.2+0.2+0.6+0.6+1.4+0.13

Table 25. Paired item-level differences pooled over four checkpoints (each item’s mean difference over the steps; bootstrap over items). +CF: with counterfactual supervision; caps: caption and QA mixture alone. 95% intervals exclude zero for every entry with ∣Δ∣≥1.6|\Delta| \ge1.6 (e.g. TOMATO −13.0 [−15.2, −10.8], SSv2 −6.5 [−7.7, −5.4], NExT-QA +1.7 [+1.0, +2.3] in the first row). “All items” weights benchmarks by size (NExT-QA has 8.6k items).

Comparisonspeeddirectionordercameracountstatelocaliz.antonymrotationtrackingTBSSv2NQAknowl.
CF on V-JEPA, 8k+32.8+3.7+1.9+6.8-1.5+4.1-5.0+3.8+2.8-8.9+2.6+0.3-0.4-0.4
CF on V-JEPA, final+38.1+4.0+1.9+3.6+1.6+1.5-3.1-1.0+2.1-4.0+0.2+1.3-0.4+0.0
AoT only on V-JEPA, 8k+0.6-3.9-0.4+3.4-6.5-0.4-4.7+1.2+1.7-3.3+2.8-2.6-0.9-0.5
CF on SigLIP, 8k+11.4+0.1+1.7+0.0+1.2-0.4-0.6+7.1+3.8-8.0+0.5+3.9+0.7-0.8
CF on SigLIP, 24k+18.0+3.0+3.3+1.9+1.4-0.3+1.7+5.4-0.3+3.8+2.4+0.4+0.1-0.3
CF on SigLIP, final+16.9+3.2+3.4+1.2+2.3+1.5+2.5+5.2-1.0+0.2+1.7-0.1+0.4-0.4
V-JEPA − SigLIP, caps, 8k-1.5+19.7+3.4+4.1+7.7-2.3+5.6+0.8+5.9+0.9+0.7+11.5-1.4+0.6
V-JEPA − SigLIP, +CF, 8k+19.9+23.3+3.6+10.9+5.0+2.3+1.1-2.5+4.9+0.0+2.8+8.0-2.4+1.1
V-JEPA − SigLIP, +CF, 24k+15.7+17.9+2.5+8.5+7.9+3.4+2.5-0.6-1.0+0.0+0.1+5.7-1.4+0.8

Table 26. Ability groups over the natural benchmarks’ own categories (2B; final: step 27,382). Items: speed 527, direction 1,170, order 1,457, camera 585, count 1,776, state change 711, localization 360, reversal/antonym 520, rotation 286, tracking 425, TemporalBench 2,765, SSv2 2,088, NExT-QA 8,564, knowledge (objects, scenes, knowledge) 9,254. Paired bootstrap 95% intervals: speed on V-JEPA [+27.9, +37.8], on SigLIP [+6.3, +16.5] (8k) and [+12.7, +23.3] (24k); direction between encoders without CF [+16.7, +22.7]; speed between encoders without CF [−4.9, +1.7].

What the language model reads

We test the propositions at the answer position on same-question pairs (Table 28). Proposition 3: with CF, 0.94 to 0.98 of the relative-speed answer is differential; without CF the common part dominates and the answer does not move when the video changes (89 to 91% stuck). Geometry, not information: the bridge input has the same speed consistency in both models (0.139, 0.131), and CF builds a shared speed axis inside the language model (κ\kappa 0.035 → 0.648). Intervention (Table 8): removing 8 speed directions (estimated on training clips) takes unseen speed combinations from 98.9 to 65.7 and TOMATO from 50.9 to 46.4; a random subspace of the same energy leaves 97.2 and 49.9, and the top content directions, at 2.6 times the energy, cost speed less and content more.

Removed subspaceenergyheld-out speed ratioheld-out speed comb.TOMNQAPT
none98.798.950.974.075.3
speed directions (8)16.691.565.746.473.273.8
random, same energy (24)16.296.797.249.973.274.3
top content directions (24)43.695.084.344.872.472.0

Table 8. Removing a subspace from every visual token of Ours-2B-Q at inference. Energy: % removed. Full: Table 29.

Pair set: share / stuck / leakSigLIP +CFV-JEPA caps+QAV-JEPA +CF
Unseen speed ratios.065 / 81.7% / 44.8.022 / 89.4% / 52.7.983 / 1.1% / 0.08
Speed, new source (Kinetics-700).611 / 73.2% / 1.24.081 / 90.8% / 5.3.943 / 6.3% / 0.29
Accel./decel. (+reversal).482 / 33.1% / 2.0.081 / 79.5% / 9.1.745 / 5.4% / 0.83
Direction combinations.918 / 51.0% / .087.900 / 9.2% / .095.980 / 2.7% / .021
κ\kappa / proto 8 at layer 18
Unseen speed ratios.008 / 50.1.035 / 55.2.648 / 99.3
Speed, new source.105 / 42.0.056 / 47.1.324 / 63.1
Accel./decel..229 / 78.9.100 / 58.6.207 / 89.4
Direction combinations.076 / 51.0.288 / 93.1.300 / 93.8
Direction, bridge input.0006 / 49.9.361 / 90.5.330 / 90.6
Parallelogram cosine (layer 27)
mirror effect−.293−.894−.938
reversal effect+.252−.877−.888
mirror effect, bridge input+.777−.242−.235
Composition acc. / original69.9 / 65.193.8 / 94.595.2 / 98.6

Table 28. Internal measurements. Parallelogram: cos⁡(hplain−hmirror,hrev−hmirror+rev)\cos(h_{\mathrm{plain}}-h_{\mathrm{mirror}},h_{\mathrm{rev}}-h_{\mathrm{mirror+rev}}); −1-1 means mirror and reversal flip the same subspace (Proposition 2); clips paired with other clips give about 0.03.

Explicit pairs versus balanced rows

Two more 2B models draw the same rows per source and answer class from one pool, one using every scene at most once, the other keeping same-scene rows with different answers together (Proposition 4, Sec. H). Pairing shows most in consistency: both halves of an unseen speed-ratio pair are correct in 96.7% against 84.4%, with gains on state change (+4.5) and reversal (+4.6).

Discussion and Conclusion

Captions and questions leave a world-model encoder’s dynamics unread. The encoder decides what can be read (direction, counting, fine actions) and the supervision what is read (speed, camera motion, state change), consistent with the measured κ\kappa; with counterfactual supervision, a frozen predictive encoder, one stage and far less data, the model leads every size class. Scope and future work. The encoder predicts video rather than reading text, so text recognition and single images are outside our target; hour-long videos, too costly to train on at our scale, are future work.

Acknowledgements

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP, AI Computing Support Project for R&D) grant funded by the Korea government (MSIT) (No. RS-2026-25505492, High-Performance Research AI Computing Infrastructure Support at the 2 PFLOPS Scale).

Appendix

Theory: statements and proofs

Setting. z(v)z(v) are the frozen encoder tokens of video vv; a transformation TT has an answer map πT\pi_T with y(Tv)=πT(y(v))y(Tv)=\pi_T(y(v)). For a binary answer that flips, y(Tv)=−y(v)y(Tv)=-y(v). The answer log-odds at the scoring position is s(v)=ℓP−ℓQs(v)=\ell_P-\ell_Q, the difference of the two option-letter logits; in the linear regime s(v)≈s0+g⋅z(v)s(v)\approx s_0+g\cdot z(v) with g=∂s/∂zg=\partial s/\partial z the readout through bridge and language model; we append a constant coordinate to zz, so that s0s_0 is part of g⋅cg\cdot c. The pair decomposition c=12(z(v)+z(Tv))c=\frac{1}{2}(z(v)+z(Tv)), d=12(z(v)−z(Tv))d=\frac{1}{2}(z(v)-z(Tv)) is a definition. Here z(Tv)z(Tv) is the input the model actually receives, and gg reads tokens at their positions; the encoder analysis of § 3.1 uses the aligned zT(v)=ΠT−1z(Tv)z_T(v)=\Pi_T^{-1}z(Tv) instead, to separate token content from token position. Since g⋅z(Tv)=(ΠTTg)⋅zT(v)g\cdot z(Tv)=(\Pi_T^{\mathsf{T}}g)\cdot z_T(v), the two decompositions give the same mm and dsd_s for a readout that weights all token positions alike (ΠTTg=g(\Pi_T^{\mathsf{T}}g=g, e.g. through pooling) or when TT keeps positions (ΠT=I)(\Pi_T=I); then the subspaces of Proposition 2 apply to the pair loss. A position-dependent readout can in addition compare tokens across positions, the relational route of § 3, which the propositions do not model; our bridge adds position codes, so the propositions describe the position-invariant part of the readout and the rest is measured (Sec. G). The log-odds splits the same way: m=12(s(v)+s(Tv))=g⋅cm=\frac{1}{2}(s(v)+s(Tv))=g\cdot c and ds=12(s(v)−s(Tv))=g⋅dd_s=\frac{1}{2}(s(v)-s(Tv))=g\cdot d.

Proof of Proposition 3. (i) With logistic ℓ(u)=log⁡(1+e−u)\ell(u)=\log(1+e^{-u}), ∂gℓ(y s(v))=ℓ′(y s(v)) y z(v)\partial_g\ell(y\,s(v))=\ell'(y\,s(v))\,y\,z(v) and ∂gℓ(−y s(Tv))=ℓ′(−y s(Tv)) y z(Tv)\partial_g\ell(-y\,s(Tv))=\ell'(-y\,s(Tv))\,y\,z(Tv). Writing a=−ℓ′(y s(v))a=-\ell'(y\,s(v)), b=−ℓ′(−y s(Tv))b=-\ell'(-y\,s(Tv)) and substituting z(v)=c+dz(v)=c+d, z(Tv)=c−dz(Tv)=c-d gives ∇gL=−y[a(c+d)−b(c−d)]=−y[(a−b)c+(a+b)d]\nabla_gL=-y[a(c+d)-b(c-d)]=-y[(a-b)c+(a+b)d]. The two margins are y s(v)=y(m+ds)y\,s(v)=y(m+d_s) and −y s(Tv)=−y(m−ds)-y\,s(Tv)=-y(m-d_s), equal exactly when m=0m=0, and then a=ba=b. (ii) As a function of (m,ds)(m,d_s), L=ℓ(yds+ym)+ℓ(yds−ym)L=\ell(yd_s+ym)+\ell(yd_s-ym). Replacing mm by −m-m swaps the two terms, so LL is even in mm; each term is convex, so LL is convex in mm, and an even convex function is minimized at m=0m=0. If the readout can change m=g⋅cm=g\cdot c without changing ds=g⋅dd_s=g\cdot d, which holds when cc and dd lie in subspaces the readout can address separately (e.g. V+V_+ and V−V_- of Proposition 2), the minimizer over gg therefore has g⋅c=0g\cdot c=0. Without separability. If cc and dd are collinear, mm and dsd_s move together and m=0m=0 is not implied. One-dimensional example: c=2c=2, d=1d=1, y=1y=1, s=gzs=gz, so L(g)=ℓ(3g)+ℓ(−g)L(g)=\ell(3g)+\ell(-g); then L′(0)=−1L'(0)=-1 and the minimizer g∗≈0.454g^*\approx0.454 has m=g∗c≈0.91≠0m=g^*c\approx0.91\ne0. This is why we measure the content share at the answer position (§ 6.2) rather than assume it. Answer-balanced sets. For binary answers and a set of variants zj=c+djz_j=c+d_j of one scene with labels yjy_j and equal error weights aj=aa_j=a, the content coefficient of the summed gradient is −a c∑jyj-a\,c\sum_j y_j, which is zero when the labels are balanced. Multi-class answers are treated as pairs of options and are not covered by that.

Proposition 4 (unpaired supervision admits shortcuts). A single clip’s gradient is −a y(c+d)-a\,y(c+d); if scene type correlates with the answer (E[y c]≠0)(\mathbb{E}[y\,c]\ne0), at equal error weights the readout acquires a content component and a counterfactual pair stays where the content puts it.

Proof of Proposition 4. For a single clip, ∇gℓ(y s(v))=−a y(c+d)\nabla_g\ell(y\,s(v))=-a\,y(c+d). Averaged over the training distribution, the content part is −E[a y c]-\mathbb{E}[a\,y\,c]. At equal error weights (as at the start of training) this is −a E[y c]-a\,\mathbb{E}[y\,c], nonzero whenever scene type and answer correlate, so the readout can acquire the component E[y c]\mathbb{E}[y\,c]; later the weights aa depend on the scores and the term need not vanish. On a counterfactual pair the two videos share cc, so the common term g⋅cg\cdot c is the same for both and pushes them to the same answer: the pair is “stuck”.

Proposition 5 (readability from few pairs, Gaussian model). From a zero readout all pair margins are equal, so the first gradient step on kk pairs is the prototype gk∝∑iyidig_k\propto\sum_i y_i d_i. If d=yμ+ϵd=y\mu+\epsilon with ϵ∼N(0,σ2ID)\epsilon\sim\mathcal{N}(0,\sigma^2 I_D) independent of yy, its accuracy in classifying the change dd of a new pair is ≈Φ(r/r(1+1/k)+D/k)\approx\Phi\left(r/\sqrt{r(1+1/k)+D/k}\right) with r=∥μ∥2/σ2=Dκ/(1−κ)r=\lVert\mu\rVert^2/\sigma^2=D\kappa/(1-\kappa): at fixed DD and kk it increases with κ\kappa, and it is 12\frac{1}{2} at κ=0\kappa=0. Applied to a single clip z~=c+d\tilde z=c+d (with threshold at the mean content cˉ\bar c), the decision uses the content term y gk⋅(c−cˉ)y\,g_k\cdot(c-\bar c), which vanishes when gkg_k is orthogonal to the content variation and is small when the change direction overlaps little with the leading content directions (ρ\rho, Table 22).

Proof of Proposition 5. At g=0g=0 every margin is zero, so a=b=−ℓ′(0)=12a=b=-\ell'(0)=\frac{1}{2} for every pair and, by Proposition 3(i), ∇gL=−12∑iyidi\nabla_gL=-\frac{1}{2}\sum_i y_i d_i: the first gradient step is the prototype gk=1k∑iyidig_k=\frac{1}{k}\sum_i y_i d_i (later steps reweight pairs by their margins, which the proposition does not cover). Under the Gaussian model, gk=μ+ηg_k=\mu+\eta with η∼N(0,σ2kID)\eta\sim\mathcal{N}(0,\frac{\sigma^2}{k}I_D), and for the change of a new pair the decision is y gk⋅d=(μ+η)⋅(μ+ϵ′)y\,g_k\cdot d=(\mu+\eta)\cdot(\mu+\epsilon'), with mean ∥μ∥2\lVert\mu\rVert^2 and variance σ2∥μ∥2(1+1k)+Dσ4/k\sigma^2\lVert\mu\rVert^2(1+\frac{1}{k})+D\sigma^4/k (the term η⋅ϵ′\eta\cdot\epsilon' dominates at large DD). Dividing the mean by the standard deviation gives the stated Φ(r/r(1+1/k)+D/k)\Phi\left(r/\sqrt{r(1+1/k)+D/k}\right), which tends to the Bayes accuracy Φ(r)\Phi(\sqrt r) as k→∞k\to\infty; this is a normal approximation of the decision, which in simulation (k=8,r≤16)(k = 8, r \le16) is within 0.4 points for D≥64D \ge64, the regime of the encoders, and within 2.5 points down to D=1D = 1. With κ=∥μ∥2/(∥μ∥2+Dσ2)\kappa= \lVert\mu\rVert^2/(\lVert\mu\rVert^2 + D\sigma^2), r=Dκ/(1−κ)r = D\kappa/(1-\kappa). At κ=0\kappa= 0, μ=0\mu= 0 and the decision is symmetric about zero, so the accuracy is 12\frac{1}{2}. Symmetry matters: with a skewed residual, κ=0\kappa= 0 does not imply chance. The statement is about this linear estimator; a nonlinear readout of relations between tokens is not covered.

Supervision and the answer layers (stated in § 4). At equal margins the pair gradient of the readout lies in span⁡{d}\operatorname{span}\{d\} (Proposition 3(i)). This is a statement about the readout, not about every update of the language model; whether training raises the cross-video alignment κL\kappa_L inside the model is measured, not derived (Sec. G).

Proof of Proposition 2. If the loss satisfies L(E∘T)=L(ΠT∘E)\mathcal{L}(E \circ T) = \mathcal{L}(\Pi_T \circ E) and EE is a minimizer, then ΠT−1∘E∘T\Pi_T^{-1} \circ E \circ T is also a minimizer. By the identifiability assumption, ΠT−1E(Tv)=ATE(v)\Pi_T^{-1}E(Tv) = A_T E(v) for an invertible ATA_T that acts on token channels, the same at every position, so it commutes with every token permutation. Applying the identity twice gives AT1T2=AT1AT2A_{T_1T_2} = A_{T_1}A_{T_2}, a representation. For T2=1T^2 = 1, AT2=IA_T^2 = I and ATA_T is diagonalizable with eigenvalues ±1\pm1, so RD=V+⊕V−\mathbb{R}^D = V_+ \oplus V_-. Then c=P+z∈V+c = P_+z \in V_+ and d=P−z∈V−d = P_-z \in V_- for every vv. Readout. Let y=sign⁡(w⋅z)y = \operatorname{sign}(w \cdot z) with y(Tv)=−y(v)y(Tv) = -y(v) for all vv, and let ww weight all token positions alike (ΠT⊤w=w\Pi_T^\top w = w), so that w⋅z(Tv)=w⋅ATz(v)w \cdot z(Tv) = w \cdot A_Tz(v); a position-dependent readout can instead compare tokens across positions (the relational route). With A=ATA = A_T this reads sign⁡(w⋅Az)=−sign⁡(w⋅z)\operatorname{sign}(w \cdot Az) = -\operatorname{sign}(w \cdot z). Then w−⋅z=12(w⋅z−w⋅Az)w_- \cdot z = \frac{1}{2}(w \cdot z - w \cdot Az) is a sum of two terms that both have the sign of yy, so w−w_- gives the same answer on every video; and since A⊤w−=−w−A^\top w_- = -w_-, w−⋅c=w−⋅12(I+A)z=12[(I+A⊤)w−]⋅z=0w_- \cdot c = w_- \cdot\frac{1}{2}(I + A)z = \frac{1}{2}[(I + A^\top)w_-] \cdot z = 0. A sign flip alone does not give w⋅Az=−w⋅zw \cdot Az = -w \cdot z pointwise, so ww itself may carry content when the data avoid the boundary. If the relation holds on an open set that meets {w⋅z=0}\{w \cdot z = 0\}, the two linear forms w⋅zw \cdot z and (A⊤w)⋅z(A^\top w) \cdot z vanish on the same hyperplane there with opposite signs, so A⊤w=−λwA^\top w = -\lambda w with λ>0\lambda> 0; (A⊤)2=I(A^\top)^2 = I gives λ=1\lambda= 1, and w=w−w = w_-. The readout lies in the −1-1 eigenspace of A⊤A^\top, which annihilates V+V_+ (it coincides with V−V_- when AA is orthogonal). Speed. If playback rates act as a one-parameter group τa\tau_a, the same argument gives Aτa=exp⁡(aK)A_{\tau_a} = \exp(aK), and a small rate change moves the features by aKzaKz. This direction depends on zz, so the proposition does not give speed a direction shared by all videos (it would for a rank-one KK, which nothing forces); how consistent the speed change is in a real encoder is an empirical question, answered by κ\kappa in Table 22. Order. For tokens that see only a temporally local window, a permutation of segments longer than that window acts on tokens by Π\Pi only (A=IA = I); for encoders with temporal context across segments it also changes token content near the boundaries. Composition. If mirror MM and reversal RR both act as −1-1 on the motion subspace, then AMAR=AMAR=+1A_M A_R = A_M A_R = +1 there, so the composed video reads like the original. We test this in Sec. G. Scope of the assumptions. For V-JEPA the symmetry holds approximately: video and masks are left–right symmetric, while masks are tubes through time and the frame stride is random. § 3.1 therefore tests the consequences, κ\kappa and ρ\rho, on the frozen features directly.

Proof of Proposition 1. (i) If E(v)=[f(x1),…,f(xN)]E(v) = [f(x_1), \ldots, f(x_N)], time reversal maps E(v)E(v) to [f(xN),…,f(x1)]=ΠE(v)[f(x_N), \ldots, f(x_1)] = \Pi E(v), so Π−1E(Tv)=E(v)\Pi^{-1}E(Tv) = E(v) and d≡0d \equiv0. For our SigLIP tokens, which pack two consecutive frames, Π\Pi also swaps the two frames inside each token. Without that swap, as the model receives them, d=12[f(x2t−1)−f(x2t)]d = \frac{1}{2}[f(x_{2t-1}) - f(x_{2t})] per frame block, a scene-dependent frame difference (rd=0.055r_d = 0.055, κ=0.0012\kappa= 0.0012 measured). This is a statement about token content after the positions are aligned: the order of the tokens still differs between vv and TvTv, and a model that compares tokens across positions can recover the arrow of time (the frozen SigLIP probe reads it at 65.3%, see Sec. D). A change of rate selects other frames, d=12[f(xt)−f(xt′)]d = \frac{1}{2}[f(x_t) - f(x_{t'})], a frame difference that depends on scene content; nothing in the objective aligns it across videos, so a shared direction is not implied (measured: κ≈0.002\kappa\approx0.002, Table 22). (ii) The contrastive loss depends on xx only through f(x)⋅g(caption)f(x) \cdot g(\text{caption}). If captions are invariant to mirroring in distribution, the expected loss is the same for ff and f∘f \circ mirror. For unnormalized embeddings the contrastive loss is convex in ff (a log-sum-exp of linear terms), so 12(f+f∘mirror)\frac{1}{2}(f + f \circ\text{mirror}) does no worse on the caption-aligned subspace. For normalized embeddings the average lies on the sphere and the argument does not apply as stated; there we rely on the measured mirror consistency (κ=0.0014\kappa= 0.0014 for SigLIP, Table 22). A minimizer can therefore be mirror-invariant there, and the remaining mirror component is unconstrained residual.

Counterfactual data

Relation to the closest work. Table 9 sets the counterfactual rows against the closest training signals and analyses.

WorkTraining signalWhere the answer comes fromEncoderWhat is analysed
Pretext tasks [7, 19, 45, 61, 65, 68, 71]order, arrow-of-time or speed predictionthe transformationtrained by the tasklearned features
Tracing the Arrow of Time [23]arrow-of-time questions, time-preserving connectorreversalvideo- vs. frame-centricinformation flow, encoder to answer
Which Way Did It Move? [32]projector objective on frame-feature deltas (DeltaDirect), direction instructions2-D motion direction—direction readable at each stage, binding gap, direction concept vectors
TIME [64]answer-balanced temporal instructions, five dimensionsconstructed instructions; reversal for direction—benchmark gains
SynRL [28]CoT fine-tuning and RL on rendered shapesthe rendering program—synthetic-to-real transfer
Ours14 constructions in 6 families on public videos, added to captions and QAthe transformation of the same clip, so content cancelsfrozen predictive vs. frame-wisechange structure (κ\kappa, ρ\rho), stage-wise probes, subspace removal, encoder × supervision

Table 9. The counterfactual rows and the closest training signals and analyses.

Adding a video collection. A new source is a short script: it lists the collection’s clips (path and, for constructions on actions and events, the source’s own action label or time span) and names the constructions to apply. The generator then renders the variants, writes question, options and answer, and applies the answer, option-length and text-prior balancing of

§ 4; no manual annotation and no language model are involved, so any public video collection becomes counterfactual training data.

Table 10 lists all 33 sources with their construction, size and a real question. Figure 6 shows one rendered training row per construction. The sources draw on SSv2, SA-V, Charades-Ego, EPIC-Kitchens, HoloAssist, Kinetics-700, CLEVRER and a physics simulator. Each row is rendered at training time from its specification (clip, window, transformation), so the same file yields the original and its counterfactual without storing extra video. Figure 2 in the main text is a sign-flip row (state change played backwards).

One training row per construction

Figure 6. One training row per construction (4 of 32 frames, after the transformation), with its question and answer.

SourceConstructionRows% mixpaired %Example question (real row)
Sign flip (12 sources, 117,066 rows, 1.47% of the mixture)
tq4_odir_ssv2object direction; clip mirrored or reversed with the answer remapped16,1160.150In which direction does the roll move?
tq4_cdir_ssv2camera direction under mirroring / reversal5,8950.050What is the camera motion in the video?
tq4_dir2_ssv2two concatenated clips, each optionally reversed: direction A then B7,2040.070How would you describe the bowl’s directional movement?
tq4_state_ssv2state-change action reversed: open/close, put/take24,6590.230Which action is performed in the video?
chego_revegocentric actions with reversible verbs, clip reversed16,7800.267Which action does the person perform?
epic_revkitchen actions with reversible verbs (on/off, open/close)3,4060.050What am I doing in this video?
chego_stateend state of a reversible action (open/closed)4,3830.074What state is the closet / cabinet in at the end?
camera_clevrersynthetic camera move (pan, tilt, zoom, static) over a static scene5,0000.080How does the camera move in this video?
camera_chexoas above, egocentric source3,9270.060Which camera movement is shown?
camera_chexo_q2camera move, second question template3,6680.060How does the camera move as the scene unfolds?
camera_clevrer_q2camera move, second question template4,6620.070Is there any camera movement in the video?
v17_finemotion_dir_ssv2fine motion direction of object or camera21,3660.330Is the camera panning to the left?
Order (4 sources, 85,579 rows, 0.92% of the mixture)
tq4_order_ssv2two events concatenated or stacked: which first / simultaneous29,6480.280What is happening first in the video?
tq4_order_llavaas above, open-domain clips20,0000.190What is happening in the first event of the video?
tq4_order_natnatural event order within one clip15,6610.150What order do the events occur in?
order_capmatchtwo clips joined; a caption names two actions in order: which clip it describes20,2700.3180The video has two segments separated by black frames. Which . . .
Position (4 sources, 48,386 rows, 0.74% of the mixture)
loc_compthree clips joined: where does the action occur23,7000.3690During which part of the video does the action ’wash the . . .
v15_half_k700_acttwo clips around a black gap: what happens in the first/second part5,9720.095What happens in the first part of the video (after / before . . .
v15_whichseg_k700_actwhich of two segments shows an action5,9620.095The video has two segments separated by black frames. Which . . .
loc_comp_q2localization, second question template12,7520.2086When does the action ’wash the knife’ occur in the video?
Magnitude (8 sources, 141,084 rows, 1.58% of the mixture)
tq4_rspeed_ssv2one clip twice, above/below, over different spans: relative speed30,0000.280Compared to the video below, how fast is the tea bag above . . .
tq4_rspeed_savas above, SA-V source18,0000.170What is the speed of the video above compared to the video . . .
tq4_dspeed_ssv2span split 1/3 : 3/1 : 1/1: accelerates, decelerates, constant30,0000.280What is the empty plastic container’s speed pattern?
tq4_dspeed_savas above, SA-V source18,0000.170How does the speed of the motion change?
v16_rspeed_w---tq4_rspeed_savrelative speed (2:1 / 3:1 / equal), varied question wording7,9260.120What is the speed of the video above?
v16_rspeed_w---tq4_rspeed_ssv2relative speed (2:1 / 3:1 / equal), varied question wording7,9680.120What is the speed of the bottle above?
v16_tempo---epicabsolute playback rate: slow motion / normal / time-lapse11,7900.180What is the relative speed of the video compared to real-time?
v17_finemotion_speed---ssv2fine relative speed of the same object17,4000.270How does the speed of the men’s inner vest above compare to . . .
Count (3 sources, 35,614 rows, 0.55% of the mixture)
rep_loopone action segment repeated k times: how many times12,0000.1886How many times does the person load the inside water . . .
v15_ndistinct_k700_actsegments of 1-4 distinct activities: how many11,7760.18100How many distinct activities appear in this video?
v16_rep6---rep_looprepetitions up to 711,8380.1886How many times does the person insert the hex socket head?
Stop/cut (2 sources, 81,464 rows, 0.85% of the mixture)
plausibilityphysically impossible edit (teleport, vanish, ...) vs. its unedited render65,8000.61100Is the motion of the objects in this video physically . . .
chego_doneclip truncated before the action completes15,6640.245By the end of the clip, has the person finished opening a . . .

Table 10. All 33 counterfactual sources of the training mixture (509,193 rows, 6.11% of sampled rows). Paired %: share of a source’s rows whose counterpart from the same clip with a different answer is also in that source (29.4% overall); the other rows are single members of answer-balanced constructions (§ 4).

How each construction is rendered. All constructions are rendered on the fly by the training loader from a source clip (and, where noted, a second clip), at 32 frames, with frame times replaced by even spacing so that the answer cannot be read off time codes.

C1. State change played backwards (sign flip). Actions with an inverse (open/close, put/take, turn on/off) from SSv2, Charades-Ego and EPIC; the clip is played forward or reversed, and the answer becomes the inverse action when reversed. A variant asks for the end state.

C2. Object direction under mirroring or reversal (sign flip). SSv2 clips with a direction in the template (left/right, up/down, towards/away). The clip is mirrored (left↔right) or reversed (all directions flip), and the answer is remapped accordingly.

C3. Camera move (sign flip). A square window of 0.7 of the short side moves over static footage (CLEVRER, Charades-Ego): pan left/right, tilt up/down, zoom in (window shrinks to 60%) or out, or stays static. The amount travelled is drawn per row, and SSv2 camera-direction clips are additionally mirrored or reversed.

C4. Two clips joined, each optionally reversed (sign flip). The first and second halves come from two SSv2 clips; the answer is “direction A then direction B”.

C5. Event order (order). Two clips are joined in sequence or stacked above and below in the same time span (simultaneous). The answer is first, second, or at the same time, and natural clips with annotated event order are added. For simultaneous rows the layout itself shows both events at once; the order answers (first or second) are the ones only the change decides.

C6. Caption–segment matching (order). Two HoloAssist segments are separated by black frames, and a caption names two actions in order; the matching segment is the answer.

C7. Temporal position (position). Three clips are joined; the queried action sits in one, and the answer is beginning, middle or end. Variants place two Kinetics clips around a black gap and ask what happens in a part or which part shows an action.

C8. Relative speed (magnitude). One clip is shown twice, above and below, with the frames of each copy spread evenly over a span [a,b][a,b] of the clip; the copy with the longer span moves faster. The spans are nested, the shorter one centred inside the longer one, so both copies show the same motion and the longer span plays more of it in the same number of frames; the audit below checks that apparent motion follows the label on every faster and slower row. Swapping the spans flips the answer, and equal spans give “the same”. Span ratios are 2:1, 3:1 or equal; the fine-motion source compares spans with ratios between 1.0 and 1.4.

C9. Speed change (magnitude). The time span is split 1:3, 3:1 or 1:1 between the two halves of the frames, so the second half moves three times as fast, a third as fast, or as fast as the first.

C10. Playback rate (magnitude). The frames are spread over a short, normal or long window: slow motion, normal or time-lapse.

C11. Repetitions (count). One action segment is repeated kk times (k≤7k \le7), and the answer is kk.

C12. Distinct activities (count). One to four segments of different Kinetics activities are joined, and the answer is their number.

C13. Completion (stop/cut). The clip is cut before or after the end of an annotated action, and the answer is complete or incomplete.

C14. Physical plausibility (stop/cut). Simulated scenes are rendered with and without an impossible edit (teleport, vanishing object, …); both versions are in the data, so every row is paired.

Across all constructions the generator balances answer letters, the rank of the correct option by length, and each option’s prior under the text-only language model.

Label audit for speed. The speed constructions manipulate playback, and the questions ask about apparent motion. On 300 random training rows of each SSV2 speed source, rendered by the training loader, we use the mean absolute difference between consecutive frames as a proxy for apparent motion (top versus bottom panel; second versus first half). For relative speed it points the way of the label in 100% of “faster” and “slower” rows (median motion ratio 1.83 and 0.52), and 92.5% of “same” rows fall within a ratio of 0.8–1.25; for speed change it agrees on 97.7% of accelerating and 95.0% of decelerating rows (median ratios 2.53 and 0.39). Speed-change labels assume the source moves at a roughly steady rate over the clip; the audit confirms the label on 95–98% of these rows. Relative-speed labels are the playback-rate ratio the rendering sets for two playbacks of the same clip, and apparent motion follows them on every audited faster and slower row; absolute-rate rows (slow motion / normal / time-lapse) take the source as recorded at natural speed. The readout carries over to speed items that external benchmarks build with their own pipelines: there CF adds 32.8 points at 8k and 38.1 at the end (Table 26).

Training and evaluation details

Released models. The six models are released as Llama-V-JEPA-LM-Qwen3-1.7B, -Qwen3-4B, -Qwen3-8B, -Llama3.2-1B, -Llama3.2-3B and -Llama3.1-8B (Ours-2B-Q, -4B-Q, -8B-Q, -1B-L, -3B-L, -8B-L), with weights, inference code and model cards (https://huggingface.co/collections/kkt94/llama-v-jepa-lm-6af41fb2a548b444c6255d63); code and the technical report are at https://github.com/kkt94/Llama-V-JEPA-LM.

Training curves and metrics. Figure 7 plots the training loss, the gradient norm and the language model’s learning rate of the released models against samples seen, and Table 12 lists throughput, hours, memory and loss. Each model is trained once, in one continuous stage. The loss falls steadily without divergence and ends lower for larger language models, from 0.43 (1B) to 0.38 (8B); the Qwen3 and Llama models of the same size end within 0.02 of each other. The language model’s learning rate is zero during the bridge-only warm-up, then decays by cosine.

Spectral bridge. The frozen encoder’s states are layer-normalized per level, pooled from the 24×2424 \times24 patch grid to 8×88 \times8 cells and kept per tubelet, so 32 frames give 16 tubelets × 64 cells = 1,024 tokens. Each cell state is centered, rotated to its principal axes and scaled by one global factor to unit mean variance; the statistics are computed once on 2.3M cell states from 4,500 training-mix clips and then fixed. ViT-L (1B, 2B) keeps all 1,024 directions; ViT-g (3B–8B; four levels, 5,632 dimensions) keeps the leading 2,048. A trainable MLP (LayerNorm, linear to 2,048, GELU, linear to the language-model width) maps each cell to a token, and learned codes for tubelet, row, column, time stamp and cell geometry are added. The SigLIP arm uses the same pipeline: each frame is encoded on its own, and the two frames of a V-JEPA tubelet form the two levels of one token (2,304 dimensions, leading 2,048 kept), so frames, token layout and token count are identical.

Mixture. The 175 sources comprise open video caption and QA collections—LLaVA-Video [78], ShareGPTVideo (LLaVA-Hound) [77], PerceptionLM data [12], the VideoChat-Flash and TimeIT mixtures [39, 52], ActivityNet-QA and ActivityNet Captions [30, 73], STAR [66], VidLN [60] and the PerceptionTest training split [47]—; questions built from the original labels of SSv2 [21], Kinetics-700 [10], Charades-Ego [56], EPIC-KITCHENS-100 [14], HoloAssist [61], EgoDex [25], SA-V [51], CLEVRER [72], RoboVQA [54], Language-Table [43], Oops! [17], Countinu [16] and HT-Step [1]; text-only replay on UltraChat prompts [15]; and the 33 counterfactual sources. Every source is used through its training split. Video encoders and language models are used as released [22, 46, 70, 74]. Sampling weights fix each source’s share of a step: the 4.53M drawn rows come from 7.2M unique rows (a median source row is drawn 0.7 times), and the 509k counterfactual rows give about 277k draws.

Evaluation. Every model, ours and open, is evaluated by the same harness: 32 uniformly sampled frames, the benchmark’s question and options with the instruction “Answer with the option’s letter from the given choices directly”, and scoring on the option letter. Training samples in Figure 3 are as each paper reports them: InternVL3.5 116M pre-training + 56M SFT [27]; PLM 1M + 72M + 19M [12]; VideoLLaMA3 the sum of its four stage tables [75]; Molmo2 its SFT mixture, since its image pre-training count is not reported, so its 9.3M is a lower bound [2]; Qwen3-VL reports 2.2T pre-training tokens [49] and is not plotted (58.4), and Meta’s V-JEPA 2 LLM uses 88.5M samples (Table 5). Open models use their released instruction-tuned weights from the Hugging Face Hub, their own processors and their own chat templates at 32 frames, with thinking off where a template offers it (Gemma 4); our models use their chat template without a thinking block. Scoring takes the next-token logits at the first assistant position and picks the option letter with the highest logit among the item’s options, with no generation and no answer parsing, so every model gives a valid answer for every item. As a check, greedy generation (16 tokens) with answer parsing gives the same prediction as the letter logits on 100% of TOMATO and MVBench items for Qwen3-VL-4B, Qwen3-VL-8B, InternVL3.5-4B and PLM-3B and on 98.7–99.5% for Ours-4B-Q and Ours-8B-Q, with no unparsed answer; the four open models’ scores are unchanged and ours move by at most 0.3 (TOMATO 53.2 and 56.6, MVBench 72.0 and 72.8). Table 14 sets the numbers the authors report, under their own frame settings and splits, beside ours: of 51 reported scores, 40 lie within 3 points of our measurement (median difference 1.3). Resampling the items of every benchmark (whole pairs for pair metrics; 1,000 draws, the same draws for every model) gives 95% intervals of the 12-score average of ±0.68\pm0.68–0.750.75 points for every model in Table 4, so the margins over the best open model of each class (5.3 to 13.6 points) lie far outside them. Pair metrics (MVP, Vinoground) count a pair as correct only if both members are.

Training splits and a train–evaluation audit

Every source enters training with the training split its authors release; no evaluation split of any benchmark is used. Public collections reuse source videos, so we also matched video identifiers (normalised file names; YouTube identifiers without time suffixes) between all 175 training sources and all 35,488 evaluation items (Table 15). Nine of the twelve sets share no video with training. MVBench and TVBench reuse Charades clips through STAR that LLaVA-Video-178K also contains, and from which one counterfactual source draws clips; TVBench also reuses PerceptionTest training videos; TemporalBench shares YouTube videos with Kinetics-700. In total 147 items (0.41%) are affected, and removing them changes every model by at most 0.42 points on a benchmark and 0.04 on the twelve-score average, ours and the open models alike. Video LLM reports rarely give such an item-level account of their training data; we report ours in full, and the released manifests list every training clip. TemporalBench uses the binary short-QA split. SSv2

BenchmarkItemsSharedShared source videos
MVBench4,00060 (1.5%)Charades (STAR) clips also in LLaVA-Video-178K
TVBench2,52557 (2.3%)the same Charades clips; PerceptionTest train
TemporalBench2,76530 (1.1%)YouTube videos also in Kinetics-700
The other nine sets26,1980–
All twelve35,488147 (0.41%)

Table 15. Video-level audit of all 175 training sources against every evaluation item. Rescoring without the shared items moves each model by at most 0.42 points on a benchmark and 0.04 on the twelve-score average, for ours and the open models alike.

is posed as multiple choice over action labels: 12 validation clips for each of the 174 action templates (2,088 items), with four options, the true template and the three most similar other templates by word overlap (e.g. “from left to right” against “from right to left”), so the options differ mainly in the action or its direction; the correct letter is spread about evenly over A–D. Visual tokens per 32-frame clip, measured with each model’s processor on 21 benchmark clips from 270p to 1080p:

ModelMVBPTTVBTCTOMMoBNQAMVPTBSSv2VG-TVG-VAvg
Gemma 4 E2B39.846.336.451.518.741.151.48.161.437.024.022.436.51
Gemma 4 E4B46.552.041.557.422.849.464.716.561.043.826.424.042.17

caption: Gemma 4 under the common protocol (thinking off, its own processor and chat template). Its pair-accuracy scores fall far below those of the other open models of its size (MVP 8.1 and 16.5), so it is listed here rather than in Table 4; it would rank last in its class either way.

:::

ours 1,024; Gemma 4 2,048–2,112; Molmo2 2,592; VideoLLaMA3 1,006–3,840 (mean 3,168); PLM and InternVL3.5 8,192; Qwen3-VL 1,920–11,664 (mean 7,801), growing with resolution.

Held-out sets. Table 17 defines the six sets. Each group is one source clip under all its variants, so a set tests one novelty with clip content held fixed. Every chosen source video (by file name; for simulation, the whole run) is absent from every training manifest and every evaluation manifest, and the six sets use disjoint clips.

Successive playback. Training rows convey a relative speed by stacking two playbacks of a clip and an acceleration by a continuous time warp. Table 20 keeps the 180 clips and unseen ratios (1.6, 2.5, 4.0) of the speed-ratio set but changes the construction: the clip is played twice one after the other in one 32-frame input, its short central span and then its full span, or the reverse, so the motion speeds up or slows down halfway through. No training row builds speed this way. The question and options are the acceleration family’s (“How does the motion change over the video?”; speeding up, slowing down, not moving). Our six models reach 92.5–99.2 (pair 85.0–98.3) against 19.4–51.9 for open models, whose pair accuracy of 0.6–21.7 (chance 11.1) shows that they mostly give both orders of a clip the same answer; without CF the V-JEPA arm stays at 39.4.

Single-frame control. With the middle frame repeated (no motion, layout and composition unchanged), the six held-out sets drop from 95.0 to 37.9 (2B) and from 96.3 to 37.6 (4B), against a chance level of 33.3: layout and single-frame cues do not solve them.

Inference cost. Table 21 separates the vision encoder, the connector and the language model; all models are timed and their FLOPs counted by the same script on the same two benchmark clips (MVBench, 480p; TVBench, 720p). The two parts behave differently. The encoder’s cost is a property of the chosen encoder: V-JEPA 2.1 attends jointly over all 9,216 tubelet patches of a 32-frame clip (attention is 60% of its 13.9 TFLOPs for ViT-L and 51% of 37.8 for ViT-g), and it is the same for every recipe on that encoder and every language-model size. The open models use frame-wise image encoders (SigLIP-2-based in Qwen3-VL [49, 59], InternViT in InternVL3.5, PE in PLM) that attend within one frame or frame pair at a time, so their encoder cost grows linearly with the number of frames and no attention crosses time. Qwen3-VL-2B’s encoder has the depth and width of our ViT-L (24 layers, width 1,024) and takes 35 ms against 60 ms at 480p; the difference is V-JEPA’s joint space-time attention, the property that carries the dynamics (§ 3.1). This alone orders the 480p first token in the 2B class (Qwen3-VL-2B 66 ms, Ours-1B-L 70, Ours-2B-Q 83); at 720p Qwen3-VL’s token count grows and the order reverses (308 against 71 and 82). The language model’s cost is set by the number of visual tokens and grows with its size; this is the part our design reduces, since the bridge passes 1,024 tokens per clip at any resolution. Per visual token, prefill costs 2.1–14.6

GFLOPs from 1B to 8B, so the saving over models with 8,192 tokens grows with scale: 22–37 TFLOPs at 2B, 62–96 at 4B and 136–145 at 8B, each more than our encoder’s 14 or 38.

Encoder measurements

Table 23 lines up, for each family, the transformation, the theoretical prediction, the measured κ\kappa and the gain from counterfactual supervision on the matching ability.

Linear probes on frozen features (time direction / speed / motion direction) give SigLIP 65.3 / 73.5 / 97.4, ViCLIP 56.9 / 70.3 / 97.5, VideoMAE 65.1 / 76.8 / 95.7 and V-JEPA ViT-L 80.8 / 81.2 / 98.8. V-JEPA reads time direction and speed best. These are two-class probes for the three pairs of § 3.1 (a clip against its reversal, its half-speed copy or its mirror image); the speed-ratio items of Table 3 are three-way between ratios, where a frozen SigLIP probe reads 36.4 and V-JEPA 86.7. Prototype readout (proto 8 / all): the readout ww is the mean of ydˉy\bar{d} over 8 (or all) fitting pairs, with dˉ\bar{d} the token-averaged change, the first gradient step of Proposition 5; it is applied to single held-out clips (token-averaged zz, threshold at the fitting clips’ mean content), so it tests whether a direction read from few pairs classifies new videos; this includes the content term of Proposition 5, which ρ\rho bounds. ViT-g gives 79.7 / 84.7 / 98.6. Motion direction is solvable from the first time step’s position alone for both encoders, so it does not separate them.

Encoder supervision in full

Table 24 lists every checkpoint and benchmark of the four 2B arms. Table 25 pools four checkpoints per comparison to average out checkpoint noise. Table 26 lists all fourteen ability groups. The speed group combines two benchmarks, and the gain holds on each: at the final step CF adds 62.9 points on TOMATO’s speed items (210; [+56.2, +69.5]) and 21.8 on TempCompass’s (317; [+14.8, +28.7]) for V-JEPA, against 41.9 and 0.3 for SigLIP.

StepMVBPTTVBTCTOMMoBNQAMVPTBSSv2VG-TVG-VAvg
SigLIP caps+QA
2k51.659.249.259.623.644.066.812.852.267.334.823.245.36
4k55.562.850.462.625.946.368.630.454.671.648.623.650.07
6k59.965.556.763.326.947.570.636.155.273.050.025.052.48
8k62.765.858.863.126.948.171.737.758.174.849.024.853.46
12k64.568.260.165.329.548.772.844.957.979.550.625.655.63
16k66.469.762.264.930.348.674.147.658.680.053.625.456.78
20k66.571.464.165.532.349.575.147.361.484.650.824.257.72
24k67.872.863.265.130.950.875.648.360.784.951.625.658.11
final67.772.564.465.633.050.875.648.261.684.551.826.658.52
SigLIP +CF
2k50.259.647.860.627.442.165.318.947.168.144.224.646.32
4k55.462.955.863.326.244.170.129.154.769.344.633.650.76
6k57.564.456.665.134.944.570.737.857.473.148.634.853.78
8k61.566.658.866.332.346.572.442.758.678.657.040.056.77
12k63.267.759.465.933.548.172.843.556.780.451.240.056.87
16k66.470.863.067.537.750.174.750.061.783.754.042.660.18
20k66.371.964.667.438.549.475.150.661.383.552.445.860.57
24k67.772.466.268.838.950.775.752.263.085.354.045.861.73
final68.371.967.670.038.950.476.053.163.484.454.846.862.13
V-JEPA caps+QA
2k53.959.653.357.026.643.963.825.054.078.538.023.448.08
4k59.164.659.062.527.848.767.835.058.682.643.823.452.74
6k62.167.663.363.430.549.068.940.558.183.551.425.855.34
8k64.268.267.663.231.750.270.342.158.886.354.622.656.65
12k66.570.365.165.135.251.971.646.660.088.454.625.658.41
16k68.271.466.765.937.853.172.948.561.189.060.423.059.83
20k68.972.970.267.537.553.473.549.962.590.358.825.060.87
24k69.373.470.766.738.454.774.250.362.990.758.825.261.27
final69.974.271.766.537.554.174.451.763.790.258.426.061.53
V-JEPA AoT only
8k63.568.262.863.529.649.469.440.661.683.658.825.456.37
final69.673.669.066.136.853.873.850.965.589.162.023.861.17
V-JEPA temporal QA in place of CF
8k65.969.166.564.537.053.170.944.960.085.257.224.258.21
V-JEPA +CF
2k49.560.653.161.334.344.360.530.354.375.550.829.250.31
4k58.265.563.264.938.745.966.939.756.979.853.237.455.86
6k59.166.360.667.345.648.268.642.059.481.456.046.658.43
8k63.069.763.469.648.649.569.945.961.486.661.445.261.18
10k64.570.864.669.549.852.470.947.963.387.460.648.262.49
12k64.770.364.969.948.051.371.246.359.688.560.637.661.07
14k67.172.567.871.748.354.071.851.562.389.361.238.863.02
16k68.373.067.073.252.052.572.851.262.689.663.648.664.53
18k68.473.367.771.549.653.273.153.862.889.564.247.864.58
20k68.074.067.373.050.353.973.355.564.290.062.848.265.04
22k69.374.370.073.249.654.373.854.964.390.766.848.865.83
24k69.375.069.674.250.453.874.354.663.191.163.844.865.33
26k69.675.270.473.649.454.274.154.564.491.564.647.465.74
final69.575.370.773.550.954.374.054.663.991.663.250.866.02

Table 24. Encoder × supervision, every evaluated checkpoint (2B, global batch 168). The V-JEPA caps+QA arm was trained to 8k first and then continued to the final step; the AoT-only arm likewise, and the temporal-QA arm was trained to 8k. Temporal QA: the counterfactual rows removed and their weight given to the 22 annotation-based temporal-QA sources of the mixture (temporal grounding, highlights, step order, before/after, next action, repetition counting, unintentional action).

4B control. The 4B model trained without the counterfactual rows, otherwise identical (mixture, schedule, global batch 150), to 8k: twelve-score average 61.28 against 66.06 with them (+4.8 [+4.0, +5.5], paired bootstrap), speed items +29.4 [+24.5, +34.2], held-out sets 61.4 against 93.0 (unseen ratios 35.7 against 93.3), and successive playback (Table 20) 33.1 against 90.0 (pair 10.6 against 80.0).

ModelSinglePair
Open models
Qwen3-VL-2B33.18.9
Qwen3-VL-4B36.91.1
InternVL3.5-4B24.20.6
Molmo2-4B19.44.4
PLM-3B29.22.8
Qwen3-VL-8B51.921.7
InternVL3.5-8B22.51.7
Molmo2-8B43.18.9
PLM-8B49.216.7
VideoLLaMA3-7B35.010.6
Controlled 2B arms (Table 6)
V-JEPA caps+QA, final39.46.7
Ours (final checkpoints)
Ours-2B-Q93.987.8
Ours-1B-L92.585.0
Ours-4B-Q96.793.3
Ours-3B-L95.891.7
Ours-8B-Q93.987.8
Ours-8B-L99.298.3

Table 20. Successive playback (360 items, 180 clips): the speed-ratio set’s clips and unseen ratios, the clip played twice one after the other, asked whether the motion speeds up or slows down. Single: chance 33.3; pair: both orders of a clip correct, chance 11.1.

AoT-only arm. The 33 counterfactual sources are replaced, at their total weight, by one source of 60k rows: 30k SSv2 training clips from the pool the counterfactual sources already use, each forward and reversed, with “Is this video playing forward or in reverse?” or one of four rewordings and two shuffled options (answers balanced); everything else is the V-JEPA arm’s recipe, trained to 8k and then to the final step like the caps+QA arm. On 2,088 unseen SSv2 validation clips, each forward and reversed, it answers this question correctly on 95.9% of videos and on both videos of 93.3% of clips at the final step (94.4 and 90.5 at 8k; caps+QA: 50.0), so it learns its target; the benchmarks do not improve (−0.4 [−1.0, +0.2] against caps+QA at the final step, paired bootstrap; Table 6), and speed items stay at the caps+QA level (34.0 against 33.4 at the end).

Speed in the caption and QA mixture. Without the counterfactual rows the mixture is not silent about speed: in the temporal-QA arm’s mixture, 1.6% of drawn rows contain faster, slower, quicker, speed, slow motion, time-lapse or accelerat-/decelerat- (mostly LLaVA-Video captions and SSv2 descriptions), about 21k rows over the first 8k steps. The answer still stays at chance on unseen speed ratios (36.1, Table 18): the words are there, but nothing ties the answer to the change.

Counterfactual consistency in full

Internal measurements

All measurements use clean pairs and the exact inputs and answer position of evaluation, with no generation (2B, 8k steps). Share is Esd2/(Esd2+Esm2)\mathrm{E}^{d^2}_{s}/(\mathrm{E}^{d^2}_{s}+\mathrm{E}^{m2}_{s}) of the answer log-odds over a pair set (Sec. A); stuck is the fraction of pairs given the same answer for both videos. Leak is the variance of the content along the answer direction u=norm⁡w⊙(WP−WQ)u=\operatorname{norm}_{w}\odot(W_{P}-W_{Q}), relative to the change.

Removing change subspaces. Table 29 removes the change directions measured on frozen features from every visual token at inference, with random and content subspaces as controls.

ModelRemoved subspacedimsenergyratioheld-out speed dom.held-out speed comb.held-out direction dom.held-out direction comb.cameraTOMTCNQAPT
Ours-2B-Qnone0098.798.598.975.798.599.850.973.574.075.3
speed816.691.577.865.775.298.598.846.470.173.273.8
random, energy-matched2416.296.795.597.274.898.199.249.971.773.274.3
reversal86.096.997.898.972.497.999.450.371.674.075.2
mirror83.098.798.799.474.497.699.650.074.274.175.2
speed + reversal + mirror2421.581.374.763.071.996.998.446.068.973.173.8
random242.498.198.399.175.497.999.850.672.574.074.7
top content PCs2443.695.085.084.375.697.496.544.868.172.472.0
V-JEPA caps+QAnone0033.344.349.169.897.192.437.566.574.474.2
speed + reversal + mirror24–33.543.550.259.889.276.333.664.973.772.6

Table 29. Removing change subspaces at the bridge input (final 2B checkpoints, no retraining). The subspaces are estimated once in the equalised space of Ours-2B-Q from 400 SSv2 training clips: the top 8 directions of the token-averaged response to playing at twice the speed, to reversal and to mirroring (on the other 20% of clips they capture 46%, 57% and 74% of that response); every visual token is projected off the subspace. Energy: share of token energy removed (%). Energy-matched random: a random 24-dimensional subspace inside a band of content principal components with the same removed energy as the speed subspace. Speed removal takes the unseen speed combinations from 98.9 to 65.7 and TOMATO from 50.9 to 46.4, while the energy-matched random subspace leaves them at 97.2 and 49.9; content benchmarks move by at most 1.5 in both. The model trained without counterfactual rows is at chance on speed with or without the subspace.

Training rows themselves. On the training distribution, held-out rows of each construction have answer loss as follows (without →\rightarrow with CF):

  • relative speed: V-JEPA 0.23 →\rightarrow 0.02, while SigLIP stays at 0.22 →\rightarrow 0.22, failing even on the training construction;

  • direction and camera: 0.10 →\rightarrow 0.03 and 0.23 →\rightarrow 0.12;

  • order, position and count: V-JEPA 0.16 →\to 0.07 and SigLIP 0.18 →\to 0.07, the same for both encoders, as Proposition 2 predicts for signals in token positions.

Explicit pairs versus unpaired rows

Construction. All counterfactual rows form one pool. A scene is the footage a row shows (an SSv2 or Kinetics clip, a CLEVRER or simulation video, or one-second bins of a long video); two rows share a scene if their footage overlaps, and two rows are counterparts if they share a scene, belong to the same question family and have different answer classes. Unpaired: rows are taken in a seeded random order, always for the (source, answer class) furthest below its original share, and a row is kept only if none of its scenes is used yet, so every scene appears in at most one row. Paired: connected groups of counterparts are taken whole, in random order, while every (source, answer class) stays within the unpaired arm’s count; rows without counterparts fill the rest. Both arms thus have the same number of rows per source and per answer class (217k in total, 43% of the original) and the same per-source weights; they differ in which rows are drawn. Counterparts are counted here by shared footage, also across sources (35.9% of the full pool’s rows have one, against 29.4% within a source in Table 10); in the paired arm 29.4% of rows have their counterpart (66.8% for direction, 100% for physics), in the unpaired arm none do.

  • Twelve-score averages: 61.56 and 60.27 (8k steps), against 61.18 for the full mixture.

  • Ability differences (paired − unpaired): state change +4.5, reversal/antonym +4.6, TemporalBench +2.8, speed +3.8 and count −3.1.

  • Held-out sets: both average 92.1.

  • Unseen speed-ratio pairs: both answers are correct for 96.7% against 84.4%, with dynamic shares 0.978 / 0.976.

The comparison is between two ways of selecting rows at the same per-source and per-class budget; the unpaired arm covers 22% more distinct videos (194k against 159k), so coverage favors it. The dynamic shares are nearly equal, so the paired arm’s gain is in consistency across a pair rather than in a larger differential share. The speed sources contain no explicit pairs in either arm (Table 10), so the speed gains of the full method come from answer-balanced construction. At equal error weights the gradient’s content term cancels in expectation in both models and within each scene with explicit pairs, which pays off where scene and answer correlate in natural video.

Scope and limitations

The aim of this work is a light video-language model that reads the dynamics a predictive world-model encoder already holds, together with the analysis that explains when and why this works. The study is scoped to that aim. Short clips. All models read 32 frames per clip, and all benchmarks pose questions about clips of seconds to a few minutes. Hour-long video understanding, which needs memory, retrieval across many clips and long-context training data, is a different problem, and training for it at our model sizes and budget would be impractical. The compact 1,024-token clip representation is a natural building block for it, which we leave to future work.

Image question answering. We do not target image question answering (documents, charts, OCR, general image QA). V-JEPA is trained only to predict masked regions of video in latent space, never to align with or read text, so text recognition lies outside what the encoder was built for; performance there is driven by very large image–text corpora. For the same reason, open models trained on 2 to 38 times more data keep an edge on knowledge-heavy video benchmarks such as NExT-QA and PerceptionTest (Table 4).

Data and compute scale. Each model is trained once on 4.5M public samples, with at most 650 H100-hours. Training and evaluating at the scale of general-purpose VLMs, with hundreds of millions of samples or trillions of tokens including long video and large image QA collections, is outside both the purpose of this study and the resources available to it. The results indicate where such scale would pay off: the counterfactual rows are generated from public videos without new manual annotation, make up 6% of the mixture, and, for the encoders compared here, their benefit is anticipated by measurements on the frozen encoder.

References

  1. [1]Triantafyllos Afouras, Efrosyni Mavroudi, Tushar Nagarajan, Huiyu Wang, and Lorenzo Torresani. HT-Step: Aligning instructional articles with how-to videos. In NeurIPS Datasets and Benchmarks, 2023.DOI
  2. [2]Ai2 Molmo Team. Molmo2: Open weights and data for vision-language models with video understanding and grounding. arXiv preprint arXiv:2601.10611, 2026.
  3. [3]Mahmoud Assran et al. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, 2023.arxiv.org/abs/2301.08243
  4. [4]Mido Assran et al. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025.arxiv.org/abs/2506.09985
  5. [5]Piyush Bagad, Makarand Tapaswi, and Cees G. M. Snoek. Test of time: Instilling video-language models with a sense of time. In CVPR, 2023.arxiv.org/abs/2301.02074
  6. [6]Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471, 2024.arxiv.org/abs/2404.08471
  7. [7]Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T. Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. SpeedNet: Learning the speediness in videos. In CVPR, 2020.arxiv.org/abs/2004.06130
  8. [8]Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the “video” in video-language understanding. In CVPR, 2022.arxiv.org/abs/2206.01720
  9. [9]Mu Cai et al. TemporalBench: Benchmarking fine-grained temporal understanding for multimodal video models. arXiv preprint arXiv:2410.10818, 2024.arxiv.org/abs/2410.10818
  10. [10]Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the Kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019.arxiv.org/abs/1907.06987
  11. [11]Delong Chen, Mustafa Shukor, et al. VL-JEPA: Joint embedding predictive architecture for vision-language. arXiv preprint arXiv:2512.10942, 2025.arxiv.org/abs/2512.10942
  12. [12]Jang Hyun Cho et al. PerceptionLM: Open-access data and models for detailed visual understanding. arXiv preprint arXiv:2504.13180, 2025.arxiv.org/abs/2504.13180
  13. [13]Daniel Cores et al. Lost in time: A new temporal benchmark for VideoLLMs. arXiv preprint arXiv:2410.07752, 2024.arxiv.org/abs/2410.07752
  14. [14]Dima Damen et al. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100. IJCV, 2022.DOI
  15. [15]Ning Ding et al. Enhancing chat language models by scaling high-quality instructional conversations. In EMNLP, 2023.arxiv.org/abs/2305.14233
  16. [16]Debidatta Dwibedi, Yusuf Aytar, Jonathan Thompson, Pierre Sermanet, and Andrew Zisserman. Counting out time: Class agnostic video repetition counting in the wild. In CVPR, 2020.arxiv.org/abs/2006.15418
  17. [17]Dave Epstein, Boyuan Chen, and Carl Vondrick. Oops! predicting unintentional action in video. In CVPR, 2020.arxiv.org/abs/1911.11206
  18. [18]David Fan et al. Scaling language-free visual representation learning. In ICCV, 2025.arxiv.org/abs/2504.01017
  19. [19]Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. In CVPR, 2017.arxiv.org/abs/1611.06646
  20. [20]Gemma Team. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026.
  21. [21]Raghav Goyal et al. The “something something” video database for learning and evaluating visual common sense. In ICCV, 2017.arxiv.org/abs/1706.04261
  22. [22]Aaron Grattafiori et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  23. [23]Peitao Han, Fei Cheng, Lis K. Pereira, Qianying Liu, and Shigeru Kitazawa. Tracing the arrow of time: Diagnosing temporal information flow in video-LLMs. arXiv preprint arXiv:2605.07568, 2026.arxiv.org/abs/2605.07568
  24. [24]Wenyi Hong et al. MotionBench: Benchmarking and improving fine-grained video motion understanding for vision language models. In CVPR, 2025.arxiv.org/abs/2501.02955
  25. [25]Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, and Jian Zhang. EgoDex: Learning dexterous manipulation from large-scale egocentric video. In ICLR, 2026.arxiv.org/abs/2505.11709
  26. [26]Yangliu Hu et al. SF2T: Self-supervised fragment finetuning of Video-LLMs for fine-grained understanding. In CVPR, 2025.arxiv.org/abs/2504.07745
  27. [27]InternVL Team. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025.
  28. [28]Songtao Jiang et al. Learning transferable temporal primitives for video reasoning via synthetic videos. In CVPR, 2026.arxiv.org/abs/2603.17693
  29. [29]Siddharth Karamcheti et al. Prismatic VLMs: Investigating the design space of visually-conditioned language models. In ICML, 2024.arxiv.org/abs/2402.07865
  30. [30]Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, 2017.arxiv.org/abs/1705.00754
  31. [31]Benno Krojer et al. A shortcut-aware video-QA benchmark for physical understanding via minimal video pairs. arXiv preprint arXiv:2506.09987, 2025.arxiv.org/abs/2506.09987
  32. [32]Jongseo Lee et al. Which way did it move? diagnosing and overcoming directional motion blindness in video-LLMs. arXiv preprint arXiv:2605.22823, 2026.arxiv.org/abs/2605.22823
  33. [33]Jie Lei, Tamara L. Berg, and Mohit Bansal. Revealing single flame bias for video-and-language learning. In ACL, 2023.arxiv.org/abs/2206.03428
  34. [34]Bo Li et al. LLaVA-OneVision: Easy visual task transfer. TMLR, 2025.arxiv.org/abs/2408.03326
  35. [35]Kunchang Li et al. MVBench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024.arxiv.org/abs/2311.17005
  36. [36]Rui Li, Xiaohan Wang, Yuhui Zhang, Orr Zohar, Zeyu Wang, and Serena Yeung-Levy. Temporal preference optimization for long-form video understanding. arXiv preprint arXiv:2501.13919, 2025.arxiv.org/abs/2501.13919
  37. [37]Shicheng Li et al. VITATECS: A diagnostic dataset for temporal concept understanding of video-language models. In ECCV, 2024.arxiv.org/abs/2311.17404
  38. [38]Shicheng Li et al. TEMPLE: Temporal preference learning of video VLLMs via difficulty scheduling and pre-SFT alignment. arXiv preprint arXiv:2503.16929, 2025.
  39. [39]Xinhao Li et al. VideoCraft-Flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00574, 2024.arxiv.org/abs/2501.00574
  40. [40]Bin Lin et al. Video-LLaVA: Learning united visual representation by alignment before projection. In EMNLP, 2024.arxiv.org/abs/2311.10122
  41. [41]Haotian Liu, Chunyuang Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
  42. [42]Yuanxin Liu et al. TempCompass: Do video LLMs really understand videos? In Findings of ACL, 2024.arxiv.org/abs/2403.00476
  43. [43]Corey Lynch et al. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023.arxiv.org/abs/2210.06407
  44. [44]Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In ACL, 2024.arxiv.org/abs/2306.05424
  45. [45]Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In ECCV, 2016.arxiv.org/abs/1603.08561
  46. [46]Lorenzo Mur-Labadia et al. V-JEPA 2.1: Unlocking dense features in video self-supervised learning. arXiv preprint arXiv:2603.14482, 2026.
  47. [47]Viorica Pătrăucean et al. Perception test: A diagnostic benchmark for multimodal video models. In NeurIPS Datasets and Benchmarks, 2023.arxiv.org/abs/2305.13786
  48. [48]Tobia Poppi et al. CounterVid: Counterfactual video generation for mitigating action and temporal hallucinations in video-language models. arXiv preprint arXiv:2601.04778, 2026.arxiv.org/abs/2601.04778
  49. [49]Qwen Team. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025.
  50. [50]Alec Radford et al. Learning transferable visual models from natural language supervision. In ICML, 2021.arxiv.org/abs/2103.00020
  51. [51]Nikhila Ravi et al. SAM 2: Segment anything in images and videos. In ICLR, 2025.arxiv.org/abs/2408.00714
  52. [52]Shuhuai Ren, Linli Yao, Sicheng Li, Xu Sun, and Lu Hou. TimeChat: A time-sensitive multimodal large language model for long video understanding. In CVPR, 2024.arxiv.org/abs/2312.02051
  53. [53]Geoffrey Roeder, Luke Metz, and Diederik P. Kingma. On linear identifiability of learned representations. In ICML, 2021.arxiv.org/abs/2007.00810
  54. [54]Pierre Sermanet et al. RoboVQA: Multimodal long-horizon reasoning for robotics. In ICRA, 2024.arxiv.org/abs/2311.00899
  55. [55]Ziyao Shangguan et al. TOMATO: Assessing visual temporal reasoning capabilities in multimodal foundation models. In ICLR, 2025.arxiv.org/abs/2410.23266
  56. [56]Gunnar A. Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-Ego: A large-scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626, 2018.arxiv.org/abs/1804.09626
  57. [57]Shengbang Tong et al. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In NeurIPS, 2024.arxiv.org/abs/2406.16860
  58. [58]Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS, 2022.arxiv.org/abs/2203.12602
  59. [59]Michael Tschannen et al. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025.arxiv.org/abs/2502.14786
  60. [60]Paul Voigtlaender, Soravit Changpinyo, Jordi Pont-Tuset, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with video localized narratives. In CVPR, 2023.arxiv.org/abs/2302.11217
  61. [61]Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu. Self-supervised video representation learning by pace prediction. In ECCV, 2020.arxiv.org/abs/2008.05861
  62. [63]Yi Wang et al. InternVid: A large-scale video-text dataset for multimodal understanding and generation. In ICLR, 2024.arxiv.org/abs/2307.06942
  63. [64]Yunxiao Wang et al. TIME: Temporal-sensitive multi-dimensional instruction tuning and robust benchmarking for Video-LLMs. In AAAI, 2026.
  64. [65]Donglai Wei, Joseph J. Lim, Andrew Zisserman, and William T. Freeman. Learning and using the arrow of time. In CVPR, 2018.DOI
  65. [66]Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B. Tenenbaum, and Chuang Gan. STAR: A benchmark for situated reasoning in real-world videos. In NeurIPS Datasets and Benchmarks, 2021.
  66. [67]Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. NExT-QA: Next phase of question-answering to explaining temporal actions. In CVPR, 2021. 6arxiv.org/abs/2105.08276
  67. [68]Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In CVPR, 2019. 3, 14DOI
  68. [69]Zihui Xue, Mi Luo, and Kristen Grauman. Seeing the arrow of time in large multimodal models. In NeurIPS, 2025. 3arxiv.org/abs/2506.03340
  69. [70]An Yang et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. 5, 18arxiv.org/abs/2505.09388
  70. [71]Yuan Yao, Chang Liu, Dezhao Luo, Yu Zhou, and Qixiang Ye. Video playback rate perception for self-supervised spatio-temporal representation learning. In CVPR, 2020. 3, 14DOI
  71. [72]Kexin Yi et al. CLEVRER: Collision events for video representation and reasoning. In ICLR, 2020. 18arxiv.org/abs/1910.01442
  72. [73]Zhou Yu et al. ActivityNet-QA: A dataset for understanding complex web videos via question answering. In AAAI, 2019. 18arxiv.org/abs/1906.02467
  73. [74]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 2, 18arxiv.org/abs/2303.15343
  74. [75]Boqiang Zhang et al. VideoLLaMA 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025. 2, 18arxiv.org/abs/2501.13106
  75. [76]Jianrui Zhang, Mu Cai, and Yong Jae Lee. Vinoground: Scrutinizing LMMs over dense temporal reasoning with short videos. arXiv preprint arXiv:2410.02763, 2024. 3, 6arxiv.org/abs/2410.02763
  76. [77]Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, et al. Direct preference optimization of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01258, 2024. 5, 18arxiv.org/abs/2404.01258
  77. [78]Yuanhan Zhang et al. LLaVA-Video: Video instruction tuning with synthetic data. TMLR, 2025. 2, 5, 18arxiv.org/abs/2410.02713
  78. [79]Orr Zohar et al. Apollo: An exploration of video understanding in large multimodal models. In CVPR, 2025. 2arxiv.org/abs/2412.10360

Paper details

vlnc:W_qy8e7kac
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents