Consistent Video-to-Video Translation via Explicit Correspondences

Introduction

Real-time video-to-video translation unlocks a large class of interactive applications, including streaming, gaming, and augmented and virtual reality. Recent advances in large scale text-to-video diffusion models [1, 6, 16, 31, 41, 57], and subsequently distilling them into few-step, autoregressive video models [12, 20, 22, 47, 51], have enabled high-quality generation at interactive speeds.

Autoregressive video generation relies on a history of past frames to ensure temporal continuity. However, determining an efficient mechanism for maintaining this history remains a fundamental challenge. Naïvely retaining all past outputs causes memory to scale linearly and runtime quadratically with sequence length, both of which are prohibitive for real-time applications. To maintain real-time latency, many methods resort to a fixed-length sliding window. While this ensures a fixed memory and runtime per frame, it fundamentally sacrifices long-range consistency: once an object leaves the window, often after only a few seconds, the model “forgets” it forever.

Several lines of work have attempted to bridge this gap. First, attention sinks [22, 38, 45, 47] heuristically retain the initial frames as a global anchor.

However, this approach inherently cannot address novel content that first appears in a later frame, even in cases like simple disocclusions. Another approach involves models with learned hidden states, such as Mamba [4, 8], which compress history into fixed-size bottlenecks. While efficient, these models often forgo the expressive power of transformers and remain largely limited to scenes with lower temporal complexity [32].

Finally, some methods use pose conditioning [52] or explicit world representation (e.g., 3D point clouds) to maintain a consistent world [42, 46]. These methods struggle to model dynamic scenes and require additional privileged inputs that are hard to obtain.

In this paper, we address the challenge of long-range consistency in video-to-video translation. Our insight is that long-range consistency cues are provided in the input video itself. That is, if two regions match in the input video, their generated outputs should match as well. The input space can therefore serve as an index for retrieving previously generated content, allowing consistency to be enforced, without requiring access to the full output history.

As such, we introduce vid2vid-long, a correspondence-based dynamic memory mechanism that enables long-term consistency while preserving real-time performance. Concretely, we compute patch-level correspondences in the input video and use them to index a compressed global representation of prior outputs. This representation grows only when novel content appears, rather than with video length, resulting in sub-linear memory growth in practice. The approach is complementary to existing autoregressive distillation backbones.

We evaluate our approach in combination with multiple autoregressive distillation frameworks, including Self-Forcing, LongLive, and Rolling Forcing. In all settings, incorporating our technique substantially improves long-range consistency, while maintaining real-time inference speed and high visual quality. Our main contributions are:

• We identify a structural property of video-to-video translation: correspondences in input space induce correspondences in output space, enabling input-driven retrieval for long-range consistency.

• We propose a correspondence-based mechanism that maintains a compressed global representation of prior outputs whose size scales with visual novelty rather than sequence length.

• We demonstrate consistent improvements in long-range temporal coherence across multiple real-time autoregressive backbones.

Our project website is available at https://www.cs.cmu.edu/~vid2vid-long/.

Related Works

Controllable Generation. The rapid advancements in text-to-image [18, 34, 44] and text-to-video [1, 10, 41] have been accompanied by controllable adapters that allow a more fine-grained control over the generation. In the image domain, works like ControlNet [54], Sketch-Guided-Diffusion [40], T2I adapter [25], img2img-turbo [28] show the ability to add structure control like edge maps, depth maps, and hand drawn sketches. Models such as IP-adapter [48], BLIP-diffusion [19], SynCD [17], VisualComposer [29] show ability to condition the generation of subject identity. These methods have also been extended to videos in recent methods like ControlVideo [56], Control-A-Video [3], VACE [14], and COSMOS-T [26]. However, existing video control methods are primarily designed for offline generation, often relying on non-causal processing that makes them unsuitable for interactive applications. We aim to address this gap in this paper.

Diffusion Distillation. Several prior works have explored speeding up diffusion models by reducing the number of denoising steps. Initial image domain approaches like consistency models [39], progressive distillation [35], *adversarial distillation [37], and distribution matching distillation [49, 50], moment matching distillation [36] enabled high-quality inference in 1–4 steps. Recent methods like CausVid [51] extend distillation to video generation, while Diffusion Forcing [2] explores autoregressive video generation. However, these approaches can still suffer from error accumulation during autoregressive inference. To address this, Self-Forcing [12] trains the student model on its own generated rollout, rather than ground truth video. Our work is complementary to these methods, focusing on maintaining long-range consistency during autoregressive inference.

Long Context Generation. Maintaining long-term consistency in autoregressive generation is computationally challenging due to the quadratic complexity of transformer attention. This is particularly severe for videos: five seconds of 480p video can exceed 32,000 tokens in modern architectures like WAN2.1 [41]. Existing methods such as Streaming-T2V [11] or LongLive [47] typically use sliding windows or hierarchical latent memory. However, these often suffer from recency bias, where the model forgets the appearance of objects or environments during loop closures or long-duration occlusions. While the language modeling literature has explored KV cache pruning and compression [7, 23], these methods often ignore the spatiotemporal continuity of video.

Rather than deciding which history to retain solely based on temporal position, learned compression, or token importance, we use input-space correspondences to retrieve the specific past outputs that are relevant to the current frame. This makes the memory mechanism task-aware and particularly well-suited to video-to-video translation, where repeated input regions should induce repeated output appearances.

Method

Our objective is to design an autoregressive video-to-video diffusion model equipped with a correspondence-based memory mechanism. Our architecture enables long-range temporal consistency while maintaining the low-latency performance required for real-time deployment. We begin by reviewing the autoregressive formulation in § 3.1. In § 3.2, we introduce our explicit correspondence-based memory component. We then detail our strategy for conditioning on this historical context in § 3.3. Finally, we describe the specific training objectives and real-time inference algorithm in § 3.4 and § 3.5, respectively.

Preliminaries

Bidirectional Video Generation. Our method builds upon pre-trained flow-matching models designed for video-to-video translation. Specifically, we adopt VACE [14], which is a latent flow-matching model, operating within the latent space of a pre-trained Variational Autoencoder (VAE) [34]. Given a block of input frames $I^{(b)} \in\mathbb{R}^{T \times H \times W \times3}$, the VAE encoder $\mathcal{E}$ maps the pixel data to a latent representation $z^{(b)} \in\mathbb{R}^{T \times h \times w \times c}$. The latent dimensions are determined by the spatial and temporal compression factors, which for VACE are given as follows: $h = H/8$, $w = W/8$, and $t = 1 + (T - 1)/4$. Following common practice in vision transformers [30], the latent block is further patchified into a sequence of tokens. Each latent frame is partitioned spatially into non-overlapping patches of size $1 \times p \times p$, where the temporal dimension of a patch remains singular. This process yields a total of $N = t \times(h/p) \times(w/p)$ tokens per block. Specifically for VACE, $p = 2$.

For the generative model itself, VACE utilizes a bidirectional spatiotemporal transformer to generate an entire video at once. This design implies that every generated token depends on the full context of the video, including both past and future tokens. This design is expensive in computation and memory, limiting their operation to short, fixed-length sequences, typically up to five seconds.

Autoregressive Video Generation. To enable long-form generation, recent methods [12, 22, 47, 51] replace bidirectional attention with causal attention. These models generate videos autoregressively in temporal blocks, where each block $z^{(b)}$ is generated by conditioning on the previously generated blocks $z^{(<b)}$. Access to previously generated blocks is typically implemented via a causal KV cache. While theoretically sound, maintaining the entire history $z^{(<b)}$ causes the runtime to grow quadratically with the video length, and the memory usage linearly. To ensure a constant per-frame runtime necessary for streaming applications, the causal attention is typically restricted to a smaller set of past frames, implemented with a sliding window of size $W$. Under this constraint, the generation of block $b$ conditions only on the most recent context $z^{(b-W:b-1)}$. Although this approach bounds the computational cost, it fundamentally limits temporal consistency to the window size. Any information that falls outside this window becomes inaccessible to the model, leading to identity drift and “forgetting” over long horizons. Our method addresses this limitation by providing an explicit correspondence-based memory that allows any autoregressive model to access the history arbitrarily far beyond its fixed window.

Compressed Patch History

In video-to-video translation, the availability of the input stream allows us to move beyond temporal proximity as the sole proxy for relevance. While a sliding window approach utilizes only recent frames for context, we can use explicit correspondences in the input space to identify relevant history from any point in the past. This is a reliable strategy because regions that match in the input video should match in the generated output (see Figure 2).

Vector (PDF)

By explicitly hard-coding these correspondences, we offload the burden of discovery from the transformer’s expensive attention layers. Rather than forcing the model to rediscover these relationships, we provide them directly via an external index. We represent this memory as an evolving set of input-output pairs. Each entry consists of a descriptor for an input patch, which acts as a retrieval key, and the corresponding generated output patch, which serves as the value for future conditioning. To ensure the system remains scalable for long sequences, we only update the memory when the input video presents novel information. If an incoming patch is already sufficiently represented in the history, we do not save it. This ensures that the memory footprint scales with the visual novelty of the scene, rather than the raw sequence length. (See history growth analysis in the appendix.)

To implement this mechanism, we define three concrete operations that govern the memory lifecycle. We first specify the patch-level representation for the keys and values, followed by the retrieval protocol for reading from history, and finally the novelty-based policy for writing to the global store.

Patch Descriptor. Our memory mechanism leverages the spatial discretization already present in the diffusion transformer (see § 3.1). In this framework, every input video spatiotemporal patch is already mapped to a small latent patch of shape $(1, 2, 2)$ that corresponds to a single token. From here, we use term “latent patch” for these token-corresponding patches. We assign each latent patch a global index $g$ that uniquely identifies its spatiotemporal coordinates. For every input latent patch at index $g$, denoted as $z_g$, we define a descriptor $k_g$ to serve as a retrieval key. We evaluated several feature extractors $\phi$ to produce these keys, including external vision backbones applied in pixel space. However, our experiments show that the most effective representation is the latent patch itself (further analysis is included in the appendix). Consequently, we define the feature extraction function as the identity mapping, where $k_g = z_g$. This approach also avoids the computational cost of decoding latents back to pixels and the inference run of additional feature extraction backbones. Under this formulation, the memory stores pairs $(k_g, v_g)$, where the key $k_g$ is the input latent patch and the value $v_g$ is the corresponding to the generated output latent patch.

Memory Management. To maintain temporal coherence throughout the generation process, we define the memory as a dynamic set of key-value pairs that evolves alongside the autoregressive blocks. Let $\mathcal{H}_b$ represent the memory state after the generation of block $b$, initialized as an empty set $\mathcal{H}_0 = \emptyset$. We similarly define $\mathcal{G}_b$ as the set of all global indices $g$ associated with the latent tokens generated within block $b$. Under this formulation, the generation of each subsequent block $b+1$ utilizes the existing history $\mathcal{H}_b$ for retrieval and concludes by updating the state to $\mathcal{H}_{b+1}$ using the newly generated tokens in $\mathcal{G}_{b+1}$.

Memory Write Policy. The memory state is updated after the generation of each block by evaluating the novelty of every new latent patch relative to the existing stored patches. For each generated latent patch at index $g \in\mathcal{G}_{b+1}$, we pair the input descriptor $k_g$ with the corresponding output latent patch $v_g$. The similarity between a query latent patch $k_g$ and the current memory $\mathcal{H}_b$ is defined as the maximum cosine similarity:

For an empty memory, we define $\operatorname{sim}(k_g,\mathcal{H}_0) = -\infty$, ensuring that all patches from the first generated block are added to memory. The latent patch and its corresponding generated output $\mathbf{v}_g$ are committed to the memory only if this similarity falls below a novelty threshold $\tau_h$. The updated memory $\mathcal{H}_{b+1}$ is thus defined as:

This policy ensures that memory footprint and inference cost scales with discovery of unique visual content rather than total sequence length.

Memory Read Mechanism. Before generating the output latents for block $b+1$, we query $\mathcal{H}_b$ using the descriptors of the current input latent. For each patch index $g$, we identify the set of $k$ nearest neighbors $\mathcal{N}_k(g)$ whose similarity exceeds a retrieval threshold $\tau_m$:

For every query latent patch that satisfies this condition, we retrieve the stored output values $\mathbf{v}_{g'}$. If no entries in $\mathcal{H}_b$ meet the threshold $\tau_m$, the neighborhood $\mathcal{N}_k(g)$ is empty. We track these occurrences with a binary validity mask $\mathbf{m}_g$, where $\mathbf{m}_g=1$ if $\mathcal{N}_k(g)\ne\varnothing$ and 0 otherwise. This mask and the retrieved values are then used to condition the diffusion transformer as described in § 3.3.

Conditioning via Correspondence Cross-Attention

We now describe how these retrieved values are used to condition the video translation model. The retrieved output patches $\{\mathbf{v}_{g'}\}$ are tokenized using the same patch embedding as the input latents, producing history tokens $\mathbf{h}_{g'}$ in a shared representation space.

To incorporate these history tokens into the video-model architecture, we add correspondence cross-attention layers to every transformer block between self-attention and text cross-attention layers. Each token attends only to its own $k$ retrieved neighbors. Concretely, for each token index $g$, the current hidden state $\mathbf{x}_g$ provides the query. Let $\mathcal{H}_g=\{\mathbf{h}_{g'}\}_{(\mathbf{k}_{g'},\mathbf{v}_{g'})\in\mathcal{N}_k(g)}$ denote the set of retrieved history tokens for query $g$. We define

where $\mathbf{m}_g$ is the validity mask that zeros out tokens without valid correspondences. The output is added as a residual: $\mathbf{x}_g \leftarrow\mathbf{x}_g + \operatorname{CorrXA}(g)$. The output projection $W_o$ is zero-initialized [54], so the layer acts as an identity at the start of training, preserving the behavior of the pretrained model. Since each token attends only to its $k$ neighbors, the computational overhead of these layers is minimal. We’ve described the cross-attention operation for a single token. To implement this efficiently, we reshape the query and memory tensors to collapse the sequence length into the batch dimension, effectively treating each token’s cross-attention as an independent $1 \times k$ operation.

Training

We incorporate our correspondence-based memory into several different autoregressive video model. Unless stated otherwise, we use Rolling Forcing [22] distillation with 5 steps as the default configuration. The student model is initialized from pretrained VACE 1.3B and all model parameters are finetuned jointly, including the newly introduced correspondence cross-attention layers. We adopt the standard Rolling Forcing objective without modification or extra loss terms. The correspondence memory is active during training, and memory read and write operations are performed as described in § 3.2. The student conditions on retrieved correspondences from its own previously generated blocks, and newly generated patches are written to memory according to the novelty threshold. The teacher model is a bidirectional VACE 14B model and it does not use correspondence memory.

Inference

At inference time, the model generates the video autoregressively, processing fixed-size blocks sequentially. At the start of each video, we set $H_0 = \emptyset$. For each block $b$, we compute the input latent tokens $z_g$, $g \in G_b$. Using $z_g$, we query $H_{b-1}$ to compute neighbors $N_k(g)$ for each $g \in G_b$. The retrieved neighbors are used to condition the translation network through correspondence cross-attention layers during each denoising step of block $b$. Retrieval is performed once per block and the resulting neighbors are reused across all denoising steps. After block $b$ is fully denoised, we obtain the output latents $v_g$, $g \in G_b$, and update the memory according to Eq. (2). The updated memory $H_b$ is used to generate the next block $b + 1$. A complete inference pseudocode is provided in the appendix.

Experiments

Here, we first evaluate our method on depth-to-video translation, focusing on long-range temporal consistency on long videos. Our mechanism is agnostic to the conditioning modality. Experiments are conducted on two benchmarks: VACE-Benchmark [14], covering diverse real-world scenes, and DL3DV [21], a geometry-grounded 3D dataset enabling controlled pose revisits. We compare against bidirectional diffusion models, prior video translation methods, and recent autoregressive distillation approaches, and ablate the key components of our correspondence mechanism. Unless stated otherwise, all experiments use VACE 1.3B at 480p resolution.

Datasets. We evaluate on two benchmarks. VACE-Benchmark [14] contains diverse real-world video sequences with varying motion and scene complexity. DL3DV [21] is a geometry-grounded 3D dataset with calibrated camera poses, enabling controlled evaluation of viewpoint revisits and long-range consistency. Additional dataset details and preprocessing are provided in appendix.

Evaluation Protocol. We evaluate our method in terms of video quality, long-range consistency under revisits, and inference efficiency. Video quality is measured on full generated videos using V-Bench [13] metrics (subject consistency, background consistency, temporal flickering, motion smoothness, aesthetic quality, and imaging quality). Experiments report average of scores, individual metrics values are provided in the appendix. Long-range consistency is evaluated using controlled revisit protocols that require model to reproduce previously observed content. On VACE-Benchmark, following [9], we construct synthetic long sequences by concatenating two distinct 5-second clips in an interleaved order (A→B→A→B), and measure reconstruction consistency between repeated occurrences of the same clip. Consistency is quantified using LPIPS [55], DreamSim [5], and CLIP [33] image similarity. On DL3DV, we use pretrained 3D Gaussian Splatting models [15] using calibrated camera poses to render exact pose revisits and measure consistency between first-visit and revisit frames. We measure inference latency and throughput. Inference latency is measured as time to produce first frame and throughput is reported in frames per second (FPS). All measurements are performed on a single NVIDIA H100 GPU.

Implementation Details. We use VACE 1.3B [14] as the student model and VACE 14B as the teacher. The student is initialized from the pretrained VACE 1.3B checkpoint and trained using Rolling Forcing [22]. For correspondence retrieval, we use VAE encoder’s latent features as keys, with $k = 3$ retrieved neighbors, history threshold $\tau_h = 0.75$, and memory threshold $\tau_m = 0.75$. Training is performed using AdamW [24] with batch size 64, a learning rate of $2 \times10^{-6}$ for the student network and $4 \times10^{-7}$ for the critic.

Baseline Comparisons

We evaluate our method against several video-to-video translation frameworks. Quantitative and qualitative results are in Table 1 and Figure 3. First, we compare to Control-A-Video [3], VideoComposer [43], and ControlVideo [56].

Family	Method	Base Model	LPIPS↓	DreamSim↓	CLIP-Sim↑	Quality Score↑	Latency(s)↓	FPS↑
	VACE 1.3B	N/A	0.463	0.195	0.901	0.835	79	1.03
Offline	Control-A-Video	SD 1.5	0.288	0.169	0.907	0.722	11	0.89
	VideoComposer	N/A	0.664	0.333	0.847	0.724	11	1.36
	ControlVideo	SD 1.5	0.174	0.065	0.954	0.778	550	0.59
Autoregressive	Self-Forcing (4 steps)	VACE 1.3B	0.548	0.203	0.881	0.789	0.67	13.74
	LongLive (4 steps)	VACE 1.3B	0.511	0.218	0.874	0.817	0.68	13.83
	Rolling Forcing (5 steps)	VACE 1.3B	0.427	0.172	0.905	0.799	2.58	10.99
	Ours (2 step)	VACE 1.3B	0.302	0.079	0.941	0.814	1.09	13.88
	Ours (5 step)	VACE 1.3B	0.240	0.066	0.955	0.828	3.72	8.10

Vector (PDF)

They operate in an offline bidirectional setting with multi-step diffusion and global attention over fixed-length sequences. In our experiments, they produce lower visual quality and struggle to maintain long-range consistency (Figure 3). ControlVideo achieves relatively strong reconstruction scores due to its hierarchical sampling strategy over long sequences. However, this comes at the cost of high latency ($\approx42$ minutes in our setting) and reduced perceptual quality, making it impractical for real-time video-to-video translation.

We next compare to newer, state-of-the-art video translation models, such as VACE [14]. As discussed before, VACE is a bidirectional model which is limited to a fixed-length, short video sequences. Unfortunately, there is no publicly available adaption of VACE to an unbounded-length, streaming use case. In order to still apply VACE on our setting, we adapt it in two ways.

First, we directly evaluate VACE by applying it on non-overlapping video segments. We find that this naive adaptation of VACE produces high-quality results, but unsurprisingly has poor long-range consistency. For example, note row 2 in Figure 3 where each individual frame is of high-quality, but objects change their colors drastically over the length of the video.

Second, we adapt VACE to a streaming use-case with three recent causal, autoregressive distillation methods: Self-Forcing [12], LongLive [47], and Rolling Forcing [22]. These methods were originally designed for text-to-video generation and aren’t applied to video-to-video yet. Here, we implement and adapt the three methods to video-to-video with VACE. We find that all methods achieve competitive quality scores and substantially lower latency than the bidirectional VACE baseline (Table 1, Table 2). Nevertheless, these methods still perform poorly on long-range consistency. On VACE-Bench, Self-Forcing and LongLive under-perform bidirectional VACE, while Rolling Forcing only slightly outperforms it in long-range reconstruction.

Family	Method	Base Model	LPIPS↓	DreamSim↓	CLIP-Sim↑	Quality Score↑	Latency(s)↓	FPS↑
Offline	VACE	1.3B	N/A	0.592	0.240	0.881	0.819	79	1.01
	Control-A-Video	SD 1.5	0.437	0.188	0.895	0.727	14	0.89
	VideoComposer	N/A	0.672	0.303	0.854	0.718	12	1.39
	ControlVideo	SD 1.5	0.205	0.057	0.957	0.794	2518	0.38
Autoregressive	Self-Forcing	VACE 1.3B	0.616	0.238	0.870	0.790	0.67	13.74
	LongLive	VACE 1.3B	0.534	0.170	0.896	0.806	0.68	13.83
	Rolling Forcing	VACE 1.3B	0.445	0.163	0.907	0.806	2.58	10.99
	Ours (2 step)	VACE 1.3B	0.379	0.083	0.932	0.799	1.09	13.88
	Ours (5 step)	VACE 1.3B	0.305	0.058	0.952	0.821	3.72	8.10

Our main contribution, correspondence mechanism, can be readily applied with any autoregressive method. In Table 1, Table 2 and Figure 3, we report results of our best configuration, where our method is applied with Rolling Forcing. Among autoregressive methods, our approach achieves the strongest long-range consistency and visual quality while maintaining real-time performance. In Table 3, we report the results of our method when applied on top of all autoregressive methods. Adding our correspondence mechanism consistently improves long-term consistency across all autoregressive backbones while preserving visual quality. Table 3 results demonstrate that our method is orthogonal to choice of autoregressive backbone. Please refer to the appendix for a comparison of different conditioning mechanisms.

	VACE-Bench LPIPS↓	VACE-Bench DreamSim↓	VACE-Bench CLIP-Sim↑	VACE-Bench Quality Score↑	DL3DV LPIPS↓	DL3DV DreamSim↓	DL3DV CLIP-Sim↑	DL3DV Quality Score↑
Self-Forcing	0.548	0.203	0.881	0.789	0.616	0.238	0.870	0.790
+ Correspondence	0.343↓37%	0.097↓52%	0.930↑6%	0.794	0.451↓27%	0.151↓37%	0.897↑3%	0.775
LongLive	0.511	0.218	0.874	0.817	0.534	0.170	0.896	0.806
+ Correspondence	0.272↓47%	0.057↓74%	0.953↑9%	0.807	0.345↓35%	0.064↓63%	0.945↑5%	0.786
Rolling Forcing	0.427	0.172	0.905	0.799	0.445	0.163	0.907	0.806
+ Correspondence	0.240↓44%	0.066↓62%	0.955↑6%	0.828	0.305↓32%	0.058↓65%	0.952↑5%	0.821

Alternate Consistency Mechanisms

We compare our correspondence mechanism against alternative strategies for extending temporal context beyond the default sliding window. Qualitative and quantitative results are shown in Figure 4 and Table 4, respectively.

Method	VACE-Bench				DL3DV
	LPIPS↓	DreamSim↓	CLIP-Sim↑	Quality↑	LPIPS↓	DreamSim↓	CLIP-Sim↑	Quality↑
KV cache (2×)	0.350	0.139	0.910	0.795	0.710	0.364	0.814	0.759
KV cache (3×)	0.308	0.120	0.920	0.793	0.770	0.496	0.744	0.752
KV cache (4×)	0.330	0.134	0.913	0.795	0.759	0.543	0.714	0.753
Keyframe (every 3rd)	0.329	0.115	0.923	0.790	0.607	0.257	0.857	0.776
Keyframe (every 5th)	0.350	0.128	0.914	0.789	0.581	0.240	0.857	0.776
Keyframe (every 10th)	0.373	0.134	0.912	0.789	0.523	0.190	0.880	0.777
Global avg. pool	0.450	0.176	0.891	0.753	0.457	0.145	0.909	0.772
Attention Sink	0.436	0.159	0.899	0.793	0.502	0.156	0.901	0.788
FramePack	0.304	0.095	0.938	0.775	0.344	0.105	0.932	0.802
Random retrieval	0.396	0.144	0.916	0.807	0.451	0.156	0.911	0.801
Correspondence (Ours)	0.240	0.066	0.955	0.828	0.305	0.058	0.952	0.821

Larger KV Cache window. We increase the sliding window size from 21 latent frames to 42 $(\times2)$, 63 $(\times3)$, and 84 $(\times4)$. On VACE-Bench, larger windows improve reconstruction consistency slightly, although the gains are non-monotonic. However, on DL3DV, where sequences are substantially longer, performance degrades sharply in both quality and long-range consistency.

Keyframe retention. Next, instead of sliding window, we retain KV cache of every $N$-th block ($N=3, 5, 10$), providing a sparse but more complete coverage of the history. While this improves over simply enlarging the window, it remains worse than correspondence-based retrieval in long-range reconstruction (Table 4).

Global Pooling. We maintain a single block of KV tokens as a running cumulative average of all previous blocks, providing constant memory access to the entire history. Although this improves long-range consistency on DL3DV degrades visual quality on both datasets (quality scores drop to 0.753 and 0.772).

Random Retrieval. To isolate the role of correspondence matching from the cross-attention layers, we replace the similarity-based retrieval with uniform random sampling from the memory bank. Each query patch receives $k$ randomly selected history patches instead of the top-$k$ most similar, the history creation process remains the same. This significantly degrades long-range reconstruction performance, demonstrating that accurate correspondence matching is essential.

Attention Sink. Recent methods such as LongLive, MotionStream, and Rolling Forcing retain initial frames as an attention sink to provide global context. However, this mechanism cannot capture novel content that appears later in the sequence and is insufficient to prevent identity drift in long videos.

FramePack. FramePack [53] is applied to the task of video to video translation, storing past blocks at geometrically decreasing spatial resolution. Recent blocks maintain full resolution while older blocks are progressively downsampled. While this retains the entire history, aggressive spatial compression reduces detail, leading to weaker long-range reconstruction.

Overall, correspondence-based retrieval outperforms all alternative mechanisms for extending temporal context, achieving stronger long-range consistency without sacrificing visual quality.

Analysis and Ablations

Importance of Correspondence. First, we show the importance of using correspondence patches through an ablation where we remove the correspondences and disable the correspondence cross-attention layer (Equation (4)). As in Table 5, without using correspondence the long context consistency gets significantly worse. The LPIPS reconstruction improves by 37% and 30% on the two datasets. DreamSim and CLIP similarity scores are also similarly effected. Figure 5 shows this qualitatively. The top row shows the generated video when the correspondence is disabled. In this video, the dog starts with a yellow snout, but as the video progresses and the dog re-enters the scene, the snout turns blue. In contrast, our full method with correspondence is shown in the bottom row. Here the dog retains the identity and re-enters the scene with the same yellow snout.

Method	VACE-Bench				DL3DV
	LPIPS↓	DreamSim↓	CLIP-Sim↑	Quality↑	LPIPS↓	DreamSim↓	CLIP-Sim↑	Quality↑
Ours w/o Corr.	0.379	0.160	0.910	0.829	0.437	0.124	0.921	0.822
Ours	0.240↓37%	0.066↓59%	0.955↑5%	0.828	0.305↓30%	0.058↓53%	0.952↑3%	0.821

Vector (PDF)

Sensitivity to different distillation methods. Next, we show that our method is not specific to any single distillation method. We apply our method to three different distillation methods: SelfForcing [12], LongLive [47], and RollingForcing [22]. Add our correspondence method improves the consistency across each of the base distillation methods as seen in Table 3.

Conclusion and Limitations

We introduce vid2vid-long, a correspondence-based memory mechanism for long-range consistent video-to-video translation. By exploiting a structural property of the task, matching regions in the input should match in the output, we replace implicit long-range attention with explicit input-driven retrieval. Our dynamic memory grows with visual novelty rather than length of the video, enabling sub-linear scaling while preserving real-time performance. Across multiple autoregressive distillation backbones, our method consistently improves long-range reconstruction and reduces identity drift while maintaining high visual quality and real-time inference performance. Compared to alternative context extension strategies, explicit correspondence retrieval proves both more effective and more efficient.

Limitations. Our model struggles on fast videos with large motion, where reliable cross-frame correspondences become harder to establish. It can also fail when the text prompt is inconsistent with the input depth video, since the model must reconcile conflicting semantic and structural conditioning signals.

Acknowledgment. We thank Sheng-Yu Wang, Nupur Kumari, Maxwell Jones, and Xun Huang for their helpful comments and discussion. The project was partially done during an internship with Adobe Research. The project is partially funded by NSF IIS-2239076, NSF ISS-2403303, the IITP grant funded by the Korean Government (MSIT) (No. RS-2024-00457882, National AI Research Lab Project), and the Packard Fellowship.

Appendix

We provide additional details and results that complement the main paper. Section A presents the architecture diagram and runtime analysis. Section B analyzes the history growth and memory scaling behavior. Section C shows results on other conditioning inputs. Sections D and E provide training and inference details. Section F compares different feature extractors for correspondence matching. Section G details the dataset preprocessing steps. Section H reports per-metric V-Bench scores. The video results are primarily shown in the accompanying website.

Model Architecture and Runtime Analysis

§ 3.3 of the main paper describes conditioning with correspondence-cross attention layers. In Figure 6, we show the corresponding architecture diagram. The only architectural modification to the pretrained VACE transformer is the insertion of a correspondence cross-attention layer in each block. Note that each token attends only to its $k$ retrieved neighbors.

Vector (PDF)

Figure 7 reports the per-component inference time for generating a 20-second video on a single H100 GPU at 480p resolution, for all three autoregressive backbones with our correspondence mechanism. The denoising forward passes dominate the total runtime across all methods, accounting for over 77% of the total. The correspondence-specific components (memory read, correspondence cross-attention, and memory write) add a small overhead. Memory read and write operations have a negligible inference cost ($\sim50\text{ms}$ total).

Vector (PDF)

History Growth Analysis

As discussed in § 3 of the main paper, the memory footprint scales with the visual novelty of the scene rather than the raw sequence length. Here, we empirically validate this claim.

VRAM Scaling. Table 6 compares the total GPU memory footprint of a naive full KV-cache approach against our dynamic memory mechanism as video length increases. The naive approach retains KV entries for every past token across all 45 transformer layers, causing memory to grow linearly with sequence length. At 480p resolution, it exceeds the 80 GB capacity of an H100 GPU after roughly 30 seconds; at 720p the limit is reached even sooner. Our method replaces the large KV cache with an attention sink and a fixed-size sliding window ($W=21$ latent frames), plus a correspondence memory that stores only input–output latent patch pairs. Because the correspondence memory bypasses the per-layer KV overhead, its footprint is negligible: even after 10 minutes of video (9,600 frames), total VRAM increases by only 0.1 GB at 480p and 0.3 GB at 720p compared to the initial cost. As a result, total VRAM remains nearly constant regardless of video duration at both resolutions.

		480p ($480 \times832$)		720p ($720 \times1280$)		$\lvert\mathcal{H}\rvert$
Duration	Frames	Full KV	Ours	Full KV	Ours
10 s	160	42.2	34.1	63.7	45.2	2 K
30 s	480	74.3	34.1	OOM	45.2	4 K
1 min	960	OOM	34.1	OOM	45.2	6 K
2 min	1920	OOM	34.1	OOM	45.3	9 K
5 min	4800	OOM	34.2	OOM	45.4	16 K
10 min	9600	OOM	34.2	OOM	45.5	26 K

History Growth and Read/Write Cost. Figure 8 (left) shows the history size $\lvert\mathcal{H}\rvert$ over time with and without our novelty-based compression, evaluated on 30 videos spanning three categories (webcam, surveillance, and action camera). Without compression, every generated patch is stored, causing $\lvert\mathcal{H}\rvert$ to grow linearly to over 936 K entries at 9 600 frames (10 minutes). With our write policy ($\tau_h=0.75$), the history grows sub-linearly, reaching roughly 26 K entries at the same point ($\sim36\times$ reduction). This directly translates to runtime savings: Figure 8 (right) shows that the per-block memory read and write time grows linearly without compression, exceeding 30 ms at 10 minutes, while with compression it remains below 2 ms throughout.

Vector (PDF)

Table 7: Hyperparameter analysis. We show the effects of varying the thresholds $\tau_h$ and $\tau_m$ on our method. As the threshold $\tau_h$ is increased, our method stores more patches in the history bank $\mathcal{H}$ and improves the reconstruction performance.

$\tau_h$	$|\mathcal{H}|$	LPIPS$\downarrow$	CLIP$\uparrow$	DreamSim$\downarrow$	$\tau_m$	LPIPS$\downarrow$	CLIP$\uparrow$	DreamSim$\downarrow$
0.25	413	0.301	0.940	0.092	0.25	0.248	0.956	0.059
0.50	682	0.284	0.946	0.081	0.50	0.249	0.957	0.058
0.75	3,897	0.256	0.957	0.060	0.75	0.256	0.957	0.060
0.90	15,822	0.204	0.963	0.049	0.90	0.286	0.946	0.084

History Growth and thresholds $\tau_h$ and $\tau_m$ Increasing $\tau_h$ admits more patches into the memory, trading memory size for better reconstruction. LPIPS improves from 0.301 to 0.204 as $|\mathcal{H}|$ grows from 413 to 15.8K patches. For $\tau_m$, the reconstruction quality is stable over a wide range of values. But, a strict threshold ($\tau_m = 0.90$) rejects many valid matches, degrading reconstruction (LPIPS 0.286). The same defaults perform consistently for both datasets with no scene-specific tuning needed.

Other Conditioning Inputs

We show the evaluation of different methods when other conditioning inputs such as sketch and optical flow are used in Table 8. Our method is robust and shows an improvement in the long range consistency across other conditioning inputs (sketch and optical flow) too.

Method	LPIPS$\downarrow$	CLIP$\uparrow$	DreamSim$\downarrow$
Self-Forcing	0.474	0.906	0.167
LongLive	0.384	0.911	0.156
Rolling Forcing	0.284	0.940	0.100
Ours	0.191	0.960	0.050
Method	LPIPS$\downarrow$	CLIP$\uparrow$	DreamSim$\downarrow$
Self-Forcing	0.535	0.852	0.253
LongLive	0.516	0.858	0.235
Rolling Forcing	0.494	0.868	0.226
Ours	0.263	0.934	0.092

Training Details

Table 9 lists all training and model hyperparameters. We train the student model using Rolling Forcing [22] distillation with 5 uniformly spaced denoising steps ($\{1000, 800, 600, 400, 200\}$) that are shifted following standard practice. The student is initialized from the pretrained VACE 1.3B checkpoint and all parameters are finetuned jointly, including the newly introduced correspondence cross-attention layers. The student is initialized with Diffusion Forcing training before timestep distillation. The teacher is the bidirectional VACE 14B model and does not use correspondence memory. During training, the correspondence memory is active: for each rollout, memory read and write operations are performed as described in § 3.2 of the main paper. We use the standard Rolling Forcing objective without modification or additional loss terms. We apply EMA with weight 0.99 starting at iteration 200. Training uses mixed precision and gradient checkpointing on 32 A100 80GB GPUs across 4 nodes. All inference benchmarks are measured on a single H100 80GB GPU.

Hyperparameter	Value
Student model	VACE 1.3B
Teacher model	VACE 14B
Distillation method	Rolling Forcing
Distillation loss	DMD
Denoising steps	5
Guidance scale	5.0
Patch descriptor $\phi$	VAE latent
Retrieved neighbors $k$	3
History threshold $\tau_h$	0.75
Matching threshold $\tau_m$	0.75
Per-patch correspondence dropout	0.1
Full-frame correspondence dropout	0.1
Optimizer	AdamW
Student learning rate	$2 \times10^{-6}$
Critic learning rate	$4 \times10^{-7}$
$(\beta_1, \beta_2)$	(0.0, 0.999)
Weight decay	0.01
Batch size	64
Training iterations	2000
Training resolution	480p
Training latent frames	21
Latent frames per block	3
Sliding window $W$ (latent frames)	21

Negative Prompt. Following standard practice, we use a fixed negative prompt: “Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards.”

Inference Details

Algorithm 1 provides the complete inference procedure. The model processes the input conditioning video block-by-block. For each block, we retrieve matching past outputs from the correspondence memory (memory read), denoise with correspondence cross-attention conditioning, and update the memory with novel input-output pairs (memory write). Retrieval is performed once per block and the retrieved tokens are reused across all denoising steps.

Feature Extractor Comparison

In § 3.2 of the main paper, we noted that VAE latent features outperform alternative feature extractors for correspondence matching. Here we provide the full comparison.

Table 10 compares three choices for the feature extractor $\phi$ used for computing the patch descriptor: VAE encoder’s latent representation, pretrained DINOv2 [27], and CLIP [33]. VAE latent features achieve the best reconstruction on VACE-Bench and competitive performance on DL3DV, without requiring an external feature model.

Feature Extractor $\phi$		VACE-Bench				DL3DV
	LPIPS ↓	DreamSim ↓	CLIP-Sim ↑	Quality ↑	LPIPS ↓	DreamSim ↓	CLIP-Sim ↑	Quality ↑
VAE latent	0.240	0.066	0.955	0.828	0.305	0.058	0.952	0.821
DINOv2 [27]	0.336	0.122	0.926	0.812	0.296	0.068	0.943	0.798
CLIP [33]	0.297	0.107	0.931	0.821	0.288	0.057	0.952	0.810

Table 11 reports the corresponding runtime overhead. Using DINOv2 or CLIP requires decoding each block’s latents back to RGB via the VAE decoder and then running the vision backbone, adding 79.7 ms and 32.2 ms per block respectively. Over a 20-second video (29 blocks), this amounts to 2.3 seconds for DINOv2 and 0.9 seconds for CLIP. VAE latent features incur no additional cost since they are already available from the encoding step.

Feature Extractor $\phi$	VAE Decode	Backbone	Total (20s video)
VAE latent	—	—	0.3 ms
DINOv2 [27]	17.4 ms	62.2 ms	2.3 s
CLIP [33]	17.4 ms	14.7 ms	0.9 s

# model: Diffusion transformer w/ corr. cross-attn
# vae: Pretrained encoder E / decoder D
# video: Input conditioning (e.g., depth maps)
# text: Text prompt for stylization
# tau_h, tau_m: History / matching thresholds
# k: Number of retrieved neighbors per patch
# W: Sliding window size (latent frames)
# ts: Denoising schedule [1000, 800, 600, 400, 200]

def vid2vid_long(model, vae, video, text,
                 tau_h, tau_m, k, W, ts):
    H_k, H_v = [], [] # memory H: keys + values
    kv_cache = None

    for block in chunk(video, block_size):
        # Encode input to latent patches
        z_in = vae.encode(block)
        q = patchify(z_in) # [N, c]

        # --- Memory Read (Eq. 3) ---
        corr, mask = zeros(N, k, c), zeros(N)
        if len(H_k) > 0:
            S = cosine_sim(q, stack(H_k)) # [N, |H|]
            vals, idx = topk(S, k, dim=1)
            hit = vals >= tau_m
            corr[hit] = stack(H_v)[idx[hit]]
            mask[hit] = 1.0

        # --- Denoise (corr reused across all steps) ---
        z = randn_like(z_in)
        for t in ts:
            z = denoise(z, t, z_in, text, corr, mask, kv_cache)

        # --- Memory Write (Eq. 2) ---
        v = patchify(z)
        s = max_cosine_sim(q, H_k)
        novel = s < tau_h
        H_k.extend(q[novel])
        H_v.extend(v[novel])

        # Update sliding window KV cache
        kv_cache = slide_kv(kv_cache, W)
        yield vae.decode(z)

Algorithm 1: Inference of vid2vid-long

Dataset Details and Preprocessing

VACE-Benchmark (ABAB). We follow the standard VACE-Benchmark [14] evaluation protocol for video preprocessing: each source video is uniformly sampled to 81 frames and resized to $480 \times832$ with center cropping. We retain all source videos with at least 81 frames. To evaluate long-range consistency, we follow the concatenation protocol of [9] and construct 100 evaluation sequences (with a fixed random seed) by randomly pairing two source clips and concatenating them in an A→B→A→B order, yielding $4 \times81 = 324$ frames per sequence at 16 fps ($\sim20$ seconds). The repeated structure produces ground-truth correspondence pairs, where each pair maps a frame in the first occurrence of a clip to the same frame in its second occurrence. Conditioning signals (depth maps, optical flow, and scribble maps) are extracted using the official VACE annotators applied to the concatenated sequence. Text captions are obtained from the original VACE-Benchmark English prompts and stylized into five artistic variants per clip using Qwen2.5-14B-Instruct.

DL3DV. We use the DL3DV-10K dataset [21] to evaluate long-range consistency under controlled, geometry-grounded viewpoint revisits. We select scenes spanning diverse indoor and outdoor environments from the DL3DV-GS-960P split, which provides calibrated COLMAP camera poses and pre-trained 3D Gaussian Splatting (3DGS) models [15]. For each scene, we construct a 60-second looping video (960 frames at 16 fps) as follows. We first extract the first loop from the camera trajectory by finding the point where the camera first returns closest to its starting position after traveling at least 25% of the total path length. The last 20 cameras are smoothly blended toward the initial camera (linear interpolation for translation, SLERP for rotation) to close the loop. The overlap segment yields ground-truth correspondence pairs per scene, with each pair separated by approximately 48 seconds. Conditioning signals are extracted using the same VACE annotator pipeline as VACE-Benchmark. Text captions are generated by captioning the first rendered frame with Qwen2.5-VL-7B-Instruct and stylizing into five variants with Qwen2.5-14B-Instruct. This dataset contains 180 samples in total.

Per-Metric V-Bench Scores

Tables in the main paper report the average V-Bench quality score. Table 12 provides the individual metric breakdown for all methods on both benchmarks. Table 13 provides the corresponding per-metric breakdown for the distillation method sensitivity experiment.

Table 12: Per-metric V-Bench quality scores. We report individual normalized V-Bench [13] metrics for all methods on both benchmarks. The rightmost column shows the average across all six metrics (reported as the aggregate quality score in the main paper). Best among autoregressive methods is bold, second best is underlined.

Benchmark	Category	Method	Subj.↑	BG↑	Flicker↑	Motion↑	Aesth.↑	Imaging↑	Avg.↑
VACE-Bench
	Offline	VACE 1.3B	0.929	0.935	0.934	0.962	0.606	0.643	0.835
	Offline	Control-A-Video	0.668	0.799	0.875	0.878	0.458	0.657	0.722
	Offline	VideoComposer	0.761	0.864	0.853	0.852	0.475	0.537	0.724
	Offline	ControlVideo	0.799	0.867	0.911	0.927	0.573	0.591	0.778
	Autoregressive	Self-Forcing (4 steps)	0.779	0.835	0.914	0.944	0.623	0.640	0.789
	Autoregressive	LongLive (4 steps)	0.873	0.891	0.916	0.953	0.619	0.650	0.817
	Autoregressive	Rolling Forcing (5 steps)	0.819	0.863	0.936	0.962	0.597	0.616	0.799
	Autoregressive	Ours (2 step)	0.883	0.900	0.918	0.947	0.594	0.642	0.814
	Autoregressive	Ours (5 step)	0.888	0.903	0.931	0.958	0.601	0.687	0.828
DL3DV
	Offline	VACE 1.3B	0.924	0.928	0.886	0.948	0.587	0.644	0.819
	Offline	Control-A-Video	0.771	0.857	0.844	0.858	0.413	0.619	0.727
	Offline	VideoComposer	0.771	0.864	0.836	0.850	0.475	0.514	0.718
	Offline	ControlVideo	0.876	0.909	0.899	0.919	0.583	0.580	0.794
	Autoregressive	Self-Forcing (4 steps)	0.899	0.912	0.885	0.933	0.559	0.550	0.790
	Autoregressive	LongLive (4 steps)	0.914	0.914	0.878	0.945	0.567	0.618	0.806
	Autoregressive	Rolling Forcing (5 steps)	0.904	0.908	0.919	0.956	0.580	0.568	0.806
	Autoregressive	Ours (2 step)	0.917	0.915	0.875	0.936	0.565	0.584	0.799
	Autoregressive	Ours (5 step)	0.915	0.915	0.899	0.953	0.590	0.653	0.821

VACE-Bench
	Subj.$\uparrow$	BG$\uparrow$	Flicker$\uparrow$	Motion$\uparrow$	Aesth.$\uparrow$	Imaging$\uparrow$	Avg.$\uparrow$
Self-Forcing	0.779	0.835	0.914	0.944	0.623	0.640	0.789
+ Correspondence	0.849	0.877	0.912	0.941	0.601	0.581	0.794
LongLive	0.873	0.891	0.916	0.953	0.619	0.650	0.817
+ Correspondence	0.875	0.894	0.906	0.935	0.603	0.628	0.807
Rolling Forcing	0.819	0.863	0.936	0.962	0.597	0.616	0.799
+ Correspondence	0.888	0.903	0.931	0.958	0.601	0.687	0.828
DL3DV
	Subj.$\uparrow$	BG$\uparrow$	Flicker$\uparrow$	Motion$\uparrow$	Aesth.$\uparrow$	Imaging$\uparrow$	Avg.$\uparrow$
Self-Forcing	0.899	0.912	0.885	0.933	0.559	0.550	0.790
+ Correspondence	0.914	0.910	0.873	0.938	0.533	0.483	0.775
LongLive	0.914	0.914	0.878	0.945	0.567	0.618	0.806
+ Correspondence	0.912	0.915	0.872	0.928	0.548	0.539	0.786
Rolling Forcing	0.904	0.908	0.919	0.956	0.580	0.568	0.806
+ Correspondence	0.915	0.915	0.899	0.953	0.590	0.653	0.821