betaFeedback

WorksComputer Science

Fast Training-Free Low-Light Video Enhancement with a Frozen Image Diffusion Prior

Abstract

Pretrained image diffusion models can enhance low-light video without low-light training data and without retraining for a new camera. ZVRD, the only published training-free method for low-light video, is too slow to use: our implementation needs over a minute per 256×256256 \times 256 frame. We present a causal, training-free pipeline that adds two temporal components to a frozen MR. Illuminate image prior. Temporal input fusion recursively averages motion-compensated low-light frames before diffusion inversion. Late-start offset propagation runs the full sampling trajectory only on key frames. Each non-key frame inverts the low-noise part of its own trajectory and adds an enhancement offset warped from the latest key frame. At ZVRD's own resolution the pipeline is 261×261\times faster than ZVRD and scores better on all five metrics we report. On 13 held-out SDSD clips at 1280×7201280 \times 720, it improves on framewise eight-step processing by 0.55 dB in PSNR and 0.034 in LPIPS and halves the warping error, in 10% less time. Measurements inside the model show why, and a linear-Gaussian model of inversion explains them. Fusion removes frame-to-frame noise before inversion amplifies it. A flow error perturbs the transported offset ten times less than it perturbs a transported state. The perceptual gain does not transfer to low resolution: at 346×260346 \times 260, LPIPS is worse by 0.028. At 0.65 frames per second, the pipeline is an offline method.

Introduction

Low-light video enhancement must improve visibility, keep detail, and avoid distracting changes between frames. Learned video models do this after training on low-light video [9, 26, 40]. Such data are expensive to capture, and supporting a new camera or condition can call for new data and retraining. Pretrained image diffusion models offer another route. MR. Illuminate [5], hereafter MR, enhances a low-light image with a frozen latent diffusion model through inversion, attention injection, and decoding, with no training or test-time optimization.

Extending such priors to video has been costly. To our knowledge, ZVRD [2] is the only published training-free method for low-light video enhancement. It adds temporal attention, temporal guidance, and noise sharing to a pre-trained image diffusion model, samples with 950 steps, and optimizes enhancement parameters at test time. ZVRD has no public code. Our implementation takes 90.8 s per frame at the published 256×256256 \times256 geometry on one NVIDIA L40 (Table 1), and about 2,450 s at 1280×7201280 \times720, where each frame is tiled into 50 patches (Figure 1). Its temporal components also do not make it stable: at matched resolution its flicker is 2.7×2.7 \times and its LPIPS almost twice that of framewise MR with eight steps, and it leaves visible noise and color blotches (Table 1 and Figure 1).

Figure 1. Training-free low-light video enhancement at a usable speed. Left: a low-light SDSD frame, shown without brightening, and our enhancement. Right, top: the boxed region from the input, ZVRD (our implementation), our pipeline, and the reference; ZVRD leaves visible noise and color blotches. Right, bottom: seconds per frame on one NVIDIA L40 (log scale). At 256×256256 \times256 the times are those of Table 1, where our pipeline is 261×261 \times faster than ZVRD. At 1280×7201280 \times720, ZVRD tiles each frame and was timed on this clip only; AGLLDiff and our pipeline are averages over SDSD clips, and the gap to ZVRD is about 1,600×1{,}600 \times. The clip is one of the design clips of § 4.1.

MethodPSNR†SSIM†LPIPS†Warp†Flicker†NFEs/frame†Speedup
ZVRD [21] (our impl.)15.81°0.5120.3121.510.28095090.781×
MR E1519.08°0.681°0.1561.000.107300.67135×
MR E818.91°0.6720.1630.940.103160.38239×
Ours19.180.6880.1780.510.087120.35261×

Table 1. Comparison with ZVRD at 256×256256 \times256, six SDSD clips of 30 frames, all methods training-free. MR E15 and MR E8 are framewise MR with 15 and 8 sampling steps. Warp is warping error in units of 10−210^{-2}; NFE is the number of network evaluations per frame; s/frame is measured on one NVIDIA L40. Speedup is the geometric mean over clips of the time ratio to ZVRD. Bold: best in column. ◦: the 95% paired-bootstrap interval of the difference from our pipeline includes zero. The six clips are design clips (§ 4.1); no setting was chosen at this resolution.

We ask what a practical training-free diffusion pipeline for low-light video requires. The starting point is simple: framewise MR with eight sampling steps is already 239×239 \times faster than ZVRD at matched resolution and scores better on every metric (§ 5.1). Two observations then guide the temporal design.

First, sensor noise is the dominant per-frame degradation in low-light video. MR inverts each frame into its diffusion state, so each frame’s noise realization enters its inversion latent, its attention features, and its decoder skips. We therefore apply a classical remedy, motion-compensated recursive temporal filtering [6], to the input of the diffusion model.

Second, the enhancement that a full diffusion trajectory applies changes slowly between adjacent frames, while the trajectory is expensive. SDEdit [32] starts denoising from an intermediate noise level; we use this idea across frames. A key frame runs the full trajectory and records how far its denoised state at an intermediate time departs from its own inversion at that time. Each non-key frame inverts only to that time, adds the warped departure, and denoises from there with its own encoding, attention features, and decoder skips.

The two components are complementary: alone, late-start propagation raises warping error and input fusion adds time; together they are faster and better than framewise processing (Table 3). Our contributions are:

wmax⁡w_{\max}TrajectoryCalib.PSNR↑\uparrowSSIM↑\uparrowLPIPS↓\downarrowWarp↓\downarrowFlicker↓\downarrowTime↓\downarrow
−E8yes17.210.6740.2521.010.1261.00
−E15yes17.350.6730.235°1.150.1321.45‡1.45^\ddagger
0.8E8yes17.49∘17.49^\circ0.7070.2280.500.1091.03
0.8E15yes17.71°0.7100.2130.540.1101.49‡1.49^\ddagger
−late start, offsetyes17.340.6730.235∘0.235^\circ1.120.1310.87
0.8late start, key stateyes17.570.6960.2490.560.1120.91∗0.91^\ast
0.5late start, offsetyes17.560.6980.2060.690.1140.90
0.8late start, offsetno16.240.6740.2280.570.1090.89
0.8late start, offset (ours)yes17.760.7100.2180.520.1090.90

Table 3. Ablation on the 13 held-out clips at 1280×720. wmax⁡w_{\max}: fusion ceiling (− means no fusion). “Key state” starts non-key frames from the warped key-frame state instead of Eq. (6). E8 and E15 are framewise schedules with 8 and 15 steps; late start is § 3.4. Calib.: with exposure calibration. Warp is in units of 10−210^{-2}. Time is relative to E8 timed in the same process, except *, a ratio of separate single runs, and ‡, from the repeated timing of Supp. Tab. 6. Bold: best in column. °: the 95% paired-bootstrap interval of the difference from our pipeline (last row) includes zero.

  • A causal, training-free pipeline for low-light video that needs no training data, retraining, or test-time optimization. At ZVRD’s own 256×256256 \times256 resolution it is 261×261 \times faster than ZVRD and better on all five metrics we report (§ 5.1).

  • Two temporal components for a frozen diffusion prior: temporal input fusion, which filters sensor noise before inversion, and late-start offset propagation, which starts non-key frames from an intermediate time with a transported offset. On held-out 1280×720 clips they improve peak signal-to-noise ratio (PSNR), learned perceptual image patch similarity (LPIPS), and warping error over framewise processing in 10% less time, and the gain holds over 300-frame sequences (Secs. 5.2 and § 5.6).

  • An analysis of why the components work together, with measurements inside the model and a linear-Gaussian model of inversion, and of where they fail: the perceptual gain is lost at low resolution (§ 5.4 and § 5.6).

Related Work

Low-light enhancement and training data. Low-light image enhancement ranges from Retinex-based estimation without learning [12] to models trained on paired images [1, 42, 45], on unpaired images [17], or without references but still on low-light data [11, 30]. Low-light video enhancement has mostly followed the learned route, with paired videos from dedicated rigs [4, 16, 40], synthesized from images [47], or collected in larger benchmarks [9, 26]. Zero-TIG [24] needs no paired references but trains its network on the BVI-RLV video dataset. ReDDiT [23] distills a diffusion enhancer into a few-step student trained on LoLv2 pairs. Among training-free image methods, MR [5] is our backbone, FourierDiff [29] embeds Fourier priors in a pretrained diffusion model, and AGL LDiff [27] guides sampling with attribute gradients at inference. Image enhancers applied frame by frame are unstable over time: on held-out SDSD clips the three ReDDiT models have 3.8–4.8× our warping error (Table 2). Zero-TIG’s flow feedback helps, but its warping error is still twice ours and its LPIPS higher. AGLLDiff’s guidance gradients cost 67 s per 1280×720 frame, and it has the lowest PSNR and highest LPIPS of all methods we ran.

MethodLow-light training dataPSNR↑\uparrowSSIM↑\uparrowLPIPS↓\downarrowWarp↓\downarrowFlicker↓\downarrowNFEs/frame↓\downarrow
ReDDiT-2 [23]LOLv2 pairs17.25∘17.25^\circ0.6170.2861.990.16620.47
ReDDiT-4 [23]LOLv2 pairs18.04∘18.04^\circ0.5820.3022.480.18440.87
ReDDiT-8 [23]LOLv2 pairs18.51∘18.51^\circ0.6090.2952.330.18781.67
Zero-TIG [24]BVI-RLV videos19.09∘19.09^\circ0.7390.2961.100.131n/a0.12
AGLLDiff [27]none13.350.5890.3781.390.13449.967.45
MR A25 [5]none17.470.6720.233∘0.233^\circ1.230.136503.81
MR E15none17.350.6730.235∘0.235^\circ1.150.132302.62
MR E8none17.210.6740.2521.010.126161.70
MR state reusenone16.990.6650.2811.210.13716.51.87
Oursnone17.760.7100.2180.520.109121.52

Table 2. Held-out SDSD clips, 13 clips of 30 frames at 1280×720. The first four rows use low-light training data and are references; they are not ranked. MR A25, E15, and E8 are framewise MR with 25, 15, and 8 sampling steps; MR state reuse warps the key frame’s inversion state to the other frames (§ 4.2). Warp is warping error in units of 10−210^{-2}. NFE is the number of network evaluations per frame, averaged over frames (Zero-TIG is not a diffusion model). Bold: best among training-free methods. °: the 95% paired-bootstrap interval of the difference from our pipeline includes zero. All MR rows and ours include the exposure calibration of § 3.5; the external methods run as released. Times are single runs on one NVIDIA L40 and indicate magnitude only, except that E8 was timed in the same process as our pipeline. The E15 run does not skip replaced attention layers; with the skip E15 takes 2.44 s (Supp. Tab. 6).

Training-free diffusion video restoration and editing. Zero-shot restoration with diffusion priors solves inverse problems with known degradation operators [8, 19, 35]. ZVRD [2] extends it to video with temporal attention, temporal guidance, and noise sharing, and its follow-up adds video diffusion models [3]. Zero-shot video generation and editing methods share attention or features across frames [20, 43]. TokenFlow [10] propagates diffusion features along inter-frame correspondences, FRESCO [44] adds intra-frame correspondence and feature optimization, RAVE [18] shuffles frames between grids during denoising, and DiffIR2VR-Zero [46] adapts restoration diffusion models to video with latent warping and token merging. These methods carry attention features or latent states from frame to frame. For low-light enhancement, a baseline in this spirit, which warps the key frame’s inversion state and attention bank to the other frames, is slower than framewise MR and worse on every metric (Table 2): the transported state carries the key frame’s noise. Late-start propagation differs in what it transports: one latent offset at an intermediate time, added to the current frame’s own partial inversion, while attention features always come from the current frame.

Diffusion acceleration and feature reuse. Denoising diffusion models [14] originally needed hundreds of sampling steps; DDIM [36] and higher-order solvers [28] trade steps against quality. Fewer steps alone leave each frame’s noise in place: with 8, 15, or 25 steps, framewise MR has 1.9–2.4× our warping error (Table 2). Deterministic DDIM inversion maps an image to a noise latent; editing methods refine it [33] or inject features captured during it [39], as MR does. SDEdit [32] starts generation from an intermediate noise level. Late-start propagation is closest to it, but its starting state combines the frame’s own inversion with an offset from a key frame. DeepCache [31] and Clockwork Diffusion [13] reuse U-Net features across the steps of one image and are complementary to reuse across frames.

Temporal filtering of noisy video. Motion-compensated recursive filtering averages each frame with the motion-compensated previous output and lowers the weight where compensation fails [6]. Learned video denoisers [37] require training. We apply the classical filter to the input of a training-free diffusion prior.

Method

Setting and overview

Let xix_i be the ii-th low-light RGB frame, yiy_i its enhanced output, and y~i\tilde{y}_i the output after exposure calibration. Inference is causal: it uses x0,…,xix_0,\ldots,x_i and never a future frame or a normal-light reference. By training-free we mean that no network weight is trained or fine-tuned for low-light enhancement, no low-light training data are collected, and no gradient-based optimization runs at test time. The pipeline has a small number of hand-set constants; § 4.1 states which clips were used to choose them.

Figure 2 shows the four parts: a frozen MR backbone, temporal input fusion, late-start offset propagation, and a causal exposure gain.

Figure 2. Pipeline overview on real SDSD frames (low-light images shown with a display gain). (a) Temporal input fusion averages the current frame xix_i with the flow-warped previous fused input, using a per-pixel weight wiw_i (dark: flow unreliable or content changed). (b) A key frame (orange) runs the full trajectory and stores the latent offset Δk\Delta_k at time tst_s; a non-key frame (purple) inverts only to tst_s, adds the warped offset, and denoises from there. Trajectory thumbnails are illustrative, and Δk\Delta_k is drawn in image space for display. (c) A causal exposure gain gives the output. Key frames cost 30 U-Net evaluations and non-key frames 10.

Framewise MR backbone

Each frame is encoded by the variational autoencoder (VAE) of Stable Diffusion v1.5 [34]; its encoder skip features are kept for decoding. Deterministic inversion with denoising diffusion implicit models (DDIM) [36] maps the latent towards noise with the denoising U-Net and records, at each inversion time, the self-attention outputs of the nine up-block layers. We call this record the attention bank. The inversion endpoint is normalized to the channel statistics of a fixed Gaussian noise sample by adaptive instance normalization (AdaIN) [15]. Denoising then injects the attention bank, and the final latent is decoded with the frame’s own skips [5].

A schedule is a set of diffusion times used for inversion and, reversed, for denoising. MR’s default schedule, A25, has 25 times and costs 50 U-Net evaluations per frame. We use two shorter framewise schedules as controls: E15 with T15={1,67,133,…,925}\mathcal{T}_{15} = \{1, 67, 133, \ldots, 925\} (30 evaluations) and E8 with T8={1,126,…,876}\mathcal{T}_{8} = \{1, 126, \ldots, 876\} (16 evaluations). We write NFE for the number of network evaluations per frame. Implementation constants are in Supp. Sec. 8.

Temporal input fusion

The model input fif_i is a causal recursive average of the low-light frames, with f0=x0f_0=x_0 and

fi=(1−wi)⊙xi+wi⊙f^i,(1)f_i = (1-w_i) \odot x_i + w_i \odot\hat{f}_i, \tag*{(1)}
f^i=Wi−1→i(fi−1).(2)\hat{f}_i = W_{i-1\to i}(f_{i-1}). \tag*{(2)}

Here Wi−1→iW_{i-1\to i} is a backward bilinear warp along Farnebäck flow [7] between xi−1x_{i-1} and xix_i, so fif_i is the motion-compensated history. Let Ri∈[0,1]R_i \in[0,1] mark pixels that pass a forward-backward consistency check. The per-pixel weight is

wi=wmax⁡Riexp⁡(−(di/(2σ^i+10−3))2),(3)w_i = w_{\max} R_i \exp\left(-\left(d_i/(2\hat{\sigma}_i+10^{-3})\right)^2\right), \tag*{(3)}
di=∣G2∗(Y(xi)−Y(f^i))∣,(4)d_i = \left|G_2 * \left(Y(x_i)-Y(\hat{f}_i)\right)\right|, \tag*{(4)}

where YY is the Rec. 709 luma of sRGB values in [0,1][0,1], GsG_s is a Gaussian blur with standard deviation ss pixels, and σ^i=1.4826median⁡∣Y(xi)−G1.5∗Y(xi)∣\hat{\sigma}_i = 1.4826\operatorname{median}|Y(x_i)-G_{1.5}*Y(x_i)| is a robust high-frequency scale of the frame. The exponential term lowers the weight where the warped history disagrees with the current frame; at a difference of twice this scale the weight has fallen to wmax⁡/ew_{\max}/e. We use wmax⁡=0.8w_{\max}=0.8, an effective memory of about five frames. Fusion changes only the input of the diffusion model; calibration and evaluation use the original xix_i.

Late-start offset propagation

Frames with i≡0(modK)i \equiv0 \pmod K, K=10K=10, are key frames. A key frame runs the full E15 trajectory on its fused input, up to the endpoint T=925T=925, and records two latents at an intermediate time ts=265t_s=265: the inversion latent zkinv(ts)z_k^{\mathrm{inv}}(t_s) and the denoising latent zkden(ts)z_k^{\mathrm{den}}(t_s). Their difference

Δk=zkden(ts)−zkinv(ts)(5)\Delta_k = z_k^{\mathrm{den}}(t_s)-z_k^{\mathrm{inv}}(t_s) \tag*{(5)}

is the enhancement that the full trajectory has applied by the time it returns to tst_s. We store Δk\Delta_k, its per-channel spatial mean Δˉk\bar{\Delta}_k, and the key frame’s fused input.

A non-key frame ii encodes its fused input and inverts only over Ts={t∈T15:t≤ts}={1,67,133,199,265}\mathcal{T}_s = \{t \in\mathcal{T}_{15}: t \le t_s\} = \{1,67,133,199,265\}, recording its own attention bank. Flow from the key frame’s fused input to the current one is resized to the latent grid; Wk→iW_{k\to i} is bilinear sampling on this grid, and Mi=1M_i=1 marks latent locations with reliable flow over at least 99.9% of their image footprint. The starting state is

zi(ts)=ziinv(ts)+Mi⊙Wk→i(Δk)+(1−Mi)⊙Δˉk.(6)z_i(t_s) = z_i^{\mathrm{inv}}(t_s) + M_i \odot W_{k\to i}(\Delta_k) + (1-M_i)\odot\bar{\Delta}_k. \tag*{(6)}

Denoising runs over TsT_s in reverse with the frame’s own attention bank, without AdaIN, and the result is decoded with the frame’s own skips.

Only one latent offset is transported per key interval. Attention features and enhanced images are never transported, and the offset is replaced at the next key frame, so errors do not accumulate. A non-key frame costs 10 U-Net evaluations and a key frame 30. A sequence of NN frames therefore needs 10+20⌈N/K⌉/N10 + 20\lceil N/K\rceil/N evaluations per frame on average: 12 for N=30N = 30 or 300, against 16 for E8 and 30 for E15.

Exposure calibration

A causal post-process corrects global brightness with one scalar gain 2ei2^{e_i} applied to linear RGB. Let μx\mu_x and μy\mu_y be the mean luminances of xix_i and yiy_i, and qcq_c the 0.995 luminance quantile of xix_i. The requested correction is

ri=12log⁡20.9μx/qcμy,(7)r_i = \frac{1}{2}\log_2\frac{0.9\mu_x/q_c}{\mu_y}, \tag*{(7)}

clipped to [−0.5,0.5][-0.5, 0.5] exposure values. A guard cancels a brightening request when the 0.999 quantile would no longer call for it, and the gain is smoothed over time as ei=0.8ei−1+0.2e^ie_i = 0.8e_{i-1} + 0.2\hat{e}_i, where e^i\hat{e}_i is the guarded request (Supp. § 9). The calibrator never sees a reference. It is applied identically to every MR configuration in this paper, and we do not claim it as a contribution.

Experimental Setup

Data and splits

SDSD. SDSD [40] provides paired low- and normal-light videos of dynamic scenes. We use its 25 test scenes (12 indoor, 13 outdoor), the first 30 frames of each, resized from 1920×1080 to 1280×720. We split them into 12 design clips and 13 held-out clips (Supp. Tab. 5). The settings of the two temporal components were chosen on the design clips only: wmax⁡w_{\max} and tst_s each from two candidate values, and KK from {5,10,20}\{5, 10, 20\} for the state-reuse baseline of § 4.2. The gate constants of Eqs. (3) and (4) were set by hand and not varied. All SDSD results at 1280×720 are reported on the held-out clips. The E8 schedule and the calibration constants were also set on SDSD: the calibration used six of the held-out clips, and E8 was chosen from framewise measurements on the SDSD test scenes. We therefore also report the seven held-out clips that informed neither the temporal components nor the calibration.

SDE. The independent dataset is the RGB part of SDE [25]: the first 30 frames of its 15 test scenes at the native 346×260, without events. No setting was chosen on SDE. Our full-frame RGB protocol is not comparable with published event-guided results.

Long sequences and stress tests. The long-sequence set contains the first 300 frames of four outdoor SDSD scenes that are outside both splits. Two stress sets reuse these windows, subsampled with stride 2 and 4 (30 and 4 (30 frames each), to simulate faster motion. A third joins the first five frames of one held-out clip to the first 25 frames of another, six times, so that a scene cut falls inside a key interval.

We did not evaluate on DID [9], which we could not obtain, or on BVI-RLV [26], Zero-TIG’s training set.

Compared methods

The only training-free method for low-light video is ZVRD, which we compare at its own resolution (§ 5.1). At 1280×7201280 \times720 the training-free baselines are AGLLDiff [27], run framewise, and MR with its default A25 schedule and with E15 and E8. We add a state-reuse baseline on the same backbone, in the spirit of feature propagation [10]: non-key frames warp the key frame’s inversion endpoint and attention bank and run all 15 denoising steps (16.5 NFE). Zero-TIG [24] and ReDDiT with 2, 4, and 8 steps [23] use low-light training data; we run their public checkpoints as references. DiffIR2VR-Zero, TokenFlow, and VISION-XL [10, 21, 46] have no low-light configuration, and adapting them to MR’s backbone would be a new method, so we do not include them. We did not run FourierDiff [29]; AGLLDiff represents guided training-free image diffusion.

Metrics and timing

We report peak signal-to-noise ratio (PSNR) on full RGB, the structural similarity index (SSIM) [41], and the learned perceptual image patch similarity (LPIPS) [48] with AlexNet. No reference-derived brightness correction enters any score. The two temporal metrics use Farnebäck flow estimated on the normal-light references, with one validity mask shared by all methods. Warping error [22] is the mean absolute RGB difference between the current output and the warped previous output. Flicker is the mean absolute frame-to-frame change, along the reference flow, of the log-luminance ratio between output and reference, so brightness changes that the reference also shows are not counted.

Clips are weighted equally. For a difference between two methods we give a 95% interval from a paired bootstrap over clips (10,000 draws); on sets of four or six sequences the intervals are indicative only. All tables share one format: bold marks the best value among training-free methods, and ◦ marks a value whose difference from our pipeline has an interval that includes zero.

Times are seconds per frame on one NVIDIA L40 with batch size one, from image decoding to the final output, including flow, fusion, and calibration. Where we state that a time is relative to E8 in the same process, E8 was re-timed in the process that timed the method; all other times are separate single runs.

Results

Comparison with ZVRD at its own resolution

At 1280×7201280 \times720 ZVRD’s cost is dominated by tiling, so we compare at its published 256×256256 \times256 geometry. Six SDSD clips are center-cropped and resized to 256×256256 \times256. ZVRD runs with its 950-step configuration and RAFT flow [38] (Supp. § 10).

Table 1 shows that framewise E8 already scores better than ZVRD on all five metrics, at 239× lower cost. Our pipeline is 261× faster than ZVRD and also better on all five metrics; it is better in PSNR and in LPIPS on five of the six clips, and the PSNR interval includes zero. Compared with E8, the pipeline lowers warping error by 46% and flicker by 16% at this resolution, but its LPIPS is worse by 0.015. § 5.6 analyzes this loss.

Held-out SDSD clips at 1280×7201280 \times720

Against framewise processing. On the 13 held-out clips (Table 2), the pipeline improves on E8 by 0.55 dB in PSNR (95% interval 0.27–0.79) and by 0.034 in LPIPS (0.018–0.050). It lowers warping error by 49% and flicker by 14%, and takes 0.90× the time of E8 run in the same process. PSNR is higher on 11 clips, LPIPS lower on 12, and both temporal metrics are lower on all 13. On the seven clips that informed neither the temporal components nor the calibration, the LPIPS gain (0.027, interval 0.016–0.039) and the temporal gains hold on every clip; the PSNR gain is 0.38 dB and its interval includes zero.

Longer schedules do not close the gap. Against E15 and against MR’s default A25, the pipeline gains 0.41 and 0.30 dB in PSNR and lowers warping error by 55% and 58%, in roughly 60% and 40% of their time. Its LPIPS is lower than theirs by 0.017 and 0.015, but these two differences are within the bootstrap interval.

Against other methods. Among training-free methods the pipeline is best on all five metrics. The trained Zero-TIG has higher PSNR, by 1.33 dB, although this interval narrowly includes zero; it also has higher SSIM and is about 12× faster. Our pipeline has lower LPIPS, warping error, and flicker than Zero-TIG on all 13 clips.

Calibration matters for these PSNR comparisons. It accounts for 1.52 dB of the pipeline’s PSNR (Table 3), and the external models have no comparable step. Without calibration the pipeline’s PSNR (16.24 dB) is below that of ReDDiT and Zero-TIG, while its LPIPS (0.228) remains lower than that of every external method and of calibrated E8. The perceptual and temporal gains thus do not depend on calibration; the PSNR ranking against external methods does. Figure 3 shows the lower residual grain on typical frames.

Qualitative comparison on frame 15

Figure 3. Qualitative comparison on frame 15, a non-key frame, of the held-out indoor and outdoor clips whose LPIPS change relative to E8 is the median of their group. Inputs are shown without brightening; insets enlarge the most textured window of the reference (yellow box). † marks methods that require training. Our pipeline shows less residual grain than framewise MR E8 on the shelves and walls; the trained references render brighter images with higher contrast.

Ablation

Each component alone fails one requirement. Table 3 shows that fusion alone improves every quality metric but is slower than E8 (1.03×). Late-start propagation alone is faster (0.87×) but raises warping error by 11%, on 12 of the 13 clips. Combined, they are faster than E8 and better on every metric.

Late start is an accelerator. Fusion on the full E15 trajectory is the quality reference for our pipeline. The pipeline matches its PSNR (the difference of 0.05 dB is within the interval) and loses 0.005 in LPIPS, in 61% of its time. (Supp. § 14).

The transported quantity must be the offset. If non-key frames start from the warped key-frame state zkden(ts)z_k^{\mathrm{den}}(t_s), the LPIPS gain over E8 disappears (0.003, within the interval) at the same cost. Our pipeline has lower LPIPS than this variant on all 13 clips, by 0.031. § 5.4 explains why.

Fusion weight. A lower ceiling, wmax⁡=0.5w_{\max}=0.5, gives the best LPIPS in Table 3 but keeps less of the temporal gain: warping error falls by 31% relative to E8 instead of 49%, and PSNR rises by 0.35 dB instead of 0.55.

Where the temporal gain occurs. Figure 4b shows warping error per frame. On non-key transitions it is about half that of E8 (0.49 against 1.01, in units of 10−210^{-2}). At key-frame refreshes it jumps. At the first refresh, frame 10, it exceeds E8 (1.23 against 0.99, on 10 of the 13 clips); at the second it stays below (0.70 against 1.04). We attribute this to start-up: the first key frame is unfused, f0=x0f_0=x_0, so the fused key frame at frame 10 changes the appearance most (Figure 4a).

Figure 4. Temporal behavior. (a) Temporal residual ∣yi−W(yi−1)∣\lvert y_i - W(y_{i-1})\rvert on a held-out clip, warped with the reference flow, at a non-key transition (top) and at the first key-frame refresh (bottom), for E8, late start without fusion, our pipeline, and the reference; numbers are mean residuals in units of 10−210^{-2}. (b) Warping error per frame index, averaged over the 13 held-out clips, for E8, each component alone, and our pipeline; dotted lines mark key frames. (c) LPIPS of our pipeline minus E8 per frame index on the six scene-cut sequences (solid) and on the same frames without a cut (dashed); the red line marks the cut.

Why the components work together

Two measurements inside the model, made on the six design clips of Table 1 at 1280×7201280 \times720, explain the ablation (Supp. § 14).

Inversion amplifies frame-to-frame noise. We define the relative temporal variation (RTV) of an internal state as the mean squared difference between the state and its flow-warped predecessor, divided by the state’s spatial variance. For raw inputs, RTV grows along inversion from 0.48 at the VAE latent to 1.09 at the endpoint. Fusion removes 52–81% of it in every latent, attention-bank, and VAE-skip state we measured, on all six clips (Supp. Tab. 9).

The offset tolerates flow errors. A flow error of one latent pixel, eight image pixels, changes the transported offset by 0.10 in root mean square (RMS) and a transported state by 1.06. Where flow is reliable, starting from the state differs from Eq. (6) by exactly the content mismatch ci=Wk→i(zkinv(ts))−ziinv(ts)c_i = W_{k\to i}(z_k^{\mathrm{inv}}(t_s))-z_i^{\mathrm{inv}}(t_s). This mismatch grows with the distance to the key frames, as does the LPIPS gap between the two variants, from 0.023 to 0.060 (Supp. Tab. 10).

A linear-Gaussian model. Both effects follow from a simple model (Supp. § 15). Assume the latent prior is Gaussian with variance λk\lambda_k along direction kk. The ordinary differential equation that DDIM discretizes then multiplies direction kk by

gk(t)=αˉt+σt2λk,(8)g_k(t)=\sqrt{\frac{\bar{\alpha}_t+\sigma_t^2}{\lambda_k}}, \tag*{(8)}

where αˉt\bar{\alpha}_t is the cumulative signal coefficient at time tt and σt2=1−αˉt\sigma_t^2=1-\bar{\alpha}_t. Coarse, high-variance content is scaled by about αˉt\sqrt{\bar{\alpha}_t}, whereas low-variance directions, where per-frame noise lives, are raised to the noise level. This is the amplification; fusion removes the noise before it happens. The enhancement enters as AdaIN at the endpoint TT. Carried back to tst_s, Eq. (8) amplifies a constant shift 7.7 times but the finest scales only 0.6 times. Indeed 97% of the offset’s energy lies in its channel means, which no flow error can move.

On the same clips, from the two ends of inversion alone, the model predicts the variance, temporal energy, and band power of all 14 interior inversion states with median errors of 2.3–2.4%. The offset’s channel means are 6.75 times AdaIN’s shift, below the predicted bound of 7.73.

Runtime

On the six design clips of Table 1 at 1280×7201280 \times720, timed in three processes with rotated method order, the pipeline takes 1.54 s per frame, against 1.69 s for E8 and 2.44 s for E15 (Supp. Tab. 6). Key frames take 2.47 s. A non-key frame needs only 10 U-Net evaluations but still 1.43 s, of which VAE encoding and decoding take about 0.68 s.

Transfer, long sequences, and stress tests

Low resolution. The perceptual gain does not transfer to SDE (Table 4). The pipeline raises PSNR by 0.63 dB (interval 0.43–0.83; on 14 of 15 scenes) and lowers warping error by 29% and flicker by 13%. Its LPIPS, however, is worse than that of E8 by 0.028 (0.015–0.042), on 13 of 15 scenes. The single-component rows place most of the loss in fusion: fusion alone costs 0.027 and late start alone 0.007. The same holds at 256×256256 \times256, where fusion alone costs 0.022 (Supp. Tab. 7). The fusion gate does not respond to resolution: the mean applied weight stays between 0.71 and 0.77 on every set (Supp. Tab. 8). Yet a 256×256256 \times256 input carries about half as much frame-to-frame variation as a 1280×7201280 \times720 one (RTV 0.005 against 0.011), so the same averaging plausibly removes more detail relative to noise. Lowering the ceiling to wmax⁡=0.5w_{\max}=0.5 reduces the LPIPS loss on SDE to 0.010 but gives up most of the temporal gain.

SetMethodPSNR↑\uparrowSSIM↑\uparrowLPIPS↓\downarrowWarp↓\downarrowFlicker↓\downarrow
SDEMR E1516.440.530°0.3532.520.202
MR E815.980.537°0.3442.270.185
MR state reuse14.850.4810.4163.760.278
Fusion only16.010.537°0.371°1.500.153
Late start only16.550.536°0.3512.460.199
Ours, wmax⁡=0.5w_{\max}=0.516.61°0.541°0.3532.140.188
Ours16.610.5370.3721.610.160
LongMR E1516.110.4840.347°1.320.167
MR E815.820.4800.3901.130.161
MR state reuse15.820.4800.3981.250.169
Ours16.530.5300.3480.580.145
Stride 2MR E815.520.4830.3841.240.172
Ours16.290.5340.3540.710.158
Stride 4MR E815.590.4790.385°1.400.192
Ours16.360.5280.3690.910.180
Scene cutMR E817.330.6930.2541.560.143
Ours17.830.7260.2231.130.130

Table 4. Transfer and stress tests. SDE: 15 scenes of 30 frames at 346×260346 \times260. Long: 300 frames at 1280×7201280 \times720. Stride 2 and 4: the long windows subsampled, 30 frames each. Long: four SDSD scenes outside the design and held-out splits. Scene cut: six sequences built from held-out clips, with a cut between frames 4 and 5; temporal metrics include the cut transition. All rows include exposure calibration. Fusion only: wmax⁡=0.8w_{\max}=0.8 on E8. Late start only: § 3.4 without fusion. MR state reuse: § 4.2. Warp is in units of 10−210^{-2}. Bold: best in column within each set. °: the 95% paired-bootstrap interval of the difference from our pipeline includes zero.

Long sequences. On the 300-frame windows the pipeline improves on E8 by 0.71 dB in PSNR and 0.042 in LPIPS, on all four scenes, and halves warping error. The gain does not decay with length: over ten consecutive 30-frame bins, the PSNR gain stays between 0.62 and 0.77 dB.

Faster motion and scene cuts. At stride 2 and 4 the pipeline gains 0.77 dB in PSNR in both cases and lowers warping error by 43% and 35%. The LPIPS gain shrinks from 0.030 to 0.016, and at stride 4 it is within the interval. A scene cut costs the pipeline the rest of its key interval (Figure 4c). It is briefly worse than E8 right after the cut, and the next key frame restores the full advantage. Over whole sequences it remains better than E8 (0.50 dB, 0.031 LPIPS).

Limitations

With wmax⁡=0.8w_{\max}=0.8 the perceptual gain does not transfer below 1280×7201280 \times720: LPIPS is worse than that of E8 by 0.028 on SDE and by 0.015 at 256×256256 \times256. All constants were chosen on clips from the same dataset and capture rig as the held-out clips, and the ZVRD comparison uses six of the design clips. SDSD and SDE are robot-captured with slow motion, and fast motion and scene cuts are tested only with sets derived from SDSD. Each key-frame refresh changes the output more than a non-key transition, and a scene cut costs the rest of its key interval. ZVRD is compared only through our implementation, as it has no public code. The trained Zero-TIG has higher PSNR and SSIM and is about 12× faster, and against the longer MR schedules the LPIPS gain is not statistically resolved on 13 clips; on SDE and at 256×256256 \times256, E15 has lower LPIPS than our pipeline. PSNR and flow-based metrics can reward smoothing; LPIPS guards against this only in part, and we ran no user study.

Conclusion

Training-free diffusion enhancement of low-light video optimization: eight-step framewise processing with a frozen image prior is already over two orders of magnitude faster than ZVRD and scores better. Input fusion and late-start offset propagation make it faster and better at 1280×720, on held-out clips and on 300-frame sequences.

Implementation details

Backbone. We use the released MR implementation [5] with Stable Diffusion v1.5, a float16 U-Net, a float32 VAE, and empty text conditioning. The latent is the posterior mean times 0.18215. AdaIN normalizes the inversion endpoint to style noise drawn once with a fixed seed. The two framewise schedules are

T15={1,67,133,199,265,331,397,463,529,595,661,727,793,859,925},\mathcal{T}_{15} = \{1, 67, 133, 199, 265, 331, 397, 463,\\ 529, 595, 661, 727, 793, 859, 925\},
T8={1,126,251,376,501,626,751,876}.\mathcal{T}_{8} = \{1, 126, 251, 376, 501, 626, 751, 876\}.

When every token of a self-attention layer is replaced by the injected bank, we skip the layer’s own attention computation; this leaves the output unchanged. SDE frames (346×260) are encoded at 352×264 and decoded back to the original size.

Flow for fusion. Farnebäck flow [7] is estimated on grayscale frames downsampled so that the long side is at most 320 pixels (pyramid scale 0.5, three levels, window 21, three iterations). A location is reliable if its warp coordinate is in bounds and its forward-backward error is at most 1.5+0.05∥b∥1.5 + 0.05\lVert b\rVert pixels, where bb is the backward displacement. The flow is resized bilinearly to the working resolution, with vectors rescaled per axis; the mask is resized by nearest neighbor.

Latent transport. Flow from the key frame’s fused input to the current fused input is estimated in the same way and resized to the latent grid (160×90 for 1280×720). A latent location is reliable, Mi=1M_i = 1, only if at least 99.9% of its image footprint is reliable and its warp coordinate is in bounds.

State-reuse baseline. At non-key frames this baseline warps the key frame’s inversion endpoint and attention bank, falls back to current-frame noise and attention where flow is unreliable, and runs all 15 denoising steps. With K=10K = 10 it needs (3⋅30+27⋅15)/30=16.5(3 \cdot30 + 27 \cdot15)/30 = 16.5 U-Net evaluations per frame on a 30-frame clip.

Algorithm. For each frame i=0,1,…i = 0, 1, \ldots:

  1. Compute the fused input fif_i with Eqs. (1) to (4) and encode it, keeping the VAE skips.

  2. If i≡0(modK)i \equiv0 \pmod{K}: invert over T15\mathcal{T}_{15} and record the attention bank, apply AdaIN, denoise over T15\mathcal{T}_{15}, and store fif_i, Δk\Delta_k of Eq. (5), and Δˉk\bar{\Delta}_k.

  3. Otherwise: invert over T8\mathcal{T}_8 and record the attention bank, estimate flow from the stored key input to fif_i, form the starting state with Eq. (6), and denoise over T8\mathcal{T}_8.

  4. Decode with the current skips to obtain yiy_i, and calibrate yiy_i against xix_i to obtain y~i\tilde{y}_i.

Exposure calibration

The calibration applies one scalar gain to the three linear RGB channels. Luminance statistics are computed on linear Rec. 709 luminance at a long side of at most 320 pixels. Let μx\mu_x and μy\mu_y be the input and output means and qcq_c, qhq_h the 0.995 and 0.999 quantiles of the input. With ε=10−6\varepsilon= 10^{-6},

mi=0.9μxqc,(9)m_i = 0.9\frac{\mu_x}{q_c}, \tag*{(9)}
ri=12log⁡2max⁡(mi,ε)max⁡(μy,ε),(10)r_i = \frac{1}{2}\log_2\frac{\max(m_i,\varepsilon)}{\max(\mu_y,\varepsilon)}, \tag*{(10)}
bi=clip⁡(ri,−0.5,0.5),(11)b_i = \operatorname{clip}(r_i,-0.5,0.5), \tag*{(11)}
Li=0.9μxmax⁡(qh,ε).(12)L_i = 0.9\frac{\mu_x}{\max(q_h,\varepsilon)}. \tag*{(12)}

The guard holds when Li≤μy≤miL_i \le\mu_y \le m_i: the output is darker than the white point qcq_c asks for, but a brighter white point qhq_h would no longer be safe for brightening. The guarded request is e^i=0\hat{e}_i = 0 if the guard holds and e^i=bi\hat{e}_i = b_i otherwise. The state starts at e0=e^0e_0 = \hat{e}_0, and for i>0i > 0

ei=0.8ei−1′+0.2e^i,(13)e_i = 0.8e'_{i-1} + 0.2\hat{e}_i, \tag*{(13)}

where ei−1′=0e'_{i-1} = 0 if the guard holds and ei−1>0e_{i-1} > 0, or if the guard does not hold and ei−1ri<0e_{i-1}r_i < 0; otherwise ei−1′=ei−1e'_{i-1} = e_{i-1}. We apply 2ei2^{e_i} in linear RGB, clip to [0,1][0,1], and convert to sRGB. The state is reset only at the start of a sequence. The heuristic can fail, for example by darkening an output that is already dark.

ZVRD implementation

ZVRD [2] has no official implementation; at the time of writing the repository linked by its authors contains only a README. We implemented it from the paper, on the Generative Diffusion Prior code [8] that its low-light experiments use, with its 950-step sampling configuration and RAFT flow [38]. Settings that the paper leaves open were fixed without access to ground truth. At 256×256 each frame is one full-frame patch, which we verified to give

SplitSDSD scenesUsed to choose
Design (12)indoor 13, 15, 21, 23, 31, 33; outdoor 1, 5, 14, 36, 46, 48wmax⁡w_{\max}, tst_s, KK; calibration
Held-out (6)indoor 50, 52, 58; outdoor 49, 60, 62calibration
Held-out (7)indoor 60, 68, 70; outdoor 63, 66, 75, 76neither

caption: Roles of the 25 SDSD clips: which clips were used to choose the settings of the temporal components and of the exposure calibration. The temporal components never ran on a held-out clip before their settings were fixed. The E8 schedule was chosen from framewise measurements on the SDSD test scenes.

The same output as the tiled code path at this size. At 1280×720 the patch strategy tiles each frame into 50 overlapping 256×256 patches at every step. Because of the cost, the 1280×720 run covers one clip of 30 frames, the one in Figure 1, and is used only for the time shown there. All ZVRD and MR runs use the same NVIDIA L40.

Data splits

Table 5 lists the clips. On the design clips we compared wmax⁡∈{0.5,0.8}w_{\max} \in\{0.5, 0.8\} and ts∈{265,397}t_s \in\{265, 397\} and chose wmax⁡=0.8w_{\max} = 0.8 and ts=265t_s = 265. The key interval K=10K = 10 is that of the state-reuse baseline, for which K∈{5,10,20}K \in\{5, 10, 20\} were compared on six design clips. The gate constants of Eqs. (3) and (4) were set by hand and not varied. The long windows are outdoor scenes 2, 19, 22, and 37, which are not in the SDSD test configuration and were used for no setting. Each scene-cut sequence joins two held-out clips of the same indoor or outdoor group. SDE was used for no setting.

Evaluation and timing details

SSIM uses an 11×11 Gaussian window with standard deviation 1.5 and is averaged over RGB channels. Evaluation flow is bidirectional Farnebäck flow on the grayscale references at a long side of at most 640 pixels. A pixel is valid if its warp coordinate is in bounds and its forward-backward error is at most 1.5+0.01(∥b∥+∥f∥)1.5 + 0.01(\lVert b \rVert+ \lVert f \rVert) pixels, with bb and ff the backward and forward flows. The same mask is used for every method. For flicker, let pip_i be the output and gig_i the reference, YY linear Rec. 709 luminance, and ai=ln⁡max⁡(Y(pi),10−4)−ln⁡max⁡(Y(gi),10−4)a_i = \ln\max(Y(p_i), 10^{-4}) - \ln\max(Y(g_i), 10^{-4}). Flicker is the mean of ∣ai−Wi−1→i(ai−1)∣\lvert a_i - W_{i-1 \to i}(a_{i-1}) \rvert over valid pixels. The reference flow is an estimate, not ground-truth motion, and is never an input to inference.

Timing starts when the low-light image is decoded and ends when the calibrated output is on the CPU. Model loading, warm-up, and file writing are excluded. The timings of E8, E15, and our pipeline ran on a GPU with no other

Methods/frame↓P95 (s)NFEMemory (GiB)
MR E81.691.70166.1
MR E152.442.46306.4
Ours1.542.49126.4

Table 6. Repeated timing at 1280×720: the six design clips of Tab. 1, 30 frames each, three processes with rotated method order on one NVIDIA L40. NFE is the number of network evaluations per frame. P95 is the 95th-percentile frame time. Memory is the peak allocated GPU memory. The three process means differ by at most 0.013 s for each method. Bold: best in column.

process. Bootstrap intervals are descriptive and are not adjusted for multiple comparisons.

Additional results

Table 7 gives the single-component rows at 256×256. Fusion alone raises LPIPS relative to E8 by 0.022, on all six clips, and late start alone raises it by 0.007. Table 8 shows that the fusion gate rarely closes on these slow-motion data, so fusion acts almost as plain motion-compensated recursive averaging, and that the applied weight is nearly the same at every resolution.

MethodPSNR↑SSIM↑LPIPS↓Warp↓Flicker↓NFEs/frame↓
MR E818.91°0.6720.1630.940.103160.38
Fusion only19.10°0.683°0.1860.500.086160.41
Late start only18.970.6780.1560.960.105120.32
Ours, wmax⁡=0.5w_{\max}=0.519.080.688°0.1560.650.091120.35
Ours19.180.6880.1780.510.087120.35

Table 7. Single components at 256×256, the six design clips of Tab. 1. Fusion only: wmax⁡=0.8w_{\max}=0.8 on E8; late start only: no fusion. Warp is in units of 10−210^{-2}; NFE is the number of network evaluations per frame. Bold: best in column. °: the 95% paired-bootstrap interval of the difference from our pipeline includes zero.

SetClipsσ^i\hat{\sigma}_iMean wiw_iReliable
SDSD, 1280×720, 30 frames250.00550.7550.992
SDSD, 1280×720, 300 frames*60.00570.7600.992
SDSD, 256×25660.00590.7700.989
SDE, 346×260150.00220.7080.990

Table 8. Fusion statistics of our pipeline, averaged over non-initial frames and then clips: the scale σ^i\hat{\sigma}_i of Eq. (3), the applied weight, and the fraction of pixels with reliable flow. The SDSD rows include design clips. *The four long scenes and two design scenes.

Measurements inside the model

RTV. For a state sis_i, RTV at frame ii is the masked mean of ∣si−W(si−1)∣2\lvert s_i - W(s_{i-1}) \rvert^2 divided by the masked spatial variance of sis_i, where WW warps with the reference flow of § 12

State1280×720 Raw1280×720 Fused1280×720 Reduction256×256 Raw256×256 Fused256×256 Reduction
Input luminance0.0110.00278%0.0050.00174%
VAE latent z0z_00.4770.17165%0.1680.06659%
Inversion latent, t=67t=670.6070.23462%0.2220.08462%
Inversion latent, ts=265t_s=2650.8520.36458%0.3550.13464%
Inversion endpoint, t=925t=9251.0920.50655%0.4910.19263%
VAE skip, level 00.0160.00381%0.0090.00278%
VAE skip, level 10.5890.12579%0.1710.03678%
VAE skip, level 20.6690.17274%0.2210.04579%
VAE skip, level 30.6420.19071%0.2010.04777%
Attention bank (range)0.08–0.290.03–0.1152–62%0.19–0.390.09–0.1746–64%

caption: Relative temporal variation (RTV) of internal states for raw and fused input, six design clips of 30 frames. The attention row gives the range over three latent resolutions and three times, t∈{1,265,925}t \in\{1,265,925\}. Reduction is 1−RTVfused/RTVraw1-\mathrm{RTV}_{\mathrm{fused}}/\mathrm{RTV}_{\mathrm{raw}} per clip, averaged; it is positive on every clip in every row.

:::

resized to the state’s grid. Each frame of the six design clips was inverted over T15T_{15} twice, from the raw frame and from the fused input. Table 9 reports the result.

Transport. On the 18 key frames of the same clips we recorded zkinv(ts)z_k^{\mathrm{inv}}(t_s) and Δk\Delta_k. After removing channel means, the spatial RMS of the offset is 0.108, against 0.871 for the state zkden(ts)z_k^{\mathrm{den}}(t_s). A displacement of one latent pixel changes the offset by 0.097 RMS and the state by 1.059. Table 10 reports the content mismatch cic_i by key age.

Key ageMismatch / offsetMismatch / contentLPIPS gap
10.690.590.023
20.850.720.037
30.940.800.043
41.000.840.049
51.030.870.050
61.060.890.051
71.080.920.056
81.100.930.059
91.120.940.060

Table 10. Content mismatch of state transport by key age (frames since the key frame), six design clips, 18 frames per age. Columns: RMS of cic_i relative to the RMS of the transported offset; RMS of cic_i relative to the spatial RMS of the current inversion latent; mean LPIPS of the key-state variant minus that of the offset variant.

A linear-Gaussian model of inversion

Assumptions. (A1) Near the data, the latent prior is Gaussian with mean μ\mu and covariance UΛU⊤U\Lambda U^\top, Λ=diag⁡(λk)\Lambda=\operatorname{diag}(\lambda_k). (A2) A low-light frame is xi=si+nix_i=s_i+n_i, where sis_i follows the flow and nin_i is noise of variance ν\nu per pixel, independent across frames and of sis_i. (A3) The prior is stationary, so UU is the Fourier basis, and warps are translations, which preserve variance and commute with the gain of inversion.

Gain of inversion. Let utu_t be the coordinate of zt−αˉtμz_t-\sqrt{\bar{\alpha}_t}\mu along direction kk, and write λ\lambda for λk\lambda_k. The forward process gives ut=αˉtu0+σtϵu_t=\sqrt{\bar{\alpha}_t}u_0+\sigma_t\epsilon with σt2=1−αˉt\sigma_t^2=1-\bar{\alpha}_t. Then

Var⁡(ut)=αˉtλ+σt2(14)\operatorname{Var}(u_t)=\bar{\alpha}_t\lambda+\sigma_t^2 \tag*{(14)}
∇log⁡pt(ut)=−utαˉtλ+σt2(15)\nabla\log p_t(u_t)=-\frac{u_t}{\bar{\alpha}_t\lambda+\sigma_t^2} \tag*{(15)}
ϵ∗(ut)=σtutαˉtλ+σt2(16)\epsilon^*(u_t)=\frac{\sigma_tu_t}{\bar{\alpha}_t\lambda+\sigma_t^2} \tag*{(16)}

ϵ∗=−σt∇log⁡pt\epsilon^*=-\sigma_t\nabla\log p_t.

Deterministic DDIM discretizes duˉ/dτ=ϵ∗d\bar{u}/d\tau=\epsilon^* in the variables uˉ=ut/αˉt\bar{u}=u_t/\sqrt{\bar{\alpha}_t} and τ=σt/αˉt\tau=\sigma_t/\sqrt{\bar{\alpha}_t} [36]. Substituting ut=αˉtuˉu_t=\sqrt{\bar{\alpha}_t}\bar{u}, with uˉ(0)=u0\bar{u}(0)=u_0,

duˉdτ=σtαˉt uˉαˉtλ+σt2(17)\frac{d\bar{u}}{d\tau}=\frac{\sigma_t\sqrt{\bar{\alpha}_t}\,\bar{u}}{\bar{\alpha}_t\lambda+\sigma_t^2} \tag*{(17)}
=τλ+τ2uˉ(18)=\frac{\tau}{\lambda+\tau^2}\bar{u} \tag*{(18)}
ln⁡uˉ(τ)uˉ(0)=12ln⁡λ+τ2λ(19)\ln\frac{\bar{u}(\tau)}{\bar{u}(0)}=\frac{1}{2}\ln\frac{\lambda+\tau^2}{\lambda} \tag*{(19)}
uˉ(τ)=uˉ(0)1+τ2/λ(20)\bar{u}(\tau)=\bar{u}(0)\sqrt{1+\tau^2/\lambda} \tag*{(20)}
ut=u0αˉt+σt2/λ(21)u_t=u_0\sqrt{\bar{\alpha}_t+\sigma_t^2/\lambda} \tag*{(21)}

Equation (21) is Eq. (8) of the main paper.

Affine law. Let QtQ_t be a quadratic statistic that is diagonal in UU, with per-direction value qkq_k at t=0t=0, and let Q∞=∑kqk/λkQ_\infty=\sum_k q_k/\lambda_k.

Qt=∑kgk(t)2qk(22)Q_t=\sum_k g_k(t)^2q_k \tag*{(22)}
=αˉt∑kqk+σt2∑kqk/λk(23)=\bar{\alpha}_t\sum_k q_k+\sigma_t^2\sum_k q_k/\lambda_k \tag*{(23)}
=αˉtQ0+σt2Q∞(24)=\bar{\alpha}_tQ_0+\sigma_t^2Q_\infty \tag*{(24)}

Under (A3), spatial variance, temporal-difference energy, lag-one autocovariance, and band power are such statistics and obey Eq. (24). RTV is a ratio of two such statistics. As tt grows, the weight σt2/λk\sigma_t^2/\lambda_k favors low-variance directions, so RTV rises whenever frame-to-frame change is relatively larger there, as it is for noise. This is the amplification in Tab. 9.

Fusion as exponential smoothing. With a constant weight ww in Eq. (1), (A2), and (A3), the residual noise ηi\eta_i of the fused input obeys the following along a motion trajectory, where ν\nu is its stationary variance and δi=ηi−ηi−1\delta_i=\eta_i-\eta_{i-1}.

ηi=(1−w)ηi+wηi−1(25)\eta_i=(1-w)\eta_i+w\eta_{i-1} \tag*{(25)}
ν=(1−w)2ν+w2ν(26)\nu=(1-w)^2\nu+w^2\nu \tag*{(26)}
ν=ν1−w1+w(27)\nu=\nu\frac{1-w}{1+w} \tag*{(27)}
E δi2=2ν(1−w)(28)\mathbb{E}\,\delta_i^2=2\nu(1-w) \tag*{(28)}
=2ν(1−w)21+w(29)=2\nu\frac{(1-w)^2}{1+w} \tag*{(29)}

Raw input has frame-difference noise energy 2ν2\nu. At the mean weight measured on the six design clips, w=0.754w = 0.754, Eq. (29) leaves about 3.5% of it; this is approximate, because the weight varies over pixels.

The offset is mostly a constant. The enhancement enters as AdaIN at T=925T = 925, whose shift is constant in space. Carried back to tst_s, a perturbation along direction kk is multiplied by Gk=gk(ts)/gk(T)G_k = g_k(t_s)/g_k(T). For the constant direction (λk→∞\lambda_k \to\infty), Gk→αˉts/αˉT=7.73G_k \to\sqrt{\bar{\alpha}_{t_s}/\bar{\alpha}_T} = 7.73; for the finest scales (λk→0\lambda_k \to0), Gk→σts/σT=0.60G_k \to\sigma_{t_s}/\sigma_T = 0.60. Denoising thus acts on the enhancement as a low-pass filter. On the 18 key frames, 97.4% of the offset’s energy is in its channel means.

Testing the model. Table 11 tests Eq. (24): from the VAE latent and the endpoint t=925t = 925 alone, we predict four statistics at the 14 interior times of T15\mathcal{T}_{15}. The largest errors are an underestimate of temporal energy by 12–15% at t=67t = 67–199 at 1280×720: the first inversion steps amplify frame-to-frame variation somewhat more than the model does. Regressing the offset’s channel means on AdaIN’s channel shifts through the origin gives a slope of 6.75 (correlation 0.997, 72 values), below the bound of 7.73.

ResolutionInputVV and NN (median / max)Band power (median / max)ρ1\rho_1 (median / max)
1280×720Raw4.2% / 14.0%3.5% / 11.5%0.001 / 0.027
1280×720Fused3.5% / 14.6%3.3% / 11.5%0.002 / 0.027
256×256Raw1.6% / 8.3%1.7% / 4.1%0.001 / 0.008
256×256Fused2.1% / 5.4%2.1% / 4.8%0.001 / 0.006

Table 11. Errors of the affine law (Eq. (24)) at the 14 interior times of T15\mathcal{T}_{15}, predicted from the two ends of inversion, six design clips. VV: spatial variance; NN: temporal-difference energy; band power: eight radial frequency bands; ρ1\rho_1: lag-one spatial autocorrelation. Errors are relative for VV, NN, and band power and absolute for ρ1\rho_1. Over all cells the median error is 2.4% for VV and NN and 2.3% for band power.

The model treats noise propagation as linear. Table 12 shows that this fails: doubling injected noise multiplies the response energy by 4.00 only in the first VAE convolution, and by 1.75–2.77 in every later state. The model therefore explains where noise is amplified and why the offset transports well, but not how much noise each state receives.

StateEnergy ratioRelative response
VAE skip, level 0 (first convolution)4.000.005
VAE skip, levels 1–32.20–2.490.14–0.18
VAE latent z0z_02.210.10
Inversion latent, t=67t = 672.110.13
Inversion latent, ts=265t_s = 2651.880.21
Inversion endpoint, t=925t = 9251.750.42
Attention bank (range)2.19–2.770.06–0.14

Table 12. Response to injected noise, 60 fused frames of the six design clips at 1280×720. White noise ξ\xi with standard deviation 0.01 is added to the input. The attention row gives the range over three latent resolutions and three times. Energy ratio: median over frames of the response energy for 2ξ2\xi over that for ξ\xi; a linear map gives 4. Relative response: median response energy for ξ\xi divided by the state’s variance.

References

  1. [1]Yuanhao Cai, Hao Bian, Jing Lin, Haoqian Wang, Radu Timofte, and Yulun Zhang. Retinexformer: One-stage retinex-based transformer for low-light image enhancement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12470–12479, 2023. 2arxiv.org/abs/2303.06705
  2. [2]Cong Cao, Huangjing Yue, Xin Liu, and Jingyu Yang. Zero-shot video restoration and enhancement using pre-trained image diffusion model. Proceedings of the AAAI Conference on Artificial Intelligence, 39(2):1935–1943, 2025. 1, 2, 5
  3. [3]Cong Cao, Huangjing Yue, Shangbin Xie, Xin Liu, and Jingyu Yang. Zero-shot video restoration and enhancement with assistance of video diffusion models. arXiv preprint arXiv:2601.21922, 2026. 2arxiv.org/abs/2601.21922
  4. [4]Chen Chen, Qifeng Chen, Minh N. Do, and Vladlen Koltun. Seeing motion in the dark. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3184–3193, 2019. 2DOI
  5. [5]Joshua Cho, Sara Aghajanzadeh, Zhen Zhu, and David Forsyth. MR. Illuminate: Zero-shot low-light image enhancement with diffusion prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8460–8470, 2026. 1, 2, 3, 6
  6. [6]E. Dubois and S. Sabri. Noise reduction in image sequences using motion-compensated temporal filtering. IEEE Transactions on Communications, 32(7):826–831, 1984. 2, 3
  7. [7]Gunnar Farnebäck. Two-frame motion estimation based on polynomial expansion. In Image Analysis (SCIA 2003), Lecture Notes in Computer Science, pages 363–370, 2003. 3, 1DOI
  8. [8]Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang, Tianyue Luo, Bo Zhang, and Bo Dai. Generative diffusion prior for unified image restoration and enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9935–9946, 2023. 2, 1arxiv.org/abs/2304.01247
  9. [9]Huiyuan Fu, Wenkai Zheng, Xicong Wang, Jiaxuan Wang, Heng Zhang, and Huadong Ma. Dancing in the dark: A benchmark towards general low-light video enhancement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12831–12840, 2023. 1, 2, 5DOI
  10. [10]Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. TokenFlow: Consistent diffusion features for consistent video editing. In International Conference on Learning Representations (ICLR), 2024. 2, 5arxiv.org/abs/2307.10373
  11. [11]Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Rummnin Cong. Zero-reference deep curve estimation for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1777–1786, 2020.arxiv.org/abs/2001.06826
  12. [12]Xiaojie Guo, Yu Li, and Haibin Ling. LIME: Low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing, 26(2):982–993, 2017.DOI
  13. [13]Amirhossein Habibian, Amir Ghodrati, Noor Fathima, Guillaume Sautiere, Risheek Garrepalli, Fatih Porikli, and Jens Petersen. Clockwork diffusion: Efficient generation with model-step distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8352–8361, 2024.arxiv.org/abs/2312.08128
  14. [14]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), pages 6840–6851, 2020.arxiv.org/abs/2006.11239
  15. [15]Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1510–1519, 2017.arxiv.org/abs/1703.06868
  16. [16]Haiyang Jiang and Yinqiang Zheng. Learning to see moving objects in the dark. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7323–7332, 2019.DOI
  17. [17]Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. EnlightenGAN: Deep light enhancement without paired supervision. IEEE Transactions on Image Processing, 30:2340–2349, 2021.DOI
  18. [18]Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James M. Rehg, and Pinar Yanardag. RAVE: Randomized noise shuffling for fast and consistent video editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6507–6516, 2024.arxiv.org/abs/2312.04524
  19. [19]Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems (NeurIPS), pages 23593–23606, 2022.arxiv.org/abs/2201.11793
  20. [20]Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphey Shi. Text2Video-Zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15908–15918, 2023.arxiv.org/abs/2303.13439
  21. [21]Taesung Kwon and Jong Chul Ye. VISION-XL: High definition video inverse problem solver using latent image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10465–10474, 2025.arxiv.org/abs/2412.00156
  22. [22]Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. Learning blind video temporal consistency. In European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, pages 179–195, 2018.arxiv.org/abs/1808.00449
  23. [23]Guanzhou Lan, Qianli Ma, Yuqi Yang, Zhigang Wang, Dong Wang, Xuelong Li, and Bin Zhao. Efficient diffusion as low light enhancer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21277–21286, 2025. 2, 5, 6arxiv.org/abs/2410.12346
  24. [24]Yini Li and Lianxuan Anantrasirichai. Zero-TIG: Temporal consistency-aware zero-shot illumination-guided low-light video enhancement. In European Signal Processing Conference, 2025. 2, 5, 6
  25. [25]Guoqiang Liang, Kanghao Chen, Hangyu Li, Yunfan Lu, and Lin Wang. Towards robust event-guided low-light image enhancement: A large-scale real-world event-image dataset and novel approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23–33, 2024. 4arxiv.org/abs/2404.00834
  26. [26]Ruirui Lin, Guoxi Huang, Joanne Lin, Qi Sun, Alexandra Malyugina, David R. Bull, and Nantheera Anantrasirichai. BVI-RLV: A fully registered dataset for low-light video enhancement. arXiv preprint arXiv:2407.03535, 2024. 1, 2, 5arxiv.org/abs/2407.03535
  27. [27]Yunlong Lin, Tian Ye, Sixiang Chen, Zhenqi Fu, Yingying Wang, Wenhao Chai, Zhaohu Xing, Lei Zhu, and Xinghao Ding. AGLLDiff: Guiding diffusion models towards unsur-pervised training-free real-world low-light image enhancement. Proceedings of the AAAI Conference on Artificial Intelligence, 39(5):5307–5315, 2025. 2, 5, 6
  28. [28]Cheng Lu, Yuhua Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems (NeurIPS), pages 5775–5787, 2022. 2arxiv.org/abs/2206.00927
  29. [29]Xiaoqian Lv, Shengping Zhang, Chenyang Wang, Yichen Zheng, Bineng Zhong, Chongyi Li, and Liqiang Nie. Fourier priors-guided diffusion for zero-shot joint low-light enhancement and deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 25378–25388, 2024. 2, 5DOI
  30. [30]Long Ma, Teng Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo. Toward fast, flexible, and robust low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5627–5636, 2022. 2arxiv.org/abs/2204.10137
  31. [31]Xinyin Ma, Gongfan Fang, and Xinchao Wang. DeepCache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15762–15772, 2024. 3arxiv.org/abs/2312.00858
  32. [32]Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations (ICLR), 2022. 2, 3arxiv.org/abs/2108.01073
  33. [33]Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6038–6047, 2023. 3arxiv.org/abs/2211.09794
  34. [34]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022. 3arxiv.org/abs/2112.10752
  35. [35]Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), pages 49960–49990, 2023. 2arxiv.org/abs/2307.00619
  36. [36]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021. 2, 3arxiv.org/abs/2010.02502
  37. [37]Matias Tassano, Julie Delon, and Thomas Veit. FastDVDnet: Towards real-time deep video denoising without flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1351–1360, 2020. 3arxiv.org/abs/1907.01361
  38. [38]Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, pages 402–419, 2020. 5, 1arxiv.org/abs/2003.12039
  39. [39]Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1921–1930, 2023. 3arxiv.org/abs/2211.12572
  40. [40]Ruixing Wang, Xiaogang Xu, Chi-Wing Fu, Jiangbo Lu, Bei Yu, and Jiaya Jia. Seeing dynamic scene in the dark: High-quality video dataset with mechatronic alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 1, 2, 4DOI
  41. [41]Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004. 5pubmed.ncbi.nlm.nih.gov/15376593
  42. [42]Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. In British Machine Vision Conference (BMVC), 2018. 2arxiv.org/abs/1808.04560
  43. [43]Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Renderer a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia 2023 Conference Papers, pages 1–11, 2023. 2arxiv.org/abs/2306.07954
  44. [44]Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. FRESCO: Spatial-temporal correspondence for zero-shot video translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8703–8712, 2024. 2arxiv.org/abs/2403.12962
  45. [45]Wenhan Yang, Wenjing Wang, Haofeng Huang, Shiqi Wang, and Jiaying Liu. Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Transactions on Image Processing, 30:2072–2086, 2021. 2DOI
  46. [46]Chang-Han Yeh, Hau-Shiang Shiu, Chin-Yang Lin, Zhixiang Wang, Chi-Wei Hsiao, Ting-Hsuan Chen, and Yu-Lun Liu. DiffIR2VR-Zero: Zero-shot video restoration with diffusion-based image restoration models. arXiv preprint arXiv:2407.01519, 2025. 2, 5arxiv.org/abs/2407.01519
  47. [47]Fan Zhang, Yu Li, Shaodi You, and Ying Fu. Learning temporal consistency for low light video enhancement from single images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4965–4974, 2021. 2DOI
  48. [48]Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 586–595, 2018. 5arxiv.org/abs/1801.03924

Paper details

vlnc:W_drtqqfue
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents