betaFeedback

WorksComputer Science

From Retouched Images to Paired Videos: A Synthetic Benchmark for Face Video Retouching

Abstract

Paired unretouched/retouched face videos are costly to produce because professional editing must remain consistent over time, while releasing identifiable face footage raises privacy, consent, and governance concerns. We present DeepVFR, an initial controlled synthetic benchmark-not a new retouching architecture-for paired face video retouching. Starting from paired synthetic DeepFRR images, Wan2.2 generates mild-motion source clips, and backend-specific face transfer constructs synchronized proxy retouching targets, followed by automatic validity screening and manual visual inspection. DeepVFR contains 1,024 pairs of 150-frame clips at 30 fps, evaluated under an identity-disjoint protocol with explicit frame correspondence; release metadata will record the construction route. Within-benchmark baselines, external checkpoints applied to DeepVFR, qualitative analysis, and a 17-participant ranking study establish reference behavior in the controlled domain. The constructed targets exhibit a substantial low-preference tail in the human study. Because both streams inherit synthetic content and the targets additionally inherit commercial-API or face-transfer artifacts, the benchmark neither replaces artist-retouched captured video nor establishes synthetic-to-real generalization. DeepVFR provides a starting point for paired supervision and controlled evaluation. Data, individual study rankings, and training logs will be made publicly available upon acceptance. Broader motion coverage and real-video validation remain future work.

Introduction

Portrait videos have become a central form of visual self-presentation on social media, turning face retouching from a photo-editing operation into a large-scale video-processing problem. With the rapid growth of short-form video platforms and mobile content creation [13], users increasingly expect face-centric videos to be enhanced in terms of skin texture, facial tone, blemish removal, and overall portrait quality. However, manual video retouching remains labor-intensive and difficult to scale, since edits must be applied consistently across frames while preserving identity, expression, pose, and natural facial motion.

Automatic face and portrait retouching has been extensively studied in the image domain. General and portrait photo retouching datasets, such as MIT-Adobe FiveK [4] and PPR10K [19], have enabled learning-based photographic enhancement and portrait-specific retouching. Face-oriented resources cover a broader range of retouching-related settings. EFR studies exemplar-based freckle retouching using before-and-after facial images from cosmetic laser treatment [21]. FFHQ provides high-quality face images for generative modeling [11], and FFHQ-R builds upon it with professionally retouched face images for automatic face retouching [26]. RetouchingFFHQ further scales face retouching data to over half a million retouched face images for fine-grained retouching detection [50]. DigiFace-1M provides more than one million synthetic face images with diverse poses, lighting, and backgrounds, mainly for face recognition [3]. More recently, HGFR compares these representative resources, including FRTDD, FFHQ, and FFHQR, and introduces 25,000 high-definition paired face retouching images with rich blemish types (Xu et al., 2025).

Despite this growing dataset landscape, existing resources remain fundamentally image-centric. They focus on portrait photo retouching, face generation, face recognition, retouching detection, or paired image-level blemish removal, but do not provide temporally aligned before-and-after face videos. This limitation is critical for video face retouching. Applying an image retouching model frame by frame often causes temporal flickering, unstable facial details, and inconsistent identity-related attributes, a problem also observed in general video processing and portrait video stylization tasks [16, 48]. Therefore, learning a practical video face retouching model requires paired video-level supervision, where the input and retouched videos share the same identity, expression, pose, motion, and camera trajectory while differing mainly in retouching style.

Constructing such paired video data in the real world is difficult. Producing high-quality retouched videos requires professional editing and careful temporal adjustment, making large-scale annotation expensive. Moreover, face videos contain sensitive identity information, and collecting paired unretouched/retouched videos raises additional privacy, consent, and dataset governance concerns [9]. As a result, although image-level face retouching datasets have become increasingly available, there is still no widely adopted paired benchmark for video face retouching.

The most related supervision sources are FFHQR-Seq and MRFV introduced in VRetouchEr [45]. As summarized in Table 1, FFHQR-Seq is constructed from FFHQR image pairs by applying random cropping, flipping, and translation to generate short paired sequences. Although it provides many paired samples, its temporal variations mainly come from geometric augmentation rather than facial motion, expression dynamics, or camera movement. MRFV contains 200 in-the-wild portrait videos with manually retouched frames and is therefore closer to the intended real-video task, but its construction is costly and the data have not been publicly released in the official repository. Existing resources consequently leave a practical gap for a fixed, inspectable benchmark with seconds-long paired clips.

BenchmarkScaleTemporal unitMotion sourcePublic splitPaired beauty task
FFHQR-Seq ([45])56k/7k/7k seq.5 frames/sampleGeometric augmentationYesYes
MRFV ([45])200 videos≥500\ge500 frames/videoReal portrait videosNoYes
DeepVFR (ours)1,024 videos150 frames/videoPaired I2V videosPlannedYes

Table 1. Representative supervision sources for video face retouching.

To address this gap, we present DeepVFR (video face retouching), distinct from its image source DeepFRR, an initial controlled synthetic benchmark that converts paired image supervision into paired video supervision. Starting from each raw/retouched image pair, an image-to-video generator supplies a shared mild-motion trajectory, after which backend-specific face transfer constructs a synchronized proxy retouching target. The off-the-shelf generators are construction tools rather than a proposed retouching architecture. Our contribution is the paired benchmark, its supervision interface, and a unified evaluation protocol. The resulting targets are not professionally edited captured videos, and we therefore restrict our claims to the controlled synthetic domain. Our contributions are summarized as follows:

  • We introduce an initial controlled paired-video benchmark for face retouching. It contains 1,024 raw and constructed-target clip pairs, each with 150 frame correspondences, and uses identity-disjoint splits. The 1,024 clips—rather than the correlated frame count—are the benchmark’s clip-level temporal units.

  • We specify a benchmark-construction and planned-release protocol that combines image-to-video synthesis, Akool-based primary transfer, a smaller complementary SimSwap route, quality control, per-clip route metadata, and explicit raw/target frame correspondence. This protocol supplies paired supervision without claiming a new generation model.

  • We report controlled-domain baselines using frame-fidelity and temporal metrics, released external checkpoints, qualitative analysis, and a 17-participant ranking study. The experiments establish reference behavior on DeepVFR; they do not establish transfer from DeepVFR to artist-retouched or in-the-wild captured videos.

Related Work

Image face retouching datasets. Image-level retouching predates video-level supervision and has been studied extensively. Representative image-level resources related to facial retouching are summarized in Table 2. General photo-retouching datasets such as MIT-Adobe FiveK [4] and portrait-specific datasets such as PPR10K [19] provide paired image supervision, but they mainly focus on global photographic enhancement or portrait tone adjustment rather than face-level retouching. Face-oriented resources cover beauty perception, makeup transfer, and face retouching detection. SCUT-FBP and SCUT-FBP5500 provide facial attractiveness annotations ([20, 39]); EFrR studies freckle removal and skin-tone adjustment using before/after cosmetic-treatment exemplars ([21]); FFHQ and FFHQR provide high-quality face images and retouched face images ([11, 26]); RetouchingFFHQ scales fine-grained face retouching detection to over half a million conditionally retouched images ([50]). More recently, HGFR summarizes image-level face retouching resources and introduces 25,000 high-definition paired face retouching images (Xu et al., 2025). These datasets are valuable for image retouching, but they do not provide frame-synchronized raw/retouched video pairs.

DatasetScalePairedMain focus
MIT-Adobe FiveK [4]5k imagesYesGeneral photo retouching
PPR10K [19]11,161 portraitsYesPortrait photo retouching
SCUT-FBP5500 [20]5,500 facesNoBeauty perception
EFfR [21]Small exemplar setYesFreckle/skin retouching
FFHQ/FFHQR [11, 26]Face imagesPartialFace generation/retouching
RetouchingFFHQ [50]>0.5M imagesConditionalRetouching detection
DeepFRR [40]50k pairsYesFace retouching reversal
HGFR (Xu et al., 2025)25k pairsYesHigh-definition face retouching

Table 2. Representative image-level resources related to facial beauty and retouching. “Paired” indicates whether an original/beauty or before/after image relation is available.

Among these resources, we choose DeepFRR as the image source for DeepVFR for three reasons ([40]). First, DeepFRR provides one-to-one raw/retouched face pairs, which is necessary for constructing paired supervision. Second, its images are high-resolution synthetic faces generated by StyleGAN, reducing the privacy risk compared with directly collected real identities ([12]). Third, its 50k-pair scale gives a sufficiently large candidate pool for quality filtering, demographic balancing, and replacement of failed video-generation or transfer cases. Therefore, DeepFRR is more suitable than unpaired beauty-score datasets, detection-oriented retouching datasets, or general portrait-retouching datasets for constructing paired face video retouching supervision.

Face swapping and identity transfer. Face swapping aims to transfer the identity or facial appearance of a source face to a target face while preserving target pose, expression, illumination, and background. Early and representative studies include automatic face reenactment ([8]), Face2Face ([30]), X2Face ([36]), Deep Video Portraits ([15]), FSGAN ([23]), and Neural Textures ([29]). Later approaches include FaceShifter (Li et al., 2020), SimSwap ([5]), MegaFS ([55]), HifiFace ([33]), high-resolution latent face swapping (Xu et al., 2022), FaceDancer ([24]), 3D-aware face swapping (Li et al., 2023), and BlendFace ([27]). In addition to academic models, commercial APIs such as the Akool Face Swap API provide face-transfer interfaces for image and video processing ([1]). In this work, face swapping is a construction tool for transferring the paired retouched appearance, not the retouching model being benchmarked.

Video face retouching datasets. Compared with image face retouching, video face retouching requires temporally aligned supervision. Existing face video datasets such as FaceForensics++ ([25]) and CelebV-HQ ([54]) contain facial videos, but they are not designed for paired raw and retouched video supervision. The most related resources are FFHQR-Seq and MRFV introduced in VRetouchER ([45]). FFHQR-Seq is built from FFHQR image pairs by random cropping, flipping, and translation, producing short pseudo-sequences rather than natural facial motion. MRFV contains 200 manually retouched portrait videos, but it is small and not publicly released in the official repository ([44]). These limitations motivate DeepVFR, which converts paired image supervision into seconds-long, frame-synchronized video pairs.

Recent model-centric work explores mixtures of diffusion models, instruction-following retouching agents, and reward-guided aesthetic alignment ([22, 47] (Wu et al. 2026)). These methods broaden the space of retouching operators, whereas DeepVFR addresses the complementary problem of paired temporal supervision and controlled evaluation. Deepfake resources such as DF40 and FakeAVCeleb focus on identity manipulation or forgery analysis ([14, 46]). Their real/fake labels and identity-replacement targets do not provide the same-identity raw/beauty correspondence required for supervised face video retouching.

Benchmark Construction and Protocol

This section specifies how the DeepVFR collection is constructed and how its paired supervision is used. It does not introduce a new retouching network. We describe source-pair curation, I2V-based motion synthesis, backend-specific target construction, quality control, and the common training/evaluation interface; release-specific identifiers and integrity records will accompany the data package planned for release upon acceptance.

Overview of the DeepVFR Construction Pipeline

We construct DeepVFR through a paired image-to-video pipeline, as illustrated in Figure 1. Given a raw/retouched image pair from DeepFRR, we first generate a raw video from the raw image and then construct a temporally aligned proxy target using the paired retouched image. Specifically, the raw face image is animated into a short portrait video, after which the retouched appearance is transferred frame by frame while attempting to retain the generated pose, expression, and motion trajectory. This produces synchronized pairs for training and evaluating face video retouching models under controlled paired supervision.

Overview of the DeepVFR construction pipeline

Figure 1. Overview of the DeepVFR construction pipeline. An I2V model animates the raw member of each DeepFRR pair; backend-specific transfer and parsing-guided blending then construct a frame-synchronized proxy target from the paired retouched image. The encoder/ID-block/decoder schematic illustrates the complementary SimSwap route; it does not describe the proprietary Akool backend. The right side shows representative raw/target frames.

Source Pair Curation from DeepFRR

Let Dimg={(xi,yib)}i=1ND_{\mathrm{img}} = \{(x_i, y_i^b)\}_{i=1}^{N} denote the DeepFRR image corpus, where xix_i is the raw face image and yiby_i^b is its corresponding retouched image [40]. In the actual construction process, we select 1,024 image pairs from DeepFRR through a combination of quality filtering and stratified sampling. The filtering procedure follows four criteria. First, each sample pair must contain a single clear frontal or near-frontal face in both the raw image and the retouched image to ensure stable alignment and transfer. Second, the facial region should have sufficient resolution, complete facial components, and no severe occlusion, watermark, or obvious generation artifacts. Third, the retouched image should preserve identity while exhibiting visible but not excessive retouching effects, so as to avoid style distortion in the constructed results. Fourth, the selected subset should cover diverse ages, genders, skin tones, illumination conditions, and expression tendencies as much as possible, thereby improving the overall diversity of the dataset distribution.

If a sample fails a later generation or transfer check, it is replaced with the next valid candidate from the same stratum. This procedure yields 1,024 retained pairs that satisfy the stated inclusion criteria. It does not make the groups perfectly balanced; the remaining demographic imbalance is reported in § 4.

I2V-Based Raw Video Synthesis

For each selected raw image xix_i, we generate a corresponding raw video using the Tongyi Wanxiang 2.2 image-to-video model [2, 31, 32]. It should be emphasized that the I2V model is used here only as a motion-prior generator rather than as the final retouching model. To induce natural micro-expressions and slight head motion while avoiding large pose changes that may break identity alignment, we adopt a mild expression prompt, such as smile, during video generation. For each sample, the generated video is uniformly set to T=150T = 150 frames with a frame rate of f=30f = 30 fps, corresponding to a clip of approximately 5 seconds:

Oi={Oi,t}t=1T=GI2V(xi;p,T,f),T=150,f=30,(1)O_i = \{O_{i,t}\}_{t=1}^{T} = G_{\mathrm{I2V}}(x_i; p, T, f), \quad T = 150, \quad f = 30, \tag*{(1)}

where GI2VG_{\mathrm{I2V}} denotes the image-to-video generator, pp denotes the text prompt, and Oi,tO_{i,t} denotes the raw video generated in frame tt.

We adopt this setting for three reasons. First, 150-frame clips expose adjacent-frame flicker and appearance drift over a longer span than five-frame pseudo-sequences. Second, 30 fps is compatible with common video-processing pipelines. Third, about 5 seconds balances temporal coverage, generation failures, storage, and training cost. The resulting trajectories are deliberately mild; they are useful for controlled temporal evaluation but do not cover rapid speech, large pose changes, occlusion, or dynamic lighting.

Face-Transfer-Based Retouched Video Construction

After obtaining the raw video OiO_i and its corresponding retouched image yiby_i^b, we construct the proxy retouched video Bi={Bi,t}t=1TB_i = \{B_{i,t}\}_{t=1}^{T}. For each Oi,tO_{i,t}, we use yiby_i^b as the source appearance and the current raw frame as the motion/background target, then apply a backend-specific face-transfer operator TbiT_{b_i}, where bib_i denotes SimSwap or the Akool API. The process is

B~i,t=Tbi(Abi(yib,Oi,t)),(2)\widetilde{B}_{i,t} = T_{b_i}\left(\mathcal{A}_{b_i}(y_i^b, O_{i,t})\right), \tag*{(2)}
Bi,t=α(Mi,t)⊙B~i,t+(1−α(Mi,t))⊙Oi,t,(3)B_{i,t} = \alpha(M_{i,t}) \odot\widetilde{B}_{i,t} + \left(1-\alpha(M_{i,t})\right) \odot O_{i,t}, \tag*{(3)}

where Abi\mathcal{A}_{b_i} denotes the detector-and-alignment transform required by backend bib_i, implemented with SCRFD or RetinaFace [6, 10]; Mi,tM_{i,t} is the BiSeNet face-parsing mask (Yu et al., 2018); α(Mi,t)\alpha(M_{i,t}) is its feathered alpha mask; and ⊙\odot is element-wise multiplication. The blend retains the background, hair, and non-face regions of Oi,tO_{i,t} while inserting the transferred facial region.

Akool is the primary production transfer route, while SimSwap supplies a smaller complementary subset [1, 5]. Under SimSwap, identity features from the paired retouched image condition the generator while the target frame supplies pose and expression. Identity conditioning does not explicitly encode skin-retouching edits, so the transferred appearance remains proxy supervision. Both routes are converted to the same clip format, and the planned release manifest will identify the backend used for each clip. Because Akool is a mutable commercial service, a future versioned fixed package and its provenance could support repeatable scoring, but exact regeneration may change when the service changes.

After transfer, automatic checks verify detection and alignment, while visual inspection rejects severe motion collapse, obvious boundary artifacts, loss of the intended retouched appearance, or clear identity drift. Rejected clips are replaced from the same sampling stratum. This hybrid screen removes conspicuous failures from the current 1,024 pairs; it cannot detect every latent synthetic or backend artifact, and its manual component is a bottleneck for expansion.

Representative DeepVFR samples

Figure 2. Representative DeepVFR samples. Each group contains three frames from one clip; the six examples were selected from metadata strata crossing age group (child, youth, elderly) and gender (male, female).

Paired Supervision Protocols Supported by DeepVFR

After the above steps, the resulting DeepVFR dataset can be expressed as

DVFR={(Oi,Bi)}i=11024,(4)D_{\mathrm{VFR}} = \left\{(O_i, B_i)\right\}_{i=1}^{1024}, \tag*{(4)}

where OiO_i denotes the raw video and BiB_i denotes the frame-aligned retouched video. Each sample contains 150 synchronized frames, so that every raw frame Oi,tO_{i,t} has a corresponding retouched target Bi,tB_{i,t}. This representation makes DeepVFR directly usable under both image-level and video-level retouching settings.

For image-based retouching methods, DeepVFR can be decomposed into individual frame pairs (Oi,t,Bi,t)(O_{i,t}, B_{i,t}), allowing existing image models to be trained without changing their input–output formulation. For video-based methods, the ordered sequences (Oi,Bi)(O_i, B_i) share frame indices and are constructed to follow the same nominal motion and expression trajectory, enabling models to exploit cross-frame information rather than relying only on pseudo-sequences or frame-wise augmentation.

We keep this protocol intentionally simple. All methods take raw frames or clips as input and use the corresponding retouched frames or clips as supervision, with the sampling unit being the only difference between single-frame and temporal-sequence settings. In this way, DeepVFR separates the benchmark contribution from any particular network design and provides a controlled basis for comparing image-based and video-based face retouching methods under the same paired supervision source.

DeepVFR Dataset

Dataset Contents and Scope

DeepVFR is a controlled synthetic benchmark for video face retouching. Each sample contains a generated raw portrait clip and a frame-synchronized proxy retouching target constructed from the same nominal trajectory. The streams therefore share frame indices and are constructed to follow corresponding pose, expression, and motion, while differing primarily in transferred facial appearance. VFR in “DeepVFR” stands for video face retouching; the name refers to the benchmark task rather than to a new model.

The dataset contains 1,024 paired clips. Each clip has 150 frames at 30 fps, corresponding to approximately 5 seconds. This gives 153,600 paired frame correspondences, or 307,200 individual frames across both streams. Frames within a clip are temporally correlated, however, so 153,600 must not be interpreted as the number of independent examples; the benchmark has 1,024 clip-level source pairs and motion trajectories. The planned release manifest will include paired-clip identifiers, split membership, frame indices, demographic tags, the transfer-route label, and quality-control status. Once released with a version and integrity record, this provenance would make scoring on the fixed package repeatable even when a third-party construction backend cannot be regenerated bit-for-bit.

Splits and Statistics

DeepVFR adopts an identity-disjoint evaluation protocol to reduce identity leakage between training and testing. The split assigns 885 clips to training and 139 clips to testing, for 1,024 clips in total, and within-benchmark experiments follow this protocol. The external-checkpoint diagnostic in § 5.3 is the stated exception: because no model is fitted on DeepVFR, it scores the complete fixed collection. Identity-disjoint testing reduces memorization but does not by itself measure synthetic-to-real generalization.

The dataset is organized with appearance-based stratification tags for age group and perceived gender presentation, including child, youth, and elderly groups crossed with the retained male/female labels. These tags are provided for analysis rather than as prediction targets or claims about a person’s self-identified gender. They allow future work to report not only overall performance but also subgroup behavior across different facial appearances. At the same time, the current distribution is not perfectly balanced. In particular, young-child samples are relatively fewer because current image-to-video generation and face-transfer pipelines are less stable on very young faces. This imbalance should be considered when interpreting subgroup results.

Benchmark and Boundaries

DeepVFR is designed to support both image-based and video-based face retouching research. Image-based methods can use the dataset as a set of paired raw/retouched frames, while video-based methods can use temporally ordered clips to model cross-frame consistency. The benchmark therefore provides a common supervision source for comparing different retouching paradigms under the same raw/retouched target definition.

Under the identity-disjoint split, each model takes the raw stream as input and is evaluated against the synchronized proxy target. This supports paired fidelity, adjacent-frame stability, qualitative inspection, and human preference studies under a common target definition. DeepVFR is not a substitute for artist-retouched captured video: both streams inherit synthetic-image and I2V characteristics, while the target additionally inherits face-transfer and blending artifacts. The benchmark therefore does not cover uncontrolled camera motion, lighting changes, sensor noise, broad pose and occlusion patterns, or the full range of subjective editing preferences.

Experimental Results and Analyses

We report baseline experiments that characterize DeepVFR within its controlled scope. We first describe the datasets, methods, implementation, and metrics; then report within-benchmark results, external-checkpoint transfer into DeepVFR, a human ranking study, and qualitative cases. These experiments are diagnostic rather than evidence of real-video deployment performance.

Experimental Setup

Datasets Description

The primary benchmark used in this section is DeepVFR. As introduced in § 4, it contains 1,024 generated-raw/constructed-target face-video pairs, with 150 frames per clip at 30 fps. Unless otherwise specified, experiments use the identity-disjoint protocol.

For the external-to-DeepVFR diagnostic, we directly apply released weights from prior work without retraining. MRFV is unavailable and FFHQR-Seq contains geometric pseudo-sequences, so a fairness-controlled external training setup cannot be reconstructed from the public materials. Because no weights are fitted on DeepVFR in this diagnostic, we evaluate all 1,024 fixed clips. This direction is external supervision →\to DeepVFR; it neither tests DeepVFR-trained models on captured video nor resolves the synthetic-to-real gap.

Comparison Methods

We compare methods spanning face restoration, image face retouching, and video face retouching. RestoreFormer and GPEN are restoration-oriented baselines [34, 49]; RetouchFormer and RetouchGPT are image-retouching methods [35, 43]; and VRetouchEr is a video-retouching method [45]. For within-benchmark experiments, the architectures are trained on the same DeepVFR paired source when applicable; the VRetouchEr setting retains its cross-frame and imperfection-flow design.

We separately test released reference weights trained on external image pairs, instructions, or pseudo-sequences. Architecture, pretraining, checkpoint, and supervision all differ, so this diagnostic is not a strict ranking and cannot isolate a causal benefit of any training dataset.

Implementation and Parameter Details

The experiments use the authors’ released implementations and configurations where available. Within-benchmark models use raw DeepVFR frames or clips as input and the synchronized constructed targets as supervision, and are evaluated on the identity-disjoint test split. The external-checkpoint diagnostic instead follows the full-collection protocol described above. This separation prevents the external transfer table from being conflated with the official train/test benchmark protocol.

For the subjective study, 17 participants rank four method-anonymized results on 100 samples from the external-checkpoint evaluation. The Wenjuanxing configuration randomized both trial order and the mapping of the four anonymous candidates to the 2×22 \times2 grid positions; we do not claim exact counterbalancing. The supplement gives the remaining instructions and timing.

Performance Metrics

PSNR, SSIM, and LPIPS [52] measure frame-level agreement with the constructed target. Higher PSNR/SSIM and lower LPIPS indicate closer pixel, structure, and feature agreement. For the output-only temporal statistics, let Y^i,t\hat{Y}_{i,t} be prediction frame tt, M=∑i(Ti−1)M = \sum_i (T_i - 1), and μc(⋅)\mu_c(\cdot) the spatial mean of RGB channel cc:

LPIPS-T⁡=1M∑i∑t=1Ti−1dLPIPS⁡(Y^i,t,Y^i,t+1),RGBFlicker⁡=13M∑i∑t=1Ti−1∑c∈{R,G,B}∣μc(Y^i,t+1)−μc(Y^i,t)∣.(5)\operatorname{LPIPS\text{-}T} = \frac{1}{M} \sum_i \sum_{t=1}^{T_i-1} d_{\operatorname{LPIPS}}(\hat{Y}_{i,t}, \hat{Y}_{i,t+1}), \qquad \operatorname{RGBFlicker} = \frac{1}{3M} \sum_i \sum_{t=1}^{T_i-1} \sum_{c \in\{R,G,B\}} \left|\mu_c(\hat{Y}_{i,t+1}) - \mu_c(\hat{Y}_{i,t})\right|. \tag*{(5)}

Predictions are decoded as RGB, area-resized to 256×256256 \times256, and scaled to [0,1][0,1] (LPIPS input: [−1,1][-1,1]). LPIPS uses lpips 0.1.4, AlexNet, version 0.1. We pool all M=20,711M = 20{,}711 pairs (139 clips, 149 pairs each); neither metric uses a reference, mask, or flow. They therefore conflate genuine motion with flicker. A frozen output has zero adjacent-frame change under both definitions; RGBFlicker can also miss local changes that preserve channel means. These statistics do not establish motion preservation or perceptual temporal quality.

Performance Evaluation on Individual Datasets

In this subsection, we evaluate the face video retouching performance of the comparison methods on the DeepVFR benchmark, where the training and testing sets are drawn from the same dataset under the identity-disjoint protocol. The experimental results are reported in Table 3, and the following observations can be made.

ModelPSNR↑SSIM↑LPIPS↓LPIPS-TRGBFlicker
RestoreFormer24.03680.70230.23830.0108970.001062
GPEN23.97880.72990.22830.0108960.000828
RetouchFormer27.90780.85290.09370.0056480.000810
RetouchGPT33.74260.95050.03350.0054350.000816
VRetouchEr33.18040.95210.03810.0035070.000642

Table 3. Results on the official identity-disjoint DeepVFR test split. Bold values mark the best target-referenced fidelity scores. Temporal columns describe unaligned adjacent-frame change; smaller values do not establish better temporal quality.

  • RetouchFormer, VRetouchEr, and RetouchGPT obtain PSNR values from 27.9078 to 33.7426 and SSIM values from 0.8529 to 0.9521, while LPIPS decreases from 0.0937 to 0.0335. The protocol therefore reports distinct metric profiles under common paired supervision. These values do not by themselves establish that the constructed targets are artifact-free or representative of artist retouching.

  • RetouchGPT has the highest PSNR and lowest LPIPS, whereas VRetouchEr has the highest SSIM and smallest observed adjacent-frame changes. These results describe target agreement and output variation, rather than an established fidelity–temporal-quality trade-off. Whole-frame fidelity includes the copied background, and neither identity preservation nor the contribution of facial retouching is isolated by these scores.

  • RestoreFormer and GPEN score below the dedicated retouching methods on target-referenced metrics. This observation is consistent with their restoration objectives being different from the benchmark’s raw-to-constructed-target mapping. It should not be read as a general ranking of restoration and retouching systems outside DeepVFR.

External-to-DeepVFR Transfer Evaluation

We next apply released external checkpoints to the complete DeepVFR collection without retraining. This is a compatibility diagnostic under mismatched supervision, not an evaluation of a DeepVFR-trained model on real video. The original method papers describe the source supervision summarized in Table 4; the models nevertheless differ in architecture, pretraining, and checkpoint selection.

ModelSource supervisionPSNR↑\uparrowSSIM↑\uparrowLPIPS↓\downarrow
RetouchFormerFFHQR image pairs26.54130.80870.1405
RetouchGPTFFHQR + instructions26.65540.81200.1374
VRetouchErFFHQR-Seq pseudo-sequences35.82510.94690.0451

Table 4. External checkpoints applied to all 1,024 DeepVFR clips (153,600 frames). “Source supervision” follows the corresponding method paper; the comparison is not fairness-controlled.

  • All three external checkpoints can be scored against the DeepVFR constructed targets, but their values reflect compatibility under supervision mismatch. They do not show that DeepVFR captures real-camera statistics or that training on it transfers to real videos.

  • The released VRetouchEr checkpoint has the best values within this external-only table, but architecture, pretraining, and supervision are confounded. Its advantage therefore cannot be attributed specifically to temporal modeling or to FFHQR-Seq.

  • Table 3 uses the identity-disjoint test split, whereas this diagnostic uses all 1,024 clips. Values must not be compared numerically across the two tables. A controlled supervision study would evaluate every checkpoint on the same split and hold architecture and preprocessing fixed.

Subjective Experimental Analysis

Seventeen participants rank four method-anonymized candidates on 100 clips from the external-checkpoint diagnostic according to visual quality, imperfection removal, naturalness, and temporal consistency. Table 5 reports the aggregated ranks; the supplementary material documents the retained protocol.

CandidateRank-1 (%)Rank-2 (%)Rank-3 (%)Rank-4 (%)
DeepVFR target57.8213.947.1821.06
VRetouchEr22.3535.5931.5310.53
RetouchGPT11.5933.8842.2412.29
RetouchFormer8.2416.5919.0656.12

Table 5. Human subjective ranking results (%) on 100 DeepVFR study clips. Higher Rank-1 indicates stronger preference within this stimulus set.

The target receives 57.82% Rank-1 but also 21.06% Rank-4, approximately its combined Rank-2/3 share, revealing a substantial low-preference tail. VRetouchEr most often ranks second, whereas RetouchFormer has the largest Rank-4 share, but architecture and external supervision are confounded. Both trial order and anonymous candidate positions were randomized. The percentages nevertheless remain descriptive of this 17-participant stimulus set rather than a population estimate. The 21.06% Rank-4 share is a fraction of judgments, not an estimated fraction of defective clips; clip-specific artifacts and participant preferences cannot be separated from these aggregates.

Qualitative Analyses

The supplementary comparisons expose local mismatch and boundary artifacts hidden by averages. Selected frames do not establish temporal consistency, identity preservation, or captured-video performance, and are not used for cross-table conclusions.

Discussion

Scope, artifacts, and scalability. DeepVFR contains 1,024 correlated, mild-motion trajectories, not a large-scale substitute for diverse captured video. Rapid speech, large pose changes, occlusion, dynamic lighting, sensor noise, camera motion, and some age groups remain underrepresented. Automatic checks and manual screening remove conspicuous failures but cannot eliminate cascaded DeepFRR/Wan2.2/transfer artifacts; manual review is the scaling bottleneck. Akool is closed and mutable, limiting exact regeneration, although a released fixed package could remain scoreable. The current work reports neither backend-stratified sensitivity nor an artist-retouched captured-video testbed.

Evaluation boundary. PSNR, SSIM, and LPIPS measure agreement with a constructed reference, not identity or aesthetic correctness. LPIPS-T and RGBFlicker measure raw adjacent-frame change and do not separate motion from flicker. External checkpoints are tested only on DeepVFR, and the randomized 17-participant study remains descriptive because of its scale and stimulus scope. Stronger validation should add ArcFace identity similarity [7], motion-compensated error [16, 28], video quality [53] (Wu et al. 2023), a real artist-retouched testbed, and a larger human study with an explicit balance audit.

Conclusion

DeepVFR is an initial controlled synthetic benchmark—not a new architecture—with 1,024 synchronized clip pairs, an identity-disjoint protocol, and planned construction provenance. Its baselines describe target agreement, adjacent-frame output variation, external-checkpoint compatibility, and heterogeneous human preference within this domain. Synthetic and third-party components prevent claims of real-video generalization; DeepVFR is a starting point for broader, verifiable benchmarks.

CRediT authorship contribution statement

Peiyuan Zeng: Investigation, Writing – original draft. Fengchuan Xing: Supervision, Writing – review and editing. Shiming Wu: Investigation, Writing – original draft. Hongwei Yue: Supervision, Writing – review and editing. Chao Zhou: Supervision, Writing – review and editing.

Declaration of competing interest

Peiyuan Zeng is affiliated with Cloudbutterfly Technology.

Funding

This work was supported by the Guangdong Provincial Key Discipline Research Capacity Enhancement Project [grant number 2025ZDJS139]; and the 2026 Provincial-Level College Student Innovation and Entrepreneurship Training Program [grant number S202614278026].

Data availability

The DeepVFR paired videos, individual study rankings, and training logs will be made publicly available upon acceptance of the manuscript. The release will include the split manifest and construction metadata described in this article. Repository details and access information will be added when the materials are released.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

OpenAI Codex was used to assist with language editing and LaTeX formatting. The authors retain full responsibility for the final manuscript.

References

  1. [1]Akool, 2026. Akool face swap api overview. Official API documentation. URL: https://docs.akool.com/ai-tools-suite/faceswap. accessed: 2026-05-04.
  2. [2]Alibaba Cloud, 2026. Wan image-to-video api reference. Official product documentation. URL: https://www.alibabacloud.com/help/en/model-studio/image-to-video-api-reference/. accessed: 2026-05-04.
  3. [3]Bae, G., de La Gorce, M., Baltrušaitis, T., et al., 2023. DigiFace-1M: 1 million digital face images for face recognition, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3526–3535.arxiv.org/abs/2210.02579
  4. [4]Bychkovsky, V., Paris, S., Chan, E., et al., 2011. Learning photographic global tonal adjustment with a database of input / output image pairs, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 97–104.
  5. [5]Chen, R., Chen, X., Ni, B., et al., 2020. SimSwap: An efficient framework for high fidelity face swapping, in: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2003–2011.DOI
  6. [6]Deng, J., Guo, J., Ververas, E., et al., 2020. RetinaFace: Single-shot multi-level face localisation in the wild, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5203–5212.DOI
  7. [7]Deng, J., Guo, J., Xue, N., Zafeiriou, S., 2019. ArcFace: Additive angular margin loss for deep face recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4690–4699.DOI
  8. [8]Garrido, P., Valgaerts, L., Rehmsen, O., Thormählen, T., Perez, P., Theobalt, C., 2014. Automatic face reenactment, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4217–4224. URL: https://openaccess.thecvf.com/content_cvpr_2014/html/Garrido_Automatic_Face_Reenactment_2014_CVPR_paper.html.DOI
  9. [9]Gebru, T., Morgenstern, J., Vecchione, B., et al., 2021. Datasheets for datasets. Communications of the ACM 64, 86–92.arxiv.org/abs/1803.09010
  10. [10]Guo, J., Deng, J., Lattas, A., et al., 2022. Sample and computation redistribution for efficient face detection, in: The Tenth International Conference on Learning Representations.arxiv.org/abs/2105.04714
  11. [11]Karras, T., Laine, S., Aila, T., 2019. A style-based generator architecture for generative adversarial networks, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401–4410.arxiv.org/abs/1812.04948
  12. [12]Karras, T., Laine, S., Aittala, M., et al., 2020. Analyzing and improving the image quality of stylegan, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8107–8116.arxiv.org/abs/1912.04958
  13. [13]Kemp, S., 2025. Digital 2025: Global overview report. DataReportal, We Are Social, and Meltwater.
  14. [14]Khalid, H., Tariq, S., Kim, M., Woo, S.S., 2021. FakeAVCeleb: A novel audio-video multimodal deepfake dataset, in: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. URL: https://openreview.net/forum?id=TAxrSfg62a01.arxiv.org/abs/2108.05080
  15. [15]Kim, H., Garrido, P., Tewari, A., et al., 2018. Deep video portraits. ACM Transactions on Graphics 37, 1–14.arxiv.org/abs/1805.11714
  16. [16]Lai, W.S., Huang, J.B., Wang, O., et al., 2018. Learning blind video temporal consistency, in: Proceedings of the European Conference on Computer Vision, pp. 170–185.arxiv.org/abs/1808.00449
  17. [19]Liang, J., Zeng, H., Cui, M., et al., 2021. PPR10K: A large-scale portrait photo retouching dataset with human-region mask and group-level consistency, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 653–661.arxiv.org/abs/2105.09180
  18. [20]Liang, L., Lin, L., Jin, L., et al., 2018. SCUT-FBP5500: A diverse benchmark dataset for multi-paradigm facial beauty prediction, in: Proceedings of the International Conference on Pattern Recognition, pp. 1598–1603.arxiv.org/abs/1801.06345
  19. [21]Lin, T.Y., Tsai, Y.T., Huang, T.S., et al., 2019. Exemplar-based freckle retouching and skin tone adjustment. Computers & Graphics 78, 54–63.DOI
  20. [22]Liu, J., Ying, Q., Qian, Z., Li, S., Zhang, R., Liu, J., Zhang, X., 2025. MoFRR: Mixture of diffusion models for face retouching restoration, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12842–12851. URL: https://openaccess.thecvf.com/content/ICCV2025/html/Liu_MoFRR_Mixture_of_Diffusion_Models_for_Face_Retouching_Restoration_ICCV_2025_paper.html.arxiv.org/abs/2507.19770
  21. [23]Nirkin, Y., Keller, Y., Hassner, T., 2019. FSGAN: Subject agnostic face swapping and reenactment, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7184–7193.arxiv.org/abs/1908.05932
  22. [24]Rosberg, F., Aksoy, E.E., Alonso-Fernandez, F., et al., 2023. FaceDancer: Pose- and occlusion-aware high fidelity face swapping, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3454–3463.arxiv.org/abs/2210.10473
  23. [25]Rössler, A., Cozzolino, D., Verdoliva, L., et al., 2019. FaceForensics++: Learning to detect manipulated facial images, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1–11.arxiv.org/abs/1901.08971
  24. [26]Shafaei, A., Little, J.J., Schmidt, M., 2021. AutoRetouch: Automatic professional face retouching, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 990–998.DOI
  25. [27]Shiohara, K., Yang, X., Taketomi, T., 2023. BlendFace: Re-designing identity encoders for face-swapping, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7634–7644.arxiv.org/abs/2307.10854
  26. [28]Teed, Z., Deng, J., 2020. RAFT: Recurrent all-pairs field transforms for optical flow, in: Proceedings of the European Conference on Computer Vision, pp. 402–419.arxiv.org/abs/2003.12039
  27. [29]Thies, J., Zollhöfer, M., Nießner, M., 2019. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics 38, 1–12.arxiv.org/abs/1904.12356
  28. [30]Thies, J., Zollhöfer, M., Stamminger, M., et al., 2016. Face2Face: Real-time face capture and reenactment of rgb videos, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387–2395.arxiv.org/abs/2007.14808
  29. [31]Wan Team, 2025. Wan2.2: Open and advanced large-scale video generative models. Official GitHub repository. URL: https://github.com/Wan-Video/Wan2.2. accessed: 2026-05-04.
  30. [32]Wan Team, Wang, A., Ai, B., et al., 2025. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 arXiv:2503.20314.arxiv.org/abs/2503.20314
  31. [33]Wang, Y., Chen, X., Zhu, J., et al., 2021. HifiFace: 3d shape and semantic prior guided high fidelity face swapping, in: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pp. 1136–1142.arxiv.org/abs/2106.09965
  32. [34]Wang, Z., Zhang, J., Chen, R., Wang, W., Luo, P., 2022. RestoreFormer: High-quality blind face restoration from undegraded key-value pairs, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17512–17521. URL: https://openaccess.thecvf.com/content/CVPR2022/html/Wang_RestoreFormer_High-Quality_Blind_Face_Restoration_From_Undegraded_Key-Value_Pairs_CVPR_2022_paper.html.arxiv.org/abs/2201.06374
  33. [35]Wen, X., Xie, L., Jiang, L., Chen, T., Wu, S., Liu, C., Wong, H.S., 2024. RetouchFormer: Semi-supervised high-quality face retouching transform with prior-based selective self-attention. Proceedings of the AAAI Conference on Artificial Intelligence 38, 5903–5911. URL: https://ojs.aaai.org/index.php/AAAI/article/view/28404, doi:10.1609/aaai.v38i6.28404.
  34. [36]Wiles, O., Koepke, A.S., Zisserman, A., 2018. X2Face: A network for controlling face generation using images, audio, and pose codes, in: Proceedings of the European Conference on Computer Vision, pp. 670–686.arxiv.org/abs/1807.10550
  35. [39]Xie, D., Liang, L., Jin, L., et al., 2015. SCUT-FBP: A benchmark dataset for facial beauty perception, in: IEEE International Conference on Systems, Man, and Cybernetics, Hong Kong, China. pp. 1821–1826.arxiv.org/abs/1511.02459
  36. [40]Xing, F., Shi, X., Wang, Y.G., et al., 2025. Unveiling authenticity with diffusion-based face retouching reversal. Applied Soft Computing 174, 113062.arxiv.org/abs/2405.07582
  37. [43]Xue, W., Ding, C., Xu, R., Wu, S., Xu, Y., Wong, H.S., 2025. RetouchGPT: LLM-based interactive high-fidelity face retouching via imperfection prompting. Proceedings of the AAAI Conference on Artificial Intelligence 39, 9059–9067. URL: https://ojs.aaai.org/index.php/AAAI/article/view/32980, doi:10.1609/aaai.v39i9.32980.
  38. [44]Xue, W., Jiang, L., Xie, L., et al., 2024a. Official implementation of VRetoucher. GitHub repository. URL: https://github.com/Davidcoach/VRetoucher_CVPR_2024. accessed: 2026-05-04.
  39. [45]Xue, W., Jiang, L., Xie, L., et al., 2024b. VRetoucher: Learning cross-frame feature interdependence with imperfection flow for face retouching in videos, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9141–9150.DOI
  40. [46]Yan, Z., Yao, T., Chen, S., Zhao, Y., Fu, X., Zhu, J., Luo, D., Wang, C., Ding, S., Wu, Y., Yuan, L., 2024. DF40: Toward next-generation deepfake detection, in: Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 29387–29434. URL: https://proceedings.neurips.cc/paper_files/paper/2024/hash/34239f60eca7ce9bee5280aaf81362d8-Abstract-Datasets_and_Benchmarks_Track.html, doi:10.52202/079017-0925.DOI
  41. [47]Yang, J., Lin, X., Dong, Y., Zheng, Z., Liu, X., Gu, H., Fang, Y., 2026. BeautyGRPO: Aesthetic alignment for face retouching via dynamic path guidance and fine-grained preference modeling, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25110–25120. URL: https://openaccess.thecvf.com/content/CVPR2026/papers/Yang_BeautyGRPO_Aesthetic_Alignment_for_Face_Retouching_via_Dynamic_Path_Guidance_CVPR_2026_paper.pdf.arxiv.org/abs/2603.01163
  42. [48]Yang, S., Jiang, L., Liu, Z., et al., 2022. VToonify: Controllable high-resolution portrait video style transfer. ACM Transactions on Graphics 41, 1–15.arxiv.org/abs/2209.11224
  43. [49]Yang, T., Ren, P., Xie, X., Zhang, L., 2021. GAN prior embedded network for blind face restoration in the wild, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 672–681. URL: https://openaccess.thecvf.com/content/CVPR2021/html/Yang_GAN_Prior_Embedded_Network_for_Blind_Face_Restoration_in_the_Wild_CVPR_2021_paper.html.arxiv.org/abs/2105.06070
  44. [50]Ying, Q., Liu, J., Li, S., et al., 2023. RetouchingFFHQ: A large-scale dataset for fine-grained face retouching detection, in: Proceedings of the 31st ACM International Conference on Multimedia, pp. 737–746.arxiv.org/abs/2307.10642
  45. [52]Zhang, R., Isola, P., Efros, A.A., et al., 2018. The unreasonable effectiveness of deep features as a perceptual metric, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595.arxiv.org/abs/1801.03924
  46. [53]Zhou, Y., Zhang, Z., Sun, W., Liu, X., Min, X., Wang, Z., Zhang, X.P., Zhai, G., 2024. THQA: A perceptual quality assessment database for talking heads, in: 2024 IEEE International Conference on Image Processing, pp. 15–21. URL: https://cmuworkshops.com/ICIP2024/view_paper.php?PaperNum=1437, doi:10.1109/ICIP51287.2024.10647507.DOI
  47. [54]Zhu, H., Wu, W., Zhu, W., et al., 2022. CelebV-HQ: A large-scale video facial attributes dataset, in: Proceedings of the European Conference on Computer Vision, pp. 650–667.arxiv.org/abs/2207.12393
  48. [55]Zhu, Y., Li, Q., Wang, J., et al., 2021. One shot face swapping on megapixels, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4834–4844.arxiv.org/abs/2105.04932

Paper details

vlnc:W_tw2kjg86
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex

View statement
Looking for related papers…
Contents