WorksComputer Science
A Threshold-Free Visual-Semantic Matching Metric for Evaluating Consistency in Subject-Driven Image Generation
Abstract
We propose a Threshold-Free Visual-Sematic Matching (TF-VSM) metric, to quantitatively evaluate localized visual consistency across varying poses and environments in subject-driven text-to-image generation. TF-VSM is a fully continuous, threshold-free visual consistency metric paired with an extended feature-disentanglement retraining schedule for dual-branch ResNet feature-aggregation networks. Existing metrics suffer from point-alignment dropouts, threshold sensitivity, and score instability. To address them, our TF-VSM, rather than enforcing rigid binary cutoffs (, ), weights every object point by a non-linear semantic match confidence to yield a closed-form confidence-weighted score without any hand-tuned hyperparameters. Experimental results show that, TF-VSM attains Pearson 0.638 and Spearman 0.674 oracle correlation, improving over the pretrained VSM baseline (0.454/0.589) by +0.171/ + 0.116 after retraining, and drastically outperforming CLIP (-0.052/ -0.004), DINO (0.087/0.121), and VLMs (0.096/0.003). Six diagnostic ablations confirm that smooth confidence weighting is the right mechanism, that hard semantic gating is brittle (dropping to 0.533 Pearson at =0.8), and that the threshold-free metric is essentially invariant to the training time loss balance . Code is available at: https://github.com/bwshan/TF-VSM.
Introduction
Subject-driven text-to-image generation has advanced rapidly, enabling users to personalize diffusion models to synthesize specific target subjects across diverse poses, lighting, and background environments [44, 17, 28, 56, 19, 29, 50, 63]. A central requirement in evaluating these models is assessing visual consistency—determining whether the visual identity and subtle textural details of a subject are faithfully preserved across reference and generated image pairs [52, 4]. An ideal metric must remain invariant to variations in pose, background, scale, and perspective while remaining sensitive to fine-grained visual inconsistencies such as altered surface patterns, modified logos, or subtle structural artifacts. Despite its importance, quantifying visual consistency remains an open challenge. Prior evaluation protocols predominantly rely on two paradigms:
Global feature similarity & vision-language models. Standard frameworks compute distances between global image embeddings such as CLIP-I [44, 17] or DINO-I [7, 39]. Because these representations pool features over spatial dimensions, they conflate high-level identity with layout and are blind to localized inconsistencies [1, 9]. Likewise, VLMs (e.g., GPT-4o, Gemini) and VLM-driven suites [26, 40, 22] offer impressionistic judgments that struggle to track fine attributes and aggregate local visual evidence across images [30, 12].
Threshold-based point alignment & feature disentanglement. To overcome spatial pooling, recent approaches such as Visual-Semantic Matching (VSM) [14] extract internal features from pre-trained diffusion backbones [43, 48] and train dual aggregation networks to disentangle semantic structure from visual appearance. Standard VSM, however, computes consistency by applying hand-tuned binary thresholds () to filter semantic keypoints and to binarize visual similarities. As decision-boundary studies show [2, 35], hard thresholding creates step-discontinuities that cause severe point-alignment dropouts and score instability whenever valid correspondences fall slightly below arbitrary cutoffs.
To address these limitations, we introduce TF-VSM (Threshold-Free Visual-Semantic Matching), a continuous, fully threshold-free visual semantic consistency metric. Rather than gating correspondences with a rigid cutoff , TF-VSM assigns every object point a smooth semantic-match weight , so each point contributes according to its alignment confidence. It further eliminates the visual threshold by integrating the visual indicator over all consistency levels , which reduces in closed form to a confidence-weighted soft visual similarity with no hand-tuned cutoffs. In tandem, we retrain the dual feature-aggregation networks on the Subjects200k-derived dataset [50, 14] with an extended schedule, identifying an optimal convergence checkpoint (epoch 8) that markedly improves feature disentanglement. Through controlled evaluation on a 100-pair test set with ground-truth inpainting oracles, TF-VSM achieves Pearson 0.638 and Spearman 0.674 correlation with the oracle, markedly outperforming global CLIP (), DINO (0.087/0.121), Gemini VLM (0.096/0.003), and the original pretrained VSM (0.454/0.589). Our contributions are:
A fully threshold-free visual-consistency metric that removes both and via non-linear confidence weighting and continuous visual integration, yielding a simple closed form.
An extended disentanglement retraining recipe that lifts oracle correlation from 0.467/0.558 to 0.638/0.674 (+0.171/ +0.116) over the released model.
Six diagnostic ablations isolating the confidence weighting, the semantic/visual readout transforms, the schedule length, the data scale, and the loss balance , showing TF-VSM matches oracle-tuned hard gating while removing its brittle hyperparameter dependence.
A leakage-free evaluation protocol (disjoint selection/test splits) that independently recovers the epoch-8 checkpoint and confirms the gain on a strictly held-out split, together with foreground-masked baselines showing that background removal explains almost none of the gap to global metrics.
Related Work
We contextualize our work across three paradigms: global feature similarity and VLM evaluation, dense semantic correspondence via diffusion features, and threshold-based consistency metrics.
Global feature similarity & vision-language evaluation. Subject-driven personalization models [44, 17, 28, 56, 19, 29, 33] synthesize user-specified subjects across diverse contexts, and their evaluation requires standardized benchmarks [27, 23, 47]. Fidelity is historically quantified via global embedding distances using CLIP-I [44] or self-supervised backbones such as DINO/DINOv2 [7, 39, 58, 54]. However, global pooling makes these metrics insensitive to localized textural discrepancies [1, 15, 37], motivating localized perceptual variants such as matched-points LPIPS [8], human-aligned customized LPIPS [53], and foreground-masked patch matching [5]. In parallel, VLM-driven protocols such as DreamBench++ [40], VIEScore [26], VQAScore [30], MMIG-Bench [22], and FineMatch [21] assess image–text alignment, yet exhibit spatial-reasoning bottlenecks [9], cross-image aggregation failures [12], and reliance on text-mediated pathways [45, 38]. Recent suites such as USO-Bench [57] and RefVNLI [47] evaluate subject consistency but do not provide dense point-wise disentanglement of appearance from structure.
Dense semantic correspondence via diffusion features. Pretrained diffusion backbones [43, 41] implicitly learn rich spatial-semantic representations. DIFT [51] and Plug-and-Play [55] showed that internal UNet features enable zero-shot dense keypoint matching across challenging poses. Subsequent work expanded these capabilities via SD–DINO fusion [60], Diffusion Hyperfeatures [32], geometry-aware modeling [61], spherical priors [34], lightweight ViT feature transforms [49], prompt-attention optimization [20], and noise-free extraction via CleanDIFT [48], with distillation [16] and conditional matching [36] improving efficiency. Surveys [62, 10, 13, 11, 24] note that self-supervised ViT descriptors [3] and diffusion features achieve strong semantic alignment but intentionally discard appearance variation. Consequently, correspondence-driven consistency methods—ConsiStory [52], Eye-for-an-eye [18], ReStyle3D [64], The Chosen One [4], Self-Cross Guidance [42], and Grounding DINO [31]—focus on driving synthesis rather than explicitly evaluating visual consistency.
Threshold-based consistency metrics & subject-driven benchmarks. Automated generation and evaluation workflows [6, 46, 63] often leverage synthetic dataset pipelines. Pairing identity-consistent datasets such as Subjects200k [50] with SAM [25] and SDXL inpainting [41] yields controlled pairs with localized variations [14]. However, perceptual metrics such as LPIPS [59] and the original VSM [14] rely on hand-tuned scalar thresholds ; decision-boundary studies [2, 35] show that single-threshold binarization introduces selection bias where minor cutoff shifts drastically alter rankings. TF-VSM removes these artifacts by replacing binary cutoffs with continuous, confidence-weighted integration.
Methodology
We present the technical framework of TF-VSM: an automated data-generation pipeline producing paired data with known localized inconsistencies (section 3.1); a dual-ResNet aggregation architecture with a contrastive disentanglement objective (section 3.2); and the continuous, threshold-free TF-VSM metric (section 3.3). Figure 1 gives an overview.

Figure 1. Overview of the proposed framework. Given an input pair with synthetic inpainted regions , multi-layer features are extracted from a frozen diffusion backbone and processed by dual-branch ResNet aggregation networks into semantic features and visual features . A contrastive disentanglement module enforces pose-invariant semantic matching over all correspondences while penalizing visual similarity inside inconsistent regions. The threshold-free TF-VSM unit then reads out a consistency score by soft-weighting visual similarities with squared semantic confidence , avoiding brittle binary cutoffs and producing a global score alongside a diagnostic heatmap.
Data Generation Pipeline for Inconsistency Supervision
To learn to separate semantic structure from visual appearance, we construct a supervision dataset of paired images with precise spatial masks marking consistent and inconsistent regions, starting from identity-consistent pairs in Subjects200k [50].
We first isolate the foreground subject in both images with Grounded-SAM [25, 31], yielding binary subject masks . To establish semantic correspondences, we extract features from the 6th UNet decoder layer of Stable Diffusion 2.1 [43, 48], and , and form the dense similarity matrix with . Dense correspondences are recovered by spatial argmax:
To create localized inconsistencies, we select a high-confidence match and prompt SAM to segment corresponding sub-regions , returning multiple hierarchical candidates and picking the smallest-footprint mask to avoid segmenting the whole object.
Ambiguity filtering via similarity skewness. On flat, textureless regions (plain surfaces, uniform clothing) point matches diffusely. We measure the skewness of ,
with , , ; high skewness indicates sharp localized matches, low skewness ambiguous ones. We keep samples with .
Inpainting and perceptual validation. For regions passing structural checks we crop a padded patch and perform localized SDXL inpainting [41], so that new textures are synthesized without global object context. We then validate perceptual change via LPIPS [59], discarding pairs with LPIPS . The resulting dataset contains 4,912 usable training pairs with subject masks and inconsistency masks .
Disentangled Feature Aggregation and Retraining Objective
Given an inconsistent pair with masks , we learn descriptors separating pose-invariant semantic structure from visual appearance. Multi-layer feature maps across decoder layers are passed through two aggregation networks (semantic) and (visual), each a per-layer mini-ResNet [14] combined via trainable scalar weights :
with projected channel dimension and . We form spatial correlation matrices , , , , and partition correspondences into interior (, inside ) and exterior () subsets:
Semantic features are pulled together across all matches via a cross-entropy loss over , while visual features are pulled together outside and pushed apart inside inconsistent regions:
Combining terms symmetrically over both directions gives , where prioritizes visual learning. We optimize with AdamW () over an extended 14-epoch schedule and select the peak checkpoint by oracle correlation.
Fully Threshold-Free TF-VSM Formulation
Given disentangled features , we extract per-point best-match similarities for all foreground points : and . Standard VSM [14] applies hard thresholds:
which introduces step-discontinuities: points with just below are discarded, and visual scores just below drop abruptly to zero. We resolve both. First, instead of gating with , every point receives a smooth non-linear confidence weight
continuously suppressing unaligned clutter while keeping non-zero weight on valid regions. Second, instead of an arbitrary , we integrate the visual indicator over all levels ; with , the fundamental theorem of calculus gives . Combining yields the closed-form TF-VSM:
TF-VSM is fully continuous, parameter-free (no or ), smooth in the features, and robust to partial spatial misalignments.
Experiments
We evaluate TF-VSM on a controlled inconsistency benchmark (§ 4.2), realistic subject-driven generators (§ 4.4), and six diagnostic ablations (§ 4.6).
Experimental Setup and Baselines
Training. We use the 4,912 pairs from § 3.1. Features are extracted from a frozen Stable Diffusion 2.1 backbone [43, 48] at resolution () and projected to ; . Models train with AdamW () for 14 epochs on a single A100 (172.7 min total).
Ground-truth oracle. Following Eldesokey et al. [14], we evaluate on a verified 100-pair test set. For each pair we compute an exact oracle from the relative footprint of the inconsistent region within the subject mask :
An accurate metric must correlate strongly (Pearson , Spearman ) with this oracle.
Baselines. We compare against: (i) CLIP global cosine similarity (OpenCLIP ViT-B/32) [44]; (ii) DINOv2 global similarity (ViT-L/14) [39]; (iii) VLM judgment (Gemini 2.0 Flash prompted for a score ) [26, 40]; and (iv) standard VSM with scalar thresholds , [14], both pretrained and retrained.
Controlled Inconsistency Evaluation
Figure 2 and table 1 report oracle correlations on the 100-pair controlled test set. Global embedding metrics fail on localized inconsistency: CLIP is near-zero/negative (−0.052/ −0.004) and DINOv2 weak (0.087/0.121), because spatial pooling conflates scene context with subject detail. The Gemini VLM likewise shows minimal correlation (0.096/0.003) due to score saturation and poor fine-grained spatial tracking. Among recently proposed localized/personalized metrics (RefVNLI, MaSC, CLPIPS), only RefVNLI [47] is reproducible—MaSC and CLPIPS have released no public code or checkpoints—and its single-prompt VLM decision protocol still lands in the same weak band as the global baselines (), far below TF-VSM, confirming that localized consistency requires dense point-wise disentanglement rather than holistic judgment (section B). Our retrained TF-VSM attains , , improving by +0.171/ + 0.116 over the released pretrained model (0.467/0.558) and far exceeding all CLIP/DINO/VLM baselines. Hard threshold VSM tuned specifically to reaches 0.728/0.705 on this test set, but that tuning is only realizable with oracle access and is highly brittle (§ 4.6); TF-VSM provides competitive, stable correlation with zero tuning.
Figure 2. Oracle correlation of each metric on the 100-pair controlled test set. Global metrics (CLIP, DINOv2) and the VLM fail on localized inconsistency, whereas retrained TF-VSM (threshold-free, highlighted) reaches 0.638/0.674, a +0.171/ + 0.116 gain over its pretrained version. The hand-tuned VSM (, retrained) is higher on this set but—as section 4.6 shows—is brittle to its thresholds.
Why adopt TF-VSM over oracle-tuned hard VSM? Although hard VSM tuned to edges out TF-VSM on this benchmark, that peak is unattainable in practice: fixing requires ground truth inconsistency masks that do not exist for open-ended generation, and the optimum is fragile. TF-VSM instead (i) generalizes better on realistic generators with no tuning (§ 4.4), (ii) is nearly invariant to the loss balance and gate placement, whereas hard VSM swings by up to 0.10 Pearson (§ 4.6), and (iii) is fully differentiable, usable directly as a loss or RL reward—a role the step-discontinuous indicator VSM cannot fill.
Leakage-Free Model Selection on a Held-Out Split
The residual gap is an artifact of the area-ratio oracle, not miscalibration. The synthetic oracle measures the spatial extent of corruption, agnostic to graded severity, so a binary step readout is structurally matched to it. Keeping TF-VSM’s threshold-free weights but replacing the continuous clamp with a step at raises Pearson from 0.638 to 0.709, essentially closing the gap to tuned hard VSM (0.728)—so the difference is the oracle’s step-wise area construction, not feature miscalibration (power-family sharpening does not help; Ablation 3). Crucially, on realistic generators with human-annotated oracles that penalize graded severity, the continuous readout outperforms tuned hard VSM (§ 4.4), as does the human-preference evaluation below.
Because the main protocol selects the peak checkpoint (epoch 8) by oracle correlation, one might worry that this selection touches the same 100 pairs used for reporting. We therefore re-run selection under a strict, leakage-free protocol. The 4,912 supervision pairs are split 90/10 into 4,421 training and 491 validation pairs (used only to monitor concontrastive loss); the 100-pair benchmark is split into a 20-pair selection split (used exclusively to pick the epoch over epochs 1–14) and an 80-pair held-out test split (reported only). The selection split independently identifies epoch 8 as the peak, identical to the main paper without observing any test sample. Table 2 reports the held-out results: retrained TF-VSM attains , , a +0.140/ + 0.111 gain over the pretrained model and far above all CLIP/DINO/VLM baselines. The epoch-8 selection is thus not an artifact of test-set leakage, and all conclusions hold under strict split separation.
Table 2. Leakage-free evaluation on the held-out 80-pair test split (). The epoch is selected only on a disjoint 20-pair split; the 80 test pairs are never seen during selection. Retrained TF-VSM retains its large gain over the pretrained model and all global/VLM baselines.
Subject-Driven Image Generation Benchmarking
To probe generalization to realistic synthesis, we benchmark three subject-driven generators: Diptych Prompting [46] (FLUX-dev, ), DSD-Diffusion [6] (), and EasyControl [63] (). We generate 100 pairs per generator and manually annotate inconsistent sub-regions to form the oracle. Table 3 shows TF-VSM maintains robust correlation (, ) under real-world distributions, whereas the VLM saturates, scoring nearly
Table 3. Evaluation on realistic subject-driven generation (Diptych, DSD-Diffusion, EasyControl) against human-annotated identity-consistency oracles. VLM = Gemini.
all images in [?, ?] regardless of corruption.
Human-Preference Evaluation on DreamBench++
The controlled oracle measures corruption area; a natural question is whether TF-VSM tracks human judgments of subject fidelity. We therefore evaluate on DreamBench++ [40], the standard human-aligned benchmark for subject-driven generation, comprising 9,441 rated pairs (150 concepts 9 prompts 7 personalization methods) with independent human Concept-Preservation (CP) ratings (0–4). Following the official protocol, we correlate each metric with the mean human CP per method and average across methods; foreground masks for VSM/TF-VSM are obtained with the same Grounded-SAM pipeline. Table 4 and Figure 5a report the results. TF-VSM correlates with real human preference at , , a decisive +0.101/+0.092 gain over standard hard-thresholded VSM (0.238/0.258)—removing brittle cutoffs improves alignment with human judgment, not merely with the synthetic area oracle. As expected on a benchmark rating holistic identity, global embeddings and the VLM score highest overall; yet on the two hardest IP-Adapter generators they collapse (CLIP-I , DINOv2 ) while TF-VSM stays robust (, 0.284; section C). The two benchmarks are complementary: global metrics capture coarse identity but fail on the localized texture inconsistencies (; Table 1) TF-VSM targets.
| Evaluation metric | Pearson () | Spearman () |
| CLIP image similarity | −0.052 | −0.004 |
| DINOv2 image similarity | 0.087 | 0.121 |
| VLM judgment | 0.096 | 0.003 |
| RefVNLI protocol | 0.124 | 0.031 |
| Standard VSM (pretrained) | 0.454 | 0.589 |
| Standard VSM (retrained) | 0.728 | 0.705 |
| TF-VSM (pretrained) | 0.467 | 0.558 |
| TF-VSM (retrained) | 0.638 | 0.674 |
Table 1. Correlation with the ground-truth oracle on the 100-pair controlled test set. TF-VSM achieves strong correlation with no scalar thresholds (). ∗VLM = Gemini 2.0 Flash. †RefVNLI evaluated via its published subject-preservation decision protocol (section B); MaSC/CLPIPS have no public code or checkpoints.
| Evaluation metric | Pearson () | Spearman () | Pairwise agr. % | Pooled (=9.4k) |
| CLIP-I (OpenCLIP ViT-B/32) | 0.409 | 0.427 | 67.7 | 0.622 |
| DINOv2 (ViT-L/14) | 0.343 | 0.376 | 65.8 | 0.558 |
| VLM (Gemini 2.0/2.5 Flash) | 0.451 | 0.477 | 71.1 | 0.589 |
| Standard VSM (retrained, =0.7, =0.6) | 0.238 | 0.258 | 60.7 | 0.387 |
| TF-VSM (ours, retrained, threshold-free) | 0.339 | 0.350 | 64.3 | 0.517 |
Table 4. Human-preference evaluation on DreamBench++ [40] (9,441 pairs, mean-CP oracle; visualized in Figure 5a). Among dense point-wise metrics, threshold-free TF-VSM beats standard hard VSM by +0.101/+0.092; global metrics/VLM rate holistic identity (their native regime) but collapse on hard IP-Adapter generators (section C).
Diagnostic Ablation Studies
We perform six controlled ablations on the full 100-pair test set with the epoch-8 model unless noted: (1) semantic confidence weighting, (2) hard vs. soft semantic gating, (3) the visual-score transform, (4) schedule length, (5) data scaling, and (6) the loss balance .
Ablation 1: semantic confidence weighting . We compare uniform (), linear (), quadratic (, default), cubic (), and temperature Softmax (). As Figure 3a shows, any smooth non-linear weighting beats uniform (0.631/0.653) by down-weighting unaligned background, while peaky Softmax overconcentrates mass onto a few keypoints and degrades to 0.593/0.644. Within the smooth family the choice is near-flat ( spread), so quadratic is a robust, defensible default.
Figure 3. (a) Semantic confidence weighting (Ablation 1). (b) Hard gating vs. soft weighting (Ablation 2).
Ablation 2: soft weighting vs. hard semantic gating . We compare threshold-free TF-VSM against hard gates applied before the same soft visual integration (Figure 3b, Table 5). Raising from 0.6 to 0.8 collapses Pearson from 0.652 to 0.533 as 66% of valid points are discarded (403.7 149.5 per pair). TF-VSM matches the oracle-tuned best gate () to within 0.013/0.012 without any cutoff, sidestepping this brittleness.
| Semantic filtering | Pts/pair | ||
| Hard =0.5 + soft int. | 0.644 | 0.680 | 444.6 |
| Hard =0.6 + soft int. | 0.652 | 0.686 | 403.7 |
| Hard =0.7 + soft int. | 0.638 | 0.673 | 309.0 |
| Hard =0.8 + soft int. | 0.533 | 0.617 | 149.5 |
| TF-VSM (no ) | 0.638 | 0.674 | 468.1 |
Table 5. Ablation 2: hard semantic gate vs. threshold-free TF-VSM. “Pts kept” is the mean object points surviving the gate per pair.
Ablation 3: visual-score transform . We compare raw dot-product (), linear clamp (, default), sigmoid (), and quadratic clamp () in fig. 4a and Table 6. Raw and clamp are identical here (matched visual cosines are already in [0,1]), confirming linear clamping is information-preserving and parameter-free. Sigmoid gains slightly (0.672/0.690) but reintroduces a tuned slope/center—exactly the cut-off TF-VSM removes—while quadratic clamp compresses the discriminative mid-range and drops to 0.568/0.630. Linear clamp is the safe parameter-free choice.
| Visual transform | Checkpoint | Loss | TF-VSM | |||
| Raw () | 0.638 | 0.674 | Epoch 0 (pretrained) | — | 0.467/0.558 | 0.454/0.589 |
| Linear clamp [TF-VSM] | 0.638 | 0.674 | Epoch 2 (early) | 129.35 | 0.571/0.649 | 0.586/0.623 |
| Sigmoid | 0.672 | 0.690 | Epoch 5 (intermediate) | 111.01 | 0.609/0.665 | 0.620/0.633 |
| Quadratic clamp | 0.568 | 0.630 | Epoch 8 (peak) | 100.20 | 0.638/0.674 | 0.728/0.705 |
| Epoch 11 (over-fit) | 93.51 | 0.575/0.649 | 0.607/0.646 | |||
| Epoch 14 (final) | 88.63 | 0.607/0.655 | 0.658/0.636 |
Table 6. Ablations 3 (left) and 4 (right). Left: visual-score transform at epoch 8. Right: train loss vs. oracle correlation at milestone epochs.
**Ablation 4: retraining schedule length and
Figure 3: (a) Any smooth confidence weighting beats uniform, and peaky Softmax is worst; quadratic is a robust default. (b) Hard semantic gating is brittle—correlation collapses at =0.8—while threshold-free TF-VSM matches the oracle-tuned best gate with no cutoff.
over-fitting. Figure 4b and Table 6 (right) trace all 14 epochs. Train loss falls monotonically (141.5 → 88.6) but oracle correlation peaks at epoch 8 and then drifts down—continued loss minimization buys no generalization. Both TF-VSM and the tuned VSM peak at epoch 8, so the selection is not an artifact of the threshold-free metric, and over-fitting is gradual rather than catastrophic.
Ablation 5: training-data scaling. Retraining the identical recipe on nested subsets shows high sample efficiency: half the data recovers over 90% of the disentanglement gain, and the curve flattens toward the full split, indicating the ∼5,000-pair set is near saturation for this task.
Ablation 6: sensitivity to the visual loss scale . Sweeping during retraining (Table 7) reveals a key property: threshold-free TF-
| Visual loss scale | TF-VSM () | Hard () |
| (equal) | 0.647/0.679 | 0.679/0.672 |
| (moderate) | 0.644/0.675 | 0.719/0.711 |
| (default) | 0.638/0.674 | 0.728/0.705 |
| (strong) | 0.635/0.679 | 0.679/0.681 |
Table 7. Ablation 6: sensitivity to at the epoch-8 checkpoint. TF-VSM is nearly -invariant; hard VSM is not.
Figure 4a. Visual-score transform (Ablation 3).
Figure 4b. Retraining dynamics over 14 epochs (Ablation 4).
fig:4: (a) Linear clamp equals raw dot-products with maximal stability; sigmoid edges higher but re-adds tuned parameters; quadratic clamp over-compresses mid-range scores. (b) Contrastive train loss (grey, right axis) decreases monotonically while TF-VSM oracle correlation peaks at epoch 8 before mild over-fitting.
VSM is essentially invariant to (Pearson range [0.635, 0.647]), while hard-thresholded VSM () swings from 0.679 to 0.728—about 4× the sensitivity. Up-weighting the visual branch is thus not needed to disentangle features; it matters only to line up a well-scaled visual logit with the fixed cutoff that TF-VSM eliminates.
Conclusion
We presented TF-VSM, a continuous, threshold-free metric for evaluating visual consistency in subject-driven generation. Eliminating both binary thresholds () via non-linear semantic weighting () and continuous visual integration, paired with an extended disentanglement schedule (peaking at epoch 8), TF-VSM reaches , with the ground-truth oracle, far above CLIP/DINO/VLM and the pretrained baseline, while removing all hand-tuned cutoffs. It matches oracle-tuned hard gating without its brittleness, is invariant to the loss balance , survives a strict leakage-free protocol, and—critically—transfers to real human preference on DreamBench++ (+0.101 Pearson over standard VSM) at competitive latency, where the step-readout analysis shows the residual synthetic-oracle gap is an area-ratio artifact rather than miscalibration.
Limitations and future work. Multi-layer UNet feature extraction costs more than a single global encoder pass, and TF-VSM relies on foreground subject masks. Future work includes distilling the disentanglement into lightweight ViT backbones and extending threshold-free integration to 3D-consistent scene and video identity evaluation.
Acknowledgement
This work was supported by the National Science Foundation of China under Grants 62550002 and 62350069, and Natural Science Basic Research Project in Shaanxi Province of China under Grant 2025JC-YBMS-790 and Grant 2025JC-YBMS-710.
Appendix
Computational Efficiency: Latency and Memory
We benchmark per-pair latency (end-to-end wall-clock and GPU CUDA-event kernel timing) and peak VRAM on a single NVIDIA A100-80GB over the 100-pair controlled test set (Table 8; visualized in Figure 5b). TF-VSM runs in ms/pair with a 6.4 GB footprint—slightly faster than standard VSM, which incurs extra similarity-matrix binarization, at an identical memory cost. Global encoders (CLIP/DINOv2/LPIPS) are 5–25 cheaper but are insensitive to localized inconsistency (), while the Gemini VLM is slower than TF-VSM (cloud API) yet remains near-zero correlated. TF-VSM is the only strongly-correlated metric, and its threshold-free readout adds no cost over standard VSM.
| Metric | End-to-end (ms/pair) | GPU kernel (ms) | Peak VRAM (MB) | Oracle |
| LPIPS (AlexNet) | 9.86 | 4,349 | ||
| CLIP-I (OpenCLIP ViT-B/32) | 28.46 | 4,691 | ||
| DINOv2 (ViT-L/14) | 44.24 | 5,571 | ||
| Standard VSM () | 270.35 | 6,451 | ||
| TF-VSM (ours) | 228.67 | 6,450 | ||
| VLM (Gemini 2.5 Flash, API) | N/A | N/A |
Table 8. Per-pair latency and peak VRAM on an NVIDIA A100-80GB (100-pair controlled set). TF-VSM is the only strongly oracle-correlated metric and is slightly faster than standard VSM at the same memory footprint.
RefVNLI Protocol and Contemporary-Baseline Availability
We attempted to benchmark the three contemporary localized/personalized metrics cited in § 2. MaSC [5] and CLPIPS [53] have no publicly released code or pre-trained checkpoints at submission time and could not be reproduced. RefVNLI [47] provides no inference scripts or public checkpoints, and its backbone (paligemma_3b_ft_nlvr2_448) is access-restricted; we therefore faithfully reimplement its published subject-preservation decision protocol on Gemini (Vertex AI), scoring each reference–target pair by the log-probability ratio with the subject wrapped in the prescribed tags. RefVNLI ( controlled, held-out) improves marginally over unguided CLIP but remains in the same weak band as global metrics, since a single holistic VLM verdict cannot localize sub-region texture corruption.
DreamBench++ Robustness Across Personalization Methods
DreamBench++ spans 7 personalization methods of varying difficulty. On the two hardest IP-Adapter variants (ip_adapter_plus_vit_h_sdxl, ip_adapter_vit_g_sdxl), global embeddings collapse—CLIP-I drops to and DINOv2 to —because coarse pooled identity becomes unreliable when the generator injects strong appearance drift. TF-VSM remains robust in exactly this regime ( and ), outperforming both global baselines. This mirrors our controlled-benchmark finding: dense, disentangled point-wise correspondence degrades gracefully where spatial pooling fails abruptly.
Multi-Subject Scenarios and Cross-Instance Drift
TF-VSM handles multi-subject scenes through two mechanisms. (i) Foreground isolation: Grounded-SAM uses an open-vocabulary target prompt to extract the specific subject mask , restricting the comparison to the target instance and excluding surrounding subjects. (ii) Semantic-confidence suppression: for multiple similar instances inside , diffusion features carry spatial positional information, so cross-instance matches yield low or negative semantic cosine ; the quadratic weight drives these ambiguous point drifts to zero. This is the same mechanism validated by the similarity-skewness filter (§ 3.1) and Ablation 1 (§ 4.6), where any smooth non-linear weighting suppresses unaligned clutter.
On Technical Novelty
While TF-VSM builds on diffusion-feature extraction and the dual-branch aggregation of Eldesokey et al. [14], its contributions target metric formulation and stability rather than backbone design. (i) Formulation: both scalar thresholds for diffusion personalization are removed in closed form via non-linear confidence weighting and continuous visual integration
(a) Human-preference correlation (DreamBench++).
(b) Accuracy vs. latency & VRAM.
Figure 5. (a) On DreamBench++ (9,441 human-rated pairs; Sec. 4.5), threshold-free TF-VSM beats standard hard VSM by +0.101 Pearson, confirming the gain transfers to real human preference. (b) Accuracy–efficiency trade-off on an A100-80GB: TF-VSM attains strong localized sensitivity at ∼229 ms/pair (slightly faster than standard VSM at identical VRAM), while global metrics are cheaper but insensitive and the VLM is ∼25× slower.
eliminating step-discontinuities and point dropouts. (ii) Robustness: Ablation 6 shows hard VSM swings by up to 0.05 Pearson across visual-loss weights , whereas TF-VSM is virtually invariant (). (iii) Training: an extended retraining protocol lifts oracle correlation from 0.467/0.558 to 0.638/0.674. (iv) Differentiability: TF-VSM is the first fully differentiable dense visual-semantic consistency metric, usable directly as a loss or RL reward
References
- [1]Reza Abbasi, Ali Nazari, Aminreza Sefid, Mohammadali Banayeanzade, M. H. Rohban, and M. Baghshah. Analyzing CLIP’s performance limitations in multi-object scenarios: A controlled high-resolution study. ArXiv, abs/2502.19828, 2025.arxiv.org/abs/2502.19828
- [2]Serra Aksoy. Systematic evaluation of attribution methods: Eliminating threshold bias and revealing method-dependent performance patterns. ArXiv, abs/2509.03176, 2025.arxiv.org/abs/2509.03176
- [3]Shirzad Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep ViT features as dense visual descriptors. ArXiv, abs/2112.05814, 2021.arxiv.org/abs/2112.05814
- [4]Omri Avrahami, Amir Hertz, Yael Vinker, Moab Arar, Shlomi Fruchter, Ohad Fried, D. Cohen-Or, and D. Lischinski. The chosen one: Consistent characters in text-to-image diffusion models. ACM SIGGRAPH 2024 Conference Papers, 2023.DOI
- [5]Patryk Bartkowiak, Lennart Petersen, B. Kotrys, D. Michels, Soren Pirk, and Wojciech Palubicki. MaSC: A masked similarity metric for evaluating concept-driven generation. ArXiv, abs/2605.22469, 2026.arxiv.org/abs/2605.22469
- [6]Shengqu Cai, E. Chan, Yunzhi Zhang, Leonidas J. Guibas, Jiajun Wu, and Gordon Wetzstein. Diffusion self-distillation for zero-shot customized image generation. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18434–18443, 2024.arxiv.org/abs/2411.18616
- [7]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, J. Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9630–9640, 2021.arxiv.org/abs/2104.14294
- [8]Weifeng Chen, Tao Gu, Yuhao Xu, and Chengcai Chen. Magic clothing: Controllable garment-driven image synthesis. Proceedings of the 32nd ACM International Conference on Multimedia, 2024.arxiv.org/abs/2404.09512
- [9]Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. Why is spatial reasoning hard for VLMs? An attention mechanism perspective on focus areas. ArXiv, abs/2503.01773, 2025.arxiv.org/abs/2503.01773
- [10]Wenyue Chong. Towards robust semantic correspondence: A benchmark and insights. ArXiv, abs/2508.00272, 2025.arxiv.org/abs/2508.00272
- [11]C. Cuttano, Gabriele Trivigno, Carlo Masone, and Stefan Roth. Marco: Navigating the unseen space of semantic correspondence. ArXiv, abs/2604.18267, 2026.arxiv.org/abs/2604.18267
- [12]Anurag Das, Adrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas, B. Schiele, Georgios Tzimiropoulos, and Brais Martínez. More images, more problems? A controlled analysis of VLM failure modes. ArXiv, abs/2601.07812, 2026.arxiv.org/abs/2601.07812
- [13]Olaf Dunkel, Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, C. Theobalt, and A. Kotlyarovskiy. SoCo: Benchmarking semantic object correspondence in vision foundation models. ArXiv, abs/2605.31597, 2026.arxiv.org/abs/2605.31597
- [14]A. Eldesokey, Aleksandar Cvejic, Bernard Ghanem, and Peter Wonka. Mind-the-glitch: Visual correspondence for detecting inconsistencies in subject-driven generation. ArXiv, abs/2509.21989, 2025.DOI
- [15]Stephanie Fu, Netanel Y. Tamir, Shobhitha Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. DreamSim: Learning new dimensions of human visual similarity using synthetic data. ArXiv, abs/2306.09344, 2023.
- [16]Frank Fundel, Johannes Schusterbauer, Vincent Tao Hu, and Bjorn Ommer. Distillation of diffusion features for semantic correspondence. 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6762–6774, 2024.arxiv.org/abs/2412.03512
- [17]Rinon Gal, Yuval Alaluf, Y. Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and D. Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. ArXiv, abs/2208.01618, 2022.arxiv.org/abs/2208.01618
- [18]Sooyeon Go, Kyungmook Choi, Minjung Shin, and Youngjung Uh. Eye-for-an-eye: Appearance transfer with dense semantic correspondence in diffusion models. 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4641–4650, 2024.
- [19]Shaozhe Hao, K. Han, Shihao Zhao, and K. K. Wong. ViCo: Plug-and-play visual condition for personalized text-to-image generation. In arXiv, 2023.arxiv.org/abs/2306.00971
- [20]Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam N. Isack, Abhishek Kar, Andrea Tagliasacchi, and K. M. Yi. Unsupervised semantic correspondence using stable diffusion. ArXiv, abs/2305.15581, 2023.arxiv.org/abs/2305.15581
- [21]Hang Hua, Jing Shi, Kushal Kafle, Simon Jenni, Daoan Zhang, John P. Collomosse, Scott Cohen, and Jiebo Luo. FineMatch: Aspect-based fine-grained image and text mismatch detection and correction. In European Conference on Computer Vision, pp. 474–491, 2024.arxiv.org/abs/2404.14715
- [22]Hang Hua, Ziyun Zeng, Yizhi Song, Yunlong Tang, Liu He, Daniel G. Aliaga, Wei Xiong, and Jiebo Luo. MMIG-Bench: Towards comprehensive and explainable evaluation of multi-modal image generation models. ArXiv, abs/2505.19415, 2025.arxiv.org/abs/2505.19415
- [23]Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2I-CompBench: A comprehensive benchmark for open-world compositional text-to-image generation. ArXiv, abs/2307.06350, 2023a.arxiv.org/abs/2307.06350
- [24]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qin Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21807–21818, 2023b.arxiv.org/abs/2311.17982
- [25]A. Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, A. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3992–4003, 2023.arxiv.org/abs/2304.02643
- [26]Max W.F. Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. VIEScore: Towards explainable metrics for conditional image synthesis evaluation. ArXiv, abs/2312.14867, 2023a.arxiv.org/abs/2312.14867
- [27]Max W.F. Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. ImagenHub: Standardizing the evaluation of conditional image generation models. ArXiv, abs/2310.01596, 2023b.arxiv.org/abs/2310.01596
- [28]Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1931–1941, 2022.arxiv.org/abs/2212.04488
- [29]Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. PhotoMaker: Customizing realistic human photos via stacked ID embedding. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8640–8650, 2023.arxiv.org/abs/2312.04461
- [30]Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and D. Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pp. 366–384, 2024.arxiv.org/abs/2404.01291
- [31]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyu Li, Jianwei Yang, Hang Su, Jun-Juan Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. ArXiv, abs/2303.05499, 2023.arxiv.org/abs/2303.05499
- [32]Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. ArXiv, abs/2305.14334, 2023.arxiv.org/abs/2305.14334
- [33]Jiancang Ma, Junhao Liang, Chen Chen, and H. Lu. Subject-diffusion: Open domain personalized text- [34] Octave Mariotti, Oisin Mac Aodha, and Hakan Bilen. Improving semantic correspondence with viewpoint-guided spherical maps. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19521–19530, 2023.
- [35]Mohammad Hossein Moslemi and Mostafa Milani. Threshold-independent fair matching through score calibration. Proceedings of the Conference on Governance, Understanding and Integration of Data for Effective and Responsible AI, 2024.DOI
- [36]Jisu Nam, Gyuseong Lee, Sunwoo Kim, Ines Hyeonsu Kim, Hyoungwon Cho, Seyeong Kim, and Seungryong Kim. Diffusion model for dense matching. In International Conference on Learning Representations, 2023.arxiv.org/abs/2305.19094
- [37]J. Napolitano and Pascal Nguyen. Gram-MMD: A texture-aware metric for image realism assessment. ArXiv, abs/2604.03064, 2026.arxiv.org/abs/2604.03064
- [38]Yeongtak Oh, Sangwon Yu, Junsung Park, Han Cheol Moon, J. Mok, and Sungroh Yoon. Contextualized visual personalization in vision-language models. ArXiv, abs/2602.03454, 2026.arxiv.org/abs/2602.03454
- [39]M. Oquab, Timothee Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russ Howes, Po-Yao (Bernie) Huang, Shang-Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, J. Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. ArXiv, abs/2304.07193, 2023.arxiv.org/abs/2304.07193
- [40]Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. DreamBench++: A human-aligned benchmark for personalized image generation. ArXiv, abs/2406.16855, 2024.arxiv.org/abs/2406.16855
- [41]Dustin Podell, Zion English, Kyle Lacey, A. Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. ArXiv, abs/2307.01952, 2023.
- [42]Weimin Qiu, Jieke Wang, and Meng Tang. Self-cross diffusion guidance for text-to-image generation without test-time fine-tuning. ACM SIGGRAPH 2024 Conference Papers, 2023.arxiv.org/abs/2411.18936
- [43]Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674–10685, 2021.arxiv.org/abs/2112.10752
- [44]Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Y. Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22500–22510, 2022.arxiv.org/abs/2208.12242
- [45]Israel Salazar, Stella Frank, Dan Oneata, Desmond Elliott, and Constanza Fierro. Pathways of visual information flow in vision-language models. In arXiv, 2026.arxiv.org/abs/2607.03358
- [46]Chaehun Shin, Jooyoung Choi, Heeseung Kim, and Sungroh Yoon. Large-scale text-to-image model with inpainting is a zero-shot subject-driven image generator. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7986–7996, 2024.arxiv.org/abs/2411.15466
- [47]Aviv Slobodkin, Hagai Taitelbaum, Yonatan Bitton, Brian Gordon, Michal Sokolik, N. Guetta, Almog Gueta, Royi Rassin, Itay Laish, Dani Lischinski, and Idan Szpektor. RefVNLI: Towards scalable evaluation of subject-driven text-to-image generation. In Conference on Empirical Methods in Natural Language Processing, pp. 8420–8438, 2025.
- [48]Nick Stracke, Stefan Andreas Baumann, Kolja Bauer, Frank Fundel, and Bjorn Ommer. CleanDIFT: Diffusion features without noise. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 117–127, 2024.arxiv.org/abs/2412.03439
- [49]Saksham Suri, Matthew Walmer, Kamal Gupta, and Abhinav Shrivastava. LiFT: A surprisingly simple lightweight feature transform for dense ViT descriptors. In European Conference on Computer Vision, pp. 110–128, 2024.arxiv.org/abs/2403.14625
- [50]Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. OmniControl: Minimal and universal control for diffusion transformer. 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14940–14950, 2024.arxiv.org/abs/2411.15098
- [51]Luming Tang, Menglin Jia, Qianqian Wang, Cheng Peng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. ArXiv, abs/2306.03881, 2023.arxiv.org/abs/2306.03881
- [52]Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Y. Atzmon. Training-free consistent text-to-image generation. ACM Transactions on Graphics (TOG), 43:1–18, 2024.arxiv.org/abs/2402.03286
- [53]Khoi Trinh, Jay C. Rothenberger, Scott Seidenberger, Dimitrios I. Diochnos, and Anindya Maiti. CLIPIPS: A personalized metric for AI-generated image similarity. ArXiv, abs/2604.01234, 2026.arxiv.org/abs/2604.01234
- [54]Michael Tschannen, Alexey Gritsenko, Xiao Wang, M. Naeem, Ibrahim M. Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, A. Steiner, and Xiao-Qi Zhai. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. ArXiv, abs/2502.14786, 2025.arxiv.org/abs/2502.14786
- [55]Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1921–1930, 2022.arxiv.org/abs/2211.12572
- [56]Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and W. Zuo. ELITE: Encoding visual concepts into textual embeddings for customized text-to-image generation. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15897–15907, 2023.arxiv.org/abs/2302.13848
- [57]Shaojin Wu, Mengqi Huang, Yufeng Cheng, Wenxu Wu, Jiahe Tian, Yiming Luo, Fei Ding, and Qian He. USQ: Unified style and subject-driven generation via disentangled and reward learning. ArXiv, abs/2508.18966, 2025.arxiv.org/abs/2508.18966
- [58]Monika Wysoczańska, Oriane Simeoni, Michael Ramamonjisoa, Andrei Bursuc, Tomasz Trzcinski, and Patrick Pérez. CLIP-DINOiser: Teaching CLIP a few DINO tricks. In European Conference on Computer Vision, pp. 320–337, 2023.
- [59]Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 586–595, 2018.arxiv.org/abs/1801.03924
- [60]Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming Yang. A tale of two features: Stable diffusion complements DINO for zero-shot semantic correspondence. ArXiv, abs/2305.15347, 2023a.arxiv.org/abs/2305.15347
- [61]Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming Hsuan Yang. Telling left from right: Identifying geometry-aware semantic correspondence. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3076–3085, 2023b.arxiv.org/abs/2311.17034
- [62]Kai Zhang, Xinghui Li, Jingyi Lu, and Kai Han. Semantic correspondence: Unified benchmarking and a strong baseline. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48:3911–3930, 2025a.DOI
- [63]Yuxuan Zhang, Yiru Yuan, Yiren Song, Haofan Wang, and Jiaming Liu. EasyControl: Adding efficient and flexible control for diffusion transformer. 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 19513–19524, 2025b.arxiv.org/abs/2503.07027
- [64]Liyuan Zhu, Shengqu Cai, Shengyu Huang, Gordon Wetzstein, Naji Khosravan, and Iro Armeni. ReStyle3D: Scene-level appearance transfer with semantic correspondences. ArXiv, abs/2502.10377, 2025.arxiv.org/abs/2502.10377





