betaFeedback

WorksComputer Science

A Threshold-Free Visual-Semantic Matching Metric for Evaluating Consistency in Subject-Driven Image Generation

Abstract

We propose a Threshold-Free Visual-Sematic Matching (TF-VSM) metric, to quantitatively evaluate localized visual consistency across varying poses and environments in subject-driven text-to-image generation. TF-VSM is a fully continuous, threshold-free visual consistency metric paired with an extended feature-disentanglement retraining schedule for dual-branch ResNet feature-aggregation networks. Existing metrics suffer from point-alignment dropouts, threshold sensitivity, and score instability. To address them, our TF-VSM, rather than enforcing rigid binary cutoffs (TsT_s, TvT_v), weights every object point by a non-linear semantic match confidence wj=ReLU(s^j)2w_j = ReLU(\hat s_j)^2 to yield a closed-form confidence-weighted score without any hand-tuned hyperparameters. Experimental results show that, TF-VSM attains Pearson 0.638 and Spearman 0.674 oracle correlation, improving over the pretrained VSM baseline (0.454/0.589) by +0.171/ + 0.116 after retraining, and drastically outperforming CLIP (-0.052/ -0.004), DINO (0.087/0.121), and VLMs (0.096/0.003). Six diagnostic ablations confirm that smooth confidence weighting is the right mechanism, that hard semantic gating is brittle (dropping to 0.533 Pearson at TsT_s=0.8), and that the threshold-free metric is essentially invariant to the training time loss balance αα. Code is available at: https://github.com/bwshan/TF-VSM.

Introduction

Subject-driven text-to-image generation has advanced rapidly, enabling users to personalize diffusion models to synthesize specific target subjects across diverse poses, lighting, and background environments [44, 17, 28, 56, 19, 29, 50, 63]. A central requirement in evaluating these models is assessing visual consistency—determining whether the visual identity and subtle textural details of a subject are faithfully preserved across reference and generated image pairs [52, 4]. An ideal metric must remain invariant to variations in pose, background, scale, and perspective while remaining sensitive to fine-grained visual inconsistencies such as altered surface patterns, modified logos, or subtle structural artifacts. Despite its importance, quantifying visual consistency remains an open challenge. Prior evaluation protocols predominantly rely on two paradigms:

  1. Global feature similarity & vision-language models. Standard frameworks compute distances between global image embeddings such as CLIP-I [44, 17] or DINO-I [7, 39]. Because these representations pool features over spatial dimensions, they conflate high-level identity with layout and are blind to localized inconsistencies [1, 9]. Likewise, VLMs (e.g., GPT-4o, Gemini) and VLM-driven suites [26, 40, 22] offer impressionistic judgments that struggle to track fine attributes and aggregate local visual evidence across images [30, 12].

  1. Threshold-based point alignment & feature disentanglement. To overcome spatial pooling, recent approaches such as Visual-Semantic Matching (VSM) [14] extract internal features from pre-trained diffusion backbones [43, 48] and train dual aggregation networks to disentangle semantic structure from visual appearance. Standard VSM, however, computes consistency by applying hand-tuned binary thresholds (Ts,TvT_s,T_v) to filter semantic keypoints and to binarize visual similarities. As decision-boundary studies show [2, 35], hard thresholding creates step-discontinuities that cause severe point-alignment dropouts and score instability whenever valid correspondences fall slightly below arbitrary cutoffs.

To address these limitations, we introduce TF-VSM (Threshold-Free Visual-Semantic Matching), a continuous, fully threshold-free visual semantic consistency metric. Rather than gating correspondences with a rigid cutoff TsT_s, TF-VSM assigns every object point a smooth semantic-match weight wj=ReLU(s^j2)w_j = ReLU(\hat{s}_j^2), so each point contributes according to its alignment confidence. It further eliminates the visual threshold TvT_v by integrating the visual indicator over all consistency levels r∈[0,1]r \in[0,1], which reduces in closed form to a confidence-weighted soft visual similarity with no hand-tuned cutoffs. In tandem, we retrain the dual feature-aggregation networks on the Subjects200k-derived dataset [50, 14] with an extended schedule, identifying an optimal convergence checkpoint (epoch 8) that markedly improves feature disentanglement. Through controlled evaluation on a 100-pair test set with ground-truth inpainting oracles, TF-VSM achieves Pearson 0.638 and Spearman 0.674 correlation with the oracle, markedly outperforming global CLIP (−0.052/−0.004-0.052/-0.004), DINO (0.087/0.121), Gemini VLM (0.096/0.003), and the original pretrained VSM (0.454/0.589). Our contributions are:

  • A fully threshold-free visual-consistency metric that removes both TsT_s and TvT_v via non-linear confidence weighting and continuous visual integration, yielding a simple closed form.

  • An extended disentanglement retraining recipe that lifts oracle correlation from 0.467/0.558 to 0.638/0.674 (+0.171/ +0.116) over the released model.

  • Six diagnostic ablations isolating the confidence weighting, the semantic/visual readout transforms, the schedule length, the data scale, and the loss balance α\alpha, showing TF-VSM matches oracle-tuned hard gating while removing its brittle hyperparameter dependence.

  • A leakage-free evaluation protocol (disjoint selection/test splits) that independently recovers the epoch-8 checkpoint and confirms the gain on a strictly held-out split, together with foreground-masked baselines showing that background removal explains almost none of the gap to global metrics.

Related Work

We contextualize our work across three paradigms: global feature similarity and VLM evaluation, dense semantic correspondence via diffusion features, and threshold-based consistency metrics.

Global feature similarity & vision-language evaluation. Subject-driven personalization models [44, 17, 28, 56, 19, 29, 33] synthesize user-specified subjects across diverse contexts, and their evaluation requires standardized benchmarks [27, 23, 47]. Fidelity is historically quantified via global embedding distances using CLIP-I [44] or self-supervised backbones such as DINO/DINOv2 [7, 39, 58, 54]. However, global pooling makes these metrics insensitive to localized textural discrepancies [1, 15, 37], motivating localized perceptual variants such as matched-points LPIPS [8], human-aligned customized LPIPS [53], and foreground-masked patch matching [5]. In parallel, VLM-driven protocols such as DreamBench++ [40], VIEScore [26], VQAScore [30], MMIG-Bench [22], and FineMatch [21] assess image–text alignment, yet exhibit spatial-reasoning bottlenecks [9], cross-image aggregation failures [12], and reliance on text-mediated pathways [45, 38]. Recent suites such as USO-Bench [57] and RefVNLI [47] evaluate subject consistency but do not provide dense point-wise disentanglement of appearance from structure.

Dense semantic correspondence via diffusion features. Pretrained diffusion backbones [43, 41] implicitly learn rich spatial-semantic representations. DIFT [51] and Plug-and-Play [55] showed that internal UNet features enable zero-shot dense keypoint matching across challenging poses. Subsequent work expanded these capabilities via SD–DINO fusion [60], Diffusion Hyperfeatures [32], geometry-aware modeling [61], spherical priors [34], lightweight ViT feature transforms [49], prompt-attention optimization [20], and noise-free extraction via CleanDIFT [48], with distillation [16] and conditional matching [36] improving efficiency. Surveys [62, 10, 13, 11, 24] note that self-supervised ViT descriptors [3] and diffusion features achieve strong semantic alignment but intentionally discard appearance variation. Consequently, correspondence-driven consistency methods—ConsiStory [52], Eye-for-an-eye [18], ReStyle3D [64], The Chosen One [4], Self-Cross Guidance [42], and Grounding DINO [31]—focus on driving synthesis rather than explicitly evaluating visual consistency.

Threshold-based consistency metrics & subject-driven benchmarks. Automated generation and evaluation workflows [6, 46, 63] often leverage synthetic dataset pipelines. Pairing identity-consistent datasets such as Subjects200k [50] with SAM [25] and SDXL inpainting [41] yields controlled pairs with localized variations [14]. However, perceptual metrics such as LPIPS [59] and the original VSM [14] rely on hand-tuned scalar thresholds (Ts,Tv)(T_s,T_v); decision-boundary studies [2, 35] show that single-threshold binarization introduces selection bias where minor cutoff shifts drastically alter rankings. TF-VSM removes these artifacts by replacing binary cutoffs with continuous, confidence-weighted integration.

Methodology

We present the technical framework of TF-VSM: an automated data-generation pipeline producing paired data with known localized inconsistencies (section 3.1); a dual-ResNet aggregation architecture with a contrastive disentanglement objective (section 3.2); and the continuous, threshold-free TF-VSM metric (section 3.3). Figure 1 gives an overview.

Overview of the proposed framework

Figure 1. Overview of the proposed framework. Given an input pair (I1′,I2′)(I'_1,I'_2) with synthetic inpainted regions (R1,R2)(R_1,R_2), multi-layer features are extracted from a frozen diffusion backbone Φ\Phi and processed by dual-branch ResNet aggregation networks (Ψs,Ψv)(\Psi_s,\Psi_v) into semantic features SiS_i and visual features ViV_i. A contrastive disentanglement module enforces pose-invariant semantic matching over all correspondences while penalizing visual similarity inside inconsistent regions. The threshold-free TF-VSM unit then reads out a consistency score by soft-weighting visual similarities with squared semantic confidence wj=ReLU(s^j)2w_j = ReLU(\hat{s}_j)^2, avoiding brittle binary cutoffs (Ts,Tv)(T_s,T_v) and producing a global score alongside a diagnostic heatmap.

Data Generation Pipeline for Inconsistency Supervision

To learn to separate semantic structure from visual appearance, we construct a supervision dataset of paired images with precise spatial masks marking consistent and inconsistent regions, starting from identity-consistent pairs (I1,I2)(I_1,I_2) in Subjects200k [50].

We first isolate the foreground subject in both images with Grounded-SAM [25, 31], yielding binary subject masks O1,O2∈{0,1}h×wO_1,O_2 \in\{0,1\}^{h \times w}. To establish semantic correspondences, we extract features from the 6th UNet decoder layer of Stable Diffusion 2.1 [43, 48], F61=Φ(I1)F_6^1 = \Phi(I_1) and F62=Φ(I2)F_6^2 = \Phi(I_2), and form the dense similarity matrix D=F61(F62)T∈Rd×dD = F_6^1(F_6^2)^\mathsf{T} \in\mathbb{R}^{d \times d} with d=h×w=48×48=2304d = h \times w = 48 \times48 = 2304. Dense correspondences C1,C2∈RN×2C_1,C_2 \in\mathbb{R}^{N \times2} are recovered by spatial argmax:

C1={(x11,y11),…,(xN1,yN1)},C2={(x12,y12),…,(xN2,yN2)}.(1)\begin{aligned} C_1 &= \{(x_1^1,y_1^1),\ldots,(x_N^1,y_N^1)\},\\ C_2 &= \{(x_1^2,y_1^2),\ldots,(x_N^2,y_N^2)\}. \tag*{(1)} \end{aligned}

To create localized inconsistencies, we select a high-confidence match kk and prompt SAM to segment corresponding sub-regions Ri=SAM⁡(Ii,(xki,yki))R_i = \operatorname{SAM}(I_i,(x_k^i,y_k^i)), returning multiple hierarchical candidates and picking the smallest-footprint mask to avoid segmenting the whole object.

Ambiguity filtering via similarity skewness. On flat, textureless regions (plain surfaces, uniform clothing) point kk matches diffusely. We measure the skewness of D[k]D[k],

Skew⁡(D[k])=n(n−1)(n−2)∑i=1n(D[k]i−μσ)3,(2)\operatorname{Skew}(D[k]) = \frac{n}{(n-1)(n-2)} \sum_{i=1}^{n} \left(\frac{D[k]_i-\mu}{\sigma}\right)^3, \tag*{(2)}

with n=∣D[k]∣n = |D[k]|, μ=mean⁡(D[k])\mu= \operatorname{mean}(D[k]), σ=STD⁡(D[k])\sigma= \operatorname{STD}(D[k]); high skewness indicates sharp localized matches, low skewness ambiguous ones. We keep samples with Skew⁡(D[k])>1.3\operatorname{Skew}(D[k]) > 1.3.

Inpainting and perceptual validation. For regions passing structural checks we crop a padded patch and perform localized SDXL inpainting [41], so that new textures are synthesized without global object context. We then validate perceptual change via LPIPS [59], discarding pairs with LPIPS ≤0.15\leq0.15. The resulting dataset contains 4,912 usable training pairs (I1′,I2′)(I'_1,I'_2) with subject masks (O1,O2)(O_1,O_2) and inconsistency masks (R1,R2)(R_1,R_2).

Disentangled Feature Aggregation and Retraining Objective

Given an inconsistent pair (I1′,I2′)(I'_1,I'_2) with masks R1,R2R_1,R_2, we learn descriptors separating pose-invariant semantic structure from visual appearance. Multi-layer feature maps Fi=Φ(Ii′)F_i = \Phi(I'_i) across decoder layers l∈Ll \in\mathcal{L} are passed through two aggregation networks Ψs\Psi_s (semantic) and Ψv\Psi_v (visual), each a per-layer mini-ResNet [14] combined via trainable scalar weights wls,wlvw_l^s,w_l^v:

Si=∑l∈LwlsΨs(Fil),Vi=∑l∈LwlvΨv(Fil),Si,Vi∈Rd×q,(3)S_i = \sum_{l\in\mathcal{L}} w_l^s \Psi_s(F_i^l), \qquad V_i = \sum_{l\in\mathcal{L}} w_l^v \Psi_v(F_i^l), \qquad S_i,V_i \in\mathbb{R}^{d \times q}, \tag*{(3)}

with projected channel dimension q=384q = 384 and d=2304d = 2304. We form spatial correlation matrices Ds12=S1S2TD_s^{12} = S_1S_2^\mathsf{T}, Ds21=S2S1TD_s^{21} = S_2S_1^\mathsf{T}, Dv12=V1V2TD_v^{12} = V_1V_2^\mathsf{T}, Dv21=V2V1TD_v^{21} = V_2V_1^\mathsf{T}, and partition correspondences into interior (PiniP_{\mathrm{in}}^i, inside RiR_i) and exterior (PoutiP_{\mathrm{out}}^i) subsets:

Pini={(xji,yji)∈Ci∣Ri(xji,yji)=1},Pouti={(xji,yji)∈Ci∣Ri(xji,yji)=0}.(4)\begin{aligned} P_{\mathrm{in}}^i &= \{(x_j^i,y_j^i)\in C_i \mid R_i(x_j^i,y_j^i)=1\},\\ P_{\mathrm{out}}^i &= \{(x_j^i,y_j^i)\in C_i \mid R_i(x_j^i,y_j^i)=0\}. \tag*{(4)} \end{aligned}

Semantic features are pulled together across all matches via a cross-entropy loss over Pi=Pini∪PoutiP_i=P_{\mathrm{in}}^i\cup P_{\mathrm{out}}^i, while visual features are pulled together outside and pushed apart inside inconsistent regions:

Ls=CE(Ds12(P1),P2),Linv=CE(−Dv12(Pin1),Pin2),Loutv=CE(Dv12(Pout1),Pout2).(5)\begin{aligned} \mathcal{L}_s &= CE(D_s^{12}(P_1),P_2),\\ \mathcal{L}_{\mathrm{in}}^v &= CE(-D_v^{12}(P_{\mathrm{in}}^1),P_{\mathrm{in}}^2),\\ \mathcal{L}_{\mathrm{out}}^v &= CE(D_v^{12}(P_{\mathrm{out}}^1),P_{\mathrm{out}}^2). \tag*{(5)} \end{aligned}

Combining terms symmetrically over both directions gives L=Ls+α(Linv+Loutv)\mathcal{L}=\mathcal{L}_s+\alpha(\mathcal{L}_{\mathrm{in}}^v+\mathcal{L}_{\mathrm{out}}^v), where α=10\alpha=10 prioritizes visual learning. We optimize with AdamW (η=5×10−4\eta=5\times10^{-4}) over an extended 14-epoch schedule and select the peak checkpoint by oracle correlation.

Fully Threshold-Free TF-VSM Formulation

Given disentangled features Si,ViS_i,V_i, we extract per-point best-match similarities for all foreground points j∈O1j\in O_1: s^j=max⁡kDs(j,k)\hat{s}_j=\max_k D_s(j,k) and v^j=max⁡kDv(j,k)\hat{v}_j=\max_k D_v(j,k). Standard VSM [14] applies hard thresholds:

VSM(Tv)=1∣Js∣∑j∈Js1[v^j>Tv],Js={j∈O1∣s^j>Ts}.(6)VSM(T_v)=\frac{1}{|J_s|}\sum_{j\in J_s}\mathbb{1}[\hat{v}_j>T_v],\qquad J_s=\{j\in O_1\mid\hat{s}_j>T_s\}. \tag*{(6)}

which introduces step-discontinuities: points with s^j\hat{s}_j just below Ts=0.7T_s=0.7 are discarded, and visual scores just below Tv=0.6T_v=0.6 drop abruptly to zero. We resolve both. First, instead of gating with TsT_s, every point receives a smooth non-linear confidence weight

wj=ReLU⁡(s^j)2,(7)w_j=\operatorname{ReLU}(\hat{s}_j)^2, \tag*{(7)}

continuously suppressing unaligned clutter while keeping non-zero weight on valid regions. Second, instead of an arbitrary TvT_v, we integrate the visual indicator over all levels r∈[0,1]r\in[0,1]; with g(v^j)=clamp⁡(v^j,0,1)g(\hat{v}_j)=\operatorname{clamp}(\hat{v}_j,0,1), the fundamental theorem of calculus gives ∫011[g(v^j)>r] dr=g(v^j)\int_0^1\mathbb{1}[g(\hat{v}_j)>r]\,dr=g(\hat{v}_j). Combining yields the closed-form TF-VSM:

Soft-VSM=∑j∈O1wjg(v^j)∑j∈O1wj=∑j∈O1ReLU⁡(s^j)2clamp⁡(v^j,0,1)∑j∈O1ReLU⁡(s^j)2.(8)\begin{aligned} Soft\text{-}VSM &= \frac{\sum_{j\in O_1}w_jg(\hat{v}_j)}{\sum_{j\in O_1}w_j}\\ &=\frac{\sum_{j\in O_1}\operatorname{ReLU}(\hat{s}_j)^2\operatorname{clamp}(\hat{v}_j,0,1)}{\sum_{j\in O_1}\operatorname{ReLU}(\hat{s}_j)^2}. \tag*{(8)} \end{aligned}

TF-VSM is fully continuous, parameter-free (no TsT_s or TvT_v), smooth in the features, and robust to partial spatial misalignments.

Experiments

We evaluate TF-VSM on a controlled inconsistency benchmark (§ 4.2), realistic subject-driven generators (§ 4.4), and six diagnostic ablations (§ 4.6).

Experimental Setup and Baselines

Training. We use the 4,912 pairs from § 3.1. Features are extracted from a frozen Stable Diffusion 2.1 backbone [43, 48] at resolution h=w=48h=w=48 (d=2304d=2304) and projected to q=384q=384; α=10\alpha=10. Models train with AdamW (η=5×10−4\eta=5\times10^{-4}) for 14 epochs on a single A100 (172.7 min total).

Ground-truth oracle. Following Eldesokey et al. [14], we evaluate on a verified 100-pair test set. For each pair we compute an exact oracle from the relative footprint of the inconsistent region R1R_1 within the subject mask O1O_1:

Oracle=1−∑R1∑O1.(9)Oracle=1-\frac{\sum R_1}{\sum O_1}. \tag*{(9)}

An accurate metric must correlate strongly (Pearson rr, Spearman ρ\rho) with this oracle.

Baselines. We compare against: (i) CLIP global cosine similarity (OpenCLIP ViT-B/32) [44]; (ii) DINOv2 global similarity (ViT-L/14) [39]; (iii) VLM judgment (Gemini 2.0 Flash prompted for a score [0,100][0,100]) [26, 40]; and (iv) standard VSM with scalar thresholds Ts=0.7T_s=0.7, Tv=0.6T_v=0.6 [14], both pretrained and retrained.

Controlled Inconsistency Evaluation

Figure 2 and table 1 report oracle correlations on the 100-pair controlled test set. Global embedding metrics fail on localized inconsistency: CLIP is near-zero/negative (−0.052/ −0.004) and DINOv2 weak (0.087/0.121), because spatial pooling conflates scene context with subject detail. The Gemini VLM likewise shows minimal correlation (0.096/0.003) due to score saturation and poor fine-grained spatial tracking. Among recently proposed localized/personalized metrics (RefVNLI, MaSC, CLPIPS), only RefVNLI [47] is reproducible—MaSC and CLPIPS have released no public code or checkpoints—and its single-prompt VLM decision protocol still lands in the same weak band as the global baselines (r=0.124r=0.124), far below TF-VSM, confirming that localized consistency requires dense point-wise disentanglement rather than holistic judgment (section B). Our retrained TF-VSM attains r=0.638r = 0.638, ρ=0.674\rho= 0.674, improving by +0.171/ + 0.116 over the released pretrained model (0.467/0.558) and far exceeding all CLIP/DINO/VLM baselines. Hard threshold VSM tuned specifically to Tv=0.6T_v=0.6 reaches 0.728/0.705 on this test set, but that tuning is only realizable with oracle access and is highly brittle (§ 4.6); TF-VSM provides competitive, stable correlation with zero tuning.

Figure 2. Oracle correlation of each metric on the 100-pair controlled test set. Global metrics (CLIP, DINOv2) and the VLM fail on localized inconsistency, whereas retrained TF-VSM (threshold-free, highlighted) reaches 0.638/0.674, a +0.171/ + 0.116 gain over its pretrained version. The hand-tuned VSM (Tv=0.6T_v=0.6, retrained) is higher on this set but—as section 4.6 shows—is brittle to its thresholds.

Why adopt TF-VSM over oracle-tuned hard VSM? Although hard VSM tuned to Tv=0.6T_v=0.6 edges out TF-VSM on this benchmark, that peak is unattainable in practice: fixing Tv,TsT_v,T_s requires ground truth inconsistency masks that do not exist for open-ended generation, and the optimum is fragile. TF-VSM instead (i) generalizes better on realistic generators with no tuning (§ 4.4), (ii) is nearly invariant to the loss balance α\alpha and gate placement, whereas hard VSM swings by up to 0.10 Pearson (§ 4.6), and (iii) is fully differentiable, usable directly as a loss or RL reward—a role the step-discontinuous indicator VSM cannot fill.

Leakage-Free Model Selection on a Held-Out Split

The residual gap is an artifact of the area-ratio oracle, not miscalibration. The synthetic oracle Oracle=1−∥R1∥/∥O1∥Oracle = 1-\lVert R_1\rVert/\lVert O_1\rVert measures the spatial extent of corruption, agnostic to graded severity, so a binary step readout 1[v^j>Tv]\mathbb{1}[\hat{v}_j > T_v] is structurally matched to it. Keeping TF-VSM’s threshold-free weights but replacing the continuous clamp g(v^j)g(\hat{v}_j) with a step at Tv=0.5T_v=0.5 raises Pearson from 0.638 to 0.709, essentially closing the gap to tuned hard VSM (0.728)—so the difference is the oracle’s step-wise area construction, not feature miscalibration (power-family sharpening does not help; Ablation 3). Crucially, on realistic generators with human-annotated oracles that penalize graded severity, the continuous readout outperforms tuned hard VSM (§ 4.4), as does the human-preference evaluation below.

Because the main protocol selects the peak checkpoint (epoch 8) by oracle correlation, one might worry that this selection touches the same 100 pairs used for reporting. We therefore re-run selection under a strict, leakage-free protocol. The 4,912 supervision pairs are split 90/10 into 4,421 training and 491 validation pairs (used only to monitor concontrastive loss); the 100-pair benchmark is split into a 20-pair selection split (used exclusively to pick the epoch over epochs 1–14) and an 80-pair held-out test split (reported only). The selection split independently identifies epoch 8 as the peak, identical to the main paper without observing any test sample. Table 2 reports the held-out results: retrained TF-VSM attains r=0.634r=0.634, ρ=0.697\rho=0.697, a +0.140/ + 0.111 gain over the pretrained model and far above all CLIP/DINO/VLM baselines. The epoch-8 selection is thus not an artifact of test-set leakage, and all conclusions hold under strict split separation.

Metricrrρ\rho
CLIP-I [44]0.0140.051
DINOv2 [39]0.1100.150
VLM (Gemini) [40]0.1080.029
RefVNLI [47]0.073−0.013
Pretrained VSM [14]0.4570.603
TF-VSM (pretrained)0.4940.586
Retrained VSM (val-sel.)0.6550.677
TF-VSM (ours, val-sel.)0.6340.697

Table 2. Leakage-free evaluation on the held-out 80-pair test split (n=80n=80). The epoch is selected only on a disjoint 20-pair split; the 80 test pairs are never seen during selection. Retrained TF-VSM retains its large gain over the pretrained model and all global/VLM baselines.

Subject-Driven Image Generation Benchmarking

To probe generalization to realistic synthesis, we benchmark three subject-driven generators: Diptych Prompting [46] (FLUX-dev, 768×1536768 \times1536), DSD-Diffusion [6] (512×512512 \times512), and EasyControl [63] (768×768768 \times768). We generate 100 pairs per generator and manually annotate inconsistent sub-regions to form the oracle. Table 3 shows TF-VSM maintains robust correlation (r=0.438r=0.438, ρ=0.412\rho=0.412) under real-world distributions, whereas the VLM saturates, scoring nearly

Metric paradigmPearson (rr)Spearman (ρ\rho)Score rangeTuning?
CLIP-I [44]0.1560.112[0.62, 0.88]No
DINO-I [39]0.1640.146[0.51, 0.82]No
VLM (Gemini)∗^{*} [26]0.0790.073[?, ?] (saturated)No
Pretrained VSM (Tv=0.6T_{\mathrm{v}}=0.6) [14]0.4050.369[0.00, 0.91]Yes (Ts,TvT_{\mathrm{s}}, T_{\mathrm{v}})
TF-VSM (ours, retrained)0.4380.412[0.04, 0.94]No

Table 3. Evaluation on realistic subject-driven generation (Diptych, DSD-Diffusion, EasyControl) against human-annotated identity-consistency oracles. ∗^{*}VLM = Gemini.

all images in [?, ?] regardless of corruption.

Human-Preference Evaluation on DreamBench++

The controlled oracle measures corruption area; a natural question is whether TF-VSM tracks human judgments of subject fidelity. We therefore evaluate on DreamBench++ [40], the standard human-aligned benchmark for subject-driven generation, comprising 9,441 rated pairs (150 concepts ×\times 9 prompts ×\times 7 personalization methods) with independent human Concept-Preservation (CP) ratings (0–4). Following the official protocol, we correlate each metric with the mean human CP per method and average across methods; foreground masks for VSM/TF-VSM are obtained with the same Grounded-SAM pipeline. Table 4 and Figure 5a report the results. TF-VSM correlates with real human preference at r=0.339r=0.339, ρ=0.350\rho=0.350, a decisive +0.101/+0.092 gain over standard hard-thresholded VSM (0.238/0.258)—removing brittle cutoffs improves alignment with human judgment, not merely with the synthetic area oracle. As expected on a benchmark rating holistic identity, global embeddings and the VLM score highest overall; yet on the two hardest IP-Adapter generators they collapse (CLIP-I r→0.132r \to0.132, DINOv2 r→−0.005r \to-0.005) while TF-VSM stays robust (r=0.220r=0.220, 0.284; section C). The two benchmarks are complementary: global metrics capture coarse identity but fail on the localized texture inconsistencies (r≈−0.05r \approx-0.05; Table 1) TF-VSM targets.

Evaluation metricPearson (rr)Spearman (ρ\rho)
CLIP image similarity−0.052−0.004
DINOv2 image similarity0.0870.121
VLM judgment0.0960.003
RefVNLI protocol0.1240.031
Standard VSM (pretrained)0.4540.589
Standard VSM (retrained)0.7280.705
TF-VSM (pretrained)0.4670.558
TF-VSM (retrained)0.6380.674

Table 1. Correlation with the ground-truth oracle on the 100-pair controlled test set. TF-VSM achieves strong correlation with no scalar thresholds (Ts,TvT_s,T_v). ∗VLM = Gemini 2.0 Flash. †RefVNLI evaluated via its published subject-preservation decision protocol (section B); MaSC/CLPIPS have no public code or checkpoints.

Evaluation metricPearson (rr)Spearman (ρ\rho)Pairwise agr. %Pooled rr (nn=9.4k)
CLIP-I (OpenCLIP ViT-B/32)0.4090.42767.70.622
DINOv2 (ViT-L/14)0.3430.37665.80.558
VLM (Gemini 2.0/2.5 Flash)0.4510.47771.10.589
Standard VSM (retrained, TsT_s=0.7, TvT_v=0.6)0.2380.25860.70.387
TF-VSM (ours, retrained, threshold-free)0.3390.35064.30.517

Table 4. Human-preference evaluation on DreamBench++ [40] (9,441 pairs, mean-CP oracle; visualized in Figure 5a). Among dense point-wise metrics, threshold-free TF-VSM beats standard hard VSM by +0.101/+0.092; global metrics/VLM rate holistic identity (their native regime) but collapse on hard IP-Adapter generators (section C).

Diagnostic Ablation Studies

We perform six controlled ablations on the full 100-pair test set with the epoch-8 model unless noted: (1) semantic confidence weighting, (2) hard vs. soft semantic gating, (3) the visual-score transform, (4) schedule length, (5) data scaling, and (6) the loss balance α\alpha.

Ablation 1: semantic confidence weighting wj=f(s^j)w_j = f(\hat{s}_j). We compare uniform (wj=1w_j=1), linear (ReLU⁡(s^j)\operatorname{ReLU}(\hat{s}_j)), quadratic (ReLU⁡(s^j)2\operatorname{ReLU}(\hat{s}_j)^2, default), cubic (ReLU⁡(s^j)3\operatorname{ReLU}(\hat{s}_j)^3), and temperature Softmax (τ=0.1\tau=0.1). As Figure 3a shows, any smooth non-linear weighting beats uniform (0.631/0.653) by down-weighting unaligned background, while peaky Softmax overconcentrates mass onto a few keypoints and degrades to 0.593/0.644. Within the smooth family the choice is near-flat (≤0.008\leq0.008 spread), so quadratic is a robust, defensible default.

Figure 3. (a) Semantic confidence weighting (Ablation 1). (b) Hard TsT_s gating vs. soft weighting (Ablation 2).

Ablation 2: soft weighting vs. hard semantic gating TsT_{\mathrm{s}}. We compare threshold-free TF-VSM against hard gates Ts∈{0.5,0.6,0.7,0.8}T_{\mathrm{s}} \in\{0.5, 0.6, 0.7, 0.8\} applied before the same soft visual integration (Figure 3b, Table 5). Raising TsT_{\mathrm{s}} from 0.6 to 0.8 collapses Pearson from 0.652 to 0.533 as 66% of valid points are discarded (403.7 →\to 149.5 per pair). TF-VSM matches the oracle-tuned best gate (Ts=0.6T_{\mathrm{s}}=0.6) to within 0.013/0.012 without any cutoff, sidestepping this brittleness.

Semantic filteringrrρ\rhoPts/pair
Hard TsT_s=0.5 + soft int.0.6440.680444.6
Hard TsT_s=0.6 + soft int.0.6520.686403.7
Hard TsT_s=0.7 + soft int.0.6380.673309.0
Hard TsT_s=0.8 + soft int.0.5330.617149.5
TF-VSM (no TsT_s)0.6380.674468.1

Table 5. Ablation 2: hard semantic gate TsT_s vs. threshold-free TF-VSM. “Pts kept” is the mean object points surviving the gate per pair.

Ablation 3: visual-score transform g(v^j)g(\hat{v}_j). We compare raw dot-product (g(v)=vg(v)=v), linear clamp (clamp⁡(v,0,1)\operatorname{clamp}(v,0,1), default), sigmoid (σ(10(v−0.5))\sigma(10(v-0.5))), and quadratic clamp (clamp⁡(v,0,1)2\operatorname{clamp}(v,0,1)^2) in fig. 4a and Table 6. Raw and clamp are identical here (matched visual cosines are already in [0,1]), confirming linear clamping is information-preserving and parameter-free. Sigmoid gains slightly (0.672/0.690) but reintroduces a tuned slope/center—exactly the cut-off TF-VSM removes—while quadratic clamp compresses the discriminative mid-range and drops to 0.568/0.630. Linear clamp is the safe parameter-free choice.

Visual transform g(v)g(v)rrρ\rhoCheckpointLossTF-VSMVSMTv=0.6VSM_{T_v=0.6}
Raw (g(v)=vg(v)=v)0.6380.674Epoch 0 (pretrained)—0.467/0.5580.454/0.589
Linear clamp [TF-VSM]0.6380.674Epoch 2 (early)129.350.571/0.6490.586/0.623
Sigmoid σ(10(v−0.5))\sigma(10(v-0.5))0.6720.690Epoch 5 (intermediate)111.010.609/0.6650.620/0.633
Quadratic clamp (⋅)2(\cdot)^20.5680.630Epoch 8 (peak)100.200.638/0.6740.728/0.705
Epoch 11 (over-fit)93.510.575/0.6490.607/0.646
Epoch 14 (final)88.630.607/0.6550.658/0.636

Table 6. Ablations 3 (left) and 4 (right). Left: visual-score transform g(v)g(v) at epoch 8. Right: train loss vs. oracle correlation at milestone epochs.

**Ablation 4: retraining schedule length and

Figure 3: (a) Any smooth confidence weighting beats uniform, and peaky Softmax is worst; quadratic is a robust default. (b) Hard semantic gating is brittle—correlation collapses at TsT_s=0.8—while threshold-free TF-VSM matches the oracle-tuned best gate with no cutoff.

over-fitting. Figure 4b and Table 6 (right) trace all 14 epochs. Train loss falls monotonically (141.5 → 88.6) but oracle correlation peaks at epoch 8 and then drifts down—continued loss minimization buys no generalization. Both TF-VSM and the tuned VSM peak at epoch 8, so the selection is not an artifact of the threshold-free metric, and over-fitting is gradual rather than catastrophic.

Ablation 5: training-data scaling. Retraining the identical recipe on nested subsets shows high sample efficiency: half the data recovers over 90% of the disentanglement gain, and the curve flattens toward the full split, indicating the ∼5,000-pair set is near saturation for this task.

Ablation 6: sensitivity to the visual loss scale α\alpha. Sweeping α∈{1,5,10,20}\alpha\in\{1, 5, 10, 20\} during retraining (Table 7) reveals a key property: threshold-free TF-

Visual loss scale α\alphaTF-VSM (r/ρr/\rho)Hard VSMTv=0.6VSM_{T_v=0.6} (r/ρr/\rho)
α=1\alpha=1 (equal)0.647/0.6790.679/0.672
α=5\alpha=5 (moderate)0.644/0.6750.719/0.711
α=10\alpha=10 (default)0.638/0.6740.728/0.705
α=20\alpha=20 (strong)0.635/0.6790.679/0.681

Table 7. Ablation 6: sensitivity to α\alpha at the epoch-8 checkpoint. TF-VSM is nearly α\alpha-invariant; hard VSM is not.

Figure 4a. Visual-score transform g(v)g(v) (Ablation 3).

Figure 4b. Retraining dynamics over 14 epochs (Ablation 4).

fig:4: (a) Linear clamp equals raw dot-products with maximal stability; sigmoid edges higher but re-adds tuned parameters; quadratic clamp over-compresses mid-range scores. (b) Contrastive train loss (grey, right axis) decreases monotonically while TF-VSM oracle correlation peaks at epoch 8 before mild over-fitting.

VSM is essentially invariant to α\alpha (Pearson range [0.635, 0.647]), while hard-thresholded VSM (Tv=0.6T_v=0.6) swings from 0.679 to 0.728—about 4× the sensitivity. Up-weighting the visual branch is thus not needed to disentangle features; it matters only to line up a well-scaled visual logit with the fixed TvT_v cutoff that TF-VSM eliminates.

Conclusion

We presented TF-VSM, a continuous, threshold-free metric for evaluating visual consistency in subject-driven generation. Eliminating both binary thresholds (Ts,TvT_s,T_v) via non-linear semantic weighting (wj=ReLU(s^j)2w_j = ReLU(\hat{s}_j)^2) and continuous visual integration, paired with an extended disentanglement schedule (peaking at epoch 8), TF-VSM reaches r=0.638r=0.638, ρ=0.674\rho=0.674 with the ground-truth oracle, far above CLIP/DINO/VLM and the pretrained baseline, while removing all hand-tuned cutoffs. It matches oracle-tuned hard gating without its brittleness, is invariant to the loss balance α\alpha, survives a strict leakage-free protocol, and—critically—transfers to real human preference on DreamBench++ (+0.101 Pearson over standard VSM) at competitive latency, where the step-readout analysis shows the residual synthetic-oracle gap is an area-ratio artifact rather than miscalibration.

Limitations and future work. Multi-layer UNet feature extraction costs more than a single global encoder pass, and TF-VSM relies on foreground subject masks. Future work includes distilling the disentanglement into lightweight ViT backbones and extending threshold-free integration to 3D-consistent scene and video identity evaluation.

Acknowledgement

This work was supported by the National Science Foundation of China under Grants 62550002 and 62350069, and Natural Science Basic Research Project in Shaanxi Province of China under Grant 2025JC-YBMS-790 and Grant 2025JC-YBMS-710.

Appendix

Computational Efficiency: Latency and Memory

We benchmark per-pair latency (end-to-end wall-clock and GPU CUDA-event kernel timing) and peak VRAM on a single NVIDIA A100-80GB over the 100-pair controlled test set (Table 8; visualized in Figure 5b). TF-VSM runs in ∼229\sim229 ms/pair with a 6.4 GB footprint—slightly faster than standard VSM, which incurs extra similarity-matrix binarization, at an identical memory cost. Global encoders (CLIP/DINOv2/LPIPS) are 5–25×\times cheaper but are insensitive to localized inconsistency (r≤0.12r \le0.12), while the Gemini VLM is ∼25×\sim25\times slower than TF-VSM (cloud API) yet remains near-zero correlated. TF-VSM is the only strongly-correlated metric, and its threshold-free readout adds no cost over standard VSM.

MetricEnd-to-end (ms/pair)GPU kernel (ms)Peak VRAM (MB)Oracle r/ρr/\rho
LPIPS (AlexNet)9.90±0.329.90 \pm0.329.864,3490.021/−0.0330.021/-0.033
CLIP-I (OpenCLIP ViT-B/32)28.50±1.1528.50 \pm1.1528.464,691−0.052/−0.004-0.052/-0.004
DINOv2 (ViT-L/14)44.29±0.1144.29 \pm0.1144.245,5710.121/0.1590.121/0.159
Standard VSM (Tv=0.6T_v=0.6)270.40±13.91270.40 \pm13.91270.356,4510.728/0.7050.728/0.705
TF-VSM (ours)228.73±1.06228.73 \pm1.06228.676,4500.638/0.6740.638/0.674
VLM (Gemini 2.5 Flash, API)5738.84±3867.555738.84 \pm3867.55N/AN/A0.096/0.0030.096/0.003

Table 8. Per-pair latency and peak VRAM on an NVIDIA A100-80GB (100-pair controlled set). TF-VSM is the only strongly oracle-correlated metric and is slightly faster than standard VSM at the same memory footprint.

RefVNLI Protocol and Contemporary-Baseline Availability

We attempted to benchmark the three contemporary localized/personalized metrics cited in § 2. MaSC [5] and CLPIPS [53] have no publicly released code or pre-trained checkpoints at submission time and could not be reproduced. RefVNLI [47] provides no inference scripts or public checkpoints, and its backbone (paligemma_3b_ft_nlvr2_448) is access-restricted; we therefore faithfully reimplement its published subject-preservation decision protocol on Gemini (Vertex AI), scoring each reference–target pair by the log-probability ratio P(“1”)/(P(“1”)+P(“0”))P(\text{“1”})/(P(\text{“1”}) + P(\text{“0”})) with the subject wrapped in the prescribed tags. RefVNLI (r=0.124r=0.124 controlled, 0.0730.073 held-out) improves marginally over unguided CLIP but remains in the same weak band as global metrics, since a single holistic VLM verdict cannot localize sub-region texture corruption.

DreamBench++ Robustness Across Personalization Methods

DreamBench++ spans 7 personalization methods of varying difficulty. On the two hardest IP-Adapter variants (ip_adapter_plus_vit_h_sdxl, ip_adapter_vit_g_sdxl), global embeddings collapse—CLIP-I drops to r=0.132r=0.132 and DINOv2 to r=−0.005r=-0.005—because coarse pooled identity becomes unreliable when the generator injects strong appearance drift. TF-VSM remains robust in exactly this regime (r=0.220r=0.220 and r=0.284r=0.284), outperforming both global baselines. This mirrors our controlled-benchmark finding: dense, disentangled point-wise correspondence degrades gracefully where spatial pooling fails abruptly.

Multi-Subject Scenarios and Cross-Instance Drift

TF-VSM handles multi-subject scenes through two mechanisms. (i) Foreground isolation: Grounded-SAM uses an open-vocabulary target prompt to extract the specific subject mask O1O_1, restricting the comparison to the target instance and excluding surrounding subjects. (ii) Semantic-confidence suppression: for multiple similar instances inside O1O_1, diffusion features carry spatial positional information, so cross-instance matches yield low or negative semantic cosine s^j\hat{s}_j; the quadratic weight wj=ReLU⁡(s^j)2w_j=\operatorname{ReLU}(\hat{s}_j)^2 drives these ambiguous point drifts to zero. This is the same mechanism validated by the similarity-skewness filter (§ 3.1) and Ablation 1 (§ 4.6), where any smooth non-linear weighting suppresses unaligned clutter.

On Technical Novelty

While TF-VSM builds on diffusion-feature extraction and the dual-branch aggregation of Eldesokey et al. [14], its contributions target metric formulation and stability rather than backbone design. (i) Formulation: both scalar thresholds for diffusion personalization are removed in closed form via non-linear confidence weighting and continuous visual integration

(a) Human-preference correlation (DreamBench++).

(b) Accuracy vs. latency & VRAM.

Figure 5. (a) On DreamBench++ (9,441 human-rated pairs; Sec. 4.5), threshold-free TF-VSM beats standard hard VSM by +0.101 Pearson, confirming the gain transfers to real human preference. (b) Accuracy–efficiency trade-off on an A100-80GB: TF-VSM attains strong localized sensitivity at ∼229 ms/pair (slightly faster than standard VSM at identical VRAM), while global metrics are cheaper but insensitive and the VLM is ∼25× slower.

∫011[clamp⁡(v^j,0,1)>r] dr=clamp⁡(v^j,0,1),\int_{0}^{1} \mathbb{1}\left[\operatorname{clamp}(\hat{v}_j,0,1)>r\right]\,dr=\operatorname{clamp}(\hat{v}_j,0,1),

eliminating step-discontinuities and point dropouts. (ii) Robustness: Ablation 6 shows hard VSM swings by up to 0.05 Pearson across visual-loss weights α\alpha, whereas TF-VSM is virtually invariant (r∈[0.635,0.647]r \in[0.635,0.647]). (iii) Training: an extended retraining protocol lifts oracle correlation from 0.467/0.558 to 0.638/0.674. (iv) Differentiability: TF-VSM is the first fully differentiable dense visual-semantic consistency metric, usable directly as a loss or RL reward

References

  1. [1]Reza Abbasi, Ali Nazari, Aminreza Sefid, Mohammadali Banayeanzade, M. H. Rohban, and M. Baghshah. Analyzing CLIP’s performance limitations in multi-object scenarios: A controlled high-resolution study. ArXiv, abs/2502.19828, 2025.arxiv.org/abs/2502.19828
  2. [2]Serra Aksoy. Systematic evaluation of attribution methods: Eliminating threshold bias and revealing method-dependent performance patterns. ArXiv, abs/2509.03176, 2025.arxiv.org/abs/2509.03176
  3. [3]Shirzad Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep ViT features as dense visual descriptors. ArXiv, abs/2112.05814, 2021.arxiv.org/abs/2112.05814
  4. [4]Omri Avrahami, Amir Hertz, Yael Vinker, Moab Arar, Shlomi Fruchter, Ohad Fried, D. Cohen-Or, and D. Lischinski. The chosen one: Consistent characters in text-to-image diffusion models. ACM SIGGRAPH 2024 Conference Papers, 2023.DOI
  5. [5]Patryk Bartkowiak, Lennart Petersen, B. Kotrys, D. Michels, Soren Pirk, and Wojciech Palubicki. MaSC: A masked similarity metric for evaluating concept-driven generation. ArXiv, abs/2605.22469, 2026.arxiv.org/abs/2605.22469
  6. [6]Shengqu Cai, E. Chan, Yunzhi Zhang, Leonidas J. Guibas, Jiajun Wu, and Gordon Wetzstein. Diffusion self-distillation for zero-shot customized image generation. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18434–18443, 2024.arxiv.org/abs/2411.18616
  7. [7]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, J. Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9630–9640, 2021.arxiv.org/abs/2104.14294
  8. [8]Weifeng Chen, Tao Gu, Yuhao Xu, and Chengcai Chen. Magic clothing: Controllable garment-driven image synthesis. Proceedings of the 32nd ACM International Conference on Multimedia, 2024.arxiv.org/abs/2404.09512
  9. [9]Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. Why is spatial reasoning hard for VLMs? An attention mechanism perspective on focus areas. ArXiv, abs/2503.01773, 2025.arxiv.org/abs/2503.01773
  10. [10]Wenyue Chong. Towards robust semantic correspondence: A benchmark and insights. ArXiv, abs/2508.00272, 2025.arxiv.org/abs/2508.00272
  11. [11]C. Cuttano, Gabriele Trivigno, Carlo Masone, and Stefan Roth. Marco: Navigating the unseen space of semantic correspondence. ArXiv, abs/2604.18267, 2026.arxiv.org/abs/2604.18267
  12. [12]Anurag Das, Adrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas, B. Schiele, Georgios Tzimiropoulos, and Brais Martínez. More images, more problems? A controlled analysis of VLM failure modes. ArXiv, abs/2601.07812, 2026.arxiv.org/abs/2601.07812
  13. [13]Olaf Dunkel, Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, C. Theobalt, and A. Kotlyarovskiy. SoCo: Benchmarking semantic object correspondence in vision foundation models. ArXiv, abs/2605.31597, 2026.arxiv.org/abs/2605.31597
  14. [14]A. Eldesokey, Aleksandar Cvejic, Bernard Ghanem, and Peter Wonka. Mind-the-glitch: Visual correspondence for detecting inconsistencies in subject-driven generation. ArXiv, abs/2509.21989, 2025.DOI
  15. [15]Stephanie Fu, Netanel Y. Tamir, Shobhitha Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. DreamSim: Learning new dimensions of human visual similarity using synthetic data. ArXiv, abs/2306.09344, 2023.
  16. [16]Frank Fundel, Johannes Schusterbauer, Vincent Tao Hu, and Bjorn Ommer. Distillation of diffusion features for semantic correspondence. 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6762–6774, 2024.arxiv.org/abs/2412.03512
  17. [17]Rinon Gal, Yuval Alaluf, Y. Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and D. Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. ArXiv, abs/2208.01618, 2022.arxiv.org/abs/2208.01618
  18. [18]Sooyeon Go, Kyungmook Choi, Minjung Shin, and Youngjung Uh. Eye-for-an-eye: Appearance transfer with dense semantic correspondence in diffusion models. 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4641–4650, 2024.
  19. [19]Shaozhe Hao, K. Han, Shihao Zhao, and K. K. Wong. ViCo: Plug-and-play visual condition for personalized text-to-image generation. In arXiv, 2023.arxiv.org/abs/2306.00971
  20. [20]Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam N. Isack, Abhishek Kar, Andrea Tagliasacchi, and K. M. Yi. Unsupervised semantic correspondence using stable diffusion. ArXiv, abs/2305.15581, 2023.arxiv.org/abs/2305.15581
  21. [21]Hang Hua, Jing Shi, Kushal Kafle, Simon Jenni, Daoan Zhang, John P. Collomosse, Scott Cohen, and Jiebo Luo. FineMatch: Aspect-based fine-grained image and text mismatch detection and correction. In European Conference on Computer Vision, pp. 474–491, 2024.arxiv.org/abs/2404.14715
  22. [22]Hang Hua, Ziyun Zeng, Yizhi Song, Yunlong Tang, Liu He, Daniel G. Aliaga, Wei Xiong, and Jiebo Luo. MMIG-Bench: Towards comprehensive and explainable evaluation of multi-modal image generation models. ArXiv, abs/2505.19415, 2025.arxiv.org/abs/2505.19415
  23. [23]Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2I-CompBench: A comprehensive benchmark for open-world compositional text-to-image generation. ArXiv, abs/2307.06350, 2023a.arxiv.org/abs/2307.06350
  24. [24]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qin Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21807–21818, 2023b.arxiv.org/abs/2311.17982
  25. [25]A. Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, A. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3992–4003, 2023.arxiv.org/abs/2304.02643
  26. [26]Max W.F. Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. VIEScore: Towards explainable metrics for conditional image synthesis evaluation. ArXiv, abs/2312.14867, 2023a.arxiv.org/abs/2312.14867
  27. [27]Max W.F. Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. ImagenHub: Standardizing the evaluation of conditional image generation models. ArXiv, abs/2310.01596, 2023b.arxiv.org/abs/2310.01596
  28. [28]Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1931–1941, 2022.arxiv.org/abs/2212.04488
  29. [29]Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. PhotoMaker: Customizing realistic human photos via stacked ID embedding. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8640–8650, 2023.arxiv.org/abs/2312.04461
  30. [30]Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and D. Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pp. 366–384, 2024.arxiv.org/abs/2404.01291
  31. [31]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyu Li, Jianwei Yang, Hang Su, Jun-Juan Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. ArXiv, abs/2303.05499, 2023.arxiv.org/abs/2303.05499
  32. [32]Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. ArXiv, abs/2305.14334, 2023.arxiv.org/abs/2305.14334
  33. [33]Jiancang Ma, Junhao Liang, Chen Chen, and H. Lu. Subject-diffusion: Open domain personalized text- [34] Octave Mariotti, Oisin Mac Aodha, and Hakan Bilen. Improving semantic correspondence with viewpoint-guided spherical maps. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19521–19530, 2023.
  34. [35]Mohammad Hossein Moslemi and Mostafa Milani. Threshold-independent fair matching through score calibration. Proceedings of the Conference on Governance, Understanding and Integration of Data for Effective and Responsible AI, 2024.DOI
  35. [36]Jisu Nam, Gyu­seong Lee, Sunwoo Kim, Ines Hyeonsu Kim, Hyoungwon Cho, Seyeong Kim, and Seungryong Kim. Diffusion model for dense matching. In International Conference on Learning Representations, 2023.arxiv.org/abs/2305.19094
  36. [37]J. Napolitano and Pascal Nguyen. Gram-MMD: A texture-aware metric for image realism assessment. ArXiv, abs/2604.03064, 2026.arxiv.org/abs/2604.03064
  37. [38]Yeongtak Oh, Sangwon Yu, Junsung Park, Han Cheol Moon, J. Mok, and Sungroh Yoon. Contextualized visual personalization in vision-language models. ArXiv, abs/2602.03454, 2026.arxiv.org/abs/2602.03454
  38. [39]M. Oquab, Timothee Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Bal­las, Wojciech Galuba, Russ Howes, Po-Yao (Bernie) Huang, Shang-Wen Li, Ishan Misra, Michael G. Rab­bat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, J. Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. ArXiv, abs/2304.07193, 2023.arxiv.org/abs/2304.07193
  39. [40]Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. DreamBench++: A human-aligned benchmark for personalized image generation. ArXiv, abs/2406.16855, 2024.arxiv.org/abs/2406.16855
  40. [41]Dustin Podell, Zion English, Kyle Lacey, A. Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. ArXiv, abs/2307.01952, 2023.
  41. [42]Weimin Qiu, Jieke Wang, and Meng Tang. Self-cross diffusion guidance for text-to-image generation without test-time fine-tuning. ACM SIGGRAPH 2024 Conference Papers, 2023.arxiv.org/abs/2411.18936
  42. [43]Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674–10685, 2021.arxiv.org/abs/2112.10752
  43. [44]Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Y. Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22500–22510, 2022.arxiv.org/abs/2208.12242
  44. [45]Israel Salazar, Stella Frank, Dan Oneata, Desmond Elliott, and Constanza Fierro. Pathways of visual information flow in vision-language models. In arXiv, 2026.arxiv.org/abs/2607.03358
  45. [46]Chaehun Shin, Jooyoung Choi, Heeseung Kim, and Sungroh Yoon. Large-scale text-to-image model with inpainting is a zero-shot subject-driven image generator. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7986–7996, 2024.arxiv.org/abs/2411.15466
  46. [47]Aviv Slobodkin, Hagai Taitelbaum, Yonatan Bitton, Brian Gordon, Michal Sokolik, N. Guetta, Almog Gueta, Royi Rassin, Itay Laish, Dani Lischinski, and Idan Szpektor. RefVNLI: Towards scalable evaluation of subject-driven text-to-image generation. In Conference on Empirical Methods in Natural Language Processing, pp. 8420–8438, 2025.
  47. [48]Nick Stracke, Stefan Andreas Baumann, Kolja Bauer, Frank Fundel, and Bjorn Ommer. CleanDIFT: Diffusion features without noise. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 117–127, 2024.arxiv.org/abs/2412.03439
  48. [49]Saksham Suri, Matthew Walmer, Kamal Gupta, and Abhinav Shrivastava. LiFT: A surprisingly simple lightweight feature transform for dense ViT descriptors. In European Conference on Computer Vision, pp. 110–128, 2024.arxiv.org/abs/2403.14625
  49. [50]Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. OmniControl: Minimal and universal control for diffusion transformer. 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14940–14950, 2024.arxiv.org/abs/2411.15098
  50. [51]Luming Tang, Menglin Jia, Qianqian Wang, Cheng Peng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. ArXiv, abs/2306.03881, 2023.arxiv.org/abs/2306.03881
  51. [52]Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Y. Atzmon. Training-free consistent text-to-image generation. ACM Transactions on Graphics (TOG), 43:1–18, 2024.arxiv.org/abs/2402.03286
  52. [53]Khoi Trinh, Jay C. Rothenberger, Scott Seidenberger, Dimitrios I. Diochnos, and Anindya Maiti. CLIPIPS: A personalized metric for AI-generated image similarity. ArXiv, abs/2604.01234, 2026.arxiv.org/abs/2604.01234
  53. [54]Michael Tschannen, Alexey Gritsenko, Xiao Wang, M. Naeem, Ibrahim M. Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, A. Steiner, and Xiao-Qi Zhai. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. ArXiv, abs/2502.14786, 2025.arxiv.org/abs/2502.14786
  54. [55]Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1921–1930, 2022.arxiv.org/abs/2211.12572
  55. [56]Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and W. Zuo. ELITE: Encoding visual concepts into textual embeddings for customized text-to-image generation. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15897–15907, 2023.arxiv.org/abs/2302.13848
  56. [57]Shaojin Wu, Mengqi Huang, Yufeng Cheng, Wenxu Wu, Jiahe Tian, Yiming Luo, Fei Ding, and Qian He. USQ: Unified style and subject-driven generation via disentangled and reward learning. ArXiv, abs/2508.18966, 2025.arxiv.org/abs/2508.18966
  57. [58]Monika Wysoczańska, Oriane Simeoni, Michael Ramamonjisoa, Andrei Bursuc, Tomasz Trzcinski, and Patrick Pérez. CLIP-DINOiser: Teaching CLIP a few DINO tricks. In European Conference on Computer Vision, pp. 320–337, 2023.
  58. [59]Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 586–595, 2018.arxiv.org/abs/1801.03924
  59. [60]Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming Yang. A tale of two features: Stable diffusion complements DINO for zero-shot semantic correspondence. ArXiv, abs/2305.15347, 2023a.arxiv.org/abs/2305.15347
  60. [61]Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming Hsuan Yang. Telling left from right: Identifying geometry-aware semantic correspondence. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3076–3085, 2023b.arxiv.org/abs/2311.17034
  61. [62]Kai Zhang, Xinghui Li, Jingyi Lu, and Kai Han. Semantic correspondence: Unified benchmarking and a strong baseline. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48:3911–3930, 2025a.DOI
  62. [63]Yuxuan Zhang, Yiru Yuan, Yiren Song, Haofan Wang, and Jiaming Liu. EasyControl: Adding efficient and flexible control for diffusion transformer. 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 19513–19524, 2025b.arxiv.org/abs/2503.07027
  63. [64]Liyuan Zhu, Shengqu Cai, Shengyu Huang, Gordon Wetzstein, Naji Khosravan, and Iro Armeni. ReStyle3D: Scene-level appearance transfer with semantic correspondences. ArXiv, abs/2502.10377, 2025.arxiv.org/abs/2502.10377

Paper details

vlnc:W_jsaysp5t
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents