Introduction
In recent years, Multimodal Large Language Models (MLLMs) [1 ] , [2 ] , [3 ] have undergone rapid advancement, with their performance increasingly governed by the quality of multimodal encoders. As a crucial component, vision encoders map images into compact visual representations that facilitate visual understanding and reasoning in downstream tasks [4 ] , [5 ] , [6 ] , [7 ] , [8 ] , [9 ] , [10 ] . Most encoders leverage contrastive pretraining, benefiting from strong zero-shot generalization capabilities and effective cross-modal alignment [11 ] . Nonetheless, the self-supervised text-image alignment achieved by these models remains relatively coarse [12 ] . Consequently, the resulting representations are often suboptimal, posing a major bottleneck for broader deployment [13 ] .
A key challenge in representation learning is that the goal of a good representation is inherently abstract. The seminal review by Bengio et al. suggests that effective representations are linked to their capacity to capture the underlying factors of variation within data [14 ] . In addition, LeCun, Bengio, and Hinton’s Nature paper demonstrates that deep neural networks learn hierarchical representations: lower layers encode local structures while higher layers capture object-level abstractions [15 ] . This hierarchy exposes a fundamental tension between semantic abstraction and perceptual preservation. Representations must be invariant enough to enable category-level discrimination, while still retaining local details necessary for fine-grained understanding. This phenomenon is also observed in Kornblith, Shlens, and Quoc V. Le’s work, which finds that ImageNet-trained features are less general than previously thought, especially for fine-grained tasks {citep}[16 ] `. Building on these insights, recent foundation models increasingly advocate that a robust visual representation should simultaneously exhibit two complementary properties. The first is Discriminative Ability (D-Ability), which assesses whether the representation can identify the primary category or detect the presence of a target concept. The second is Fine-grained Perceptual Ability (P-Ability ), which evaluates whether the representation retains pose, orientation, local attributes, and subtle visual differences.
Existing approaches employ distinct optimization strategies for these two capabilities. (1) Various forms of contrastive learning [17 , 18 , 19 , 20 , 21 ] encourage models to suppress local variations and low-level noise, thereby promoting stable semantic representations and strengthening D-Ability . As depicted in Figure 1 (b), this process tends to cluster learned representations into compact semantic groups. (2) With the emergence of generative models, recent research leverages generative feedback for further enhancement [22 , 23 , 24 , 25 ] . In this setting, generative models reconstruct images conditioned on visual features, where accurate reconstruction implies that the visual representation preserves detailed information from the input. This approach defines a reconstruction loss via a mean squared error (MSE) objective, thereby enhancing P-Ability . As shown in Figure 1 (a), this objective produces a sparser representation space that maintains fine-grained variations across samples.
Figure 1. Motivation of our method. (a) Reconstructive learning improves P-Ability by preserving a sparser representation space. (b) Contrastive learning improves D-Ability by organizing representations into compact semantic clusters. (c) The naive method searches for a balanced representation through heuristic weight tuning between contrastive learning and reconstructive learning, but the proper weights are difficult to determine. (d) The target balanced representation, which is directly optimized by our method without heuristic weight tuning.
However, these two approaches typically optimize only one objective at a time, resulting in a trade-off between D-Ability and P-Ability . As illustrated in Figure 1 (d), an ideal balanced representation should be sparse while maintaining clear semantic separability. This leads to a natural question:
Is there an intuitive optimization strategy that embraces both discriminative and fine-grained perceptual ability?
In this paper, we answer this question affirmatively. We first motivate our approach by proposing a straightforward solution that linearly combines these two seemingly contradictory objectives within one framework, as depicted in Figure 1 (c). However, both theoretical analysis (§ 5.1.1 ) and empirical results (§ 5.1.2 ) indicate that this naive solution cannot directly resolve the conflict of the two goals.
To circumvent the need for explicit balancing, we propose a unified supervision strategy within generative reconstruction. The key insight is that reconstruction intrinsically promotes P-Ability , while comparison-based supervision fundamentally underpins the strong D-Ability observed in contrastive learning. Rather than introducing an additional contrastive branch, we seamlessly embed comparison signals into the reconstruction process. Specifically, for each image, we sample a generative timestep alongside its corresponding noisy observation, and compare different one-step denoising results under varying conditioning. (1) When the condition is derived from the same image , such as an augmented view, the reconstruction should remain highly consistent. These results are treated as positive examples . (2) When the condition originates from a different image , the reconstruction should diverge. These results serve as negative examples . By integrating comparisons directly into the generative process, we avoid directly optimizing two potentially conflicting objectives.
We instantiate this comparison structure using the area under the ROC curve (AUC), as depicted in Figure 3 (b). Based on the definition of positive/negative examples shown above, optimizing the AUC objective encourages the score of a randomly chosen positive to exceed that of a randomly chosen negative [26 ] . This formulation precisely enforces the desired ordering within the generated image space at an arbitrary scale (since the comparison timestep is randomly chosen each time). On top of this, we introduce AUC Generative Reconstruction (AUCGR ) as a tractable training objective. Theoretical analysis further demonstrates that this AUC-driven comparison structure supports both D-Ability and P-Ability .
Figure 3. An overview of our method. (a) A generic comparison principle for generative reconstruction. (b) AUC-oriented instantiations, including AUCGR and pAUCGR. (c) Reconstruction learning pipeline. Our method serves as a plug-and-play optimization module that replaces the original optimization strategy in existing reconstruction learning methods, thereby yielding balanced enhanced visual representations.
AUCGR still encounters a practical challenge in mini-batch training: the pseudo-negative set generated above may include noisy negatives that are semantically similar to the anchor, especially in long-tailed data distributions. Blindly telling apart those noisy examples from positives could introduce additional bias into supervision. To address this, we propose Partial AUC Generative Reconstruction (pAUCGR ), yielding a partial-AUC interpretation that preserves uniform generalization guarantees.
Finally, we conduct comprehensive experiments across diverse benchmark datasets and multimodal large language models to verify the effectiveness of our proposed method.
This work extends our CVPR 2026 paper [27 ] , where we initially proposed a unified contrastive reconstruction objective to resolve the gradient conflict between contrastive learning and generative reconstruction. In this version, we incorporate systematic enhancements in experimental settings, methodology, and empirical evaluation, resulting in a more general and robust generative reconstruction framework for visual representation learning. In summary, the new contributions are as follows:
New Methods. This version extends the contrastive reconstruction objective into a generic comparison-based theoretical framework (see § 6 ). As a fundamental principle, it can be instantiated by different comparison strategies. Moreover, it serves as a plug-and-play optimization module that enhances existing generative reconstruction methods and produces more balanced visual representations. We develop an AUC-oriented instantiation of the comparison strategy in the unified paradigm, termed AUC Generative Reconstruction (AUCGR). To mitigate biased supervision arising from noisy negatives, we further introduce Partial AUC Generative Reconstruction (pAUCGR), which restricts negative comparisons to reliable regions.
New Theory. We further provide theoretical analyses for the effectiveness of AUCGR and pAUCGR, establish corresponding generalization bounds (see § 7 ), and show that the conference version is naturally covered as the logistic-surrogate case of AUCGR.
New Empirical Results. We conduct a broad range of new empirical studies to evaluate the effectiveness of our method. Specifically, we introduce more generative reconstruction backbones, extend the evaluation to over 30 benchmarks (including zero-shot clustering, zero-shot classification, zero-shot retrieval, fine-grained perception, and MLLM evaluation), and perform more fine-grained ablation studies and visualization analyses.
Miscellaneous Contents. We also improve some existing contents to make our work more complete, including the abstract, introduction, review of prior arts (§ 2 and § 3 ), methodology and theory (Sec. 4, § 6 and § 7 ), and experiments (§ 8 ).
Related Work
Vision Encoders
Vision encoders serve as the visual backbone of MLLMs [1 , 28 ] , transforming raw images into high-level feature representations. Among various architectures [11 , 29 , 30 , 31 , 32 , 33 ] , CLIP [11 ] is one of the most widely adopted models. It also provides transferable representations for a wide range of tasks, including detection [34 , 35 , 36 , 37 , 38 ] , segmentation [39 , 40 ] , video understanding [41 , 42 ] , and generation [43 , 44 , 45 , 46 , 47 , 48 , 49 ] . Reliability-aware visual learning has also received increasing attention. SELongVLM [50 ] effectively suppresses task-irrelevant visual content through self-corrective clip selection, while [51 ] explicitly model knowledge uncertainty via Monte Carlo dropout to reduce unreliable supervision in object detection distillation. Overall, recent studies have focused on enhancing vision encoders’ understanding capability to improve downstream performance. Some primarily strengthen its discriminative ability [17 , 18 , 52 ] , while others improve its fine-grained perceptual ability [22 , 23 , 24 , 25 , 53 ] . However, an ideal visual encoder should achieve a balance between both aspects. To bridge this gap, this paper aims to develop an enhanced vision encoder with balanced understanding abilities, providing a stronger foundation for MLLMs.
Generative Reconstruction Learning
Generative modeling has recently become a powerful paradigm for improving visual representations. Early studies [54 , 55 , 56 ] use generative models as data augmenters to expand training data and boost generalization. More recent works [22 , 23 , 24 , 25 ] add self-supervised reconstruction objectives to diffusion frameworks, enabling models to learn richer spatial details and fine-grained semantics. Among them, DIVA [23 ] introduces diffusion-based visual feedback conditioned on CLIP embeddings, which refines internal representations through generative reconstruction. GenHancer [24 ] extends this reconstruction process to discrete latent spaces, while un2 ^2 2 CLIP [25 ] resolves the mismatch between CLIP’s pretrained feature space and generative model latents. However, these methods often overemphasize fine-grained reconstruction while neglecting discriminative learning, leading to suboptimal representations. In contrast, our work simultaneously strengthens discriminative ability and fine-grained perceptual ability, producing a more effective vision representation.
Vision-understanding Benchmarks
The benchmarks can be broadly divided into two camps. The first focuses on assessing discriminative ability. This ability can be evaluated through zero-shot clustering, zero-shot classification, and zero-shot retrieval. Zero-shot clustering examines whether image representations form separable semantic structures on datasets such as ImageNet [57 ] , Caltech-101 [58 ] , and DTD [59 ] . Zero-shot classification [11 ] further tests whether the learned visual features support class-level recognition. Zero-shot retrieval [60 , 61 ] evaluates whether visual representations can distinguish correct image-text pairs from many candidates. The second group emphasizes fine-grained perceptual ability. MMVP [53 ] introduces a challenging benchmark with nine visual patterns, CV-Bench [62 ] extends evaluation to spatial relations, counting, and 3D understanding, and NaturalBench [63 ] adds natural adversarial examples to assess perceptual robustness. Together, these benchmarks provide a comprehensive evaluation of visual representations from both discriminative and fine-grained perceptual perspectives.
AUC Optimization
AUC optimization develops from classical machine learning to modern deep learning. Early work clarifies the distinct motivation for this line of research by showing that maximizing AUC is generally inconsistent with minimizing classification error [64 ] . This observation motivates a series of optimization methods for linear models, including logistic regression [65 ] and support vector machines [66 , 67 ] . Subsequent studies extend AUC maximization to online and stochastic settings [68 , 69 , 70 , 71 ] , while generalization and consistency analyses [72 ] , [73 ] , [74 ] , [75 ] , [76 ] provide the corresponding theoretical foundations. With the rise of deep models, AUC-oriented training further extends to deep neural networks [77 ] , and a range of variants such as partial AUC, compositional AUC, and long-tailed or class-balanced extensions gradually enriches this line of research [78 ] , [79 ] , [80 ] . More recently, AUC optimization also extends beyond instance-level prediction to the pixel level [81 ] , [82 ] , [83 ] . In practice, AUC-based optimization demonstrates effectiveness in recommendation systems [84 ] , [85 ] , disease prediction [86 ] , [87 ] , dense-level segmentation [81 ] , [82 ] , and adversarial training [88 ] , [89 ] , [90 ] . In this paper, we introduce AUC optimization into generative reconstruction for visual representation enhancement.
Preliminaries
Notation
Let ϵ θ \epsilon_\theta ϵ θ denote the generative model, f ϕ f_\phi f ϕ denote the visual encoder, and h ω h_\omega h ω denote the projection module that maps visual features into the generative space. The training dataset is defined as D = { x i ∈ R H × W × 3 } i = 1 n \mathcal{D} = \{x^i \in\mathbb{R}^{H \times W \times3}\}_{i=1}^{n} D = { x i ∈ R H × W × 3 } i = 1 n , where H H H and W W W denote the image height and width. A Variational autoencoder (VAE) encodes each image x x x into a latent x ~ \tilde{x} x ~ . We denote the visual feature as z = f ϕ ( x ) z = f_\phi(x) z = f ϕ ( x ) , the condition as c = h ω ( z ) c = h_\omega(z) c = h ω ( z ) , and the model prediction as ϵ ^ θ = ϵ θ ( x ~ t , c , t ) \hat{\epsilon}_\theta= \epsilon_\theta(\tilde{x}_t, c, t) ϵ ^ θ = ϵ θ ( x ~ t , c , t ) at timestep t t t , with the ground-truth ϵ t g t \epsilon_t^{\mathrm{gt}} ϵ t gt . For clarity, we include a table of symbol definitions in Sec. A.
Generative Models
We review two typical generative models: diffusion models and flow-based models.
Diffusion Models. Diffusion models [43 ] , [91 ] learn a generative process by reversing a gradual noising procedure over a finite number of timesteps. Given a latent representation x ~ 0 \tilde{x}_0 x ~ 0 encoded by a VAE, the model progressively adds Gaussian noise to obtain x ~ t \tilde{x}_t x ~ t at timestep t t t . A noise prediction network ϵ θ ( x ~ t , c , t ) \epsilon_\theta(\tilde{x}_t, c, t) ϵ θ ( x ~ t , c , t ) parameterizes the reverse process and predicts the ground-truth noise ϵ t g t \epsilon_t^{\mathrm{gt}} ϵ t gt added at timestep t t t . Training diffusion models amounts to minimizing the following denoising objective:
L D M = E x ~ 0 , ϵ g t , t [ ∥ ϵ t g t − ϵ ^ θ ∥ 2 2 ] , (1) \mathcal{L}_{\mathrm{DM}} = \mathbb{E}_{\tilde{x}_0,\epsilon^{\mathrm{gt}},t}\left[\left\lVert\epsilon_t^{\mathrm{gt}}-\hat{\epsilon}_\theta\right\rVert_2^2\right], \tag*{(1)} L DM = E x ~ 0 , ϵ gt , t [ ϵ t gt − ϵ ^ θ 2 2 ] , (1)
where t t t is sampled from a predefined timestep distribution. This objective promotes accurate noise prediction at each diffusion step and implicitly defines the reverse generative dynamics. The model achieves conditional generation by injecting the condition c c c into the noise prediction network.
Flow-based Models. Flow-based models [92 ] , [93 ] learn a continuous transformation that maps samples from a base distribution to the target data distribution through deterministic dynamics. In the conditional setting, this transformation is often written as an ordinary differential equation:
d x ~ t d t = ϵ θ ( x ~ t , c , t ) , (2) \frac{d\tilde{x}_t}{dt} = \epsilon_\theta(\tilde{x}_t,c,t), \tag*{(2)} d t d x ~ t = ϵ θ ( x ~ t , c , t ) , (2)
where ϵ θ ( x ~ t , c , t ) \epsilon_\theta(\tilde{x}_t,c,t) ϵ θ ( x ~ t , c , t ) denotes a time-dependent conditional velocity function. Rectified Flow uses a simpler formulation by constructing linear interpolations between samples from the base distribution and the data distribution. Specifically,
let x ~ 0 ∼ N ( 0 , I ) \tilde{x}_0 \sim\mathcal{N}(0,\mathbf{I}) x ~ 0 ∼ N ( 0 , I ) and x ~ 1 ∼ p d a t a \tilde{x}_1 \sim p_{\mathrm{data}} x ~ 1 ∼ p data , and define x ~ t = t x ~ 1 + ( 1 − t ) x ~ 0 \tilde{x}_t = t\tilde{x}_1 + (1-t)\tilde{x}_0 x ~ t = t x ~ 1 + ( 1 − t ) x ~ 0 . The conditional velocity field is learned by minimizing the regression objective:
L F M = E t , x ~ 0 , x ~ 1 [ ∥ ϵ t g t − ϵ ^ θ ∥ 2 2 ] , (3) \mathcal{L}_{\mathrm{FM}} = \mathbb{E}_{t,\tilde{x}_0,\tilde{x}_1}\left[\left\lVert\epsilon_t^{\mathrm{gt}}-\hat{\epsilon}_\theta\right\rVert_2^2\right], \tag*{(3)} L FM = E t , x ~ 0 , x ~ 1 [ ϵ t gt − ϵ ^ θ 2 2 ] , (3)
where ϵ t g t = x ~ 1 − x ~ 0 \epsilon_t^{\mathrm{gt}} = \tilde{x}_1-\tilde{x}_0 ϵ t gt = x ~ 1 − x ~ 0 . It directly supervises the instantaneous velocity along straight-line paths conditioned on c c c , defining a deterministic conditional generative process without requiring likelihood estimation or numerical solvers.
Although diffusion and flow-based models define different generative processes, their training objectives share the same basic form. Both supervise the model prediction to match a target signal. They provide a reliable supervision signal for generative reconstruction.
AUC Optimization
Area under the Receiver Operating Characteristic Curve (AUC) is a ranking metric widely used in binary classification [26 ] . It measures the probability that a scoring function assigns a higher score to a positive sample than to a negative sample, namely
AUC ( h θ ) = P ( h θ ( x + ) > h θ ( x − ) ∣ y + = 1 , y − = 0 ) , (4) \operatorname{AUC}(h_\theta) = \mathbb{P}\left(h_\theta(x^+) > h_\theta(x^-) \mid y^+ = 1,\ y^- = 0\right), \tag*{(4)} AUC ( h θ ) = P ( h θ ( x + ) > h θ ( x − ) ∣ y + = 1 , y − = 0 ) , (4)
where ( x + , y + ) (x^+,y^+) ( x + , y + ) and ( x − , y − ) (x^-,y^-) ( x − , y − ) denote positive and negative samples, respectively. A larger AUC indicates a stronger ability to rank positives ahead of negatives.
For a finite dataset, AUC maximization is commonly implemented through an empirical pairwise surrogate objective [78 ] , [94 ] , [95 ] . Let { x i + } i = 1 n + \{x_i^+\}_{i=1}^{n^+} { x i + } i = 1 n + and { x j − } j = 1 n − \{x_j^-\}_{j=1}^{n^-} { x j − } j = 1 n − denote the positive and negative sets. The corresponding empirical objective can be written as
L ^ A U C ( h θ ) = 1 n + n − ∑ i = 1 n + ∑ j = 1 n − ℓ ( h θ ( x i + ) − h θ ( x j − ) ) , (5) \widehat{\mathcal{L}}_{\mathrm{AUC}}(h_\theta) = \frac{1}{n^+n^-}\sum_{i=1}^{n^+}\sum_{j=1}^{n^-}\ell\left(h_\theta(x_i^+)-h_\theta(x_j^-)\right), \tag*{(5)} L AUC ( h θ ) = n + n − 1 i = 1 ∑ n + j = 1 ∑ n − ℓ ( h θ ( x i + ) − h θ ( x j − ) ) , (5)
where ℓ \ell ℓ denotes a differentiable surrogate loss [94 ] . Minimizing this objective reduces pairwise ordering violations and therefore improves the induced AUC.
This formulation shows that AUC optimization depends on the relative order between positive and negative scores rather than their absolute calibration. Such a property is particularly valuable under imbalanced or long-tailed data distributions [96 ] , [97 ] , [98 ] , [99 ] , where comparison-based objectives can alleviate the bias caused by skewed class frequencies. This perspective provides the foundation for our later AUC instantiation of comparison-based generative reconstruction.
The Task of Visual Representation Learning
Visual representation learning aims to learn an encoder f ϕ : X → Z f_\phi:\mathcal{X}\to\mathcal{Z} f ϕ : X → Z that maps an image x x x into a visual representation z z z . The goal is to obtain a representation with rich understanding capacity, typically characterized by two key properties: strong Discriminative Ability (D-Ability) and Fine-grained Perceptual Ability (P-Ability) [100 ] , [101 ] .
Property 1 (D-Ability): Discriminative ability describes a model’s ability to accurately and robustly distinguish between different visual categories. A representation with strong discriminative power can effectively separate distinct
classes in the feature space, keeping samples from the same class tightly clustered while pushing samples from different classes farther apart [11 , 102 ] . Mathematically, for a mini-batch with class set Y \mathcal{Y} Y and feature set Z y = { z i ∣ y i = y } \mathcal{Z}_y = \{z_i \mid y_i = y\} Z y = { z i ∣ y i = y } , we define the class center as μ y = 1 ∣ Z y ∣ ∑ z ∈ Z y z \mu_y = \frac{1}{|\mathcal{Z}_y|} \sum_{z \in\mathcal{Z}_y} z μ y = ∣ Z y ∣ 1 ∑ z ∈ Z y z . It is formulated as
min S i n n e r , max S i n t e r , (6) \min S_{\mathrm{inner}},\quad\max S_{\mathrm{inter}}, \tag*{(6)} min S inner , max S inter , (6)
S i n n e r = 1 ∣ Y ∣ ∑ y ∈ Y 1 ∣ Z y ∣ ∑ z ∈ Z y ∥ z − μ y ∥ 2 2 , (7) S_{\mathrm{inner}} = \frac{1}{|\mathcal{Y}|} \sum_{y \in\mathcal{Y}} \frac{1}{|\mathcal{Z}_y|} \sum_{z \in\mathcal{Z}_y} \lVert z-\mu_y\rVert_2^2, \tag*{(7)} S inner = ∣ Y ∣ 1 y ∈ Y ∑ ∣ Z y ∣ 1 z ∈ Z y ∑ ∥ z − μ y ∥ 2 2 , (7)
S i n t e r = 1 ∣ Y ∣ ( ∣ Y ∣ − 1 ) ∑ y , y ′ ∈ Y , y ≠ y ′ ∥ μ y − μ y ′ ∥ 2 2 . (8) S_{\mathrm{inter}} = \frac{1}{|\mathcal{Y}|(|\mathcal{Y}|-1)} \sum_{y,y' \in\mathcal{Y},y\ne y'} \lVert\mu_y-\mu_{y'}\rVert_2^2. \tag*{(8)} S inter = ∣ Y ∣ ( ∣ Y ∣ − 1 ) 1 y , y ′ ∈ Y , y = y ′ ∑ ∥ μ y − μ y ′ ∥ 2 2 . (8)
Property 2 (P-Ability): Fine-grained perceptual ability reflects the model’s capacity to capture fine-grained visual details, enabling a comprehensive understanding of object attributes, spatial relations, and subtle semantic cues within an image. Following previous studies [22 , 23 , 24 , 25 ] , this perception is usually evaluated through reconstruction consistency. When a generative model can accurately reconstruct the original image from its learned representation, the representation is considered sufficiently rich. It is formulated as
min E t [ ∥ ϵ θ ( x ~ t , h ω ( f ϕ ( x ) ) , t ) − ϵ t g t ∥ 2 2 ] , (9) \min\mathbb{E}_t\left[\left\lVert\epsilon_\theta(\tilde{x}_t,h_\omega(f_\phi(x)),t)-\epsilon_t^{\mathrm{gt}}\right\rVert_2^2\right], \tag*{(9)} min E t [ ϵ θ ( x ~ t , h ω ( f ϕ ( x )) , t ) − ϵ t gt 2 2 ] , (9)
where t t t denotes the timestep, f ϕ f_\phi f ϕ denotes the CLIP visual encoder, and h ω h_\omega h ω denotes the projector.
A Naive Joint Optimization Method
In this section, we propose a naive joint optimization method that aims to pursue the two properties in § 3.4 . We focus on self-supervised training, since dense supervision is expensive and does not scale well to broad data sources [103 , 104 , 105 , 106 , 107 ] . We consider a classical and widely used visual representation framework, namely CLIP [11 , 108 , 109 ] , as the foundation of our analysis.
Linear Combination of Two Objectives
We start from two widely adopted self-supervised signals that naturally target the two properties.
D-Ability: As shown in Figure 1 (b), mainstream approaches commonly use a contrastive loss that pulls same-class features closer and pushes different-class features apart [11 ] . Specifically, for a mini-batch B \mathcal{B} B with cosine similarity sim ( z i , z j ) = z i T z j ∥ z i ∥ ∥ z j ∥ \operatorname{sim}(z_i,z_j)=\frac{z_i^\mathsf{T}z_j}{\lVert z_i\rVert\lVert z_j\rVert} sim ( z i , z j ) = ∥ z i ∥ ∥ z j ∥ z i T z j , temperature τ > 0 \tau>0 τ > 0 , and the positive set P ( i ) = { j ∈ B ∖ { i } ∣ y j = y i } \mathcal{P}(i)=\{j\in\mathcal{B}\setminus\{i\}\mid y_j=y_i\} P ( i ) = { j ∈ B ∖ { i } ∣ y j = y i } , a InfoNCE-style loss [110 ] is
L c o n = 1 ∣ B ∣ ∑ i ∈ B − log ∑ j ∈ P ( i ) exp ( sim ( z i , z j ) / τ ) ∑ k ∈ B ∖ { i } exp ( sim ( z i , z k ) / τ ) , (10) \mathcal{L}_{\mathrm{con}}=\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}-\log\frac{\sum_{j\in\mathcal{P}(i)}\exp(\operatorname{sim}(z_i,z_j)/\tau)}{\sum_{k\in\mathcal{B}\setminus\{i\}}\exp(\operatorname{sim}(z_i,z_k)/\tau)}, \tag*{(10)} L con = ∣ B ∣ 1 i ∈ B ∑ − log ∑ k ∈ B ∖ { i } exp ( sim ( z i , z k ) / τ ) ∑ j ∈ P ( i ) exp ( sim ( z i , z j ) / τ ) , (10)
which decreases S i n n e r S_{\mathrm{inner}} S inner and increases S i n t e r S_{\mathrm{inter}} S inter in Eq. (6) . Here, the class label y i y_i y i is introduced only to formalize the class-level discriminative objective and is not used in the actual self-supervised training procedure. In practice, P ( i ) \mathcal{P}(i) P ( i ) is constructed using different augmented views of the same image, while the other samples in the mini-batch are treated as negatives. Therefore, no category annotations are required.
A Naive Joint Optimization Method
P-Ability: As shown in Figure 1 (a), the objective is typically realized by enforcing consistency between the reconstructed image and the original one [23 ] , namely
L r e c = E t [ ∥ ϵ θ − ϵ t g t ∥ 2 2 ] , (11) \mathcal{L}_{\mathrm{rec}}=\mathbb{E}_t\left[\left\lVert\epsilon_\theta-\epsilon_t^{\mathrm{gt}}\right\rVert_2^2\right], \tag*{(11)} L rec = E t [ ϵ θ − ϵ t gt 2 2 ] , (11)
which is exactly Eq. (9) .
To obtain richer visual representations, we consider a naive method that combines the two losses described above:
L j o i n t = λ c o n L c o n + λ r e c L r e c , (12) \mathcal{L}_{\mathrm{joint}}=\lambda_{\mathrm{con}}\mathcal{L}_{\mathrm{con}}+\lambda_{\mathrm{rec}}\mathcal{L}_{\mathrm{rec}}, \tag*{(12)} L joint = λ con L con + λ rec L rec , (12)
where λ c o n \lambda_{\mathrm{con}} λ con and λ r e c \lambda_{\mathrm{rec}} λ rec are hyperparameters that balance the two objectives.
Limitation of the Naive Method
Eq. (12) provides a concise way to perform joint optimization. However, directly combining the two objectives does not necessarily produce balanced representations. In practice, the two losses may optimize the shared representation in different directions, leading to interference and suboptimal trade-offs [111 , 112 ] . We first analyze this issue theoretically, and then verify it empirically.
Theoretical Analysis
To analyze the interaction between the two objectives, we first decompose the information contained in the representation. A visual representation includes cues that mainly support semantic discrimination, that mainly support fine-grained perception, and a noise term entangled with both objectives. We model these sources as three independent and orthogonal unit components. Since Z ∈ S d − 1 Z\in\mathbb{S}^{d-1} Z ∈ S d − 1 , we have ∥ Z ∥ 2 2 = 1 \lVert Z\rVert_2^2=1 ∥ Z ∥ 2 2 = 1 . Under the orthogonal decomposition, the squared coefficient of each component corresponds to its contribution to the total representation energy. Therefore, we interpret the squared coefficients as the energy proportions of different information sources. Based on this observation, we define the following signal decomposition model.
Definition 1 (Signal Decomposition Model). Let the feature vector Z ∈ S d − 1 Z\in\mathbb{S}^{d-1} Z ∈ S d − 1 be decomposed into three independent and orthogonal unit components:
Z = α D z D + α P z P + α N z N . (13) Z=\sqrt{\alpha_D}\,z_D+\sqrt{\alpha_P}\,z_P+\sqrt{\alpha_N}\,z_N. \tag*{(13)} Z = α D z D + α P z P + α N z N . (13)
where α D , α P , α N ≥ 0 \alpha_D,\alpha_P,\alpha_N\ge0 α D , α P , α N ≥ 0 and α D + α P + α N = 1 \alpha_D+\alpha_P+\alpha_N=1 α D + α P + α N = 1 .
Here, z D z_D z D denotes the component related to D-Ability , which contains category-separable and semantic-discriminative information. z P z_P z P denotes the component related to P-Ability , which contains fine-grained perceptual and reconstruction-related information. z N z_N z N denotes the noise component that is orthogonal to both signals, which contains task-irrelevant perturbations, sampling noise, or ineffective variations. The coefficients α D \alpha_D α D , α P \alpha_P α P , and α N \alpha_N α N denote their energy proportions in the representation.
Based on Definition 1, we show that the two objectives favor incompatible allocation regimes.
Theorem 1 (Objective Conflict in the Naive Method). *Under Assumptions 2 and 3, consider the representation model in Definition 1. There exist constants c c o n , c r e c ∈ R c_{\mathrm{con}},c_{\mathrm{rec}}\in\mathbb{R} c con , c rec ∈ R and κ P , κ N , κ D , κ N ′ > 0 \kappa_P,\kappa_N,\kappa_D,\kappa'_N>0 κ P , κ N , κ D , κ N ′ > 0 . With these constants, the achievable
envelopes of the two objectives satisfy
inf L c o n ≥ c c o n + κ P α P + κ N α N , inf L r e c ≥ c r e c + κ D ( 1 − α P ) + κ N ′ α N , (14) \begin{aligned}
\inf\mathcal{L}_{\mathrm{con}} &\ge c_{\mathrm{con}}+\kappa_P\alpha_P+\kappa_N\alpha_N,\\
\inf\mathcal{L}_{\mathrm{rec}} &\ge c_{\mathrm{rec}}+\kappa_D(1-\alpha_P)+\kappa'_N\alpha_N, \tag*{(14)}
\end{aligned} inf L con inf L rec ≥ c con + κ P α P + κ N α N , ≥ c rec + κ D ( 1 − α P ) + κ N ′ α N , (14)
where the infimum is taken over all admissible model parameters, optimizing each objective with the shared representation Z Z Z . This result implies that the two objectives favor different allocation regimes:
NOTE: Recall α D + α P + α N = 1 \alpha_D+\alpha_P+\alpha_N=1 α D + α P + α N = 1 , one only needs to include two of the three parameters in the inequality.
The proof is provided in Sec. C. The conflicting inequality in the theorem above suggests that one cannot simultaneously reduce L c o n \mathcal{L}_{\mathrm{con}} L con and L r e c \mathcal{L}_{\mathrm{rec}} L rec . Therefore, jointly optimizing L j o i n t = λ c o n L c o n + λ r e c L r e c \mathcal{L}_{\mathrm{joint}}=\lambda_{\mathrm{con}}\mathcal{L}_{\mathrm{con}}+\lambda_{\mathrm{rec}}\mathcal{L}_{\mathrm{rec}} L joint = λ con L con + λ rec L rec inevitably leads to gradient competition on the shared representation.
Empirical Analysis
Following the Tragic Triad analysis in multi-task learning [113 ] , optimization tends to fail to converge properly when three conditions co-occur: opposite gradient directions , low gradient magnitude similarity , and high multi-task curvature . Therefore, we conduct three experiments to verify whether the naive method indeed falls into this regime.
For all experiments, let z z z denote the shared visual representation. At training step t t t , the update rule for z z z is:
z t + 1 = z t − η ∇ z L j o i n t = z t − η ( λ c o n ∇ z L c o n + λ r e c ∇ z L r e c ) , (15) z^{t+1}=z^t-\eta\nabla_z\mathcal{L}_{\mathrm{joint}}=z^t-\eta\left(\lambda_{\mathrm{con}}\nabla_z\mathcal{L}_{\mathrm{con}}+\lambda_{\mathrm{rec}}\nabla_z\mathcal{L}_{\mathrm{rec}}\right), \tag*{(15)} z t + 1 = z t − η ∇ z L joint = z t − η ( λ con ∇ z L con + λ rec ∇ z L rec ) , (15)
where η \eta η is the learning rate. We compute the per-step gradients of the two objectives:
g c o n = ∇ z L c o n , g r e c = ∇ z L r e c , (16) \mathbf{g}_{\mathrm{con}}=\nabla_z\mathcal{L}_{\mathrm{con}},\qquad\mathbf{g}_{\mathrm{rec}}=\nabla_z\mathcal{L}_{\mathrm{rec}}, \tag*{(16)} g con = ∇ z L con , g rec = ∇ z L rec , (16)
Experiments are conducted on OpenAI CLIP ViT-L/14@224 [11 ] .
Experiment 1: Gradient direction. We first examine whether the two objectives induce inconsistent descent directions. Specifically, to quantify this interaction, we define the cosine similarity between the two gradient directions:
cos ( g c o n , g r e c ) = g c o n T g r e c ∥ g c o n ∥ 2 ∥ g r e c ∥ 2 . (17) \cos(\mathbf{g}_{\mathrm{con}},\mathbf{g}_{\mathrm{rec}})=\frac{\mathbf{g}_{\mathrm{con}}^{\mathsf{T}}\mathbf{g}_{\mathrm{rec}}}{\lVert\mathbf{g}_{\mathrm{con}}\rVert_2\lVert\mathbf{g}_{\mathrm{rec}}\rVert_2}. \tag*{(17)} cos ( g con , g rec ) = ∥ g con ∥ 2 ∥ g rec ∥ 2 g con T g rec . (17)
When cos ( g c o n , g r e c ) > 0 \cos(\mathbf{g}_{\mathrm{con}},\mathbf{g}_{\mathrm{rec}})>0 cos ( g con , g rec ) > 0 , the two objectives cooperate and reinforce each other. However, when cos ( g c o n , g r e c ) < 0 \cos(\mathbf{g}_{\mathrm{con}},\mathbf{g}_{\mathrm{rec}})<0 cos ( g con , g rec ) < 0 , they conflict, leading to destructive interference in the gradient updates.
As shown in Figure 2 (a), 86.3% of the training steps exhibit a negative cosine similarity cos ( g c o n , g r e c ) \cos(\mathbf{g}_{\mathrm{con}},\mathbf{g}_{\mathrm{rec}}) cos ( g con , g rec ) between the gradients of the two losses, indicating that gradient directional conflict is pervasive throughout the optimization process. Moreover, as training progresses, this value becomes almost entirely negative, suggesting that gradient directional conflicts further intensify in the later stages of optimization.
Figure 2. Training results of linearly combining contrastive learning and reconstructive learning (Naive Method) on the OpenAI CLIP ViT-L@224 backbone, which verify that the naive method indeed falls into the Tragic Triad regime.
Experiment 2: Gradient magnitude similarity. We next examine whether the optimization is dominated by one task gradient. To quantify this effect, we measure the gradient magnitude similarity
Φ ( g c o n , g r e c ) = 2 ∥ g c o n ∥ 2 ∥ g r e c ∥ 2 ∥ g c o n ∥ 2 2 + ∥ g r e c ∥ 2 2 , (18) \Phi(\mathbf{g}_{\mathrm{con}},\mathbf{g}_{\mathrm{rec}})=\frac{2\lVert\mathbf{g}_{\mathrm{con}}\rVert_2\lVert\mathbf{g}_{\mathrm{rec}}\rVert_2}{\lVert\mathbf{g}_{\mathrm{con}}\rVert_2^2+\lVert\mathbf{g}_{\mathrm{rec}}\rVert_2^2}, \tag*{(18)} Φ ( g con , g rec ) = ∥ g con ∥ 2 2 + ∥ g rec ∥ 2 2 2 ∥ g con ∥ 2 ∥ g rec ∥ 2 , (18)
where Φ ∈ ( 0 , 1 ] \Phi\in(0,1] Φ ∈ ( 0 , 1 ] , and larger values indicate more balanced gradients. When Φ \Phi Φ is small, the joint update is dominated by one objective, while the other objective contributes little to the actual descent direction.
We find that the naive method exhibits low gradient magnitude similarity in 51.7% of training steps. It means that even when the two objectives are optimized together, their contributions are highly imbalanced for a large portion of training. As a result, the stronger gradient can override the weaker one, causing the shared encoder to drift toward one objective while sacrificing the other. As shown in Figure 2 (b)(Figure 2 c), L c o n \mathcal{L}_{\mathrm{con}} L con dominates the optimization process, while L r e c \mathcal{L}_{\mathrm{rec}} L rec is suppressed and fails to converge.
Experiment 3: Multi-task curvature. We finally examine whether the joint update is taken in a locally sharp region. Following [113 ] , we measure the multi-task curvature along the update direction of L j o i n t \mathcal{L}_{\mathrm{joint}} L joint . A large curvature implies that even a small step along the joint gradient can deviate substantially from the first-order approximation. In that case, the apparent improvement on the dominant objective may be overestimated, while the degradation on the suppressed objective may be underestimated.
We observe that the naive method reaches a multi-task curvature on the order of 10 6 10^6 1 0 6 during training, indicating an extremely sharp landscape around the joint descent direction. This result shows that the naive method not only suffers from directional conflict and gradient imbalance, but also operates in a regime where these two effects are strongly amplified by curvature. Consequently, once the update direction is biased toward one objective and misaligned with the other, the optimization error is rapidly magnified, leading to unstable convergence.
Taken together, Experiments 1–3 verify that the naive method indeed falls into the Tragic Triad regime of multi-task optimization, further demonstrating the existence of severe gradient conflicts between the two objectives.
Learning with Comparisons in Generative Reconstruction
In the previous section, we analyze the limitations of the naive linear weighting strategy. To pursue a unified form of supervision, we inject comparison information into the reconstruction signal, so that generative reconstruction itself can carry relative supervision in the predicted-noise space. A brief overview is provided in Figure 3 .
A Generic Comparison Principle for Generative Reconstruction
As discussed in § 4.1 , standard reconstruction learning enhances P-Ability by enforcing that each image is reconstructed accurately under its own conditioning. However, it does not provide explicit supervision for D-Ability . Importantly, D-Ability is inherently comparison-driven, as it relies on ensuring that semantically compatible signals are closer than incompatible ones in the representation space. Rather than externally merging reconstruction and comparison through a weighted sum as in the naive approach, we embed the comparison mechanism directly into the reconstruction learning process. In this manner, both P-Ability and D-Ability are jointly supervised under a unified objective.
This comparison principle can be operationalized by holding the denoising state fixed in the diffusion process while varying only the reconstruction condition. As illustrated in Figure 3 (a), for a reference image x i x_i x i , we first sample a diffusion timestep t t t and generate the corresponding noised image:
x ~ t , i = α ˉ t x i + 1 − α ˉ t ϵ , ϵ ∼ N ( 0 , I ) , (19) \tilde{x}_{t,i} = \sqrt{\bar{\alpha}_t}x_i + \sqrt{1-\bar{\alpha}_t}\epsilon,\quad\epsilon\sim\mathcal{N}(0,I), \tag*{(19)} x ~ t , i = α ˉ t x i + 1 − α ˉ t ϵ , ϵ ∼ N ( 0 , I ) , (19)
where α ˉ t \bar{\alpha}_t α ˉ t denotes the accumulated noise schedule. Given a conditioning vector c \mathbf{c} c , the denoiser predicts the noise component and conducts one-step denoising from t t t to t − 1 t-1 t − 1 :
x ~ t − 1 , i c = 1 α t ( x ~ t , i − 1 − α t 1 − α ˉ t ϵ θ ( x ~ t , i , c , t ) ) + σ t z , (20) \tilde{x}^{\mathbf{c}}_{t-1,i} = \frac{1}{\sqrt{\alpha_t}}\left(\tilde{x}_{t,i}-\frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\epsilon_\theta(\tilde{x}_{t,i},\mathbf{c},t)\right)+\sigma_t\mathbf{z}, \tag*{(20)} x ~ t − 1 , i c = α t 1 ( x ~ t , i − 1 − α ˉ t 1 − α t ϵ θ ( x ~ t , i , c , t ) ) + σ t z , (20)
where z ∼ N ( 0 , I ) \mathbf{z}\sim\mathcal{N}(0,I) z ∼ N ( 0 , I ) . We keep x ~ t , i \tilde{x}_{t,i} x ~ t , i and t t t fixed, altering only the condition c \mathbf{c} c . The condition derived from x i x_i x i itself establishes the anchor denoising result. A condition from the same image, such as an augmented view, is expected to yield a similar one-step denoising outcome, while a condition from a different image should result in a distinct denoising output. To facilitate meaningful comparison, we note that x ~ t − 1 , i \tilde{x}_{t-1,i} x ~ t − 1 , i is inherently stochastic. To mitigate the effects of randomness, we directly compare the corresponding noise predictions ϵ θ ( x ~ t , i , c , t ) \epsilon_\theta(\tilde{x}_{t,i},\mathbf{c},t) ϵ θ ( x ~ t , i , c , t ) , since:
E [ x ~ t − 1 , i c | x ~ t , i ] ∝ ϵ θ ( x ~ t , i , c , t ) . \mathbb{E}\left[\tilde{x}^{\mathbf{c}}_{t-1,i}\middle|\tilde{x}_{t,i}\right]\propto\epsilon_\theta(\tilde{x}_{t,i},\mathbf{c},t). E [ x ~ t − 1 , i c x ~ t , i ] ∝ ϵ θ ( x ~ t , i , c , t ) .
In other words, leveraging the noise prediction enables us to average out random fluctuations, thereby embedding the comparison directly within the diffusion reconstruction process.
We now formalize this mechanism. For an image x i x_i x i , let
ϵ ^ i = ϵ θ ( x ~ t , i , h ω ( f ϕ ( x i ) ) , t ) (21) \hat{\epsilon}_i=\epsilon_\theta\left(\tilde{x}_{t,i},h_\omega(f_\phi(x_i)),t\right) \tag*{(21)} ϵ ^ i = ϵ θ ( x ~ t , i , h ω ( f ϕ ( x i )) , t ) (21)
denote the anchor predicted noise. We then consider two sets of responses around this anchor in the same predicted-noise space. The positive set P i \mathcal{P}_i P i comprises responses generated by conditions originating from x i x_i x i . The negative set N i \mathcal{N}_i N i consists of responses induced by conditions from images distinct from x i x_i x i . Specifically, P i \mathcal{P}_i P i may include the predicted noise from augmented views of x i x_i x i as well as the ground-truth diffusion noise, whereas N i \mathcal{N}_i N i is constructed using conditions derived from other images.
With this notation, the comparison structure for x i x_i x i is defined as
C i = ( ϵ ^ i , P i , N i , d ) , (22) \mathcal{C}_i = (\hat{\epsilon}_i,\mathcal{P}_i,\mathcal{N}_i,d), \tag*{(22)} C i = ( ϵ ^ i , P i , N i , d ) , (22)
where d d d denotes a generic metric in the predicted-noise space. The corresponding generic comparison objective is written as
L = 1 B ∑ i = 1 B Ψ ( ϵ ^ i , P i , N i ; d ) , (23) \mathcal{L} = \frac{1}{B}\sum_{i=1}^{B}\Psi(\hat{\epsilon}_i,\mathcal{P}_i,\mathcal{N}_i;d), \tag*{(23)} L = B 1 i = 1 ∑ B Ψ ( ϵ ^ i , P i , N i ; d ) , (23)
where B B B is the number of samples and Ψ ( ⋅ ) \Psi(\cdot) Ψ ( ⋅ ) represents a comparison functional induced by d d d . Eq. (23) arranges positive and negative responses with respect to the same anchor in the predicted-noise space. Thus, reconstructive and comparative information are unified within a single objective.
This formulation provides a general template for comparison-based generative reconstruction. Variations in the choice of P i \mathcal{P}_i P i , N i \mathcal{N}_i N i , d d d , and Ψ \Psi Ψ yield different realizations, while consistently adhering to the same foundational principle.
AUC-oriented Instantiation of the Comparison Principle
The general principle in Eq. (23) arranges an anchor and its corresponding positive and negative responses within a unified comparison objective. The remaining challenge is to determine an appropriate metric for quantifying their relative relationships. Building upon § 3.3 , we instantiate this metric using the Area Under the ROC Curve (AUC). As illustrated in Figure 3 (b), the AUC criterion naturally models whether positive responses are consistently ranked above negative ones. This comparison-based structure is directly aligned with the objective of enhancing D-Ability .
In the AUC framework, the score is defined as the similarity between the anchor i i i and a candidate response v v v .
q i ( v ) = d ( ϵ ^ i , v ) , (24) q_i(v) = d(\hat{\epsilon}_i,v), \tag*{(24)} q i ( v ) = d ( ϵ ^ i , v ) , (24)
where v v v denotes a sample drawn from either P i \mathcal{P}_i P i or N i \mathcal{N}_i N i , and d d d is the comparison metric introduced in Eq. (23) . Here, d d d is instantiated as cosine similarity, such that a higher score indicates greater alignment with the anchor. The corresponding true positive rate (TPR) and false positive rate (FPR) at threshold τ \tau τ are given by
TPR ( τ ) = P i , p ∼ P i ( q i ( p ) > τ ) , \operatorname{TPR}(\tau) = \mathbb{P}_{i,p\sim\mathcal{P}_i}\left(q_i(p)>\tau\right), TPR ( τ ) = P i , p ∼ P i ( q i ( p ) > τ ) ,
FPR ( τ ) = P i , n ∼ N i ( q i ( n ) > τ ) . (25) \operatorname{FPR}(\tau) = \mathbb{P}_{i,n\sim\mathcal{N}_i}\left(q_i(n)>\tau\right). \tag*{(25)} FPR ( τ ) = P i , n ∼ N i ( q i ( n ) > τ ) . (25)
Accordingly, the overall AUC is the area under the resulting ROC curve:
AUC = ∫ 0 1 TPR ( FPR − 1 ( u ) ) d u . (26) \operatorname{AUC} = \int_{0}^{1}\operatorname{TPR}\left(\operatorname{FPR}^{-1}(u)\right)\,du. \tag*{(26)} AUC = ∫ 0 1 TPR ( FPR − 1 ( u ) ) d u . (26)
Equivalently, following [26 ] , it can be written as
AUC = P i , p ∼ P i , n ∼ N i ( q i ( p ) > q i ( n ) ) . (27) \operatorname{AUC} = \mathbb{P}_{i,p\sim\mathcal{P}_i,n\sim\mathcal{N}_i}\left(q_i(p)>q_i(n)\right). \tag*{(27)} AUC = P i , p ∼ P i , n ∼ N i ( q i ( p ) > q i ( n ) ) . (27)
However, Eq. (27) is defined through an indicator function and is therefore difficult to optimize directly. By replacing its indicator with a tractable pairwise surrogate, we obtain the loss of AUC Generative Reconstruction (AUCGR) as
L A U C G R = 1 B ∑ i = 1 B 1 ∣ P i ∣ ∣ N i ∣ ∑ p ∈ P i ∑ n ∈ N i ℓ ( q i ( p ) − q i ( n ) ) , (28) \mathcal{L}_{\mathrm{AUCGR}} = \frac{1}{B}\sum_{i=1}^{B}\frac{1}{\lvert\mathcal{P}_i\rvert\lvert\mathcal{N}_i\rvert}\sum_{p\in\mathcal{P}_i}\sum_{n\in\mathcal{N}_i}\ell\left(q_i(p)-q_i(n)\right), \tag*{(28)} L AUCGR = B 1 i = 1 ∑ B ∣ P i ∣ ∣ N i ∣ 1 p ∈ P i ∑ n ∈ N i ∑ ℓ ( q i ( p ) − q i ( n ) ) , (28)
where ℓ \ell ℓ is a pairwise surrogate loss function defined on the ordering gap q i ( p ) − q i ( n ) q_i(p)-q_i(n) q i ( p ) − q i ( n ) . The function ℓ \ell ℓ is monotonically decreasing , as a larger gap indicates that the positive sample is ranked above the negative, thus incurring a smaller penalty. This objective promotes higher scores for positive samples relative to negative samples with respect to the same anchor.
We further provide a theoretical analysis of AUCGR, demonstrating that it is specifically constructed to fulfill the dual objectives discussed in § 3.4 . The following two theorems summarize these findings. Proofs are deferred to Sec. D and Sec. E for completeness.
Theorem 2 (AUCGR Supports D-Ability ). Fix a diffusion step t t t and define T ( z ) = ϵ θ ( x ^ t , h ω ( z ) , t ) T(z)=\epsilon_\theta(\hat{x}_t,h_\omega(z),t) T ( z ) = ϵ θ ( x ^ t , h ω ( z ) , t ) . Under Assumption 4, the intra-class scatter and inter-class scatter in the feature space ( S i n n e r , S i n t e r ) (S_{\mathrm{inner}},S_{\mathrm{inter}}) ( S inner , S inter ) can be bounded by those in the noise space ( S i n n e r ( ϵ ) ( t ) , S i n t e r ( ϵ ) ( t ) ) (S_{\mathrm{inner}}^{(\epsilon)}(t),S_{\mathrm{inter}}^{(\epsilon)}(t)) ( S inner ( ϵ ) ( t ) , S inter ( ϵ ) ( t )) as
S i n n e r ≤ 1 m 2 S i n n e r ( ϵ ) ( t ) , (29) S_{\mathrm{inner}}\leq\frac{1}{m^2}S_{\mathrm{inner}}^{(\epsilon)}(t), \tag*{(29)} S inner ≤ m 2 1 S inner ( ϵ ) ( t ) , (29) S i n t e r ≥ κ S i n t e r ( ϵ ) ( t ) − η S i n n e r ( ϵ ) ( t ) , (30) S_{\mathrm{inter}}\geq\kappa S_{\mathrm{inter}}^{(\epsilon)}(t)-\eta S_{\mathrm{inner}}^{(\epsilon)}(t), \tag*{(30)} S inter ≥ κ S inter ( ϵ ) ( t ) − η S inner ( ϵ ) ( t ) , (30) where m m m , κ \kappa κ , and η \eta η are positive constants depending on the Lipschitz continuity of T ( ⋅ ) T(\cdot) T ( ⋅ ) .
Theorem 3 (AUCGR Supports P-Ability ). Under the assumptions that negatives in the mini-batch are well separated from the anchor (i.e., sim ( ϵ ^ , ϵ ^ j ) ≪ sim ( ϵ ^ , ϵ g t ) \operatorname{sim}(\hat{\epsilon},\hat{\epsilon}_j)\ll\operatorname{sim}(\hat{\epsilon},\epsilon_{\mathrm{gt}}) sim ( ϵ ^ , ϵ ^ j ) ≪ sim ( ϵ ^ , ϵ gt ) for all j ≠ i j\ne i j = i ), predicted-noise norms are bounded away from 0 0 0 and ∞ \infty ∞ , and the surrogate loss is decreasing with bounded slope for the pairwise gaps used by AUCGR, then the AUCGR loss is bounded by a scaled reconstruction loss up to additive constants:
λ min R r e c + c min ≤ L A U C G R ≤ λ max R r e c + c max , (31) \lambda_{\min}R_{\mathrm{rec}}+c_{\min}\leq\mathcal{L}_{\mathrm{AUCGR}}\leq\lambda_{\max}R_{\mathrm{rec}}+c_{\max}, \tag*{(31)} λ m i n R rec + c m i n ≤ L AUCGR ≤ λ m a x R rec + c m a x , (31) where R r e c = 1 B ∑ i = 1 B ∥ ϵ ^ i − ϵ t , i g t ∥ 2 2 R_{\mathrm{rec}}=\frac{1}{B}\sum_{i=1}^{B}\lVert\hat{\epsilon}_i-\epsilon_{t,i}^{\mathrm{gt}}\rVert_2^2 R rec = B 1 ∑ i = 1 B ∥ ϵ ^ i − ϵ t , i gt ∥ 2 2 , λ min , λ max > 0 \lambda_{\min},\lambda_{\max}>0 λ m i n , λ m a x > 0 , and c min , c max c_{\min},c_{\max} c m i n , c m a x are constants.
Remark 1. Theorem 2 establishes that as L A U C G R \mathcal{L}_{\mathrm{AUCGR}} L AUCGR decreases, the intra-class scatter S i n n e r S_{\mathrm{inner}} S inner decreases and the inter-class scatter S i n t e r S_{\mathrm{inter}} S inter increases in the feature space, directly supporting the discriminative objective in Eq. (6) . Theorem 3 shows that L A U C G R \mathcal{L}_{\mathrm{AUCGR}} L AUCGR is bounded by the reconstruction error up to positive scaling constants, thereby fulfilling the fine-grained perceptual objective in Eq. (9) .
Partial AUC-oriented Generative Reconstruction
The preceding analysis establishes AUCGR by instantiating d d d as cosine similarity and ℓ \ell ℓ as an AUC-oriented pairwise functional. In this formulation, P i \mathcal{P}_i P i and N i \mathcal{N}_i N i act as the positive and negative comparison sets sampled from each mini-batch. However, this default construction presents a limitation: noisy negatives are intermixed with true negatives and may dominate the optimization process. This motivates a partial-AUC reformulation, which adheres to the same comparison principle while restricting optimization to reliable negatives.
Limitation of AUCGR
Although AUCGR resolves the objective conflict of the naive method, it still inherits a weakness from the default construction of the comparison sets. The AUC objective treats all negatives in N i \mathcal{N}_i N i equally. In practice, some in-batch negatives remain semantically compatible with the anchor and should not be treated as equally reliable negatives. Below, we formalize the issue in the predicted-noise space. Recall that the anchor predicted noise in Eq. (21) is ϵ ^ i = ϵ θ ( x ~ t , i , h ω ( f ϕ ( x i ) ) , t ) \hat{\epsilon}_i = \epsilon_\theta(\tilde{x}_{t,i}, h_\omega(f_\phi(x_i)), t) ϵ ^ i = ϵ θ ( x ~ t , i , h ω ( f ϕ ( x i )) , t ) . Under the default AUCGR construction, the in-batch negative set is
N i = { ϵ ^ i − k } k ≠ i , ϵ ^ i − k = ϵ θ ( x ~ t , i , h ω ( f ϕ ( x k ) ) , t ) . (32) \mathcal{N}_i = \{\hat{\epsilon}_i^{-k}\}_{k\ne i}, \qquad\hat{\epsilon}_i^{-k} = \epsilon_\theta(\tilde{x}_{t,i}, h_\omega(f_\phi(x_k)), t). \tag*{(32)} N i = { ϵ ^ i − k } k = i , ϵ ^ i − k = ϵ θ ( x ~ t , i , h ω ( f ϕ ( x k )) , t ) . (32)
Here, x ~ t , i \tilde{x}_{t,i} x ~ t , i is fixed and only the condition changes. A noisy negative corresponds to an index k ≠ i k \ne i k = i whose condition remains compatible with x i x_i x i . In the predicted-noise space, this means that
q i ( ϵ ^ i − k ) = d ( ϵ ^ i , ϵ ^ i − k ) ≈ 1. (33) q_i(\hat{\epsilon}_i^{-k}) = d(\hat{\epsilon}_i,\hat{\epsilon}_i^{-k}) \approx1. \tag*{(33)} q i ( ϵ ^ i − k ) = d ( ϵ ^ i , ϵ ^ i − k ) ≈ 1. (33)
Since this section instantiates d d d as cosine similarity, such a negative receives a high anchor-wise similarity score. We can make its effect explicit from the pairwise AUCGR objective. For anchor i i i , the corresponding loss is
L A U C G R ( i ) = 1 ∣ P i ∣ ∣ N i ∣ ∑ p ∈ P i ∑ n ∈ N i ℓ ( q i ( p ) − q i ( n ) ) . (34) \mathcal{L}_{\mathrm{AUCGR}}^{(i)} = \frac{1}{|\mathcal{P}_i||\mathcal{N}_i|}\sum_{p\in\mathcal{P}_i}\sum_{n\in\mathcal{N}_i}\ell(q_i(p)-q_i(n)). \tag*{(34)} L AUCGR ( i ) = ∣ P i ∣∣ N i ∣ 1 p ∈ P i ∑ n ∈ N i ∑ ℓ ( q i ( p ) − q i ( n )) . (34)
The contribution of a specific negative ϵ ^ i − k \hat{\epsilon}_i^{-k} ϵ ^ i − k to this loss can be written as
ρ i , k = 1 ∣ P i ∣ ∑ p ∈ P i ℓ ( q i ( p ) − q i ( ϵ ^ i − k ) ) . (35) \rho_{i,k} = \frac{1}{|\mathcal{P}_i|}\sum_{p\in\mathcal{P}_i}\ell(q_i(p)-q_i(\hat{\epsilon}_i^{-k})). \tag*{(35)} ρ i , k = ∣ P i ∣ 1 p ∈ P i ∑ ℓ ( q i ( p ) − q i ( ϵ ^ i − k )) . (35)
By the decreasing property of ℓ ( ⋅ ) \ell(\cdot) ℓ ( ⋅ ) in Eq. (28) , a larger negative score q i ( ϵ ^ i − k ) q_i(\hat{\epsilon}_i^{-k}) q i ( ϵ ^ i − k ) yields a smaller gap and therefore a larger contribution ρ i , k \rho_{i,k} ρ i , k . Consequently, optimization is dominated by the highest-score negatives. If these negatives are noisy or ambiguous, the ranking signal becomes systematically biased.
We further verify this argument empirically in Figure 4 . Noisy negatives mainly arise from the data distribution and mini-batch sampling. We first examine the sample frequency of semantic categories in the training data. The resulting distribution exhibits a clear long-tail pattern. We then randomly sample one mini-batch and compute the pairwise image similarity and label-text similarity within the batch. The statistics show that highly similar image pairs indeed appear in a normal mini-batch. These highly similar pairs can become noisy negatives under the default in-batch negative construction.
Figure 4. Empirical verification of noisy negatives. (a) The training data exhibits a clear long-tail distribution. (b) We randomly sample one mini-batch with batch size 128 and compute pairwise image similarity and label-text similarity within the batch. The results show that highly similar image pairs indeed appear during training.
Therefore, the limitation of AUCGR is not the AUC principle itself. The limitation is that AUCGR still allows the least reliable negative comparisons to dominate optimization. This observation motivates a partial-AUC reformulation that keeps all positives while restricting the negative side to the reliable region.
From AUCGR to pAUCGR
To suppress this bias, we keep all positives unchanged and truncate only the negative side. Given a truncation ratio α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) , let γ ( α ) \gamma(\alpha) γ ( α ) denote the ( 1 − α ) (1-\alpha) ( 1 − α ) -quantile of the negative-score distribution { q i ( n ) } n ∈ N i \{q_i(n)\}_{n\in\mathcal{N}_i} { q i ( n ) } n ∈ N i .
P n ∼ N i ( q i ( n ) ≤ γ ( α ) ) = 1 − α . (36) \mathbb{P}_{n\sim\mathcal{N}_i}(q_i(n)\le\gamma(\alpha))=1-\alpha. \tag*{(36)} P n ∼ N i ( q i ( n ) ≤ γ ( α )) = 1 − α . (36)
We then retain only the low-score negative subset
N i ( α ) = { n ∈ N i : q i ( n ) ≤ γ ( α ) } . (37) \mathcal{N}_i^{(\alpha)}=\{n\in\mathcal{N}_i:q_i(n)\le\gamma(\alpha)\}. \tag*{(37)} N i ( α ) = { n ∈ N i : q i ( n ) ≤ γ ( α )} . (37)
This operation removes the top α \alpha α fraction of highest-score negatives and keeps the remaining 1 − α 1-\alpha 1 − α fraction.
The following proposition makes the partial-AUC interpretation explicit.
Proposition 1. Let q : X → R q:\mathcal{X}\to\mathbb{R} q : X → R be a scoring function, and let x + ∼ P + x^+\sim\mathcal{P}_+ x + ∼ P + and x − ∼ P − x^-\sim\mathcal{P}_- x − ∼ P − denote a positive and a negative sample. Assume that the negative score distribution F − ( t ) = P ( q ( x − ) ≤ t ) F_-(t)=\mathbb{P}(q(x^-)\le t) F − ( t ) = P ( q ( x − ) ≤ t ) is continuous. For any measurable negative subset N \mathcal{N} N with positive probability, define the conditional ranking objective
AUC ( N ) = P ( q ( x + ) > q ( x − ) ∣ x − ∈ N ) . (38) \operatorname{AUC}(\mathcal{N})=\mathbb{P}(q(x^+)>q(x^-)\mid x^-\in\mathcal{N}). \tag*{(38)} AUC ( N ) = P ( q ( x + ) > q ( x − ) ∣ x − ∈ N ) . (38) For any β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] , let
N l o w β = { x − : F − ( q ( x − ) ) ≤ β } . (39) \mathcal{N}_{\mathrm{low}}^\beta=\{x^-:F_-(q(x^-))\le\beta\}. \tag*{(39)} N low β = { x − : F − ( q ( x − )) ≤ β } . (39) Then the partial AUC over [ 1 − β , 1 ] [1-\beta,1] [ 1 − β , 1 ] is defined as
pAUC [ 1 − β , 1 ] : = ∫ 1 − β 1 TPR ( FPR − 1 ( ξ ) ) d ξ , (40) \operatorname{pAUC}_{[1-\beta,1]}:=\int_{1-\beta}^{1}\operatorname{TPR}(\operatorname{FPR}^{-1}(\xi))\,\mathrm{d}\xi, \tag*{(40)} pAUC [ 1 − β , 1 ] := ∫ 1 − β 1 TPR ( FPR − 1 ( ξ )) d ξ , (40) where TPR ( t ) = P ( q ( x + ) > t ) \operatorname{TPR}(t)=\mathbb{P}(q(x^+)>t) TPR ( t ) = P ( q ( x + ) > t ) , FPR ( t ) = P ( q ( x − ) > t ) \operatorname{FPR}(t)=\mathbb{P}(q(x^-)>t) FPR ( t ) = P ( q ( x − ) > t ) , and FPR − 1 ( ξ ) \operatorname{FPR}^{-1}(\xi) FPR − 1 ( ξ ) denotes the threshold associated with false positive rate ξ \xi ξ .
The proof is provided in Sec. F. In our setting, N i ( α ) \mathcal{N}_i^{(\alpha)} N i ( α ) keeps exactly the lowest-score 1 − α 1-\alpha 1 − α fraction of negatives. Therefore, by Proposition 1 with β = 1 − α \beta= 1-\alpha β = 1 − α , the retained negative set corresponds to the partial AUC over the interval
pAUC [ α , 1 ] = ∫ α 1 TPR ( FPR − 1 ( ξ ) ) d ξ . (41) \operatorname{pAUC}_{[\alpha,1]} = \int_{\alpha}^{1} \operatorname{TPR}\left(\operatorname{FPR}^{-1}(\xi)\right)\,\mathrm{d}\xi. \tag*{(41)} pAUC [ α , 1 ] = ∫ α 1 TPR ( FPR − 1 ( ξ ) ) d ξ . (41)
up to the constant factor 1 − α 1-\alpha 1 − α . Since this factor does not depend on the model, the retained negative set does not define a heuristic filter. It restricts optimization to the reliable part of the AUC objective.
Based on Eq. (37) , we define the final robust objective, termed Partial AUC Generative Reconstruction (pAUCGR) , as
L p A U C G R = 1 B ∑ i = 1 B 1 ∣ P i ∣ ∣ N i ( α ) ∣ ∑ p ∈ P i ∑ n ∈ N i ( α ) ℓ ( q i ( p ) − q i ( n ) ) . (42) \mathcal{L}_{\mathrm{pAUCGR}} = \frac{1}{B}\sum_{i=1}^{B}\frac{1}{\lvert\mathcal{P}_i\rvert\lvert\mathcal{N}_i^{(\alpha)}\rvert}\sum_{p\in\mathcal{P}_i}\sum_{n\in\mathcal{N}_i^{(\alpha)}}\ell\left(q_i(p)-q_i(n)\right). \tag*{(42)} L pAUCGR = B 1 i = 1 ∑ B ∣ P i ∣ ∣ N i ( α ) ∣ 1 p ∈ P i ∑ n ∈ N i ( α ) ∑ ℓ ( q i ( p ) − q i ( n ) ) . (42)
When α = 0 \alpha= 0 α = 0 , Eq. (42) collapses to the AUCGR objective in Eq. (28) . As α \alpha α increases, the optimization process incrementally excludes negative comparisons that are most likely to introduce noise. In this manner, pAUCGR maintains the comparison principle outlined in Eq. (23) , inherits the AUC-based interpretation established previously, and mitigates the dominant bias caused by ambiguous negatives. Consequently, pAUCGR serves as the final optimization framework in our approach.
Remark 2. Our goal is to remove highly similar, potentially noisy negatives using a consistent similarity criterion across mini-batches. The ratio α \alpha α denotes the fraction of high-score negatives in the full training distribution. A mini-batch only provides a local estimate of this distribution. Fixing α \alpha α in each batch targets the same removed fraction, but its empirical quantile threshold can vary. We instead fix the cosine similarity threshold at γ 0 = 0.9 \gamma_0 = 0.9 γ 0 = 0.9 and allow the removed fraction to vary across batches:
α ^ i , B ( γ 0 ) = 1 ∣ N i , B ∣ ∑ n ∈ N i , B 1 [ q i ( n ) > γ 0 ] . (43) \widehat{\alpha}_{i,B}(\gamma_0)=\frac{1}{\lvert\mathcal{N}_{i,B}\rvert}\sum_{n\in\mathcal{N}_{i,B}}\mathbf{1}[q_i(n)>\gamma_0]. \tag*{(43)} α i , B ( γ 0 ) = ∣ N i , B ∣ 1 n ∈ N i , B ∑ 1 [ q i ( n ) > γ 0 ] . (43)
Consequently, each batch removes negatives above a fixed similarity threshold, whose correspondence to the induced population truncation ratio is proved in Sec. K.
Surrogate Loss Instantiation of AUCGR
The AUCGR and pAUCGR objectives in Eq. (28) and Eq. (42) are obtained by replacing the non-differentiable AUC indicator with a surrogate loss ℓ ( ⋅ ) \ell(\cdot) ℓ ( ⋅ ) . Each surrogate converts the original 0-1 ranking event into a tractable pairwise surrogate risk, while preserving the preference that positive samples should be ranked ahead of negative samples. Different choices of ℓ ( ⋅ ) \ell(\cdot) ℓ ( ⋅ ) correspond to different optimization strategies, which provide AUCGR with a broader family of realizations.
Following the AUC optimization literature [94 ] , for the pairwise gap u = q i ( p ) − q i ( n ) u=q_i(p)-q_i(n) u = q i ( p ) − q i ( n ) , the surrogate loss ℓ ( ⋅ ) \ell(\cdot) ℓ ( ⋅ ) can take the following choices.
Square. ℓ s q u a r e ( u ) = [ λ − u ] 2 \ell_{\mathrm{square}}(u)=[\lambda-u]^2 ℓ square ( u ) = [ λ − u ] 2 .
Hinge. ℓ h i n g e ( u ) = [ λ − u ] + \ell_{\mathrm{hinge}}(u)=[\lambda-u]_+ ℓ hinge ( u ) = [ λ − u ] + .
Exponential. ℓ e x p ( u ) = exp ( − u ) \ell_{\mathrm{exp}}(u)=\exp(-u) ℓ exp ( u ) = exp ( − u ) .
Logistic. ℓ l o g i s t i c ( u ) = log ( 1 + exp ( − u / τ ) ) \ell_{\mathrm{logistic}}(u)=\log(1+\exp(-u/\tau)) ℓ logistic ( u ) = log ( 1 + exp ( − u / τ )) .
where λ > 0 \lambda>0 λ > 0 denotes a safety margin and τ > 0 \tau>0 τ > 0 denotes a temperature.
This view also clarifies the relation to our conference version. The conference version employs a contrastive objective that compares positive and negative reconstruction signals. This objective can be naturally connected to logistic AUCGR through the pairwise ranking formulation. Specifically, the anchor-wise loss L D C R ( i ) \mathcal{L}_{\mathrm{DCR}}^{(i)} L DCR ( i ) is defined as
L D C R ( i ) = − 1 ∣ P i ∣ ∑ p ∈ P i log exp ( q i ( p ) / τ ) ∑ v ∈ P i ∪ N i exp ( q i ( v ) / τ ) . (44) \mathcal{L}_{\mathrm{DCR}}^{(i)}=-\frac{1}{\lvert\mathcal{P}_i\rvert}\sum_{p\in\mathcal{P}_i}\log\frac{\exp(q_i(p)/\tau)}{\sum_{v\in\mathcal{P}_i\cup\mathcal{N}_i}\exp(q_i(v)/\tau)}. \tag*{(44)} L DCR ( i ) = − ∣ P i ∣ 1 p ∈ P i ∑ log ∑ v ∈ P i ∪ N i exp ( q i ( v ) / τ ) exp ( q i ( p ) / τ ) . (44)
The following theorem establishes this correspondence.
Theorem 4. For anchor i i i , let P i \mathcal{P}_i P i denote the positive set and let N i \mathcal{N}_i N i denote the negative set. Then we have the following result:
L D C R ( i ) ≥ 1 ∣ P i ∣ ∣ N i ∣ ∑ p ∈ P i ∑ n ∈ N i ℓ ( q i ( p ) − q i ( n ) ) , (45) \mathcal{L}_{\mathrm{DCR}}^{(i)}\geq\frac{1}{\lvert\mathcal{P}_i\rvert\lvert\mathcal{N}_i\rvert}\sum_{p\in\mathcal{P}_i}\sum_{n\in\mathcal{N}_i}\ell(q_i(p)-q_i(n)), \tag*{(45)} L DCR ( i ) ≥ ∣ P i ∣ ∣ N i ∣ 1 p ∈ P i ∑ n ∈ N i ∑ ℓ ( q i ( p ) − q i ( n )) , (45) where ℓ ( ⋅ ) \ell(\cdot) ℓ ( ⋅ ) is instantiated as the logistic surrogate function
ℓ l o g i s t i c ( u ) = log ( 1 + exp ( − u τ ) ) . (46) \ell_{\mathrm{logistic}}(u)=\log\left(1+\exp\left(-\frac{u}{\tau}\right)\right). \tag*{(46)} ℓ logistic ( u ) = log ( 1 + exp ( − τ u ) ) . (46) Moreover, the bound is tight if and only if ∣ P i ∣ = ∣ N i ∣ = 1 \lvert\mathcal{P}_i\rvert=\lvert\mathcal{N}_i\rvert=1 ∣ P i ∣ = ∣ N i ∣ = 1 .
The proof is provided in Sec. G. Theorem 4 shows that L D C R \mathcal{L}_{\mathrm{DCR}} L DCR is a special case of AUCGR when ∣ P i ∣ = ∣ N i ∣ = 1 \lvert\mathcal{P}_i\rvert=\lvert\mathcal{N}_i\rvert=1 ∣ P i ∣ = ∣ N i ∣ = 1 . Meanwhile, from the ROC perspective, AUCGR supports a broader family of surrogate losses, making it a more general formulation with a clearer theoretical interpretation . We provide a detailed theoretical analysis in the next section.
Generalization Bound
To further establish that one-way negative truncation does not compromise statistical reliability, we analyze the generalization properties of pAUCGR. Let f ∈ F f\in\mathcal{F} f ∈ F denote a comparison model parameterized by the learnable modules. Each training instance is represented by an anchor tuple
z i = ( ϵ ^ i , ϵ ^ i + , ϵ ^ i − ) , ϵ ^ i + ∈ P , ϵ ^ i − ∈ N − (47) z_i=(\hat{\epsilon}_i,\hat{\epsilon}_i^+,\hat{\epsilon}_i^-),\qquad\hat{\epsilon}_i^+\in\mathcal{P},\hat{\epsilon}_i^-\in\mathcal{N}^- \tag*{(47)} z i = ( ϵ ^ i , ϵ ^ i + , ϵ ^ i − ) , ϵ ^ i + ∈ P , ϵ ^ i − ∈ N − (47)
where P \mathcal{P} P follows the same positive-set construction as above, N − = { n j } j = 1 J \mathcal{N}^-=\{n_j\}_{j=1}^{J} N − = { n j } j = 1 J is a candidate negative pool sampled from the negative condition distribution, and ϵ ^ \hat{\epsilon} ϵ ^ is the anchor predicted noise induced by f f f .
For a fixed truncation ratio α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) , γ ( α ) \gamma(\alpha) γ ( α ) denote the ( 1 − α ) (1-\alpha) ( 1 − α ) -quantile of the negative scores { d ( ϵ ^ , n ) : n ∈ N − } \{d(\hat{\epsilon},n):n\in\mathcal{N}^-\} { d ( ϵ ^ , n ) : n ∈ N − } , and define the retained negative set
N α ( z ) = { n ∈ N − : d ( ϵ ^ , n ) ≤ γ ( α ) } . (48) \mathcal{N}_{\alpha}(z)=\{n\in\mathcal{N}^-:d(\hat{\epsilon},n)\leq\gamma(\alpha)\}. \tag*{(48)} N α ( z ) = { n ∈ N − : d ( ϵ ^ , n ) ≤ γ ( α )} . (48)
The corresponding anchor-wise pAUCGR loss is
g f ( α ) ( z ) = 1 ∣ P ∣ ∣ N α ( z ) ∣ ∑ p ∈ P ∑ n ∈ N α ( z ) ℓ ( d ( ϵ ^ , p ) − d ( ϵ ^ , n ) ) . (49) g_f^{(\alpha)}(z)=\frac{1}{\lvert\mathcal{P}\rvert\lvert\mathcal{N}_{\alpha}(z)\rvert}\sum_{p\in\mathcal{P}}\sum_{n\in\mathcal{N}_{\alpha}(z)}\ell\left(d(\hat{\epsilon},p)-d(\hat{\epsilon},n)\right). \tag*{(49)} g f ( α ) ( z ) = ∣ P ∣ ∣ N α ( z )∣ 1 p ∈ P ∑ n ∈ N α ( z ) ∑ ℓ ( d ( ϵ ^ , p ) − d ( ϵ ^ , n ) ) . (49)
Accordingly, the population risk and empirical risk are
R α ( f ) = E z ∼ Q [ g f ( α ) ( z ) ] , R ^ α ( f ) = 1 N ∑ i = 1 N g f ( α ) ( z i ) , (50) \mathcal{R}_{\alpha}(f)=\mathbb{E}_{z\sim\mathcal{Q}}\left[g_f^{(\alpha)}(z)\right],\qquad\widehat{\mathcal{R}}_{\alpha}(f)=\frac{1}{N}\sum_{i=1}^{N}g_f^{(\alpha)}(z_i), \tag*{(50)} R α ( f ) = E z ∼ Q [ g f ( α ) ( z ) ] , R α ( f ) = N 1 i = 1 ∑ N g f ( α ) ( z i ) , (50)
where Q \mathcal{Q} Q denotes the distribution of such anchor tuples, and { z i } i = 1 B \{z_i\}_{i=1}^{B} { z i } i = 1 B are i.i.d. samples from Q \mathcal{Q} Q .
Assumption 1. We adopt the following assumptions.
For every f ∈ F f \in\mathcal{F} f ∈ F , the similarity score satisfies d ( ϵ ^ , v ) ∈ [ − 1 , 1 ] d(\hat{\epsilon}, v) \in[-1,1] d ( ϵ ^ , v ) ∈ [ − 1 , 1 ] . Hence the pairwise gap in Eq. (49) belongs to [ − 2 , 2 ] [-2,2] [ − 2 , 2 ] . The surrogate ℓ \ell ℓ is continuous on [ − 2 , 2 ] [-2,2] [ − 2 , 2 ] , and there exists a constant M > 0 M > 0 M > 0 such that 0 ≤ ℓ ( u ) ≤ M 0 \le\ell(u) \le M 0 ≤ ℓ ( u ) ≤ M for all u ∈ [ − 2 , 2 ] u \in[-2,2] u ∈ [ − 2 , 2 ] .
For every f ∈ F f \in\mathcal{F} f ∈ F and every anchor tuple z z z in the support of Q \mathcal{Q} Q , the negative-score distribution is continuous. Therefore, γ ( α ) \gamma(\alpha) γ ( α ) is well defined and the truncation threshold is unambiguous.
Let G α = { g f α : f ∈ F } \mathcal{G}_{\alpha} = \{g_{f}^{\alpha}: f \in\mathcal{F}\} G α = { g f α : f ∈ F } . Its covering number under the sup norm satisfies
log N ( G α , ϵ , ∥ ⋅ ∥ ∞ ) ≤ A log R ϵ , ∀ ϵ ∈ ( 0 , R ] , (51) \log\mathcal{N}(\mathcal{G}_{\alpha}, \epsilon, \lVert\cdot\rVert_{\infty}) \le A \log\frac{R}{\epsilon}, \qquad\forall\epsilon\in(0,R], \tag*{(51)} log N ( G α , ϵ , ∥ ⋅ ∥ ∞ ) ≤ A log ϵ R , ∀ ϵ ∈ ( 0 , R ] , (51)
for some constants A > 0 A > 0 A > 0 and R > 0 R > 0 R > 0 .
These assumptions are mild and reasonable in our context. The boundedness condition is justified by the use of cosine similarity and a bounded surrogate loss over the relevant interval. The continuity condition simply excludes degenerate score distributions. The covering number requirement is a standard complexity constraint to ensure uniform convergence.
Theorem 5 (Generalization of pAUCGR). Under Assumption 1, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and any fixed ϵ ∈ ( 0 , R ] \epsilon\in(0,R] ϵ ∈ ( 0 , R ] , with probability at least 1 − δ 1-\delta 1 − δ , the following inequality holds simultaneously for all f ∈ F f \in\mathcal{F} f ∈ F :
∣ R α ( f ) − R ^ α ( f ) ∣ ≤ 2 ϵ + M A log ( R / ϵ ) + log ( 2 / δ ) 2 N . (52) \left|R_{\alpha}(f)-\widehat{R}_{\alpha}(f)\right| \le2\epsilon+ M\sqrt{\frac{A\log(R/\epsilon)+\log(2/\delta)}{2N}}. \tag*{(52)} R α ( f ) − R α ( f ) ≤ 2 ϵ + M 2 N A log ( R / ϵ ) + log ( 2/ δ ) . (52) In particular, choosing ϵ = R / N \epsilon= R/N ϵ = R / N yields
∣ R α ( f ) − R ^ α ( f ) ∣ ≤ 2 R N + M A log N + log ( 2 / δ ) 2 N . (53) \left|R_{\alpha}(f)-\widehat{R}_{\alpha}(f)\right| \le\frac{2R}{N} + M\sqrt{\frac{A\log N+\log(2/\delta)}{2N}}. \tag*{(53)} R α ( f ) − R α ( f ) ≤ N 2 R + M 2 N A log N + log ( 2/ δ ) . (53)
The proof is presented in Sec. H. This theorem demonstrates that pAUCGR retains the standard uniform convergence rate of O ( log N / N ) O(\sqrt{\log N/N}) O ( log N / N ) even in the presence of one-way truncation, where N N N denotes the number of samples.
Experiments
In this section, we describe some details of the experiments and present our results. Due to space limitations, please refer to Sec. I and Sec. J for the extended version.
Experimental Setups
Implementation Details. We implement our method as plug-and-play optimization modules for existing generative reconstruction methods. In the main experiments, we apply them to the following two base methods:
GenHancer [24 ] systematically investigates how generative models can enhance CLIP by refining conditioning design, denoising strategies, and generation paradigms. It shows that using global conditions and lightweight denoisers leads to more stable reconstruction-based representation learning. In addition, it extends the reconstruction process to flow-based generative models.
For both base methods, we replace the original training objective with our objective, while keeping the network architecture, training data, and other optimization settings unchanged whenever possible for fair comparison. We denote the enhanced versions as +AUCGR and +pAUCGR. For AUCGR, we use the hinge surrogate listed above. For pAUCGR, we set γ ( α ) = 0.9 \gamma(\alpha)=0.9 γ ( α ) = 0.9 . This protocol directly evaluates whether the proposed AUC-oriented optimization improves existing generative reconstruction pipelines. We retrain all models using public code on eight NVIDIA A100 80GB GPUs.
Evaluation Protocol. Following [23 ] , [24 ] , we enhance 6 CLIP backbones: OpenAI CLIP ViT-L@224/[11 ] , MetaCLIP ViT-L/H@224 [108 ] , and SigLIP ViT-SO@224/@384 [109 ] . To evaluate the P-Ability, we use MMVP-VLM [53 ] . To evaluate the D-Ability, we conduct zero-shot clustering and zero-shot classification on 26 standard datasets. For clustering, we report Normalized Mutual Information (NMI), Accuracy (ACC), and Adjusted Rand Index (ARI). For classification, we report Top-1 Accuracy (ACC@1), Top-5 Accuracy (ACC@5), and Mean Recall (mRecall). We also evaluate zero-shot retrieval on Flickr30k [60 ] and COCO [61 ] , covering both image-to-text and text-to-image retrieval. Finally, we train MLLMs with our enhanced CLIP encoders using the official LLaVA-1.5 [1 ] recipe. We then evaluate these MLLMs on the same benchmarks [53 ] , [62 ] , [63 ] , [133 ] , [134 ] , [135 ] .
Overall Performance
Performance of P-Ability
Quantitative results. As shown in Table 1 , AUCGR consistently improves the original generative reconstruction baselines on MMVP-VLM. For OpenAI CLIP ViT-L@224, AUCGR improves GenHancer from 25.2% to 26.7%, and raises the color-and-appearance score from 66.7% to 73.3%. These gains show that AUCGR improves fine-grained perceptual ability across visual patterns such as color, state, structure, and viewpoint. Our conference version, DCR, also yields clear improvements over the corresponding baselines. pAUCGR further improves the results by filtering ambiguous negatives with a suitable similarity threshold. On MetaCLIP ViT-L@224, pAUCGR improves DIVA from 25.2% to 29.6% and GenHancer from 28.2% to 31.8%. Overall, AUCGR brings consistent gains over generative reconstruction baselines, and pAUCGR achieves stronger performance by reducing unreliable negative comparisons.
Table 1. TABLE 1 Performance of Fine-grained Perceptual Ability (P-Ability) on the MMVP-VLM benchmark in terms of ACC. ‘+DCR’ denotes our conference version. The visual patterns are symbolized as: ☼: Orientation and Direction, Q: Presence of Specific Features, ⟳: State and Condition, ↑↓: Quantity and Count, ⬤: Positional and Relational Context, 🎨: Color and Appearance, ⚙: Structural and Physical Characteristics, A: Texts, ▣: Viewpoint and Perspective.
Qualitative results. Figure 5 further shows representative examples from MMVP-VLM. The original CLIP often fails when the question depends on local visual evidence, such as color, viewpoint, and subtle object appearance. After applying our plug-in optimization, the enhanced CLIP gives more accurate predictions on these detail-sensitive cases. The qualitative results are consistent with the quantitative gains and indicate that our method improves fine-grained perceptual ability without changing the model architecture. Additional qualitative results are provided in Sec. J.1.
Figure 5. Qualitative results of P-Ability on the MMVP-VLM benchmark. The predictions from the original CLIP and our improved version are indicated by red and green arrows, respectively. The improved CLIP effectively addresses the original model’s limitations in capturing fine-grained visual details. More images are provided in Figure 12 and Figure 13 .
Performance of D-Ability
We evaluate D-Ability from three complementary perspectives, including zero-shot clustering, zero-shot classification, and zero-shot retrieval.
Zero-shot clustering. We first evaluate whether image representations form separable semantic structures in image clustering tasks. As shown in Table 2 , AUCGR improves the two generative reconstruction baselines on 26 standard datasets. Specifically, on SigLIP ViT-SO@224, AUCGR improves DIVA from 26.88% to 55.81% Avg ACC. On MetaCLIP ViT-L@224, AUCGR improves GenHancer from 30.39% to 44.61% Avg ACC. pAUCGR further improves the results by filtering ambiguous negatives. For example, it raises the DIVA result on SigLIP ViT-SO@224 to 59.30% Avg ACC and the GenHancer result on MetaCLIP ViT-L@224 to 52.78% Avg ACC. Compared with the original pretrained models, pAUCGR largely restores their discriminative ability in most settings and even achieves improvements on some datasets. The ARI and NMI results in Table 5 and Table 6 show that the proposed comparison-based optimization improves the quality of visual representations and remains effective in the more demanding classification setting. This indicates that the comparison-based supervision encourages samples with related semantics to be grouped more compactly and separates samples with different semantics more clearly. The t t t -SNE visualization in Figure 6 further supports this observation.
Table 2. Performance of Discriminative Ability (D-Ability) on 26 standard zero-shot clustering benchmarks in terms of ACC. ‘+DCR’ denotes our conference version. O-1, M-1, and S-1 separately represent OpenAI CLIP ViT-L@224, MetaCLIP ViT-L@224, and SigLIP ViT-SO@224. Results in terms of ARI and NMI are provided in Table 5 and Table 6 .
Figure 6. Qualitative results of D-Ability on the CIFAR-10 zero-shot clustering benchmark by using the t-SNE method. The first three panels use DIVA with SigLIP ViT-SO@224 (S-1), and the last three panels use GenHancer with MetaCLIP ViT-L@224 (M-1). The improved CLIP achieves better class separability.
Table 5. Performance of Discriminative Ability (D-Ability) on 26 standard zero-shot clustering benchmarks in terms of ARI. ‘+DCR’ denotes our conference. This table provides supplementary results for Table 2 .
Table 6. Performance of Discriminative Ability (D-Ability) on 26 standard zero-shot clustering benchmarks in terms of NMI. ‘+DCR’ denotes our conference version. This table provides supplementary results for Table 2 .
Zero-shot classification. We next examine the performance on zero-shot classification. As shown in Figure 8 , AUCGR improves the average classification accuracy of both DIVA and GenHancer, showing that comparison-based supervision helps improve the discriminative ability weakened by plain generative reconstruction supervision. Specifically, on MetaCLIP ViT-L@224, AUCGR improves GenHancer from 20.69% to 69.26% Avg ACC@1. On SigLIP ViT-SO@224, AUCGR improves DIVA from 40.97% to 70.86% Avg ACC@1. pAUCGR further improves the results by filtering ambiguous negatives with the similarity threshold, reaching 72.42% and 74.28% Avg ACC@1, respectively. These results
Figure 8. Average performance on 26 standard zero-shot classification benchmarks. O-1, M-1, and S-1 separately represent OpenAI CLIP ViT-L@224, MetaCLIP ViT-L@224, and SigLIP ViT-SO@224. Detailed results are provided in Table 7 , Table 8 and Table 9 .
the proposed comparison-based optimization improves the quality of visual representations and remains effective in the more demanding classification setting. The complete results are provided in Table 7 , Table 8 and Table 9 .
Table 7. Performance on 26 standard zero-shot classification benchmarks in terms of ACC@1. This table presents the detailed results corresponding to Figure 8 .
Table 8. TABLE 8 Performance on 26 standard zero-shot classification benchmarks in terms of ACC@5. This table presents the detailed results corresponding to Figure 8 . Since PCam and Rendered SST2 contain only two categories, ACC@5 cannot be computed on these two datasets.
Table 9. Performance on 26 standard zero-shot classification benchmarks in terms of mRecall. This table presents the detailed results corresponding to Figure 8 .
Zero-shot retrieval. Finally, we evaluate cross-modal alignment on Flickr30k and COCO. As shown in Figure 7 , our method improves both image-to-text and text-to-image retrieval on OpenAI CLIP ViT-L@224. Specifically, for Text@5 on COCO, AUCGR improves the DIVA baseline from 63.92 to 71.64, and pAUCGR further reaches 79.34. For Image@5 on Flickr30k, AUCGR improves the GenHancer baseline from 75.70 to 81.34, and pAUCGR further improves it to 83.40. The results on MetaCLIP ViT-L@224 and SigLIP ViT-SO@224 are provided in Figure 14 and Figure 15 , where the same trend is observed. Since retrieval requires distinguishing correct image-text pairs from many candidates, these results further show that the enhanced visual representations have stronger discriminative ability.
Figure 7. Performance on zero-shot retrieval on the OpenAI CLIP ViT-L@224. Compared with the original DIVA/GenHancer model, our method consistently improves both image-to-text and text-to-image retrieval on Flickr30k and COCO. The results on the MetaCLIP ViT-L@224 and SigLIP ViT-SO@224 are provided in Figure 14 and Figure 15 .
Figure 14. Performance on zero-shot retrieval on MetaCLIP ViT-L@224. Compared with the original DIVA/GenHancer model, our method consistently improves both image-to-text and text-to-image retrieval on Flickr30k and COCO.
Figure 15. Performance on zero-shot retrieval on SigLIP ViT-SO@224. Compared with the original DIVA/GenHancer model, our method consistently improves both image-to-text and text-to-image retrieval on Flickr30k and COCO.
Multimodal Large Language Model Evaluation
We further examine whether the enhanced visual representations learned by our method can be effectively transferred to Multimodal Large Language Models (MLLMs) . Specifically, we substitute the original CLIP vision encoder in the LLaVA-1.5 [1 ] framework with our improved CLIP while keeping all training configurations unchanged to ensure a fair comparison. The evaluation is performed on a series of vision-centric and general multimodal benchmarks, covering both vision-centric and general tasks. As shown in Table 3 , the integration of our enhanced CLIP substantially boosts MLLM performance, particularly in benchmarks that require detailed visual understanding. These results demonstrate that improving the granularity of visual representations not only benefits standalone vision models but also strengthens the visual grounding and reasoning ability of multimodal systems. Some qualitative results are provided in Sec. J.3.
Table 3. Performance of MLLMs (LLaVA-1.5 [1 ] ) on various benchmarks. The champion is highlighted in bold.
Ablation Studies
Results on Naive and Gradient-Balancing Methods. Figure 9 compares our AUCGR with the naive linear weighting method and three representative gradient-balancing methods, i.e., PCGrad [113 ] , CAGrad [136 ] , and GradNorm [137 ] . Although these methods can improve D-Ability or P-Ability in some cases, their gains are generally limited or inconsistent across different baselines and backbones. This suggests that the conflict between the two optimization objectives remains insufficiently resolved, making it difficult to approach the Pareto frontier. In contrast, AUCGR and pAUCGR consistently achieve a better balance between the two abilities through a unified optimization objective. These results further demonstrate the effectiveness of the proposed AUCGR framework.
Figure 9. Ablation comparison with naive linear weighting and gradient-balancing methods. We evaluate the trade-off between P-Ability and D-Ability on two generative reconstruction baselines, DIVA and GenHancer, across OpenAI CLIP ViT-L@224 (O-1), MetaCLIP ViT-L@224 (M-1), and SigLIP ViT-SO@224 (S-1). Here, D-Ability is measured by ACC on zero-shot clustering benchmarks, while +N, +P, +C, and +G denote +Naive, +PCGrad, +CAGrad, and +GradNorm, respectively. Compared with both naive linear weighting and gradient-balancing methods, AUCGR and pAUCGR achieve a better balance between the two abilities.
Results on Different Generative Model Structures. Figure 10 evaluates our method with different generative reconstruction backbones, including diffusion models and flow-based models. Specifically, we test Stable Diffusion v1.5, Stable Diffusion v2.1, and Rectified Flow. The results show that our plug-in consistently improves both D-Ability and P-Ability across these generative structures. This indicates that the proposed optimization is not tied to a specific architecture, but can be applied to different generative reconstruction models in a plug-and-play manner.
Figure 10. Ablation study on different generative model structures. We evaluate our method with Stable Diffusion v1.5, Stable Diffusion v2.1, and Rectified Flow as the generative reconstruction backbone. All experiments use OpenAI CLIP ViT-L@224 as the visual backbone. The results show that our framework is not tied to a specific generative architecture and consistently improves both P-Ability and D-Ability.
Ablation Study on Hyper-Parameter γ ( α ) \gamma(\alpha) γ ( α ) in pAUCGR. The hyper-parameter γ ( α ) \gamma(\alpha) γ ( α ) controls which negatives are filtered in pAUCGR. A larger value keeps more high-similarity negatives, but it may fail to remove noisy negatives that share similar visual or semantic cues with the anchor. A smaller value removes more negatives, but it may also discard useful mismatched samples that provide valid comparison signals. As shown in Figure 11 , both extremes lead to suboptimal performance. In our experiments, γ ( α ) = 0.9 \gamma(\alpha)=0.9 γ ( α ) = 0.9 provides a better balance between removing noisy negatives and retaining reliable negative comparisons. We therefore use this setting as the default choice.
Figure 11. Ablation study on the pAUCGR parameter. We vary the retained-negative threshold γ ( α ) \gamma(\alpha) γ ( α ) . All experiments use OpenAI CLIP ViT-L@224 as the visual backbone.
Due to space limitations, experiments validating negative filtering and robustness to surrogate choices and random seeds are presented in Sec. J.5. The computational cost analysis is provided in Sec. J.5.
Conclusion
In this paper, we focus on balanced visual representation learning by jointly considering semantic discrimination and fine-grained perception. Our analysis shows that the balance cannot be reliably achieved by a naive weighted
combination of contrastive learning and generative reconstruction, because the resulting optimization is structurally conflicted. To address this issue, we formulate generative reconstruction as a comparison-based optimization problem and instantiate it with AUCGR. We further introduce pAUCGR to reduce biased supervision from noisy negatives through one-way partial AUC. This formulation provides a plug-and-play optimization module for existing generative reconstruction methods. Theoretical results and empirical studies consistently support the proposed framework and demonstrate its effectiveness across CLIP backbones, visual benchmarks, and multimodal large language models.
Acknowledgments
This work was supported in part by National Natural Science Foundation of China: 62525212, U23B2051, 62236008, 62441232, 62521007, U21B2038, 62502496, and 6257633, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680201, in part by the project ZR2025ZD01 supported by Shandong Provincial Natural Science Foundation, in part by the Young Elite Scientists Sponsorship Program of the Beijing High Innovation Plan No. 20250687, and in part by Beijing Natural Science Foundation-Xiaomi Innovation Joint Fund (Grant No. 26L030072).
References
Symbol Definitions
In this section, Table 4 summarizes the key notations used in the paper.
Table 4. A summary of key notations and descriptions in the paper.
Appendix B: Algorithms of AUCGR and pAUCGR
For clarity, we provide the plug-and-play training procedures of AUCGR and pAUCGR in Algorithms 1 and 2. Both algorithms only require an existing generative reconstruction framework to produce an anchor response, a matched set, and a mismatched set. Therefore, they can be inserted into different reconstruction-based enhancement pipelines without assuming a specific generative architecture.
Algorithm 1: AUCGR Algorithm.
Input: Dataset D \mathcal{D} D ; existing generative reconstruction framework with trainable modules Θ \Theta Θ ; comparison measure d d d ; decreasing surrogate loss ℓ \ell ℓ ; batch size B B B ; training iterations K K K .
Output: Enhanced visual encoder in the base framework.
for k = 1 k = 1 k = 1 to K K K do
Sample a mini-batch { x i } i = 1 B \{x_i\}_{i=1}^{B} { x i } i = 1 B from D \mathcal{D} D .
for i = 1 i = 1 i = 1 to B B B do
Use the base framework to obtain the anchor response ϵ ^ i \hat{\epsilon}_i ϵ ^ i .
Build the positive set P i \mathcal{P}_i P i and negative set N i \mathcal{N}_i N i using the base framework.
for each v ∈ P i ∪ N i v \in\mathcal{P}_i \cup\mathcal{N}_i v ∈ P i ∪ N i do
Compute the anchor-wise score q i ( v ) ← d ( ϵ ^ i , v ) q_i(v) \leftarrow d(\hat{\epsilon}_i,v) q i ( v ) ← d ( ϵ ^ i , v ) .
end for
Compute the AUCGR anchor loss
L A U C G R ( i ) ← 1 ∣ P i ∣ ∣ N i ∣ ∑ p ∈ P i ∑ n ∈ N i ℓ ( q i ( p ) − q i ( n ) ) . \mathcal{L}_{\mathrm{AUCGR}}^{(i)} \leftarrow\frac{1}{|\mathcal{P}_i||\mathcal{N}_i|}\sum_{p\in\mathcal{P}_i}\sum_{n\in\mathcal{N}_i}\ell(q_i(p)-q_i(n)). L AUCGR ( i ) ← ∣ P i ∣∣ N i ∣ 1 p ∈ P i ∑ n ∈ N i ∑ ℓ ( q i ( p ) − q i ( n )) .
end for
L A U C G R ← 1 B ∑ i = 1 B L A U C G R ( i ) \mathcal{L}_{\mathrm{AUCGR}} \leftarrow\frac{1}{B}\sum_{i=1}^{B}\mathcal{L}_{\mathrm{AUCGR}}^{(i)} L AUCGR ← B 1 ∑ i = 1 B L AUCGR ( i ) .
Update the trainable modules Θ \Theta Θ .
end for
return the enhanced visual encoder.
Algorithm 2: pAUCGR Algorithm.
Input: Dataset D \mathcal{D} D ; existing generative reconstruction framework with trainable modules Θ \Theta Θ ; comparison measure d d d ; decreasing surrogate loss ℓ \ell ℓ ; threshold γ ( α ) \gamma(\alpha) γ ( α ) ; batch size B B B ; training iterations K K K .
Output: Enhanced visual encoder in the base framework.
for k = 1 k = 1 k = 1 to K K K do
Sample a mini-batch { x i } i = 1 B \{x_i\}_{i=1}^{B} { x i } i = 1 B from D \mathcal{D} D .
for i = 1 i = 1 i = 1 to B B B do
Use the base framework to obtain the anchor response ϵ ^ i \hat{\epsilon}_i ϵ ^ i .
Build the positive set P i \mathcal{P}_i P i and candidate negative set N i \mathcal{N}_i N i using the base framework.
for each v ∈ P i ∪ N i v \in\mathcal{P}_i \cup\mathcal{N}_i v ∈ P i ∪ N i do
Compute the anchor-wise score q i ( v ) ← d ( ϵ ^ i , v ) q_i(v) \leftarrow d(\hat{\epsilon}_i,v) q i ( v ) ← d ( ϵ ^ i , v ) .
end for
Retain reliable negatives N i ( α ) ← { n ∈ N i : q i ( n ) ≤ γ ( α ) } \mathcal{N}_i^{(\alpha)} \leftarrow\{n \in\mathcal{N}_i : q_i(n) \le\gamma(\alpha)\} N i ( α ) ← { n ∈ N i : q i ( n ) ≤ γ ( α )} .
Compute the pAUCGR anchor loss
L p A U C G R ( i ) ← 1 ∣ P i ∣ ∣ N i ( α ) ∣ ∑ p ∈ P i ∑ n ∈ N i ( α ) ℓ ( q i ( p ) − q i ( n ) ) . \mathcal{L}_{\mathrm{pAUCGR}}^{(i)} \leftarrow\frac{1}{|\mathcal{P}_i||\mathcal{N}_i^{(\alpha)}|}\sum_{p\in\mathcal{P}_i}\sum_{n\in\mathcal{N}_i^{(\alpha)}}\ell(q_i(p)-q_i(n)). L pAUCGR ( i ) ← ∣ P i ∣∣ N i ( α ) ∣ 1 p ∈ P i ∑ n ∈ N i ( α ) ∑ ℓ ( q i ( p ) − q i ( n )) .
end for
L p A U C G R ← 1 B ∑ i = 1 B L p A U C G R ( i ) \mathcal{L}_{\mathrm{pAUCGR}} \leftarrow\frac{1}{B}\sum_{i=1}^{B}\mathcal{L}_{\mathrm{pAUCGR}}^{(i)} L pAUCGR ← B 1 ∑ i = 1 B L pAUCGR ( i ) .
Update the trainable modules.
end for
return the enhanced visual encoder.
Proof of Theorem 1
Assumption 2 (Three-Component Orthogonality). Let
Z = α D z D + α P z P + α N z N , Z + = α D z D + α P z P + + α N z N + , (54) Z = \sqrt{\alpha_D}\,z_D + \sqrt{\alpha_P}\,z_P + \sqrt{\alpha_N}\,z_N,\qquad Z^+ = \sqrt{\alpha_D}\,z_D + \sqrt{\alpha_P}\,z_P^+ + \sqrt{\alpha_N}\,z_N^+, \tag*{(54)} Z = α D z D + α P z P + α N z N , Z + = α D z D + α P z P + + α N z N + , (54) where z D , z P , z N , z P + , z N + ∈ S d − 1 z_D,z_P,z_N,z_P^+,z_N^+ \in\mathbb{S}^{d-1} z D , z P , z N , z P + , z N + ∈ S d − 1 , α D , α P , α N ≥ 0 \alpha_D,\alpha_P,\alpha_N \ge0 α D , α P , α N ≥ 0 , and α D + α P + α N = 1 \alpha_D+\alpha_P+\alpha_N=1 α D + α P + α N = 1 . We impose the following conditions.
(i) z D z_D z D , z P z_P z P , and z N z_N z N are mutually independent and orthogonal.
(ii) The positive-view perceptual and noise components are conditionally centered in the sense that
E [ z P + ∣ z D , z P , z N ] = 0 , E [ z N + ∣ z D , z P , z N ] = 0. (55) \mathbb{E}[z_P^+ \mid z_D,z_P,z_N] = 0,\qquad\mathbb{E}[z_N^+ \mid z_D,z_P,z_N] = 0. \tag*{(55)} E [ z P + ∣ z D , z P , z N ] = 0 , E [ z N + ∣ z D , z P , z N ] = 0. (55) (iii) Conditioned on Z Z Z , the component of z P z_P z P orthogonal to Z Z Z is isotropic in Z ⊥ Z^\perp Z ⊥ .
Assumption 3 (Recoverability of the Reconstruction Target). For each diffusion step t t t , there exist constants λ , η > 0 \lambda,\eta> 0 λ , η > 0 and a measurable function r t ( ⋅ ) r_t(\cdot) r t ( ⋅ ) such that
ϵ t g t = r t ( x t , z D , t ) + λ z P + η α N z N . (56) \epsilon_t^{\mathrm{gt}} = r_t(x_t,z_D,t) + \lambda z_P + \eta\sqrt{\alpha_N}\,z_N. \tag*{(56)} ϵ t gt = r t ( x t , z D , t ) + λ z P + η α N z N . (56) Let c = h ω ( Z ) c=h_\omega(Z) c = h ω ( Z ) and U = ( x t , c , t ) U=(x_t,c,t) U = ( x t , c , t ) . We assume that
E [ z P ∣ U ] = E [ z P ∣ Z ] , (57) \mathbb{E}[z_P \mid U] = \mathbb{E}[z_P \mid Z], \tag*{(57)} E [ z P ∣ U ] = E [ z P ∣ Z ] , (57) and that the cross-noise residual is non-degenerate:
E [ ∥ z N − E [ z N ∣ U ] ∥ 2 2 ] ≥ m N (58) \mathbb{E}\left[\left\|z_N-\mathbb{E}[z_N\mid U]\right\|_2^2\right] \ge m_N \tag*{(58)} E [ ∥ z N − E [ z N ∣ U ] ∥ 2 2 ] ≥ m N (58) for some constant m N > 0 m_N>0 m N > 0 . Define
R D = r t ( x t , z D , t ) − E [ r t ( x t , z D , t ) ∣ U ] , R_D=r_t(x_t,z_D,t)-\mathbb{E}[r_t(x_t,z_D,t)\mid U], R D = r t ( x t , z D , t ) − E [ r t ( x t , z D , t ) ∣ U ] , R P = z P − E [ z P ∣ U ] , R_P=z_P-\mathbb{E}[z_P\mid U], R P = z P − E [ z P ∣ U ] , R N = z N − E [ z N ∣ U ] . (59) R_N=z_N-\mathbb{E}[z_N\mid U]. \tag*{(59)} R N = z N − E [ z N ∣ U ] . (59) We assume that these three residuals are conditionally pairwise orthogonal:
E [ ⟨ R a , R b ⟩ ∣ U ] = 0 , a ≠ b , a , b ∈ { D , P , N } . (60) \mathbb{E}[\langle R_a,R_b\rangle\mid U]=0,\qquad a\ne b,\quad a,b\in\{D,P,N\}. \tag*{(60)} E [⟨ R a , R b ⟩ ∣ U ] = 0 , a = b , a , b ∈ { D , P , N } . (60)
Lemma 1 (Conditional Mean Minimizes Mean-Squared Error). Let Y Y Y be a random vector and let U U U be an information variable. Then, for every U U U -measurable predictor Y ^ ( U ) \widehat{Y}(U) Y ( U ) ,
E [ ∥ Y − Y ^ ( U ) ∥ 2 2 ] ≥ E [ ∥ Y − E [ Y ∣ U ] ∥ 2 2 ] . (61) \mathbb{E}\left[\|Y-\widehat{Y}(U)\|_2^2\right]\ge\mathbb{E}\left[\|Y-\mathbb{E}[Y\mid U]\|_2^2\right]. \tag*{(61)} E [ ∥ Y − Y ( U ) ∥ 2 2 ] ≥ E [ ∥ Y − E [ Y ∣ U ] ∥ 2 2 ] . (61) Proof. Let m ( U ) = E [ Y ∣ U ] m(U)=\mathbb{E}[Y\mid U] m ( U ) = E [ Y ∣ U ] . Then
Y − Y ^ ( U ) = ( Y − m ( U ) ) + ( m ( U ) − Y ^ ( U ) ) . (62) Y-\widehat{Y}(U)=(Y-m(U))+(m(U)-\widehat{Y}(U)). \tag*{(62)} Y − Y ( U ) = ( Y − m ( U )) + ( m ( U ) − Y ( U )) . (62) Taking squared norms gives
∥ Y − Y ^ ( U ) ∥ 2 2 = ∥ Y − m ( U ) ∥ 2 2 + ∥ m ( U ) − Y ^ ( U ) ∥ 2 2 \|Y-\widehat{Y}(U)\|_2^2=\|Y-m(U)\|_2^2+\|m(U)-\widehat{Y}(U)\|_2^2 ∥ Y − Y ( U ) ∥ 2 2 = ∥ Y − m ( U ) ∥ 2 2 + ∥ m ( U ) − Y ( U ) ∥ 2 2 + 2 ⟨ Y − m ( U ) , m ( U ) − Y ^ ( U ) ⟩ . (63) \qquad+2\langle Y-m(U),m(U)-\widehat{Y}(U)\rangle. \tag*{(63)} + 2 ⟨ Y − m ( U ) , m ( U ) − Y ( U )⟩ . (63) We next take the conditional expectation with respect to U U U . Since m ( U ) − Y ^ ( U ) m(U)-\widehat{Y}(U) m ( U ) − Y ( U ) is U U U -measurable, we have
E [ ⟨ Y − m ( U ) , m ( U ) − Y ^ ( U ) ⟩ ∣ U ] = ⟨ E [ Y − m ( U ) ∣ U ] , m ( U ) − Y ^ ( U ) ⟩ \mathbb{E}\left[\left\langle Y-m(U),m(U)-\widehat{Y}(U)\right\rangle\mid U\right]
=\left\langle\mathbb{E}[Y-m(U)\mid U],m(U)-\widehat{Y}(U)\right\rangle E [ ⟨ Y − m ( U ) , m ( U ) − Y ( U ) ⟩ ∣ U ] = ⟨ E [ Y − m ( U ) ∣ U ] , m ( U ) − Y ( U ) ⟩ = ⟨ E [ Y ∣ U ] − m ( U ) , m ( U ) − Y ^ ( U ) ⟩ =\left\langle\mathbb{E}[Y\mid U]-m(U),m(U)-\widehat{Y}(U)\right\rangle = ⟨ E [ Y ∣ U ] − m ( U ) , m ( U ) − Y ( U ) ⟩ = 0. (64) =0. \tag*{(64)} = 0. (64) Therefore,
E [ ∥ Y − Y ^ ( U ) ∥ 2 2 ∣ U ] = E [ ∥ Y − m ( U ) ∥ 2 2 ∣ U ] + ∥ m ( U ) − Y ^ ( U ) ∥ 2 2 . (65) \mathbb{E}\left[\|Y-\widehat{Y}(U)\|_2^2\mid U\right]
=\mathbb{E}\left[\|Y-m(U)\|_2^2\mid U\right]+\|m(U)-\widehat{Y}(U)\|_2^2. \tag*{(65)} E [ ∥ Y − Y ( U ) ∥ 2 2 ∣ U ] = E [ ∥ Y − m ( U ) ∥ 2 2 ∣ U ] + ∥ m ( U ) − Y ( U ) ∥ 2 2 . (65) Taking the full expectation yields
E ∥ Y − Y ^ ( U ) ∥ 2 2 = E ∥ Y − m ( U ) ∥ 2 2 + E ∥ m ( U ) − Y ^ ( U ) ∥ 2 2 ≥ E ∥ Y − m ( U ) ∥ 2 2 . (66) \mathbb{E}\|Y-\widehat{Y}(U)\|_2^2=\mathbb{E}\|Y-m(U)\|_2^2+\mathbb{E}\|m(U)-\widehat{Y}(U)\|_2^2\ge\mathbb{E}\|Y-m(U)\|_2^2. \tag*{(66)} E ∥ Y − Y ( U ) ∥ 2 2 = E ∥ Y − m ( U ) ∥ 2 2 + E ∥ m ( U ) − Y ( U ) ∥ 2 2 ≥ E ∥ Y − m ( U ) ∥ 2 2 . (66) This completes the proof.
Lemma 2 (Perceptual Residual Under the Signal Decomposition). Under Assumption 2 , the perceptual component admits the decomposition
z P = α P Z + z P , ⊥ , z P , ⊥ ⊥ Z , ∥ z P , ⊥ ∥ 2 2 = 1 − α P . (67) z_P = \sqrt{\alpha_P}Z + z_{P,\perp}, \qquad z_{P,\perp} \perp Z, \qquad\lVert z_{P,\perp}\rVert_2^2 = 1-\alpha_P. \tag*{(67)} z P = α P Z + z P , ⊥ , z P , ⊥ ⊥ Z , ∥ z P , ⊥ ∥ 2 2 = 1 − α P . (67) Moreover,
E [ z P ∣ Z ] = α P Z , E [ ∥ z P − E [ z P ∣ Z ] ∥ 2 2 ] = 1 − α P . (68) \mathbb{E}[z_P \mid Z] = \sqrt{\alpha_P}Z, \qquad\mathbb{E}\left[\left\lVert z_P-\mathbb{E}[z_P\mid Z]\right\rVert_2^2\right] = 1-\alpha_P. \tag*{(68)} E [ z P ∣ Z ] = α P Z , E [ ∥ z P − E [ z P ∣ Z ] ∥ 2 2 ] = 1 − α P . (68) Proof. By Eq. (13) and Assumption 2,
⟨ z P , Z ⟩ = ⟨ z P , α D z D + α P z P + α N z N ⟩ = α D ⟨ z P , z D ⟩ + α P ⟨ z P , z P ⟩ + α N ⟨ z P , z N ⟩ = α P . (69) \begin{aligned}
\langle z_P,Z\rangle&= \left\langle z_P,\sqrt{\alpha_D}z_D+\sqrt{\alpha_P}z_P+\sqrt{\alpha_N}z_N\right\rangle\\
&= \sqrt{\alpha_D}\langle z_P,z_D\rangle+\sqrt{\alpha_P}\langle z_P,z_P\rangle+\sqrt{\alpha_N}\langle z_P,z_N\rangle\\
&= \sqrt{\alpha_P}. \tag*{(69)}
\end{aligned} ⟨ z P , Z ⟩ = ⟨ z P , α D z D + α P z P + α N z N ⟩ = α D ⟨ z P , z D ⟩ + α P ⟨ z P , z P ⟩ + α N ⟨ z P , z N ⟩ = α P . (69) Therefore, the projection of z P z_P z P onto the direction Z Z Z is α P Z \sqrt{\alpha_P}Z α P Z . Define
z P , ⊥ ≔ z P − α P Z . (70) z_{P,\perp} \coloneqq z_P-\sqrt{\alpha_P}Z. \tag*{(70)} z P , ⊥ : = z P − α P Z . (70) Then z P , ⊥ ⊥ Z z_{P,\perp}\perp Z z P , ⊥ ⊥ Z , and
z P = α P Z + z P , ⊥ . (71) z_P=\sqrt{\alpha_P}Z+z_{P,\perp}. \tag*{(71)} z P = α P Z + z P , ⊥ . (71) Since ∥ Z ∥ 2 = ∥ z P ∥ 2 = 1 \lVert Z\rVert_2=\lVert z_P\rVert_2=1 ∥ Z ∥ 2 = ∥ z P ∥ 2 = 1 and z P , ⊥ ⊥ Z z_{P,\perp}\perp Z z P , ⊥ ⊥ Z , the Pythagorean identity gives
1 = ∥ z P ∥ 2 2 = ∥ α P Z ∥ 2 2 + ∥ z P , ⊥ ∥ 2 2 = α P + ∥ z P , ⊥ ∥ 2 2 , (72) 1=\lVert z_P\rVert_2^2=\lVert\sqrt{\alpha_P}Z\rVert_2^2+\lVert z_{P,\perp}\rVert_2^2=\alpha_P+\lVert z_{P,\perp}\rVert_2^2, \tag*{(72)} 1 = ∥ z P ∥ 2 2 = ∥ α P Z ∥ 2 2 + ∥ z P , ⊥ ∥ 2 2 = α P + ∥ z P , ⊥ ∥ 2 2 , (72) which implies
∥ z P , ⊥ ∥ 2 2 = 1 − α P . (73) \lVert z_{P,\perp}\rVert_2^2=1-\alpha_P. \tag*{(73)} ∥ z P , ⊥ ∥ 2 2 = 1 − α P . (73) By item (iii) of Assumption 2, conditioned on Z Z Z , the vector z P , ⊥ z_{P,\perp} z P , ⊥ is isotropic in Z ⊥ Z^\perp Z ⊥ . Hence its conditional mean vanishes:
E [ z P , ⊥ ∣ Z ] = 0. (74) \mathbb{E}[z_{P,\perp}\mid Z]=0. \tag*{(74)} E [ z P , ⊥ ∣ Z ] = 0. (74) Combining this relation with Eq. (67) gives
E [ z P ∣ Z ] = E [ α P Z + z P , ⊥ ∣ Z ] = α P Z . (75) \mathbb{E}[z_P\mid Z]=\mathbb{E}[\sqrt{\alpha_P}Z+z_{P,\perp}\mid Z]=\sqrt{\alpha_P}Z. \tag*{(75)} E [ z P ∣ Z ] = E [ α P Z + z P , ⊥ ∣ Z ] = α P Z . (75) Finally,
z P − E [ z P ∣ Z ] = z P − α P Z = z P , ⊥ , (76) z_P-\mathbb{E}[z_P\mid Z]=z_P-\sqrt{\alpha_P}Z=z_{P,\perp}, \tag*{(76)} z P − E [ z P ∣ Z ] = z P − α P Z = z P , ⊥ , (76) so
E [ ∥ z P − E [ z P ∣ Z ] ∥ 2 2 ] = E ∥ z P , ⊥ ∥ 2 2 = 1 − α P . (77) \mathbb{E}\left[\left\lVert z_P-\mathbb{E}[z_P\mid Z]\right\rVert_2^2\right]=\mathbb{E}\lVert z_{P,\perp}\rVert_2^2=1-\alpha_P. \tag*{(77)} E [ ∥ z P − E [ z P ∣ Z ] ∥ 2 2 ] = E ∥ z P , ⊥ ∥ 2 2 = 1 − α P . (77) This completes the proof.
Lemma 3 (Lower Envelope of the Contrastive Objective). Under Assumption 2 , there exist constants c c o n ∈ R c_{\mathrm{con}}\in\mathbb{R} c con ∈ R and κ P , κ N > 0 \kappa_P,\kappa_N>0 κ P , κ N > 0 , which depend only on the temperature τ \tau τ and the number K K K of negatives, such that
L c o n ≥ c c o n + κ P α P + κ N α N . (78) \mathcal{L}_{\mathrm{con}}\ge c_{\mathrm{con}}+\kappa_P\alpha_P+\kappa_N\alpha_N. \tag*{(78)} L con ≥ c con + κ P α P + κ N α N . (78) Proof. Consider one anchor together with one positive and K K K negatives. Let
s + = sim ( Z , Z + ) , s k − = sim ( Z , Z k − ) , k = 1 , … , K . (79) s^+=\operatorname{sim}(Z,Z^+), \qquad s_k^-=\operatorname{sim}(Z,Z_k^-), \qquad k=1,\ldots,K. \tag*{(79)} s + = sim ( Z , Z + ) , s k − = sim ( Z , Z k − ) , k = 1 , … , K . (79) The corresponding InfoNCE term is
ℓ c o n = − log exp ( s + / τ ) exp ( s + / τ ) + ∑ k = 1 K exp ( s k − / τ ) . (80) \ell_{\mathrm{con}}=-\log\frac{\exp(s^+/\tau)}{\exp(s^+/\tau)+\sum_{k=1}^{K}\exp(s_k^-/\tau)}. \tag*{(80)} ℓ con = − log exp ( s + / τ ) + ∑ k = 1 K exp ( s k − / τ ) exp ( s + / τ ) . (80) Factoring out exp ( s + / τ ) \exp(s^+/\tau) exp ( s + / τ ) from the denominator yields
ℓ c o n = log ( 1 + ∑ k = 1 K exp ( s k − − s + τ ) ) . (81) \ell_{\mathrm{con}}=\log\left(1+\sum_{k=1}^{K}\exp\left(\frac{s_k^- -s^+}{\tau}\right)\right). \tag*{(81)} ℓ con = log ( 1 + k = 1 ∑ K exp ( τ s k − − s + ) ) . (81) Since all similarities lie in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] , we have s k − ≥ − 1 s_k^-\ge-1 s k − ≥ − 1 for every k k k . Therefore,
ℓ c o n ≥ log ( 1 + K exp ( − 1 − s + τ ) ) . (82) \ell_{\mathrm{con}}\ge\log\left(1+K\exp\left(\frac{-1-s^+}{\tau}\right)\right). \tag*{(82)} ℓ con ≥ log ( 1 + K exp ( τ − 1 − s + ) ) . (82)
We next compute the positive similarity. By Eq. (54) ,
s + = ⟨ α D z D + α P z P + α N z N , α D z D + α P z P + + α N z N + ⟩ = α D ⟨ z D , z D ⟩ + α P α D ( ⟨ z D , z P + ⟩ + ⟨ z P , z D ⟩ ) + α N α D ( ⟨ z D , z N + ⟩ + ⟨ z N , z D ⟩ ) + α P ⟨ z P , z P + ⟩ + α N ⟨ z N , z N + ⟩ + α P α N ( ⟨ z P , z N + ⟩ + ⟨ z N , z P + ⟩ ) . (83) \begin{aligned}
s^{+} &= \left\langle\sqrt{\alpha_D}z_D+\sqrt{\alpha_P}z_P+\sqrt{\alpha_N}z_N,\sqrt{\alpha_D}z_D+\sqrt{\alpha_P}z_P^{+}+\sqrt{\alpha_N}z_N^{+}\right\rangle\\
&=\alpha_D\langle z_D,z_D\rangle+\sqrt{\alpha_P\alpha_D}\left(\langle z_D,z_P^{+}\rangle+\langle z_P,z_D\rangle\right)+\sqrt{\alpha_N\alpha_D}\left(\langle z_D,z_N^{+}\rangle+\langle z_N,z_D\rangle\right)\\
&\quad+\alpha_P\langle z_P,z_P^{+}\rangle+\alpha_N\langle z_N,z_N^{+}\rangle\\
&\quad+\sqrt{\alpha_P\alpha_N}\left(\langle z_P,z_N^{+}\rangle+\langle z_N,z_P^{+}\rangle\right). \tag*{(83)}
\end{aligned} s + = ⟨ α D z D + α P z P + α N z N , α D z D + α P z P + + α N z N + ⟩ = α D ⟨ z D , z D ⟩ + α P α D ( ⟨ z D , z P + ⟩ + ⟨ z P , z D ⟩ ) + α N α D ( ⟨ z D , z N + ⟩ + ⟨ z N , z D ⟩ ) + α P ⟨ z P , z P + ⟩ + α N ⟨ z N , z N + ⟩ + α P α N ( ⟨ z P , z N + ⟩ + ⟨ z N , z P + ⟩ ) . (83)
Taking the conditional expectation with respect to z D z_D z D , z P z_P z P , z N z_N z N , and using Eq. (55) , we obtain
E [ s + ∣ z D , z P , z N ] = α D = 1 − α P − α N . (84) \mathbb{E}\left[s^{+}\mid z_D,z_P,z_N\right]=\alpha_D=1-\alpha_P-\alpha_N. \tag*{(84)} E [ s + ∣ z D , z P , z N ] = α D = 1 − α P − α N . (84)
Define
g ( u ) = log ( 1 + K exp ( − 1 − u τ ) ) . (85) g(u)=\log\left(1+K\exp\left(\frac{-1-u}{\tau}\right)\right). \tag*{(85)} g ( u ) = log ( 1 + K exp ( τ − 1 − u ) ) . (85)
Then Eq. (82) becomes ℓ c o n ≥ g ( s + ) \ell_{\mathrm{con}}\ge g(s^{+}) ℓ con ≥ g ( s + ) . A direct calculation gives
g ′ ′ ( u ) = K exp ( − 1 − u τ ) τ 2 ( 1 + K exp ( − 1 − u τ ) ) 2 > 0 , (86) g^{\prime\prime}(u)=\frac{K\exp\left(\frac{-1-u}{\tau}\right)}{\tau^2\left(1+K\exp\left(\frac{-1-u}{\tau}\right)\right)^2}>0, \tag*{(86)} g ′′ ( u ) = τ 2 ( 1 + K exp ( τ − 1 − u ) ) 2 K exp ( τ − 1 − u ) > 0 , (86)
so g g g is convex. Applying Jensen’s inequality together with Eq. (84) gives
E [ ℓ c o n ∣ z D , z P , z N ] ≥ E [ g ( s + ) ∣ z D , z P , z N ] ≥ g ( E [ s + ∣ z D , z P , z N ] ) = log ( 1 + K exp ( − 2 + α P + α N τ ) ) . (87) \begin{aligned}
\mathbb{E}\left[\ell_{\mathrm{con}}\mid z_D,z_P,z_N\right]&\ge\mathbb{E}\left[g(s^{+})\mid z_D,z_P,z_N\right]\\
&\ge g\left(\mathbb{E}\left[s^{+}\mid z_D,z_P,z_N\right]\right)\\
&=\log\left(1+K\exp\left(\frac{-2+\alpha_P+\alpha_N}{\tau}\right)\right). \tag*{(87)}
\end{aligned} E [ ℓ con ∣ z D , z P , z N ] ≥ E [ g ( s + ) ∣ z D , z P , z N ] ≥ g ( E [ s + ∣ z D , z P , z N ] ) = log ( 1 + K exp ( τ − 2 + α P + α N ) ) . (87)
Set
C = K exp ( − 2 τ ) , h ( x ) = log ( 1 + C e x ) . (88) C=K\exp\left(-\frac{2}{\tau}\right),\qquad h(x)=\log(1+Ce^x). \tag*{(88)} C = K exp ( − τ 2 ) , h ( x ) = log ( 1 + C e x ) . (88)
Then Eq. (87) becomes
E [ ℓ c o n ∣ z D , z P , z N ] ≥ h ( α P + α N τ ) . (89) \mathbb{E}\left[\ell_{\mathrm{con}}\mid z_D,z_P,z_N\right]\ge h\left(\frac{\alpha_P+\alpha_N}{\tau}\right). \tag*{(89)} E [ ℓ con ∣ z D , z P , z N ] ≥ h ( τ α P + α N ) . (89)
Since
h ′ ( x ) = C e x 1 + C e x , (90) h^{\prime}(x)=\frac{Ce^x}{1+Ce^x}, \tag*{(90)} h ′ ( x ) = 1 + C e x C e x , (90)
we have
h ′ ( x ) ≥ h ′ ( 0 ) = C 1 + C for all x ≥ 0. (91) h^{\prime}(x)\ge h^{\prime}(0)=\frac{C}{1+C}\qquad\text{for all }x\ge0. \tag*{(91)} h ′ ( x ) ≥ h ′ ( 0 ) = 1 + C C for all x ≥ 0. (91)
Therefore,
h ( x ) = h ( 0 ) + ∫ 0 x h ′ ( u ) d u ≥ h ( 0 ) + C 1 + C x . (92) \begin{aligned}
h(x)&=h(0)+\int_0^x h^{\prime}(u)\,du\\
&\ge h(0)+\frac{C}{1+C}x. \tag*{(92)}
\end{aligned} h ( x ) = h ( 0 ) + ∫ 0 x h ′ ( u ) d u ≥ h ( 0 ) + 1 + C C x . (92)
Substituting x = ( α P + α N ) / τ x=(\alpha_P+\alpha_N)/\tau x = ( α P + α N ) / τ gives
E [ ℓ c o n ∣ z D , z P , z N ] ≥ log ( 1 + C ) + C τ ( 1 + C ) ( α P + α N ) . (93) \mathbb{E}\left[\ell_{\mathrm{con}}\mid z_D,z_P,z_N\right]\ge\log(1+C)+\frac{C}{\tau(1+C)}(\alpha_P+\alpha_N). \tag*{(93)} E [ ℓ con ∣ z D , z P , z N ] ≥ log ( 1 + C ) + τ ( 1 + C ) C ( α P + α N ) . (93)
Finally, taking the full expectation over anchors yields
L c o n ≥ log ( 1 + C ) + C τ ( 1 + C ) ( α P + α N ) . (94) L_{\mathrm{con}}\ge\log(1+C)+\frac{C}{\tau(1+C)}(\alpha_P+\alpha_N). \tag*{(94)} L con ≥ log ( 1 + C ) + τ ( 1 + C ) C ( α P + α N ) . (94)
Hence Eq. (78) holds with
c c o n = log ( 1 + C ) , κ P = κ N = C τ ( 1 + C ) . (95) c_{\mathrm{con}}=\log(1+C),\qquad\kappa_P=\kappa_N=\frac{C}{\tau(1+C)}. \tag*{(95)} c con = log ( 1 + C ) , κ P = κ N = τ ( 1 + C ) C . (95)
This completes the proof.
Lemma 4 (Lower Envelope of the Reconstruction Objective). Under Assumptions 2 and 3, there exist constants c r e c ∈ R c_{\mathrm{rec}}\in\mathbb{R} c rec ∈ R and κ D , κ N ′ > 0 \kappa_D,\kappa_N^{\prime}>0 κ D , κ N ′ > 0 such that
L r e c ≥ c r e c + κ D ( 1 − α P ) + κ N ′ α N . (96) L_{\mathrm{rec}}\ge c_{\mathrm{rec}}+\kappa_D(1-\alpha_P)+\kappa_N^{\prime}\alpha_N. \tag*{(96)} L rec ≥ c rec + κ D ( 1 − α P ) + κ N ′ α N . (96)
Proof. Recall from the main text that
L r e c = E t [ ∥ ϵ ^ θ − ϵ t g t ∥ 2 2 ] . (97) \mathcal{L}_{\mathrm{rec}}=\mathbb{E}_{t}\left[\left\|\hat{\epsilon}_{\theta}-\epsilon_{t}^{\mathrm{gt}}\right\|_{2}^{2}\right]. \tag*{(97)} L rec = E t [ ϵ ^ θ − ϵ t gt 2 2 ] . (97)
Let
U = ( x t , c , t ) , Y = ϵ t g t , Y ^ ( U ) = ϵ ^ θ . (98) U=(x_{t},c,t),\qquad Y=\epsilon_{t}^{\mathrm{gt}},\qquad\hat{Y}(U)=\hat{\epsilon}_{\theta}. \tag*{(98)} U = ( x t , c , t ) , Y = ϵ t gt , Y ^ ( U ) = ϵ ^ θ . (98)
By Lemma 1,
L r e c ≥ E ∥ ϵ t g t − E [ ϵ t g t ∣ U ] ∥ 2 2 . (99) \mathcal{L}_{\mathrm{rec}}\geq\mathbb{E}\left\|\epsilon_{t}^{\mathrm{gt}}-\mathbb{E}\left[\epsilon_{t}^{\mathrm{gt}}\mid U\right]\right\|_{2}^{2}. \tag*{(99)} L rec ≥ E ϵ t gt − E [ ϵ t gt ∣ U ] 2 2 . (99)
We next expand the conditional residual. By Eq. (56) ,
ϵ t g t − E [ ϵ t g t ∣ U ] = ( r t ( x t , z D , t ) − E [ r t ( x t , z D , t ) ∣ U ] ) + λ ( z P − E [ z P ∣ U ] ) + η α N ( z N − E [ z N ∣ U ] ) . (100) \epsilon_{t}^{\mathrm{gt}}-\mathbb{E}\left[\epsilon_{t}^{\mathrm{gt}}\mid U\right]=\left(r_{t}(x_{t},z_{D},t)-\mathbb{E}\left[r_{t}(x_{t},z_{D},t)\mid U\right]\right)+\lambda\left(z_{P}-\mathbb{E}\left[z_{P}\mid U\right]\right)+\eta\sqrt{\alpha_{N}}\left(z_{N}-\mathbb{E}\left[z_{N}\mid U\right]\right). \tag*{(100)} ϵ t gt − E [ ϵ t gt ∣ U ] = ( r t ( x t , z D , t ) − E [ r t ( x t , z D , t ) ∣ U ] ) + λ ( z P − E [ z P ∣ U ] ) + η α N ( z N − E [ z N ∣ U ] ) . (100)
Taking squared norms and then conditioning on U U U gives
E [ ∥ ϵ t g t − E [ ϵ t g t ∣ U ] ∥ 2 2 ∣ U ] = E [ ∥ r t ( x t , z D , t ) − E [ r t ( x t , z D , t ) ∣ U ] ∥ 2 2 ∣ U ] + λ 2 E [ ∥ z P − E [ z P ∣ U ] ∥ 2 2 ∣ U ] + η 2 α N E [ ∥ z N − E [ z N ∣ U ] ∥ 2 2 ∣ U ] (101) \begin{aligned}
\mathbb{E}\left[\left\|\epsilon_{t}^{\mathrm{gt}}-\mathbb{E}\left[\epsilon_{t}^{\mathrm{gt}}\mid U\right]\right\|_{2}^{2}\mid U\right]
={}&\mathbb{E}\left[\left\|r_{t}(x_{t},z_{D},t)-\mathbb{E}\left[r_{t}(x_{t},z_{D},t)\mid U\right]\right\|_{2}^{2}\mid U\right]\\
&+\lambda^{2}\mathbb{E}\left[\left\|z_{P}-\mathbb{E}\left[z_{P}\mid U\right]\right\|_{2}^{2}\mid U\right]\\
&+\eta^{2}\alpha_{N}\mathbb{E}\left[\left\|z_{N}-\mathbb{E}\left[z_{N}\mid U\right]\right\|_{2}^{2}\mid U\right] \tag*{(101)}
\end{aligned} E [ ϵ t gt − E [ ϵ t gt ∣ U ] 2 2 ∣ U ] = E [ ∥ r t ( x t , z D , t ) − E [ r t ( x t , z D , t ) ∣ U ] ∥ 2 2 ∣ U ] + λ 2 E [ ∥ z P − E [ z P ∣ U ] ∥ 2 2 ∣ U ] + η 2 α N E [ ∥ z N − E [ z N ∣ U ] ∥ 2 2 ∣ U ] (101)
where all cross terms vanish by Eq. (60) . Hence
E [ ∥ ϵ t g t − E [ ϵ t g t ∣ U ] ∥ 2 2 ∣ U ] ≥ λ 2 E [ ∥ z P − E [ z P ∣ U ] ∥ 2 2 ∣ U ] + η 2 α N E [ ∥ z N − E [ z N ∣ U ] ∥ 2 2 ∣ U ] . (102) \mathbb{E}\left[\left\|\epsilon_{t}^{\mathrm{gt}}-\mathbb{E}\left[\epsilon_{t}^{\mathrm{gt}}\mid U\right]\right\|_{2}^{2}\mid U\right]\geq\lambda^{2}\mathbb{E}\left[\left\|z_{P}-\mathbb{E}\left[z_{P}\mid U\right]\right\|_{2}^{2}\mid U\right]+\eta^{2}\alpha_{N}\mathbb{E}\left[\left\|z_{N}-\mathbb{E}\left[z_{N}\mid U\right]\right\|_{2}^{2}\mid U\right]. \tag*{(102)} E [ ϵ t gt − E [ ϵ t gt ∣ U ] 2 2 ∣ U ] ≥ λ 2 E [ ∥ z P − E [ z P ∣ U ] ∥ 2 2 ∣ U ] + η 2 α N E [ ∥ z N − E [ z N ∣ U ] ∥ 2 2 ∣ U ] . (102)
Taking the full expectation and using Eq. (57) yields
L r e c ≥ λ 2 E ∥ z P − E [ z P ∣ Z ] ∥ 2 2 + η 2 α N E ∥ z N − E [ z N ∣ U ] ∥ 2 2 . (103) \mathcal{L}_{\mathrm{rec}}\geq\lambda^{2}\mathbb{E}\left\|z_{P}-\mathbb{E}\left[z_{P}\mid Z\right]\right\|_{2}^{2}+\eta^{2}\alpha_{N}\mathbb{E}\left\|z_{N}-\mathbb{E}\left[z_{N}\mid U\right]\right\|_{2}^{2}. \tag*{(103)} L rec ≥ λ 2 E ∥ z P − E [ z P ∣ Z ] ∥ 2 2 + η 2 α N E ∥ z N − E [ z N ∣ U ] ∥ 2 2 . (103)
Combining Lemma 2 with Eq. (58) then implies
L r e c ≥ λ 2 ( 1 − α P ) + η 2 m N α N . (104) \mathcal{L}_{\mathrm{rec}}\geq\lambda^{2}(1-\alpha_{P})+\eta^{2}m_{N}\alpha_{N}. \tag*{(104)} L rec ≥ λ 2 ( 1 − α P ) + η 2 m N α N . (104)
Therefore, Eq. (96) holds with
c r e c = 0 , κ D = λ 2 , κ N ′ = η 2 m N . (105) c_{\mathrm{rec}}=0,\qquad\kappa_{D}=\lambda^{2},\qquad\kappa_{N}^{\prime}=\eta^{2}m_{N}. \tag*{(105)} c rec = 0 , κ D = λ 2 , κ N ′ = η 2 m N . (105)
This completes the proof.
Restate of Theorem 1. Under Assumptions 2 and 3, consider the representation model in Definition 1. There exist constants c c o n , c r e c ∈ R c_{\mathrm{con}},c_{\mathrm{rec}}\in\mathbb{R} c con , c rec ∈ R and κ P , κ N , κ D , κ N ′ > 0 \kappa_{P},\kappa_{N},\kappa_{D},\kappa_{N}^{\prime}>0 κ P , κ N , κ D , κ N ′ > 0 . With these constants, the achievable envelopes of the two objectives satisfy
inf L c o n ≥ c c o n + κ P α P + κ N α N , inf L r e c ≥ c r e c + κ D ( 1 − α P ) + κ N ′ α N , (106) \begin{aligned}
\inf\mathcal{L}_{\mathrm{con}}&\geq c_{\mathrm{con}}+\kappa_{P}\alpha_{P}+\kappa_{N}\alpha_{N},\\
\inf\mathcal{L}_{\mathrm{rec}}&\geq c_{\mathrm{rec}}+\kappa_{D}(1-\alpha_{P})+\kappa_{N}^{\prime}\alpha_{N}, \tag*{(106)}
\end{aligned} inf L con inf L rec ≥ c con + κ P α P + κ N α N , ≥ c rec + κ D ( 1 − α P ) + κ N ′ α N , (106) where the infimum is taken over all admissible model parameters, optimizing each objective with the shared representation Z Z Z . This result implies that the two objectives favor different allocation regimes:
Proof. Lemma 3 shows that every admissible choice of model parameters satisfies
L c o n ≥ c c o n + κ P α P + κ N α N (107) \mathcal{L}_{\mathrm{con}}\geq c_{\mathrm{con}}+\kappa_{P}\alpha_{P}+\kappa_{N}\alpha_{N} \tag*{(107)} L con ≥ c con + κ P α P + κ N α N (107)
for some constants c c o n ∈ R c_{\mathrm{con}}\in\mathbb{R} c con ∈ R and κ P , κ N > 0 \kappa_{P},\kappa_{N}>0 κ P , κ N > 0 . These constants do not depend on the particular parameter choice. Taking the infimum over all admissible parameters preserves this lower bound, and therefore
inf L c o n ≥ c c o n + κ P α P + κ N α N . (108) \inf\mathcal{L}_{\mathrm{con}}\geq c_{\mathrm{con}}+\kappa_{P}\alpha_{P}+\kappa_{N}\alpha_{N}. \tag*{(108)} inf L con ≥ c con + κ P α P + κ N α N . (108)
Lemma 4 similarly gives
L r e c ≥ c r e c + κ D ( 1 − α P ) + κ N ′ α N (109) \mathcal{L}_{\mathrm{rec}}\geq c_{\mathrm{rec}}+\kappa_{D}(1-\alpha_{P})+\kappa_{N}^{\prime}\alpha_{N} \tag*{(109)} L rec ≥ c rec + κ D ( 1 − α P ) + κ N ′ α N (109)
for constants c r e c ∈ R c_{\mathrm{rec}}\in\mathbb{R} c rec ∈ R and κ D , κ N ′ > 0 \kappa_{D},\kappa_{N}^{\prime}>0 κ D , κ N ′ > 0 that are independent of the admissible parameter choice. Taking the infimum again yields
inf L r e c ≥ c r e c + κ D ( 1 − α P ) + κ N ′ α N . (110) \inf\mathcal{L}_{\mathrm{rec}}\geq c_{\mathrm{rec}}+\kappa_{D}(1-\alpha_{P})+\kappa_{N}^{\prime}\alpha_{N}. \tag*{(110)} inf L rec ≥ c rec + κ D ( 1 − α P ) + κ N ′ α N . (110)
The first lower envelope is an increasing function of α P \alpha_{P} α P . Therefore, minimizing L c o n \mathcal{L}_{\mathrm{con}} L con favors smaller α P \alpha_{P} α P . The second lower envelope is a decreasing function of α P \alpha_{P} α P . Therefore, minimizing L r e c \mathcal{L}_{\mathrm{rec}} L rec favors larger α P \alpha_{P} α P . The two objectives thus prefer incompatible allocations of the shared representation. This is the structural source of the objective conflict in the naive method.
This completes the proof.
Proof of Theorem 2
Definition 2 (Bi-Lipschitz Mapping [138 ] ). Let T : X → Y T:\mathcal{X}\to\mathcal{Y} T : X → Y be a mapping between two metric spaces ( X , ∥ ⋅ ∥ X ) (\mathcal{X},\|\cdot\|_{\mathcal{X}}) ( X , ∥ ⋅ ∥ X ) and ( Y , ∥ ⋅ ∥ Y ) (\mathcal{Y},\|\cdot\|_{\mathcal{Y}}) ( Y , ∥ ⋅ ∥ Y ) . We say that T T T is bi-Lipschitz on a subset M ⊆ X \mathcal{M}\subseteq\mathcal{X} M ⊆ X if there exist constants 0 < m ≤ L < ∞ 0<m\leq L<\infty 0 < m ≤ L < ∞ such that, for all z 1 , z 2 ∈ M z_1,z_2\in\mathcal{M} z 1 , z 2 ∈ M ,
m ∥ z 1 − z 2 ∥ X ≤ ∥ T ( z 1 ) − T ( z 2 ) ∥ Y ≤ L ∥ z 1 − z 2 ∥ X . (111) m\|z_1-z_2\|_{\mathcal{X}}\leq\|T(z_1)-T(z_2)\|_{\mathcal{Y}}\leq L\|z_1-z_2\|_{\mathcal{X}}. \tag*{(111)} m ∥ z 1 − z 2 ∥ X ≤ ∥ T ( z 1 ) − T ( z 2 ) ∥ Y ≤ L ∥ z 1 − z 2 ∥ X . (111) In particular, a bi-Lipschitz map preserves pairwise distances up to constant factors and is injective on M \mathcal{M} M .
Assumption 4. Fix a diffusion step t t t and define T ( z ) = ϵ θ ( x ~ t , h ω ( z ) , t ) T(z)=\epsilon_\theta(\tilde{x}_t,h_\omega(z),t) T ( z ) = ϵ θ ( x ~ t , h ω ( z ) , t ) . According to Definition 2, we assume that T ( z ) T(z) T ( z ) satisfies the bi-Lipschitz property on a subset that contains all feature samples and their class means.
Lemma 5 (Equivalent Variance Identity [139 ] ). Let e 1 , … , e n ∈ R d e_1,\ldots,e_n\in\mathbb{R}^d e 1 , … , e n ∈ R d be arbitrary vectors and denote their mean by e ˉ = 1 n ∑ i = 1 n e i \bar{e}=\frac{1}{n}\sum_{i=1}^{n}e_i e ˉ = n 1 ∑ i = 1 n e i . Then the empirical variance can be written as the average of all pairwise squared distances:
∑ i = 1 n ∥ e i − e ˉ ∥ 2 2 = 1 2 n ∑ i = 1 n ∑ j = 1 n ∥ e i − e j ∥ 2 2 . (112) \sum_{i=1}^{n}\|e_i-\bar{e}\|_2^2=\frac{1}{2n}\sum_{i=1}^{n}\sum_{j=1}^{n}\|e_i-e_j\|_2^2. \tag*{(112)} i = 1 ∑ n ∥ e i − e ˉ ∥ 2 2 = 2 n 1 i = 1 ∑ n j = 1 ∑ n ∥ e i − e j ∥ 2 2 . (112)
Proof. For the left-hand side , using e ˉ = 1 n ∑ k = 1 n e k \bar{e}=\frac{1}{n}\sum_{k=1}^{n}e_k e ˉ = n 1 ∑ k = 1 n e k , we have:
∑ i = 1 n ∥ e i − e ˉ ∥ 2 2 = ∑ i = 1 n ⟨ e i − e ˉ , e i − e ˉ ⟩ = ∑ i = 1 n ( ∥ e i ∥ 2 2 + ∥ e ˉ ∥ 2 2 − 2 ⟨ e i , e ˉ ⟩ ) = ∑ i = 1 n ∥ e i ∥ 2 2 + n ∥ e ˉ ∥ 2 2 − 2 ⟨ ∑ i = 1 n e i , e ˉ ⟩ . (113) \begin{aligned}
\sum_{i=1}^{n}\|e_i-\bar{e}\|_2^2
&=\sum_{i=1}^{n}\langle e_i-\bar{e},e_i-\bar{e}\rangle\\
&=\sum_{i=1}^{n}\left(\|e_i\|_2^2+\|\bar{e}\|_2^2-2\langle e_i,\bar{e}\rangle\right)\\
&=\sum_{i=1}^{n}\|e_i\|_2^2+n\|\bar{e}\|_2^2-2\left\langle\sum_{i=1}^{n}e_i,\bar{e}\right\rangle. \tag*{(113)}
\end{aligned} i = 1 ∑ n ∥ e i − e ˉ ∥ 2 2 = i = 1 ∑ n ⟨ e i − e ˉ , e i − e ˉ ⟩ = i = 1 ∑ n ( ∥ e i ∥ 2 2 + ∥ e ˉ ∥ 2 2 − 2 ⟨ e i , e ˉ ⟩ ) = i = 1 ∑ n ∥ e i ∥ 2 2 + n ∥ e ˉ ∥ 2 2 − 2 ⟨ i = 1 ∑ n e i , e ˉ ⟩ . (113)
Since ∑ i = 1 n e i = n e ˉ \sum_{i=1}^{n}e_i=n\bar{e} ∑ i = 1 n e i = n e ˉ , it follows that
⟨ ∑ i = 1 n e i , e ˉ ⟩ = ⟨ n e ˉ , e ˉ ⟩ = n ∥ e ˉ ∥ 2 2 . (114) \left\langle\sum_{i=1}^{n}e_i,\bar{e}\right\rangle=\langle n\bar{e},\bar{e}\rangle=n\|\bar{e}\|_2^2. \tag*{(114)} ⟨ i = 1 ∑ n e i , e ˉ ⟩ = ⟨ n e ˉ , e ˉ ⟩ = n ∥ e ˉ ∥ 2 2 . (114)
Therefore,
∑ i = 1 n ∥ e i − e ˉ ∥ 2 2 = ∑ i = 1 n ∥ e i ∥ 2 2 − n ∥ e ˉ ∥ 2 2 . (115) \sum_{i=1}^{n}\|e_i-\bar{e}\|_2^2=\sum_{i=1}^{n}\|e_i\|_2^2-n\|\bar{e}\|_2^2. \tag*{(115)} i = 1 ∑ n ∥ e i − e ˉ ∥ 2 2 = i = 1 ∑ n ∥ e i ∥ 2 2 − n ∥ e ˉ ∥ 2 2 . (115)
For the right-hand side , using ∥ e i − e j ∥ 2 2 = ∥ e i ∥ 2 2 + ∥ e j ∥ 2 2 − 2 ⟨ e i , e j ⟩ \|e_i-e_j\|_2^2=\|e_i\|_2^2+\|e_j\|_2^2-2\langle e_i,e_j\rangle ∥ e i − e j ∥ 2 2 = ∥ e i ∥ 2 2 + ∥ e j ∥ 2 2 − 2 ⟨ e i , e j ⟩ , we obtain
∑ i = 1 n ∑ j = 1 n ∥ e i − e j ∥ 2 2 = ∑ i , j ∥ e i ∥ 2 2 + ∑ i , j ∥ e j ∥ 2 2 − 2 ∑ i , j ⟨ e i , e j ⟩ = n ∑ i = 1 n ∥ e i ∥ 2 2 + n ∑ j = 1 n ∥ e j ∥ 2 2 − 2 ⟨ ∑ i = 1 n e i , ∑ j = 1 n e j ⟩ = 2 n ∑ i = 1 n ∥ e i ∥ 2 2 − 2 ∥ ∑ i = 1 n e i ∥ 2 2 . (116) \begin{aligned}
\sum_{i=1}^{n}\sum_{j=1}^{n}\|e_i-e_j\|_2^2
&=\sum_{i,j}\|e_i\|_2^2+\sum_{i,j}\|e_j\|_2^2-2\sum_{i,j}\langle e_i,e_j\rangle\\
&=n\sum_{i=1}^{n}\|e_i\|_2^2+n\sum_{j=1}^{n}\|e_j\|_2^2-2\left\langle\sum_{i=1}^{n}e_i,\sum_{j=1}^{n}e_j\right\rangle\\
&=2n\sum_{i=1}^{n}\|e_i\|_2^2-2\left\|\sum_{i=1}^{n}e_i\right\|_2^2. \tag*{(116)}
\end{aligned} i = 1 ∑ n j = 1 ∑ n ∥ e i − e j ∥ 2 2 = i , j ∑ ∥ e i ∥ 2 2 + i , j ∑ ∥ e j ∥ 2 2 − 2 i , j ∑ ⟨ e i , e j ⟩ = n i = 1 ∑ n ∥ e i ∥ 2 2 + n j = 1 ∑ n ∥ e j ∥ 2 2 − 2 ⟨ i = 1 ∑ n e i , j = 1 ∑ n e j ⟩ = 2 n i = 1 ∑ n ∥ e i ∥ 2 2 − 2 i = 1 ∑ n e i 2 2 . (116)
Again using ∑ i = 1 n e i = n e ˉ \sum_{i=1}^{n}e_i=n\bar{e} ∑ i = 1 n e i = n e ˉ , we obtain
∥ ∑ i = 1 n e i ∥ 2 2 = ∥ n e ˉ ∥ 2 2 = n 2 ∥ e ˉ ∥ 2 2 , (117) \left\|\sum_{i=1}^{n}e_i\right\|_2^2=\|n\bar{e}\|_2^2=n^2\|\bar{e}\|_2^2, \tag*{(117)} i = 1 ∑ n e i 2 2 = ∥ n e ˉ ∥ 2 2 = n 2 ∥ e ˉ ∥ 2 2 , (117)
so
∑ i = 1 n ∑ j = 1 n ∥ e i − e j ∥ 2 2 = 2 n ∑ i = 1 n ∥ e i ∥ 2 2 − 2 n 2 ∥ e ˉ ∥ 2 2 . (118) \sum_{i=1}^{n}\sum_{j=1}^{n}\|e_i-e_j\|_2^2=2n\sum_{i=1}^{n}\|e_i\|_2^2-2n^2\|\bar{e}\|_2^2. \tag*{(118)} i = 1 ∑ n j = 1 ∑ n ∥ e i − e j ∥ 2 2 = 2 n i = 1 ∑ n ∥ e i ∥ 2 2 − 2 n 2 ∥ e ˉ ∥ 2 2 . (118)
Dividing Eq. (118) by 2 n 2n 2 n yields
1 2 n ∑ i = 1 n ∑ j = 1 n ∥ e i − e j ∥ 2 2 = ∑ i = 1 n ∥ e i ∥ 2 2 − n ∥ e ˉ ∥ 2 2 . (119) \frac{1}{2n}\sum_{i=1}^{n}\sum_{j=1}^{n}\|e_i-e_j\|_2^2=\sum_{i=1}^{n}\|e_i\|_2^2-n\|\bar{e}\|_2^2. \tag*{(119)} 2 n 1 i = 1 ∑ n j = 1 ∑ n ∥ e i − e j ∥ 2 2 = i = 1 ∑ n ∥ e i ∥ 2 2 − n ∥ e ˉ ∥ 2 2 . (119)
Therefore, by combining Eq. (115) and Eq. (119) , we can obtain:
∑ i = 1 n ∥ e i − e ˉ ∥ 2 2 = 1 2 n ∑ i = 1 n ∑ j = 1 n ∥ e i − e j ∥ 2 2 . (120) \sum_{i=1}^{n}\lVert e_i-\bar{e}\rVert_2^2=\frac{1}{2n}\sum_{i=1}^{n}\sum_{j=1}^{n}\lVert e_i-e_j\rVert_2^2. \tag*{(120)} i = 1 ∑ n ∥ e i − e ˉ ∥ 2 2 = 2 n 1 i = 1 ∑ n j = 1 ∑ n ∥ e i − e j ∥ 2 2 . (120)
This completes the proof.
Restate of Theorem 2 (AUCGR Supports D -Ability). Fix a diffusion step t t t and define T ( z ) = ϵ θ ( x ~ t , h ω ( z ) , t ) T(z)=\epsilon_\theta(\tilde{x}_t,h_\omega(z),t) T ( z ) = ϵ θ ( x ~ t , h ω ( z ) , t ) . Under Assumption 4, the intra-class scatter and inter-class scatter in the feature space ( S i n n e r , S i n t e r ) (S_{\mathrm{inner}},S_{\mathrm{inter}}) ( S inner , S inter ) can be bounded by those in the noise space ( S i n n e r ( ϵ ) ( t ) , S i n t e r ( ϵ ) ( t ) ) (S_{\mathrm{inner}}^{(\epsilon)}(t),S_{\mathrm{inter}}^{(\epsilon)}(t)) ( S inner ( ϵ ) ( t ) , S inter ( ϵ ) ( t )) as
S i n n e r ≤ 1 m 2 S i n n e r ( ϵ ) ( t ) , S_{\mathrm{inner}}\leq\frac{1}{m^2}S_{\mathrm{inner}}^{(\epsilon)}(t), S inner ≤ m 2 1 S inner ( ϵ ) ( t ) , S i n t e r ≥ κ S i n t e r ( ϵ ) ( t ) − η S i n n e r ( ϵ ) ( t ) , S_{\mathrm{inter}}\geq\kappa S_{\mathrm{inter}}^{(\epsilon)}(t)-\eta S_{\mathrm{inner}}^{(\epsilon)}(t), S inter ≥ κ S inter ( ϵ ) ( t ) − η S inner ( ϵ ) ( t ) , where m m m , κ \kappa κ , and η \eta η are positive constants depending on the Lipschitz continuity of T ( ⋅ ) T(\cdot) T ( ⋅ ) .
Proof. We proceed in three steps. Step 1 identifies the pairwise ordering target induced by AUCGR. Steps 2 and 3 establish the two scatter bounds stated in the theorem. Throughout, fix a diffusion step t t t . We also fix a mini-batch with class set Y \mathcal{Y} Y .
For class y ∈ Y y\in\mathcal{Y} y ∈ Y , let the feature set be Z y = { z i } i = 1 n y \mathcal{Z}_y=\{z_i\}_{i=1}^{n_y} Z y = { z i } i = 1 n y with z i = f ϕ ( x i ) z_i=f_\phi(x_i) z i = f ϕ ( x i ) and the noise set be E y , t = { ϵ ^ i } i = 1 n y \mathcal{E}_{y,t}=\{\hat{\epsilon}_i\}_{i=1}^{n_y} E y , t = { ϵ ^ i } i = 1 n y with ϵ ^ i = ϵ θ ( x ~ t , h ω ( z i ) , t ) \hat{\epsilon}_i=\epsilon_\theta(\tilde{x}_t,h_\omega(z_i),t) ϵ ^ i = ϵ θ ( x ~ t , h ω ( z i ) , t ) . Denote the class means by μ y = 1 n y ∑ i = 1 n y z i \mu_y=\frac{1}{n_y}\sum_{i=1}^{n_y}z_i μ y = n y 1 ∑ i = 1 n y z i and ν y , t = 1 n y ∑ i = 1 n y ϵ ^ i \nu_{y,t}=\frac{1}{n_y}\sum_{i=1}^{n_y}\hat{\epsilon}_i ν y , t = n y 1 ∑ i = 1 n y ϵ ^ i . As shown in Eqs. (7) and (8) in the main text, we define the intra-/inter-class scatters in the CLIP feature space by
S i n n e r = 1 ∣ Y ∣ ∑ y ∈ Y 1 n y ∑ i = 1 n y ∥ z i − μ y ∥ 2 2 , (121) S_{\mathrm{inner}}=\frac{1}{\lvert\mathcal{Y}\rvert}\sum_{y\in\mathcal{Y}}\frac{1}{n_y}\sum_{i=1}^{n_y}\lVert z_i-\mu_y\rVert_2^2, \tag*{(121)} S inner = ∣ Y ∣ 1 y ∈ Y ∑ n y 1 i = 1 ∑ n y ∥ z i − μ y ∥ 2 2 , (121)
S i n t e r = 1 ∣ Y ∣ ( ∣ Y ∣ − 1 ) ∑ y , y ′ ∈ Y y ≠ y ′ ∥ μ y − μ y ′ ∥ 2 2 . (122) S_{\mathrm{inter}}=\frac{1}{\lvert\mathcal{Y}\rvert(\lvert\mathcal{Y}\rvert-1)}\sum_{\substack{y,y'\in\mathcal{Y}\\y\ne y'}}\lVert\mu_y-\mu_{y'}\rVert_2^2. \tag*{(122)} S inter = ∣ Y ∣ (∣ Y ∣ − 1 ) 1 y , y ′ ∈ Y y = y ′ ∑ ∥ μ y − μ y ′ ∥ 2 2 . (122)
Similarly, we define the intra-/inter-class scatters in the diffusion noise space by
S i n n e r ( ϵ ) ( t ) = 1 ∣ Y ∣ ∑ y ∈ Y 1 n y ∑ i = 1 n y ∥ ϵ ^ i − ν y , t ∥ 2 2 , (123) S_{\mathrm{inner}}^{(\epsilon)}(t)=\frac{1}{\lvert\mathcal{Y}\rvert}\sum_{y\in\mathcal{Y}}\frac{1}{n_y}\sum_{i=1}^{n_y}\lVert\hat{\epsilon}_i-\nu_{y,t}\rVert_2^2, \tag*{(123)} S inner ( ϵ ) ( t ) = ∣ Y ∣ 1 y ∈ Y ∑ n y 1 i = 1 ∑ n y ∥ ϵ ^ i − ν y , t ∥ 2 2 , (123)
S i n t e r ( ϵ ) ( t ) = 1 ∣ Y ∣ ( ∣ Y ∣ − 1 ) ∑ y , y ′ ∈ Y y ≠ y ′ ∥ ν y , t − ν y ′ , t ∥ 2 2 . (124) S_{\mathrm{inter}}^{(\epsilon)}(t)=\frac{1}{\lvert\mathcal{Y}\rvert(\lvert\mathcal{Y}\rvert-1)}\sum_{\substack{y,y'\in\mathcal{Y}\\y\ne y'}}\lVert\nu_{y,t}-\nu_{y',t}\rVert_2^2. \tag*{(124)} S inter ( ϵ ) ( t ) = ∣ Y ∣ (∣ Y ∣ − 1 ) 1 y , y ′ ∈ Y y = y ′ ∑ ∥ ν y , t − ν y ′ , t ∥ 2 2 . (124)
Step 1: AUCGR induces an ordering target in the predicted-noise space.
Recall the anchor-wise AUCGR term
L A U C G R ( i ) = 1 ∣ P i ∣ ∣ N i ∣ ∑ p ∈ P i ∑ n ∈ N i ℓ ( q i ( p ) − q i ( n ) ) , (125) \mathcal{L}_{\mathrm{AUCGR}}^{(i)}=\frac{1}{\lvert\mathcal{P}_i\rvert\lvert\mathcal{N}_i\rvert}\sum_{p\in\mathcal{P}_i}\sum_{n\in\mathcal{N}_i}\ell(q_i(p)-q_i(n)), \tag*{(125)} L AUCGR ( i ) = ∣ P i ∣ ∣ N i ∣ 1 p ∈ P i ∑ n ∈ N i ∑ ℓ ( q i ( p ) − q i ( n )) , (125)
where q i ( v ) = sim ( ϵ ^ i , v ) q_i(v)=\operatorname{sim}(\hat{\epsilon}_i,v) q i ( v ) = sim ( ϵ ^ i , v ) and sim ( ⋅ , ⋅ ) \operatorname{sim}(\cdot,\cdot) sim ( ⋅ , ⋅ ) denotes cosine similarity. For a fixed anchor i i i , define the pairwise margin
Δ i p n : = q i ( p ) − q i ( n ) , ∀ p ∈ P i , ∀ n ∈ N i . (126) \Delta_{ipn}:=q_i(p)-q_i(n),\qquad\forall p\in\mathcal{P}_i,\ \forall n\in\mathcal{N}_i. \tag*{(126)} Δ i p n := q i ( p ) − q i ( n ) , ∀ p ∈ P i , ∀ n ∈ N i . (126)
Then Eq. (125) becomes
L A U C G R ( i ) = 1 ∣ P i ∣ ∣ N i ∣ ∑ p ∈ P i ∑ n ∈ N i ℓ ( Δ i p n ) . (127) \mathcal{L}_{\mathrm{AUCGR}}^{(i)}=\frac{1}{\lvert\mathcal{P}_i\rvert\lvert\mathcal{N}_i\rvert}\sum_{p\in\mathcal{P}_i}\sum_{n\in\mathcal{N}_i}\ell(\Delta_{ipn}). \tag*{(127)} L AUCGR ( i ) = ∣ P i ∣ ∣ N i ∣ 1 p ∈ P i ∑ n ∈ N i ∑ ℓ ( Δ i p n ) . (127)
Since ℓ ( ⋅ ) \ell(\cdot) ℓ ( ⋅ ) is decreasing, a smaller AUCGR loss is obtained by increasing the ordering gaps Δ i p n \Delta_{ipn} Δ i p n in the active comparison pairs. Thus, an ordering-consistent solution satisfies
q i ( p ) ≥ q i ( n ) , ∀ p ∈ P i , ∀ n ∈ N i . (128) q_i(p)\geq q_i(n),\qquad\forall p\in\mathcal{P}_i,\ \forall n\in\mathcal{N}_i. \tag*{(128)} q i ( p ) ≥ q i ( n ) , ∀ p ∈ P i , ∀ n ∈ N i . (128)
Equivalently, AUCGR promotes the expected ordering
E p ∼ P i [ q i ( p ) ] ≥ E n ∼ N i [ q i ( n ) ] , (129) \mathbb{E}_{p\sim\mathcal{P}_i}[q_i(p)]\geq\mathbb{E}_{n\sim\mathcal{N}_i}[q_i(n)], \tag*{(129)} E p ∼ P i [ q i ( p )] ≥ E n ∼ N i [ q i ( n )] , (129)
for the active comparisons optimized by the surrogate.
Since cosine similarity compares normalized vectors, for any nonzero u , v u,v u , v ,
sim ( u , v ) = ⟨ u ˉ , v ˉ ⟩ , u ˉ = u ∥ u ∥ 2 , v ˉ = v ∥ v ∥ 2 , (130) \operatorname{sim}(u,v)=\langle\bar{u},\bar{v}\rangle,\qquad\bar{u}=\frac{u}{\lVert u\rVert_2},\qquad\bar{v}=\frac{v}{\lVert v\rVert_2}, \tag*{(130)} sim ( u , v ) = ⟨ u ˉ , v ˉ ⟩ , u ˉ = ∥ u ∥ 2 u , v ˉ = ∥ v ∥ 2 v , (130)
and hence
∥ u ˉ − v ˉ ∥ 2 2 = 2 ( 1 − sim ( u , v ) ) . (131) \lVert\bar{u}-\bar{v}\rVert_2^2=2(1-\operatorname{sim}(u,v)). \tag*{(131)} ∥ u ˉ − v ˉ ∥ 2 2 = 2 ( 1 − sim ( u , v )) . (131)
Applying Eq. (131) to anchor-positive and anchor-negative pairs yields
∥ ϵ ˉ ^ i − p ˉ ∥ 2 2 = 2 ( 1 − q i ( p ) ) , ∥ ϵ ˉ ^ i − n ˉ ∥ 2 2 = 2 ( 1 − q i ( n ) ) . (132) \lVert\hat{\bar{\epsilon}}_i-\bar{p}\rVert_2^2=2(1-q_i(p)),\qquad\lVert\hat{\bar{\epsilon}}_i-\bar{n}\rVert_2^2=2(1-q_i(n)). \tag*{(132)} ∥ ϵ ˉ ^ i − p ˉ ∥ 2 2 = 2 ( 1 − q i ( p )) , ∥ ϵ ˉ ^ i − n ˉ ∥ 2 2 = 2 ( 1 − q i ( n )) . (132)
Hence the ordering relation in Eq. (128) is equivalent to
∥ ϵ ˉ ^ i − p ˉ ∥ 2 2 ≤ ∥ ϵ ˉ ^ i − n ˉ ∥ 2 2 , ∀ p ∈ P i , ∀ n ∈ N i . (133) \lVert\hat{\bar{\epsilon}}_i-\bar{p}\rVert_2^2\leq\lVert\hat{\bar{\epsilon}}_i-\bar{n}\rVert_2^2,\qquad\forall p\in\mathcal{P}_i,\ \forall n\in\mathcal{N}_i. \tag*{(133)} ∥ ϵ ˉ ^ i − p ˉ ∥ 2 2 ≤ ∥ ϵ ˉ ^ i − n ˉ ∥ 2 2 , ∀ p ∈ P i , ∀ n ∈ N i . (133)
In expectation, this gives
E i , p [ ∥ ϵ ˉ ^ i − p ˉ ∥ 2 2 ] ≤ E i , n [ ∥ ϵ ˉ ^ i − n ˉ ∥ 2 2 ] . (134) \mathbb{E}_{i,p}\left[\lVert\hat{\bar{\epsilon}}_i-\bar{p}\rVert_2^2\right]\leq\mathbb{E}_{i,n}\left[\lVert\hat{\bar{\epsilon}}_i-\bar{n}\rVert_2^2\right]. \tag*{(134)} E i , p [ ∥ ϵ ˉ ^ i − p ˉ ∥ 2 2 ] ≤ E i , n [ ∥ ϵ ˉ ^ i − n ˉ ∥ 2 2 ] . (134)
Thus, AUCGR explicitly encodes the desired ordering between matched and mismatched responses in the predicted-noise space. We next use the bi-Lipschitz property of T ( ⋅ ) T(\cdot) T ( ⋅ ) to transfer the relevant scatter quantities to the feature space.
Step 2: Bounding feature-space intra-class scatter.
Fix a class y ∈ Y y\in\mathcal{Y} y ∈ Y , and write its feature set as Z y = { z i } i = 1 n y \mathcal{Z}_y=\{z_i\}_{i=1}^{n_y} Z y = { z i } i = 1 n y and its noise set as E y , t = { ϵ ^ i } i = 1 n y \mathcal{E}_{y,t}=\{\hat{\epsilon}_i\}_{i=1}^{n_y} E y , t = { ϵ ^ i } i = 1 n y with ϵ ^ i = T ( z i ) \hat{\epsilon}_i=T(z_i) ϵ ^ i = T ( z i ) . By Lemma 5 , the intra-class variance in the feature space can be written as
1 n y ∑ i = 1 n y ∥ z i − μ y ∥ 2 2 = 1 2 n y 2 ∑ i = 1 n y ∑ j = 1 n y ∥ z i − z j ∥ 2 2 . (135) \frac{1}{n_y}\sum_{i=1}^{n_y}\lVert z_i-\mu_y\rVert_2^2=\frac{1}{2n_y^2}\sum_{i=1}^{n_y}\sum_{j=1}^{n_y}\lVert z_i-z_j\rVert_2^2. \tag*{(135)} n y 1 i = 1 ∑ n y ∥ z i − μ y ∥ 2 2 = 2 n y 2 1 i = 1 ∑ n y j = 1 ∑ n y ∥ z i − z j ∥ 2 2 . (135)
Similarly, the intra-class variance in the noise space is
1 n y ∑ i = 1 n y ∥ ϵ ^ i − ν y , t ∥ 2 2 = 1 2 n y 2 ∑ i = 1 n y ∑ j = 1 n y ∥ ϵ ^ i − ϵ ^ j ∥ 2 2 . (136) \frac{1}{n_y}\sum_{i=1}^{n_y}\lVert\hat{\epsilon}_i-\nu_{y,t}\rVert_2^2=\frac{1}{2n_y^2}\sum_{i=1}^{n_y}\sum_{j=1}^{n_y}\lVert\hat{\epsilon}_i-\hat{\epsilon}_j\rVert_2^2. \tag*{(136)} n y 1 i = 1 ∑ n y ∥ ϵ ^ i − ν y , t ∥ 2 2 = 2 n y 2 1 i = 1 ∑ n y j = 1 ∑ n y ∥ ϵ ^ i − ϵ ^ j ∥ 2 2 . (136)
By Assumption 4, there exists m > 0 m>0 m > 0 such that, for all z 1 , z 2 ∈ M z_1,z_2\in\mathcal{M} z 1 , z 2 ∈ M ,
m ∥ z 1 − z 2 ∥ ≤ ∥ T ( z 1 ) − T ( z 2 ) ∥ . (137) m\lVert z_1-z_2\rVert\leq\lVert T(z_1)-T(z_2)\rVert. \tag*{(137)} m ∥ z 1 − z 2 ∥ ≤ ∥ T ( z 1 ) − T ( z 2 )∥ . (137)
Equivalently,
∥ z 1 − z 2 ∥ ≤ 1 m ∥ T ( z 1 ) − T ( z 2 ) ∥ . (138) \lVert z_1-z_2\rVert\leq\frac{1}{m}\lVert T(z_1)-T(z_2)\rVert. \tag*{(138)} ∥ z 1 − z 2 ∥ ≤ m 1 ∥ T ( z 1 ) − T ( z 2 )∥ . (138)
Applying this inequality to each pair ( z i , z j ) (z_i,z_j) ( z i , z j ) gives
∥ z i − z j ∥ 2 ≤ 1 m 2 ∥ T ( z i ) − T ( z j ) ∥ 2 = 1 m 2 ∥ ϵ ^ i − ϵ ^ j ∥ 2 . (139) \lVert z_i-z_j\rVert^2\leq\frac{1}{m^2}\lVert T(z_i)-T(z_j)\rVert^2=\frac{1}{m^2}\lVert\hat{\epsilon}_i-\hat{\epsilon}_j\rVert^2. \tag*{(139)} ∥ z i − z j ∥ 2 ≤ m 2 1 ∥ T ( z i ) − T ( z j ) ∥ 2 = m 2 1 ∥ ϵ ^ i − ϵ ^ j ∥ 2 . (139)
Substituting Eq. (139) into Eq. (135) and comparing with Eq. (136) , we obtain
1 n y ∑ i = 1 n y ∥ z i − μ y ∥ 2 2 ≤ 1 2 n y 2 ∑ i = 1 n y ∑ j = 1 n y 1 m 2 ∥ ϵ ^ i − ϵ ^ j ∥ 2 2 = 1 m 2 ⋅ 1 n y ∑ i = 1 n y ∥ ϵ ^ i − ν y , t ∥ 2 2 . (140) \begin{aligned}
\frac{1}{n_y}\sum_{i=1}^{n_y}\lVert z_i-\mu_y\rVert_2^2
&\leq\frac{1}{2n_y^2}\sum_{i=1}^{n_y}\sum_{j=1}^{n_y}\frac{1}{m^2}\lVert\hat{\epsilon}_i-\hat{\epsilon}_j\rVert_2^2\\
&=\frac{1}{m^2}\cdot\frac{1}{n_y}\sum_{i=1}^{n_y}\lVert\hat{\epsilon}_i-\nu_{y,t}\rVert_2^2. \tag*{(140)}
\end{aligned} n y 1 i = 1 ∑ n y ∥ z i − μ y ∥ 2 2 ≤ 2 n y 2 1 i = 1 ∑ n y j = 1 ∑ n y m 2 1 ∥ ϵ ^ i − ϵ ^ j ∥ 2 2 = m 2 1 ⋅ n y 1 i = 1 ∑ n y ∥ ϵ ^ i − ν y , t ∥ 2 2 . (140)
Averaging over all classes y ∈ Y y\in\mathcal{Y} y ∈ Y yields
S i n n e r = 1 ∣ Y ∣ ∑ y ∈ Y 1 n y ∑ i = 1 n y ∥ z i − μ y ∥ 2 2 ≤ 1 m 2 ⋅ 1 ∣ Y ∣ ∑ y ∈ Y 1 n y ∑ i = 1 n y ∥ ϵ ^ i − ν y , t ∥ 2 2 = 1 m 2 S i n n e r ( ϵ ) ( t ) , (141) \begin{aligned}
S_{\mathrm{inner}}
&=\frac{1}{|\mathcal{Y}|}\sum_{y\in\mathcal{Y}}\frac{1}{n_y}\sum_{i=1}^{n_y}\lVert z_i-\mu_y\rVert_2^2\\
&\leq\frac{1}{m^2}\cdot\frac{1}{|\mathcal{Y}|}\sum_{y\in\mathcal{Y}}\frac{1}{n_y}\sum_{i=1}^{n_y}\lVert\hat{\epsilon}_i-\nu_{y,t}\rVert_2^2\\
&=\frac{1}{m^2}S_{\mathrm{inner}}^{(\epsilon)}(t), \tag*{(141)}
\end{aligned} S inner = ∣ Y ∣ 1 y ∈ Y ∑ n y 1 i = 1 ∑ n y ∥ z i − μ y ∥ 2 2 ≤ m 2 1 ⋅ ∣ Y ∣ 1 y ∈ Y ∑ n y 1 i = 1 ∑ n y ∥ ϵ ^ i − ν y , t ∥ 2 2 = m 2 1 S inner ( ϵ ) ( t ) , (141)
which proves the first inequality in Theorem 2.
Step 3: Bounding feature-space inter-class scatter.
By Assumption 4, there exists L > 0 L>0 L > 0 such that, for all z 1 , z 2 ∈ M z_1,z_2\in\mathcal{M} z 1 , z 2 ∈ M ,
∥ T ( z 1 ) − T ( z 2 ) ∥ ≤ L ∥ z 1 − z 2 ∥ . (142) \lVert T(z_1)-T(z_2)\rVert\leq L\lVert z_1-z_2\rVert. \tag*{(142)} ∥ T ( z 1 ) − T ( z 2 )∥ ≤ L ∥ z 1 − z 2 ∥ . (142)
Equivalently,
∥ z 1 − z 2 ∥ ≥ 1 L ∥ T ( z 1 ) − T ( z 2 ) ∥ . (143) \lVert z_1-z_2\rVert\geq\frac{1}{L}\lVert T(z_1)-T(z_2)\rVert. \tag*{(143)} ∥ z 1 − z 2 ∥ ≥ L 1 ∥ T ( z 1 ) − T ( z 2 )∥ . (143)
Fix two distinct classes y , y ′ ∈ Y y, y' \in\mathcal{Y} y , y ′ ∈ Y . Applying Eq. (143) with z 1 = μ y z_1 = \mu_y z 1 = μ y , z 2 = μ y ′ z_2 = \mu_{y'} z 2 = μ y ′ yields
∥ μ y − μ y ′ ∥ ≥ 1 L ∥ T ( μ y ) − T ( μ y ′ ) ∥ . (144) \lVert\mu_y-\mu_{y'}\rVert\ge\frac{1}{L}\lVert T(\mu_y)-T(\mu_{y'})\rVert. \tag*{(144)} ∥ μ y − μ y ′ ∥ ≥ L 1 ∥ T ( μ y ) − T ( μ y ′ )∥ . (144)
We next relate T ( μ y ) T(\mu_y) T ( μ y ) to ν y , t \nu_{y,t} ν y , t . We have
ν y , t = 1 n y ∑ i = 1 n y ϵ ^ i = 1 n y ∑ i = 1 n y T ( z i ) . (145) \nu_{y,t}=\frac{1}{n_y}\sum_{i=1}^{n_y}\hat{\epsilon}_i=\frac{1}{n_y}\sum_{i=1}^{n_y}T(z_i). \tag*{(145)} ν y , t = n y 1 i = 1 ∑ n y ϵ ^ i = n y 1 i = 1 ∑ n y T ( z i ) . (145)
Then
∥ T ( μ y ) − ν y , t ∥ 2 = ∥ T ( μ y ) − 1 n y ∑ i = 1 n y T ( z i ) ∥ 2 = ∥ 1 n y ∑ i = 1 n y ( T ( μ y ) − T ( z i ) ) ∥ 2 ≤ 1 n y ∑ i = 1 n y ∥ T ( μ y ) − T ( z i ) ∥ 2 ≤ L n y ∑ i = 1 n y ∥ μ y − z i ∥ 2 , (146) \begin{aligned}
\lVert T(\mu_y)-\nu_{y,t}\rVert_2
&=\left\lVert T(\mu_y)-\frac{1}{n_y}\sum_{i=1}^{n_y}T(z_i)\right\rVert_2\\
&=\left\lVert\frac{1}{n_y}\sum_{i=1}^{n_y}\left(T(\mu_y)-T(z_i)\right)\right\rVert_2\\
&\le\frac{1}{n_y}\sum_{i=1}^{n_y}\lVert T(\mu_y)-T(z_i)\rVert_2\\
&\le\frac{L}{n_y}\sum_{i=1}^{n_y}\lVert\mu_y-z_i\rVert_2, \tag*{(146)}
\end{aligned} ∥ T ( μ y ) − ν y , t ∥ 2 = T ( μ y ) − n y 1 i = 1 ∑ n y T ( z i ) 2 = n y 1 i = 1 ∑ n y ( T ( μ y ) − T ( z i ) ) 2 ≤ n y 1 i = 1 ∑ n y ∥ T ( μ y ) − T ( z i ) ∥ 2 ≤ n y L i = 1 ∑ n y ∥ μ y − z i ∥ 2 , (146)
where we used the triangle inequality and then the Lipschitz property of T T T . By the Cauchy-Schwarz inequality,
∑ i = 1 n y ∥ μ y − z i ∥ 2 ≤ n y ( ∑ i = 1 n y ∥ μ y − z i ∥ 2 2 ) 1 / 2 . (147) \sum_{i=1}^{n_y}\lVert\mu_y-z_i\rVert_2\le\sqrt{n_y}\left(\sum_{i=1}^{n_y}\lVert\mu_y-z_i\rVert_2^2\right)^{1/2}. \tag*{(147)} i = 1 ∑ n y ∥ μ y − z i ∥ 2 ≤ n y ( i = 1 ∑ n y ∥ μ y − z i ∥ 2 2 ) 1/2 . (147)
Hence,
∥ T ( μ y ) − ν y , t ∥ 2 ≤ L n y ( ∑ i = 1 n y ∥ μ y − z i ∥ 2 2 ) 1 / 2 . (148) \lVert T(\mu_y)-\nu_{y,t}\rVert_2\le\frac{L}{\sqrt{n_y}}\left(\sum_{i=1}^{n_y}\lVert\mu_y-z_i\rVert_2^2\right)^{1/2}. \tag*{(148)} ∥ T ( μ y ) − ν y , t ∥ 2 ≤ n y L ( i = 1 ∑ n y ∥ μ y − z i ∥ 2 2 ) 1/2 . (148)
Similarly, for class y ′ y' y ′ , we have:
∥ T ( μ y ′ ) − ν y ′ , t ∥ 2 ≤ L n y ′ ( ∑ i = 1 n y ′ ∥ μ y ′ − z i ∥ 2 2 ) 1 / 2 . (149) \lVert T(\mu_{y'})-\nu_{y',t}\rVert_2\le\frac{L}{\sqrt{n_{y'}}}\left(\sum_{i=1}^{n_{y'}}\lVert\mu_{y'}-z_i\rVert_2^2\right)^{1/2}. \tag*{(149)} ∥ T ( μ y ′ ) − ν y ′ , t ∥ 2 ≤ n y ′ L ( i = 1 ∑ n y ′ ∥ μ y ′ − z i ∥ 2 2 ) 1/2 . (149)
Using the triangle inequality, we get
∥ T ( μ y ) − T ( μ y ′ ) ∥ 2 ≥ ∥ ν y , t − ν y ′ , t ∥ 2 − ∥ T ( μ y ) − ν y , t ∥ 2 − ∥ T ( μ y ′ ) − ν y ′ , t ∥ 2 . (150) \begin{aligned}
\lVert T(\mu_y)-T(\mu_{y'})\rVert_2
&\ge\lVert\nu_{y,t}-\nu_{y',t}\rVert_2-\lVert T(\mu_y)-\nu_{y,t}\rVert_2-\lVert T(\mu_{y'})-\nu_{y',t}\rVert_2. \tag*{(150)}
\end{aligned} ∥ T ( μ y ) − T ( μ y ′ ) ∥ 2 ≥ ∥ ν y , t − ν y ′ , t ∥ 2 − ∥ T ( μ y ) − ν y , t ∥ 2 − ∥ T ( μ y ′ ) − ν y ′ , t ∥ 2 . (150)
Combining this with Eqs. (148) and (149) , we obtain
∥ T ( μ y ) − T ( μ y ′ ) ∥ 2 ≥ ∥ ν y , t − ν y ′ , t ∥ 2 − L n y ( ∑ i = 1 n y ∥ μ y − z i ∥ 2 2 ) 1 / 2 − L n y ′ ( ∑ i = 1 n y ′ ∥ μ y ′ − z i ∥ 2 2 ) 1 / 2 . (151) \lVert T(\mu_y)-T(\mu_{y'})\rVert_2\ge\lVert\nu_{y,t}-\nu_{y',t}\rVert_2-\frac{L}{\sqrt{n_y}}\left(\sum_{i=1}^{n_y}\lVert\mu_y-z_i\rVert_2^2\right)^{1/2}-\frac{L}{\sqrt{n_{y'}}}\left(\sum_{i=1}^{n_{y'}}\lVert\mu_{y'}-z_i\rVert_2^2\right)^{1/2}. \tag*{(151)} ∥ T ( μ y ) − T ( μ y ′ ) ∥ 2 ≥ ∥ ν y , t − ν y ′ , t ∥ 2 − n y L ( i = 1 ∑ n y ∥ μ y − z i ∥ 2 2 ) 1/2 − n y ′ L ( i = 1 ∑ n y ′ ∥ μ y ′ − z i ∥ 2 2 ) 1/2 . (151)
Substituting Eq. (151) into Eq. (144) , we obtain
∥ μ y − μ y ′ ∥ 2 ≥ 1 L ∥ ν y , t − ν y ′ , t ∥ 2 − 1 n y ( ∑ i = 1 n y ∥ μ y − z i ∥ 2 2 ) 1 / 2 − 1 n y ′ ( ∑ i = 1 n y ′ ∥ μ y ′ − z i ∥ 2 2 ) 1 / 2 . (152) \lVert\mu_y-\mu_{y'}\rVert_2\ge\frac{1}{L}\lVert\nu_{y,t}-\nu_{y',t}\rVert_2-\frac{1}{\sqrt{n_y}}\left(\sum_{i=1}^{n_y}\lVert\mu_y-z_i\rVert_2^2\right)^{1/2}-\frac{1}{\sqrt{n_{y'}}}\left(\sum_{i=1}^{n_{y'}}\lVert\mu_{y'}-z_i\rVert_2^2\right)^{1/2}. \tag*{(152)} ∥ μ y − μ y ′ ∥ 2 ≥ L 1 ∥ ν y , t − ν y ′ , t ∥ 2 − n y 1 ( i = 1 ∑ n y ∥ μ y − z i ∥ 2 2 ) 1/2 − n y ′ 1 ( i = 1 ∑ n y ′ ∥ μ y ′ − z i ∥ 2 2 ) 1/2 . (152)
Let
a = 1 L ∥ ν y , t − ν y ′ , t ∥ 2 , b = 1 n y ( ∑ i = 1 n y ∥ μ y − z i ∥ 2 2 ) 1 / 2 , c = 1 n y ′ ( ∑ i = 1 n y ′ ∥ μ y ′ − z i ∥ 2 2 ) 1 / 2 . (153) a=\frac{1}{L}\lVert\nu_{y,t}-\nu_{y',t}\rVert_2,\qquad b=\frac{1}{\sqrt{n_y}}\left(\sum_{i=1}^{n_y}\lVert\mu_y-z_i\rVert_2^2\right)^{1/2},\qquad c=\frac{1}{\sqrt{n_{y'}}}\left(\sum_{i=1}^{n_{y'}}\lVert\mu_{y'}-z_i\rVert_2^2\right)^{1/2}. \tag*{(153)} a = L 1 ∥ ν y , t − ν y ′ , t ∥ 2 , b = n y 1 ( i = 1 ∑ n y ∥ μ y − z i ∥ 2 2 ) 1/2 , c = n y ′ 1 ( i = 1 ∑ n y ′ ∥ μ y ′ − z i ∥ 2 2 ) 1/2 . (153)
If a − b − c < 0 a-b-c<0 a − b − c < 0 , then 1 2 a 2 − 2 ( b 2 + c 2 ) ≤ 0 \frac{1}{2}a^2-2(b^2+c^2)\le0 2 1 a 2 − 2 ( b 2 + c 2 ) ≤ 0 and the following lower bound is trivial. If a − b − c ≥ 0 a-b-c\ge0 a − b − c ≥ 0 , then we square the preceding inequality and use ( a − b − c ) 2 ≥ 1 2 a 2 − 2 ( b 2 + c 2 ) (a-b-c)^2\ge\frac{1}{2}a^2-2(b^2+c^2) ( a − b − c ) 2 ≥ 2 1 a 2 − 2 ( b 2 + c 2 ) . In both cases, we obtain
∥ μ y − μ y ′ ∥ 2 2 ≥ 1 2 L 2 ∥ ν y , t − ν y ′ , t ∥ 2 2 − 2 ( 1 n y ∑ i = 1 n y ∥ μ y − z i ∥ 2 2 ) − 2 ( 1 n y ′ ∑ i = 1 n y ′ ∥ μ y ′ − z i ∥ 2 2 ) . (154) \lVert\mu_y-\mu_{y'}\rVert_2^2\ge\frac{1}{2L^2}\lVert\nu_{y,t}-\nu_{y',t}\rVert_2^2-2\left(\frac{1}{n_y}\sum_{i=1}^{n_y}\lVert\mu_y-z_i\rVert_2^2\right)-2\left(\frac{1}{n_{y'}}\sum_{i=1}^{n_{y'}}\lVert\mu_{y'}-z_i\rVert_2^2\right). \tag*{(154)} ∥ μ y − μ y ′ ∥ 2 2 ≥ 2 L 2 1 ∥ ν y , t − ν y ′ , t ∥ 2 2 − 2 ( n y 1 i = 1 ∑ n y ∥ μ y − z i ∥ 2 2 ) − 2 ( n y ′ 1 i = 1 ∑ n y ′ ∥ μ y ′ − z i ∥ 2 2 ) . (154)
Averaging the above inequality over all ordered pairs ( y , y ′ ) (y,y') ( y , y ′ ) with y ≠ y ′ y \ne y' y = y ′ , and recalling the definition of S i n t e r ( ϵ ) S_{\mathrm{inter}}^{(\epsilon)} S inter ( ϵ ) and S i n t e r ( ϵ ) ( t ) S_{\mathrm{inter}}^{(\epsilon)}(t) S inter ( ϵ ) ( t ) , we obtain
S i n t e r = 1 ∣ Y ∣ ( ∣ Y ∣ − 1 ) ∑ y , y ′ ∈ Y y ≠ y ′ ∥ μ y − μ y ′ ∥ 2 2 ≥ 1 ∣ Y ∣ ( ∣ Y ∣ − 1 ) ∑ y , y ′ ∈ Y y ≠ y ′ [ 1 2 L 2 ∥ ν y , t − ν y ′ , t ∥ 2 2 − 2 B y , y ′ ] = 1 2 L 2 S i n t e r ( ϵ ) ( t ) − 2 ∣ Y ∣ ( ∣ Y ∣ − 1 ) ∑ y , y ′ ∈ Y y ≠ y ′ B y , y ′ . (155) \begin{aligned}
S_{\mathrm{inter}} &=
\frac{1}{|\mathcal{Y}|(|\mathcal{Y}|-1)}
\sum_{\substack{y,y'\in\mathcal{Y}\\y\ne y'}}
\left\|\mu_y-\mu_{y'}\right\|_2^2\\
&\geq
\frac{1}{|\mathcal{Y}|(|\mathcal{Y}|-1)}
\sum_{\substack{y,y'\in\mathcal{Y}\\y\ne y'}}
\left[\frac{1}{2L^2}\left\|\nu_{y,t}-\nu_{y',t}\right\|_2^2-2B_{y,y'}\right]\\
&=\frac{1}{2L^2}S_{\mathrm{inter}}^{(\epsilon)}(t)
-\frac{2}{|\mathcal{Y}|(|\mathcal{Y}|-1)}
\sum_{\substack{y,y'\in\mathcal{Y}\\y\ne y'}}B_{y,y'}. \tag*{(155)}
\end{aligned} S inter = ∣ Y ∣ ( ∣ Y ∣ − 1 ) 1 y , y ′ ∈ Y y = y ′ ∑ ∥ μ y − μ y ′ ∥ 2 2 ≥ ∣ Y ∣ ( ∣ Y ∣ − 1 ) 1 y , y ′ ∈ Y y = y ′ ∑ [ 2 L 2 1 ∥ ν y , t − ν y ′ , t ∥ 2 2 − 2 B y , y ′ ] = 2 L 2 1 S inter ( ϵ ) ( t ) − ∣ Y ∣ ( ∣ Y ∣ − 1 ) 2 y , y ′ ∈ Y y = y ′ ∑ B y , y ′ . (155)
where B y , y ′ = 1 n y ∑ i ∈ I y ∥ μ y − z i ∥ 2 2 + 1 n y ′ ∑ i ∈ I y ′ ∥ μ y ′ − z i ∥ 2 2 B_{y,y'}=\frac{1}{n_y}\sum_{i\in I_y}\left\|\mu_y-z_i\right\|_2^2+\frac{1}{n_{y'}}\sum_{i\in I_{y'}}\left\|\mu_{y'}-z_i\right\|_2^2 B y , y ′ = n y 1 ∑ i ∈ I y ∥ μ y − z i ∥ 2 2 + n y ′ 1 ∑ i ∈ I y ′ ∥ μ y ′ − z i ∥ 2 2 .
For the second term, note that for each fixed y y y , the quantity A y = 1 n y ∑ i ∈ I y ∥ μ y − z i ∥ 2 2 A_y=\frac{1}{n_y}\sum_{i\in I_y}\left\|\mu_y-z_i\right\|_2^2 A y = n y 1 ∑ i ∈ I y ∥ μ y − z i ∥ 2 2 appears exactly ( ∣ Y ∣ − 1 ) (|\mathcal{Y}|-1) ( ∣ Y ∣ − 1 ) times as the first index and ( ∣ Y ∣ − 1 ) (|\mathcal{Y}|-1) ( ∣ Y ∣ − 1 ) times as the second index when summing over all ordered pairs ( y , y ′ ) (y,y') ( y , y ′ ) with y ≠ y ′ y\ne y' y = y ′ . Hence
∑ y , y ′ ∈ Y y ≠ y ′ ( A y + A y ′ ) = 2 ( ∣ Y ∣ − 1 ) ∑ y ∈ Y A y . (156) \sum_{\substack{y,y'\in\mathcal{Y}\\y\ne y'}}(A_y+A_{y'})=2(|\mathcal{Y}|-1)\sum_{y\in\mathcal{Y}}A_y. \tag*{(156)} y , y ′ ∈ Y y = y ′ ∑ ( A y + A y ′ ) = 2 ( ∣ Y ∣ − 1 ) y ∈ Y ∑ A y . (156)
Therefore,
2 ∣ Y ∣ ( ∣ Y ∣ − 1 ) ∑ y , y ′ ∈ Y y ≠ y ′ ( A y + A y ′ ) = 2 ∣ Y ∣ ( ∣ Y ∣ − 1 ) 2 ( ∣ Y ∣ − 1 ) ∑ y ∈ Y A y = 4 ∣ Y ∣ ∑ y ∈ Y A y = 4 S i n n e r . (157) \begin{aligned}
\frac{2}{|\mathcal{Y}|(|\mathcal{Y}|-1)}
\sum_{\substack{y,y'\in\mathcal{Y}\\y\ne y'}}(A_y+A_{y'})
&=\frac{2}{|\mathcal{Y}|(|\mathcal{Y}|-1)}2(|\mathcal{Y}|-1)\sum_{y\in\mathcal{Y}}A_y\\
&=\frac{4}{|\mathcal{Y}|}\sum_{y\in\mathcal{Y}}A_y\\
&=4S_{\mathrm{inner}}. \tag*{(157)}
\end{aligned} ∣ Y ∣ ( ∣ Y ∣ − 1 ) 2 y , y ′ ∈ Y y = y ′ ∑ ( A y + A y ′ ) = ∣ Y ∣ ( ∣ Y ∣ − 1 ) 2 2 ( ∣ Y ∣ − 1 ) y ∈ Y ∑ A y = ∣ Y ∣ 4 y ∈ Y ∑ A y = 4 S inner . (157)
Substituting this back, we obtain the explicit bound
S i n t e r ≥ 1 2 L 2 S i n t e r ( ϵ ) ( t ) − 4 S i n n e r . (158) S_{\mathrm{inter}}\geq\frac{1}{2L^2}S_{\mathrm{inter}}^{(\epsilon)}(t)-4S_{\mathrm{inner}}. \tag*{(158)} S inter ≥ 2 L 2 1 S inter ( ϵ ) ( t ) − 4 S inner . (158)
Finally, using the bound from Step 2, S i n n e r ≤ 1 m 2 S i n n e r ( ϵ ) ( t ) S_{\mathrm{inner}}\leq\frac{1}{m^2}S_{\mathrm{inner}}^{(\epsilon)}(t) S inner ≤ m 2 1 S inner ( ϵ ) ( t ) , we arrive at
S i n t e r ≥ 1 2 L 2 S i n t e r ( ϵ ) ( t ) − 4 m 2 S i n n e r ( ϵ ) ( t ) . (159) S_{\mathrm{inter}}\geq\frac{1}{2L^2}S_{\mathrm{inter}}^{(\epsilon)}(t)-\frac{4}{m^2}S_{\mathrm{inner}}^{(\epsilon)}(t). \tag*{(159)} S inter ≥ 2 L 2 1 S inter ( ϵ ) ( t ) − m 2 4 S inner ( ϵ ) ( t ) . (159)
Thus, we have:
S i n t e r ≥ κ S i n t e r ( ϵ ) ( t ) − η S i n n e r ( ϵ ) ( t ) , (160) S_{\mathrm{inter}}\geq\kappa S_{\mathrm{inter}}^{(\epsilon)}(t)-\eta S_{\mathrm{inner}}^{(\epsilon)}(t), \tag*{(160)} S inter ≥ κ S inter ( ϵ ) ( t ) − η S inner ( ϵ ) ( t ) , (160)
where κ = 1 2 L 2 \kappa=\frac{1}{2L^2} κ = 2 L 2 1 and η = 4 m 2 \eta=\frac{4}{m^2} η = m 2 4 . This proves the second inequality in Theorem 2.
This completes the proof.
Appendix E: Proof of Theorem 3
Restate of Theorem 3 (AUCGR Supports P-Ability). Under the assumptions that negatives in the mini-batch are well separated from the anchor (i.e., sim ( ϵ ^ , ϵ ^ j ) ≪ sim ( ϵ ^ , ϵ g t ) \operatorname{sim}(\hat{\epsilon},\hat{\epsilon}^{j}) \ll\operatorname{sim}(\hat{\epsilon},\epsilon_{\mathrm{gt}}) sim ( ϵ ^ , ϵ ^ j ) ≪ sim ( ϵ ^ , ϵ gt ) for all j ≠ i j \ne i j = i ), predicted-noise norms are bounded away from 0 0 0 and ∞ \infty ∞ , and the surrogate loss is decreasing with bounded slope for the pairwise gaps used by AUCGR, then the AUCGR loss is bounded by a scaled reconstruction loss up to additive constants:
λ min R r e c + c min ≤ L A U C G R ≤ λ max R r e c + c max , \lambda_{\min}R_{\mathrm{rec}} + c_{\min} \le\mathcal{L}_{\mathrm{AUCGR}} \le\lambda_{\max}R_{\mathrm{rec}} + c_{\max}, λ m i n R rec + c m i n ≤ L AUCGR ≤ λ m a x R rec + c m a x , where R r e c = 1 B ∑ i = 1 B ∥ ϵ ^ i − ϵ t , i g t ∥ 2 2 R_{\mathrm{rec}} = \frac{1}{B}\sum_{i=1}^{B}\lVert\hat{\epsilon}_{i}-\epsilon_{t,i}^{\mathrm{gt}}\rVert_{2}^{2} R rec = B 1 ∑ i = 1 B ∥ ϵ ^ i − ϵ t , i gt ∥ 2 2 , λ min , λ max > 0 \lambda_{\min},\lambda_{\max} > 0 λ m i n , λ m a x > 0 , and c min , c max c_{\min},c_{\max} c m i n , c m a x are constants.
Proof. We consider a single anchor sample and drop the sample index for brevity.
Recall the anchor predicted noise
ϵ ^ = ϵ θ ( x ~ t , h ω ( f ϕ ( x ) ) , t ) . (161) \hat{\epsilon} = \epsilon_{\theta}\left(\tilde{x}_{t},h_{\omega}(f_{\phi}(x)),t\right). \tag*{(161)} ϵ ^ = ϵ θ ( x ~ t , h ω ( f ϕ ( x )) , t ) . (161)
Let ϵ g t \epsilon_{\mathrm{gt}} ϵ gt denote the ground-truth predicted-noise target contained in the positive set. Let P a u x \mathcal{P}_{\mathrm{aux}} P aux denote the remaining positives, let P = { ϵ g t } ∪ P a u x \mathcal{P}=\{\epsilon_{\mathrm{gt}}\}\cup\mathcal{P}_{\mathrm{aux}} P = { ϵ gt } ∪ P aux , and let N = { ϵ ^ j } j \mathcal{N}=\{\hat{\epsilon}^{j}\}_{j} N = { ϵ ^ j } j denote the set of negatives in the mini-batch. The corresponding anchor-wise AUCGR loss can be decomposed as
L A U C G R ( i ) = L g t + L a u x , (162) \mathcal{L}_{\mathrm{AUCGR}}^{(i)}=\mathcal{L}_{\mathrm{gt}}+\mathcal{L}_{\mathrm{aux}}, \tag*{(162)} L AUCGR ( i ) = L gt + L aux , (162)
where
u = sim ( ϵ ^ , ϵ g t ) , r j = sim ( ϵ ^ , ϵ ^ j ) , q ( p ) = sim ( ϵ ^ , p ) . (163) u=\operatorname{sim}(\hat{\epsilon},\epsilon_{\mathrm{gt}}),\qquad r_{j}=\operatorname{sim}(\hat{\epsilon},\hat{\epsilon}^{j}),\qquad q(p)=\operatorname{sim}(\hat{\epsilon},p). \tag*{(163)} u = sim ( ϵ ^ , ϵ gt ) , r j = sim ( ϵ ^ , ϵ ^ j ) , q ( p ) = sim ( ϵ ^ , p ) . (163)
L g t = 1 ∣ P ∣ ∣ N ∣ ∑ j ℓ ( u − r j ) , L a u x = 1 ∣ P ∣ ∣ N ∣ ∑ p ∈ P a u x ∑ j ℓ ( q ( p ) − r j ) . (164) \mathcal{L}_{\mathrm{gt}}=\frac{1}{\lvert\mathcal{P}\rvert\lvert\mathcal{N}\rvert}\sum_{j}\ell(u-r_{j}),\qquad
\mathcal{L}_{\mathrm{aux}}=\frac{1}{\lvert\mathcal{P}\rvert\lvert\mathcal{N}\rvert}\sum_{p\in\mathcal{P}_{\mathrm{aux}}}\sum_{j}\ell(q(p)-r_{j}). \tag*{(164)} L gt = ∣ P ∣ ∣ N ∣ 1 j ∑ ℓ ( u − r j ) , L aux = ∣ P ∣ ∣ N ∣ 1 p ∈ P aux ∑ j ∑ ℓ ( q ( p ) − r j ) . (164)
Since cosine similarity lies in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] and the surrogate is bounded on compact intervals, the auxiliary term L a u x \mathcal{L}_{\mathrm{aux}} L aux is bounded. Therefore, there exist constants C a u x min , C a u x max ∈ R C_{\mathrm{aux}}^{\min},C_{\mathrm{aux}}^{\max}\in\mathbb{R} C aux m i n , C aux m a x ∈ R such that
C a u x min ≤ L a u x ≤ C a u x max . (165) C_{\mathrm{aux}}^{\min}\le\mathcal{L}_{\mathrm{aux}}\le C_{\mathrm{aux}}^{\max}. \tag*{(165)} C aux m i n ≤ L aux ≤ C aux m a x . (165)
We now analyze the ground-truth term. By the separation assumption, there exists a constant Δ > 0 \Delta>0 Δ > 0 such that
r j ≤ u − Δ , ∀ j . (166) r_{j}\le u-\Delta,\qquad\forall j. \tag*{(166)} r j ≤ u − Δ , ∀ j . (166)
Hence, for each j j j , the score gap u − r j u-r_{j} u − r j belongs to the pairwise gaps used by AUCGR. We assume that the surrogate loss is decreasing with bounded slope on these gaps. That is, there exist constants 0 < μ ℓ ≤ L ℓ < ∞ 0<\mu_{\ell}\le L_{\ell}<\infty 0 < μ ℓ ≤ L ℓ < ∞ such that, for every such gap s s s ,
μ ℓ ≤ − ℓ ′ ( s ) ≤ L ℓ . (167) \mu_{\ell}\le-\ell'(s)\le L_{\ell}. \tag*{(167)} μ ℓ ≤ − ℓ ′ ( s ) ≤ L ℓ . (167)
Next, define the normalized vectors
e ^ = ϵ ^ ∥ ϵ ^ ∥ 2 , g = ϵ g t ∥ ϵ g t ∥ 2 . (168) \hat{e}=\frac{\hat{\epsilon}}{\lVert\hat{\epsilon}\rVert_{2}},\qquad
g=\frac{\epsilon_{\mathrm{gt}}}{\lVert\epsilon_{\mathrm{gt}}\rVert_{2}}. \tag*{(168)} e ^ = ∥ ϵ ^ ∥ 2 ϵ ^ , g = ∥ ϵ gt ∥ 2 ϵ gt . (168)
Let
d g t 2 = ∥ e ^ − g ∥ 2 2 . (169) d_{\mathrm{gt}}^{2}=\lVert\hat{e}-g\rVert_{2}^{2}. \tag*{(169)} d gt 2 = ∥ e ^ − g ∥ 2 2 . (169)
Since e ^ \hat{e} e ^ and g g g are unit vectors, we have
u = ⟨ e ^ , g ⟩ = 1 − 1 2 d g t 2 , 0 ≤ d g t 2 ≤ 4. (170) u=\langle\hat{e},g\rangle=1-\frac{1}{2}d_{\mathrm{gt}}^{2},\qquad0\le d_{\mathrm{gt}}^{2}\le4. \tag*{(170)} u = ⟨ e ^ , g ⟩ = 1 − 2 1 d gt 2 , 0 ≤ d gt 2 ≤ 4. (170)
Substituting Eq. (170) into each term of L g t \mathcal{L}_{\mathrm{gt}} L gt gives
ℓ ( u − r j ) = ℓ ( 1 − 1 2 d g t 2 − r j ) . (171) \ell(u-r_{j})=\ell\left(1-\frac{1}{2}d_{\mathrm{gt}}^{2}-r_{j}\right). \tag*{(171)} ℓ ( u − r j ) = ℓ ( 1 − 2 1 d gt 2 − r j ) . (171)
By the mean value theorem and Eq. (167) , the right-hand side of Eq. (171) is bounded above and below by affine functions of d g t 2 d_{\mathrm{gt}}^{2} d gt 2 . Therefore, there exist positive constants a j , b j a_{j},b_{j} a j , b j and constants c j , d j c_{j},d_{j} c j , d j such that
a j d g t 2 + c j ≤ ℓ ( u − r j ) ≤ b j d g t 2 + d j . (172) a_{j}d_{\mathrm{gt}}^{2}+c_{j}\le\ell(u-r_{j})\le b_{j}d_{\mathrm{gt}}^{2}+d_{j}. \tag*{(172)} a j d gt 2 + c j ≤ ℓ ( u − r j ) ≤ b j d gt 2 + d j . (172)
Averaging over all j j j in Eq. (172) , there exist positive constants λ ~ min , λ ~ max \widetilde{\lambda}_{\min},\widetilde{\lambda}_{\max} λ m i n , λ m a x and constants c ~ min , c ~ max \widetilde{c}_{\min},\widetilde{c}_{\max} c m i n , c m a x such that
λ ~ min d g t 2 + c ~ min ≤ L g t ≤ λ ~ max d g t 2 + c ~ max . (173) \widetilde{\lambda}_{\min}d_{\mathrm{gt}}^{2}+\widetilde{c}_{\min}\le\mathcal{L}_{\mathrm{gt}}\le\widetilde{\lambda}_{\max}d_{\mathrm{gt}}^{2}+\widetilde{c}_{\max}. \tag*{(173)} λ m i n d gt 2 + c m i n ≤ L gt ≤ λ m a x d gt 2 + c m a x . (173)
Combining Eq. (165) and Eq. (173) , we obtain
λ ‾ min d g t 2 + c ‾ min ≤ L A U C G R ( i ) ≤ λ ‾ max d g t 2 + c ‾ max , (174) \overline{\lambda}_{\min}d_{\mathrm{gt}}^{2}+\overline{c}_{\min}\le\mathcal{L}_{\mathrm{AUCGR}}^{(i)}\le\overline{\lambda}_{\max}d_{\mathrm{gt}}^{2}+\overline{c}_{\max}, \tag*{(174)} λ m i n d gt 2 + c m i n ≤ L AUCGR ( i ) ≤ λ m a x d gt 2 + c m a x , (174)
for some positive constants λ ‾ min , λ ‾ max \overline{\lambda}_{\min},\overline{\lambda}_{\max} λ m i n , λ m a x and constants c ‾ min , c ‾ max \overline{c}_{\min},\overline{c}_{\max} c m i n , c m a x .
We now relate d g t 2 d_{gt}^{2} d g t 2 to the true reconstruction error ∥ ϵ ^ − ϵ g t ∥ 2 2 \lVert\hat{\epsilon}-\epsilon_{gt}\rVert_{2}^{2} ∥ ϵ ^ − ϵ g t ∥ 2 2 . For any nonzero vectors u , v u,v u , v , the law of cosines gives
∥ u − v ∥ 2 2 = ∥ u ∥ 2 2 + ∥ v ∥ 2 2 − 2 ∥ u ∥ 2 ∥ v ∥ 2 sim ( u , v ) . (175) \lVert u-v\rVert_{2}^{2}=\lVert u\rVert_{2}^{2}+\lVert v\rVert_{2}^{2}-2\lVert u\rVert_{2}\lVert v\rVert_{2}\operatorname{sim}(u,v). \tag*{(175)} ∥ u − v ∥ 2 2 = ∥ u ∥ 2 2 + ∥ v ∥ 2 2 − 2 ∥ u ∥ 2 ∥ v ∥ 2 sim ( u , v ) . (175)
With u = ϵ ^ u=\hat{\epsilon} u = ϵ ^ , v = ϵ g t v=\epsilon_{gt} v = ϵ g t and sim ( u , v ) = 1 − d g t 2 / 2 \operatorname{sim}(u,v)=1-d_{gt}^{2}/2 sim ( u , v ) = 1 − d g t 2 /2 , we obtain
∥ ϵ ^ − ϵ g t ∥ 2 2 = ∥ ϵ ^ ∥ 2 2 + ∥ ϵ g t ∥ 2 2 − 2 ∥ ϵ ^ ∥ 2 ∥ ϵ g t ∥ 2 ( 1 − d g t 2 2 ) = ( ∥ ϵ ^ ∥ 2 − ∥ ϵ g t ∥ 2 ) 2 + ∥ ϵ ^ ∥ 2 ∥ ϵ g t ∥ 2 d g t 2 . (176) \begin{aligned}
\lVert\hat{\epsilon}-\epsilon_{gt}\rVert_{2}^{2}
&=\lVert\hat{\epsilon}\rVert_{2}^{2}+\lVert\epsilon_{gt}\rVert_{2}^{2}-2\lVert\hat{\epsilon}\rVert_{2}\lVert\epsilon_{gt}\rVert_{2}\left(1-\frac{d_{gt}^{2}}{2}\right)\\
&=\left(\lVert\hat{\epsilon}\rVert_{2}-\lVert\epsilon_{gt}\rVert_{2}\right)^{2}+\lVert\hat{\epsilon}\rVert_{2}\lVert\epsilon_{gt}\rVert_{2}d_{gt}^{2}. \tag*{(176)}
\end{aligned} ∥ ϵ ^ − ϵ g t ∥ 2 2 = ∥ ϵ ^ ∥ 2 2 + ∥ ϵ g t ∥ 2 2 − 2 ∥ ϵ ^ ∥ 2 ∥ ϵ g t ∥ 2 ( 1 − 2 d g t 2 ) = ( ∥ ϵ ^ ∥ 2 − ∥ ϵ g t ∥ 2 ) 2 + ∥ ϵ ^ ∥ 2 ∥ ϵ g t ∥ 2 d g t 2 . (176)
By assumption, predicted-noise norms are bounded away from 0 and ∞ \infty ∞ . Thus, there exist constants 0 < α ≤ β < ∞ 0<\alpha\le\beta<\infty 0 < α ≤ β < ∞ such that
α ≤ ∥ ϵ ^ ∥ 2 , ∥ ϵ g t ∥ 2 ≤ β . (177) \alpha\le\lVert\hat{\epsilon}\rVert_{2},\lVert\epsilon_{gt}\rVert_{2}\le\beta. \tag*{(177)} α ≤ ∥ ϵ ^ ∥ 2 , ∥ ϵ g t ∥ 2 ≤ β . (177)
Using Eq. (176) and the fact that ( ∥ ϵ ^ ∥ 2 − ∥ ϵ g t ∥ 2 ) 2 ≥ 0 \left(\lVert\hat{\epsilon}\rVert_{2}-\lVert\epsilon_{gt}\rVert_{2}\right)^{2}\ge0 ( ∥ ϵ ^ ∥ 2 − ∥ ϵ g t ∥ 2 ) 2 ≥ 0 , we have
∥ ϵ ^ − ϵ g t ∥ 2 2 ≥ ∥ ϵ ^ ∥ 2 ∥ ϵ g t ∥ 2 d g t 2 ≥ α 2 d g t 2 . (178) \lVert\hat{\epsilon}-\epsilon_{gt}\rVert_{2}^{2}\ge\lVert\hat{\epsilon}\rVert_{2}\lVert\epsilon_{gt}\rVert_{2}d_{gt}^{2}\ge\alpha^{2}d_{gt}^{2}. \tag*{(178)} ∥ ϵ ^ − ϵ g t ∥ 2 2 ≥ ∥ ϵ ^ ∥ 2 ∥ ϵ g t ∥ 2 d g t 2 ≥ α 2 d g t 2 . (178)
On the other hand, using ∥ ϵ ^ ∥ 2 2 + ∥ ϵ g t ∥ 2 2 ≤ 2 β 2 \lVert\hat{\epsilon}\rVert_{2}^{2}+\lVert\epsilon_{gt}\rVert_{2}^{2}\le2\beta^{2} ∥ ϵ ^ ∥ 2 2 + ∥ ϵ g t ∥ 2 2 ≤ 2 β 2 and ∥ ϵ ^ ∥ 2 ∥ ϵ g t ∥ 2 ≤ β 2 \lVert\hat{\epsilon}\rVert_{2}\lVert\epsilon_{gt}\rVert_{2}\le\beta^{2} ∥ ϵ ^ ∥ 2 ∥ ϵ g t ∥ 2 ≤ β 2 , we obtain
∥ ϵ ^ − ϵ g t ∥ 2 2 ≤ 2 β 2 + β 2 d g t 2 . (179) \lVert\hat{\epsilon}-\epsilon_{gt}\rVert_{2}^{2}\le2\beta^{2}+\beta^{2}d_{gt}^{2}. \tag*{(179)} ∥ ϵ ^ − ϵ g t ∥ 2 2 ≤ 2 β 2 + β 2 d g t 2 . (179)
Rearranging these inequalities yields
1 β 2 ( ∥ ϵ ^ − ϵ g t ∥ 2 2 − 2 β 2 ) ≤ d g t 2 ≤ 1 α 2 ∥ ϵ ^ − ϵ g t ∥ 2 2 . (180) \frac{1}{\beta^{2}}\left(\lVert\hat{\epsilon}-\epsilon_{gt}\rVert_{2}^{2}-2\beta^{2}\right)\le d_{gt}^{2}\le\frac{1}{\alpha^{2}}\lVert\hat{\epsilon}-\epsilon_{gt}\rVert_{2}^{2}. \tag*{(180)} β 2 1 ( ∥ ϵ ^ − ϵ g t ∥ 2 2 − 2 β 2 ) ≤ d g t 2 ≤ α 2 1 ∥ ϵ ^ − ϵ g t ∥ 2 2 . (180)
Combining Eq. (174) and Eq. (180) , we obtain affine bounds on the anchor-wise AUCGR loss in terms of the reconstruction error. Therefore, there exist positive constants λ min ( i ) , λ max ( i ) \lambda_{\min}^{(i)},\lambda_{\max}^{(i)} λ m i n ( i ) , λ m a x ( i ) and constants c min ( i ) , c max ( i ) c_{\min}^{(i)},c_{\max}^{(i)} c m i n ( i ) , c m a x ( i ) such that
λ min ( i ) ∥ ϵ ^ − ϵ g t ∥ 2 2 + c min ( i ) ≤ L A U C G R ( i ) ≤ λ max ( i ) ∥ ϵ ^ − ϵ g t ∥ 2 2 + c max ( i ) . (181) \lambda_{\min}^{(i)}\lVert\hat{\epsilon}-\epsilon_{gt}\rVert_{2}^{2}+c_{\min}^{(i)}\le L_{\mathrm{AUCGR}}^{(i)}\le\lambda_{\max}^{(i)}\lVert\hat{\epsilon}-\epsilon_{gt}\rVert_{2}^{2}+c_{\max}^{(i)}. \tag*{(181)} λ m i n ( i ) ∥ ϵ ^ − ϵ g t ∥ 2 2 + c m i n ( i ) ≤ L AUCGR ( i ) ≤ λ m a x ( i ) ∥ ϵ ^ − ϵ g t ∥ 2 2 + c m a x ( i ) . (181)
Averaging Eq. (181) over all anchors in the mini-batch and using the smallest lower coefficient and the largest upper coefficient yields
λ min 1 B ∑ i = 1 B ∥ ϵ ^ i − ϵ t , i g t ∥ 2 2 + c min ≤ L A U C G R ≤ λ max 1 B ∑ i = 1 B ∥ ϵ ^ i − ϵ t , i g t ∥ 2 2 + c max , (182) \lambda_{\min}\frac{1}{B}\sum_{i=1}^{B}\lVert\hat{\epsilon}_{i}-\epsilon_{t,i}^{gt}\rVert_{2}^{2}+c_{\min}\le L_{\mathrm{AUCGR}}\le\lambda_{\max}\frac{1}{B}\sum_{i=1}^{B}\lVert\hat{\epsilon}_{i}-\epsilon_{t,i}^{gt}\rVert_{2}^{2}+c_{\max}, \tag*{(182)} λ m i n B 1 i = 1 ∑ B ∥ ϵ ^ i − ϵ t , i g t ∥ 2 2 + c m i n ≤ L AUCGR ≤ λ m a x B 1 i = 1 ∑ B ∥ ϵ ^ i − ϵ t , i g t ∥ 2 2 + c m a x , (182)
for some positive constants λ min , λ max \lambda_{\min},\lambda_{\max} λ m i n , λ m a x and constants c min , c max c_{\min},c_{\max} c m i n , c m a x . This proves the claimed affine control by the reconstruction error.
This completes the proof.
Proof of Proposition 1
Restate of Proposition 1. Let q : X → R q:\mathcal{X}\to\mathbb{R} q : X → R be a scoring function, and let x + ∼ P + x^{+}\sim P_{+} x + ∼ P + and x − ∼ P − x^{-}\sim P_{-} x − ∼ P − denote a positive and a negative sample. Assume that the negative score distribution F − ( t ) = P ( q ( x − ) ≤ t ) F_{-}(t)=\mathbb{P}(q(x^{-})\le t) F − ( t ) = P ( q ( x − ) ≤ t ) is continuous. For any measurable negative subset N \mathcal{N} N with positive probability, define the conditional ranking objective
AUC ( N ) = P ( q ( x + ) > q ( x − ) ∣ x − ∈ N ) . \operatorname{AUC}(\mathcal{N})=\mathbb{P}(q(x^{+})>q(x^{-})\mid x^{-}\in\mathcal{N}). AUC ( N ) = P ( q ( x + ) > q ( x − ) ∣ x − ∈ N ) . For any β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] , let
N l o w β = { x − : F − ( q ( x − ) ) ≤ β } . \mathcal{N}_{\mathrm{low}}^{\beta}=\{x^{-}:F_{-}(q(x^{-}))\le\beta\}. N low β = { x − : F − ( q ( x − )) ≤ β } . Then the partial AUC over [ 1 − β , 1 ] [1-\beta,1] [ 1 − β , 1 ] is defined as
pAUC [ 1 − β , 1 ] : = ∫ 1 − β 1 TPR ( FPR − 1 ( ξ ) ) d ξ . \operatorname{pAUC}_{[1-\beta,1]}:=\int_{1-\beta}^{1}\operatorname{TPR}(\operatorname{FPR}^{-1}(\xi))\,d\xi. pAUC [ 1 − β , 1 ] := ∫ 1 − β 1 TPR ( FPR − 1 ( ξ )) d ξ . where TPR ( t ) = P ( q ( x + ) > t ) \operatorname{TPR}(t)=\mathbb{P}(q(x^{+})>t) TPR ( t ) = P ( q ( x + ) > t ) , FPR ( t ) = P ( q ( x − ) > t ) \operatorname{FPR}(t)=\mathbb{P}(q(x^{-})>t) FPR ( t ) = P ( q ( x − ) > t ) , and FPR − 1 ( ξ ) \operatorname{FPR}^{-1}(\xi) FPR − 1 ( ξ ) denotes the threshold associated with false positive rate ξ \xi ξ .
Proof. Let
Y − = q ( x − ) , Y + = q ( x + ) . (183) Y^{-}=q(x^{-}),\qquad Y^{+}=q(x^{+}). \tag*{(183)} Y − = q ( x − ) , Y + = q ( x + ) . (183)
Then the negative-score cumulative distribution function is
F − ( t ) = P ( Y − ≤ t ) . (184) F_{-}(t)=\mathbb{P}(Y^{-}\le t). \tag*{(184)} F − ( t ) = P ( Y − ≤ t ) . (184)
We also define the false positive rate and the true positive rate at threshold t t t as
FPR ( t ) = P ( Y − > t ) = 1 − F − ( t ) , TPR ( t ) = P ( Y + > t ) . (185) \operatorname{FPR}(t)=\mathbb{P}(Y^{-}>t)=1-F_{-}(t),\qquad\operatorname{TPR}(t)=\mathbb{P}(Y^{+}>t). \tag*{(185)} FPR ( t ) = P ( Y − > t ) = 1 − F − ( t ) , TPR ( t ) = P ( Y + > t ) . (185)
Since F − F_{-} F − is continuous, the transformed variable
U = F − ( Y − ) (186) U=F_{-}(Y^{-}) \tag*{(186)} U = F − ( Y − ) (186)
is uniformly distributed on [ 0 , 1 ] [0,1] [ 0 , 1 ] . We also define the quantile function
Q − ( u ) = inf { t ∈ R : F − ( t ) ≥ u } , u ∈ [ 0 , 1 ] . (187) Q_{-}(u)=\inf\{t\in\mathbb{R}:F_{-}(t)\ge u\},\qquad u\in[0,1]. \tag*{(187)} Q − ( u ) = inf { t ∈ R : F − ( t ) ≥ u } , u ∈ [ 0 , 1 ] . (187)
For any score level z z z , define
G ( z ) = P ( Y + > z ) . (188) G(z)=\mathbb{P}(Y^{+}>z). \tag*{(188)} G ( z ) = P ( Y + > z ) . (188)
The quantity G ( z ) G(z) G ( z ) is exactly TPR ( z ) \operatorname{TPR}(z) TPR ( z ) .
Step 1. Conditioning on low-score negatives.
Consider
N l o w β = { x − : F − ( Y − ) ≤ β } = { x − : U ≤ β } . (189) \mathcal{N}_{\mathrm{low}}^{\beta}=\{x^{-}:F_{-}(Y^{-})\le\beta\}=\{x^{-}:U\le\beta\}. \tag*{(189)} N low β = { x − : F − ( Y − ) ≤ β } = { x − : U ≤ β } . (189)
Because U ∼ Unif [ 0 , 1 ] U\sim\operatorname{Unif}[0,1] U ∼ Unif [ 0 , 1 ] , we have
P ( U ≤ β ) = β . (190) \mathbb{P}(U\le\beta)=\beta. \tag*{(190)} P ( U ≤ β ) = β . (190)
Therefore, conditioned on the event U ≤ β U\le\beta U ≤ β , the variable U U U is uniformly distributed on [ 0 , β ] [0,\beta] [ 0 , β ] with density 1 / β 1/\beta 1/ β . Hence
AUC ( N l o w β ) = P ( Y + > Y − ∣ U ≤ β ) = 1 β ∫ 0 β G ( Q − ( u ) ) d u . (191) \begin{aligned}
\operatorname{AUC}(\mathcal{N}_{\mathrm{low}}^{\beta})
&=\mathbb{P}(Y^{+}>Y^{-}\mid U\le\beta)\\
&=\frac{1}{\beta}\int_{0}^{\beta}G(Q_{-}(u))\,du. \tag*{(191)}
\end{aligned} AUC ( N low β ) = P ( Y + > Y − ∣ U ≤ β ) = β 1 ∫ 0 β G ( Q − ( u )) d u . (191)
This identity means that, after restricting negatives to the lowest-score β \beta β fraction, we average the ranking probability over the quantile range u ∈ [ 0 , β ] u\in[0,\beta] u ∈ [ 0 , β ] .
Now make the change of variable
ξ = 1 − u . (192) \xi=1-u. \tag*{(192)} ξ = 1 − u . (192)
When u = 0 u=0 u = 0 , we have ξ = 1 \xi=1 ξ = 1 . When u = β u=\beta u = β , we have ξ = 1 − β \xi=1-\beta ξ = 1 − β . Since d u = − d ξ du=-d\xi d u = − d ξ , reversing the integration limits removes the minus sign. Therefore,
AUC ( N l o w β ) = 1 β ∫ 1 − β 1 G ( Q − ( 1 − ξ ) ) d ξ . (193) \operatorname{AUC}(\mathcal{N}_{\mathrm{low}}^{\beta})=\frac{1}{\beta}\int_{1-\beta}^{1}G(Q_{-}(1-\xi))\,d\xi. \tag*{(193)} AUC ( N low β ) = β 1 ∫ 1 − β 1 G ( Q − ( 1 − ξ )) d ξ . (193)
Step 2. Connecting the conditional objective to partial AUC.
For any ξ ∈ [ 0 , 1 ] \xi\in[0,1] ξ ∈ [ 0 , 1 ] , define the generalized inverse of the false positive rate as
FPR − 1 ( ξ ) = Q − ( 1 − ξ ) . (194) \operatorname{FPR}^{-1}(\xi)=Q_{-}(1-\xi). \tag*{(194)} FPR − 1 ( ξ ) = Q − ( 1 − ξ ) . (194)
We now show that FPR − 1 ( ξ ) \operatorname{FPR}^{-1}(\xi) FPR − 1 ( ξ ) is exactly the score threshold whose false positive rate is ξ \xi ξ . By definition,
FPR ( FPR − 1 ( ξ ) ) = P ( Y − > Q − ( 1 − ξ ) ) = 1 − P ( Y − ≤ Q − ( 1 − ξ ) ) . (195) \operatorname{FPR}\left(\operatorname{FPR}^{-1}(\xi)\right) = \mathbb{P}\left(Y^{-} > Q_{-}(1-\xi)\right) = 1-\mathbb{P}\left(Y^{-} \le Q_{-}(1-\xi)\right). \tag*{(195)} FPR ( FPR − 1 ( ξ ) ) = P ( Y − > Q − ( 1 − ξ ) ) = 1 − P ( Y − ≤ Q − ( 1 − ξ ) ) . (195)
Since F − F_{-} F − is continuous, we have
P ( Y − ≤ Q − ( 1 − ξ ) ) = F − ( Q − ( 1 − ξ ) ) = 1 − ξ . (196) \mathbb{P}\left(Y^{-} \le Q_{-}(1-\xi)\right) = F_{-}\left(Q_{-}(1-\xi)\right) = 1-\xi. \tag*{(196)} P ( Y − ≤ Q − ( 1 − ξ ) ) = F − ( Q − ( 1 − ξ ) ) = 1 − ξ . (196)
Substituting this identity above yields
FPR ( FPR − 1 ( ξ ) ) = ξ . (197) \operatorname{FPR}\left(\operatorname{FPR}^{-1}(\xi)\right) = \xi. \tag*{(197)} FPR ( FPR − 1 ( ξ ) ) = ξ . (197)
Therefore, FPR − 1 ( ξ ) \operatorname{FPR}^{-1}(\xi) FPR − 1 ( ξ ) is the threshold associated with false positive rate ξ \xi ξ . The corresponding true positive rate is
TPR ( FPR − 1 ( ξ ) ) = P ( Y + > FPR − 1 ( ξ ) ) = P ( Y + > Q − ( 1 − ξ ) ) = G ( Q − ( 1 − ξ ) ) . (198) \operatorname{TPR}\left(\operatorname{FPR}^{-1}(\xi)\right) = \mathbb{P}\left(Y^{+} > \operatorname{FPR}^{-1}(\xi)\right) = \mathbb{P}\left(Y^{+} > Q_{-}(1-\xi)\right) = G\left(Q_{-}(1-\xi)\right). \tag*{(198)} TPR ( FPR − 1 ( ξ ) ) = P ( Y + > FPR − 1 ( ξ ) ) = P ( Y + > Q − ( 1 − ξ ) ) = G ( Q − ( 1 − ξ ) ) . (198)
Substituting Eq. (198) into Eq. (193) , we obtain
AUC ( N l o w β ) = 1 β ∫ 1 − β 1 TPR ( FPR − 1 ( ξ ) ) d ξ . (199) \operatorname{AUC}\left(\mathcal{N}_{\mathrm{low}}^{\beta}\right) = \frac{1}{\beta}\int_{1-\beta}^{1} \operatorname{TPR}\left(\operatorname{FPR}^{-1}(\xi)\right)\,d\xi. \tag*{(199)} AUC ( N low β ) = β 1 ∫ 1 − β 1 TPR ( FPR − 1 ( ξ ) ) d ξ . (199)
Multiplying both sides by β \beta β gives
pAUC [ 1 − β , 1 ] : = ∫ 1 − β 1 TPR ( FPR − 1 ( ξ ) ) d ξ = β AUC ( N l o w β ) . (200) \operatorname{pAUC}_{[1-\beta,1]} := \int_{1-\beta}^{1} \operatorname{TPR}\left(\operatorname{FPR}^{-1}(\xi)\right)\,d\xi= \beta\operatorname{AUC}\left(\mathcal{N}_{\mathrm{low}}^{\beta}\right). \tag*{(200)} pAUC [ 1 − β , 1 ] := ∫ 1 − β 1 TPR ( FPR − 1 ( ξ ) ) d ξ = β AUC ( N low β ) . (200)
This completes the proof.
Appendix G: Proof of Theorem 4
Restate of Theorem 4. For anchor i i i , let P i \mathcal{P}_i P i denote the positive set and let N i \mathcal{N}_i N i denote the negative set. Then we have the following result:
L D C R ( i ) ≥ 1 ∣ P i ∣ ∣ N i ∣ ∑ p ∈ P i ∑ n ∈ N i ℓ ( q i ( p ) − q i ( n ) ) . \mathcal{L}_{\mathrm{DCR}}^{(i)} \geq\frac{1}{|\mathcal{P}_i||\mathcal{N}_i|}\sum_{p\in\mathcal{P}_i}\sum_{n\in\mathcal{N}_i}\ell(q_i(p)-q_i(n)). L DCR ( i ) ≥ ∣ P i ∣∣ N i ∣ 1 p ∈ P i ∑ n ∈ N i ∑ ℓ ( q i ( p ) − q i ( n )) .
where ℓ ( ⋅ ) \ell(\cdot) ℓ ( ⋅ ) is instantiated as the logistic surrogate function
ℓ l o g i s t i c ( u ) = log ( 1 + exp ( − u τ ) ) . \ell_{\mathrm{logistic}}(u)=\log\left(1+\exp\left(-\frac{u}{\tau}\right)\right). ℓ logistic ( u ) = log ( 1 + exp ( − τ u ) ) .
Moreover, the bound is tight if and only if ∣ P i ∣ = ∣ N i ∣ = 1 |\mathcal{P}_i|=|\mathcal{N}_i|=1 ∣ P i ∣ = ∣ N i ∣ = 1 .
Proof. Let C i = P i ∪ N i C_i=\mathcal{P}_i\cup\mathcal{N}_i C i = P i ∪ N i denote the comparison set. The anchor-wise conference loss takes the form
L D C R ( i ) = − 1 ∣ P i ∣ ∑ p ∈ P i log exp ( q i ( p ) / τ ) ∑ c ∈ C i exp ( q i ( c ) / τ ) . (201) \mathcal{L}_{\mathrm{DCR}}^{(i)}=-\frac{1}{|\mathcal{P}_i|}\sum_{p\in\mathcal{P}_i}\log\frac{\exp(q_i(p)/\tau)}{\sum_{c\in C_i}\exp(q_i(c)/\tau)}. \tag*{(201)} L DCR ( i ) = − ∣ P i ∣ 1 p ∈ P i ∑ log ∑ c ∈ C i exp ( q i ( c ) / τ ) exp ( q i ( p ) / τ ) . (201)
Fix a positive p ∈ P i p\in\mathcal{P}_i p ∈ P i . The corresponding summand in L D C R ( i ) \mathcal{L}_{\mathrm{DCR}}^{(i)} L DCR ( i ) is
T i , p : = − log exp ( q i ( p ) / τ ) ∑ c ∈ C i exp ( q i ( c ) / τ ) . (202) T_{i,p}:=-\log\frac{\exp(q_i(p)/\tau)}{\sum_{c\in C_i}\exp(q_i(c)/\tau)}. \tag*{(202)} T i , p := − log ∑ c ∈ C i exp ( q i ( c ) / τ ) exp ( q i ( p ) / τ ) . (202)
By direct expansion,
T i , p = − log exp ( q i ( p ) / τ ) ∑ c ∈ P i ∪ N i exp ( q i ( c ) / τ ) = log ( 1 + ∑ p ′ ∈ P i ∖ { p } exp ( q i ( p ′ ) − q i ( p ) τ ) + ∑ n ∈ N i exp ( q i ( n ) − q i ( p ) τ ) ) . (203) \begin{aligned}
T_{i,p}&=-\log\frac{\exp(q_i(p)/\tau)}{\sum_{c\in\mathcal{P}_i\cup\mathcal{N}_i}\exp(q_i(c)/\tau)}\\
&=\log\left(1+\sum_{p'\in\mathcal{P}_i\setminus\{p\}}\exp\left(\frac{q_i(p')-q_i(p)}{\tau}\right)+\sum_{n\in\mathcal{N}_i}\exp\left(\frac{q_i(n)-q_i(p)}{\tau}\right)\right). \tag*{(203)}
\end{aligned} T i , p = − log ∑ c ∈ P i ∪ N i exp ( q i ( c ) / τ ) exp ( q i ( p ) / τ ) = log 1 + p ′ ∈ P i ∖ { p } ∑ exp ( τ q i ( p ′ ) − q i ( p ) ) + n ∈ N i ∑ exp ( τ q i ( n ) − q i ( p ) ) . (203)
Since the terms contributed by P i ∖ { p } \mathcal{P}_i\setminus\{p\} P i ∖ { p } are nonnegative, we obtain
T i , p ≥ log ( 1 + ∑ n ∈ N i exp ( q i ( n ) − q i ( p ) τ ) ) . (204) T_{i,p}\geq\log\left(1+\sum_{n\in\mathcal{N}_i}\exp\left(\frac{q_i(n)-q_i(p)}{\tau}\right)\right). \tag*{(204)} T i , p ≥ log ( 1 + n ∈ N i ∑ exp ( τ q i ( n ) − q i ( p ) ) ) . (204)
By the definition of the pairwise logistic surrogate, we have
ℓ ( q i ( p ) − q i ( n ) ) = log ( 1 + exp ( q i ( n ) − q i ( p ) τ ) ) . (205) \ell(q_i(p)-q_i(n))=\log\left(1+\exp\left(\frac{q_i(n)-q_i(p)}{\tau}\right)\right). \tag*{(205)} ℓ ( q i ( p ) − q i ( n )) = log ( 1 + exp ( τ q i ( n ) − q i ( p ) ) ) . (205)
Averaging over n ∈ N i n\in\mathcal{N}_i n ∈ N i and applying Jensen’s inequality to the concave function log ( ⋅ ) \log(\cdot) log ( ⋅ ) yields
1 ∣ N i ∣ ∑ n ∈ N i ℓ ( q i ( p ) − q i ( n ) ) = 1 ∣ N i ∣ ∑ n ∈ N i log ( 1 + exp ( q i ( n ) − q i ( p ) τ ) ) ≤ log ( 1 + 1 ∣ N i ∣ ∑ n ∈ N i exp ( q i ( n ) − q i ( p ) τ ) ) . (206) \begin{aligned}
\frac{1}{|\mathcal{N}_i|}\sum_{n\in\mathcal{N}_i}\ell(q_i(p)-q_i(n))
&=\frac{1}{|\mathcal{N}_i|}\sum_{n\in\mathcal{N}_i}\log\left(1+\exp\left(\frac{q_i(n)-q_i(p)}{\tau}\right)\right)\\
&\leq\log\left(1+\frac{1}{|\mathcal{N}_i|}\sum_{n\in\mathcal{N}_i}\exp\left(\frac{q_i(n)-q_i(p)}{\tau}\right)\right). \tag*{(206)}
\end{aligned} ∣ N i ∣ 1 n ∈ N i ∑ ℓ ( q i ( p ) − q i ( n )) = ∣ N i ∣ 1 n ∈ N i ∑ log ( 1 + exp ( τ q i ( n ) − q i ( p ) ) ) ≤ log ( 1 + ∣ N i ∣ 1 n ∈ N i ∑ exp ( τ q i ( n ) − q i ( p ) ) ) . (206)
Because ∣ N i ∣ ≥ 1 |\mathcal{N}_i|\geq1 ∣ N i ∣ ≥ 1 , we further have
log ( 1 + 1 ∣ N i ∣ ∑ n ∈ N i exp ( q i ( n ) − q i ( p ) τ ) ) ≤ log ( 1 + ∑ n ∈ N i exp ( q i ( n ) − q i ( p ) τ ) ) ≤ T i , p . (207) \log\left(1+\frac{1}{|\mathcal{N}_i|}\sum_{n\in\mathcal{N}_i}\exp\left(\frac{q_i(n)-q_i(p)}{\tau}\right)\right)\leq\log\left(1+\sum_{n\in\mathcal{N}_i}\exp\left(\frac{q_i(n)-q_i(p)}{\tau}\right)\right)\leq T_{i,p}. \tag*{(207)} log ( 1 + ∣ N i ∣ 1 n ∈ N i ∑ exp ( τ q i ( n ) − q i ( p ) ) ) ≤ log ( 1 + n ∈ N i ∑ exp ( τ q i ( n ) − q i ( p ) ) ) ≤ T i , p . (207)
Therefore,
1 ∣ N i ∣ ∑ n ∈ N i ℓ ( q i ( p ) − q i ( n ) ) ≤ T i , p . (208) \frac{1}{|\mathcal{N}_i|}\sum_{n\in\mathcal{N}_i}\ell(q_i(p)-q_i(n))\leq T_{i,p}. \tag*{(208)} ∣ N i ∣ 1 n ∈ N i ∑ ℓ ( q i ( p ) − q i ( n )) ≤ T i , p . (208)
Averaging the above inequality over p ∈ P i p\in\mathcal{P}_i p ∈ P i gives
1 ∣ P i ∣ ∣ N i ∣ ∑ p ∈ P i ∑ n ∈ N i ℓ ( q i ( p ) − q i ( n ) ) ≤ L D C R ( i ) . (209) \frac{1}{|\mathcal{P}_i||\mathcal{N}_i|}\sum_{p\in\mathcal{P}_i}\sum_{n\in\mathcal{N}_i}\ell(q_i(p)-q_i(n))\leq\mathcal{L}_{\mathrm{DCR}}^{(i)}. \tag*{(209)} ∣ P i ∣∣ N i ∣ 1 p ∈ P i ∑ n ∈ N i ∑ ℓ ( q i ( p ) − q i ( n )) ≤ L DCR ( i ) . (209)
We finally characterize when the bound is tight. Assume τ > 0 \tau>0 τ > 0 and all scores are finite, so every exponential term is strictly positive. For a fixed p ∈ P i p \in\mathcal{P}_i p ∈ P i , the inequality
T i , p ≥ log ( 1 + ∑ n ∈ N i exp ( q i ( n ) − q i ( p ) τ ) ) (210) T_{i,p} \ge\log\left(1+\sum_{n\in\mathcal{N}_i}\exp\left(\frac{q_i(n)-q_i(p)}{\tau}\right)\right) \tag*{(210)} T i , p ≥ log ( 1 + n ∈ N i ∑ exp ( τ q i ( n ) − q i ( p ) ) ) (210)
is an equality if and only if
P i ∖ { p } = ∅ , (211) \mathcal{P}_i \setminus\{p\} = \varnothing, \tag*{(211)} P i ∖ { p } = ∅ , (211)
i.e., ∣ P i ∣ = 1 |\mathcal{P}_i|=1 ∣ P i ∣ = 1 .
Moreover, since
log ( 1 + 1 ∣ N i ∣ ∑ n ∈ N i exp ( q i ( n ) − q i ( p ) τ ) ) ≤ log ( 1 + ∑ n ∈ N i exp ( q i ( n ) − q i ( p ) τ ) ) , (212) \log\left(1+\frac{1}{|\mathcal{N}_i|}\sum_{n\in\mathcal{N}_i}\exp\left(\frac{q_i(n)-q_i(p)}{\tau}\right)\right)
\leq
\log\left(1+\sum_{n\in\mathcal{N}_i}\exp\left(\frac{q_i(n)-q_i(p)}{\tau}\right)\right), \tag*{(212)} log ( 1 + ∣ N i ∣ 1 n ∈ N i ∑ exp ( τ q i ( n ) − q i ( p ) ) ) ≤ log ( 1 + n ∈ N i ∑ exp ( τ q i ( n ) − q i ( p ) ) ) , (212)
and the summed exponential term is strictly positive, equality holds if and only if ∣ N i ∣ = 1 |\mathcal{N}_i|=1 ∣ N i ∣ = 1 . Therefore, the bound is tight if and only if
∣ P i ∣ = ∣ N i ∣ = 1. (213) |\mathcal{P}_i|=|\mathcal{N}_i|=1. \tag*{(213)} ∣ P i ∣ = ∣ N i ∣ = 1. (213)
In this case, letting P i = { p } \mathcal{P}_i=\{p\} P i = { p } and N i = { n } \mathcal{N}_i=\{n\} N i = { n } ,
L D C R ( i ) = log ( 1 + exp ( q i ( n ) − q i ( p ) τ ) ) = ℓ ( q i ( p ) − q i ( n ) ) . (214) \mathcal{L}_{\mathrm{DCR}}^{(i)}=\log\left(1+\exp\left(\frac{q_i(n)-q_i(p)}{\tau}\right)\right)=\ell(q_i(p)-q_i(n)). \tag*{(214)} L DCR ( i ) = log ( 1 + exp ( τ q i ( n ) − q i ( p ) ) ) = ℓ ( q i ( p ) − q i ( n )) . (214)
Hence, DCR and logistic AUCGR are exactly equivalent when each anchor has one positive and one negative; otherwise, the upper bound is strict.
This completes the proof.
Proof of Theorem 5
Lemma 6 (Hoeffding Inequality [140 ] ). Let X 1 , … , X N X_1,\ldots,X_N X 1 , … , X N be independent random variables such that
X i ∈ [ a i , b i ] for all i = 1 , … , N . (215) X_i \in[a_i,b_i]\qquad\text{for all }i=1,\ldots,N. \tag*{(215)} X i ∈ [ a i , b i ] for all i = 1 , … , N . (215) Then, for any t > 0 t>0 t > 0 ,
P ( ∣ 1 N ∑ i = 1 N X i − 1 N ∑ i = 1 N E [ X i ] ∣ ≥ t ) ≤ 2 exp ( − 2 N 2 t 2 ∑ i = 1 N ( b i − a i ) 2 ) . (216) \mathbb{P}\left(\left|\frac{1}{N}\sum_{i=1}^{N}X_i-\frac{1}{N}\sum_{i=1}^{N}\mathbb{E}[X_i]\right|\geq t\right)
\leq2\exp\left(-\frac{2N^2t^2}{\sum_{i=1}^{N}(b_i-a_i)^2}\right). \tag*{(216)} P ( N 1 i = 1 ∑ N X i − N 1 i = 1 ∑ N E [ X i ] ≥ t ) ≤ 2 exp ( − ∑ i = 1 N ( b i − a i ) 2 2 N 2 t 2 ) . (216)
Lemma 7 (Uniform Convergence over a Finite Function Class). Let { g 1 , … , g N G } \{g_1,\ldots,g_{N_{\mathcal{G}}}\} { g 1 , … , g N G } be a finite collection of measurable functions on Z \mathcal{Z} Z . Assume that, for every k k k and every z z z ,
0 ≤ g k ( z ) ≤ M . (217) 0\leq g_k(z)\leq M. \tag*{(217)} 0 ≤ g k ( z ) ≤ M . (217) Then, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ ,
sup 1 ≤ k ≤ N G ∣ E [ g k ( z ) ] − E ^ N [ g k ] ∣ ≤ M log ( 2 N G / δ ) 2 N . (218) \sup_{1\leq k\leq N_{\mathcal{G}}}\left|\mathbb{E}[g_k(z)]-\widehat{\mathbb{E}}_N[g_k]\right|
\leq M\sqrt{\frac{\log(2N_{\mathcal{G}}/\delta)}{2N}}. \tag*{(218)} 1 ≤ k ≤ N G sup E [ g k ( z )] − E N [ g k ] ≤ M 2 N log ( 2 N G / δ ) . (218)
Proof. Fix one function g k g_k g k . Since the anchor tuples z 1 , … , z N z_1,\ldots,z_N z 1 , … , z N are i.i.d., define
X i : = g k ( z i ) , i = 1 , … , N . (219) X_i:=g_k(z_i),\qquad i=1,\ldots,N. \tag*{(219)} X i := g k ( z i ) , i = 1 , … , N . (219)
Then X 1 , … , X N X_1,\ldots,X_N X 1 , … , X N are independent random variables, and they have the same expectation:
E [ X i ] = E [ g k ( z ) ] . (220) \mathbb{E}[X_i]=\mathbb{E}[g_k(z)]. \tag*{(220)} E [ X i ] = E [ g k ( z )] . (220)
Moreover, the assumption 0 ≤ g k ( z ) ≤ M 0\leq g_k(z)\leq M 0 ≤ g k ( z ) ≤ M implies
0 ≤ X i ≤ M for all i = 1 , … , N . (221) 0\leq X_i\leq M\qquad\text{for all }i=1,\ldots,N. \tag*{(221)} 0 ≤ X i ≤ M for all i = 1 , … , N . (221)
Therefore, Lemma 6 applies with
a i = 0 , b i = M for all i = 1 , … , N . (222) a_i=0,\qquad b_i=M\qquad\text{for all }i=1,\ldots,N. \tag*{(222)} a i = 0 , b i = M for all i = 1 , … , N . (222)
Under this choice,
b i − a i = M for all i = 1 , … , N , (223) b_i-a_i=M\qquad\text{for all }i=1,\ldots,N, \tag*{(223)} b i − a i = M for all i = 1 , … , N , (223)
and hence
∑ i = 1 N ( b i − a i ) 2 = ∑ i = 1 N M 2 = N M 2 . (224) \sum_{i=1}^{N}(b_i-a_i)^2=\sum_{i=1}^{N}M^2=NM^2. \tag*{(224)} i = 1 ∑ N ( b i − a i ) 2 = i = 1 ∑ N M 2 = N M 2 . (224)
Substituting these identities into Eq. (216) yields
P ( ∣ 1 N ∑ i = 1 N X i − 1 N ∑ i = 1 N E [ X i ] ∣ > t ) ≤ 2 exp ( − 2 N t 2 M 2 ) . (225) \mathbb{P}\left(\left|\frac{1}{N}\sum_{i=1}^{N}X_i-\frac{1}{N}\sum_{i=1}^{N}\mathbb{E}[X_i]\right|>t\right)\le2\exp\left(-\frac{2Nt^2}{M^2}\right). \tag*{(225)} P ( N 1 i = 1 ∑ N X i − N 1 i = 1 ∑ N E [ X i ] > t ) ≤ 2 exp ( − M 2 2 N t 2 ) . (225)
Since
1 N ∑ i = 1 N X i = 1 N ∑ i = 1 N g k ( z i ) = E ^ N [ g k ] (226) \frac{1}{N}\sum_{i=1}^{N}X_i=\frac{1}{N}\sum_{i=1}^{N}g_k(z_i)=\widehat{\mathbb{E}}_N[g_k] \tag*{(226)} N 1 i = 1 ∑ N X i = N 1 i = 1 ∑ N g k ( z i ) = E N [ g k ] (226)
and
1 N ∑ i = 1 N E [ X i ] = E [ g k ( z ) ] , (227) \frac{1}{N}\sum_{i=1}^{N}\mathbb{E}[X_i]=\mathbb{E}[g_k(z)], \tag*{(227)} N 1 i = 1 ∑ N E [ X i ] = E [ g k ( z )] , (227)
we obtain
P ( ∣ E [ g k ( z ) ] − E ^ N [ g k ] ∣ > t ) ≤ 2 exp ( − 2 N t 2 M 2 ) . (228) \mathbb{P}\left(\left|\mathbb{E}[g_k(z)]-\widehat{\mathbb{E}}_N[g_k]\right|>t\right)\le2\exp\left(-\frac{2Nt^2}{M^2}\right). \tag*{(228)} P ( E [ g k ( z )] − E N [ g k ] > t ) ≤ 2 exp ( − M 2 2 N t 2 ) . (228)
This bound holds for every index k k k . We now require the same event to hold simultaneously for all k = 1 , … , N G k=1,\ldots,N_{\mathcal{G}} k = 1 , … , N G . Using the union bound,
P ( sup 1 ≤ k ≤ N G ∣ E [ g k ( z ) ] − E ^ N [ g k ] ∣ > t ) ≤ 2 N G exp ( − 2 N t 2 M 2 ) . (229) \mathbb{P}\left(\sup_{1\le k\le N_{\mathcal{G}}}\left|\mathbb{E}[g_k(z)]-\widehat{\mathbb{E}}_N[g_k]\right|>t\right)\le2N_{\mathcal{G}}\exp\left(-\frac{2Nt^2}{M^2}\right). \tag*{(229)} P ( 1 ≤ k ≤ N G sup E [ g k ( z )] − E N [ g k ] > t ) ≤ 2 N G exp ( − M 2 2 N t 2 ) . (229)
Indeed, the event
sup 1 ≤ k ≤ N G ∣ E [ g k ( z ) ] − E ^ N [ g k ] ∣ > t (230) \sup_{1\le k\le N_{\mathcal{G}}}\left|\mathbb{E}[g_k(z)]-\widehat{\mathbb{E}}_N[g_k]\right|>t \tag*{(230)} 1 ≤ k ≤ N G sup E [ g k ( z )] − E N [ g k ] > t (230)
means that there exists at least one index k k k such that
∣ E [ g k ( z ) ] − E ^ N [ g k ] ∣ > t . (231) \left|\mathbb{E}[g_k(z)]-\widehat{\mathbb{E}}_N[g_k]\right|>t. \tag*{(231)} E [ g k ( z )] − E N [ g k ] > t . (231)
Therefore,
P ( sup 1 ≤ k ≤ N G ∣ E [ g k ( z ) ] − E ^ N [ g k ] ∣ > t ) ≤ ∑ k = 1 N G P ( ∣ E [ g k ( z ) ] − E ^ N [ g k ] ∣ > t ) , (232) \mathbb{P}\left(\sup_{1\le k\le N_{\mathcal{G}}}\left|\mathbb{E}[g_k(z)]-\widehat{\mathbb{E}}_N[g_k]\right|>t\right)\le\sum_{k=1}^{N_{\mathcal{G}}}\mathbb{P}\left(\left|\mathbb{E}[g_k(z)]-\widehat{\mathbb{E}}_N[g_k]\right|>t\right), \tag*{(232)} P ( 1 ≤ k ≤ N G sup E [ g k ( z )] − E N [ g k ] > t ) ≤ k = 1 ∑ N G P ( E [ g k ( z )] − E N [ g k ] > t ) , (232)
and each summand is bounded by Eq. (228) .
To make the right hand side of the union bound no larger than δ \delta δ , it is sufficient to require
2 N G exp ( − 2 N t 2 M 2 ) ≤ δ . (233) 2N_{\mathcal{G}}\exp\left(-\frac{2Nt^2}{M^2}\right)\le\delta. \tag*{(233)} 2 N G exp ( − M 2 2 N t 2 ) ≤ δ . (233)
Since δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and N G ≥ 1 N_{\mathcal{G}}\ge1 N G ≥ 1 , this requirement is equivalent to
exp ( − 2 N t 2 M 2 ) ≤ δ 2 N G , (234) \exp\left(-\frac{2Nt^2}{M^2}\right)\le\frac{\delta}{2N_{\mathcal{G}}}, \tag*{(234)} exp ( − M 2 2 N t 2 ) ≤ 2 N G δ , (234)
which gives
− 2 N t 2 M 2 ≤ log δ 2 N G = − log 2 N G δ . (235) -\frac{2Nt^2}{M^2}\le\log\frac{\delta}{2N_{\mathcal{G}}}=-\log\frac{2N_{\mathcal{G}}}{\delta}. \tag*{(235)} − M 2 2 N t 2 ≤ log 2 N G δ = − log δ 2 N G . (235)
Equivalently,
2 N t 2 M 2 ≥ log 2 N G δ . (236) \frac{2Nt^2}{M^2}\ge\log\frac{2N_{\mathcal{G}}}{\delta}. \tag*{(236)} M 2 2 N t 2 ≥ log δ 2 N G . (236)
Solving for the nonnegative threshold yields
t ≥ M log ( 2 N G / δ ) 2 N . (237) t\ge M\sqrt{\frac{\log(2N_{\mathcal{G}}/\delta)}{2N}}. \tag*{(237)} t ≥ M 2 N log ( 2 N G / δ ) . (237)
Thus the smallest threshold that ensures a failure probability at most δ \delta δ is
t = M log ( 2 N G / δ ) 2 N . (238) t=M\sqrt{\frac{\log(2N_{\mathcal{G}}/\delta)}{2N}}. \tag*{(238)} t = M 2 N log ( 2 N G / δ ) . (238)
Hence, with probability at least 1 − δ 1-\delta 1 − δ ,
sup 1 ≤ k ≤ N G ∣ E [ g k ( z ) ] − E ^ N [ g k ] ∣ ≤ M log ( 2 N G / δ ) 2 N . (239) \sup_{1\le k\le N_{\mathcal{G}}}\left|\mathbb{E}[g_k(z)]-\widehat{\mathbb{E}}_N[g_k]\right|\le M\sqrt{\frac{\log(2N_{\mathcal{G}}/\delta)}{2N}}. \tag*{(239)} 1 ≤ k ≤ N G sup E [ g k ( z )] − E N [ g k ] ≤ M 2 N log ( 2 N G / δ ) . (239)
This completes the proof.
Restate of Theorem 5. Under Assumption 1, for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and any fixed ε ∈ ( 0 , R ] \varepsilon\in(0,R] ε ∈ ( 0 , R ] , with probability at least 1 − δ 1-\delta 1 − δ , the following inequality holds simultaneously for all f ∈ F f \in\mathcal{F} f ∈ F :
∣ R α ( f ) − R ^ α ( f ) ∣ ≤ 2 ε + M A log ( R / ε ) + log ( 2 / δ ) 2 N . (240) \left\lvert\mathcal{R}_{\alpha}(f)-\widehat{\mathcal{R}}_{\alpha}(f)\right\rvert\le2\varepsilon
+M\sqrt{\frac{A\log(R/\varepsilon)+\log(2/\delta)}{2N}}. \tag*{(240)} R α ( f ) − R α ( f ) ≤ 2 ε + M 2 N A log ( R / ε ) + log ( 2/ δ ) . (240)
In particular, choosing ε = R / N \varepsilon=R/N ε = R / N yields
∣ R α ( f ) − R ^ α ( f ) ∣ ≤ 2 R N + M A log N + log ( 2 / δ ) 2 N . (241) \left\lvert\mathcal{R}_{\alpha}(f)-\widehat{\mathcal{R}}_{\alpha}(f)\right\rvert\le\frac{2R}{N}
+M\sqrt{\frac{A\log N+\log(2/\delta)}{2N}}. \tag*{(241)} R α ( f ) − R α ( f ) ≤ N 2 R + M 2 N A log N + log ( 2/ δ ) . (241)
Proof. Fix any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and any ε ∈ ( 0 , R ] \varepsilon\in(0,R] ε ∈ ( 0 , R ] .
For convenience, define
E ^ N [ g ] = 1 N ∑ i = 1 N g ( z i ) . (242) \widehat{\mathbb{E}}_{N}[g]=\frac{1}{N}\sum_{i=1}^{N}g(z_i). \tag*{(242)} E N [ g ] = N 1 i = 1 ∑ N g ( z i ) . (242)
Write g f ( z ) = g f ( α ) ( z ) g_f(z)=g_f^{(\alpha)}(z) g f ( z ) = g f ( α ) ( z ) . Then
R α ( f ) = E [ g f ( z ) ] , R ^ α ( f ) = E ^ N [ g f ] . (243) \mathcal{R}_{\alpha}(f)=\mathbb{E}[g_f(z)],\qquad\widehat{\mathcal{R}}_{\alpha}(f)=\widehat{\mathbb{E}}_{N}[g_f]. \tag*{(243)} R α ( f ) = E [ g f ( z )] , R α ( f ) = E N [ g f ] . (243)
For every anchor tuple z z z and every retained pair ( p , n ) (p,n) ( p , n ) , define
Δ f ( z ; p , n ) = d ( e ^ , p ) − d ( e ^ , n ) . (244) \Delta_f(z;p,n)=d(\hat{e},p)-d(\hat{e},n). \tag*{(244)} Δ f ( z ; p , n ) = d ( e ^ , p ) − d ( e ^ , n ) . (244)
Assumption 1 implies
− 2 ≤ Δ f ( z ; p , n ) ≤ 2 , 0 ≤ ℓ ( Δ f ( z ; p , n ) ) ≤ M . (245) -2\le\Delta_f(z;p,n)\le2,\qquad0\le\ell(\Delta_f(z;p,n))\le M. \tag*{(245)} − 2 ≤ Δ f ( z ; p , n ) ≤ 2 , 0 ≤ ℓ ( Δ f ( z ; p , n )) ≤ M . (245)
Since g f ( z ) g_f(z) g f ( z ) is the average of these surrogate terms over all retained positive-negative pairs, it follows that
0 ≤ g f ( z ) ≤ M . (246) 0\le g_f(z)\le M. \tag*{(246)} 0 ≤ g f ( z ) ≤ M . (246)
Let
{ g 1 , … , g N G } ⊂ G α (247) \{g_1,\ldots,g_{N_{\mathcal{G}}}\}\subset\mathcal{G}_{\alpha} \tag*{(247)} { g 1 , … , g N G } ⊂ G α (247)
be an ε \varepsilon ε -cover of G α \mathcal{G}_{\alpha} G α under the sup norm, where
N G = N ( G α , ε , ∥ ⋅ ∥ ∞ ) . (248) N_{\mathcal{G}}=\mathcal{N}(\mathcal{G}_{\alpha},\varepsilon,\|\cdot\|_{\infty}). \tag*{(248)} N G = N ( G α , ε , ∥ ⋅ ∥ ∞ ) . (248)
Applying Lemma 7 to the bound in Eq. (246) , we obtain that with probability at least 1 − δ 1-\delta 1 − δ ,
sup 1 ≤ k ≤ N G ∣ E [ g k ( z ) ] − E ^ N [ g k ] ∣ ≤ M log ( 2 N G / δ ) 2 N . (249) \sup_{1\le k\le N_{\mathcal{G}}}\left\lvert\mathbb{E}[g_k(z)]-\widehat{\mathbb{E}}_{N}[g_k]\right\rvert
\le M\sqrt{\frac{\log(2N_{\mathcal{G}}/\delta)}{2N}}. \tag*{(249)} 1 ≤ k ≤ N G sup E [ g k ( z )] − E N [ g k ] ≤ M 2 N log ( 2 N G / δ ) . (249)
Now take any g ∈ G α g\in\mathcal{G}_{\alpha} g ∈ G α . Since { g 1 , … , g N G } \{g_1,\ldots,g_{N_{\mathcal{G}}}\} { g 1 , … , g N G } is an ε \varepsilon ε -cover, there exists some g k g_k g k such that
∥ g − g k ∥ ∞ ≤ ε . (250) \|g-g_k\|_{\infty}\le\varepsilon. \tag*{(250)} ∥ g − g k ∥ ∞ ≤ ε . (250)
This means that, for every anchor tuple z z z ,
∣ g ( z ) − g k ( z ) ∣ ≤ ε . (251) |g(z)-g_k(z)|\le\varepsilon. \tag*{(251)} ∣ g ( z ) − g k ( z ) ∣ ≤ ε . (251)
Taking expectation on both sides gives
∣ E [ g ( z ) ] − E [ g k ( z ) ] ∣ ≤ E [ ∣ g ( z ) − g k ( z ) ∣ ] ≤ ε . (252) |\mathbb{E}[g(z)]-\mathbb{E}[g_k(z)]|\le\mathbb{E}[|g(z)-g_k(z)|]\le\varepsilon. \tag*{(252)} ∣ E [ g ( z )] − E [ g k ( z )] ∣ ≤ E [ ∣ g ( z ) − g k ( z ) ∣ ] ≤ ε . (252)
For the empirical averages, we have
∣ E ^ N [ g ] − E ^ N [ g k ] ∣ = ∣ 1 N ∑ i = 1 N ( g ( z i ) − g k ( z i ) ) ∣ ≤ 1 N ∑ i = 1 N ∣ g ( z i ) − g k ( z i ) ∣ ≤ 1 N ∑ i = 1 N ε = ε . (253) \begin{aligned}
\left|\widehat{\mathbb{E}}_{N}[g]-\widehat{\mathbb{E}}_{N}[g_k]\right|
&=\left|\frac{1}{N}\sum_{i=1}^{N}(g(z_i)-g_k(z_i))\right|\\
&\le\frac{1}{N}\sum_{i=1}^{N}|g(z_i)-g_k(z_i)|\\
&\le\frac{1}{N}\sum_{i=1}^{N}\varepsilon\\
&=\varepsilon. \tag*{(253)}
\end{aligned} E N [ g ] − E N [ g k ] = N 1 i = 1 ∑ N ( g ( z i ) − g k ( z i )) ≤ N 1 i = 1 ∑ N ∣ g ( z i ) − g k ( z i ) ∣ ≤ N 1 i = 1 ∑ N ε = ε . (253)
We now insert g k g_k g k between the population and empirical terms for g g g . By adding and subtracting E [ g k ( z ) ] \mathbb{E}[g_k(z)] E [ g k ( z )] and E ^ N [ g k ] \widehat{\mathbb{E}}_N[g_k] E N [ g k ] , we obtain the exact identity
E [ g ( z ) ] − E ^ N [ g ] = ( E [ g ( z ) ] − E [ g k ( z ) ] ) + ( E [ g k ( z ) ] − E ^ N [ g k ] ) + ( E ^ N [ g k ] − E ^ N [ g ] ) . (254) \mathbb{E}[g(z)]-\widehat{\mathbb{E}}_N[g] = \left(\mathbb{E}[g(z)]-\mathbb{E}[g_k(z)]\right)
\\
\quad+ \left(\mathbb{E}[g_k(z)]-\widehat{\mathbb{E}}_N[g_k]\right)
\\
\quad+ \left(\widehat{\mathbb{E}}_N[g_k]-\widehat{\mathbb{E}}_N[g]\right). \tag*{(254)} E [ g ( z )] − E N [ g ] = ( E [ g ( z )] − E [ g k ( z )] ) + ( E [ g k ( z )] − E N [ g k ] ) + ( E N [ g k ] − E N [ g ] ) . (254)
Taking absolute values on both sides and applying the triangle inequality gives
∣ E [ g ( z ) ] − E ^ N [ g ] ∣ ≤ ∣ E [ g ( z ) ] − E [ g k ( z ) ] ∣ + ∣ E [ g k ( z ) ] − E ^ N [ g k ] ∣ + ∣ E ^ N [ g k ] − E ^ N [ g ] ∣ . (255) \left|\mathbb{E}[g(z)]-\widehat{\mathbb{E}}_N[g]\right| \leq\left|\mathbb{E}[g(z)]-\mathbb{E}[g_k(z)]\right|
\\
\quad+ \left|\mathbb{E}[g_k(z)]-\widehat{\mathbb{E}}_N[g_k]\right|
\\
\quad+ \left|\widehat{\mathbb{E}}_N[g_k]-\widehat{\mathbb{E}}_N[g]\right|. \tag*{(255)} E [ g ( z )] − E N [ g ] ≤ ∣ E [ g ( z )] − E [ g k ( z )] ∣ + E [ g k ( z )] − E N [ g k ] + E N [ g k ] − E N [ g ] . (255)
The first term is bounded by ε \varepsilon ε from the expectation estimate above. The third term is also bounded by ε \varepsilon ε from the empirical estimate above. For the middle term, Eq. (249) gives
∣ E [ g k ( z ) ] − E ^ N [ g k ] ∣ ≤ M log ( 2 N G / δ ) 2 N . (256) \left|\mathbb{E}[g_k(z)]-\widehat{\mathbb{E}}_N[g_k]\right| \leq M\sqrt{\frac{\log(2N_{\mathcal{G}}/\delta)}{2N}}. \tag*{(256)} E [ g k ( z )] − E N [ g k ] ≤ M 2 N log ( 2 N G / δ ) . (256)
Substituting these three bounds into Eq. (255) yields
∣ E [ g ( z ) ] − E ^ N [ g ] ∣ ≤ 2 ε + M log ( 2 N G / δ ) 2 N . (257) \left|\mathbb{E}[g(z)]-\widehat{\mathbb{E}}_N[g]\right| \leq2\varepsilon+ M\sqrt{\frac{\log(2N_{\mathcal{G}}/\delta)}{2N}}. \tag*{(257)} E [ g ( z )] − E N [ g ] ≤ 2 ε + M 2 N log ( 2 N G / δ ) . (257)
Because this bound holds for every g ∈ G α g \in\mathcal{G}_\alpha g ∈ G α , we may take the supremum over G α \mathcal{G}_\alpha G α :
sup g ∈ G α ∣ E [ g ( z ) ] − E ^ N [ g ] ∣ ≤ 2 ε + M log ( 2 N G / δ ) 2 N . (258) \sup_{g\in\mathcal{G}_\alpha}\left|\mathbb{E}[g(z)]-\widehat{\mathbb{E}}_N[g]\right| \leq2\varepsilon+ M\sqrt{\frac{\log(2N_{\mathcal{G}}/\delta)}{2N}}. \tag*{(258)} g ∈ G α sup E [ g ( z )] − E N [ g ] ≤ 2 ε + M 2 N log ( 2 N G / δ ) . (258)
Every element of G α \mathcal{G}_\alpha G α has the form g f g_f g f for some f ∈ F f \in\mathcal{F} f ∈ F . Therefore,
sup f ∈ F ∣ R α ( f ) − R ^ α ( f ) ∣ ≤ 2 ε + M log ( 2 N G / δ ) 2 N . (259) \sup_{f\in\mathcal{F}}\left|\mathcal{R}_\alpha(f)-\widehat{\mathcal{R}}_\alpha(f)\right| \leq2\varepsilon+ M\sqrt{\frac{\log(2N_{\mathcal{G}}/\delta)}{2N}}. \tag*{(259)} f ∈ F sup R α ( f ) − R α ( f ) ≤ 2 ε + M 2 N log ( 2 N G / δ ) . (259)
By Assumption 1,
log N G ≤ A log ( R / ε ) . (260) \log N_{\mathcal{G}} \leq A\log(R/\varepsilon). \tag*{(260)} log N G ≤ A log ( R / ε ) . (260)
Hence,
log ( 2 N G / δ ) = log N G + log ( 2 / δ ) ≤ A log ( R / ε ) + log ( 2 / δ ) . (261) \log(2N_{\mathcal{G}}/\delta)=\log N_{\mathcal{G}}+\log(2/\delta)\leq A\log(R/\varepsilon)+\log(2/\delta). \tag*{(261)} log ( 2 N G / δ ) = log N G + log ( 2/ δ ) ≤ A log ( R / ε ) + log ( 2/ δ ) . (261)
Substituting this estimate into the previous display yields
sup f ∈ F ∣ R α ( f ) − R ^ α ( f ) ∣ ≤ 2 ε + M A log ( R / ε ) + log ( 2 / δ ) 2 N . (262) \sup_{f\in\mathcal{F}}\left|\mathcal{R}_\alpha(f)-\widehat{\mathcal{R}}_\alpha(f)\right| \leq2\varepsilon+ M\sqrt{\frac{A\log(R/\varepsilon)+\log(2/\delta)}{2N}}. \tag*{(262)} f ∈ F sup R α ( f ) − R α ( f ) ≤ 2 ε + M 2 N A log ( R / ε ) + log ( 2/ δ ) . (262)
The above argument is valid for any fixed ε ∈ ( 0 , R ] \varepsilon\in(0,R] ε ∈ ( 0 , R ] . We now take
ε = R N . (263) \varepsilon=\frac{R}{N}. \tag*{(263)} ε = N R . (263)
Since N ≥ 1 N\geq1 N ≥ 1 , this choice belongs to ( 0 , R ] (0,R] ( 0 , R ] , and the supremum bound implies that for every f ∈ F f\in\mathcal{F} f ∈ F ,
∣ R α ( f ) − R ^ α ( f ) ∣ ≤ 2 R N + M A log N + log ( 2 / δ ) 2 N . (264) \left|\mathcal{R}_\alpha(f)-\widehat{\mathcal{R}}_\alpha(f)\right| \leq\frac{2R}{N}+M\sqrt{\frac{A\log N+\log(2/\delta)}{2N}}. \tag*{(264)} R α ( f ) − R α ( f ) ≤ N 2 R + M 2 N A log N + log ( 2/ δ ) . (264)
This completes the proof.
Appendix I: Additional Experimental Settings
In this section, we make a supplementation to Sec. 7.1.
CLIP Backbones
Here, we provide a detailed summary of the 6 types of CLIP backbones used in our experiments. They fall into 3 categories:
OpenAI CLIP ViT-L@224 and OpenAI CLIP ViT-L@336 [11 ] establish the canonical formulation of contrastive language-image pretraining, defining the feature space that later CLIP variants build upon. Their large-scale but noisy training data gives them strong open-world generalization, while their ability to perceive fine-grained visual details remains limited. The 336-resolution variant uniquely extends CLIP’s receptive field without altering the architecture, making it a standard choice for studying high-resolution alignment.
MetaCLIP ViT-L@224 and MetaCLIP ViT-H@224 [108 ] distinguishes itself by reconstructing a high-quality, well-aligned training corpus using a principled filtering pipeline rather than relying on raw web data. This dataset-centric redesign leads to representations that are more stable, less noisy, and better calibrated than those of OpenAI CLIP. The ViT-H model further pushes scaling laws within the CLIP paradigm.
SigLIP ViT-SO@224 and SigLIP ViT-SO@384 [109 ] departs from the classic softmax contrastive loss by introducing a sigmoid-based objective, fundamentally altering how positive and negative pairs influence training. This probabilistic formulation enables more fine-grained pairwise alignment and reduces overconfidence artifacts commonly observed in CLIP-like models. Its higher-resolution variant leverages the smoother objective to maintain stable training.
Evaluation Protocol for P-Ability
Datasets. Following [23 ] , [24 ] , [25 ] , we use MMVP-VLM [53 ] to evaluate the P-Ability. It is a benchmark designed to evaluate fine-grained visual perception by testing VLMs on a wide range of visual patterns. It contains human-designed symbols, shapes, transformations, and composites that isolate specific perceptual abilities such as symmetry, color matching, and geometric relations. The dataset includes tens of thousands of diverse pattern-label pairs, providing a controlled environment for assessing detailed visual understanding.
Evaluation metrics. Following [53 ] , we report Accuracy (ACC) as the evaluation metric. ACC measures the proportion of samples for which the model correctly recognizes the underlying visual pattern, serving as a direct indicator of fine-grained visual perception.
Evaluation Protocol for D-Ability
We evaluate D-Ability from three complementary perspectives. Zero-shot clustering examines whether image representations form separable semantic structures without using textual class names. Zero-shot classification evaluates whether image features can be aligned with class-name prompts. Zero-shot retrieval further tests cross-modal alignment between images and natural language descriptions.
Zero-shot Clustering
Datasets. Following [11 ] , we perform zero-shot clustering on 26 standard datasets to evaluate whether the learned visual representations form separable semantic structures. These datasets cover handwritten digits, generic objects, fine-grained categories, textures, scenes, remote sensing images, medical images, and distribution-shifted recognition benchmarks.
CIFAR-10 [114 ] is a natural image dataset designed for general object recognition across 10 categories. Its low-resolution 32 × 32 images contain large appearance variations, making it a compact benchmark for evaluating semantic separability.
CIFAR-100 [114 ] extends CIFAR-10 to 100 fine-grained categories. Compared with CIFAR-10, it provides a more challenging test of whether visual features can distinguish visually similar classes.
MNIST [115 ] contains grayscale handwritten digits from 0 to 9, covering 10 categories. It is a standard benchmark for testing whether representations preserve simple but clearly separable visual categories.
Caltech-101 [58 ] contains images from 101 object categories and a background class, covering 102 classes in total. The dataset includes substantial variation in object scale, pose, and background, making it useful for evaluating general object-level discrimination.
SUN397 [116 ] is a large-scale scene recognition dataset covering 397 scene categories. It evaluates whether visual representations can separate scene-level semantics rather than only object-centric categories.
FGVC Aircraft [117 ] focuses on fine-grained aircraft recognition over 100 aircraft variants. The categories are visually similar, so the dataset tests whether representations can capture subtle appearance cues.
Country-211 [11 ] contains geolocation-related visual categories from 211 countries and regions. It evaluates whether representations encode high-level geographic and cultural visual cues.
Stanford Cars [118 ] is a fine-grained vehicle recognition dataset with 196 car categories. It requires distinguishing car models with small differences in shape, texture, and local details.
• Describable Textures (DTD) [59 ] focuses on texture classification over 47 human-interpretable texture categories. It is useful for testing whether representations capture local visual patterns.
• Eurosat [119 ] is a satellite image dataset for land-use and land-cover recognition. It contains 10 diverse remote-sensing categories such as forests, rivers, and residential areas.
• Facial Emotion Recognition 2013 (FER2013) [120 ] is a facial expression recognition dataset with 7 expression categories. It evaluates whether representations can separate subtle facial states under variations in identity, illumination, and pose.
• Oxford Flowers 102 (Flowers-102) [121 ] is a fine-grained flower classification dataset with 102 categories. The dataset emphasizes color, shape, and petal-level differences.
• Food-101 [122 ] contains food images from 101 categories. It tests recognition under large intra-class appearance variation caused by cooking style, viewpoint, and presentation.
• GTSRB [123 ] is a traffic sign recognition dataset with 43 traffic sign categories. It evaluates the ability to distinguish symbolic visual categories with strong shape and color regularities.
• PatchCamelyon (PCam) [124 ] is a medical image dataset for patch-level histopathology recognition. It provides a binary classification setting with 2 classes and subtle texture and tissue patterns.
• Oxford-IIIT Pets (Pets) [125 ] contains 37 cat and dog breed categories. It evaluates fine-grained animal recognition with variations in pose, fur pattern, and background.
• Rendered SST2 [11 ] converts binary sentiment classification examples into rendered text images, covering 2 classes. It tests whether visual representations can handle text-like visual inputs.
• RESISC45 [126 ] is a remote sensing scene classification dataset with 45 categories. It evaluates scene-level discrimination from aerial imagery.
• STL10 [127 ] is an object recognition dataset with 10 categories and higher-resolution images than CIFAR. It provides another compact benchmark for semantic clustering.
• Pascal VOC 2007 (VOC2007) [128 ] is a multi-object recognition dataset containing 20 object categories in natural scenes. It tests representation quality under cluttered backgrounds and multi-label visual content.
• ImageNet-1K [57 ] is a large-scale recognition benchmark with 1000 categories. It provides a comprehensive test of general visual discrimination.
• ImageNet-V2 [129 ] is a distribution-shifted ImageNet test set collected with the same 1000-class label space. It evaluates whether separability is preserved under natural dataset shift.
• ImageNet-Adversarial (ImageNet-Adv.) [130 ] contains naturally occurring images from 200 ImageNet categories that are difficult for standard ImageNet classifiers. It tests robustness to challenging visual examples.
• ImageNet-Rendition (ImageNet-Ren.) [131 ] contains artistic renditions from 200 ImageNet categories. It evaluates whether representations capture semantic identity beyond natural photo statistics.
• ImageNet-OOD [130 ] evaluates out-of-distribution recognition over 200 ImageNet-related categories. It measures robustness when image statistics deviate from the original ImageNet distribution.
• ObjectNet [132 ] is an object recognition dataset with 313 object categories, designed to stress test models under changes in viewpoint, rotation, and background. It evaluates robust semantic discrimination in realistic settings.
Evaluation metrics. Following [141 ] , we adopt 3 widely-used metrics to evaluate clustering performance.
• Normalized Mutual Information (NMI) measures the mutual dependence between predicted clusters and ground-truth labels, normalized to [0, 1]. Higher values indicate better alignment between the clustering structure and the true class partition.
• Accuracy (ACC) evaluates the best-matched assignment between predicted clusters and ground-truth classes. It computes the fraction of correctly assigned samples after optimal label permutation using the Hungarian algorithm.
• Adjusted Rand Index (ARI) quantifies the similarity between two partitions by counting pairwise agreements, adjusted for random chance. An ARI of 0 corresponds to random clustering, while higher values indicate more consistent partitioning.
Zero-shot Classification
Datasets. We use the same 26 datasets introduced in the zero-shot clustering protocol. This shared dataset suite allows us to evaluate the same visual representations from two complementary perspectives. Clustering measures the intrinsic separability of image features without using text labels during grouping, while classification measures whether image features can be aligned with textual class names in a zero-shot manner.
Evaluation metrics. We adopt 3 metrics to evaluate zero-shot classification performance.
• Top-1 Accuracy (ACC@1) measures the percentage of samples whose top-ranked predicted class matches the ground-truth label. It directly reflects whether the strongest image-text match is correct.
• Top-5 Accuracy (ACC@5) measures whether the ground-truth label appears among the top 5 predicted classes. This metric is useful for large-category datasets where several classes may be visually or semantically close. For datasets with fewer than 5 classes, ACC@5 is not applicable.
• Mean Recall (mRecall) computes recall for each class and then averages over classes. It reduces the dominance of frequent classes and better reflects class-balanced recognition performance.
Zero-shot Retrieval
Datasets. We evaluate cross-modal retrieval on Flickr30k and COCO, which are widely used image-text retrieval benchmarks.
Flickr30k [60 ] contains images collected from Flickr, each paired with multiple human-written captions. It evaluates whether visual representations can align images with natural language descriptions in open-domain scenes.
COCO [61 ] contains complex natural images with rich object interactions and multiple captions per image. It provides a more diverse retrieval benchmark with dense object and scene content.
Evaluation metrics. We evaluate both image-to-text and text-to-image retrieval.
Image-to-Text Recall@5 (Image@5) measures whether at least one correct caption of a query image appears in the top 5 retrieved texts. Higher values indicate stronger image-to-language alignment.
Text-to-Image Recall@5 (Text@5) measures whether the correct image of a query caption appears in the top 5 retrieved images. Higher values indicate stronger language-to-image alignment.
Evaluation Protocol for MLLMs
Following [24 ] , [25 ] , [53 ] , we evaluate MLLMs using two complementary groups of benchmarks: Vision-Centric Benchmarks , which focus on fine-grained visual perception and scene understanding, and Conventional MLLM Benchmarks , which measure robustness, hallucination resistance, and multimodal reasoning. Together, they provide a comprehensive view of an MLLM’s visual competence, reliability, and general multimodal capability.
Vision-Centric Benchmarks. We evaluate on 4 datasets:
MMVP-MLLM [53 ] evaluates the model’s fine-grained visual perception by testing its ability to identify structured visual patterns. We report classification accuracy (ACC).
NaturalBench [63 ] examines multimodal reasoning across natural images using four metrics: overall accuracy (Acc), question-type accuracy (Q-Acc), instance-level accuracy (I-Acc), and group-level accuracy (G-Acc).
CV-Bench 2D [62 ] measures perception quality on 2D dense prediction tasks. We report the accuracy (ACC) of ADE20K [142 ] and COCO [143 ] .
CV-Bench 3D [62 ] assesses 3D spatial understanding through structured 3D reasoning queries. We report overall accuracy (ACC).
Conventional MLLM Benchmarks. We evaluate on 3 datasets:
POPE [133 ] evaluates object hallucination robustness under three settings, including random (rand), popularity-based (pop), and adversarial (adv). Higher scores indicate stronger resistance to hallucination.
SciQA-IMG [134 ] assesses scientific visual question answering involving diagrams, plots, and structured visual cues. We report accuracy (ACC).
Hallusion [135 ] measures multimodal robustness by testing whether the model avoids visually induced false inferences. We report the average accuracy across all query types.
Appendix J: Additional Experimental Results
Figure 12 and Figure 13 provide expanded qualitative results on the MMVP-VLM benchmark. Figure 12 visualizes the results obtained by AUCGR, while Figure 13 visualizes the results obtained by pAUCGR. These examples cover visual questions that depend on local and fine-grained evidence, such as object state, color, spatial relation, structure, and viewpoint. The original CLIP often gives incorrect answers when the required evidence is subtle or appears only in a local region. After applying AUCGR, the enhanced representation better preserves such detail-sensitive cues and produces more accurate predictions. pAUCGR further improves the qualitative behavior by reducing misleading negative comparisons during training. The two groups of visualizations therefore provide complementary evidence that the proposed optimization strengthens P-Ability without changing the visual encoder architecture.
Figure 12. Expanded qualitative results of AUCGR on the MMVP-VLM benchmark. The predictions from the original CLIP and the AUCGR-enhanced CLIP are indicated by red and green arrows, respectively. AUCGR helps the model capture fine-grained visual details that are missed by the original CLIP.
Figure 13. Expanded qualitative results of pAUCG R on the MMVP-VLM benchmark. The predictions from the original CLIP and the pAUCG R-enhanced CLIP are indicated by red and green arrows, respectively. pAUCG R further improves fine-grained visual perception by reducing misleading negative comparisons during optimization.
In this section, we provide a more comprehensive version of the quantitative evaluation for D-Ability, covering zero-shot clustering, zero-shot classification, and zero-shot retrieval.
Zero-shot Clustering
The main paper reports zero-shot clustering results in terms of ACC in Table 2 . To provide a more complete evaluation, Table 5 and Table 6 further report ARI and NMI on the same 26 datasets. ARI evaluates whether the predicted clusters agree with the ground-truth semantic partitions after correcting for chance. NMI measures the information overlap between predicted clusters and ground-truth labels. Across these complementary metrics, AUCGR and pAUCGR generally improve the DIVA and GenHancer baselines. The consistent gains indicate that the improvement is not tied to a single clustering metric. Instead, the comparison-based optimization encourages more coherent semantic grouping in the visual feature space.
Zero-shot Classification
The main paper summarizes average zero-shot classification performance in Figure 8 . The complete per-dataset results are reported in Table 7 , Table 8 , and Table 9 . Table 7 reports ACC@1, which measures whether the top-ranked class-name prompt matches the ground-truth label. Table 8 reports ACC@5, which measures whether the correct class appears among the top 5 predictions. Table 9 reports mean recall, which gives a class-balanced view of recognition performance. These metrics evaluate different aspects of prompt-based recognition.
The results show that AUCGR restores the discriminative information weakened by plain generative reconstruction supervision. pAUCGR further improves the results by reducing ambiguous negative comparisons.
Zero-shot Retrieval
The retrieval results evaluate whether the enhanced visual representations remain effective when the model ranks images and natural language descriptions. Figure 7 reports the results on OpenAI CLIP ViT-L@224. Figure 14 and Figure 15 provide the corresponding results on MetaCLIP ViT-L@224 and SigLIP ViT-SO@224. Each figure includes image-to-text retrieval and text-to-image retrieval on Flickr30k and COCO. The improvements over the original DIVA and GenHancer baselines show that the proposed optimization benefits not only image-only discrimination, but also retrieval-based discrimination under cross-modal candidate ranking. This further supports the effectiveness of AUCGR and pAUCGR as plug-and-play optimization modules for strengthening D-Ability.
Expanded Qualitative Results on MLLM Performance
Figure 16 shows several qualitative examples on the vision-centric benchmarks from MLLMs. The results demonstrate that our enhanced CLIP can be seamlessly integrated into MLLMs to improve their visual capabilities.
Figure 16. Expanded qualitative results on the MLLM benchmarks. Ours denotes the model that replaces the original CLIP vision encoder with the pAUCGR-enhanced CLIP encoder.
Expanded Ablation Studies
Results on Noisy Negative Filtering
We conduct two complementary studies to determine whether pAUCGR suppresses semantically noisy negatives rather than merely discarding difficult but valid comparisons.
Semantic validity of the removed negatives. Following the sampling protocol used in Figure 4 , we independently sample 40 mini-batches with a batch size of 128. Our training procedure is self-supervised and uses only images. For this diagnostic, we use the similarity between the corresponding texts as a proxy ground truth for semantic relationships. These text-based judgments are used only to evaluate the image-based filter and are not used during training. Among all negatives removed by the image-based filter, 98.75% are also identified as noisy by the text-similarity criterion. Thus, only 1.25% of the removed pairs are not flagged as semantically compatible, indicating that the filter accurately targets likely false negatives while rarely eliminating valid hard negatives. Moreover, filtering produces a 12.73% relative reduction in the false-negative rate compared with the unfiltered negative set. The strong agreement between the image- and text-based criteria confirms that image similarity is an effective surrogate for identifying noisy negatives and substantially reduces false-negative contamination.
Ablation Study on Filtering Strategies. We further disentangle the effects of AUC optimization and negative filtering in Table 10 . For the naive joint objective, we compare retaining all negatives (Naive), random negative removal (Naive+Random), and image-similarity-based removal without the AUC objective (Naive+Partial). We conduct the analogous comparison for AUC-based optimization using AUCGR, AUCGR+Random, and pAUCGR. Random removal consistently fails to improve the two abilities and often degrades both, showing that the gain does not result from simply reducing the number of negative comparisons. Similarity-based filtering alone is also insufficient: Naive+Partial does not yield a balanced improvement over Naive. In contrast, pAUCGR achieves the best P-Ability and D-Ability for both DIVA and GenHancer. These results demonstrate that reliable negative selection and AUC-oriented comparison are complementary: the former suppresses noisy supervision, while the latter converts the retained comparisons into an effective ranking objective.
Table 10. Comparison of noisy-negative filtering strategies. All experiments use OpenAI CLIP ViT-L@224. P-Ability is measured by ACC on MMVP-VLM, and D-Ability is measured by the average ACC over 26 zero-shot clustering benchmarks. Naive+Random and AUCGR+Random apply random negative removal to the corresponding objectives; Naive+Partial applies image-similarity-based filtering without AUC-oriented optimization; and pAUCGR combines the same filtering principle with AUCGR. The best result for each is highlighted in bold .
Results on Different Surrogate Losses
We evaluate the four surrogate losses introduced in § 6.4 using OpenAI CLIP ViT-L@224 with DIVA and GenHancer. As reported in Table 11 , different surrogate choices generally improve the learned representations over the corresponding generative reconstruction baselines. In particular, the square and hinge surrogates consistently improve both P-Ability and D-Ability on the two backbones, while the logistic surrogate also yields clear gains in most cases. These results demonstrate that the effectiveness of the proposed framework is not tied to a particular surrogate formulation. The exponential surrogate is comparatively less effective: although it improves D-Ability, it reduces P-Ability on both backbones. A likely reason is that its rapidly increasing weighting assigns disproportionately large gradients to a small number of high-loss hard pairs, leading to overly aggressive updates and a less balanced preservation of the two abilities.
Table 11. Performance with different surrogate losses. We instantiate pAUCGR with the exponential, square, hinge, and logistic pairwise surrogates using OpenAI CLIP ViT-L@224. P-Ability is measured by ACC on MMVP-VLM, and D-Ability is measured by the average ACC over 26 zero-shot clustering benchmarks.
We further study the margin λ \lambda λ of the hinge and square losses and the temperature τ \tau τ of the logistic loss. As shown in Figure 17 , the proposed framework achieves consistent improvements over the corresponding reconstruction baselines across a broad range of reasonable hyperparameter values. Performance deteriorates mainly when the hyperparameters are set excessively small or large, because such extreme settings either provide insufficient separation between positive and negative pairs or place excessive emphasis on their differences. Within the practical range, P-Ability remains relatively stable and the improvements in D-Ability are consistently observed, despite its larger variation. These results show that the framework does not rely on a narrowly tuned optimum and is robust to the choice of surrogate-loss hyperparameters.
Figure 17. Sensitivity to surrogate loss hyperparameters. We evaluate the trade-off between P-Ability and D-Ability of pAUCGR on DIVA and GenHancer using OpenAI CLIP ViT-L@224. Each point corresponds to a different margin λ \lambda λ for the hinge or square surrogate, or a different temperature τ \tau τ for the logistic surrogate. The dashed lines indicate the original performance of DIVA or GenHancer, respectively.
Results on Training Stability
To assess the stability of pAUCGR under training randomness, we repeat the experiments with five random seeds on both DIVA and GenHancer using OpenAI CLIP ViT-L@224. All other experimental settings remain unchanged. Figure 18 presents the results as boxplots to show the variation across runs. Averaged over the five seeds, pAUCGR improves both P-Ability and D-Ability over the corresponding original DIVA or GenHancer baseline. In particular, the D-Ability results remain above the corresponding baseline across all five runs on both reconstruction methods. P-Ability exhibits greater variation across seeds, but its average improvement is maintained on both methods. These results support the stability of pAUCGR and show that its gains in balancing the two abilities are reproducible across different random seeds.
Figure 18. Training stability across five random seeds. We evaluate pAUCGR on DIVA and GenHancer using OpenAI CLIP ViT-L@224. P-Ability is measured by ACC on MMVP-VLM, and D-Ability is measured by the average ACC over 26 zero-shot clustering benchmarks. The boxplots summarize the pAUCGR results across five runs. Boxes indicate the interquartile ranges, and the horizontal lines inside them indicate the medians. Circular markers denote the corresponding original DIVA or GenHancer results. The mean P-Ability and D-Ability of pAUCGR over the five seeds both exceed the corresponding baseline values.
Computational Costs
Table 12 reports the computational costs of our method. Since AUCGR and pAUCGR are optimization methods, they do not introduce extra trainable modules. Therefore, the number of trainable parameters remains the same as that of the original generative reconstruction backbone. The additional training cost mainly comes from pairwise comparison and negative filtering. Relative to DIVA, AUCGR increases the training time by 25.70%–54.79% and GPU memory by 35.53%–74.34%. pAUCGR increases the training time by 58.88%–77.11% and GPU memory by 39.50%–85.38%. Nevertheless, the overall computation remains within a practical range for the NVIDIA A100 80GB GPU.
Table 12. Training costs of our method on DIVA. Time is measured in hours (h), and GPU memory is measured in gigabytes (GB). The percentages in parentheses denote the relative increases over DIVA with the same backbone and cost metric.
For a batch size B B B , the pairwise loss constructs a B × B B \times B B × B score matrix. Its per-batch comparison and intermediate-storage complexities therefore scale as O ( B 2 ) O(B^2) O ( B 2 ) . As shown in Figure 19 , training time and GPU memory increase accordingly with the batch size. A batch size of 16 provides a favorable balance between performance and resource consumption. We therefore use B = 16 B = 16 B = 16 in all experiments. Moreover, pairwise comparison and negative filtering are used only to compute the training objective. They do not change the encoder architecture or inference path and thus introduce no additional inference cost. Overall, our method incurs moderate and manageable training-only overhead while keeping the model size and inference cost unchanged.
Figure 19. Computational costs under different batch sizes. Training time and GPU memory are measured for our method on DIVA with OpenAI CLIP ViT-L@224. The pairwise objective constructs a B × B B \times B B × B score matrix, so its per-batch computation and intermediate storage scale as O ( B 2 ) O(B^2) O ( B 2 ) . We use B = 16 B = 16 B = 16 in all experiments because it provides a favorable balance between performance and resource consumption.
Appendix K: Relationship Between the Truncation Ratio α \alpha α and Threshold γ ( α ) \gamma(\alpha) γ ( α ) in Expectation
Our goal is to apply the same similarity criterion across mini-batches when removing potentially noisy negatives. Fixing α \alpha α in each batch controls the removed fraction but allows the threshold to vary. Fixing γ 0 \gamma_0 γ 0 instead keeps the threshold unchanged and allows the removed fraction to vary. We establish their correspondence below for a fixed negative-score distribution.
Let n ∼ N − n \sim\mathcal{N}_{-} n ∼ N − be drawn from the negative-sample distribution and define its score as Q = q i ( n ) Q = q_i(n) Q = q i ( n ) . The cumulative distribution function of Q Q Q is
F − ( s ) = P ( Q ≤ s ) . (265) F_{-}(s) = \mathbb{P}(Q \le s). \tag*{(265)} F − ( s ) = P ( Q ≤ s ) . (265)
For a fixed truncation ratio α \alpha α , the corresponding quantile threshold is
γ ( α ) = F − − 1 ( 1 − α ) . (266) \gamma(\alpha) = F_{-}^{-1}(1-\alpha). \tag*{(266)} γ ( α ) = F − − 1 ( 1 − α ) . (266)
For a fixed threshold γ 0 \gamma_0 γ 0 , the induced truncation ratio is
α ( γ 0 ) = P ( Q > γ 0 ) = 1 − F − ( γ 0 ) . (267) \alpha(\gamma_0) = \mathbb{P}(Q > \gamma_0) = 1 - F_{-}(\gamma_0). \tag*{(267)} α ( γ 0 ) = P ( Q > γ 0 ) = 1 − F − ( γ 0 ) . (267)
Assume that F − F_{-} F − is continuous and strictly increasing over the relevant score range. Substituting Eq. (267) into the quantile definition gives
γ ( α ( γ 0 ) ) = F − − 1 ( F − ( γ 0 ) ) = γ 0 . (268) \gamma(\alpha(\gamma_0)) = F_{-}^{-1}(F_{-}(\gamma_0)) = \gamma_0. \tag*{(268)} γ ( α ( γ 0 )) = F − − 1 ( F − ( γ 0 )) = γ 0 . (268)
Therefore, the two parameterizations retain the same negative set when α = α ( γ 0 ) \alpha= \alpha(\gamma_0) α = α ( γ 0 ) :
{ n : q i ( n ) ≤ γ 0 } = N i ( α ( γ 0 ) ) . (269) \{n : q_i(n) \le\gamma_0\} = \mathcal{N}_{i}^{(\alpha(\gamma_0))}. \tag*{(269)} { n : q i ( n ) ≤ γ 0 } = N i ( α ( γ 0 )) . (269)
Moreover,
α ( γ 0 ) = P ( Q > γ 0 ) = FPR ( γ 0 ) . (270) \alpha(\gamma_0) = \mathbb{P}(Q > \gamma_0) = \operatorname{FPR}(\gamma_0). \tag*{(270)} α ( γ 0 ) = P ( Q > γ 0 ) = FPR ( γ 0 ) . (270)
Thus, the fixed threshold γ 0 \gamma_0 γ 0 corresponds to the partial AUC interval [ α ( γ 0 ) , 1 ] [\alpha(\gamma_0), 1] [ α ( γ 0 ) , 1 ] . For a fixed score distribution and independent batch sampling, the average empirical truncation ratio converges to α ( γ 0 ) \alpha(\gamma_0) α ( γ 0 ) . This population-level correspondence does not require every batch to remove the same fraction of negatives.