WorksComputer Science
Beyond Raw Similarity: Prior-Aware Fusion for Out-of-Distribution Detection
Abstract
Out-of-distribution (OOD) detection in vision-language models presents unique challenges due to the reliance on cross-modal similarity. Even when embeddings are well separated for in-distribution (ID) data, similarity scores can remain overconfident for OOD inputs. A common remedy is to adapt the model on ID data, but adaptation discards the pretrained geometry that made the similarity meaningful. We propose multi-modal logit fusion (MM-Log), which keeps both by pairing a gradient-free zero-shot prior with a task-adapted branch in each modality and fuses their four cross-modal similarity channels into a single OOD score. Because the zero-shot priors are never adapted, the cross-channels act as a consistency check between the pretrained and adapted geometries: an OOD input that fools the adapted branch through a shortcut must additionally agree with frozen CLIP to retain a high score. We formalize this as a margin bound showing that the fused score is provably penalized by disagreement between the zero-shot and adapted branches, a signal to which adapted-only scores are structurally blind. MM-Log requires no auxiliary OOD data, negative prompts, or test-time adaptation, and trains on few-shot ID data alone. Across Far-OOD, Near-OOD, and Hard-OOD benchmarks, MM-Log consistently outperforms strong CLIP-based baselines.
Introduction
Modern machine learning models are inherently vulnerable to inputs that fall outside their training distribution. When confronted with unseen classes, novel concepts, or shifted data conditions, these models often extrapolate with unjustified confidence, leading to erroneous and potentially harmful decisions [20]. To address this, out-of-distribution (OOD) detection seeks to determine whether a given test input originates from the model’s in-distribution (ID) data or represents an unknown case. Traditionally, OOD detection has been studied primarily in single-modality settings [5, 9], leveraging confidence scores or feature statistics to detect anomalous inputs. However, recent advancements in vision–language models (VLMs) such as CLIP [17] have substantially expanded this scope as they offer strong zero-shot transfer learning, naturally operate on both image and text inputs, and produce a shared embedding space where cross-modal similarity can be used for downstream tasks.
Early CLIP-based OOD detection methods such as Maximum Concept Matching (MCM) [12], and Global-local MCM [15] primarily rely on the model’s inherent cross-modal similarity geometries. While conceptually appealing, their detection performance is limited in practice, indicating that the raw similarity structure of pretrained VLMs is often insufficient for reliable OOD separation. Other subsequent methods move beyond the pretrained model’s natural similarity geometry. One of the common techniques is to introduce auxiliary OOD signals [3, 11], synthetic negatives [6, 24], or surrogate distributions [14] (Yu et al. 2024) to calibrate scores and shape the decision boundary. Another popular strategy is negative prompting [21], where the detector queries the model with carefully designed “non-class” or “none-of-the-above” prompts to elicit low similarity for OOD inputs. These techniques can be effective, but also complicate the pipeline as they introduce prompt sensitivity, increase dependency on curated or generated negatives, and require additional hyperparameters and heuristics for processing surrogate data which may not transfer across datasets, modalities, or deployment conditions.
In this paper, we argue that a key failure mode in multi-modal OOD detection is not merely the absence of “OOD knowledge”, but rather the miscalibration and brittleness of cross-modal similarity under distribution shift. While strong image–text alignment enables VLMs to learn discriminative embeddings across diverse ID categories, their similarity scores remain susceptible to overconfidence induced by spurious correlations, shortcut cues, and modality-specific artifacts. [22] This indicates that strong cross-modal agreement alone, while sufficient for classification, is not enough for reliable OOD detection. We therefore hypothesize that explicitly strengthening and constraining semantic agreement between modalities for ID data can make OOD detection substantially easier. This is because OOD samples tend to violate such an agreement in subtle ways, through novel concepts, mismatched descriptions, or shifts that perturb one modality while leaving the other relatively unchanged. Consequently, by enforcing tighter cross-modal consistency, even small departures from the ID manifold can be amplified and detected as OOD. With this motivation, we propose MM-Log (Multi-Modal Logit Fusion), a simple cross-modal OOD detection approach built around two concepts: (i) enforce tighter alignment between ID image and text embeddings in the shared representation space, and (ii) preserve the rich semantic knowledge of pretrained VLMs by using zero-shot representations as a regularizing prior. Specifically, instead of relying on auxiliary OOD signals or handcrafted negative prompts, we adapt the VLM solely with ID data, learning an embedding geometry that pulls matched image–text pairs closer together and pushes mismatched pairs farther apart. This creates an ID manifold with higher cross-modal coherence and a similarity landscape that is less permissive to OOD inputs.
Our key contributions are summarized as follows:
We propose MM-Log, which pairs a gradient-free zero-shot prior with a task-adapted branch in each modality and fuses their four cosine channels into a single OOD score, trained on few-shot ID data alone without auxiliary outliers, negative prompts, or test-time adaptation.
We show that the fused score admits an exact factorization whose magnitude is governed by the angular disagreement between the pretrained and adapted geometries, and prove a margin bound (Prop. 0.1) quantifying how this disagreement penalizes the score; adapted-only scores are structurally blind to this signal.
We validate MM-Log across Far-, Near-, and Hard-OOD benchmarks, where it outperforms strong CLIP-based baselines, and isolate the contribution of each cross-modal interaction through a channel-wise ablation.
Related Works
Single-modal OOD Detection Methods OOD detection has been extensively studied in single-modal settings, particularly in image classification through an image encoder such as ResNet. Early approaches on single-modal OOD detection rely on the predictive confidence of discriminative classifiers, such as maximum softmax probability (MSP) [4] and its extensions, as well as energy-based scores that reinterpret classifier logits as unnormalized densities [9]. Other lines of work leverage feature-space statistics, including distance-based methods such as the Mahalanobis distance [8], or model data likelihoods estimated by generative models [18]. Beyond post-hoc scoring functions, several methods explicitly incorporate OOD information as auxiliary outliers during training [5, 19, 13, 2]. In contrast to these single-modal approaches, our work targets OOD detection in large VLMs, where the presence of a shared cross-modal embedding space introduces both new challenges and opportunities that are not captured by traditional unimodal formulations.
Vision Language Model-based OOD Detection
With the emergence of large-scale VLMs, recent work has started to investigate OOD detection in multimodal and cross-modal settings by exploiting the shared embedding space learned by models such as CLIP. Early VLM-based OOD detection approaches primarily operate directly on CLIP-style image–text similarity scores. Representative methods, including MCM [12] and its extension GL-MCM [16], use the maximum similarity between an image and a set of class prompts as an OOD confidence score, based on the intuition that ID samples should exhibit strong alignment with at least one semantic class. Another prominent line of research focuses on prompt engineering and prompt learning, where carefully designed or learned prompts such as negative or “none-of-the-above” descriptions [6, 24, 21] are introduced to induce lower similarity responses for OOD inputs. A complementary direction incorporates auxiliary OOD data or synthetically generated outliers [11] (Miyai et al. 2023; Yu et al. 2024) to explicitly shape the decision boundary and improve score calibration. Despite their empirical success, these methods often rely heavily on the availability and quality of proxy OOD information, as well as the careful construction of negative labels, which limits their robustness in realistic and ubiquitous deployment settings. Moreover, they tend to be sensitive to prompt design choices and frequently introduce additional hyperparameters that may not generalize well across datasets or modalities. In contrast, our approach adopts a minimal and principled setting: we rely exclusively on ID data while avoiding auxiliary OOD supervision and negative prompting. Rather than introducing additional signals, we directly improve the cross-modal similarity structure by tightening image–text alignment for ID pairs.
Preliminaries
OOD Detection Setup: Let and represent a set of input images and their labels, respectively, drawn from the ID dataset with classes. Subsequently, for a test dataset drawn from OOD dataset , it holds that . Then, formally the objective of OOD detection during testing remain two-folds: (i) for , classify it into correct ID class; (ii) for , reject it as OOD data. Formally, the second objective can be achieved through a simple scoring mechanism:
where, is an OOD detector, denotes the confidence score of , and is a threshold computed based on some heuristics.
Revisiting CLIP and Prompt Learning: With prompt learning, a simple natural-language text prompt is transformed into a set of learnable prompt vectors that is optimized to encode task-specific semantics, enabling tighter and more effective alignment with the model’s visual representations. For each class , consider a manually designed textual prompt as
Following CoOp [25], prompt learning reformulates the textual prompt as a sequence of learnable context tokens, , where denotes the word embedding of class , and are learnable context vectors shared across all classes. We distinguish between two versions of this class prompt. The initialization prompt, , is constructed from the handcrafted template ”a photo of a ” and kept frozen throughout training, preserving the pretrained VLM’s zero-shot semantics. The adapted prompt, , shares the same initialization but is optimized via gradient descent, allowing it to capture task-specific adaptations from the ID training data. Then, given a pretrained CLIP model, the image and text embeddings are obtained as , , where and denote the image encoder and text encoder, respectively, and the embeddings are -normalized. The cross-modal similarity is then computed via cosine similarity. For downstream tasks such as image recognition, the probability of assigning image to class is then given by
where is a softmax temperature parameter, and represents cosine similarity between the image features and its corresponding textual prompt vector for class .
MM-Log: Multi-Modal Logit Fusion
We propose MM-Log (Multi-Modal Logit Fusion), that jointly leverages (i) intrinsic zero-shot CLIP features and (ii) task-adapted features from the few-shot ID samples of the downstream dataset. MM-Log integrates these complementary signals through a simple bilinear fusion mechanism of the logits derived from their respective multi-modal similarity function. The outcome is a model that substantially enhances both ID accuracy and OOD separability without requiring external OOD data. The schematic representation of MM-Log’s working procedure is shown in Figure 1. The MM-Log model consists of two components: (i) Cross-Modal Zero-Shot and Adapted Embeddings, and (ii) Bilinear Similarity Fusion, both of which are discussed in the following sections.
Figure 1. Schematic overview of MM-Log. The embedding outcomes are zero-shot feature (, ) and an adapted feature (, ). On the image side these come from a frozen encoder and an encoder adapter . On the text side a single shared prompt learner are evaluated at two prompt states: the template-initialized (cached, no gradient) gives , and the trained prompt gives . The four zero-shot/adapted cosine pairings form the channels , , , , summed into ; the fused score is supervised by cross-entropy during training, and its free energy serves as the OOD score at inference.
Cross-Modal Zero-Shot and Adapted Embeddings: We begin by extracting embeddings from the vanilla CLIP and simultaneously adapting to the downstream dataset on-the-fly during training. In particular, we aim in generating zero-shot and adapted embeddings for each modality. The source of MM-Log’s zero-shot text prior, lies in when the prompt is evaluated, not in a separate encoder path. Both text branches share the same prompt learner and the same text encoder; what differs is the state of the context vectors and whether gradients flow. The zero-shot text prior is the outcome of text encoder applied to the prompt at its initialization state , whose context vectors still hold the token embeddings of the template “a photo of a ”; this feature is computed once before optimization and cached under a no-gradient context, so it never updates and carries no effective trainable parameters. It therefore equals the zero-shot CLIP text embedding of the template. The adapted branch applies the same encoder to the same prompt learner as its context vectors are optimized during training.
Concretely, for vision modality the vanilla CLIP encoder is weight-frozen and replicated to obtain frozen and adapted branches. Using these, we extract the zero-shot image prior and the adapted image features . We refer to the pair as the zero-shot priors and as the adapted features. This makes the text zero-shot prior symmetric in role to the image zero-shot prior: just as is the untouched CLIP image embedding, is the untouched CLIP text embedding, both fixed throughout training. We note that all of these embeddings undergo normalization across the dimension . The zero-shot priors preserve CLIP’s robust pretrained geometry, while the adapted features capture task-specific semantics of the downstream dataset. Importantly, because and are captured with no-gradient evaluation and never recomputed, they receive no gradients and remain fixed throughout training, providing a stable semantic anchor for the fusion described next.
Bilinear Similarity Fusion: A key design choice in MM-Log is how to reconcile the stability of the zero-shot priors and the adaptability of the adapted features. Keeping the VLM entirely zero-shot maintains robust, general semantics but may fail to correct dataset-specific misalignment that harms OOD detection. Conversely, aggressive adapting can overfit to ID idiosyncrasies and degrade the pretrained geometry that makes VLMs broadly useful. We address this tension through bilinear similarity fusion: we compute cross-modal similarity using both the zero-shot features (capturing pretrained priors) and the adapted features (capturing ID-specific alignment), and then fuse these similarity signals into a single OOD score. This fusion operates directly at the logit level where detection decisions are made, making the method simple, modular, and compatible with standard VLM inference pipelines. Intuitively, zero-shot similarities provide a conservative semantic anchor, while adapted similarities sharpen ID agreement; their fusion yields a detector that is simultaneously well-calibrated and ID-sensitive.
With Bilinear similarity fusion, we form a fused representation for each modality by combining its zero-shot and adapted branches, and . Computing the cross-modal similarity of these fused representations expands into four interaction terms:
Consequently the MM-Log objective is a single cross-entropy applied to the fused logits:
We deliberately supervise the fused score rather than each channel with its own cross-entropy: the softmax of a sum is not the product of per-channel softmaxes, and applying the loss at the fusion point is exactly what couples the channels. Gradients reach the trainable parameters only through the channels that contain an adapted component (, , ), while the parameter-free enters the softmax as a fixed, input-dependent offset that shifts the per-class competition without itself receiving gradients. Following CLIP (Radford et al. [17]), all cosine similarities are scaled by the model’s learned logit scale before the softmax; we absorb this scaling into in our notation.
Why the cross-terms aid OOD detection? It is worth making explicit why the mixed terms, rather than alone, are what stabilizes detection under distribution shift. Because all embeddings are -normalized, each channel is a cosine agreement in and the fused score is their sum,
(the learned logit scale multiplies this by a positive constant and preserves all orderings). Consider first an ID sample of class . Training drives and into tight alignment, so is large; and because the loss of Eq. (3) is applied to the fused score, gradients through pull the adapted image features toward the zero-shot text embeddings while gradients through pull the adapted prompts toward the zero-shot image features. The zero-shot priors thus act as fixed targets that regularize adaptation, so the cross-terms remain large as well and all four channels agree. Now consider an OOD sample. The adapted branch, optimized purely on ID data, can still assign a spuriously high to an input that exploits a shortcut or dataset-specific artifact — precisely the overconfidence failure mode of adapted-only detectors. The zero-shot priors, however, were never adapted to that shortcut, so the cross-terms and compare an adapted embedding in one modality against a non-adapted embedding in the other. An input that fools the adapted branch must therefore additionally agree with the zero-shot CLIP geometry to keep the summed score high. The full MM-Log procedure is provided in Algorithm 1.
Theoretical Justification: By bilinearity, the four separately computed channels sum to a single inner product,
where are used only for analysis; the model never forms them. Since their summands are unit vectors, their lengths measure exactly how far adaptation has moved from the pretrained priors: and , where and .
Proposition 0.1. Let denote the larger zero-shot–adapted drift. Then
If, in addition, every cross-modal pair satisfies for all , then .
Proof. Upper bound. Assume without loss of generality , so . Then
Lower bound. Each channel equals . If every cross-modal pair satisfies , monotonicity of on gives for each , and summing the four channels yields .
Proposition 0.1 formalizes the argument above. An input whose adaptation has drifted by in either modality is capped at and cannot recover a high fused score however confidently the adapted branch aligns it, since that confidence enters only through ; an input whose four features mutually agree to within retains at least . Whenever the first bound falls below the second, guaranteeing a separating threshold. The cross-terms are what make this possible: involves neither nor , so it is invariant to the zero-shot geometry and cannot distinguish two inputs that differ only in how far they have drifted from it. The bound establishes sensitivity rather than detection: whether OOD inputs actually exhibit larger than ID inputs is an empirical property of the data and the trained model. Table 4 and 5 confirm the predicted ordering, with variants omitting the cross-terms (e.g., ) most vulnerable and the progressive addition of , , and yielding the most reliable separation.
| Fusion Type | iNaturalist AUROC ↑ | iNaturalist FPR95 ↓ | SUN AUROC ↑ | SUN FPR95 ↓ | Places AUROC ↑ | Places FPR95 ↓ | Textures AUROC ↑ | Textures FPR95 ↓ | Average AUROC ↑ | Average FPR95 ↓ |
| 94.16±1.12 | 31.95±0.45 | 92.55±3.44 | 37.22±2.34 | 90.10±3.12 | 42.98±1.18 | 85.83±2.73 | 58.35±0.98 | 90.66±2.60 | 42.63±1.23 | |
| 91.35±2.33 | 41.68±1.27 | 90.79±2.70 | 43.79±3.42 | 87.90±1.83 | 50.47±0.87 | 88.76±2.12 | 47.59±3.40 | 89.70±2.24 | 45.88±2.23 | |
| 93.40±3.47 | 34.39±2.83 | 91.50±1.77 | 43.36±2.16 | 88.86±0.93 | 49.97±1.06 | 86.23±3.81 | 58.71±2.77 | 90.00±2.49 | 46.61±2.20 | |
| 95.75±0.45 | 17.79±1.47 | 95.85±1.89 | 16.38±3.60 | 94.10±0.79 | 23.10±5.04 | 92.72±0.82 | 28.23±2.22 | 94.60±0.98 | 21.37±2.95 | |
| 94.82±3.67 | 25.82±4.48 | 90.39±3.73 | 47.01±4.82 | 89.32±2.79 | 46.88±4.76 | 87.79±2.86 | 35.67±4.73 | 89.85±6.16 | 41.77±4.23 | |
| 96.39±0.77 | 14.06±3.98 | 96.01±0.77 | 16.01±3.30 | 94.18±0.98 | 21.29±3.32 | 93.66±1.24 | 28.10±3.24 | 95.06±0.94 | 19.86±3.46 | |
| 97.49±0.39 | 10.53±2.09 | 96.13±0.59 | 15.12±3.85 | 94.60±0.75 | 20.89±3.79 | 94.68±1.59 | 27.61±6.29 | 95.72±0.83 | 18.53±4.00 | |
| Fusion Type | OOD Dataset | |||||||||
| OpenImage-O | NINCO | Imagenet-O | Average | |||||||
| AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | |||
| 92.342.08 | 28.653.44 | 73.961.03 | 79.122.73 | 80.360.87 | 68.541.91 | 82.221.32 | 58.772.69 | |||
| 91.132.37 | 43.713.90 | 71.692.13 | 80.370.98 | 79.821.78 | 71.251.34 | 80.882.09 | 65.112.07 | |||
| 90.702.31 | 46.901.91 | 72.063.49 | 81.610.68 | 79.292.70 | 77.551.03 | 80.682.83 | 68.691.20 | |||
| 95.490.52 | 19.332.19 | 81.531.50 | 62.322.68 | 86.871.58 | 50.154.49 | 87.961.20 | 43.933.12 | |||
| 92.060.85 | 38.953.77 | 74.921.82 | 79.233.81 | 79.084.95 | 74.453.38 | 82.022.54 | 64.213.65 | |||
| 95.620.11 | 18.910.13 | 82.990.73 | 59.160.62 | 88.020.51 | 46.780.69 | 88.870.45 | 41.610.48 | |||
| 96.070.68 | 17.663.33 | 83.390.50 | 59.382.19 | 87.772.30 | 46.936.24 | 89.071.16 | 41.323.92 |
Table 4. Comparison of OOD detection results across different fusion types for Near-OOD datasets. Subscripts denote the two branches contributing to each similarity term (0: zero-shot, : adapted; e.g., ). Rows highlighted in color represent performance without logit fusion. Best and second-best results are in bold and underline respectively.
Inference: During inference, we score a test input with the free energy of the fused logits [9]:
with energy temperature , and plug into the detector of Eq. (1). Energy aggregates evidence over all classes rather than committing to the single best match, which suits the fused logit space: an ID input accumulates consistently high agreement across the four channels for its correct class, whereas an OOD input that partially fools one channel rarely sustains high fused logits across the class set, so its logsumexp remains low.
Algorithm 1: MM-Log: finetuning and OOD inference
Require: pretrained CLIP encoders , ; ID data over classes ; softmax temperature ; energy temperature ; threshold
Zero-shot priors (frozen, computed once)
frozen copy of ; initialization prompt (frozen)
for ; cache zero-shot text features; all features -normalized
Training on ID data
initialize adapted encoder and adapted prompt
repeat
for minibatch do
for
update via ; frozen
end for
until converged
OOD detection for a test input
compute for as in line 8
return ID if else OOD
Experimental setup
Following the existing studies in VLM based OOD detection (Miao et al. 2025; Ming et al. 2022; Lu et al. 2025), we utilize CLIP ViT-B/16 as the backbone model and adapt it for 50 epochs on a downstream image dataset. The adapted prompt uses context vectors initialized from the token embeddings of the template “a photo of a ”, following the CoOp initialization scheme [26]. The hyperparameters used for training are as follows: batch size is 160, learning rate is , SGD optimizer (momentum 0.9, and weight decay ) with a cosine scheduler. Cosine similarities are scaled by CLIP’s learned logit scale, and the softmax and energy temperature are set to , and respectively. All the experiments including training and testing were conducted on a NVIDIA A100 GPU cluster. For statistical significance of the experiment, we train the models for 3 trials with different random seeds and report the averaged test metrics over these trials. We provide comprehensive experimental setup including details about algorithm, datasets, baselines in the Supplementary material.
Datasets: We train MM-Log on the ImageNet-1k recognition benchmark as the ID dataset, and evaluate OOD detection across three regimes of increasing difficulty. Far-OOD uses iNaturalist, SUN, Places, and Textures, which are semantically and visually distant from ImageNet classes (fine-grained natural species, scene imagery, and surface textures). Near-OOD uses OpenImage-O, NINCO, and ImageNet-O, whose classes are visually or semantically close to the ID categories while sharing no label overlap, making them substantially harder to reject. Hard-OOD pairs the ImageNet-10, ImageNet-20, and ImageNet-100 subsets against one another (e.g., ImageNet-10 vs. ImageNet-20), so that ID and OOD distributions overlap in semantic content and the detector must separate closely related concepts. Baselines and Metrics: We compare MM-Log’s performance against popular baselines, some of them being state-of-the-art in OOD detection. These include; Coop [26], LoCoop [14], CLIPN [21], IDLike [1], NegLabel [6], LAPT [24], SCT (Yu et al. 2024), APT [11], CMA [7], and FA (Lu et al. 2025). To evaluate OOD detection performance, we use Area Under the Receiving Operator Characteristics Curve (AUROC), and False Positive Rate @ 95% of True Positive Rate (FPR95). Likewise, we also use ID accuracy to compare the classification performance of the model.
Results
Performance on Far-OOD dataset: Table 1 reports Far-OOD results with ImageNet-1k as ID. MM-Log achieves the best average AUROC and FPR95; although it is not top on iNaturalist, it outperforms all baselines on SUN, Places, and Textures. The second-best method, LAPT [24], relies on label-driven retrieval of textual descriptions from web-scale data, and its effectiveness hinges on the coverage of the retrieved labels; MM-Log requires no such external retrieval. Against the most design-similar baseline, FA (Lu et al. 2025), which does not model zero-shot priors in the image branch, MM-Log improves average AUROC by more than 2% and nearly halves FPR95.
| Method | Venues | iNaturalist AUROC ↑ | iNaturalist FPR95 ↓ | SUN AUROC ↑ | SUN FPR95 ↓ | Places AUROC ↑ | Places FPR95 ↓ | Textures AUROC ↑ | Textures FPR95 ↓ | Average AUROC ↑ | Average FPR95 ↓ |
| Coop | IJCV’22 | 91.47±2.12 | 41.14±9.39 | 91.68±0.83 | 39.06±4.17 | 89.16±1.28 | 45.38±4.88 | 87.82±1.12 | 51.37±3.09 | 90.03±0.32 | 44.24±1.39 |
| LoCoop | NeurIPS’23 | 92.78±0.93 | 36.64±4.87 | 93.50±0.92 | 31.86±3.79 | 90.70±0.97 | 38.81±3.37 | 89.43±0.68 | 48.11±2.31 | 91.59±0.58 | 38.85±2.67 |
| CLIPN† | ICCV’23 | 95.27±0.00 | 23.94±0.00 | 93.93±0.00 | 26.17±0.00 | 90.93±0.00 | 40.83±0.00 | 92.28±0.00 | 33.45±0.00 | 93.10±0.00 | 31.10±0.00 |
| IDLike | CVPR’24 | 96.68±0.31 | 17.73±1.91 | 89.53±0.58 | 48.17±1.39 | 88.27±2.27 | 50.43±6.58 | 93.25±2.16 | 29.12±7.64 | 91.93±1.18 | 36.36±3.86 |
| NegLabel † | ICLR’24 | 99.49±0.00 | 1.91±0.00 | 95.49±0.00 | 20.53±0.00 | 91.64±0.00 | 35.59±0.00 | 90.22±0.00 | 43.56±0.00 | 94.21±0.00 | 25.40±0.00 |
| LAPT† | ECCV’24 | 99.63±0.00 | 1.16±0.00 | 96.01±0.00 | 19.12±0.00 | 92.01±0.00 | 33.01±0.00 | 91.06±0.00 | 40.32±0.00 | 94.68±0.00 | 23.40±0.00 |
| SCT | NeurIPS’24 | 91.77±2.40 | 41.93±12.17 | 93.76±0.39 | 30.39±2.17 | 90.78±0.34 | 38.73±2.54 | 88.96±1.16 | 46.78±3.51 | 91.32±0.90 | 39.46±3.39 |
| APT† | ICCV’25 | 97.79±0.00 | 9.70±0.00 | 95.52±0.00 | 20.12±0.00 | 92.84±0.00 | 28.54±0.00 | 88.43±0.00 | 45.78±0.00 | 93.64±0.00 | 26.03±0.00 |
| CMA† | CVPR’25 | 95.67±0.00 | 22.81±0.00 | 91.64±0.00 | 40.60±0.00 | 88.41±0.00 | 48.71±0.00 | 89.68±0.00 | 45.23±0.00 | 91.35±0.00 | 39.33±0.00 |
| FA | ICCV’25 | 94.72±0.27 | 25.50±2.72 | 92.40±0.95 | 36.24±3.15 | 91.99±0.68 | 35.38±2.75 | 93.95±0.19 | 28.34±0.87 | 93.25±0.32 | 31.37±1.07 |
| MM-Log | - | 97.49±0.39 | 10.53±2.09 | 96.13±0.59 | 15.12±3.85 | 94.60±0.75 | 20.89±3.79 | 94.68±1.59 | 27.61±6.29 | 95.72±0.83 | 18.53±4.00 |
Table 1. Comparison of OOD Detection metrics trained on ImageNet-1k benchmark as ID and tested on Far-OOD datasets. Results of methods represented with † are directly reported form their corresponding paper.
Performance on Near-OOD dataset: In Table 2, MM-Log achieves the best AUROC and FPR95 across all Near-OOD datasets, improving over the strongest baseline, CMA, by a substantial average margin. We attribute these gains to MM-Log’s expanded similarity formulation, which jointly models frozen and learnable components of both image and text representations.
| Method | Venues | OpenImage-O AUROC ↑ | OpenImage-O FPR95 ↓ | NINCO AUROC ↑ | NINCO FPR95 ↓ | ImageNet-O AUROC ↑ | ImageNet-O FPR95 ↓ | Average AUROC ↑ | Average FPR95 ↓ |
| COOP | IJCV’22 | 91.99±0.22 | 37.95±0.59 | 73.00±0.59 | 78.00±0.23 | 81.34±0.83 | 70.14±0.86 | 82.11±0.48 | 62.03±0.38 |
| LoCoop | NeurIPS’23 | 92.22±0.25 | 32.23±1.00 | 75.35±0.88 | 72.04±0.81 | 78.44±0.94 | 67.35±1.19 | 81.99±0.58 | 57.21±0.96 |
| IDLike | CVPR’24 | 87.89±0.45 | 54.43±2.35 | 69.32±0.4 | 78.93±1.79 | 67.45±0.86 | 84.08±1.45 | 74.89±0.28 | 72.48±0.89 |
| SCT | NeurIPS’24 | 92.04±0.33 | 37.28±1.16 | 71.09±1.19 | 78.51±0.86 | 81.29±0.15 | 71.45±0.54 | 81.48±0.52 | 62.41±0.73 |
| CMA | CVPR’25 | 94.32±0.00 | 28.85±0.00 | 82.18±0.00 | 65.64±0.00 | 83.39±0.00 | 63.55±0.00 | 86.64±0.00 | 52.68±0.00 |
| FA | ICCV’25 | 91.84±0.45 | 37.93±1.65 | 77.77±0.94 | 70.44±1.24 | 82.71±0.30 | 63.84±1.36 | 84.08±0.47 | 57.01±1.34 |
| MM-Log | - | 96.07±0.68 | 17.66±3.33 | 83.39±0.50 | 59.38±2.19 | 87.77±2.30 | 46.93±6.24 | 89.27±1.16 | 41.32±3.92 |
Table 2. Comparison of OOD Detection metrics trained on ImageNet-1k benchmark as ID and tested on Near-OOD datasets. The best and second best results are reported in bold and underline respectively.
Performance on Hard-OOD dataset: Table 3 presents results on the challenging ImageNet- variants. MM-Log achieves the best average AUROC and FPR95. NegLabel occasionally surpasses it on individual splits, but by small margins and only through explicitly defined negative labels whose quality may not transfer across datasets; MM-Log needs no such supervision and remains reliable even under heavy semantic overlap.
| Method | ID OOD | ImageNet-10 AUROC ↑ | ImageNet-10 FPR95 ↓ | ImageNet-20 AUROC ↑ | ImageNet-20 FPR95 ↓ | ImageNet-10 AUROC ↑ | ImageNet-100 FPR95 ↓ | ImageNet-100 AUROC ↑ | ImageNet-10 FPR95 ↓ | Average AUROC ↑ | Average FPR95 ↓ |
| LoCoop | - | 92.75 | 28.20 | 92.34 | 34.40 | 93.00 | 30.08 | 81.97 | 61.40 | 90.01 | 38.52 |
| SCT | - | 94.33 | 25.10 | 94.95 | 25.00 | 93.90 | 26.64 | 82.60 | 57.80 | 91.44 | 33.63 |
| NegLabel | - | 98.86 | 5.10 | 98.81 | 4.60 | 99.51 | 1.68 | 90.19 | 40.20 | 96.84 | 12.89 |
| IDLike | - | 97.32 | 9.43 | 97.96 | 10.28 | 96.43 | 9.81 | 88.03 | 45.67 | 94.93 | 18.79 |
| FA | - | 98.36 | 4.36 | 98.13 | 12.05 | 97.81 | 6.84 | 94.38 | 33.76 | 97.17 | 14.25 |
| MM-Log | - | 99.38 | 0.87 | 98.17 | 11.79 | 99.06 | 5.46 | 95.37 | 22.55 | 97.99 | 10.16 |
Table 3. Hard OOD on test for ImageNet- variants. The best and second best results are reported in bold and underline respectively.
Impact on ID-Accuracy: Figure 2 compares ID classification accuracy of MM-Log on ImageNet-1k under different few-shot settings (1-shot, 4-shots, and 16-shots) across five methods. Here, we can see that MM-Log consistently achieves the highest accuracy in all shot regimes, with particularly notable improvements in the low-shot setting. These gains can be attributed to MM-Log’s explicit modeling of fine-grained cross-modal interactions between zero-shot and adapted representations. By decomposing both image and text embeddings into zero-shot and adapted components and jointly optimizing all four interaction terms, MM-Log preserves the strong semantic priors of the pretrained model while enabling effective task-specific adaptation with limited labeled data. This expanded similarity formulation leads to improved alignment for ID image–text pairs without overfitting, resulting in better sample efficiency and more stable performance as the number of training examples increases.

Figure 2. Comparison of ID classification accuracy of MM-Log with different baselines trained on ImageNet-1k dataset under different few-shot settings.
Ablation
Influence of Different Fusion Types: Table 4 presents an ablation study analyzing the contribution of different similarity components in MM-Log for OOD detection. Here, we evaluate variants that progressively incorporate cross-modal interactions between zero-shot and adapted representations, namely , , , , and the full model . In addition to those, we also compare the purely zero-shot channel , which uses the zero-shot image embedding and the zero-shot template text embedding with no trainable parameters, against its fully adapted counterpart , which uses only the two adapted branches. We can see that the methods without any similarity fusion (highlighted in color in Table 4) yield poor detection results. This highlights the shortcomings of the zero-shot-only method, and of methods that consider only the semantics of the downstream task.
Among the fusion types, the variant performs the worst, with substantially degraded OOD detection performance. This behavior can be attributed to its reliance on similarities dominated by the adapted-only and zero-shot-only representations within each modality (image, and text). Without a proper cross-modal fusion formulation, the similarity scores can become more sensitive to downstream artifacts or overfit to generic CLIP embeddings from the zero-shot priors, leading to poor calibration and unreliable separation between ID and OOD samples. These findings highlight the importance of balanced similarity fusion to preserve robust semantics and ensure stable cross-modal similarity under distribution shift. Overall, this ablation confirms that effective OOD detection benefits from fusing complementary similarity signals, and that the proportionate combination of zero-shot priors and adapted interactions yields the most re-
liable performance.
Performance Across Multiple-shots: In Figure 3, we further analyze the effect of the number of training shots on both OOD detection performance and ID accuracy using two backbone architectures, ViT-B/16 and ViT-B/32. Here, we trained multiple models under different shot-settings: , and then evaluated the average detection performance across Far-OOD benchmarks. Across both backbones, OOD detection performance as measured by AUROC and FPR95 exhibits a non-monotonic trend with respect to the number of shots: the strongest detection results are achieved in the low and mid-shot regimes, particularly at 1-shot and 16-shot settings. In contrast, increasing the number of shots consistently improves ID classification accuracy, with accuracy rising monotonically as more labeled data becomes available. Together, these results highlight a trade-off between detection and recognition: while additional shots benefit ID accuracy, they do not necessarily enhance OOD detection performance, and the 1-shot and 16-shot settings are most effective for detection. We attribute this non-monotonic behavior to a tension between the two objectives that the zero-shot priors only partially absorb.

Figure 3. Performance vs. number of shots. Results are averaged over three seeds with shaded standard deviation.
Conclusion
We presented MM-Log, a principled method for cross-modal OOD detection that explicitly addresses the miscalibration and brittleness of VLM similarity scores under distribution shift. By jointly leveraging zero-shot pre-trained representations and task-adapted branches learned on downstream data, MM-Log strengthens alignment for ID image–text pairs while discouraging spurious or non-corresponding similarities that could mislead detection. Unlike prior methods, MM-Log does not require auxiliary OOD data, negative prompts, or web-scale label retrieval, yet it achieves consistently strong performance across Far-OOD, Near-OOD, and Hard-OOD benchmarks. Empirical results demonstrate that fusing zero-shot and adapted similarities yields better-calibrated OOD scores and robust ID–OOD separation. Our findings suggest that focusing on cross-modal alignment and embedding geometry, rather than relying on external OOD supervision, is an effective strategy for scalable and reliable OOD detection in large VLMs.
References
- [1]Bai, Y.; Han, Z.; Cao, B.; Jiang, X.; Hu, Q.; and Zhang, C. 2024. Id-like prompt learning for few-shot out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 17480–17489.arxiv.org/abs/2311.15243
- [2]Chen, J.; Li, Y.; Wu, X.; Liang, Y.; and Jha, S. 2021. Atom: Robustifying out-of-distribution detection using outlier mining. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 430–445. Springer.
- [3]Esmaeilpour, S.; Liu, B.; Robertson, E.; and Shu, L. 2022. Zero-shot out-of-distribution detection based on the pre-trained model clip. In Proceedings of the AAAI conference on artificial intelligence, volume 36, 6568–6576.
- [4]Hendrycks, D.; and Gimpel, K. 2016. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In International Conference on Learning Representations.
- [5]Hendrycks, D.; Mazeika, M.; and Dietterich, T. 2018. Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606.
- [6]Jiang, X.; Liu, F.; Fang, Z.; Chen, H.; Liu, T.; Zheng, F.; and Han, B. 2024. Negative label guided ood detection with pretrained vision-language models. arXiv preprint arXiv:2403.20078.arxiv.org/abs/2403.20078
- [7]Kim, J.; and Hwang, S. 2025. Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations. In Proceedings of the Computer Vision and Pattern Recognition Conference, 29979–29988.
- [8]Lee, K.; Lee, K.; Lee, H.; and Shin, J. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31.
- [9]Liu, W.; Wang, X.; Owens, J.; and Li, Y. 2020. Energy-based out-of-distribution detection. Advances in neural information processing systems, 33: 21464–21475.
- [11]Miao, W.; Pang, G.; Wang, Z.; Zheng, J.; and Bai, X. 2025. Auxiliary Prompt Tuning of Vision-Language Models for Few-Shot Out-of-Distribution Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4776–4785.DOI
- [12]Ming, Y.; Cai, Z.; Gu, J.; Sun, Y.; Li, W.; and Li, Y. 2022. Delving into out-of-distribution detection with vision-language representations. Advances in neural information processing systems, 35: 35087–35102.arxiv.org/abs/2211.13445
- [13]Ming, Y.; Fan, Y.; and Li, Y. 2022. Poem: Out-of-distribution detection with posterior sampling. In International conference on machine learning, 15650–15665. PMLR.arxiv.org/abs/2206.13687
- [14]Miyai, A.; Yu, Q.; Irie, G.; and Aizawa, K. 2023a. Locoop: Few-shot out-of-distribution detection via prompt learning. Advances in Neural Information Processing Systems, 36: 76298–76310.arxiv.org/abs/2306.01293
- [15]Miyai, A.; Yu, Q.; Irie, G.; and Aizawa, K. 2023b. Zero-shot in-distribution detection in multi-object settings using vision-language foundation models. arXiv preprint arXiv:2304.04521.
- [16]Miyai, A.; Yu, Q.; Irie, G.; and Aizawa, K. 2025. Gl-mcm: Global and local maximum concept matching for zero-shot out-of-distribution detection. International Journal of Computer Vision, 133(6): 3586–3596.arxiv.org/abs/2304.04521
- [17]Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748–8763. PmLR.arxiv.org/abs/2103.00020
- [18]Serrà, J.; Álvarez, D.; Gómez, V.; Slizovskaia, O.; Núñez, J. F.; and Luque, J. 2019. Input complexity and out-of-distribution detection with likelihood-based generative models. arXiv preprint arXiv:1909.11480.arxiv.org/abs/1909.11480
- [19]Tamang, L.; Boudjenek, M. R.; Dazeley, R.; and Aryal, S. 2024. Margin-bounded confidence scores for out-of-distribution detection. In 2024 IEEE International Conference on Data Mining (ICDM), 1–10. IEEE.arxiv.org/abs/2410.07185
- [20]Tamang, L.; Boudjenek, M. R.; Dazeley, R.; and Aryal, S. 2025. Handling Out-of-Distribution Data: A Survey. IEEE Transactions on Knowledge and Data Engineering.arxiv.org/abs/2507.21160
- [21]Wang, H.; Li, Y.; Yao, H.; and Li, X. 2023. Clipn for zero-shot ood detection: Teaching clip to say no. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1802–1812.arxiv.org/abs/2308.12213
- [22]Yang, Y.; Lee, C. P.; Feng, S.; Zhao, D.; Wen, B.; Liu, A. Z.; Tsvetkov, Y.; and Howe, B. 2025. Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations? arXiv preprint arXiv:2506.18322.arxiv.org/abs/2506.18322
- [24]Zhang, Y.; Zhu, W.; He, C.; and Zhang, L. 2024. Lapt: Label-driven automated prompt tuning for ood detection with vision-language models. In European conference on computer vision, 271–288. Springer.arxiv.org/abs/2407.08966
- [25]Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022a. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 16816–16825.arxiv.org/abs/2203.05557
- [26]Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022b. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9): 2337–2348.DOI
