Zero-Shot Domain Generalization via Domain Prompt Learning

Introduction and Motivation

Early machine learning models assume that training data and test data are independent and identical distributions [42, 10], resulting in weak robust to significant distribution shifts between training and test distribution, e.g., style, backgrounds or day-to-night shifts. To address this problem, domain generalization (DG) was introduced [6, 61], assuming that a model is trained on data from one or multiple source domains and generalizes well to unseen target domains or styles. As shown in Figure 1 (a), given source domains containing images of dogs in sketch, cartoon, and painting styles, DG aims to learn a model that can recognize dogs in natural images or photographs. Enhancing the generalization ability of machine learning models is important in both industry and academic fields, DG has received much attention recently [53].

Standard DG methods aim to learn domain-invariant features or prompts from the given source domain data to tackle domain shift in the visual modality, as shown in Figure 1 (c). Early DG methods learn domain-invariant features to improve model generalization to target domains using various training strategies, such as data augmentation [9]; Nam et al [37]; [57, 37, 38], cross-domain feature alignment methods [20, 33 b], decomposing domain-specific and domain-invariant knowledge [40, 9, 34], meta-learning [4, 16, 56], and ensemble learning [7, 58]. Although these methods achieve significant progress in small datasets, they cannot well generalize to large-scale datasets, e.g., different variants of ImageNet [14].

Address(es) of author(s) should be given Recently, large vision-language models (VLMs), such as CLIP Radford et al [43] and ALIGN Jia et al [26], have been trained on large-scale image–text data and have demonstrated strong domain generalization and zero-shot learning capabilities on downstream tasks. Accordingly, VLM reforms the DG formulation and obtains new state-of-the-art generalization performance Baliah et al [5]; Addepalli et al [2]. There are two categories of methods to bring VLM into DG, including prompt learning and fine-tuning VLM. For example, Zhou et al [60] introduces CoOp to explore the concept of prompt learning for automating prompt engineering via fine-tuning a few prompt tokens. Subsequently, he extends CoOp by further learning a lightweight neural network to generate an input-conditional token for each image and improve model generalization Zhou et al [59]. In contrast, Jia et al [27] proposed visual prompt tuning for fine-tuning VLM. Cha et al [8] re-formulated the DG objective using mutual information with the Oracle model to any possible domain. Bai et al [3] re-framed the prompt learning framework from a generative perspective and propose a simple yet efficient method for DG.

We can find that the above methods access source domain data (i.e., visual data in some domains shared same classes in the target domain) for model optimization. However, these methods face three major limitations in realistic scenarios: i) it is unclear which source domains are ideal and how many source domains are sufficient for model learning since unseen domains are arbitrary in DG; ii) it is costly and infeasible to collect and annotate large-scale source domain data for training; iii) Lots of classes cannot be collected in advance, especially for the new or scarce classes in the real world. Naturally, it raises a new task, i.e., zero-shot domain generalization (ZSDG), which recognizes unseen classes in unseen domains without source domain data, as shown in Figure 1 (b).

The potential solution for ZSDG is directly making use of the zero-shot learning capacity of VLMs (e.g., CLIP) without accessing visual data of unseen classes in source data for training. Thus, the rest key task is how to discover the domain knowledge to improve the domain generalization of VLMs. This raises the challenge of distribution shift in text-visual modalities in the unseen domain (as shown in Figure 1 (d)), which is more challenging than the distribution shift in the visual modality of standard DG.

To tackle this challenge, our intuitive insight is to simultaneously discover target domain knowledge both in text and visual modalities under the test-time adaptation formulation Sun et al [49]; Shu et al [47]; Feng et al [19]. In the text modality, we simulate diverse distribution shifts through prompts that synthesize diverse domain knowledge. In the visual modality, we can explicitly discover the target domain knowledge via prompt by making use of test visual information with self-supervised learning during testing. More importantly, the domain knowledge learned in text modality can be further refined in the visual modality jointly.

In light of the above observations, we propose a novel Domain Prompt learning (dubbed DoP) for zero-shot domain generalization. DoP consists of two key modules to learn the domain prompt, including text domain learning (TDL) and visual domain learning (VDL). In the first stage, TDL simulates various distribution shifts via a prompt in the text space by synthesizing diverse domains knowledge, i.e., styles token generator using inter-style diversity and intra-style consistency loss and for optimization, and selects the most relevant style token ($S^*$) from the generated style tokens based on the test image. In the second stage, VDL refines the selected style token $S^*$ using the test image and domain prompt (i.e., “A $S^*$ style of a [class]”) via self-supervised learning. Unlike PromptStyler Cho et al [13], which uses all synthesized style features to construct a fixed domain-generalizable classifier, DoP treats the learned style tokens as candidate domain prompts and dynamically selects and visually refines the target-relevant token for each test image. Finally, DoP takes the refined domain prompt to predict the classes of test images using CLIP prediction Radford et al [43]. Extensive experiments demonstrate the effectiveness of our method.

The main contributions are summarized as follows:

• We introduce domain prompt learning, dubbed as DoP, which learns the domain knowledge both in text and visual spaces to tackle the domain shift in text-visual modalities of ZSDG.

• We deploy TDL to discover the domain knowledge via a prompt in the text space using inter-style diversity loss and intra-style consistency loss. Meanwhile, we design VDL to further refine the target-domain-relevant style token generated by TDL via self-supervised learning during testing. Thus, DoP can implicitly and explicitly learn the domain knowledge with the text and visual supervision.

• Extensive experimental results on four widely used DG benchmarks (i.e., PACS, VLCS, OfficeHome, and DomainNet) and ImageNet, demonstrate that our DoP learns the domain knowledge for effective ZSDG, thus leading to competitive performance over the current state-of-the-art methods.

Related Work

Classical Domain Generalization. Existing classical DG methods aim to learn domain-invariant feature representations to improve the generation of the model on the target domain with different distributions from the training data of the source domain. They focus on various training strategies on source domains, such as data augmentation, cross-domain feature alignment, decomposing domain-specific and domain-invariant knowledge, meta-learning, and ensemble learning. Data augmentation methods Chattopadhyay et al [9]; Nam et al [37]; Zhou et al [57]; Nam et al [37]; Nuriel et al [38] aim to increase the diversity of the training data and thereby enhance model generalization. For example, Zhou Zhou et al [62, 63] focuses on mixing the feature statistics of two random instances in a single forward pass during training. Cross-domain feature alignment methods Ganin et al [20]; Li et al [33] align the distributions among different domains by adversarial training. Decomposing domain-specific and domain-invariant knowledge methods Piratla et al [40]; Chattopadhyay et al [9]; Lv et al [34] disentangle the domain-specific and domain-invariant features and discard the domain-specific features for presenting the transferable features. Meta-learning methods Balaji et al [4]; Dou et al [16]; Zhang et al [56] leverage past knowledge to improve the learning of current tasks. For instance, MLDG Li et al [31] synthesizes meta-train and meta-test domains to simulate domain shift. Ensemble learning methods Cha et al [7]; Zhou et al [58] reduce the risk of overfitting by combining models learned from different domains or sampled from a series of episodes.

These methods rely on utilizing the data of source domains for learning domain-invariant features. However, it is unclear which source domains are ideal and how many source domains are sufficient for model learning, since unseen domains are arbitrary in DG. Furthermore, it is costly and sometimes even infeasible to collect and annotate large-scale source domain data for training. Most importantly, some classes are impossible to collect in advance, e.g., the new and scarce data. Thus, it is necessary to develop new domain generalization methods, which enable the model to generalize to unseen classes in unseen domains.

VLM-Based Domain Generalization. The large vision-language (VLM) is trained on great amounts of image-text pairs (e.g., CLIP was trained on 400M pairs) and has been applied to various downstream tasks, demonstrating their potential for remarkable domain generalization performance and zero-shot learning capabilities. Accordingly, VLM brings new insight for DG Baliah et al [5]; Cho et al [13]; Addepalli et al [2], but the generation capacities are still hampered by the challenges of distribution shift Chen et al [12]; Bai et al [3] or the cross-dataset bias Chen et al [10, 11]. Fine-tuning VLMs is a promising approach for further improving their generalization on DG tasks. Hu et al [23] proposed a parameter-efficient group with orthogonal regularization to preserve the generalization ability of the pre-trained VLM and learn more diverse knowledge for DG. Shu et al [48] introduced CLIPood to adapt CLIP models to DG via exploiting the semantic relations between classes from the text modality with the beta moving averaging optimization strategy. Another effective fine-tuning strategy is prompt learning, which only fine-tunes a few tokens to improve the generalization of VLM Zhou et al [60]; Jia et al [27]. Different from multimodal domain generalization methods Dong et al [15]; Fan et al [17]; Huang et al [24] that learn domain-robust representations from multiple observed modalities, DoP focuses on mitigating the cross-modal distribution mismatch between textual prompts and visual representations in unseen domains without using source-domain images.

Most existing VLM-based DG methods require source-domain images for optimization, while PromptStyler Cho et al [13] learns diverse pseudo-style words without source data and uses the synthesized text features to train a fixed classifier. In contrast, DoP further performs test-image-conditioned style token selection and visual refinement for cross-modal domain prompt learning. Thus, learning to discover the domain knowledge to improve the domain generalization capacity of existing VLM-based zero-shot learning methods is a good solution. Motivated by the test-time adaptation methods Shu et al. [47]; Feng et al. [19], we design domain prompt learning to explicitly and implicitly learn target domain knowledge via prompt for zero-shot domain generalization.

labels are available, whereas test-time adaptation describes how the domain prompt is refined using an unlabeled test instance. During VDL, only the selected style token is optimized, while the pretrained CLIP visual and text encoders remain frozen.

The Proposed Method

Overview. The framework of our domain prompt learning (DoP) is presented in Figure 2. Essentially, DoP implicitly and explicitly learns the domain knowledge of the target domain both in text and visual spaces to tackle the domain shift in text-visual modalities of ZSDG. DoP first takes text domain learning (TDL) to learn domain knowledge from the text space via inter-style diversity loss and intra-style consistency loss to generate a group of tokens of styles $\{S_1,S_2,\cdots,S_K\}$, which is further selected one style token relevant to the target domain (e.g., $S^*$) based on the test data via token selection. Subsequently, DoP employs visual domain learning (VDL) to further refine $S^*$ using test data of the target domain by self-supervised learning. Finally, the domain prompt $P^d$ (i.e., “A $S^*$ style of a [class]”) learned by VDL is used for class prediction following the standard CLIP. DoP addresses zero-shot domain generalization under a test-time prompt adaptation protocol. Specifically, ZSDG defines the considered data setting, in which no downstream source-domain images or target

Text Domain Learning

Considering there are no source domain data that can be accessed, we can simulate various distribution shifts and generate the styles via a prompt in the text space. Targeting this goal, we first construct the text samples via prompts analogously to the visual data with both style and content information, i.e., “A $S_k$ style photo of a [class]” (denoted as $P_k^{style}\circ P_m$). Where $\{S_k\}_{k=1}^{K}$ are the style tokens initialized from randomly sampled style words, and [class] is the class name in the dataset. To help the model disentangle the style information from the text samples, we further construct the style prompt “A $S_k$ style photo ” (denoted as $P_k^{style}$). Assuming we initiate $K$ style tokens and there are $M$ classes in the dataset, we can formulate $K \times M$ text samples and $K$ style prompts. We take the frozen CLIP text encoder (i.e., $E^T$) to extract their feature representations.

The first task in TDL is to learn tokens with various style concepts in the text space with two properties: i) different style features should exhibit low pairwise correlation to reduce redundancy and improve the coverage of potential target domains; ii) text samples associated with the same style concept should form compact clusters and remain distant from samples associated with other styles.

Inter-Style Diversity Loss. To enable TDL to learn diverse domain knowledge that can cover the target domains, we design inter-style diversity loss ($\mathcal{L}_{diversity}$) to optimize the style tokens of domain knowledge. For example, we finetuned the initialized style tokens $\{S_k\}_{k=1}^{K}$ by globally optimizing the styles features $\{E^T(P_k^{style})\}_{k=1}^{K}$ produced by the $K$ style token, encouraging them to be orthogonal to each other. Specifically, the cosine similarity between the $i$-th and $j$-th style features is formulated as:

Based on the pairwise style similarity, $\mathcal{L}_{diversity}$ is formulated as:

where $i \ne j$.

We can find that when $\mathcal{L}_{diversity}$ is minimized to be zero, it satisfies the orthogonality between all style features. Here, orthogonality serves as a data-independent surrogate for reducing style redundancy rather than an assumption about real-world domain distributions.

Intra-Style Consistency Loss. Due to $\mathcal{L}_{diversity}$ only based on $K$ style prompts for learning, it is not enough to make style tokens with rich domain knowledge. To learn more reliable domain knowledge via prompts, we further take the text samples to discover domain knowledge. Specifically, we enforce the text samples with the same style token to be close to their corresponding style features and far away from the other style features using intra-style consistency loss ($\mathcal{L}_{consistency}$) for optimization. Thus, the contents of text samples are compacted and the style information is highlighted. As shown in Figure 6, the text samples with the same style token are clustered together. That is, each text sample with $k$-th style token should have the highest similarity (e.g., cosine similarity score) with its corresponding style features and low similarity with other style features. For $t$-th class sample with $i$-th style (e.g., $P_i^{style} \circ P_t^{content}$), its cosine similarity score $z_{tij}$ with the $j$-th style features formulated as:

Based on the similarity, $\mathcal{L}_{consistency}$ for the $t$-th class with a contrastive loss:

Accordingly, $\mathcal{L}_{consistency}$ encourages each text sample feature to be close to its corresponding style feature by suppressing content information, thereby enabling the style tokens to capture the shared style characteristics across classes. Different from PromptStyler Cho et al. [13], which sequentially learns style words with a class-centric content-consistency objective, our TDL jointly optimizes all $K$ style tokens using the inter-style diversity and intra-style consistency objectives while keeping the pretrained CLIP text encoder frozen.

Style Token Selection. After learning the style tokens with various domain knowledge (i.e., $\{S_k\}_{k=1}^{K}$), we further select the most relevant style token for each test image. Thus, the selected style token can be further refined based on the test image on visual space. The selection process is shown in Figure 3. We first take the frozen visual encoder of CLIP (i.e., $E^V$) to extract the visual features of the test image. Then, we select the most relevant style token according to the cosine similarity scores between the test image $x$ and all learned style prompts $\{P_k^{style}\}_{k=1}^{K}$:

where

This selection is performed independently for each test image using the frozen CLIP encoders without gradient updates, and only the highest-scoring style token $S^*$ is passed to VDL for subsequent visual refinement.

Visual Domain Learning

Due to the styles token learned by TDL in the text space without the supervision of target data, they may not well capture the target domain knowledge and meet the issue of distribution shift in the visual modality. To further refine the style tokens in visual space, DoP takes VDL to fine-tune the domain prompt $P^d$ (i.e., “A $S^*$ style of a [class]”) with the supervision of test image via test-time adaptation. Different from test-time prompt tuning (TPT) Shu et al. [47] that fine-tunes multiple randomly initialized tokens for learning context information in the test image, VDL only fine-tunes the style token $S^*$ by discovering the domain knowledge from the test image. Accordingly, DoP can improve the generalization in any unseen domain.

For each test image $x$, VDL first generates $N$ augmented views, denoted by $v_1, v_2, \ldots, v_N$, and uses the visual encoder $E^V$ to extract their features. Then, it takes the selected style token $S^*$ learned by TDL and class names to construct $M$ domain prompts to extract the text features using $E^T$. Subsequently, VDL adopts the entropy to conduct self-supervision and fine-tune $S^*$ based on the gradients:

where

Here, $\tilde{p}_{S^*}(y_i \mid x)$ is the averaged prediction of $x$, $p_{S^*}(y_i \mid v_j)$ is the class prediction probabilities produced by the domain prompt to $i$-th class and the $j$-th patches of $x$.

To reduce the noise from random augmentations, VDL filters out the patches of low confidence (i.e., high-entropy) using a threshold $\gamma$ with a confidence selection percentile $\rho$. Thus the averaged prediction $\tilde{p}_{S^*}(y \mid x)$ can be rewritten as:

where $H(p_j)$ is the self-entropy of the prediction on the $j$-th patches. For each test image, VDL independently refines the selected style token $S^*$ and discards the adapted token after prediction, preventing any information from being accumulated across test samples.

Class Prediction

After refining the style token $S^*$ in VDL, our DoP performs class prediction following the standard CLIP. Differently, DoP takes the domain prompt $P^d$ as text input of $E^T$. Thus, the $i$-th class prediction score of test sample $x$ is formulated as:

Where $\tau$ is the temperature of the softmax function and is set to 100 following CLIP. According to the prediction scores, DoP obtains the predicted class for $x$.

Experiments

Datasets. We evaluate our method on four widely used DG dataset benchmarks (e.g., PACS Li et al. [30], VLCS Fang et al. [18], OfficeHome Venkateswara et al. [51], and DomainNet Peng et al. [39]) and the large-scale dataset ImageNet (Including ImageNet-V2 Recht et al. [45], ImageNet-Sketch Wang et al. [52], ImageNet-A Hendrycks et al. [22] and ImageNet-R Hendrycks et al. [21]). Their statistics are summarized in Table 1. Notably, most methods take the natural images in ImageNet Deng et al. [14] for model training and evaluate them on the other four domains. Since ZS-DG does not require source data for training, we do not take the leave-one-domain-out cross-validation evaluation protocol Peng et al. [39] and take our DoP to test directly for all samples in datasets.

	Domains/Styles	Classes	Images
PACS	4	7	9,991
VLCS	4	5	10,729
OfficeHome	4	65	15,588
DomainNet	6	345	586,575
ImageNet	4	1000	98,389

Implementation Details. We use AdamW with a learning rate of $5 \times10^{-3}$ for prompt optimization. TDL jointly optimizes all initialized style tokens for 10 epochs, whereas VDL independently updates only the selected token $S^*$ for each test image for two optimization rounds; the pretrained CLIP encoders remain frozen throughout both stages. We empirically set the number of initialized style tokens $K = 10/2/50/10/10$ for PACS, VLCS, OfficeHome, DomainNet, and ImageNet, respectively. We also experimentally set the confidence selection percentile $\rho$ to $\{0.1, 0.1, 0.2, 0.1, 0.1\}$ for PACS, VLCS, OfficeHome, DomainNet, and ImageNet, respectively.

All experiments are performed on a single NVIDIA H100 graphic card with 80GB memory. We use PyTorch² for the implementation of all experiments.

We compare DoP with domain generalization methods using various network backbones, including classical DG methods (e.g., DANN Ganin et al. [20], RSC Huang et al. [25], SAGM Wang et al. [54], SagNet Nam et al. [37], and Self-Reg Kim et al. [29]), VLM-based methods (e.g., CLIP Radford et al. [43], GVRT Min et al. [36], MIRO Cha et al. [8], CoOp Zhou et al. [60], CoCoOp Zhou et al. [59], TPT Shu et al. [47], VPT Jia et al. [27], MaPLe Khattak et al. [28], ProGrad Zhu et al. [64], CuPL Pratt et al. [41], PEGO Hu et al. [23], VL2V-SD Addepalli et al. [2], SPG Bai et al. [3], ArGue Tian et al. [50], PromptAlign Abdul Samadh et al. (2023), DPE Zhang et al. [55], and PromptStyler Cho et al. [13].

Comparison with Standard DG Methods. Table 2 reports the results of standard DG methods using different network backbones, including ResNet-50 and ViT-B/16, on four widely used DG datasets. Overall, VLM-based methods consistently outperform classical DG methods, demonstrating the strong generalization capability of pretrained VLMs for DG tasks. DoP achieves the best average accuracies of 74.9% and 82.0% with ResNet-50 and ViT-B/16, respectively. More importantly, DoP performs zero-shot domain generalization without requiring any downstream source-domain images, whereas most compared DG methods rely on source data for training. Compared with the PromptStyler Cho et al. [13], DoP improves the average accuracy by 0.2 and 1.2 percentage points with ResNet-50 and ViT-B/16, respectively, demonstrating the effectiveness of dynamically selecting and visually refining target-relevant domain prompts.

Table 2: Domain generalization results (%) of the state-of-the-art DG methods on PACS, VLCS, OfficeHome, and DomainNet, including classical DG methods and VLM-based methods. Methods takes the source data for training marked with ✓, otherwise, marked with ✗.

Method	Source Data	PACS	VLCS	OfficeHome	DomainNet	Average
ResNet-50 with pre-trained weights on ImageNet
DANN Ganin et al. [20]	✓	83.6	78.6	65.9	38.3	66.6
RSC Huang et al. [25]	✓	85.2	77.1	65.5	38.9	66.7
SagNet Nam et al. [37]	✓	86.3	77.8	68.1	40.3	68.1
SelfReg Kim et al. [29]	✓	85.6	77.8	67.9	42.8	68.5
GVRT Min et al. [36]	✓	85.1	79.0	70.1	44.1	69.6
MIRO Cha et al. [8]	✓	85.4	79.0	70.5	44.3	69.8
SAGM Wang et al. [54]	✓	86.6	80.0	70.1	45.0	70.4
ResNet-50 with pre-trained weights from CLIP
CAD Ruan et al. [46]	✓	90.0	81.2	70.5	45.5	71.8
CoOp Zhou et al. [60]	✓	91.3	81.4	73.5	49.7	74.0
CoCoOp Zhou et al. [59]	✓	91.9	81.8	73.4	49.7	74.2
PromptStyler Cho et al. [13]	✗	93.2	82.3	73.6	49.5	74.7
DoP (Ours)	✗	93.7	80.3	74.7	50.7	74.9
ViT-B / 16 with pre-trained weights from CLIP
MIRO Cha et al. [8]	✓	95.6	82.2	82.5	54.0	78.6
CoOp Zhou et al. [60]	✓	96.4	80.8	83.0	59.5	79.9
CoCoOp Zhou et al. [59]	✓	96.7	80.3	83.4	59.4	80.0
VPT Jia et al. [27]	✓	96.9	82.0	83.2	58.5	80.2
MaPLe Khattak et al. [28]	✓	96.5	82.2	83.4	59.5	80.4
VL2V-SD Addepalli et al. [2]	✓	94.9	81.9	85.7	59.4	80.5
PEGO Hu et al. [23]	✓	96.5	83.2	84.2	59.3	80.8
SPG Bai et al. [3]	✓	97.0	82.4	83.6	60.1	80.8
PromptStyler Cho et al. [13]	✗	97.2	82.9	83.6	59.4	80.8
DoP (Ours)	✗	97.3	84.3	84.7	61.4	82.0

We further report the generalization results on the more challenging ImageNet variants in Table 3. DoP achieves a competitive average accuracy of 61.0% compared with strong VLM-based methods, including ProGrad Zhu et al. [64], CuPL Pratt et al. [41], and ArGue Tian et al. [50]. These results further demonstrate that DoP can effectively learn domain knowledge to enhance the generalization capability of pretrained VLMs under diverse distribution shifts.

Method	Source Data	ImageNet-V2	ImageNet-R	ImageNet-S	ImageNet-A	Average
CLIP Radford et al [43]	❌	60.8	74.0	47.8	46.1	57.2
CoOp Zhou et al [60]	✅	64.2	75.2	47.9	49.7	59.3
CoCoOp Zhou et al [59]	✅	64.1	76.2	48.8	50.6	59.9
TPT Shu et al [47]	❌	64.3	73.9	46.4	53.6	59.5
ProGrad Zhu et al [64]	✅	64.7	74.5	47.9	49.3	59.1
CuPL Pratt et al [41]	❌	63.3	77.1	48.8	50.8	60.0
ArGue Tian et al [50]	✅	64.6	76.6	48.9	50.9	60.3
DoP (Ours)	❌	62.6	78.4	48.9	54.0	61.0

Comparison with Zero-Shot Learning Methods. Table 4 reports the zero-shot domain generalization results of different methods using ResNet-50, ViT-B/16, and ViT-L/14. DoP achieves the best average accuracies of 74.9% and 82.0% with ResNet-50 and ViT-B/16, respectively. Compared with the recent test-time generalization methods, DoP outperforms DPE Zhang et al. [55] by 1.2 percentage points with ResNet-50. With ViT-B/16, DoP exceeds PromptAlign Abdul Samadh et al. [1] and DPE by 3.1 and 0.9 percentage points, respectively. Although PromptAlign and DPE perform better on VLCS in some settings, DoP achieves the best results on PACS, OfficeHome, and DomainNet, indicating more balanced generalization across different types of domain shift. With ViT-L/14, DoP achieves a competitive average accuracy of 83.3%, compared with 84.1% obtained by TPT Shu et al. [47].

Method	PACS	VLCS	OfficeHome	DomainNet	Average
		ResNet-50 with pre-trained weights from CLIP
ZS-CLIP Radford et al [43]	90.6	76.0	68.6	45.6	70.2
ZS-CLIP(DP) Radford et al [43]	93.5	79.8	70.5	46.8	72.7
TPT Shu et al [47]	91.3	81.7	72.9	49.1	73.8
TPT(DP) Shu et al [47]	93.5	80.2	71.2	49.1	73.5
DPE Zhang et al [55]	91.8	82.0	73.5	47.3	73.7
DoP (Ours)	93.7	80.3	74.7	50.7	74.9
		ViT-B / 16 with pre-trained weights from CLIP
ZS-CLIP Radford et al [43]	96.0	82.8	82.2	57.5	79.6
ZS-CLIP(DP) Radford et al [43]	96.7	70.3	80.7	57.3	76.3
TPT Shu et al [47]	96.5	83.3	83.8	59.6	80.8
TPT(DP) Shu et al [47]	96.6	70.4	81.8	59.4	77.1
PromptAlign Abdul Samadh et al [1]	93.5	85.3	81.1	55.8	78.9
DPE Zhang et al [55]	96.7	84.3	83.9	59.6	81.1
DoP (Ours)	97.3	84.3	84.7	61.4	82.0
		ViT-L / 14 with pre-trained weights from CLIP
ZS-CLIP Radford et al [43]	98.5	82.4	86.9	64.0	83.0
ZS-CLIP(DP) Radford et al [43]	97.7	72.0	87.2	63.0	80.0
TPT Shu et al [47]	98.8	83.8	88.6	65.4	84.1
TPT(DP) Shu et al [47]	98.0	74.5	88.2	65.2	81.5
DPE Zhang et al [55]	98.5	82.0	88.7	66.2	83.9
DoP (Ours)	98.6	78.9	89.2	66.3	83.3

We additionally incorporate explicit domain knowledge into the text prompts of ZS-CLIP and TPT, denoted as DP, using the template A random style of a [class]. Although this direct prompting strategy yields slight improvements with ResNet-50, it generally degrades performance with ViT-B/16 and ViT-L/14, particularly on VLCS.³ This observation suggests that directly inserting unlearned domain descriptions may disturb the pretrained text-visual alignment. In contrast, DoP learns diverse domain knowledge in the text space and further performs target-relevant prompt selection and visual refinement, thereby effectively mitigating the cross-modal distribution shift in ZSDG.

Per-Domain Analysis. Table 5 further reports the per-domain performance on PACS. Compared with PromptStyler Cho et al. [13], DoP consistently improves Art accuracy by 1.0 and 1.2 percentage points using ResNet-50 and ViT-B/16, respectively. With ResNet-50, DoP also improves Cartoon and Sketch by 0.2 and 0.9 percentage points, while maintaining the same Photo accuracy. With ViT-B/16, the differences on Photo and Cartoon are negligible, whereas Sketch accuracy decreases by 0.8 percentage points. Overall, DoP improves the average accuracy by 0.5 and 0.1 percentage points with the two backbones, respectively.

Method	Art	Cartoon	Photo	Sketch	Average
ResNet-50 with pre-trained weights from CLIP
ZS-CLIP Radford et al. [43]	88.9	94.4	99.3	79.8	90.6
TPT Shu et al. [47]	92.5	93.0	99.4	80.4	91.3
PromptStyler Cho et al. [13]	93.7	94.7	99.4	84.9	93.2
DoP (Ours)	94.7	94.9	99.4	85.8	93.7
ViT-B / 16 with pre-trained weights from CLIP
ZS-CLIP Radford et al. [43]	96.8	98.8	99.9	88.3	96.0
TPT Shu et al. [47]	97.8	99.2	99.9	89.1	96.5
PromptStyler Cho et al. [13]	97.6	99.1	99.9	92.3	97.2
DoP (Ours)	98.8	99.0	99.9	91.5	97.3

Ablation Study

To provide further insight into DoP, we conduct ablation studies to evaluate the effects of different model components. Results on four widely used DG datasets are presented in Table 6. If DoP only takes TDL (i.e., DoP w/o VDL) for implicitly discovering domain knowledge, its average performance decreases by 6.2%. When DoP only adopts the VDL (i.e., DoP w/o TDL) to learn domain knowledge for the target domain, its average performance decreases by 4.9%. Because they failed in the distribution shift between text and visual modalities in the target domain. By jointly learning domain knowledge in the text and visual spaces, DoP improves the average accuracy by 5.6% and 4.3% over DoP w/o VDL and DoP w/o TDL, respectively. Considering the domain knowledge learned in TDL without any information supervision in the target domain, we can select the target-domain-relevant style tokens via style token selection to enable

Method	PACS	VLCS	OfficeHome	DomainNet	Average
DoP w/o VDL	93.9	77.3	79.1	52.7	75.8
DoP w/o TDL	96.6	70.4	81.8	59.4	77.1
DoP w/o TDL(STS)	97.2	84.3	83.5	60.7	81.4
DoP (Full)	97.3	84.3	84.7	61.4	82.0

able effective refinements in VDL and improve DoP (i.e., 0.6% average performance gains).

Qualitative Results

t-SNE Visualization of Text Samples in TDL. As shown in Figure 6, we presented the t-SNE visualization Maaten and Hinton [35] of text samples with randomly initialized style tokens and styles tokens learned by TDL of DoP on PACS and OfficeHome datasets. Results show that the initialized style token without obvious style means for text samples, which are clustered together according to the content information (e.g., class). In contrast, our DoP effectively compacts the content information of each text sample using the intra-style consistency loss. Accordingly, all text samples can be clustered by style. Furthermore, we can find that all the learned styles are diverse because the inter-style diverse

Qualitative Results

loss encourages DoP to learn the orthogonal styles. As such, DoP effectively learns diverse styles to provide diverse candidate style anchors for potential target domains in text space and achieve impressive domain generalization over current state-of-the-art methods.

Synthesize Images According to Domain prompts. To qualitatively illustrate the domain knowledge learned by DoP, we use Stable Diffusion (e.g., SD2.1 Ramesh et al [44]) to synthesize images from the learned domain prompts with randomly initialized style tokens (i.e., $\{S_1,S_2,\cdots,S_K\}$) and style tokens learned by DoP (i.e., $S^*$), as shown in Figure 5. Results show that the style tokens learned by DoP can accurately capture the domain knowledge of target domains, while the domain prompt with random initialized style tokens failed. The learned prompts capture representative characteristics of the four PACS domains, including painting, cartoon, photo, and sketch styles. This demonstrates DoP significantly discovers diverse target domain knowledge both in text and visual modalities, which well alleviates the challenge of domain shift in text-visual modalities. To this end, DoP conTable 6: Ablation studies for different components of our DoP on four datasets. “STS” denotes style token selection in TDL. We conduct these experiments using ViT-B/16 with pre-trained weights from CLIP.

sistently achieves impressive performance on various DG datasets for ZSDG tasks.

Hyperparameter Analysis

Number of the Initialized Style Tokens $K$. We study the number of initialized style tokens $K$ in Eq. (2) to determine its effectiveness on our DoP. As can be seen from Figure 7(a)(b), when $K$ is set to too small or too large, the performance of DoP decreases slightly. These results are explained as follows: i) when $K$ is set to too small, i.e., $K$ is smaller than the numbers of target domains, the styles token cannot well cover all possible target domains; ii) when $K$ is set to too large, e.g., $K = 200$ on PACS, the limited text samples may not sufficiently learn too many style tokens. Accordingly, we experimentally set the $K = 10$, $K = 2$, $K = 50$, $K = 10$, and $K = 10$ for PACS, VLCS, OfficeHome, DomainNet, and

ImageNet, respectively. Notably, although VLCS has four domains, its domains are the “photo” style.

Confidence Selection Percentile $\rho$. We analyze the confidence selection percentile $\rho$ in Eq. (7) to determine its effectiveness on DoP. As shown in Figure 7(c)(Figure 7d), the classification accuracy is insensitive to $\rho$. This demonstrates that DoP is robust to the choice of the confidence selection percentile because TDL provides well-initialized style tokens that can be stably refined by VDL. According to the experimental results, we experimentally set the $\rho= 0.1/0.1/0.2/0.1/0.1$ for PACS/VLCS/OfficeHome/DomainNet/ImageNet, respectively.

Computational Efficiency

Table 7 compares DoP with ZS-CLIP Radford et al. [43], PromptStyler Cho et al. [13], and TPT Shu et al. [47] on all 9,991 PACS images using CLIP ViT-B/16. Although VDL introduces additional test-time overhead, DoP requires 170.5 ms per image and reduces the testing time by 63.2% compared with TPT, since it refines only the selected style token $S^*$. Different from TPT which optimizes a large set of prompt parameters at test time, our token-selective update strategy largely cuts down redundant computation. Such efficiency gain enables our method to be more practical for real-world deployment scenarios.

Method	Training Time (s) $\downarrow$	Testing Time (s) $\downarrow$	Per Image (ms) $\downarrow$
ZS-CLIP	–	61.2	6.1
TPT	–	4,624.1	462.8
PromptStyler	125.7	70.7	7.1
DoP (Ours)	4.8	1,703.5	170.5

Conclusion

This paper proposes DoP for ZSDG, a practical DG setting in which models recognize unseen classes in unseen domains without access to source-domain data. The key challenge in ZSDG is the domain shift in text-visual modalities. Leveraging the strong zero-shot capability of pretrained VLMs (e.g., CLIP), DoP learns domain knowledge in both the text and visual modalities to improve cross-domain generalization. Specifically, DoP employs TDL and VDL to learn the domain knowledge both in text and visual modalities via a prompt. The quantitative and qualitative results on four widely used DG benchmarks show that DoP effectively learns domain knowledge to perform impressive zero-shot learning and domain generalization for the ZSDG task. These results establish domain prompt learning as a promising research direction for improving the generalization of machine learning models.

Though the performance is excellent on the simple DG datasets, the results on the challenging datasets (e.g., DomainNet and ImageNet with more domains and classes) have great rest improvements. This limitation may arise because i) the constructed text space contains limited domain knowledge, and ii) a single test image may provide insufficient information for accurately discovering target-domain characteristics. Accordingly, we hope the empirical findings and research insights in this work could pave the way for future research on discovering rich domain knowledge from large-language models (LLM) and efficient adaptation methods for domain prompt learning.

Declarations

Availability of Data and Materials: The experiments in this manuscript use eight publicly available benchmark datasets, i.e., PACS⁴, VLCS⁵, OfficeHome⁶, DomainNet⁷, ImageNet-A⁸, ImageNet-R⁹, ImageNet-S¹⁰, and ImageNet-V2¹¹, which are publicly opened benchmarks.

Competing Interests: The authors declare no competing interests.

Funding: N/A

Authors’ Contributions: Shiming Chen writes this manuscript and conduct experiments. Shuhuang Chen supports the coding. Salman Khan and Fahad Shahbaz Khan are the supervisors for this project and polish manuscript.

Acknowledgements: N/A