betaFeedback

WorksComputer Science

Towards Domain Adaptive Neural Contextual Bandits

Abstract

Contextual bandit algorithms are essential for solving real-world decision making problems. In practice, collecting a contextual bandit's feedback from different domains may involve different costs. For example, measuring drug reaction from mice (as a source domain) and humans (as a target domain). Unfortunately, adapting a contextual bandit algorithm from a source domain to a target domain with distribution shift still remains a major challenge and largely unexplored. In this paper, we introduce the first general domain adaptation method for contextual bandits. Our approach learns a bandit model for the target domain by collecting feedback from the source domain. Our theoretical analysis shows that our algorithm maintains a sub-linear regret bound even adapting across domains. Empirical results show that our approach outperforms the state-of-the-art contextual bandit algorithms on real-world datasets.

Introduction

Contextual bandit (CB) algorithms have shown great promise for naturally handling exploration/exploitation trade-off problems with optimal statistical properties. Notably, LinUCB [20] and its various adaptations [52, 1, 21, 19, 11, 17, 25, 57], have been shown to be able learn the optimal strategy when all data come from the same domain. However, these methods fall short when applied to data from a new domain. For example, a drug reaction prediction model trained by collecting feedback from mice (the source domain) may not work for humans (the target domain).

So, how does one effectively explore a high-cost target domain by only collecting feedback from a low-cost source domain, e.g., exploring drug reaction in humans by collecting feedback from mice or exploring real-world environments by collecting feedback from simulated environments? The challenge of effective cross-domain exploration is multifaceted: (1) The need for effective exploration in both the source and target domains depends on the quality of representations learned in the source domain, which in turn requires effective exploration; this leads to a chicken-and-egg dilemma. (2) Aligning source-domain representations with target-domain representations is nontrivial in bandit settings, where ground-truth may still be unknown if the action is incorrect. (3) Balancing these aspects – effective exploration and accurate alignment in bandit settings – is also nontrivial.

To address these challenges, as the first step, we allow our method to simultaneously perform effective exploration and representation alignment, leveraging unlabeled data from both the source and target domains. Interestingly, our theoretical analysis reveals that naively doing so leads to sub-optimal accuracy/regret (verified by our empirical results) and naturally leads to additional terms in the target-domain regret bound. We then follow the regret bound derived from our analysis to develop an algorithm that adaptively collect feedback from the source domain while aligning representations from the source and target domains. Our contributions are outlined as follows:

  • We identify the problem of contextual bandits across domains and propose domain-adaptive contextual bandits (DABand) as the first general method to explore a high-cost target domain while only collecting feedback from a low-cost source domain.

  • Our theoretical analysis shows that our method can achieve a sub-linear regret bound in the target domain.

  • Our empirical results on real-world datasets show our DABand significantly improve performance over the state-of-the-art contextual bandit methods when adapting across domains.

Related Work

Contextual Bandits. In the realm of adaptive decision-making, contextual bandit algorithms, epitomized by LinUCB [20], have carved a niche in efficiently balancing the exploitation-exploration paradigm leveraging estimation of reward and uncertainty [45] (Wang et al. 2016); they have been influential across a spectrum of use cases, notably in complex adaptive systems such as recommender systems [20, 45]. These algorithmic frameworks, along with their myriad adaptations [52, 1, 21, 17, 19, 11, 57, 25, 49], have decisively outperformed conventional bandit models that operate devoid of contextual awareness [2]. This superiority is underpinned by theoretical models, paralleling the insights in [2], where LinUCB variants are validated to conform to optimal regret boundaries in targeted scenarios [7]. However, a discernible gap in these methodologies is their reduced efficacy in a new domain, particularly when getting feedback involves high cost. Our proposed DABand bridges this gap by adeptly aligning both source-domain and target-domain representations in the latent space, thereby reducing performance drop when transfer across different domains.

Domain Adaptation. The landscape of domain adaptation has been extensively explored, as evidenced by a breadth of research [27, 28, 24, 33, 34, 53, 30, 6, 8, 26, 51] (Wang et al. 2020). The primary objective of these studies has been the alignment of source and target domain distributions to facilitate the effective generalization of models trained on labeled source data to unlabeled target data. This alignment is conventionally attained either through the direct matching of distributional statistics [28, 39, 37, 30, 26] or via the integration of an adversarial loss [13, 56, 40, 53, 6, 8, 50, 23, 35] (Kuroki et al. 2021; Wang et al. 2020). The latter, known as adversarial domain adaptation, has surged in prominence, bolstered by its theoretical foundation [14, 54, 53, 55], the adaptability of end-to-end training in neural network architectures, and its empirical efficacy. However, these methods only work in offline settings, and assumes complete observability of labels in the source domain. Therefore they are not applicable to our online bandit settings.

Distribution Shift in Bandits. A related study addresses the challenge of policy learning in contextual bandits using historical observational data, particularly when there is a shift in the environment [36]. The authors propose a distributionally robust approach to policy evaluation and learning, ensuring that the learned policy remains effective even under worst-case environmental shifts. However, their focus is distinct from ours. Their goal is to ensure policy robustness against environmental shifts between training and deployment within the same domain, whereas our proposed DABand focuses on adapting policies from a source domain to a different target domain, particularly when direct feedback from the target domain is limited or costly.

Domain Adaptation Related to Bandits. There is also work related to both domain adaptation and bandits. Specifically, Guo et al. [15] propose a domain adaptation method using a bandit algorithm to select which domain to use during training.

We note that their goal and setting is different from our DABand’s. (1) Guo et al. [15] focuses on improving accuracy in a typical, offline domain adaptation setting. In contrast, our DABand focuses on minimizing regret in an online bandit setting. (2) Guo et al. [15] assumes complete access to ground-truth labels in the source domain, while in our bandit setting, ground-truth may still be unknown if the action is incorrect. Therefore their work is not applicable to our setting.

Theory

In this section, we formalize the problem of contextual bandits across domains and derive the corresponding regret bound. We then develop our DABand inspired by this bound in § 4. All proofs of lemmas, theorems can be found in the Appendix.

Notation. We use [k][k] to denote the set {1,2,⋯ ,k}\{1,2,\cdots,k\}, for k∈N+k\in\mathbb{N}^{+}. The Euclidean norm of a vector x∈Rd\mathbf{x}\in\mathbb{R}^{d} is denoted as ∥x∥2=xTx\|\mathbf{x}\|_{2}=\sqrt{\mathbf{x}^{T}\mathbf{x}}. We denote the operator norm and Frobenius norm of a matrix W∈Rm×n\mathbf{W}\in\mathbb{R}^{m\times n} as ∥W∥\|\mathbf{W}\| and ∥W∥F\|\mathbf{W}\|_{F}, respectively. Given a semi-definite matrix A∈Rd×d\mathbf{A}\in\mathbb{R}^{d\times d} and a vector x∈Rd\mathbf{x}\in\mathbb{R}^{d}, we denote the Mahalanobis norm as ∥x∥A=xTAx\|\mathbf{x}\|_{\mathbf{A}}=\sqrt{\mathbf{x}^{T}\mathbf{A}\mathbf{x}}. For a function f(T)f(T) of a parameter TT, we denote as O(f(T))O(f(T)) the terms growing in the order of f(T)f(T), ignoring constant factors. We assume all the action spaces are identical and with cardinality of KK, i.e., ∣A1∣=∣A2∣=⋯=∣AN∣=K|\mathcal{A}_{1}|=|\mathcal{A}_{2}|=\cdots=|\mathcal{A}_{N}|=K. We use ⟨⋅,⋅⟩\langle\cdot,\cdot\rangle to denote the inner product of two vectors. We denote as aD,ia_{\mathcal{D},i} the action ii in domain D\mathcal{D}, and omit the subscript D\mathcal{D} when the context is clear, e.g., xi,ai∗D≡xi,aD,i∗D\mathbf{x}_{i,a_{i}^{*}}^{\mathcal{D}}\equiv\mathbf{x}_{i,a_{\mathcal{D},i}^{*}}^{\mathcal{D}}.

Preliminaries

Typical Contextual Bandit Setting. Suppose we have NN samples from an unknown domain D\mathcal{D}, which is represented as {xi,aD,ai∗}i∈[N],a∈[K]\{\mathbf{x}_{i,a}^{\mathcal{D}},a_{i}^{*}\}_{i\in[N],a\in[K]}. Here, xi,a∈D\mathbf{x}_{i,a}\in\mathcal{D} denotes the context for action a∈Aia\in\mathcal{A}_{i}, and ai∗∈Aia_{i}^{*}\in\mathcal{A}_{i} is the ground-truth action that will receive the optimal reward. We denote as ri,ar_{i,a} the received reward in round ii after performing action aa and a^i∈Ai\hat{a}_{i}\in\mathcal{A}_{i} the chose action by a bandit algorithm in round ii. The goal in typical contextual bandits is to learn a policy of choosing actions a^i\hat{a}_{i} in each round to minimize the regret after NN rounds:

R=∑i=1Nri,ai∗−∑i=1Nri,a^i.(1)R=\sum_{i=1}^{N}r_{i,a_{i}^{*}}-\sum_{i=1}^{N}r_{i,\hat{a}_{i}}. \tag*{(1)}

LinUCB. LinUCB (Li et al., [20]) is a classic method for the typical contextual bandit setting. It assumes that the reward is linear to its input context, i.e., xi,a\mathbf{x}_{i,a}. During each round ii, the agent selects an arm according to historical contexts and their corresponding rewards, ri,ar_{i,a}:

a^i=argmax⁡a∈Ai{xi,aTθi+α∥xi,a∥Ai−1−1}.\hat{a}_{i}=\underset{a\in\mathcal{A}_{i}}{\operatorname{argmax}}\left\{\mathbf{x}_{i,a}^{T}\theta_{i}+\alpha\|\mathbf{x}_{i,a}\|_{\mathbf{A}_{i-1}^{-1}}\right\}.

where Ai−1=γId+∑j=1i−1xj,a^jxj,a^jT\mathbf{A}_{i-1}=\gamma\mathbf{I}_{d}+\sum_{j=1}^{i-1}\mathbf{x}_{j,\hat{a}_{j}}\mathbf{x}_{j,\hat{a}_{j}}^{T} with γ>0\gamma>0 and xj,a^jT\mathbf{x}_{j,\hat{a}_{j}}^{T} as the historical selected action’s context, xi,a\mathbf{x}_{i,a} is the context for the candidate action aa in round ii, θi\theta_{i} is the bandit parameter in round ii and α>0\alpha>0 is a hyperparameter that adjusts the exploration rate.

LinUCB with Representation Learning. In this paper, we use Neural LinUCB (Xu et al., [49]; Wang et al., [45]) as a backbone model to handle high-dimensional context with representation learning. We assume that there exist a ground-truth encoder ϕ∗\phi^{*} to encode the raw context xi,a\mathbf{x}_{i,a} into a latent-space representation (encoding ϕ∗(xi,a)\phi^{*}(\mathbf{x}_{i,a})). Subsequently, the reward for the context xi,a\mathbf{x}_{i,a} is then r(xi,a)=⟨θ∗,ϕ∗(xi,a)⟩r(\mathbf{x}_{i,a})=\langle\theta^{*},\phi^{*}(\mathbf{x}_{i,a})\rangle plus some stochastic noise ϵ\epsilon. Furthermore, we use the same setting as in Xu et al. [49] where the ground-truth rewards are restricted to the range of [0,1][0,1]. Additionally, we further bound predicted rewards by normalizing ∥θ^∥=1\|\hat{\theta}\|=1 and ∥ϕ^(xi,a)∥=1\|\hat{\phi}(\mathbf{x}_{i,a})\|=1.

In the single-domain typical setting, the goal is to learn an encoder ϕ^\hat{\phi} and the contextual bandit parameter θ^\hat{\theta}, such that our estimated reward r^(xi,a)=⟨θ^,ϕ^(xi,a)⟩\hat{r}(\mathbf{x}_{i,a})=\langle\hat{\theta},\hat{\phi}(\mathbf{x}_{i,a})\rangle can be close to the ground-truth reward r(xi,a)r(\mathbf{x}_{i,a}). One can then use this to form derive policies to minimize the regret in Eq. (1).

For simplicity, in Definition 3.1 below, we further denote as fDf_{\mathcal{D}} the (ground-truth) labeling function for domain D\mathcal{D} and as h∈Hh\in\mathcal{H} a hypothesis such that ff is parameterized by ϕD∗,θD∗\phi_{\mathcal{D}}^{*},\theta_{\mathcal{D}}^{*}, and hh is parameterized by ϕ^,θ^\hat{\phi},\hat{\theta}.

Our Cross-Domain Contextual Bandit Setting

Typical contextual bandits operate in a single domain D\mathcal{D}. In contrast, our cross-domain bandit setting involves a source domain S\mathcal{S} and a target domain T\mathcal{T}. In this setting, one can only collect feedback (reward) from the source domain, but not from the target domain. Specifically: (1) We assume a low-cost source domain (experiments on mice), where for each round ii, one has access to the contexts {xi,aS}a∈[K]\{x^{S}_{i,a}\}_{a\in[K]} for each candidate actions, chooses one action a^i\hat{a}_i, and receives reward ri,a^iSr^{S}_{i,\hat{a}_i}. (2) Additionally, we assume a high-cost target domain where collecting feedback (reward) is expensive (experiments on humans); therefore, one only has access to the contexts {xi,aT}a∈[K]\{x^{T}_{i,a}\}_{a\in[K]} for each candidate actions for each round, but cannot collect feedback (reward) ri,a^iTr^{T}_{i,\hat{a}_i} for any action in the target domain. The goal in our cross-domain contextual bandits is to learn a policy of choosing actions a^i\hat{a}_i in the target domain to minimize the target-domain zero-shot regret for NN (hypothetical) future rounds:

RT=∑i=1Nri,ai∗T−∑i=1Nri,a^iT.(2)R_T = \sum_{i=1}^{N} r^{T}_{i,a_i^*} - \sum_{i=1}^{N} r^{T}_{i,\hat{a}_i}. \tag*{(2)}

Difference between Eq. (1) and Eq. (2). In the typical setting, Eqn. (1) uses different updated policies for each round; this is also true for the source domain. In contrast, the target regret in Eqn. (2) use the same fixed policy obtained from the source domain for all NN (hypothetical) rounds because one cannot collect feedback from the target domain. See the difference between Definition 3.4 and Definition 3.5 below for a more formal comparison.

Formal Definitions of Error

Definition 3.1 (Labeling Function and Hypothesis). We define the labeling function fDf_D and the hypothesis hh for domain DD as follows:

fD(xi,aD)=r(xi,aD)=⟨θD∗,ϕD∗(xi,aD)⟩+ϵif_D(x^D_{i,a}) = r(x^D_{i,a}) = \langle\theta_D^*, \phi_D^*(x^D_{i,a}) \rangle+ \epsilon_i
h(xi,aD)=r^(xi,aD)=⟨θ^,ϕ^(xi,aD)⟩+ϵih(x^D_{i,a}) = \hat{r}(x^D_{i,a}) = \langle\hat{\theta}, \hat{\phi}(x^D_{i,a}) \rangle+ \epsilon_i

where θD∗\theta_D^* is the optimal predictor for domain DD, θ^\hat{\theta} denotes the estimated predictor, and ϵi\epsilon_i is the random noise.

Next, we analyze how well our hypothesis hh estimates the labeling function ff (i.e., ground-truth hypothesis). Definition 3.2 below quantifies the closeness between two hypotheses by calculating the absolute difference in their estimated rewards.

Definition 3.2 (Estimated Error between Two Hypotheses). Assuming all contexts xi,a^iDx^D_{i,\hat{a}_i} are from domain DD, the error between two hypotheses h1,h2∈Hh_1, h_2 \in\mathcal{H} on domain DD given selected actions {a^i}i∈[N]∈[K]\{\hat{a}_i\}_{i\in[N]} \in[K] is

ϵD(h1,h2)=∑i=1N(∣h1(xi,a^iD)−h2(xi,a^iD)∣)=∑i=1N(∣⟨θ^1,ϕ^1(xi,a^iD)⟩−⟨θ^2,ϕ^2(xi,a^iD)⟩∣).(3)\epsilon_D(h_1,h_2) = \sum_{i=1}^{N} \left( \left|h_1(x^D_{i,\hat{a}_i}) - h_2(x^D_{i,\hat{a}_i})\right| \right) = \sum_{i=1}^{N} \left( \left|\langle\hat{\theta}_1,\hat{\phi}_1(x^D_{i,\hat{a}_i})\rangle- \langle\hat{\theta}_2,\hat{\phi}_2(x^D_{i,\hat{a}_i})\rangle\right| \right). \tag*{(3)}

Note that the domain DD above can be the source domain SS or the target domain TT. We then define the error of our estimated hypothesis in these domains.

Definition 3.3 (Source- and Target-Domain Error). With NN samples from the source domain SS, and fSf_S denoting the labeling function for SS, the source-domain error is then

ϵS(fS,h)=∑i=1N(∣fS(xi,a^iS)−h(xi,a^iS)∣)=∑i=1N(∣⟨θS∗,ϕS∗(xi,a^iS)⟩−⟨θ^,ϕ^(xi,a^iS)⟩∣),\epsilon_S(f_S,h) = \sum_{i=1}^{N} \left( \left|f_S(x^S_{i,\hat{a}_i}) - h(x^S_{i,\hat{a}_i})\right| \right) = \sum_{i=1}^{N} \left( \left|\langle\theta_S^*,\phi_S^*(x^S_{i,\hat{a}_i})\rangle- \langle\hat{\theta},\hat{\phi}(x^S_{i,\hat{a}_i})\rangle\right| \right),

where xi,⋅Sx^S_{i,\cdot} comes from SS. Furthermore, a^i=arg⁡max⁡ah(xi,aS)\hat{a}_i = \arg\max_a h(x^S_{i,a}) and ai∗=arg⁡max⁡afS(xi,aS)a_i^* = \arg\max_a f_S(x^S_{i,a}).

For simplicity, we shorten the notation ϵS(fS,h)\epsilon_S(f_S,h) to ϵS(h)\epsilon_S(h) and similarly use ϵT(h)\epsilon_T(h) for domain TT. Assuming xi,⋅Tx^T_{i,\cdot} comes from TT, the estimated error for the target domain is then:

ϵT(fT,h)=∑i=1N(∣fT(xi,a^iT)−h(xi,a^iT)∣)=∑i=1N(∣⟨θT∗,ϕT∗(xi,a^iT)⟩−⟨θ^,ϕ^(xi,a^iT)⟩∣).\epsilon_T(f_T,h) = \sum_{i=1}^{N} \left( \left|f_T(x^T_{i,\hat{a}_i}) - h(x^T_{i,\hat{a}_i})\right| \right) = \sum_{i=1}^{N} \left( \left|\langle\theta_T^*,\phi_T^*(x^T_{i,\hat{a}_i})\rangle- \langle\hat{\theta},\hat{\phi}(x^T_{i,\hat{a}_i})\rangle\right| \right).

Source Regret and Target Regret

Below define the regret for the source and target domains.

Definition 3.4 (Source Regret). Assuming xi,⋅Sx^S_{i,\cdot} comes from domain SS, the source regret (i.e., the regret in the source domain) is

RS=∑i=1N(fS(xi,ai∗S)−fS(xi,a^iS))=∑i=1N(⟨θS∗,ϕS∗(xi,ai∗S)⟩−⟨θS∗,ϕS∗(xi,a^iS)⟩).(4)R_S = \sum_{i=1}^{N} \left( f_S(x^S_{i,a_i^*}) - f_S(x^S_{i,\hat{a}_i}) \right) = \sum_{i=1}^{N} \left( \langle\theta_S^*,\phi_S^*(x^S_{i,a_i^*})\rangle- \langle\theta_S^*,\phi_S^*(x^S_{i,\hat{a}_i})\rangle\right). \tag*{(4)}

The goal in our cross-domain contextual bandits is to learn a policy of choosing actions a^i\hat{a}_i in the high-cost target domain (e.g., human experiments) by collecting feedback only in the source domain (e.g., mouse experiments). Formally, we would like to minimize the target regret as defined below.

Definition 3.5 (Target Regret and Problem Formulation). Denoting the estimated hypothesis as h^={ϕ^,θ^}\hat{h} = \{\hat{\phi}, \hat{\theta}\}, the target regret we aim to minimize is defined as

RT=∑i=1N(fT(xi,ai∗T)−fT(xi,a^iT))R_T = \sum_{i=1}^{N} \left(f_T(x_{i,a_i^*}^{T}) - f_T(x_{i,\hat{a}_i}^{T})\right)
s.t. a^i=argmax⁡a h^(xi,aT)+α∥ϕ^(xi,aT)∥[AS]−1,h^=argmin⁡h ϵS(h),\mathrm{s.t.}\ \hat{a}_i = \underset{a}{\operatorname{argmax}}\ \hat{h}(x_{i,a}^{T}) + \alpha\lVert\hat{\phi}(x_{i,a}^{T}) \rVert_{[A^S]^{-1}}, \qquad\hat{h} = \underset{h}{\operatorname{argmin}}\ \epsilon_S(h),

where AS=γI+∑i=1N[ϕ^(xi,a^iS)][ϕ^(xi,a^iS)]TA^S = \gamma I + \sum_{i=1}^{N} [\hat{\phi}(x_{i,\hat{a}_i}^{S})][\hat{\phi}(x_{i,\hat{a}_i}^{S})]^{T} denotes the context matrix accumulated by the selected context features in the source domain.

Cross-Domain Error Bound for Regression

Prior to introducing our final regret bound, it is necessary to define an additional component. A fundamental challenge in general domain adaptation problems is to manage the divergence between source and target domains. Unfortunately, previous domain adaptation theory only covers classification tasks [4, 53]. In contrast, the problem of contextual bandits is essentially a reward regression problem. To address this challenge, we introduce the following new definition to bound the error between the source and target domains.

Definition 3.6 (HΔH\mathcal{H}\Delta\mathcal{H} Hypothesis Space for Regression). For a hypothesis space H\mathcal{H}, the symmetric difference hypothesis space HΔH\mathcal{H}\Delta\mathcal{H} is the set of hypotheses s.t.

g∈HΔH⟺g(x)=∣h(x)−h′(x)∣.g \in\mathcal{H}\Delta\mathcal{H} \quad\Longleftrightarrow\quad g(x) = \lvert h(x) - h'(x) \rvert.

We then can further define the Divergence for HΔH\mathcal{H}\Delta\mathcal{H} Hypothesis Space:

d^HΔH(S,T)=2sup⁡h,h′∈H∣Ex∼S[∣h(x)−h′(x)∣]−Ex∼T[∣h(x)−h′(x)∣]∣\hat{d}_{\mathcal{H}\Delta\mathcal{H}}(S,T) = 2 \sup_{h,h' \in\mathcal{H}} \left\lvert\mathbb{E}_{x\sim S}[\lvert h(x)-h'(x)\rvert] - \mathbb{E}_{x\sim T}[\lvert h(x)-h'(x)\rvert] \right\rvert

Definition 3.7 (Optimal Hypothesis). The optimal hypothesis, denoted as h∗h^*, can balance the estimated error for both source and target domain. Formally,

h∗=argmin⁡h∈H ϵS(h)+ϵT(h),h^* = \underset{h \in\mathcal{H}}{\operatorname{argmin}}\ \epsilon_S(h) + \epsilon_T(h),

and we define the minimum estimated error for the optimal hypothesis h∗h^* as

ψ=ϵS(h∗)+ϵT(h∗).\psi= \epsilon_S(h^*) + \epsilon_T(h^*).

Final Regret Bound

With all the definitions of source/target regret in Definition 3.4 and Definition 3.5, we can then bound the target regret using Theorem 3.1 below.

Theorem 3.1 (Target Regret Bound). Denoting the contexts from the source domain SS as {xt,aS}i∈[N],a∈[K]\{x_{t,a}^{S}\}_{i\in[N],a\in[K]}, the associated ground-truth action as {ai∗}i=1N\{a_i^*\}_{i=1}^{N}, and the contexts from the target domain TT as {xt,aT}i∈[N],a∈[K]\{x_{t,a}^{T}\}_{i\in[N],a\in[K]}, the upper bound for our target regret RTR_T is

RT≜∑i=1N(∣⟨θT∗,ϕT∗(xi,ai∗T)⟩−⟨θT∗,ϕT∗(xi,a^iT)⟩∣)R_T \triangleq\sum_{i=1}^{N} \left(\left\lvert\langle\theta_T^*,\phi_T^*(x_{i,a_i^*}^{T})\rangle- \langle\theta_T^*,\phi_T^*(x_{i,\hat{a}_i}^{T})\rangle\right\rvert\right)
≤RS⏟Source Regret+2⋅ϵS(h)⏟Regression Error+N⋅d^HΔH(S,T)⏟Data Divergence+ψ⏟Constant+C+∑i=1N(∣⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1N1[ai∗≠a^i](∣⟨θ^,ϕ^(xi,ai∗S)⟩∣)(5)\leq\underbrace{R_S}_{\text{Source Regret}} + \underbrace{2\cdot\epsilon_S(h)}_{\text{Regression Error}} + \underbrace{N\cdot\hat{d}_{\mathcal{H}\Delta\mathcal{H}}(S,T)}_{\text{Data Divergence}} + \underbrace{\psi}_{\text{Constant}} + C + \sum_{i=1}^{N}\left(\left\lvert\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{T})\rangle\right\rvert\right) + \sum_{i=1}^{N}\mathbf{1}[a_i^* \ne\hat{a}_i]\left(\left\lvert\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{S})\rangle\right\rvert\right) \tag*{(5)}

where RSR_S, ϵS(h)\epsilon_S(h), and d^HΔH\hat{d}_{\mathcal{H}\Delta\mathcal{H}} are the source regret, source-domain error, and HΔH\mathcal{H}\Delta\mathcal{H} divergence defined in Definition 3.4, Definition 3.3, and Definition 3.6, respectively. ψ\psi is a constant independent to the problem and CC is a constant which can be ignored (see the Appendix for more details).

Here we provide several observations (for more analysis, please refer to the Appendix):

  1. Since RSR_S is sub-linear w.r.t. the number of samples in the source domain, so is RTR_T.

  2. Theorem 3.1 bounds the target regret using the source regret, enabling the exploration of the target domain by only collecting feedback (reward) from the source domain.

  3. Typical generalization bounds in DA naively minimize the source regret RSR_S while aligning source and target data in the latent space (i.e., minimizing dHΔHd_{\mathcal{H}\Delta\mathcal{H}}). This is not sufficient, as we need two additional terms, i.e., the regression error ϵS(h)\epsilon_S(h) and the predicted reward ∑i=1N(∣⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1N1[ai∗≠a^i](∣⟨θ^,ϕ^(xi,a^iS)⟩∣)\sum_{i=1}^{N}\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{T})\right\rangle\right|\right)+\sum_{i=1}^{N}\mathbb{1}[a_i^*\ne\hat{a}_i]\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{S})\right\rangle\right|\right), as regularization terms.

  4. Minimizing the regression error ϵS(h)\epsilon_S(h) encourages accurate prediction of source rewards.

  5. Minimizing the predicted reward ∑i=1N(∣⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1N1[ai∗≠a^i](∣⟨θ^,ϕ^(xi,a^iS)⟩∣)\sum_{i=1}^{N}\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{T})\right\rangle\right|\right)+\sum_{i=1}^{N}\mathbb{1}[a_i^*\ne\hat{a}_i]\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{S})\right\rangle\right|\right) regularizes the model to avoid overestimating rewards.

Note that Theorem 3.1 is nontrivial. While it does resemble the generalization bound in domain adaptation, there are key differences. As mentioned in Observation (3) above, our target regret bound includes two additional crucial terms not found in domain adaptation. Specifically:

  • Regression Error in the Source Domain. ∑i=1N(∣⟨θS∗,ϕS∗(xi,a^iS)⟩−⟨θ^,ϕ^(xi,a^iS)⟩∣)\sum_{i=1}^{N}\left(\left|\left\langle\theta_S^*,\phi_S^*(x_{i,\hat{a}_i}^{S})\right\rangle-\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{S})\right\rangle\right|\right) (i.e., ϵS(h)\epsilon_S(h) in Theorem 3.1) defines the difference between the true reward from selecting action a^i\hat{a}_i and the estimated reward for this action.

  • Predicted Reward. ∑i=1N(∣⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1N1[ai∗≠a^i](∣⟨θ^,ϕ^(xi,a^iS)⟩∣)\sum_{i=1}^{N}\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{T})\right\rangle\right|\right)+\sum_{i=1}^{N}\mathbb{1}[a_i^*\ne\hat{a}_i]\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{S})\right\rangle\right|\right) serves as a regularization term to regularize the model to avoid overestimating rewards.

The results of the ablation study in Table 4 in § 5 highlight the significance of these two terms. See Appendix § 14.§ 4 for more discussion on novelty as well as key differences between our DABand and classic domain adaptation.

DatasetsW/O R & PW/O RW/O PDABAND (FULL)
DIGIT0.56760.56820.57680.6002
VISDA170.40880.40960.43040.4644
S2RDA490.36910.36940.37190.3923

Table 4. Results of the ablation study in terms of accuracy (higher is better). Note that the accuracy ACC=1−1NRTACC = 1 - \frac{1}{N}R_T, where RTR_T is the target regret. “R” and “P” is short for “Regression Error” and “Predicted Reward”, respectively.

Method

With Theorem 3.1, we can then design our DABand algorithm to obtain optimal target regret. We discuss how to translate each term in Eqn. (5) to a differential loss term in § 4.1 and then put them together in § 4.2.

From Theory to Practice: Translating the Bound in Eqn. (5) to Differentiable Loss Terms

Source Regret. Inspired by Neural-LinUCB [49], we use LinUCB [20] to update θ^\hat{\theta} and the following empirical loss function when updating the encoder parameters ϕ^\hat{\phi} by back-propagation in round ii:

LiRS=∑a=1K(θ^Tϕ^(xi,a)−r(xi,a))2,(6)\mathcal{L}_i^{R_S}=\sum_{a=1}^{K}\left(\hat{\theta}^{T}\hat{\phi}(x_{i,a})-r(x_{i,a})\right)^2, \tag*{(6)}

where θ\theta is the contextual bandit parameter, and ϕ^\hat{\phi} is the encoder shared by the source and target domains, i.e., we set ϕ^S=ϕ^T\hat{\phi}_S=\hat{\phi}_T during training.

Insights. From Eqn. (6), a fundamental difference between classification problems and bandit problems becomes apparent. In classification tasks, both the ground-truth label and the estimated label are accessible. However, in bandit problems, we only know the estimated label and reward. If the estimated rewards align with our estimated label’s correctness, it suggests knowledge of the ground-truth label for that feature. If not, we can only deduce that the estimated label is inaccurate, without identifying the correct label among the other candidates. This uncertainty significantly amplifies the complexity of contextual bandit problems.

Regression Error. We use L1L_1 loss to optimize the regression error, defined as the difference between the true reward from selecting action a^i\hat{a}_i and the estimated reward for this action (see Definition 3.3 for a detailed explanation of this regression error). Minimizing the regression error term directly

Algorithm 1 DABand Training Algorithm

  1. Input: regularization parameter γ>0\gamma> 0, number of total steps NN, episode length HH, exploration parameters α\alpha.

  2. Output: parameters of the model: ϕ^N,θ^N,AN,bN\hat{\phi}_N, \hat{\theta}_N, A_N, b_N.

  3. Initialization: A0=γIA_0 = \gamma I, b0=0b_0 = 0, and for θ^0\hat{\theta}_0 and ϕ^0\hat{\phi}_0 are initialized following Xu et al. [49].

  4. for i=1,…,Ni = 1, \ldots, N do

  5. Obtain {xi,aS}a∈[K]\{x_{i,a}^{S}\}_{a \in[K]} and {xi,aT}a∈[K]\{x_{i,a}^{T}\}_{a \in[K]} from the source and the target domain, respectively.

  6. Choose an action a^i=argmax⁡a∈[K][ϕ^i−1(xi,aS)Tθ^i−1+α∥ϕ^i−1(xi,aS)∥Ai−1−1]\hat{a}_i = \underset{a \in[K]}{\operatorname{argmax}}\left[\hat{\phi}_{i-1}(x_{i,a}^{S})^{\mathsf{T}}\hat{\theta}_{i-1} + \alpha\left\|\hat{\phi}_{i-1}(x_{i,a}^{S})\right\|_{A_{i-1}^{-1}}\right].

  7. Get the reward ri=r(xi,a^i)r_i = r(x_{i,\hat{a}_i}) based on the selected action a^i\hat{a}_i.

  8. Update bandit parameters:

{Ai=Ai−1+[ϕ^i−1(xi,a^iS)][ϕ^i−1(xi,a^iS)]T,bi=bi−1+ri[ϕ^i−1(xi,a^iS)],θ^i=Ai−1bi\left\{ \begin{aligned} A_i &= A_{i-1} + \left[\hat{\phi}_{i-1}(x_{i,\hat{a}_i}^{S})\right]\left[\hat{\phi}_{i-1}(x_{i,\hat{a}_i}^{S})\right]^{\mathsf{T}},\\ b_i &= b_{i-1} + r_i\left[\hat{\phi}_{i-1}(x_{i,\hat{a}_i}^{S})\right],\\ \hat{\theta}_i &= A_i^{-1}b_i \end{aligned} \right.
  1. if mod⁡(i,H)=0\operatorname{mod}(i,H)=0 then

  2. for j=1,…,Hj = 1,\ldots,H do

  3. Calculate LDABand\mathcal{L}^{DABand} in Eqn. (11).

  4. Use the Adam optimizer Diederik [10] to update encoder ϕ^i\hat{\phi}_i and discriminator gg by back-propagation to solve the minimax optimization in Eqn. (10).

  5. end for

  6. else

  7. ϕ^i=ϕ^i−1\hat{\phi}_i = \hat{\phi}_{i-1}

  8. end if

  9. end for

  10. Output: ϕ^N,θ^N,AN,bN\hat{\phi}_N,\hat{\theta}_N,A_N,b_N.

encourages accurate prediction of source rewards using L1L_1 loss. Specifically, we use

LiϵS(h)=∣⟨θS,ϕS(xi,a^iS)⟩−⟨θ^,ϕ^(xi,a^iS)⟩∣,(7)\mathcal{L}^{\epsilon_S(h)}_i = \left|\left\langle\theta^S,\phi^S(x_{i,\hat{a}_i}^{S})\right\rangle-\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{S})\right\rangle\right|, \tag*{(7)}

where ⟨θS,ϕS(xi,a^iS)⟩\left\langle\theta^S,\phi^S(x_{i,\hat{a}_i}^{S})\right\rangle is the reward received by the agent in the source domain. Minimizing the regression error term above directly encourages accurate prediction of source rewards.

Data Divergence. The data divergence term in Eqn. (5) leads to following loss term:

Ldiv⁡=max⁡gN⋅(ES[LD(g(ϕ^(x)),0)]+ET[LD(g(ϕ^(x)),1)]),(8)\mathcal{L}^{\operatorname{div}} = \max_g N \cdot\left(\mathbb{E}^{S}\left[\mathcal{L}_{D}(g(\hat{\phi}(x)),0)\right] + \mathbb{E}^{T}\left[\mathcal{L}_{D}(g(\hat{\phi}(x)),1)\right]\right), \tag*{(8)}

where ES\mathbb{E}^{S} and ET\mathbb{E}^{T} denote expectations over the data distributions of (x,a)(x,a) in the source and target domains, respectively. gg is a discriminator that classifies whether xx is from the source domain or the target domain. LD\mathcal{L}_{D} is the binary classification accuracy, where labels 0 and 1 indicate the source and target domains, respectively. In practice, we use the cross-entropy loss as a differentiable surrogate loss. As in Ganin et al. [13], solving the minimax optimization min⁡ϕ^max⁡gLdiv⁡\min_{\hat{\phi}}\max_g \mathcal{L}^{\operatorname{div}} is equivalent to aligning source- and target-domain data distributions in the latent (encoding) space induced by the encoder ϕ^\hat{\phi}.

Predicted Reward. The predicted reward term in Eqn. (5), i.e., ∑i=1N(∣⟨θ^,ϕ^(xi,a^iT)⟩∣+1[ai∗≠a^i]∣⟨θ^,ϕ^(xi,a^iS)⟩∣)\sum_{i=1}^{N}\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{T})\right\rangle\right|+\mathbb{1}[a_i^* \ne\hat{a}_i]\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{S})\right\rangle\right|\right), can be translated to the loss term ∑i=1NLireg⁡\sum_{i=1}^{N}\mathcal{L}^{\operatorname{reg}}_i, where for each round ii we have

Lireg⁡=∣⟨θ^,ϕ^(xi,a^iT)⟩∣+1[ai∗≠a^i](∣⟨θ^,ϕ^(xi,a^iS)⟩∣).(9)\mathcal{L}^{\operatorname{reg}}_i = \left|\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{T})\right\rangle\right|+\mathbb{1}[a_i^* \ne\hat{a}_i]\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{S})\right\rangle\right|\right). \tag*{(9)}

Minimizing the predicted reward Lireg⁡\mathcal{L}^{\operatorname{reg}}_i regularizes the model to avoid overestimating rewards.

Constant Term. In Eqn. (5), ψ+C\psi+ C is the constant term, where ψ\psi is defined in Definition 3.7, and CC contains some small constants which can be ignored (see the Appendix for more details).

Putting Everything Together: DABand Training Algorithm

Putting all non-constant loss terms above together, we have the final minimax optimization problem

min⁡ϕmax⁡gLDABand,(10)\min_{\phi}\max_g \mathcal{L}^{DABand}, \tag*{(10)}

where the value function (objective function):

LDABand=∑i=1N(LicRs+2⋅Liϵs(h)+Lireq)+λ⋅Lidiv.(11)\mathcal{L}^{DABand} = \sum_{i=1}^{N}\left(\mathcal{L}_{i}^{cRs} + 2\cdot\mathcal{L}_{i}^{\epsilon s(h)} + \mathcal{L}_{i}^{req}\right) + \lambda\cdot\mathcal{L}_{i}^{div}. \tag*{(11)}

Note that we need to alternate between updating θ^\hat{\theta} using LinUCB Li et al. [20] and updating the encoder ϕ^\hat{\phi} with the discriminator gg in Eqn. (10).

Formally, our method, described in Alg. 1, operates as follows: In each iteration, we access the context in the original feature space (i.e., Rd\mathbb{R}^{d}) for each action from both source and target domains, denoted as {xi,aS}a∈[K]\{x_{i,a}^{S}\}_{a\in[K]} and {xi,aT}a∈[K]\{x_{i,a}^{T}\}_{a\in[K]} respectively. Subsequently, we compute the latent representation using the encoder ϕ(⋅)\phi(\cdot) and select the action a^i\hat{a}_{i} for iteration ii based on the LinUCB selection rule. The bandit parameters (i.e., θ\theta, AA, bb) are updated in each iteration. The weights in ϕ\phi are updated every episode of length HH (i.e., a batch with HH past iterations) using our objective function in Eqn. (10), following the same rule as in Neural-LinUCB ([49]).

Experiments

In this section, we compare DABand with existing methods on real-world datasets.

Datasets. To demonstrate the effectiveness of our DABand, We evaluate our methods in terms of prediction accuracy and zero-shot target regret on three datasets, i.e., DIGIT ([12]), VisDA17 ([29]), and S2RDA49 ([38]). See details for each dataset in Appendix § 9.

Baselines. We compare our DABand with both classic and state-of-the-art contextual bandit algorithms, including LinUCB ([20]), LinUCB with principle component analysis (PCA) pre-processing (i.e., LinUCB-P), Neural-LinUCB ([49]) (NLinUCB), and Neural-LinUCB variant that incorporates PCA (i.e., NLinUCB-P). Details for each baselines and discussion are in Appendix § 10. Note that domain adaptation baselines are not applicable to our setting because it only works in offline settings and assumes complete observability of labels in the source domain (see Appendix § 10 for details).

Zero-Shot Target Regret (Accuracy). We evaluate different methods on the zero-shot target regret setting in Definition 3.5, where all methods can only collect feedback from the source domain, but not the target domain. In this setting, accuracy is equal to 11 minus the average target regret, i.e.,

ACC=1−1NRTACC = 1 - \frac{1}{N}R_T, where RTR_T is defined in Eqn. (2) and Definition 3.5. Therefore higher accuracy indicates lower target regret and better performance.

We report the per-class accuracy and average accuracy of different methods in Table 1, Table 2 and Table 3 for DIGIT, VisDA17 and S2RDA49, respectively. Our DABand demonstrates favorable performance against NLinUCB, a leading contextual bandit algorithm with representation learning. Despite NLinUCB’s superior representation learning capabilities through neural networks (NN) over LinUCB – yielding impressive performance in single-domain applications – its efficacy in cross-domain tasks is heavily contingent on the disparity between the source and target domains. Typically, the strengths of NN, which include enhanced representation learning (feature extraction), may inadvertently amplify this domain divergence, potentially leading to overfitting on the source domain. Conversely, our DABand algorithm not only improves accuracy but also adeptly addresses the challenges posed by domain divergence. Note that LinUCB encounters an out-of-memory issue due to the high-dimensionality of the data in VisDA17 and S2RDA49, marked as “N/A” in Table 2 and Table 3, respectively.

MetricsAcc Per Class ↑\uparrowAcc ↑\uparrow
Test Accuracy0123456789Average
LinUCB ([20])0.3667±\pm0.030.4045±\pm0.020.5102±\pm0.030.3811±\pm0.010.3157±\pm0.040.3445±\pm0.010.4229±\pm0.030.4595±\pm0.030.3489±\pm0.020.2449±\pm0.010.3808±\pm0.02
LinUCB-P ([20])0.3645±\pm0.020.6398±\pm0.040.3151±\pm0.020.2709±\pm0.030.2258±\pm0.010.1995±\pm0.020.1752±\pm0.010.3786±\pm0.040.1273±\pm0.020.2992±\pm0.030.3060±\pm0.02
NLinUCB ([49])0.3781±\pm0.040.3228±\pm0.030.3569±\pm0.030.3381±\pm0.040.3663±\pm0.040.4312±\pm0.050.4766±\pm0.050.4004±\pm0.050.4034±\pm0.020.3613±\pm0.030.3816±\pm0.04
NLinUCB-P ([49])0.8588±\pm0.030.2224±\pm0.020.3880±\pm0.030.3029±\pm0.020.2978±\pm0.030.0000±\pm0.010.1869±\pm0.020.2243±\pm0.010.2148±\pm0.010.0000±\pm0.000.2706±\pm0.02
DABand (Ours)0.6891±\pm0.100.6662±\pm0.030.6188±\pm0.010.5703±\pm0.050.6008±\pm0.050.5534±\pm0.030.6010±\pm0.020.5709±\pm0.010.6266±\pm0.040.5023±\pm0.020.6002±\pm0.02
Ours vs. NLinUCB+0.3110+0.3434+0.2619+0.2322+0.2345+0.1222+0.1244+0.1705+0.2232+0.1410+0.2186

Table 1. Accuracy on the target domain for the DIGIT dataset. Note that in this zero-shot target regret setting, the accuracy ACC=1−1NRTACC = 1 - \frac{1}{N}R_T, where RTR_T is the target regret.

MetricsAcc Per Class ↑\uparrowAcc ↑\uparrow
Test AccuracyaeroplanebicyclebuscarhorseknifemotorcyclepersonplantskateboardtraintruckAverage
LinUCB ([20])N/AN/AN/AN/AN/AN/AN/AN/AN/AN/AN/AN/AN/A
LinUCB-P ([20])0.0510±\pm0.020.0190±\pm0.000.1360±\pm0.050.0440±\pm0.020.0390±\pm0.010.5580±\pm0.060.0410±\pm0.020.2050±\pm0.050.0850±\pm0.040.0920±\pm0.130.0580±\pm0.020.0460±\pm0.120.1145±\pm0.06
NLinUCB ([49])0.2870±\pm0.030.0220±\pm0.020.0000±\pm0.000.1250±\pm0.020.0000±\pm0.000.0130±\pm0.020.0470±\pm0.020.0000±\pm0.000.0460±\pm0.020.0050±\pm0.000.6550±\pm0.040.0000±\pm0.000.1001±\pm0.02
NLinUCB-P ([49])0.0060±\pm0.010.0020±\pm0.010.9222±\pm0.020.0010±\pm0.000.0080±\pm0.010.0990±\pm0.030.0030±\pm0.000.0020±\pm0.000.0180±\pm0.020.0420±\pm0.010.0000±\pm0.000.0150±\pm0.010.0932±\pm0.01
DABand (Ours)0.5504±\pm0.020.3562±\pm0.040.4462±\pm0.020.3844±\pm0.010.5582±\pm0.040.2632±\pm0.030.6474±\pm0.030.2396±\pm0.030.4882±\pm0.040.4062±\pm0.020.7685±\pm0.050.1968±\pm0.020.4644±\pm0.03
Ours vs. NLinUCB+0.2634+0.3342+0.4462+0.2594+0.5582+0.2502+0.6004+0.2396+0.4422+0.4012+0.1135+0.1968+0.3643

Table 2. Accuracy on the target domain for the VisDA17 dataset. Note that in this zero-shot target regret setting, the accuracy ACC=1−1NRTACC = 1 - \frac{1}{N}R_T, where RTR_T is the target regret.

MetricsAcc Per Class ↑Acc ↑
Test AccuracyaeroplanebicyclebuscarknifemotorcycleplantskateboardtraintruckAverage
LINUCB ([20])N/AN/AN/AN/AN/AN/AN/AN/AN/AN/AN/A
LINUCB-P ([20])0.1430±0.020.2600±0.030.0810±0.010.0240±0.010.1070±0.020.0570±0.010.1060±0.010.0390±0.010.0960±0.020.1260±0.030.1039±0.02
NLINUCB ([49])0.0350±0.000.1210±0.020.0510±0.010.0870±0.020.0270±0.000.0240±0.010.2720±0.040.0090±0.000.1730±0.020.0309±0.010.1108±0.02
NLINUCB-P ([49])0.0000±0.000.0000±0.000.0000±0.000.2990±0.030.7000±0.020.0000±0.000.0000±0.000.0000±0.000.0000±0.000.1160±0.010.1115±0.01
DABAND (OURS)0.5501±0.030.6418±0.030.5844±0.030.2346±0.010.0410±0.010.7580±0.040.5689±0.040.1345±0.020.1118±0.020.3126±0.020.3923±0.03
OURS VS. NLINUCB+0.5151+0.5208+0.5334+0.1476+0.0140+0.7340+0.2969+0.1255-0.0612+0.2817+0.2815

Table 3. Accuracy on the target domain for the S2RDA49 dataset. Note that in this zero-shot target regret setting, the accuracy ACC=1−1NRTACC = 1 - \frac{1}{N}R_T, where RTR_T is the target regret.

Performance Analysis of LinUCB and NLinUCB. Our target regret bound in Eqn. (5) consists of 5 terms, i.e., source regret, regression error, data divergence, constant, and predicted reward. After applying linear contextual bandits on the source domain, the source regret enjoys a sub-linear rate. However, in the linear model, the encoder ϕ^(x)=x\hat{\phi}(x) = x is an identity function and therefore fail to align source-domain and target-domain contexts, leading to an unbounded, large data divergence term in Eqn. (5). This is the main reason why linear contextual bandits, such as LinUCB, perform poorly in the cross-domain contextual bandit setting.

Note that even contextual bandits using a nonlinear encoder, e.g., Neural-LinUCB (NLinUCB), fail in this case because their encoders ϕ^(x)\hat{\phi}(x) are not learned to align source-domain and target-domain contexts; they therefore will still lead to an unbounded, large data divergence term in Eqn. (5), and subsequently poor performance in the cross-domain contextual bandit setting.

Continued Training in Target Domains and Cumulative Regret. Besides zero-shot target regret, we also evaluate how different methods perform when one continues their training in the high-cost target domain and starts to collect feedback (reward) after the model is trained in the source domain. A good cross-domain bandit model can provide a head-start in this setting, therefore enjoying significantly lower cumulative regret in the target domain and saving substantial cost in collecting feedback in the high-cost target domain.

Figure 1 shows the cumulative regrets of different methods on DIGIT, VisDA17 and S2RDA49, respectively. We show results averaged over five runs with different random seeds. Our DABand consistently surpasses all baselines for all rounds in terms of both cumulative regrets and their increase rates, demonstrating DABand’s potential to significantly reduce the cost in high-cost target domains.

Figure 1. The cumulative regrets of different methods for 1,920 rounds on DIGIT (left), VisDA17 (middle), and S2RDA49 (right). Results are averaged over 5 runs. LinUCB is not reported for VisDA17 and S2RDA49 due to out-of-memory issues.

Ablation Study. To verify the effectiveness of each term in our DABand’s regret bound (Theorem 3.1), we report the accuracy of our proposed DABand after removing the Regression Error term and/or the Predicted Reward term (during training) in Table 4 for DIGIT, VisDA17 and S2RDA49.

Results on all datasets show that removing either term will lead to performance drop. For example, in VisDA17, our full DABand achieves accuracy of 0.4642; the accuracy drops to 0.4238 after removing the Regression Error (R) term, and further drops to 0.4088 after removing the Predicted Reward (P) term. These results verify the effectiveness of these two terms in our DABand.

Conclusion

In this paper, we introduce a novel domain adaptive neural contextual bandit algorithm, DABand, which adeptly combines effective exploration with representation alignment, utilizing unlabeled data from both source and target domains. Our theoretical analysis demonstrates that DABand can attain a sub-linear regret bound within the target domain. This marks the first regret analysis for domain adaptation in contextual bandit problems incorporating deep representation, shallow exploration, and adversarial alignment. We show that all these elements are instrumental in the domain adaptive bandit setting on real-world datasets. Moving forward, interesting future research could be to uncover more innovative techniques for aligning the source and target domains (mentioned in Appendix § 13), particularly within the constraints of: (1) bandit settings, as opposed to classification settings, and (2) the high-dimensional and dense nature of the target domain, contrasted with the sparse and simplistic nature of the source domain. It would also be interesting to explore the effect of improved uncertainty quantification [48], potentially via Bayesian deep learning [41, 42, 16], on DABand’s performance.

Acknowledgments

We sincerely appreciate the generous support from the reviewers/AC for the constructive comments to improve the paper. ZW and XH are partially sponsored by a subcontract of NSF grant 2229876, the A. Russell Chandler III Professorship at Georgia Institute of Technology, and NIH-sponsored Georgia Clinical & Translational Science Alliance. HW is partially supported by the Microsoft Research AI & Society Fellowship, NSF Grant IIS-2127918, NSF CAREER Award IIS-2340125, NIH Grant 1R01CA297832, and the Amazon Faculty Research Award. This research is also supported by NSF National Artificial Intelligence Research Resource (NAIRR) Pilot and the Frontera supercomputer, funded by the National Science Foundation (award NSF-OAC 1818253) and hosted at the Texas Advanced Computing Center (TACC) at The University of Texas at Austin. Finally, we extend our gratitude to the Center for AI Safety (CAIS) for providing the essential computing resources that made this work possible.

Proofs

Proofs for Lemmas

Lemma A.1. For any hypotheses h,h′∈Hh, h' \in\mathcal{H} and a∈Aa \in\mathcal{A} is selected by either hh or h′h',

∣ϵS(h,h′)−ϵT(h,h′)∣≤N2d^HΔH(S,T).\left|\epsilon_{S}(h,h')-\epsilon_{T}(h,h')\right| \le\frac{N}{2}\widehat{d}_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{S},\mathcal{T}).

Proof. By definition of HΔH\mathcal{H}\Delta\mathcal{H} (as defined in Def. 3.6), we have

d^HΔH(S,T)=Def. 3.62sup⁡h,h′∈H∣Ex∼S[∣h(x)−h′(x)∣]−Ex∼T[∣h(x)−h′(x)∣]∣=2Nsup⁡h,h′∈H∣N⋅Ex∼S[∣h(x)−h′(x)∣]−N⋅Ex∼T[∣h(x)−h′(x)∣]∣=Def. 3.32Nsup⁡h,h′∈H∣ϵS(h,h′)−ϵT(h,h′)∣≥2N∣ϵS(h,h′)−ϵT(h,h′)∣.\begin{aligned} \widehat{d}_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{S},\mathcal{T}) &\mathrel{\overset{\text{Def. }3.6}{=}} 2\sup_{h,h'\in\mathcal{H}}\left|\mathbb{E}_{x\sim\mathcal{S}}[|h(x)-h'(x)|]-\mathbb{E}_{x\sim\mathcal{T}}[|h(x)-h'(x)|]\right|\\ &= \frac{2}{N}\sup_{h,h'\in\mathcal{H}}\left|N\cdot\mathbb{E}_{x\sim\mathcal{S}}[|h(x)-h'(x)|]-N\cdot\mathbb{E}_{x\sim\mathcal{T}}[|h(x)-h'(x)|]\right|\\ &\mathrel{\overset{\text{Def. }3.3}{=}} \frac{2}{N}\sup_{h,h'\in\mathcal{H}}\left|\epsilon_{S}(h,h')-\epsilon_{T}(h,h')\right|\\ &\ge\frac{2}{N}\left|\epsilon_{S}(h,h')-\epsilon_{T}(h,h')\right|. \end{aligned}

□

Lemma A.2. Given three hypotheses h1,h2,h3∈Hh_1,h_2,h_3\in\mathcal{H}, we have the following triangle inequality:

ϵD(h1,h2)≤ϵD(h1,h3)+ϵD(h2,h3).(12)\epsilon_{D}(h_1,h_2)\le\epsilon_{D}(h_1,h_3)+\epsilon_{D}(h_2,h_3). \tag*{(12)}

Proof. The hypothesis difference

ϵD(h1,h2)=∑i=1N∣h1(xiD)−h2(xiD)∣=Def. 3.2∑i=1N∣h1(xiD)−h2(xiD)+h3(xiD)−h3(xiD)∣≤∑i=1N∣h1(xiD)−h3(xiD)∣+∑i=1N∣h3(xiD)−h2(xiD)∣=ϵS(h1,h3)+ϵS(h2,h3).\begin{aligned} \epsilon_{D}(h_1,h_2) &= \sum_{i=1}^{N}\left|h_1(x_i^D)-h_2(x_i^D)\right|\\ &\mathrel{\overset{\text{Def. }3.2}{=}} \sum_{i=1}^{N}\left|h_1(x_i^D)-h_2(x_i^D)+h_3(x_i^D)-h_3(x_i^D)\right|\\ &\le\sum_{i=1}^{N}\left|h_1(x_i^D)-h_3(x_i^D)\right|+\sum_{i=1}^{N}\left|h_3(x_i^D)-h_2(x_i^D)\right|\\ &= \epsilon_{S}(h_1,h_3)+\epsilon_{S}(h_2,h_3). \end{aligned}

□

Lemma A.3. Let H\mathcal{H} be a hypothesis space. Then for every h∈Hh\in\mathcal{H}:

ϵT(h)≤ϵS(h)+N2d^HΔH(S,T)+ψ.(13)\epsilon_{T}(h)\le\epsilon_{S}(h)+\frac{N}{2}\widehat{d}_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{S},\mathcal{T})+\psi. \tag*{(13)}

where ψ\psi is defined in Definition 3.7.

Proof. By definition in Definition 3.3 in main paper, we start from ϵT(h)=ϵT(h,fT)\epsilon_T(h)=\epsilon_T(h,f_T), then we have

ϵT(h)=ϵT(fT,h)≤Lm. A.2ϵT(fT,h∗)+ϵT(h,h∗)≤ϵT(h∗)+ϵS(h,h∗)+∣ϵT(h,h∗)−ϵS(h,h∗)∣≤Lm. A.1ϵT(h∗)+ϵS(h,h∗)+N2d^HΔH(S,T)≤ϵT(h∗)+ϵS(h)+ϵS(h∗)+N2d^HΔH(S,T)=ϵS(h)+ψ+N2d^HΔH(S,T).\begin{aligned} \epsilon_T(h) &= \epsilon_T(f_T,h)\\ &\mathrel{\overset{\mathrm{Lm.}\ A.2}{\le}} \epsilon_T(f_T,h^*)+\epsilon_T(h,h^*)\\ &\le\epsilon_T(h^*)+\epsilon_S(h,h^*)+\left|\epsilon_T(h,h^*)-\epsilon_S(h,h^*)\right|\\ &\mathrel{\overset{\mathrm{Lm.}\ A.1}{\le}} \epsilon_T(h^*)+\epsilon_S(h,h^*)+\frac{N}{2}\widehat{d}_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{S},\mathcal{T})\\ &\le\epsilon_T(h^*)+\epsilon_S(h)+\epsilon_S(h^*)+\frac{N}{2}\widehat{d}_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{S},\mathcal{T})\\ &= \epsilon_S(h)+\psi+\frac{N}{2}\widehat{d}_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{S},\mathcal{T}). \end{aligned}

□

Lemma A.4. Denoting the contexts from the target domain T\mathcal{T} as {xt,j}j∈[N],a∈[K]\{x_{t,j}\}_{j\in[N],a\in[K]} and the associated ground-truth action as {ai∗}i=1N\{a^{*}_{i}\}_{i=1}^{N}, the estimated regret (i.e., using the trained model ϕ^\hat{\phi} from the source domain) for the target domain is

∑i=1N(∣⟨θ^,ϕ^(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,a^iT⟩∣)≤∑i=1N(∣⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1Nα⋅κiT.\sum_{i=1}^{N}\left(\left|\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\rangle-\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}}\right\rangle\right|\right)\leq\sum_{i=1}^{N}\left(\left|\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\rangle\right|\right)+\sum_{i=1}^{N}\alpha\cdot\kappa^{\mathcal{T}}_{i}.

where κiT\kappa^{\mathcal{T}}_{i} is max⁡{2∥ϕ^(xi,a^iT)∥Ai−1−1,∥ϕ^(xi,ai∗T)∥Ai−1−1+∥ϕ^(xi,a^iT)∥Ai−1−1}\max\left\{2\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}},\left\|\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\|_{A^{-1}_{i-1}}+\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}}\right\}.

Proof. By definition, we have

∑i=1N(∣⟨θ^,ϕ^(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,a^iT)⟩∣)=∑i=1N(∣(⟨θ^,ϕ^(xi,ai∗T)⟩+α∥ϕ^(xi,ai∗T)∥Ai−1−1)−(⟨θ^,ϕ^(xi,a^iT)⟩+α∥ϕ^(xi,a^iT)∥Ai−1−1)∣)\begin{aligned} \sum_{i=1}^{N}\left(\left|\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\rangle-\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\rangle\right|\right) \\ &=\sum_{i=1}^{N}\left(\left|\left(\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\rangle+\alpha\left\|\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\|_{A^{-1}_{i-1}}\right)-\left(\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\rangle+\alpha\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}}\right)\right|\right) \end{aligned}
+∑i=1Nα(∥ϕ^(xi,a^iT)∥Ai−1−1−∥ϕ^(xi,ai∗T)∥Ai−1−1)≤∑i=1N(∣(⟨θ^,ϕ^(xi,ai∗T)⟩+α∥ϕ^(xi,ai∗T)∥Ai−1−1)−(⟨θ^,ϕ^(xi,a^iT)⟩+α∥ϕ^(xi,a^iT)∥Ai−1−1)∣)\begin{aligned} \quad+\sum_{i=1}^{N}\alpha\left(\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}}-\left\|\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\|_{A^{-1}_{i-1}}\right) \\ &\leq\sum_{i=1}^{N}\left(\left|\left(\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\rangle+\alpha\left\|\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\|_{A^{-1}_{i-1}}\right)-\left(\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\rangle+\alpha\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}}\right)\right|\right) \end{aligned}
+∑i=1Nαmax⁡{∥ϕ^(xi,a^iT)∥Ai−1−1,∥ϕ^(xi,ai∗T)∥Ai−1−1}(14)\quad+\sum_{i=1}^{N}\alpha\max\left\{\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}},\left\|\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\|_{A^{-1}_{i-1}}\right\} \tag*{(14)}
≤∑i=1N(∣(⟨θ^,ϕ^(xi,a^iT)⟩+α∥ϕ^(xi,a^iT)∥Ai−1−1)∣)+∑i=1Nαmax⁡{∥ϕ^(xi,a^iT)∥Ai−1−1,∥ϕ^(xi,ai∗T)∥Ai−1−1}≤∑i=1N(∣⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1Nα⋅max⁡{2∥ϕ^(xi,a^iT)∥Ai−1−1,∥ϕ^(xi,ai∗T)∥Ai−1−1+∥ϕ^(xi,a^iT)∥Ai−1−1}(15)\begin{aligned} \leq\sum_{i=1}^{N}\left(\left|\left(\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\rangle+\alpha\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}}\right)\right|\right)+\sum_{i=1}^{N}\alpha\max\left\{\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}},\left\|\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\|_{A^{-1}_{i-1}}\right\} \\ &\leq\sum_{i=1}^{N}\left(\left|\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\rangle\right|\right)+\sum_{i=1}^{N}\alpha\cdot\max\left\{2\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}},\left\|\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\|_{A^{-1}_{i-1}}+\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}}\right\} \tag*{(15)} \end{aligned}
:=∑i=1N(∣⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1Nα⋅κiT.(16):=\sum_{i=1}^{N}\left(\left|\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\rangle\right|\right)+\sum_{i=1}^{N}\alpha\cdot\kappa^{\mathcal{T}}_{i}. \tag*{(16)}

where the inequality in Eqn. (14) is based on the fact that for any a,b>0a,b>0, ∣a−b∣≤max⁡{a,b}\left|a-b\right|\leq\max\{a,b\}. The inequality in Eqn. (15) is derived from the definition of a^i\hat{a}_{i}, since a^i\hat{a}_{i} is selected from the maximum value of (⟨θ^,ϕ^(xi,aT)⟩+α∥ϕ^(xi,aT)∥Ai−1−1)\left(\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,a})\right\rangle+\alpha\left\|\hat{\phi}(x^{\mathcal{T}}_{i,a})\right\|_{A^{-1}_{i-1}}\right), therefore we guarantee that

(⟨θ^,ϕ^(xi,a^iT)⟩+α∥ϕ^(xi,a^iT)∥Ai−1−1)≥(⟨θ^,ϕ^(xi,ai∗T)⟩+α∥ϕ^(xi,ai∗T)∥Ai−1−1).\left(\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\rangle+\alpha\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}}\right)\geq\left(\left\langle\hat{\theta},\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\rangle+\alpha\left\|\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\|_{A^{-1}_{i-1}}\right).

In addition, Eqn. (16) holds by definition as we define κiT\kappa^{\mathcal{T}}_{i} as max⁡{2∥ϕ^(xi,a^iT)∥Ai−1−1,∥ϕ^(xi,ai∗T)∥Ai−1−1+∥ϕ^(xi,a^iT)∥Ai−1−1}\max\left\{2\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}},\left\|\hat{\phi}(x^{\mathcal{T}}_{i,a^{*}_{i}})\right\|_{A^{-1}_{i-1}}+\left\|\hat{\phi}(x^{\mathcal{T}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}}\right\}.

Next, we discuss the property of κiT\kappa^{\mathcal{T}}_{i} in the following lemma.

Lemma A.5. For any arbitrary domain D\mathcal{D}, we denote

κiD=max⁡{2∥ϕ^(xi,a^iD)∥Ai−1−1,∥ϕ^(xi,ai∗D)∥Ai−1−1+∥ϕ^(xi,a^iD)∥Ai−1−1}.\kappa^{\mathcal{D}}_{i}=\max\left\{2\left\|\hat{\phi}(x^{\mathcal{D}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}},\left\|\hat{\phi}(x^{\mathcal{D}}_{i,a^{*}_{i}})\right\|_{A^{-1}_{i-1}}+\left\|\hat{\phi}(x^{\mathcal{D}}_{i,\hat{a}_{i}})\right\|_{A^{-1}_{i-1}}\right\}.

Then, by our DABand algorithm, as the time step i→∞i\to\infty, we have κiD→0\kappa^{\mathcal{D}}_{i}\to0.

Proof. By definition of AA, its singular value increase as ii increase, which means that the singular values in its inverse matrix, A−1A^{-1}, will decrease to 00. Therefore, for any new array uu, ∥u∥A−1=\left\|u\right\|_{A^{-1}}= uA−1uT→0\sqrt{uA^{-1}u^{\mathsf{T}}} \to0. Then when i→∞i \to\infty, we have

lim⁡i→∞κiD=max⁡{2∥ϕ^(xi,a^iD)∥Ai−1−1,∥ϕ^(xi,ai∗D)∥Ai−1−1+∥ϕ^(xi,a^iD)∥Ai−1−1}\lim_{i\to\infty}\kappa_i^{\mathcal{D}}=\max\left\{2\left\|\hat{\phi}(x_{i,\hat{a}_i}^{\mathcal{D}})\right\|_{\mathcal{A}_{i-1}^{-1}},\left\|\hat{\phi}(x_{i,a_i^*}^{\mathcal{D}})\right\|_{\mathcal{A}_{i-1}^{-1}}+\left\|\hat{\phi}(x_{i,\hat{a}_i}^{\mathcal{D}})\right\|_{\mathcal{A}_{i-1}^{-1}}\right\}

≈0\approx0.

Lemma A.6. Denoting the contexts from the source domain S\mathcal{S} as {xt,a}t∈[N],a∈[K]\{x_{t,a}\}_{t\in[N],a\in[K]}, the associated ground-truth action as {ai∗}i=1N\{a_i^*\}_{i=1}^N, we have the following inequality:

∑i=1N(∣⟨θ^,ϕ^(xi,ai∗S)⟩−⟨θ∗,ϕ∗(xi,ai∗S)⟩∣)≤ϵS(h)+∑i=1N1[ai∗≠a^i](∣⟨θ^,ϕ^(xi,a^iS)⟩∣+α⋅κiS),\sum_{i=1}^{N}\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle-\left\langle\theta^*,\phi^*(x_{i,a_i^*}^{\mathcal{S}})\right\rangle\right|\right)\le\epsilon_{\mathcal{S}}(h)+\sum_{i=1}^{N}\mathbb{1}[a_i^*\ne\hat{a}_i]\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{\mathcal{S}})\right\rangle\right|+\alpha\cdot\kappa_i^{\mathcal{S}}\right),

where κiS\kappa_i^{\mathcal{S}} is defined in Lemma A.5.

Proof. By definition, we have

∑i=1N(∣⟨θ^,ϕ^(xi,ai∗S)⟩−⟨θ∗,ϕ∗(xi,ai∗S)⟩∣)=∑i=1N[1[ai∗=a^i](⟨θ^,ϕ^(xi,ai∗S)⟩−⟨θ∗,ϕ∗(xi,ai∗S)⟩)+1[ai∗≠a^i](⟨θ^,ϕ^(xi,ai∗S)⟩−⟨θ∗,ϕ∗(xi,ai∗S)⟩)]≤∑i=1N[1[ai∗=a^i](⟨θ^,ϕ^(xi,ai∗S)⟩−1)+1[ai∗≠a^i](⟨θ^,ϕ^(xi,ai∗S)⟩−0)]≤∑i=1N[1[ai∗=a^i](⟨θ^,ϕ^(xi,ai∗S)⟩−1)+1[ai∗≠a^i](⟨θ^,ϕ^(xi,ai∗S)⟩+⟨θ^,ϕ^(xi,ai∗S)⟩−⟨θ^,ϕ^(xi,ai∗S)⟩)]≤∑i=1N[1[ai∗=a^i](⟨θ^,ϕ^(xi,ai∗S)⟩−1)+1[ai∗≠a^i](⟨θ^,ϕ^(xi,ai∗S)⟩−0)+1[ai∗≠a^i](⟨θ^,ϕ∗(xi,ai∗S)⟩−⟨θ^,ϕ^(xi,ai∗S)⟩)]\begin{aligned} \sum_{i=1}^{N}\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle-\left\langle\theta^*,\phi^*(x_{i,a_i^*}^{\mathcal{S}})\right\rangle\right|\right) \\ &=\sum_{i=1}^{N}\left[\mathbb{1}[a_i^*=\hat{a}_i]\left(\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle-\left\langle\theta^*,\phi^*(x_{i,a_i^*}^{\mathcal{S}})\right\rangle\right)+\mathbb{1}[a_i^*\ne\hat{a}_i]\left(\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle-\left\langle\theta^*,\phi^*(x_{i,a_i^*}^{\mathcal{S}})\right\rangle\right)\right] \\ &\le\sum_{i=1}^{N}\left[\mathbb{1}[a_i^*=\hat{a}_i]\left(\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle-1\right)+\mathbb{1}[a_i^*\ne\hat{a}_i]\left(\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle-0\right)\right] \\ &\le\sum_{i=1}^{N}\left[\mathbb{1}[a_i^*=\hat{a}_i]\left(\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle-1\right)+\mathbb{1}[a_i^*\ne\hat{a}_i]\left(\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle+\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle-\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle\right)\right] \\ &\le\sum_{i=1}^{N}\left[\mathbb{1}[a_i^*=\hat{a}_i]\left(\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle-1\right)+\mathbb{1}[a_i^*\ne\hat{a}_i]\left(\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle-0\right)+\mathbb{1}[a_i^*\ne\hat{a}_i]\left(\left\langle\hat{\theta},\phi^*(x_{i,a_i^*}^{\mathcal{S}})\right\rangle-\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle\right)\right] \end{aligned}

L.m. A.4

≤∑i=1N(∣⟨θ∗,ϕ∗(xi,ai∗S)⟩−⟨θ^,ϕ^(xi,ai∗S)⟩∣)+∑i=1N1[ai∗≠a^i](⟨θ^,ϕ(xi,ai∗S)⟩+α⋅κiS)≡ϵS(h)+∑i=1N1[ai∗≠a^i](⟨θ^,ϕ^(xi,ai∗S)⟩+α⋅κiS).\begin{aligned} \le\sum_{i=1}^{N}\left(\left|\left\langle\theta^*,\phi^*(x_{i,a_i^*}^{\mathcal{S}})\right\rangle-\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle\right|\right)+\sum_{i=1}^{N}\mathbb{1}[a_i^*\ne\hat{a}_i]\left(\left\langle\hat{\theta},\phi(x_{i,a_i^*}^{\mathcal{S}})\right\rangle+\alpha\cdot\kappa_i^{\mathcal{S}}\right) \\ &\equiv\epsilon_{\mathcal{S}}(h)+\sum_{i=1}^{N}\mathbb{1}[a_i^*\ne\hat{a}_i]\left(\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle+\alpha\cdot\kappa_i^{\mathcal{S}}\right). \end{aligned}

Proof for the Target Regret Bound

Theorem A.1 (Target Regret Bound). Denoting the contexts from the source domain S\mathcal{S} as {xt,aS}t∈[N],a∈[K]\{x_{t,a}^{\mathcal{S}}\}_{t\in[N],a\in[K]}, the associated ground-truth action as {ai∗}i=1N\{a_i^*\}_{i=1}^N, and the contexts from the target domain T\mathcal{T} as {xt,aT}t∈[N],a∈[K]\{x_{t,a}^{\mathcal{T}}\}_{t\in[N],a\in[K]}, the upper bound for our target regret RTR_{\mathcal{T}} is

RT≜∑i=1N(∣⟨θT∗,ϕT∗(xi,ai∗T)⟩−⟨θT∗,ϕT∗(xi,a^iT)⟩∣)R_{\mathcal{T}}\triangleq\sum_{i=1}^{N}\left(\left|\left\langle\theta_{\mathcal{T}}^*,\phi_{\mathcal{T}}^*(x_{i,a_i^*}^{\mathcal{T}})\right\rangle-\left\langle\theta_{\mathcal{T}}^*,\phi_{\mathcal{T}}^*(x_{i,\hat{a}_i}^{\mathcal{T}})\right\rangle\right|\right)
≤RS+2⋅ϵS(h)+N⋅d^HΔH(S,T)+ψ+C\le R_{\mathcal{S}}+2\cdot\epsilon_{\mathcal{S}}(h)+N\cdot\hat{d}_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{S},\mathcal{T})+\psi+C
+∑i=1N(∣⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1N1[ai∗≠a^i](∣⟨θ^,ϕ^(xi,ai∗S)⟩∣).+\sum_{i=1}^{N}\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_i}^{\mathcal{T}})\right\rangle\right|\right)+\sum_{i=1}^{N}\mathbb{1}[a_i^*\ne\hat{a}_i]\left(\left|\left\langle\hat{\theta},\hat{\phi}(x_{i,a_i^*}^{\mathcal{S}})\right\rangle\right|\right).

where RSR_{\mathcal{S}}, ϵS(h)\epsilon_{\mathcal{S}}(h), and d^HΔH\hat{d}_{\mathcal{H}\Delta\mathcal{H}} are the source regret, source-domain error, and HΔH\mathcal{H}\Delta\mathcal{H} divergence defined in Definition 3.4, Definition 3.3, and Definition 3.6, respectively. ψ\psi is a constant independent to the problem and CC is a constant which can be ignored.

Proof. Please check the full proof in the next page.

By the definition of the Target Regret Bound, we have

RT=∑i=1N(∣⟨θ∗,ϕ∗(xi,ai∗T)⟩−⟨θ∗,ϕ∗(xi,a^iT)⟩∣)=∑i=1N(∣⟨θ∗,ϕ∗(xi,ai∗T)⟩−⟨θ∗,ϕ∗(xi,a^iT)⟩+⟨θ^,ϕ^(xi,a^iT)⟩−⟨θ^,ϕ^(xi,a^iT)⟩∣)≤∑i=1N(∣⟨θ∗,ϕ∗(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1N(∣⟨θ∗,ϕ∗(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,a^iT)⟩∣)≤εT(h)+∑i=1N(∣⟨θ∗,ϕ∗(xi,aiT)⟩−⟨θ^,ϕ^(xi,a^iT)⟩+⟨θ^,ϕ^(xi,a^iT)⟩−⟨θ^,ϕ^(xi,a^iT)⟩∣)≤εT(h)+∑i=1N(∣⟨θ^,ϕ^(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1N(∣⟨θ∗,ϕ∗(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,ai∗T)⟩∣)≤εT(h)+∑i=1N(∣⟨θ^,ϕ^(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1N(∣⟨θ∗,ϕ∗(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,ai∗T)⟩∣)+∑i=1N(∣⟨θ∗,ϕ∗(xi,aiS)⟩−⟨θ^,ϕ^(xi,aiS)⟩∣)−∑i=1N(∣⟨θ∗,ϕ∗(xi,ai∗S)⟩−⟨θ^,ϕ^(xi,ai∗S)⟩∣)≤εT(h)+∑i=1N(∣⟨θ^,ϕ^(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1N(∣⟨θ∗,ϕ∗(xi,aiS)⟩−⟨θ^,ϕ^(xi,aiS)⟩∣)+∣∑i=1N(∣⟨θ∗,ϕ∗(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,ai∗T)⟩∣)−∑i=1N(∣⟨θ∗,ϕ∗(xi,ai∗S)⟩−⟨θ^,ϕ^(xi,ai∗S)⟩∣)∣≤εT(h)+N2d^HΔH(S,T)+∑i=1N(∣⟨θ^,ϕ^(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1N(∣⟨θ∗,ϕ∗(xi,ai∗S)⟩−⟨θ^,ϕ^(xi,ai∗S)⟩+⟨θ∗,ϕ∗(xi,a^iS)⟩−⟨θ∗,ϕ∗(xi,aiS)⟩∣)L.m. A.3≤εS(h)+ψ+N2d^HΔH(S,T)+N2d^HΔH(S,T)+∑i=1N(∣⟨θ^,ϕ^(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1N(∣⟨θ∗,ϕ∗(xi,ai∗S)⟩−⟨θ∗,ϕ∗(xi,a^iS)⟩∣)+∑i=1N(∣⟨θ^,ϕ^(xi,ai∗S)⟩−⟨θ∗,ϕ∗(xi,aiS)⟩∣)L.m. A.4≤εS(h)+ψ+Nd^HΔH(S,T)+∑i=1N(∣⟨θ^,ϕ^(xi,ai∗T)⟩−⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1Nα⋅κiT+RS+∑i=1N(∣⟨θ^,ϕ^(xi,ai∗S)⟩−⟨θ∗,ϕ∗(xi,ai∗S)⟩∣)L.m. A.6≤εS(h)+ψ+Nd^HΔH(S,T)+∑i=1N(∣⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1Nα⋅κiT+RS+εS(h)+∑i=1N1[ai∗≠a^i](∣⟨θ^,ϕ^(xi,aiS)⟩∣+α⋅κiS)L.m. A.5≤RS⏟Source Regret+2⋅εS(h)⏟Regression Error+N⋅d^HΔH(S,T)⏟Data Divergence+ψ+C⏟Constant+∑i=1N(∣⟨θ^,ϕ^(xi,aiT)⟩∣)+1[ai∗≠a^i](∑i=1N(∣⟨θ^,ϕ^(xi,aiS)⟩∣))⏟Predicted Rewards,\begin{aligned} R_T &= \sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i^*}^{T})\rangle-\langle\theta^*,\phi^*(x_{i,\hat a_i}^{T})\rangle\right|\right)\\ &= \sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i^*}^{T})\rangle-\langle\theta^*,\phi^*(x_{i,\hat a_i}^{T})\rangle+\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle\right|\right)\\ &\leq\sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i^*}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle\right|\right)+\sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i^*}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle\right|\right)\\ &\leq\varepsilon_T(h)+\sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle+\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle\right|\right)\\ &\leq\varepsilon_T(h)+\sum_{i=1}^{N}\left(\left|\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle\right|\right)+\sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i^*}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{T})\rangle\right|\right)\\ &\leq\varepsilon_T(h)+\sum_{i=1}^{N}\left(\left|\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle\right|\right)+\sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i^*}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{T})\rangle\right|\right)\\ &\quad+\sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i}^{S})\rangle-\langle\hat\theta,\hat\phi(x_{i,a_i}^{S})\rangle\right|\right)-\sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i^*}^{S})\rangle-\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{S})\rangle\right|\right)\\ &\leq\varepsilon_T(h)+\sum_{i=1}^{N}\left(\left|\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle\right|\right)+\sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i}^{S})\rangle-\langle\hat\theta,\hat\phi(x_{i,a_i}^{S})\rangle\right|\right)\\ &\quad+\left|\sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i^*}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{T})\rangle\right|\right)-\sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i^*}^{S})\rangle-\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{S})\rangle\right|\right)\right|\\ &\leq\varepsilon_T(h)+\frac{N}{2}\hat d_{\mathcal H\Delta\mathcal H}(S,T)+\sum_{i=1}^{N}\left(\left|\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle\right|\right)\\ &\quad+\sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i^*}^{S})\rangle-\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{S})\rangle+\langle\theta^*,\phi^*(x_{i,\hat a_i}^{S})\rangle-\langle\theta^*,\phi^*(x_{i,a_i}^{S})\rangle\right|\right)\\ \text{L.m. A.3}\quad &\leq\varepsilon_S(h)+\psi+\frac{N}{2}\hat d_{\mathcal H\Delta\mathcal H}(S,T)+\frac{N}{2}\hat d_{\mathcal H\Delta\mathcal H}(S,T)+\sum_{i=1}^{N}\left(\left|\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle\right|\right)\\ &\quad+\sum_{i=1}^{N}\left(\left|\langle\theta^*,\phi^*(x_{i,a_i^*}^{S})\rangle-\langle\theta^*,\phi^*(x_{i,\hat a_i}^{S})\rangle\right|\right)+\sum_{i=1}^{N}\left(\left|\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{S})\rangle-\langle\theta^*,\phi^*(x_{i,a_i}^{S})\rangle\right|\right)\\ \text{L.m. A.4}\quad &\leq\varepsilon_S(h)+\psi+N\hat d_{\mathcal H\Delta\mathcal H}(S,T)+\sum_{i=1}^{N}\left(\left|\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{T})\rangle-\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle\right|\right)+\sum_{i=1}^{N}\alpha\cdot\kappa_i^T\\ &\quad+R_S+\sum_{i=1}^{N}\left(\left|\langle\hat\theta,\hat\phi(x_{i,a_i^*}^{S})\rangle-\langle\theta^*,\phi^*(x_{i,a_i^*}^{S})\rangle\right|\right)\\ \text{L.m. A.6}\quad &\leq\varepsilon_S(h)+\psi+N\hat d_{\mathcal H\Delta\mathcal H}(S,T)+\sum_{i=1}^{N}\left(\left|\langle\hat\theta,\hat\phi(x_{i,\hat a_i}^{T})\rangle\right|\right)+\sum_{i=1}^{N}\alpha\cdot\kappa_i^T\\ &\quad+R_S+\varepsilon_S(h)+\sum_{i=1}^{N}\mathbb{1}[a_i^*\ne\hat a_i]\left(\left|\langle\hat\theta,\hat\phi(x_{i,a_i}^{S})\rangle\right|+\alpha\cdot\kappa_i^S\right)\\ \text{L.m. A.5}\quad &\leq\underbrace{R_S}_{\text{Source Regret}}+\underbrace{2\cdot\varepsilon_S(h)}_{\text{Regression Error}}+\underbrace{N\cdot\hat d_{\mathcal H\Delta\mathcal H}(S,T)}_{\text{Data Divergence}}+\underbrace{\psi+C}_{\text{Constant}}\\ &\quad+\underbrace{\sum_{i=1}^{N}\left(\left|\langle\hat\theta,\hat\phi(x_{i,a_i}^{T})\rangle\right|\right)+\mathbb{1}[a_i^*\ne\hat a_i]\left(\sum_{i=1}^{N}\left(\left|\langle\hat\theta,\hat\phi(x_{i,a_i}^{S})\rangle\right|\right)\right)}_{\text{Predicted Rewards}}, \end{aligned}

where C=∑i=1Nα⋅κiT+1[ai∗≠a^i](∑i=1Nα⋅κiS)C = \sum_{i=1}^{N} \alpha\cdot\kappa_i^{T} + 1[a_i^{*} \ne\hat{a}_i]\left(\sum_{i=1}^{N} \alpha\cdot\kappa_i^{S}\right) is a small number which can be ignored as i→∞i \to\infty (i.e., see Lemma A.5).

Datasets

To evaluate the effectiveness of our DABand, for each dataset, we treat the “easy” domain as the low-cost source domain and the more challenging one as the high-cost target domain. This allows us to demonstrate the efficacy of our method by adapting from a simpler to a more complex domain within a contextual bandit setting.

DIGIT. Our DIGIT dataset consists of MNIST and MNIST-M. MNIST is a gray-scale hand-written digit dataset, while MNIST-M [12] features color digits. In this paper, MNIST serves as our source domain, and MNIST-M as our target domain, offering a more challenging adaptation path. The MNIST digits are gray-scale and uniform in size, aspect ratio, and intensity range, in stark contrast to the colorful and varied digits of MNIST-M. Therefore, adapting from MNIST to MNIST-M presents a greater challenge than adapting from MNIST-M to MNIST, where the target domain is less complex.

VisDA17. The VisDA-2017 image classification challenge [29] addresses a domain adaptation problem across 12 classes, involving three distinct datasets. The training set consists of 3D rendering images, whereas the validation and test sets feature real images from the COCO [22] and YouTube Bounding Boxes [32] datasets, respectively. Ground truth labels were provided only for the training and validation sets. Scores for the test set were calculated by a server operated by the competition organizers. For our purposes, we use the training set as the source domain and the validation set as the target domain.

S2RDA49. The S2RDA49 (Synthetic-to-Real) is a new benchmark dataset [38] constructed in 2023. This dataset contains 49 classes. The source domain (i.e., the synthetic domain) is synthesized by rendering 3D models from ShapeNet [5]. The used 3D models are in the same label space as the target/real domain, and each class has 12K rendered RGB images. The target domain (i.e., the real domain) of S2RDA49 contains 60535 images from 49 classes, collected from the ImageNet validation set [9], ObjectNet [3], VisDA2017 validation set [29], and the web. In this paper, we select the only 10 class that matches the VisDA17 dataset.

Baselines

We compare our DABand with both classic and state-of-the-art contextual bandit algorithms. We start with LinUCB [20], a typical baseline for linear contextual bandit problems. To enable comprehensive comparisons, we extend our evaluation to include LinUCB augmented with pre-processed features using principle component analysis (PCA), referred to as LinUCB-P. In LinUCB-P, we first apply PCA to fit on training data (without labels/feedback) in both source and target domain. Then, with transformed, lower-dimensional context data, LinUCB is performed to see whether this pre-alignment procedure can transfer the knowledge to the target domain. Another pivotal baseline in our study is Neural-LinUCB [49] (NLinUCB), which utilizes several layers of fully connected neural networks to dynamically process the original features at the beginning of every iteration. Similar to LinUCB-P, we introduce a NLinUCB variant that incorporates PCA, i.e., NLinUCB-P. Note that domain adaptation baselines are not applicable to our setting. Specifically, domain adaptation methods only work in offline settings, and assume complete observability of labels in the source domain. In contrast, contextual bandit is an online setting where the oracle is revealed only when correctly predicted. Therefore domain adaptation methods are not applicable to our online bandit settings.

Implementation Details

In this section, we provide detailed insights into the implementation of our approach, applied to two distinct datasets: DIGIT and VisDA17. We use Pytorch to implement our method, and all experiments are run on servers with NVIDIA A5000 GPUs.

DIGIT. Within the DIGIT dataset framework, we use the MNIST dataset as the source domain and MNIST-M as the target domain. To ensure compatibility between the datasets, we standardize the channel size of images in the source domain (cS=1c_S = 1) to align with that in the target domain (cT=3c_T = 3). Each image undergoes normalization and is resized to 28×2828 \times28 pixels with 3 channels to accommodate the format requirements of both domains. Then, an encoder is utilized to diminish the data’s dimensionality to a more manageable latent space. Following this reduction, the data is processed through two fully connected neural network layers, ending in the final latent space necessary for loss computation as delineated in main paper. For the optimal hyperparameters, we set the learning rate to 1×e−51 \times e^{-5}, with λ\lambda is chosen from {1.0,5.0,10.0,15.0,20.0}\{1.0, 5.0, 10.0, 15.0, 20.0\} and kept the same for all experiments. Additionally, we set the exploration rate α\alpha to 0.05.

VisDA17. We use the VisDA17 dataset’s training set as the source domain, with the validation set functioning as the target domain. We adhere to preprocessing steps established by Prabhu et al. [31] to ensure uniformity across domains: Each image is normalized and resized to 224×224224 \times224 pixels with 3 channels, matching the requisite specifications for both source and target domains. For hyperparameters, the learning rate of 1×e−51 \times e^{-5} is applied, with λ\lambda is chosen from {1.0,5.0,10.0,15.0,20.0}\{1.0, 5.0, 10.0, 15.0, 20.0\} and then kept the same for all experiments. We set the exploration rate α\alpha to 0.05.

S2RDA49. We selects 10 classes from the original 49 classes since it matches the target domain samples in VisDA17 [29]. We adhere to preprocessing steps established by Prabhu et al. [31] to ensure uniformity across domains: Each image is normalized and resized to 224×224224 \times224 pixels with 3 channels. For hyperparameters, the learning rate of 1×e−31 \times e^{-3} is applied, with λ\lambda chosen from {1.0,5.0,10.0,15.0,20.0}\{1.0, 5.0, 10.0, 15.0, 20.0\} and then kept the same for all experiments. We set the exploration rate α\alpha to 0.01.

Full Ablation Studies

The full ablation study is shown in Table 5. Furthermore, we ran the corresponding hypothesis tests, and the p values are in the range of (3.201×10−21,1.504×10−2)(3.201 \times10^{-21}, 1.504 \times10^{-2}), much lower than the threshold of 0.05 and therefore verifying the significance of DABand’s performance improvement.

Datasetsw/o R&P&Dw/o R&Pw/o R&Dw/o P&Dw/o Rw/o Pw/o DDABAND (FULL)
DIGIT0.3816±\pm0.040.5676±\pm0.020.3793±\pm0.020.3544±\pm0.010.5682±\pm0.010.5768±\pm0.010.3649±\pm0.020.6002±\pm0.02
VisDA170.1001±\pm0.020.4088±\pm0.030.1010±\pm0.020.0936±\pm0.010.4096±\pm0.010.4304±\pm0.010.1098±\pm0.010.4644±\pm0.03
S2RDA490.1108±\pm0.020.3691±\pm0.020.0918±\pm0.010.1032±\pm0.010.3694±\pm0.030.3719±\pm0.020.1121±\pm0.020.3923±\pm0.03

Table 5. Results of the ablation studies in terms of accuracy (higher is better). Note that the accuracy ACC=1−1NRTACC = 1 - \frac{1}{N}R_T, where RTR_T is the target regret. “R”, “P” and “D” are short for “Regression Error”, “Predicted Reward” and “Data Divergence” (i.e., adversarial loss and the discriminator), respectively.

Limitation

This work contains several limitations. Specifically, we highlight below:

Intuitive Perspective. The bandit algorithm indeed enjoys interpretability [47 a] and the accuracy of its closed-form updates [20], but this is true only because it typically employs a linear model.

Unfortunately, linear models do not work very well for real-world data, which is often high-dimensional. For example, the empirical results for LinUCB and LinUCB-P in Table 1 and Table 2 show that such linear contextual bandit algorithms significantly underperform state-of-the-art neural bandit algorithms, which use back-propagation (similar results are shown in [57, 49]). In summary deep learning models with back-propagation is necessary due to the following reasons:

  • High-Dimensional Data. To enhance performance in real-world high-dimensional data (e.g., images), the integration of deep learning (and back-propagation) is necessary, though this may sacrifice a portion of the algorithm’s explanatory power.

  • Covariate Shift and Aligning Source and Target Domains in the Latent Space. In our settings where there is covariance shift between the source and target domains, a deep (nonlinear) encoder is required to transform the original context into a latent space where source-domain encodings and target-domain encoders can align. This also necessitates deep learning models with back-propagation.

Indeed, there is a trade-off between interpretability and performance. This would certainly be an interesting future direction, but it is out of the scope of this paper.

Theoretical Perspective. Our Theorem 3.1 is sharp, as all the inequalities are based on lemmas in the paper and the Cauchy inequality. Identifying the criteria under which the target regret bound reaches equality as well as how one can achieve it in practice would be interesting future work.

Empirical Perspective. The performance of DABand largely relies on the alignment quality between the source and target domains. Therefore, for two domains that cannot be aligned (for example, most domain adaptation tasks are predefined, and the data for both domains is pre-processed and cleaned, not original real-world data), it is challenging to evaluate how DABand can still transfer knowledge across different domains. Furthermore, it is still unknown whether, if we increase the domain shift in a dataset from another domain, our DABand can still perform well. These issues are beyond the scope of this paper, but they represent interesting areas for future work.

Discussions

Sublinearity for the Target Regret Bound

To see the data divergence term is sub-linear, note that our target regret bound in Eqn. (5) can be divided into two parts:

Figure 2. Sub-linearity for each term in Eqn. (5).

  • Total Source Regret: In Neural-LinUCB (NLinUCB) [49], this is shown to be O(N)O(\sqrt{N}), which is sub-linear.

  • Sum of Other Terms (Including the Data Divergence): These terms can be directly optimized to convergence and minimized to a small value using SGD variants such as Adam. As shown in Corollary 4.2 of the Adam paper [10], the Adam optimizer enjoys an average regret R(N)/NR(N)/N of O(1/N)O(1/\sqrt{N}) (here NN is equivalent to TT in the Adam paper), which leads to a sub-linear total regret O(N)O(\sqrt{N}). Since all other terms are directly optimized using Adam, they also enjoy a sub-linear regret.

Therefore, our target regret bound is also sub-linear. Moreover:

  • Key Difference between the Source Regret and Other Terms. Note that in our target regret bound in Eqn. (5), the source regret term RSR_S increases monotonically with respect to NN and cannot be directly minimized. In contrast, all other terms, e.g., the data divergence term N⋅d^HΔH(S,T)N \cdot\hat{d}_{\mathcal{H}\Delta\mathcal{H}}(S,T) can be directly minimized since all related data is training data and is already known. This is why we minimize the corresponding loss terms like Eqn. (7), Eqn. (8) and Eqn. (9) during training.

  • The data divergence term can be minimized to a small value as N→∞N \to\infty. For example, if the source and target domains are perfectly aligned after our training, the data divergence becomes 0. Therefore, the cumulative sum of the data divergence term over all rounds will be sub-linear.

Difference between Bandit and Classification Settings

In the classification setting, given a sample x\mathbf{x}, a model predicts its label y^\hat{y}. Since its ground-truth label is known, one can directly apply the cross-entropy loss to perform back-propagation and update our model. In contrast, in bandit settings, for each sample x\mathbf{x}, our model predicts an action a^\hat{a} by optimizing the estimated rewards. One then submits this predicted action to the environment and receives feedback that only indicates whether the predicted action a^\hat{a} is the optimal action a∗a^{*} or not. If not, we are informed that our prediction is incorrect, yet we do not receive information on what the correct (optimal) action should be. Moreover, during training, we only train on 300 episodes for DIGIT and VisDA17 and 100 episodes for S2RDA49. Each episode contains 64 samples. This implies that compared with those DA methods which train on multiple epochs, in our settings, all of the models only see a sample once. Such complexity makes the bandit setting much more challenging.

Why Simultaneously Training Source and Target Domains is Needed and Helpful. It is also noteworthy that our DABand is an end-to-end model that enables two-way feedback between source and target domains:

  • Source to Target: Collecting source-domain rewards helps DABand to learn a better encoder (which transforms raw contexts into encodings in the latent space) because the reward signals help the encoder extract more relevant encodings (embeddings) from target-domain contexts, thereby reducing the target-domain regret.

  • Target to Source: After aligning source-domain and target-domain encodings in the shared latent space (i.e., minimizing the data divergence term in Eqn. (5)), the information from the target-domain context can then help the source domain to explore arms that are more relevant to the target domain. Ultimately, this also helps reduce the target-domain regret even without collecting target-domain rewards.

Importance of Bandits and DABand

Bandit algorithms are designed to navigate environments where feedback is sparse, costly, and indirect. By efficiently learning from limited feedback – identifying not just when a prediction is wrong but adapting without explicit guidance on the right choice – bandit algorithms offer a strategic advantage in dynamically evolving settings. Our DABand exemplifies this by leveraging a low-cost source domain to improve performance in a high-cost target domain, thereby reducing cumulative regret (improving accuracy) while minimizing operational costs. DABand not only reduces the expense associated with acquiring and labeling vast datasets but also capitalizes on the intrinsic adaptability of bandit algorithms to learn and optimize in complex, uncertain environments.

Novelties Restatement

Theoretical Novelty. In general, DABand is the first work to perform contextual bandit in a domain adaptation setting, i.e., adapt from a source domain with feedback to a target domain without feedback. Moreover, our theoretical analysis presents two major technical challenges/novelties below:

  • Generalization of HΔH\mathcal{H}\Delta\mathcal{H} Distance to Regression. In the original multi-domain generalization bound ([4]), the HΔH\mathcal{H}\Delta\mathcal{H} divergence between source and target domains is derived for classification models. However, contextual bandit H\mathcal{H} is essentially a reward regression problem, and therefore necessitates generalizing the HΔH\mathcal{H}\Delta\mathcal{H} distance from the classification case to the regression case. This presents a significant technical challenge, and leads to a series of modifications in the proof.

  • Decomposing the Target Regret into the Source Regret with Other Terms. The subsequent major challenge our DABand addresses involves decomposing the regret bound for the target domain (i.e., the target regret) into a source regret term and other terms to upper-bound the target regret. This is necessary because we do not have access to feedback/reward from the target domain (we only have access to feedback/reward in the source domain). This process requires a nuanced understanding of the interplay between source and target domain dynamics within our model’s framework.

Other Technical Novelties. Moreover, our contribution goes beyond the theoretical analysis itself. Specifically,

  • We identify the problem of contextual bandits across domains and propose domain-adaptive contextual bandits (DABand) as the first general method to explore a high-cost target domain while only collecting feedback from a low-cost source domain.

  • Our theoretical analysis shows that our method can achieve a sub-linear regret bound in the target domain.

  • Our empirical results on real-world datasets show our DABand significantly improves performance over the state-of-the-art contextual bandit methods when adapting across domains.

Discussion on Zero-shot Target Regret Bound

Significance of Theorem 3.1

Note that Theorem 3.1 is nontrivial. While it does resemble the generalization bound in domain adaptation, there are key differences. As mentioned in Observation (3) in § 3.6, our target regret bound includes two additional crucial terms not found in domain adaptation. Specifically:

  • Regression Error in the Source Domain. ∑i=1N(∣⟨θS∗,ϕS∗(xi,a^iS)⟩−⟨θ^,ϕ^(xi,a^iS)⟩∣)\sum_{i=1}^{N}\left(\left|\langle\theta_{S}^{*},\phi_{S}^{*}(x_{i,\hat{a}_{i}}^{S})\rangle-\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_{i}}^{S})\rangle\right|\right), which defines the difference between the true reward from selecting action a^i\hat{a}_{i} and the estimated reward for this action.

  • Predicted Reward. ∑i=1N(∣⟨θ^,ϕ^(xi,a^iT)⟩∣)+∑i=1N1[ai∗≠a^i](∣⟨θ^,ϕ^(xi,a^iS)⟩∣)\sum_{i=1}^{N}\left(\left|\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_{i}}^{T})\rangle\right|\right)+\sum_{i=1}^{N}\mathbb{1}[a_{i}^{*}\ne\hat{a}_{i}]\left(\left|\langle\hat{\theta},\hat{\phi}(x_{i,\hat{a}_{i}}^{S})\rangle\right|\right), which serves as a regularization term to regularize the model to avoid overestimating rewards.

The results of the ablation study in Table 4 in § 5 highlight the significance of these two terms. Please also refer to “Technical Novelty” of Sec. G.4 above for discussion on key novelty/challenges in deriving Theorem 3.1.

Testing Phase rather than Traditional Bandit Learning Settings

Our regret bound in Theorem 3.1 is a zero-shot regret bound for the target domain (with empirical results in the Zero-Shot Target Regret paragraph of § 5), which corresponds to a testing phase.

However, we would like to clarify that extending this bound to handle continued training in the target domain (corresponding to the Continued Training in Target Domains and Cumulative Regret paragraph of § 5) is straightforward. At a high level, we can have

RTtotal=RTzero−shot+RTcontinued,R_{T}^{total}=R_{T}^{zero-shot}+R_{T}^{continued},

with the first term handled by our DABand’s Theorem 3.1 and the second term handled by typical single-domain contextual bandit.

Scale of Source Regret and Predicted Rewards

Similar to Neural-LinUCB (NLinUCB), in our DABand, the true reward is restricted to the range of [0,1][0,1] (as we mentioned in § 3.1); therefore it will not make the source regret unbounded.

Furthermore, our DABand algorithm tries to minimize the source regret while aligning the source and target domains. The minimization of the source regret is theoretically guaranteed, as discussed in recent work such as Neural-LinUCB (NLinUCB) (Xu et al., [49]).

Are the Comparisons Fair?

Leveraging Target-Domain Contexts. A lot of our baselines do leverage the contexts of the target domain. For example, our baseline LinUCB-P starts by performing PCA jointly on both source-domain and target-domain contexts and then perform LinUCB. Therefore LinUCB-P does leverage the contexts of the target domain. Similarly, NLinUCB-P starts by performing PCA jointly on both source-domain and target-domain context and then perform NLinUCB. It therefore also leverages target-domain contexts.

Seeing Target-Domain Rewards. Note that in our domain adaptive bandit settings:

  • During the zero-shot phase, target rewards are not visible for all methods (include both baselines and our DABand); it is therefore fair comparison.

  • During the continued-training phase, all methods (include both baselines and our DABand) start to see the target reward and update their parameters; it is therefore also fair comparison.

Why Minimize the Predicted Reward on the Source Domain?

While the predicted-reward term is naturally derived from our theoretical analysis, we do find interesting insights when examining this term and its relation to our model. Specifically:

  • Indirect Regularization on Bandit Parameters θ^\hat{\theta} and the Encoder ϕ^(⋅)\hat{\phi}(\cdot). One can see this term as an L1 regularization term. It does not directly regularize the bandit parameter θ^\hat{\theta}; however, minimizing the L1 norm of the predicted reward (i.e., a KK-dimensional vector for a KK-arm bandit) does indirectly regularize the bandit parameter θ^\hat{\theta} and the encoder ϕ^(⋅)\hat{\phi}(\cdot), thereby preventing the L1 norm of the predicted reward from getting too large.

  • Smaller Predicted Rewards for Smaller Variance and Better Stability. Furthermore, this regularization can help avoid predicting high rewards for all arms in the bandit.

    • Note that for the bandit algorithm to achieve low regret, predicting large rewards are not necessary. This is because one uses the argmax operation (i.e., Line 6 in Alg. 1) to select the best arm; an arm kk with a small predicted reward can still be selected as long as all other arms have even smaller predicted rewards.

    • Too higher predicted rewards are not desirable because they increase the model’s sensitivity, leading to higher variance and subsequently increasing the generalization error (i.e., the target regret’s bound).

Clarification of DABand's Contributions

Our DABand is the first general method to explore a target domain while only collecting feedback from the source domain, regardless of linear or nonlinear assumptions. No prior methods have explored this setting of exploring a target domain while only collecting feedback from the source domain.

Therefore, DABand’s contribution is two-fold: (1) DABand is the first general method to explore a target domain while only collecting feedback from the source domain; (2) DABand is also the first general method in this domain-adaptive bandit setting that works even under general nonlinear assumptions.

Importance of the Discriminator.

Our bandit setting aims to explore a target domain while only collecting feedback from the source domain. All reward feedback in the target domain is unknown, making this approach very challenging. Therefore, using a discriminator to align representations for both domains is necessary. If we ignore the discriminator, the method will not work in our settings. This is also evidenced by the whole results for ablation studies in Table 5.

How the Terms in the Regret Decay as N increases.

To see how the terms in the regret decay:

  • Decay in the Source Regret. Most of the decay occurs in the first term (i.e., Source Regret RSR_S), which has been discussed in the NLinUCB paper [49]. Specifically the cumulative source regret is O(N)O(\sqrt{N}). Then, dividing both sides of Eqn. (5) by NN, we will get the average source regret term with O(N/N)O(\sqrt{N}/N), which does decay with NN.

  • Decay in the Data Divergence Term. Decay in the Data Divergence Term. The data divergence term, d^HΔH(S,T)\widehat{d}_{\mathcal{H}\Delta\mathcal{H}}(S,T), can actually be further decomposed into

    • an empirical term that estimates the divergence using NN source-domain contexts and NN target-domain contexts, and

    • a term related to NN with complexity O(log⁡(2N)/N)\mathcal{O}(\log(2N)/N) [4]. Therefore, the second term with O(log⁡(2N)/N)\mathcal{O}(\log(2N)/N) does also decay as NN increases.

Understanding the Bound if the Source and Target Domains are equivalent

Even if the input distributions of the two domains match, it does not imply that the relationship between the input context xx and the predicted reward is the same for them, since the data divergence term is only responsible for aligning the distribution of xx (input); it is irrelevant to the output and rewards. This is why other terms are needed to characterize the difference in the relationship between the context xx and the reward in the source and target domains. The data divergence term alone is not sufficient.

Clarification on the Problem Setting.

Simultaneous Observation. In our setting, source-domain contexts and target-domain contexts are simultaneously observed, with only the reward of the source domain being observed.

Target-Domain Contexts Available Even Before Running Alg. 1. Note that in practice, target-domain contexts are usually available even before Algorithm 1 starts and before observing any reward from the source domain. (This makes it possible to train our DABand by simultaneously using source-domain and target-domain contexts.)

For example, in the case of testing drug reactions on mice (source domain) and humans (target domain), usually one already has the human subjects’ genomics, demographic, and other data as target-domain contexts, even before testing the drug on mice (i.e., the source domain) and collecting rewards.

This is because these human genomics/demographic data are easy to collect at a low cost; in contrast, testing the drug on humans (i.e., the target domain) and collecting rewards involves extremely high costs, due to the risks of fatality, side effects, and the enormous costs of conducting clinical trials.

Potential Impact of DABand

Our DABand has many potential real-world applications, especially when obtaining responses is costly. For instance, in testing new drugs, we can construct responses from mice for new drug A, and then use DABand to obtain the zero-shot hypothetical regret bound for responses in humans. In another scenario, we can collect data (responses) on humans for another published, similar drug B, and then transfer knowledge by aligning the divergence between drug A and drug B. If both regrets reveal acceptable performance, we might not need excessive costs for back-and-forth testing, which significantly speeds up the process and reduces costs.

References

  1. [1]Alekh Agarwal, Daniel J. Hsu, Satyen Kale, John Langford, Lihong Li, and Robert E. Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In ICML, pp. 1638–1646, 2014.arxiv.org/abs/1402.0555
  2. [2]Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. JMLR, 3:397–422, 2002.DOI
  3. [3]Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
  4. [4]Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151–175, 2010.
  5. [5]Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.arxiv.org/abs/1512.03012
  6. [6]Ziliang Chen, Jingyu Zhuang, Xiaodan Liang, and Liang Lin. Blending-target domain adaptation by adversarial meta-adaptation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2248–2257, 2019.arxiv.org/abs/1907.03389
  7. [7]Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In AISTATS, pp. 208–214, 2011.
  8. [8]Shuyang Dai, Kihyuk Sohn, Yi-Hsuan Tsai, Lawrence Carin, and Manmohan Chandraker. Adaptation across extreme variations using unlabeled domain bridges. arXiv preprint arXiv:1906.02238, 2019.arxiv.org/abs/1906.02238
  9. [9]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
  10. [10]P Kingma Diederik. Adam: A method for stochastic optimization. (No Title), 2014.arxiv.org/abs/1412.6980
  11. [11]Dylan J. Foster, Alekh Agarwal, Miroslav Dudík, Haipeng Luo, and Robert E. Schapire. Practical contextual bandits with regression oracles. In ICML, pp. 1534–1543, 2018.arxiv.org/abs/1803.01088
  12. [12]Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of machine learning research, 17(59):1–35, 2016a.arxiv.org/abs/1505.07818
  13. [13]Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 17(1):2096–2030, 2016b.arxiv.org/abs/1505.07818
  14. [14]Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672–2680, 2014.
  15. [15]Han Guo, Ramakanth Pasunuru, and Mohit Bansal. Multi-source domain adaptation for text classification via distancenet-bandits. In Proceedings of the AAAI conference on artificial intelligence, 2020.arxiv.org/abs/2001.04362
  16. [16]Hengguan Huang, Hao Wang, and Brian Mak. Recurrent poisson process unit for speech recognition. In AAAI, volume 33, pp. 6538–6545, 2019.DOI
  17. [17]Nathan Korda, Balazs Szorenyi, and Shuai Li. Distributed clustering of linear bandits in peer to peer networks. In ICML, pp. 1301–1309, 2016.arxiv.org/abs/1604.07706
  18. [19]Branislav Kveton, Csaba Szepesvári, Anup Rao, Zheng Wen, Yasin Abbasi-Yadkori, and S. Muthukrishnan. Stochastic low-rank bandits. CoRR, abs/1712.04644, 2017.arxiv.org/abs/1712.04644
  19. [20]Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In WWW, pp. 661–670, 2010.DOI
  20. [21]Shuai Li, Alexandros Karatzoglou, and Claudio Gentile. Collaborative filtering bandits. In SIGIR, pp. 539–548, 2016.arxiv.org/abs/1502.03473
  21. [22]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740–755. Springer, 2014.arxiv.org/abs/1405.0312
  22. [23]Tianyi Liu, Zihao Xu, Hao He, Guangyuan Hao, Guang-He Lee, and Hao Wang. Taxonomy-structured domain adaptation. In ICML, 2023.arxiv.org/abs/2306.07874
  23. [24]Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Conditional adversarial domain adaptation. In NIPS, pp. 1647–1657, 2018.arxiv.org/abs/1705.10667
  24. [25]Kanak Mahadik, Qingyun Wu, Shuai Li, and Amit Sabne. Fast distributed bandits for online recommendation systems. In SC, pp. 1–13, 2020.arxiv.org/abs/2007.08061
  25. [26]Le Thanh Nguyen-Meidine, Atif Belal, Madhu Kiran, Jose Dolz, Louis-Antoine Blais-Morin, and Eric Granger. Unsupervised multi-target domain adaptation through knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1339–1347, 2021.
  26. [27]Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. TKDE, 22(10):1345–1359, 2009.DOI
  27. [28]Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. TNN, 22(2):199–210, 2010.DOI
  28. [29]Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924, 2017.arxiv.org/abs/1710.06924
  29. [30]Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1406–1415, 2019.arxiv.org/abs/1812.01754
  30. [31]Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8558–8567, 2021.arxiv.org/abs/2012.11460
  31. [32]Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. Youtubeboundingboxes: A large high-precision human-annotated data set for object detection in video. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5296–5305, 2017.arxiv.org/abs/1702.00824
  32. [33]Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR, pp. 3723–3732, 2018.arxiv.org/abs/1712.02560
  33. [34]Swami Sankaranarayanan, Yogesh Balaji, Carlos D. Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In CVPR, pp. 8503–8512, 2018.arxiv.org/abs/1704.01705
  34. [35]Haizhou Shi and Hao Wang. A unified approach to domain incremental learning with memory: Theory and algorithm. In NeurIPS, 2023.arxiv.org/abs/2310.12244
  35. [36]Nian Si, Fan Zhang, Zhengyuan Zhou, and Jose Blanchet. Distributionally robust batch contextual bandits. Management Science, 69(10):5772–5793, 2023.arxiv.org/abs/2006.05630
  36. [37]Baochen Sun and Kate Saenko. Deep CORAL: correlation alignment for deep domain adaptation. In ICCV workshop on Transferring and Adapting Source Knowledge in Computer Vision (TASK-CV), pp. 443–450, 2016.arxiv.org/abs/1607.01719
  37. [38]Hui Tang and Kui Jia. A new benchmark: On the utility of synthetic data with blender for bare supervised learning and downstream domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15954–15964, 2023.arxiv.org/abs/2303.09165
  38. [39]Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.arxiv.org/abs/1412.3474
  39. [40]Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, pp. 7167–7176, 2017.
  40. [41]Hao Wang and Dit-Yan Yeung. Towards bayesian deep learning: A framework and some existing methods. TDKE, 28(12):3395–3408, 2016.arxiv.org/abs/1608.06884
  41. [42]Hao Wang and Dit-Yan Yeung. A survey on bayesian deep learning. CSUR, 53(5):1–37, 2020.arxiv.org/abs/1604.01662
  42. [45]Hao Wang, Yifei Ma, Hao Ding, and Yuyang Wang. Context uncertainty in contextual bandits with applications to recommender systems. In AAAI, 2022.arxiv.org/abs/2202.00805
  43. [47]Hengyi Wang, Shiwei Tan, and Hao Wang. Probabilistic conceptual explainers: Towards trustworthy conceptual explanations for vision foundation models. In ICML, 2024b.arxiv.org/abs/2406.12649
  44. [48]Ziyan Wang and Hao Wang. Variational imbalanced regression: Fair uncertainty quantification via probabilistic smoothing. In NeurIPS, 2023.arxiv.org/abs/2306.06599
  45. [49]Pan Xu, Zheng Wen, Handong Zhao, and Quanquan Gu. Neural contextual bandits with deep representation and shallow exploration. arXiv preprint arXiv:2012.01780, 2020.arxiv.org/abs/2012.01780
  46. [50]Zihao Xu, Guang-He Lee, Yuyang Wang, Hao Wang, et al. Graph-relational domain adaptation. In ICLR, 2022.arxiv.org/abs/2202.03628
  47. [51]Zihao Xu, Guangyuan Hao, Hao He, and Hao Wang. Domain indexing variational bayes: Interpretable domain index for domain adaptation. In ICLR, 2023.arxiv.org/abs/2302.02561
  48. [52]Yisong Yue and Carlos Guestrin. Linear submodular bandits and their application to diversified retrieval. In NIPS, pp. 2483–2491, 2011.
  49. [53]Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael I Jordan. Bridging theory and algorithm for domain adaptation. arXiv preprint arXiv:1904.05801, 2019.arxiv.org/abs/1904.05801
  50. [54]Han Zhao, Shanghang Zhang, Guan؟hang Wu, José M. F. Moura, João Paulo Costeira, and Geoffrey J. Gordon. Adversarial multiple source domain adaptation. In NIPS, pp. 8568–8579, 2018.
  51. [55]Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J. Gordon. On learning invariant representations for domain adaptation. In ICML, pp. 7523–7532, 2019.
  52. [56]Mingmin Zhao, Shichao Yue, Dina Katabi, Tommi S. Jaakkola, and Matt T. Bianchi. Learning sleep stages from radio signals: A conditional adversarial architecture. In ICML, pp. 4100–4109, 2017.
  53. [57]Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with upper confidence bound-based exploration. arXiv preprint arXiv:1911.04462, 2019.arxiv.org/abs/1911.04462

Paper details

vlnc:W_9wd4a8x7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents