Introduction
In many engineering applications, the goal is to classify a collection of data sources according to their underlying distributions based on sequentially collected measurements. For example, in navigation monitoring systems, one seeks to identify malfunctioning sensors that introduce noise into the measurements [1 ] . In computer network intrusion detection, deviations from a standard traffic profile, such as a reduced data transmission rate, may indicate a potential attack [2 ] . In cognitive radio spectrum sensing [3 ] , the objective is to detect available frequency bands where the signal is of very low amplitude. Such applications motivate the study of sequential multiple testing problems for composite hypotheses [4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 ] .
Composite hypothesis testing extends the classical binary hypothesis testing framework by replacing two competing distributions with two disjoint sets of distributions. This generalization introduces additional complexity because the true distribution is known only to belong to one of these sets. The study of sequential composite hypothesis testing dates back to Wald [4 ] , who considered a simple null hypothesis against a composite alternative and introduced the Weighted SPRT (WSPRT) and the Generalized SPRT (GSPRT). While the error probabilities of the WSPRT can be controlled using techniques similar to those developed for simple hypotheses, analyzing the GSPRT is more involved. Lorden [13 ] studied open-ended generalized likelihood ratio tests of a simple null hypothesis against a one-sided composite alternative in exponential families, establishing near-optimal expected sample sizes subject to a bound on the probability of rejecting the null hypothesis. Further asymptotic analyses of sequential tests include the works of Lorden [7 ] , Woodroofe [14 ] , and Chan and Lai [15 ] , the latter focusing on error-probability approximations in multiparameter exponential families. For separate parametric families that need not belong to a common exponential family, Li et al. [16 ] established logarithmic asymptotic characterizations of the GSPRT’s maximal type I and type II error probabilities, and derived first-order asymptotic expressions for its expected sample size.
The Bayesian formulation of composite hypothesis testing was introduced by Schwarz [6 ] , who proposed a first-order asymptotically Bayes-optimal procedure for parametric families in which the parameter belongs to one of two disjoint sets. Variants of this procedure were subsequently shown to preserve asymptotic optimality [5 ] , [10 ] , [17 ] , and further refinements achieved second-order optimality [7 ] . Related procedures were also developed in the frequentist setting, where first-order asymptotic optimality was established under appropriate conditions [8 ] . In a related direction, Yilmaz et al. [18 ] studied sequential joint detection and estimation, where a decision in favor of the alternative is accompanied by an estimate of a random parameter. They showed that optimal joint performance requires accounting for estimation in the design of the detection rule, rather than optimizing the two tasks separately.
Optimal sequential testing was further extended to experimental design, where the decision maker determines both when to stop and which observations to collect. Chernoff [19 ] considered a simple null hypothesis against a finite set of alternatives and proposed an asymptotically optimal procedure based on adaptive experiment selection. This framework was subsequently extended to countably infinite alternatives [20 ] and multiple competing hypotheses [21 ] , [22 ] , with later developments addressing frequentist error criteria [23 ] . More recently, Deshmukh et al. [24 ] studied controlled sequential testing of composite hypotheses in exponential family models, with the unknown parameter belonging to one of several prespecified sets. Assuming that the number of available experiments equals the number of hypotheses, they established first order asymptotic optimality using sampling rules inspired by the multi-armed bandit literature [25 ] .
In our work, we consider multiple data sources, each associated with a composite hypothesis testing problem, and adopt a frequentist formulation in which the objective is to minimize the expected stopping time as the error probabilities approach zero. The observations from each source follow a density that belongs to one of two disjoint sets, corresponding to regular and anomalous behavior, respectively. A source is declared anomalous if its underlying density belongs to the latter set. Within exponential family models, the problem of sequential anomaly identification for composite hypotheses has been studied in [26 ] , [27 ] , where all sources are observed at each time and generalized error metrics are considered. The authors propose first-order asymptotically optimal policies and provide explicit characterizations of the optimal performance.
We now consider the nonparametric setting under sampling constraints, where only a subset of sources can be observed at each time due to resource limitations. Nonparametric formulations of sequential hypothesis testing impose minimal assumptions on the underlying distributions and have been studied both in classical settings [28 ] and, more recently, through time-uniform, nonasymptotic inference methods [29 ] . These works highlight the inherent difficulty of the problem when the statistical model is not fully specified. In contrast to the existing literature, we consider a setting in which multiple sources must be monitored under explicit sampling constraints. In particular, at each time instant only a subset of sources can be observed, and the goal is to design a policy that jointly determines when to stop, which sources to declare anomalous, and how to allocate observations across sources. The stopping and decision rules must ensure that the error probabilities remain below prescribed levels, while the sampling rule must satisfy the imposed constraint. Existing works on sequential anomaly identification typically assume that all sources are observed at each time or focus on restricted settings. For example, when there is a single anomalous source and only one observation can be collected at each time, asymptotically optimal policies have been derived in [12 ] , [30 ] . In our formulation, we allow for multiple anomalous sources, simultaneous observations of multiple sources, and more general sampling constraints, expressed in terms of the ratio between the expected number of observations and the expected stopping time. Examples show that accounting for distributional shape beyond parameters is essential for efficient sampling allocation, further motivating the necessity of a nonparametric framework.
We impose regularity conditions on the nonparametric family similar in spirit to those introduced by Pavlov [10 Section 3] , who established their validity for multidimensional exponential families under compactness. Within this framework, we establish conditions for consistency and asymptotic optimality of the proposed procedures. In particular, we show that suitable convergence of the test statistics and the estimators ensures these properties. In addition, asymptotic optimality is achieved when the long-run sampling frequency of each source exceeds an explicitly computable threshold. Based on this characterization, we design probabilistic sampling rules that attain the optimal performance. Our results generalize our previous work on sequential anomaly identification for the binary (simple vs. simple) case [31 ] , and establish connections with the multi-armed bandit literature [25 , 32 , 33 , 34 , 35 ] . In classical bandit problems, the objective is to maximize cumulative reward, which requires identifying the arm with the largest mean as quickly as possible [25 , 32 , 33 ] . In contrast, in [34 ] , each arm is associated with a composite hypothesis testing problem, and the objective is to minimize the expected stopping time and switching costs. This setting naturally aligns with pure-exploration bandit problems [25 , 35 ] , where the goal is to identify a subset of arms satisfying a statistical property. Our results provide a unified characterization of asymptotically optimal sampling strategies across these settings. The proposed framework is also closely connected to recent developments in machine learning that integrate generative models with active learning for anomaly detection [36 , 37 , 38 ] , highlighting the role of adaptive data acquisition in improving detection performance. However, in these approaches, data acquisition is typically passive, with all features or data streams observed uniformly over time. In contrast, our formulation explicitly incorporates sampling constraints and enables the sequential allocation of observations to the most informative sources, providing a principled approach to active anomaly detection under resource constraints.
The remainder of the paper is organized as follows. Section II introduces the problem formulation, Section IV provides an illustrative example, and § 5 derives a universal asymptotic lower bound (UALB) on the expected stopping time. Section VI presents a family of policies, and § 7 establishes conditions under which an appropriately designed policy attains the UALB and is therefore asymptotically optimal. Sections VIII and IX present test statistics and estimators that satisfy the required asymptotic conditions. § 10 and XI evaluate the performance of the proposed policy through simulation studies. Finally, Section XII concludes the paper and outlines directions for future research.
We end this section with some notations we use throughout the paper. We use : = := := to indicate the definition of a new quantity and ≡ \equiv ≡ to indicate a duplication of notation. We set N : = { 1 , 2 , … } \mathbb{N} := \{1,2,\ldots\} N := { 1 , 2 , … } and [ n ] : = { 1 , … , n } [n] := \{1,\ldots,n\} [ n ] := { 1 , … , n } for n ∈ N n \in\mathbb{N} n ∈ N , we denote by ∣ A ∣ |A| ∣ A ∣ the size and by 2 A 2^{A} 2 A the powerset of a set A A A , by [ a ] [a] [ a ] the floor and by ⌈ a ⌉ \lceil a\rceil ⌈ a ⌉ the ceiling of a positive number a a a , and by 1 \mathbf{1} 1 the indicator of an event. The symbol ∨ \vee ∨ stands for the maximum, and the symbol ∧ \wedge ∧ for the minimum of two quantities. We write x ∼ y x\sim y x ∼ y when lim ( x / y ) = 1 \lim(x/y)=1 lim ( x / y ) = 1 , x ≳ y x\gtrsim y x ≳ y when lim inf ( x / y ) ≥ 1 \liminf(x/y)\ge1 lim inf ( x / y ) ≥ 1 , and x ≲ y x\lesssim y x ≲ y when lim sup ( x / y ) ≤ 1 \limsup(x/y)\le1 lim sup ( x / y ) ≤ 1 , where the limit is taken in some sense that will be specified. We denote by ∥ ⋅ ∥ p \|\cdot\|_p ∥ ⋅ ∥ p the L p L^p L p -norm of a function. Moreover, iid stands for independent and identically distributed, and we say that a sequence of positive numbers ( a n ) (a_n) ( a n ) is summable if ∑ n = 1 ∞ a n < ∞ \sum_{n=1}^{\infty}a_n<\infty ∑ n = 1 ∞ a n < ∞ and exponentially decaying if there are real numbers c , d > 0 c,d>0 c , d > 0 such that a n ≤ c exp { − d n } a_n\le c\exp\{-dn\} a n ≤ c exp { − d n } for every n ∈ N n\in\mathbb{N} n ∈ N . A useful property is that if ( a n ) (a_n) ( a n ) is exponentially decaying, so is the sequence ( ∑ m ≥ ζ n a m ) (\sum_{m\ge\zeta n}a_m) ( ∑ m ≥ ζ n a m ) , for any ζ ∈ ( 0 , 1 ] \zeta\in(0,1] ζ ∈ ( 0 , 1 ] .
Let ( S , S ) (\mathcal{S},\mathcal{S}) ( S , S ) be a measurable space, and let ( Ω , F , P ) (\Omega,\mathcal{F},\mathbb{P}) ( Ω , F , P ) be a probability space on which are defined M M M independent sequences of iid S \mathcal{S} S -valued random elements, { X i ( n ) : n ∈ N } \{X_i(n):n\in\mathbb{N}\} { X i ( n ) : n ∈ N } , i ∈ [ M ] i\in[M] i ∈ [ M ] , corresponding to M M M distinct data sources. In addition, let { Z i ( n ) : n ∈ N } \{Z_i(n):n\in\mathbb{N}\} { Z i ( n ) : n ∈ N } , i ∈ [ M ] i\in[M] i ∈ [ M ] , be M M M independent sequences of iid Uniform[ 0 , 1 ] [0,1] [ 0 , 1 ] random variables, used for randomization purposes, and define Z ( n ) : = ( Z 1 ( n ) , … , Z M ( n ) ) Z(n):=(Z_1(n),\ldots,Z_M(n)) Z ( n ) := ( Z 1 ( n ) , … , Z M ( n )) .
For each i ∈ [ M ] i\in[M] i ∈ [ M ] , the observations { X i ( n ) : n ∈ N } \{X_i(n):n\in\mathbb{N}\} { X i ( n ) : n ∈ N } are generated according to a density f i ∈ Φ i f_i\in\Phi_i f i ∈ Φ i , with respect to a σ \sigma σ -finite measure ν i \nu_i ν i , where the class of densities Φ i \Phi_i Φ i is equipped with the L 1 L^1 L 1 -norm ∥ ⋅ ∥ 1 \|\cdot\|_1 ∥ ⋅ ∥ 1 . We say that source i i i is anomalous if f i ∈ Φ i 1 f_i\in\Phi_i^1 f i ∈ Φ i 1 , and regular if f i ∈ Φ i 0 f_i\in\Phi_i^0 f i ∈ Φ i 0 , where Φ i : = Φ i 0 ∪ Φ i 1 \Phi_i:=\Phi_i^0\cup\Phi_i^1 Φ i := Φ i 0 ∪ Φ i 1 and Φ i 0 ∩ Φ i 1 = ∅ \Phi_i^0\cap\Phi_i^1=\varnothing Φ i 0 ∩ Φ i 1 = ∅ . Let Φ : = Φ 1 × ⋯ × Φ M \Phi:=\Phi_1\times\cdots\times\Phi_M Φ := Φ 1 × ⋯ × Φ M denote the product space. For each A ⊆ [ M ] A\subseteq[M] A ⊆ [ M ] , define
Φ A : = { ( g 1 , … , g M ) : g i ∈ Φ i 1 for i ∈ A , and g j ∈ Φ j 0 for j ∉ A } . \Phi_A:=\{(g_1,\ldots,g_M):g_i\in\Phi_i^1\text{ for }i\in A,\text{ and }g_j\in\Phi_j^0\text{ for }j\notin A\}. Φ A := {( g 1 , … , g M ) : g i ∈ Φ i 1 for i ∈ A , and g j ∈ Φ j 0 for j ∈ / A } .
where A A A stands for the subset of anomalous sources. For each A ⊆ [ M ] A\subseteq[M] A ⊆ [ M ] and f : = ( f 1 , … , f M ) ∈ Φ A f:=(f_1,\ldots,f_M)\in\Phi_A f := ( f 1 , … , f M ) ∈ Φ A , we denote by P f \mathbb{P}_f P f the probability measure and by E f \mathbb{E}_f E f the corresponding expectation, when the underlying true densities are given by f f f . When the dependence on f f f is not essential for our computations, we simply write P \mathbb{P} P and E \mathbb{E} E .
For each i ∈ [ M ] i\in[M] i ∈ [ M ] and g ∈ Φ i g\in\Phi_i g ∈ Φ i , we define the Kullback–Leibler (KL) divergence
D ( f i , g ) : = E f i [ log ( f i ( X i ) g ( X i ) ) ] . D(f_i,g):=\mathbb{E}_{f_i}\left[\log\left(\frac{f_i(X_i)}{g(X_i)}\right)\right]. D ( f i , g ) := E f i [ log ( g ( X i ) f i ( X i ) ) ] .
where X i X_i X i denotes a generic observation from source i i i . We impose the following assumptions, which are required for the development of our results:
(i) Separation of hypotheses:
I i ( f i ) : = inf g ∈ Φ i 0 D ( f i , g ) ∈ ( 0 , ∞ ) , ∀ i ∈ A , (1) I_i(f_i):=\inf_{g\in\Phi_i^0}D(f_i,g)\in(0,\infty),\qquad\forall i\in A, \tag*{(1)} I i ( f i ) := g ∈ Φ i 0 inf D ( f i , g ) ∈ ( 0 , ∞ ) , ∀ i ∈ A , (1)
J i ( f i ) : = inf g ∈ Φ i 1 D ( f i , g ) ∈ ( 0 , ∞ ) , ∀ i ∉ A . (2) J_i(f_i):=\inf_{g\in\Phi_i^1}D(f_i,g)\in(0,\infty),\qquad\forall i\notin A. \tag*{(2)} J i ( f i ) := g ∈ Φ i 1 inf D ( f i , g ) ∈ ( 0 , ∞ ) , ∀ i ∈ / A . (2)
(ii) Continuity of the infimum maps:
lim f → f i I i ( f ) = I i ( f i ) , lim f → f i J i ( f ) = J i ( f i ) . (3) \lim_{f\to f_i} I_i(f)=I_i(f_i),\qquad\lim_{f\to f_i} J_i(f)=J_i(f_i). \tag*{(3)} f → f i lim I i ( f ) = I i ( f i ) , f → f i lim J i ( f ) = J i ( f i ) . (3)
Both assumptions depend on the true density f i f_i f i , which is unknown, and thus it is required that they hold locally for every f i ∈ Φ i f_i\in\Phi_i f i ∈ Φ i , i.e., in a neighborhood of f i f_i f i .
The main objective is to identify the set of anomalous sources A A A , i.e., to determine whether f i ∈ Φ i 1 f_i\in\Phi_i^1 f i ∈ Φ i 1 or f i ∈ Φ i 0 f_i\in\Phi_i^0 f i ∈ Φ i 0 for each i ∈ [ M ] i\in[M] i ∈ [ M ] , based on sequential observations under sampling constraints that preclude observing all sources at every time. To this end, we consider policies that consist of three components: (i) a stopping time T T T , (ii) a sequence Δ : = { Δ ( n ) : n ∈ N } \Delta:=\{\Delta(n):n\in\mathbb{N}\} Δ := { Δ ( n ) : n ∈ N } , where Δ ( n ) ⊆ [ M ] \Delta(n)\subseteq[M] Δ ( n ) ⊆ [ M ] denotes the current estimate of the anomalous set and Δ ( T ) \Delta(T) Δ ( T ) is the final decision, and (iii) a sequence R : = { R ( n ) : n ∈ N } R:=\{R(n):n\in\mathbb{N}\} R := { R ( n ) : n ∈ N } , where R ( n ) ⊆ [ M ] R(n)\subseteq[M] R ( n ) ⊆ [ M ] is the set of sources sampled at time n n n . We say that R R R is a sampling rule if R ( n ) R(n) R ( n ) is F n − 1 R \mathcal{F}_{n-1}^R F n − 1 R -measurable for every n ∈ N n\in\mathbb{N} n ∈ N , where the filtration { F n R : n ∈ N } \{\mathcal{F}_n^R:n\in\mathbb{N}\} { F n R : n ∈ N } is defined recursively by
F n R : = σ ( F n − 1 R , Z ( n ) , { X i ( n ) : i ∈ R ( n ) } ) , n ∈ N , \mathcal{F}_n^R:=\sigma\bigl(\mathcal{F}_{n-1}^R,Z(n),\{X_i(n):i\in R(n)\}\bigr),\qquad n\in\mathbb{N}, F n R := σ ( F n − 1 R , Z ( n ) , { X i ( n ) : i ∈ R ( n )} ) , n ∈ N ,
F 0 R : = σ ( Z ( 0 ) ) . \mathcal{F}_0^R:=\sigma(Z(0)). F 0 R := σ ( Z ( 0 )) .
A triplet ( R , T , Δ ) (R,T,\Delta) ( R , T , Δ ) is called a policy if R R R is a sampling rule, T T T is a stopping time with respect to { F n R : n ∈ N } \{\mathcal{F}_n^R:n\in\mathbb{N}\} { F n R : n ∈ N } , i.e., { T = n } ∈ F n R \{T=n\}\in\mathcal{F}_n^R { T = n } ∈ F n R for all n n n , and Δ ( n ) \Delta(n) Δ ( n ) is F n R \mathcal{F}_n^R F n R -measurable for every n ∈ N n\in\mathbb{N} n ∈ N . In this case, we refer to T T T as the stopping rule and to Δ \Delta Δ as the decision rule . Our policies ( R , T , Δ ) (R,T,\Delta) ( R , T , Δ ) must satisfy prescribed error and sampling constraints, described next.
Error constraints: Let α , β ∈ ( 0 , 1 ) \alpha,\beta\in(0,1) α , β ∈ ( 0 , 1 ) . A policy ( R , T , Δ ) (R,T,\Delta) ( R , T , Δ ) is said to belong to the class C ( α , β ) \mathcal{C}(\alpha,\beta) C ( α , β ) if it satisfies the error constraints
P f ( ∣ Δ ( T ) ∖ A ∣ ≥ 1 ) ≤ α , P f ( ∣ A ∖ Δ ( T ) ∣ ≥ 1 ) ≤ β , (4) P_f\bigl(|\Delta(T)\setminus A|\ge1\bigr)\le\alpha,\qquad P_f\bigl(|A\setminus\Delta(T)|\ge1\bigr)\le\beta, \tag*{(4)} P f ( ∣Δ ( T ) ∖ A ∣ ≥ 1 ) ≤ α , P f ( ∣ A ∖ Δ ( T ) ∣ ≥ 1 ) ≤ β , (4)
for all A ⊆ [ M ] A\subseteq[M] A ⊆ [ M ] and all f ∈ Φ A f\in\Phi_A f ∈ Φ A . These correspond to controlling the probabilities of at least one false alarm and at least one missed detection , respectively.
Prior to stating the sampling constraint, we introduce the following notation. For any sampling rule R R R , let R i ( n ) R_i(n) R i ( n ) denote the indicator of whether source i i i is sampled at time n n n , i.e.,
R i ( n ) : = 1 { i ∈ R ( n ) } . R_i(n):=1\{i\in R(n)\}. R i ( n ) := 1 { i ∈ R ( n )} .
We define
N i R ( n ) : = ∑ m = 1 n R i ( m ) , π i R ( n ) : = N i R ( n ) n , (5) N_i^R(n):=\sum_{m=1}^{n}R_i(m),\qquad\pi_i^R(n):=\frac{N_i^R(n)}{n}, \tag*{(5)} N i R ( n ) := m = 1 ∑ n R i ( m ) , π i R ( n ) := n N i R ( n ) , (5)
where N i R ( n ) N_i^R(n) N i R ( n ) is the number of times source i i i is sampled in the first n n n time instants, and π i R ( n ) \pi_i^R(n) π i R ( n ) is the corresponding sampling proportion.
Sampling constraint: Let K ∈ ( 0 , M ] K \in(0,M] K ∈ ( 0 , M ] . We say that a policy ( R , T , Δ ) (R,T,\Delta) ( R , T , Δ ) belongs to the class C ( α , β , K ) \mathcal{C}(\alpha,\beta,K) C ( α , β , K ) if, in addition to satisfying the error constraint (4), it holds that
∑ i = 1 M E [ N i R ( T ) ] ≤ K E [ T ] . (6) \sum_{i=1}^{M} \mathbb{E}\left[N_i^R(T)\right] \le K\mathbb{E}[T]. \tag*{(6)} i = 1 ∑ M E [ N i R ( T ) ] ≤ K E [ T ] . (6)
That is, the expected total number of observations up to the stopping time does not exceed K K K times the expected stopping time. Using the identity
∑ i = 1 M E [ N i R ( T ) ] = E [ ∑ n = 1 T E [ ∣ R ( n ) ∣ ∣ F n − 1 R ] ] , \sum_{i=1}^{M} \mathbb{E}\left[N_i^R(T)\right] = \mathbb{E}\left[\sum_{n=1}^{T} \mathbb{E}\left[|R(n)| \mid\mathcal{F}_{n-1}^R\right]\right], i = 1 ∑ M E [ N i R ( T ) ] = E [ n = 1 ∑ T E [ ∣ R ( n ) ∣ ∣ F n − 1 R ] ] ,
we observe that (6) is satisfied whenever
sup n ∈ N , n ≤ T E [ ∣ R ( n ) ∣ ∣ F n − 1 R ] ≤ K , a.s. (7) \sup_{\substack{n\in\mathbb{N},\,n\le T}} \mathbb{E}\left[|R(n)| \mid\mathcal{F}_{n-1}^R\right] \le K,\qquad\text{a.s.} \tag*{(7)} n ∈ N , n ≤ T sup E [ ∣ R ( n ) ∣ ∣ F n − 1 R ] ≤ K , a.s. (7)
Condition (7) is stronger than (6), and thus (6) allows for sampling rules that do not necessarily satisfy (7).
Our goal is to design a policy ( R , T , Δ ) ∈ C ( α , β , K ) (R,T,\Delta) \in\mathcal{C}(\alpha,\beta,K) ( R , T , Δ ) ∈ C ( α , β , K ) whose expected stopping time attains the infimum
J f ( α , β , K ) : = inf ( R , T , Δ ) ∈ C ( α , β , K ) E f [ T ] (8) J_f(\alpha,\beta,K) := \inf_{(R,T,\Delta)\in\mathcal{C}(\alpha,\beta,K)} \mathbb{E}_f[T] \tag*{(8)} J f ( α , β , K ) := ( R , T , Δ ) ∈ C ( α , β , K ) inf E f [ T ] (8)
to a first-order asymptotic approximation as α , β → 0 \alpha,\beta\to0 α , β → 0 , for every f ∈ Φ f \in\Phi f ∈ Φ . Throughout, we assume that there exists r ∈ ( 0 , ∞ ) r \in(0,\infty) r ∈ ( 0 , ∞ ) such that
∣ log ( α ) ∣ ∼ r ∣ log ( β ) ∣ , α , β → 0. (9) |\log(\alpha)| \sim r|\log(\beta)|,\qquad\alpha,\beta\to0. \tag*{(9)} ∣ log ( α ) ∣ ∼ r ∣ log ( β ) ∣ , α , β → 0. (9)
Background results
In our previous work [31 ] , we studied sequential anomaly identification under sampling constraints in the simple-versus-simple setting. Each source generates independent and identically distributed observations, independently of the other sources, according to one of two specified densities: f 0 i f_{0i} f 0 i when source i i i is regular and f 1 i f_{1i} f 1 i when it is anomalous. The corresponding information numbers, I i : = D ( f 1 i , f 0 i ) I_i := D(f_{1i},f_{0i}) I i := D ( f 1 i , f 0 i ) and J i : = D ( f 0 i , f 1 i ) J_i := D(f_{0i},f_{1i}) J i := D ( f 0 i , f 1 i ) , are assumed to be positive and finite. The objective is to identify the unknown anomalous set A A A while controlling the probabilities of false positive and false negative errors below prescribed levels α \alpha α and β \beta β . Observations are collected under the sampling constraint (6).
The asymptotic optimality theory in [31 ] begins with a universal asymptotic lower bound on the expected stopping time of any procedure satisfying the error and sampling constraints. It then identifies target sampling frequencies { c i ∗ ( A ) : i ∈ [ M ] } \{c_i^*(A): i \in[M]\} { c i ∗ ( A ) : i ∈ [ M ]} and sufficient conditions under which maintaining these frequencies ensures that the lower bound is attained. These frequencies depend on the information numbers, the sampling budget, and the relative error levels. Specifically, it suffices that
∑ n = 1 ∞ P A ( π i R ( n ) < c i ∗ ( A ) − ϵ ) < ∞ , ∀ i ∈ [ M ] , ∀ ϵ > 0 , \sum_{n=1}^{\infty} \mathop{\mathrm{P}}\nolimits_A\left(\pi_i^R(n)<c_i^*(A)-\epsilon\right)<\infty,\qquad\forall i\in[M],\quad\forall\epsilon>0, n = 1 ∑ ∞ P A ( π i R ( n ) < c i ∗ ( A ) − ϵ ) < ∞ , ∀ i ∈ [ M ] , ∀ ϵ > 0 ,
where π i R ( n ) \pi_i^R(n) π i R ( n ) is the empirical sampling frequency of source i i i at time n n n . This condition controls the probabilities that a source is sampled less frequently than its target requires.
To satisfy this condition, a probabilistic sampling rule was proposed using one Bernoulli draw per source. At time n + 1 n+1 n + 1 , source i i i is sampled with probability c i ∗ ( Δ n R ) c_i^*(\Delta_n^R) c i ∗ ( Δ n R ) , where Δ n R \Delta_n^R Δ n R is the current estimate of the anomalous set. Thus, the sampling probabilities adapt to the observations through the estimated configuration. Together with the stopping and decision rules of [31 ] , this sampling rule attains first-order asymptotic optimality.
The asymptotically optimal expected stopping time is determined by the rate at which evidence is accumulated. Sampling an anomalous source i i i with frequency c i ∗ ( A ) c_i^*(A) c i ∗ ( A ) yields an information rate c i ∗ ( A ) I i c_i^*(A)I_i c i ∗ ( A ) I i ; the corresponding rate for a regular source j j j is c j ∗ ( A ) J j c_j^*(A)J_j c j ∗ ( A ) J j . When α = β → 0 \alpha=\beta\to0 α = β → 0 , the target frequencies balance these information rates, and an appropriately designed procedure satisfies
E A [ T R ] ∼ ∣ log α ∣ c i ∗ ( A ) I i , i ∈ A , E A [ T R ] ∼ ∣ log β ∣ c j ∗ ( A ) J j , j ∉ A . (10) \begin{aligned}
\mathop{\mathrm{E}}\nolimits_A[T^R] &\sim\frac{|\log\alpha|}{c_i^*(A)I_i},\qquad i\in A,\\
\mathop{\mathrm{E}}\nolimits_A[T^R] &\sim\frac{|\log\beta|}{c_j^*(A)J_j},\qquad j\notin A. \tag*{(10)}
\end{aligned} E A [ T R ] E A [ T R ] ∼ c i ∗ ( A ) I i ∣ log α ∣ , i ∈ A , ∼ c j ∗ ( A ) J j ∣ log β ∣ , j ∈ / A . (10)
These expressions hold for arbitrary K ∈ ( 0 , M ] K\in(0,M] K ∈ ( 0 , M ] , with the dependence on K K K incorporated in the target frequencies. Thus, optimal sampling accounts for both the information provided by each observation and the frequency with which each source is observed.
In the present work, we extend these results to nonparametric hypothesis testing, where the null and alternative hypotheses consist of classes of densities. The information numbers and target sampling frequencies now depend on the unknown underlying densities. We introduce a generalized criterion for asymptotic optimality that jointly addresses the sampling rule, the test statistics, and the density estimators. Specifically, it requires sampling probabilities that dominate the target frequencies, test statistics that accumulate evidence at the corresponding information rates, and density estimators that converge sufficiently quickly to the true densities. Together, these conditions guarantee that the designed procedure attains first-order asymptotic optimality.
An illustrative example
This example illustrates how distributional misspecification can affect the allocation of limited observations across sources. Although accurate mean estimation suffices for consistent hypothesis identification in this setting, a Gaussian surrogate that matches the true mean and variance can yield sampling frequencies that differ substantially from the optimal frequencies determined by the true densities. We show that the resulting allocation can cause an arbitrarily large loss in first-order asymptotic efficiency, even when the true likelihood ratios are used for testing. This highlights the importance of accounting for distributional shape in hypothesis testing under sampling constraints and motivates the use of nonparametric density estimation when a reliable parametric family is unavailable.
We consider two independent sources, each generating independent and identically distributed observations. Exactly one source is observed at each time, i.e.,
∣ R ( n ) ∣ = 1 , n ≥ 1 , |R(n)| = 1, \qquad n \ge1, ∣ R ( n ) ∣ = 1 , n ≥ 1 ,
which satisfies the sampling constraint (7) with K = 1 K = 1 K = 1 . For each i ∈ [ 2 ] i \in[2] i ∈ [ 2 ] , let f i f_i f i denote the underlying true density, and let f 0 : = N ( 0 , σ 2 ) f_0 := \mathcal{N}(0,\sigma^2) f 0 := N ( 0 , σ 2 ) denote the common Gaussian null density. Let Φ i 1 \Phi_i^1 Φ i 1 be the class of all L L L -Lipschitz densities on R \mathbb{R} R with mean in [ ϵ , B ] [\epsilon, B] [ ϵ , B ] , variance σ 2 \sigma^2 σ 2 , and fourth moment bounded by C C C , where L , C > 0 L, C > 0 L , C > 0 are fixed and sufficiently large. We consider the hypothesis-testing problem
\begin{aligned}
H_i^1 &: f_i \in\Phi_i^1,\\
H_i^0 &: f_i = f_0,
\end{aligned}
\qquad i \in[2], \tag*{(11)}\end{aligned}
where 0 < ϵ < 1 ≤ B 0 < \epsilon< 1 \le B 0 < ϵ < 1 ≤ B and σ > 0 \sigma> 0 σ > 0 are fixed.
Class Φ i 1 \Phi_i^1 Φ i 1 permits densities with the same mean and variance but different distributional shapes and information numbers. It therefore allows us to isolate the effect of distributional shape on the sampling allocation.
Suppose that both sources are anomalous and that f i f_i f i is a Laplace density with location parameter μ i \mu_i μ i , and scale parameter σ / 2 \sigma/\sqrt{2} σ / 2 , i.e.,
f i = L ( μ i , σ 2 ) , i ∈ [ 2 ] , (12) f_i = \mathcal{L}\left(\mu_i,\frac{\sigma}{\sqrt{2}}\right), \qquad i \in[2], \tag*{(12)} f i = L ( μ i , 2 σ ) , i ∈ [ 2 ] , (12)
where
μ i : = { 1 , i = 1 , ϵ , i = 2. (13) \mu_i :=
\begin{cases}
1, & i = 1,\\
\epsilon, & i = 2.
\end{cases} \tag*{(13)} μ i := { 1 , ϵ , i = 1 , i = 2. (13)
Throughout this example, dependence on ϵ \epsilon ϵ is suppressed in the notation unless needed when taking limits.
For the purposes of this illustrative example, we use the corresponding simple-versus-simple testing problems, in which each true alternative density f i f_i f i is treated as known. These provide an oracle benchmark that reflects the first-order asymptotic behavior of appropriately designed procedures for the composite problem. The results needed to establish this behavior for composite hypotheses are developed later in the paper.
In this simple-versus-simple setting, our previous work [31 ] shows that, under suitable regularity conditions, an appropriately designed procedure with limiting sampling frequencies satisfying
c 1 I 1 = c 2 I 2 , c 1 + c 2 = 1 , (14) c_1 I_1 = c_2 I_2,\qquad c_1+c_2=1, \tag*{(14)} c 1 I 1 = c 2 I 2 , c 1 + c 2 = 1 , (14)
attains the first-order asymptotically optimal expected stopping time
E [ 2 ] [ T o r ] ∼ ∣ log α ∣ c 1 I 1 , α = β → 0 , (15) \mathbb{E}_{[2]}[T_{\mathrm{or}}]\sim\frac{|\log\alpha|}{c_1I_1},\qquad\alpha=\beta\to0, \tag*{(15)} E [ 2 ] [ T or ] ∼ c 1 I 1 ∣ log α ∣ , α = β → 0 , (15)
where T o r T_{\mathrm{or}} T or denotes the stopping time of the oracle procedure, and α \alpha α and β \beta β are the prescribed error-probability bounds in (4). Here and below, E [ 2 ] \mathbb{E}_{[2]} E [ 2 ] denotes expectation when both sources are anomalous with the particular densities specified in (12). The corresponding information numbers are
I 1 : = D ( f 1 , f 0 ) = 1 2 σ 2 + log π − 1 2 , I 2 : = D ( f 2 , f 0 ) = ϵ 2 2 σ 2 + log π − 1 2 . (16) \begin{aligned}
I_1&:=D(f_1,f_0)=\frac{1}{2\sigma^2}+\frac{\log\pi-1}{2},\\
I_2&:=D(f_2,f_0)=\frac{\epsilon^2}{2\sigma^2}+\frac{\log\pi-1}{2}. \tag*{(16)}
\end{aligned} I 1 I 2 := D ( f 1 , f 0 ) = 2 σ 2 1 + 2 log π − 1 , := D ( f 2 , f 0 ) = 2 σ 2 ϵ 2 + 2 log π − 1 . (16)
For each fixed ϵ > 0 \epsilon>0 ϵ > 0 , sufficiently accurate estimation of the mean is enough to distinguish the two hypotheses with high probability. However, the mean alone does not determine the information numbers needed to compute the optimal sampling frequencies. Consequently, treating a Laplace alternative as Gaussian can lead to an inefficient sampling allocation.
To illustrate this effect, suppose that a practitioner replaces (11) by the simplified Gaussian problem
\begin{aligned}
H_i^{1,\mathrm{s}} &: f_i\in\{\mathcal{N}(\mu,\sigma^2):\mu\in[\epsilon,B]\},\\
H_i^{0,\mathrm{s}} &: f_i=f_0,
\end{aligned}
\qquad i\in[2]. \tag*{(17)}\end{aligned}
Let
f i s : = N ( μ i , σ 2 ) , i ∈ [ 2 ] , f_i^{\mathrm{s}}:=\mathcal{N}(\mu_i,\sigma^2),\qquad i\in[2], f i s := N ( μ i , σ 2 ) , i ∈ [ 2 ] ,
denote the Gaussian surrogate densities with the correct means. Under the true densities, the surrogate log-likelihood ratios have mean increments
I 1 s : = E f 1 [ log f 1 s ( X ) f 0 ( X ) ] = 1 2 σ 2 , I_1^{\mathrm{s}}:=\mathbb{E}_{f_1}\left[\log\frac{f_1^{\mathrm{s}}(X)}{f_0(X)}\right]=\frac{1}{2\sigma^2}, I 1 s := E f 1 [ log f 0 ( X ) f 1 s ( X ) ] = 2 σ 2 1 ,
I 2 s : = E f 2 [ log f 2 s ( X ) f 0 ( X ) ] = ϵ 2 2 σ 2 . I_2^{\mathrm{s}}:=\mathbb{E}_{f_2}\left[\log\frac{f_2^{\mathrm{s}}(X)}{f_0(X)}\right]=\frac{\epsilon^2}{2\sigma^2}. I 2 s := E f 2 [ log f 0 ( X ) f 2 s ( X ) ] = 2 σ 2 ϵ 2 .
Let c 1 s c_1^{\mathrm{s}} c 1 s and c 2 s c_2^{\mathrm{s}} c 2 s denote the sampling frequencies computed from these surrogate information numbers, so that
c 1 s I 1 s = c 2 s I 2 s , c 1 s + c 2 s = 1. c_1^{\mathrm{s}}I_1^{\mathrm{s}}=c_2^{\mathrm{s}}I_2^{\mathrm{s}},\qquad c_1^{\mathrm{s}}+c_2^{\mathrm{s}}=1. c 1 s I 1 s = c 2 s I 2 s , c 1 s + c 2 s = 1.
For this illustrative comparison, we use the first-order stopping-time approximation associated with these surrogate information rates:
E [ 2 ] [ T s ] ∼ ∣ log α ∣ c 1 s I 1 s , α = β → 0 , (18) \mathbb{E}_{[2]}[T_{\mathrm{s}}]\sim\frac{|\log\alpha|}{c_1^{\mathrm{s}}I_1^{\mathrm{s}}},\qquad\alpha=\beta\to0, \tag*{(18)} E [ 2 ] [ T s ] ∼ c 1 s I 1 s ∣ log α ∣ , α = β → 0 , (18)
for each fixed ϵ > 0 \epsilon> 0 ϵ > 0 , where T S T_S T S denotes the stopping time of the surrogate procedure.
Under these stopping-time asymptotics, for each fixed ϵ > 0 \epsilon> 0 ϵ > 0 , the ratio of the expected stopping times satisfies
R ( ϵ ) : = lim α → 0 E [ 2 ] [ T S ] E [ 2 ] [ T o r ] = c 1 ( ϵ ) I 1 c 1 S ( ϵ ) I 1 S , (19) \mathcal{R}(\epsilon) := \lim_{\alpha\to0} \frac{\mathbb{E}_{[2]}[T_S]}{\mathbb{E}_{[2]}[T_{\mathrm{or}}]} = \frac{c_1(\epsilon) I_1}{c_1^S(\epsilon) I_1^S}, \tag*{(19)} R ( ϵ ) := α → 0 lim E [ 2 ] [ T or ] E [ 2 ] [ T S ] = c 1 S ( ϵ ) I 1 S c 1 ( ϵ ) I 1 , (19)
where we have made the dependence of the sampling frequencies on ϵ \epsilon ϵ explicit. The quantities I 1 I_1 I 1 and I 1 S I_1^S I 1 S are fixed and strictly positive. Moreover, as ϵ ↓ 0 \epsilon\downarrow0 ϵ ↓ 0 , c 1 ( ϵ ) c_1(\epsilon) c 1 ( ϵ ) converges to a strictly positive limit, whereas c 1 S ( ϵ ) c_1^S(\epsilon) c 1 S ( ϵ ) converges to zero. It follows that
R ( ϵ ) ⟶ ∞ as ϵ ↓ 0. \mathcal{R}(\epsilon) \longrightarrow\infty\quad\text{as } \epsilon\downarrow0. R ( ϵ ) ⟶ ∞ as ϵ ↓ 0.
As ϵ \epsilon ϵ decreases, the Gaussian surrogate treats the second source as increasingly difficult to distinguish from the null and allocates nearly all observations to it. However, the true Laplace alternative remains separated from the Gaussian null through its distributional shape, so the optimal allocation continues to sample each source with a positive limiting frequency. Thus, the first-order stopping-time comparison illustrates how ignoring distributional shape can lead to an arbitrarily large loss of efficiency.
Universal asymptotic lower bound
In this section, we derive the universal asymptotic lower bound on J f ( α , β , K ) \mathcal{J}_{f}(\alpha,\beta,K) J f ( α , β , K ) as α , β → 0 \alpha,\beta\to0 α , β → 0 under the scaling condition (9). For any A ⊆ [ M ] A \subseteq[M] A ⊆ [ M ] with A ≠ ∅ A \ne\varnothing A = ∅ , we define
I A ∗ ( f ) : = min i ∈ A I i ( f i ) , K ^ A ( f ) : = ∑ i ∈ A I A ∗ ( f ) I i ( f i ) . I_A^*(f) := \min_{i \in A} I_i(f_i), \qquad\widehat{K}_A(f) := \sum_{i \in A} \frac{I_A^*(f)}{I_i(f_i)}. I A ∗ ( f ) := i ∈ A min I i ( f i ) , K A ( f ) := i ∈ A ∑ I i ( f i ) I A ∗ ( f ) .
and for any A ⊆ [ M ] A \subseteq[M] A ⊆ [ M ] with A ≠ [ M ] A \ne[M] A = [ M ] ,
J A ∗ ( f ) : = min i ∉ A J i ( f i ) , K ~ A ( f ) : = ∑ i ∉ A J A ∗ ( f ) J i ( f i ) . J_A^*(f) := \min_{i \notin A} J_i(f_i), \qquad\widetilde{K}_A(f) := \sum_{i \notin A} \frac{J_A^*(f)}{J_i(f_i)}. J A ∗ ( f ) := i ∈ / A min J i ( f i ) , K A ( f ) := i ∈ / A ∑ J i ( f i ) J A ∗ ( f ) .
The quantities I A ∗ ( f ) I_A^*(f) I A ∗ ( f ) and J A ∗ ( f ) J_A^*(f) J A ∗ ( f ) represent the minimum Kullback–Leibler divergences over the sets { I i ( f i ) : i ∈ A } \{I_i(f_i): i \in A\} { I i ( f i ) : i ∈ A } and { J i ( f i ) : i ∉ A } \{J_i(f_i): i \notin A\} { J i ( f i ) : i ∈ / A } , respectively. For 0 < ∣ A ∣ < M 0 < |A| < M 0 < ∣ A ∣ < M , we define the ratio
s A ∗ ( f ) : = I A ∗ ( f ) J A ∗ ( f ) . s_A^*(f) := \frac{I_A^*(f)}{J_A^*(f)}. s A ∗ ( f ) := J A ∗ ( f ) I A ∗ ( f ) .
In the following theorem, we present the Universal Asymptotic Lower Bound (UALB), expressed in terms of factors x A ( f ) , y A ( f ) ∈ [ 0 , 1 ] x_A(f), y_A(f) \in[0,1] x A ( f ) , y A ( f ) ∈ [ 0 , 1 ] . The terms K ^ A ( f ) \widehat{K}_A(f) K A ( f ) and K ~ A ( f ) \widetilde{K}_A(f) K A ( f ) can be interpreted as the effective sampling effort required over the sources in A A A and A c A^c A c , respectively, for achieving x A ( f ) = 1 x_A(f) = 1 x A ( f ) = 1 and y A ( f ) = 1 y_A(f) = 1 y A ( f ) = 1 . In addition to characterizing the UALB, the quantities x A ( f ) x_A(f) x A ( f ) and y A ( f ) y_A(f) y A ( f ) play a key role in the design of asymptotically optimal sampling rules, as will be seen in following section.
Theorem 5.1: Let A ⊆ [ M ] A \subseteq[M] A ⊆ [ M ] and f ∈ Φ A f \in\Phi_A f ∈ Φ A .
(i) If ∣ A ∣ = 0 |A| = 0 ∣ A ∣ = 0 , then as α , β → 0 \alpha, \beta\to0 α , β → 0 ,
J f ( α , β , K ) ≳ ∣ log β ∣ y A ( f ) J A ∗ ( f ) . \mathcal{J}_{f}(\alpha, \beta, K) \gtrsim\frac{|\log\beta|}{y_{A}(f)J_{A}^{*}(f)}. J f ( α , β , K ) ≳ y A ( f ) J A ∗ ( f ) ∣ log β ∣ .
where
y A ( f ) : = ( K K ~ A ( f ) ) ∧ 1 , x A ( f ) : = 0. y_{A}(f) := \left(\frac{K}{\widetilde{K}_{A}(f)}\right) \wedge1,\qquad x_{A}(f) := 0. y A ( f ) := ( K A ( f ) K ) ∧ 1 , x A ( f ) := 0.
(ii) If 0 < ∣ A ∣ < M 0 < |A| < M 0 < ∣ A ∣ < M , then as α , β → 0 \alpha, \beta\to0 α , β → 0 under (9),
J f ( α , β , K ) ≳ ∣ log α ∣ x A ( f ) J A ∗ ( f ) ∼ ∣ log β ∣ y A ( f ) J A ∗ ( f ) . \mathcal{J}_{f}(\alpha, \beta, K) \gtrsim\frac{|\log\alpha|}{x_{A}(f)J_{A}^{*}(f)} \sim\frac{|\log\beta|}{y_{A}(f)J_{A}^{*}(f)}. J f ( α , β , K ) ≳ x A ( f ) J A ∗ ( f ) ∣ log α ∣ ∼ y A ( f ) J A ∗ ( f ) ∣ log β ∣ .
where
x A ( f ) : = K K ^ A ( f ) + ( s A ∗ ( f ) / r ) K ~ A ( f ) ∧ r s A ∗ ( f ) ∧ 1 , x_{A}(f) := \frac{K}{\widehat{K}_{A}(f) + (s_{A}^{*}(f)/r)\widetilde{K}_{A}(f)} \wedge\frac{r}{s_{A}^{*}(f)} \wedge1, x A ( f ) := K A ( f ) + ( s A ∗ ( f ) / r ) K A ( f ) K ∧ s A ∗ ( f ) r ∧ 1 ,
y A ( f ) : = K K ~ A ( f ) + ( r / s A ∗ ( f ) ) K ^ A ( f ) ∧ s A ∗ ( f ) r ∧ 1. y_{A}(f) := \frac{K}{\widetilde{K}_{A}(f) + (r/s_{A}^{*}(f))\widehat{K}_{A}(f)} \wedge\frac{s_{A}^{*}(f)}{r} \wedge1. y A ( f ) := K A ( f ) + ( r / s A ∗ ( f )) K A ( f ) K ∧ r s A ∗ ( f ) ∧ 1.
(iii) If ∣ A ∣ = M |A| = M ∣ A ∣ = M , then as α , β → 0 \alpha, \beta\to0 α , β → 0 ,
J f ( α , β , K ) ≳ ∣ log α ∣ x A ( f ) J A ∗ ( f ) + y A ( f ) J A ∗ ( f ) . \mathcal{J}_{f}(\alpha, \beta, K) \gtrsim\frac{|\log\alpha|}{x_{A}(f)J_{A}^{*}(f) + y_{A}(f)J_{A}^{*}(f)}. J f ( α , β , K ) ≳ x A ( f ) J A ∗ ( f ) + y A ( f ) J A ∗ ( f ) ∣ log α ∣ .
where
x A ( f ) : = ( K K ^ A ( f ) ) ∧ 1 , y A ( f ) : = 0. x_{A}(f) := \left(\frac{K}{\widehat{K}_{A}(f)}\right) \wedge1,\qquad y_{A}(f) := 0. x A ( f ) := ( K A ( f ) K ) ∧ 1 , y A ( f ) := 0.
Proof: Fix A ⊆ [ M ] A \subseteq[M] A ⊆ [ M ] and f ∈ Φ A f \in\Phi_{A} f ∈ Φ A . We distinguish two cases, depending on whether the infima in (1)–(2) are attained.
Case 1: Suppose that for each i ∈ A i \in A i ∈ A (resp. i ∈ A c i \in A^{c} i ∈ A c ) there exists a function g i ∗ g_{i}^{*} g i ∗ that attains the infimum in (1) (resp. (2)). Let g ∗ : = ( g 1 ∗ , … , g M ∗ ) \boldsymbol{g}^{*} := (g_{1}^{*}, \ldots, g_{M}^{*}) g ∗ := ( g 1 ∗ , … , g M ∗ ) . Consider the following M M M -binary (simple vs. simple) hypothesis testing problem:
{ X i ( n ) : n ∈ N } ∼ f i vs. { X i ( n ) : n ∈ N } ∼ g i ∗ , i ∈ [ M ] . (20) \{X_{i}(n): n \in\mathbb{N}\} \sim f_{i}\quad\text{vs.}\quad\{X_{i}(n): n \in\mathbb{N}\} \sim g_{i}^{*},\qquad i \in[M]. \tag*{(20)} { X i ( n ) : n ∈ N } ∼ f i vs. { X i ( n ) : n ∈ N } ∼ g i ∗ , i ∈ [ M ] . (20)
Let C s i m p l e f , g ∗ ( α , β , K ) C_{\mathrm{simple}}^{f,\boldsymbol{g}^{*}}(\alpha, \beta, K) C simple f , g ∗ ( α , β , K ) denote the class of policies satisfying the error constraint (4) and the sampling constraint (6) for the problem (20). Any policy ( R , T , Δ ) ∈ C ( α , β , K ) (R,T,\Delta) \in C(\alpha, \beta, K) ( R , T , Δ ) ∈ C ( α , β , K ) for the composite problem also satisfies the same constraints for the simple problem (20). Therefore,
C ( α , β , K ) ⊆ C s i m p l e f , g ∗ ( α , β , K ) . (21) C(\alpha, \beta, K) \subseteq C_{\mathrm{simple}}^{f,\boldsymbol{g}^{*}}(\alpha, \beta, K). \tag*{(21)} C ( α , β , K ) ⊆ C simple f , g ∗ ( α , β , K ) . (21)
By [31 Theorem 5] , for any ( R , T , Δ ) ∈ C s i m p l e f , g ∗ ( α , β , K ) (R,T,\Delta) \in C_{\mathrm{simple}}^{f,\boldsymbol{g}^{*}}(\alpha, \beta, K) ( R , T , Δ ) ∈ C simple f , g ∗ ( α , β , K ) , we have E f [ T ] E_{f}[T] E f [ T ] asymptotically lower bounded by the UALB in Theorem 5.1. Combining this with (21) yields the desired result.
Case 2: Suppose that for some i ∈ [ M ] i \in[M] i ∈ [ M ] , the infimum in (1) or (2) is not attained. Under the separation of hypotheses assumption, for any ϵ > 0 \epsilon> 0 ϵ > 0 there exists g i ϵ ∈ Φ i g_i^{\epsilon} \in\Phi_i g i ϵ ∈ Φ i such that
I i ( f i ) < D ( f i , g i ϵ ) ≤ ( 1 + ϵ ) I i ( f i ) , i ∈ A , I_i(f_i) < D(f_i,g_i^{\epsilon}) \le(1+\epsilon)I_i(f_i), \qquad i \in A, I i ( f i ) < D ( f i , g i ϵ ) ≤ ( 1 + ϵ ) I i ( f i ) , i ∈ A ,
J i ( f i ) < D ( f i , g i ϵ ) ≤ ( 1 + ϵ ) J i ( f i ) , i ∈ A c . J_i(f_i) < D(f_i,g_i^{\epsilon}) \le(1+\epsilon)J_i(f_i), \qquad i \in A^c. J i ( f i ) < D ( f i , g i ϵ ) ≤ ( 1 + ϵ ) J i ( f i ) , i ∈ A c .
Let g ϵ : = ( g 1 ϵ , … , g M ϵ ) \mathbf{g}^{\epsilon} := (g_1^{\epsilon},\ldots,g_M^{\epsilon}) g ϵ := ( g 1 ϵ , … , g M ϵ ) and consider the corresponding simple testing problem,
{ X i ( n ) : n ∈ N } ∼ f i vs { X i ( n ) : n ∈ N } ∼ g i ϵ , i ∈ [ M ] . \{X_i(n): n \in\mathbb{N}\} \sim f_i \quad\text{vs} \quad\{X_i(n): n \in\mathbb{N}\} \sim g_i^{\epsilon}, \qquad i \in[M]. { X i ( n ) : n ∈ N } ∼ f i vs { X i ( n ) : n ∈ N } ∼ g i ϵ , i ∈ [ M ] .
Repeating the argument of Case 1, we obtain
E f [ T ] ≳ ( 1 − o ( ϵ ) ) UALB , \mathbb{E}_{f}[T] \gtrsim(1-o(\epsilon))\operatorname{UALB}, E f [ T ] ≳ ( 1 − o ( ϵ )) UALB ,
since I i ( f i ) I_i(f_i) I i ( f i ) and J i ( f i ) J_i(f_i) J i ( f i ) are replaced by D ( f i , g i ϵ ) D(f_i,g_i^{\epsilon}) D ( f i , g i ϵ ) . Letting ϵ → 0 \epsilon\to0 ϵ → 0 completes the proof.
A family of policies
In this section, we present a family of policies ( R , T , Δ ) (R,T,\Delta) ( R , T , Δ ) that, under an appropriate design, achieve first-order asymptotically optimal performance (8) by attaining the UALB derived in § 5 .
Test Statistics and Estimators
Throughout this work, we denote by Λ i R ( n ) \Lambda_i^R(n) Λ i R ( n ) the test statistic for source i ∈ [ M ] i \in[M] i ∈ [ M ] , based on the observations collected from source i i i up to time n n n under the sampling rule R R R . The test statistic Λ i R ( n ) \Lambda_i^R(n) Λ i R ( n ) is designed to quantify the evidence in favor of the hypothesis that source i i i is anomalous. In particular, positive values of Λ i R ( n ) \Lambda_i^R(n) Λ i R ( n ) indicate that i ∈ A i \in A i ∈ A , whereas negative values indicate that i ∉ A i \notin A i ∈ / A . Thus, Λ i R ( n ) \Lambda_i^R(n) Λ i R ( n ) serves as a sequential decision statistic, analogous to a log-likelihood ratio in classical sequential testing.
We also denote by f ^ n \hat{f}_n f ^ n the estimator of the true density vector at time n n n , constructed from the observations up to time n n n . The estimator approximates the unknown densities governing each source and is particularly important in the nonparametric setting where the density models are not known a priori.
The test statistics we apply are the Generalized Log-Likelihood Ratio (GLLR) and the Adaptive Log-Likelihood Ratio (ALLR). The key difference is that GLLR is defined via maximization over the hypothesis sets, whereas ALLR is constructed recursively using sequential estimates of the underlying densities. The density estimators we apply are the Maximum Likelihood Estimator (MLE), and the Kernel Density Estimator (KDE).
Stopping and Decision rules
The stopping rule is defined as
T R : = inf { n ≥ 1 : Λ i R ( n ) ∉ ( − a , b ) , ∀ i ∈ [ M ] } , (22) T^{R} := \inf\{n \ge1 : \Lambda_i^{R}(n) \notin(-a,b),\ \forall i \in[M]\}, \tag*{(22)} T R := inf { n ≥ 1 : Λ i R ( n ) ∈ / ( − a , b ) , ∀ i ∈ [ M ]} , (22)
i.e., sampling continues as long as there exists i ∈ [ M ] i \in[M] i ∈ [ M ] such that Λ i R ( n ) ∈ ( − a , b ) \Lambda_i^{R}(n) \in(-a,b) Λ i R ( n ) ∈ ( − a , b ) . At each time n n n , the estimate of the anomalous set is given by
Δ R ( n ) : = { i ∈ [ M ] : Λ i R ( n ) > 0 } , n ∈ N , (23) \Delta^{R}(n) := \{i \in[M] : \Lambda_i^{R}(n) > 0\},\quad n \in\mathbb{N}, \tag*{(23)} Δ R ( n ) := { i ∈ [ M ] : Λ i R ( n ) > 0 } , n ∈ N , (23)
and the final decision is Δ R ( T R ) \Delta^{R}(T^{R}) Δ R ( T R ) . We assume that the test statistics, the estimator, and the thresholds a , b a, b a , b are chosen so that ( R , T R , Δ R ) (R,T^{R},\Delta^{R}) ( R , T R , Δ R ) satisfies the error constraint (4). In addition, for asymptotic optimality, we require that
a ∼ ∣ log ( β ) ∣ , b ∼ ∣ log ( α ) ∣ , as α , β → 0. (24) a \sim|\log(\beta)|,\qquad b \sim|\log(\alpha)|,\quad\text{as }\alpha,\beta\to0. \tag*{(24)} a ∼ ∣ log ( β ) ∣ , b ∼ ∣ log ( α ) ∣ , as α , β → 0. (24)
Accordingly, any proposed choice of test statistic and estimator must be shown to satisfy these conditions.
Probabilistic Sampling Rules
We introduce a family of sampling rules, referred to as probabilistic sampling rules . Under an appropriate design, these rules guarantee that the policy ( R , T R , Δ R ) (R,T^{R},\Delta^{R}) ( R , T R , Δ R ) , with T R T^{R} T R and Δ R \Delta^{R} Δ R defined in (22)–(23), achieves the first-order asymptotic optimality in (8) by attaining the UALB.
A sampling rule R R R is called probabilistic if, for each i ∈ [ M ] i \in[M] i ∈ [ M ] , there exists a measurable function
c i R : Φ → [ 0 , 1 ] , c_i^{R}:\Phi\to[0,1], c i R : Φ → [ 0 , 1 ] ,
where c i R ( g ) c_i^{R}(g) c i R ( g ) denotes the probability of sampling source i i i at time n + 1 n+1 n + 1 , given that f ^ n = g \hat{f}_n=g f ^ n = g . Specifically,
R i ( n + 1 ) = 1 { Z i ( n ) ≤ c i R ( f ^ n ) } , i ∈ [ M ] , n ∈ N , (25) R_i(n+1)=\mathbf{1}\left\{Z_i(n)\le c_i^{R}(\hat{f}_n)\right\},\qquad i\in[M],\quad n\in\mathbb{N}, \tag*{(25)} R i ( n + 1 ) = 1 { Z i ( n ) ≤ c i R ( f ^ n ) } , i ∈ [ M ] , n ∈ N , (25)
where { Z i ( n ) : i ∈ [ M ] , n ∈ N } \{Z_i(n): i\in[M],\ n\in\mathbb{N}\} { Z i ( n ) : i ∈ [ M ] , n ∈ N } are iid uniform random variables on [ 0 , 1 ] [0,1] [ 0 , 1 ] , independent of the observations and the past sampling decisions.
We next provide a sufficient condition under which a probabilistic sampling rule satisfies the sampling constraint (6).
Proposition 6.1: If R R R is a probabilistic sampling rule such that for all g ∈ Φ g\in\Phi g ∈ Φ ,
∑ i = 1 M c i R ( g ) ≤ K , (26) \sum_{i=1}^{M} c_i^{R}(g)\le K, \tag*{(26)} i = 1 ∑ M c i R ( g ) ≤ K , (26) then (6) holds for any { F n R } \{\mathcal{F}^{R}_{n}\} { F n R } -stopping time T T T .
Proof: For any n ∈ N n \in\mathbb{N} n ∈ N ,
E [ ∣ R ( n ) ∣ ∣ F n − 1 R ] = ∑ i = 1 M P ( R i ( n ) = 1 ∣ F n − 1 R ) = ∑ i = 1 M c i R ( f ^ n − 1 ) ≤ K . \mathbb{E}\left[|R(n)| \mid\mathcal{F}^{R}_{n-1}\right] = \sum_{i=1}^{M} \mathbb{P}\left(R_i(n)=1 \mid\mathcal{F}^{R}_{n-1}\right) = \sum_{i=1}^{M} c_i^R\left(\hat{f}_{n-1}\right) \le K. E [ ∣ R ( n ) ∣ ∣ F n − 1 R ] = i = 1 ∑ M P ( R i ( n ) = 1 ∣ F n − 1 R ) = i = 1 ∑ M c i R ( f ^ n − 1 ) ≤ K .
Lemma 6.1. Let A ⊆ [ M ] A \subseteq[M] A ⊆ [ M ] , f ∈ Φ A f \in\Phi_A f ∈ Φ A , and let R R R be a probabilistic sampling rule. If, for each i ∈ [ M ] i \in[M] i ∈ [ M ] ,
min D ⊆ [ M ] inf g ∈ Φ D c i R ( g ) > 0 , (27) \min_{D \subseteq[M]} \inf_{g \in\Phi_D} c_i^R(g) > 0, \tag*{(27)} D ⊆ [ M ] min g ∈ Φ D inf c i R ( g ) > 0 , (27) then there exists ρ ∈ ( 0 , 1 ) \rho\in(0,1) ρ ∈ ( 0 , 1 ) such that, for every i ∈ [ M ] i \in[M] i ∈ [ M ] , the sequence
{ P f ( π i R ( n ) < ρ ) : n ∈ N } \left\{\mathbb{P}_f\left(\pi_i^R(n)<\rho\right): n \in\mathbb{N}\right\} { P f ( π i R ( n ) < ρ ) : n ∈ N } is exponentially decaying.
Proof: Fix A ⊆ [ M ] A \subseteq[M] A ⊆ [ M ] , f ∈ Φ A f \in\Phi_A f ∈ Φ A , a sampling rule R R R , and i ∈ [ M ] i \in[M] i ∈ [ M ] . Define
π ~ i R ( n ) : = 1 n ∑ m = 1 n ( R i ( m ) − c i R ( f ^ m − 1 ) ) = π i R ( n ) − 1 n ∑ m = 1 n c i R ( f ^ m − 1 ) ≤ π i R ( n ) − ζ , (28) \widetilde{\pi}_i^R(n) := \frac{1}{n}\sum_{m=1}^{n}\left(R_i(m)-c_i^R\left(\hat{f}_{m-1}\right)\right)=\pi_i^R(n)-\frac{1}{n}\sum_{m=1}^{n}c_i^R\left(\hat{f}_{m-1}\right)\le\pi_i^R(n)-\zeta, \tag*{(28)} π i R ( n ) := n 1 m = 1 ∑ n ( R i ( m ) − c i R ( f ^ m − 1 ) ) = π i R ( n ) − n 1 m = 1 ∑ n c i R ( f ^ m − 1 ) ≤ π i R ( n ) − ζ , (28)
where
ζ : = min D ⊆ [ M ] inf g ∈ Φ D c i R ( g ) > 0. \zeta:= \min_{D \subseteq[M]} \inf_{g \in\Phi_D} c_i^R(g)>0. ζ := D ⊆ [ M ] min g ∈ Φ D inf c i R ( g ) > 0.
Fix ϵ ∈ ( 0 , ζ ) \epsilon\in(0,\zeta) ϵ ∈ ( 0 , ζ ) . For any ρ ∈ ( 0 , ζ − ϵ ) \rho\in(0,\zeta-\epsilon) ρ ∈ ( 0 , ζ − ϵ ) , (28) implies
P f ( π i R ( n ) < ρ ) ≤ P f ( π ~ i R ( n ) < ρ − ζ ) ≤ P f ( ∑ m = 1 n ( R i ( m ) − c i R ( f ^ m − 1 ) ) < − ϵ n ) . \mathbb{P}_f\left(\pi_i^R(n)<\rho\right)\le\mathbb{P}_f\left(\widetilde{\pi}_i^R(n)<\rho-\zeta\right)\le\mathbb{P}_f\left(\sum_{m=1}^{n}\left(R_i(m)-c_i^R\left(\hat{f}_{m-1}\right)\right)<-\epsilon n\right). P f ( π i R ( n ) < ρ ) ≤ P f ( π i R ( n ) < ρ − ζ ) ≤ P f ( m = 1 ∑ n ( R i ( m ) − c i R ( f ^ m − 1 ) ) < − ϵ n ) .
By (25) ,
E f [ R i ( n ) − c i R ( f ^ n − 1 ) ∣ F n − 1 R ] = P f ( R i ( n ) = 1 ∣ F n − 1 R ) − c i R ( f ^ n − 1 ) = 0. \mathbb{E}_f\left[R_i(n)-c_i^R\left(\hat{f}_{n-1}\right)\mid\mathcal{F}^{R}_{n-1}\right]=\mathbb{P}_f\left(R_i(n)=1\mid\mathcal{F}^{R}_{n-1}\right)-c_i^R\left(\hat{f}_{n-1}\right)=0. E f [ R i ( n ) − c i R ( f ^ n − 1 ) ∣ F n − 1 R ] = P f ( R i ( n ) = 1 ∣ F n − 1 R ) − c i R ( f ^ n − 1 ) = 0.
Therefore,
{ R i ( n ) − c i R ( f ^ n − 1 ) : n ≥ 1 } \left\{R_i(n)-c_i^R\left(\hat{f}_{n-1}\right):n\ge1\right\} { R i ( n ) − c i R ( f ^ n − 1 ) : n ≥ 1 }
is a martingale difference sequence, uniformly bounded by 1. The result follows from the Azuma–Hoeffding inequality.
Asymptotic Optimality
In view of the UALB in Theorem 5.1 , we now provide sufficient conditions under which a policy ( R , T R , Δ R ) (R,T^R,\Delta^R) ( R , T R , Δ R ) attains this bound and therefore satisfies the first-order asymptotic optimality criterion in (8) . We first introduce the target sampling frequencies { c i ∗ ( f ) : i ∈ [ M ] } \{c_i^*(\mathbf{f}):i\in[M]\} { c i ∗ ( f ) : i ∈ [ M ]} through
c i ∗ ( f ) I i ( f i ) = x A ( f ) I A ∗ ( f ) , i ∈ A , c i ∗ ( f ) J i ( f i ) = y A ( f ) J A ∗ ( f ) , i ∉ A . (29) \begin{aligned}
c_i^*(\mathbf{f})I_i(f_i)&=x_A(\mathbf{f})I_A^*(\mathbf{f}), && i\in A,\\
c_i^*(\mathbf{f})J_i(f_i)&=y_A(\mathbf{f})J_A^*(\mathbf{f}), && i\notin A. \tag*{(29)}
\end{aligned} c i ∗ ( f ) I i ( f i ) c i ∗ ( f ) J i ( f i ) = x A ( f ) I A ∗ ( f ) , = y A ( f ) J A ∗ ( f ) , i ∈ A , i ∈ / A . (29)
where x A ( f ) x_A(\mathbf{f}) x A ( f ) and y A ( f ) y_A(\mathbf{f}) y A ( f ) are defined in Theorem 5.1 . By the definition of r r r in (9) and the expressions for x A ( f ) x_A(\mathbf{f}) x A ( f ) and y A ( f ) y_A(\mathbf{f}) y A ( f ) , whenever 0 < ∣ A ∣ < M 0<|A|<M 0 < ∣ A ∣ < M ,
x A ( f ) I A ∗ ( f ) = r y A ( f ) J A ∗ ( f ) . x_A(\mathbf{f})I_A^*(\mathbf{f})=r y_A(\mathbf{f})J_A^*(\mathbf{f}). x A ( f ) I A ∗ ( f ) = r y A ( f ) J A ∗ ( f ) .
Together with (29) , this implies
c i ∗ ( f ) I i ( f i ) = r c j ∗ ( f ) J j ( f j ) , i ∈ A , j ∉ A . c_i^*(\mathbf{f})I_i(f_i)=r c_j^*(\mathbf{f})J_j(f_j),\qquad i\in A,\ j\notin A. c i ∗ ( f ) I i ( f i ) = r c j ∗ ( f ) J j ( f j ) , i ∈ A , j ∈ / A .
Thus, the target frequencies balance the information accumulation rates within each group and relate the rates of the two groups through the factor r r r .
The assumptions (1)–(2) ensure that the information numbers in (29) are strictly positive and finite. Suppose, in addition, that the target sampling frequencies satisfy the uniform lower bound
min D ⊆ [ M ] inf g ∈ Φ D c i ∗ ( g ) > 0 , ∀ i ∈ [ M ] , \min_{D\subseteq[M]}\inf_{\mathbf{g}\in\Phi_D}c_i^*(\mathbf{g})>0,\qquad\forall i\in[M], D ⊆ [ M ] min g ∈ Φ D inf c i ∗ ( g ) > 0 , ∀ i ∈ [ M ] ,
then any sampling rule satisfying
c i R ( g ) ≥ c i ∗ ( g ) , ∀ i ∈ [ M ] , ∀ D ⊆ [ M ] , ∀ g ∈ Φ D , c_i^R(\mathbf{g})\ge c_i^*(\mathbf{g}),\qquad\forall i\in[M],\ \forall D\subseteq[M],\ \forall\mathbf{g}\in\Phi_D, c i R ( g ) ≥ c i ∗ ( g ) , ∀ i ∈ [ M ] , ∀ D ⊆ [ M ] , ∀ g ∈ Φ D ,
also has sampling probabilities uniformly bounded away from zero. According to Lemma 6.1, it follows that the empirical sampling frequency of every source remains bounded away from zero with high probability.
The following theorem establishes asymptotic optimality under a suitable asymptotic growth condition on the test statistics. This condition ensures that evidence accumulates at rates sufficient to attain the UALB, as determined by the target sampling frequencies { c i ∗ ( f ) : i ∈ [ M ] } \{c_i^*(\mathbf{f}):i\in[M]\} { c i ∗ ( f ) : i ∈ [ M ]} and the corresponding information numbers.
Theorem 7.1. Let A ⊆ [ M ] A\subseteq[M] A ⊆ [ M ] , f ∈ Φ A \mathbf{f}\in\Phi_A f ∈ Φ A , and let R R R be a sampling rule that satisfies (6) with T = T R T=T^R T = T R , defined in (22) with thresholds a , b a,b a , b satisfying (24). If for any ϵ > 0 \epsilon>0 ϵ > 0 the test statistics satisfy
∑ n = 1 ∞ P f ( − 1 n Λ i R ( n ) < ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ ) < ∞ , ∀ i ∈ A , \sum_{n=1}^{\infty}\mathbb{P}_{\mathbf{f}}\left(-\frac{1}{n}\Lambda_i^R(n)<(c_i^*(\mathbf{f})-\epsilon)I_i(f_i)-\epsilon\right)<\infty,\qquad\forall i\in A, n = 1 ∑ ∞ P f ( − n 1 Λ i R ( n ) < ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ ) < ∞ , ∀ i ∈ A , ∑ n = 1 ∞ P f ( − 1 n Λ j R ( n ) < ( c j ∗ ( f ) − ϵ ) J j ( f j ) − ϵ ) < ∞ , ∀ j ∉ A , (30) \sum_{n=1}^{\infty}\mathbb{P}_{\mathbf{f}}\left(-\frac{1}{n}\Lambda_j^R(n)<(c_j^*(\mathbf{f})-\epsilon)J_j(f_j)-\epsilon\right)<\infty,\qquad\forall j\notin A, \tag*{(30)} n = 1 ∑ ∞ P f ( − n 1 Λ j R ( n ) < ( c j ∗ ( f ) − ϵ ) J j ( f j ) − ϵ ) < ∞ , ∀ j ∈ / A , (30)
then R R R is asymptotically optimal under P f \mathsf{P}_{\mathbf{f}} P f .
Proof: We prove the case 0 < ∣ A ∣ < M 0<\lvert A\rvert<M 0 < ∣ A ∣ < M , as the cases ∣ A ∣ = 0 \lvert A\rvert=0 ∣ A ∣ = 0 and ∣ A ∣ = M \lvert A\rvert=M ∣ A ∣ = M follow similarly. Fix ϵ > 0 \epsilon>0 ϵ > 0 , and let a , b > 0 a,b>0 a , b > 0 be the thresholds in (22) that satisfy (24). Define
N ϵ ( a , b ) : = max i ∈ A , j ∉ A { a ( c j ∗ ( f ) − ϵ ) J j ( f j ) − ϵ , b ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ } . (31) N_{\epsilon}(a,b):=\max_{i\in A,\,j\notin A}\left\{\frac{a}{(c_j^*(\mathbf{f})-\epsilon)J_j(f_j)-\epsilon},\frac{b}{(c_i^*(\mathbf{f})-\epsilon)I_i(f_i)-\epsilon}\right\}. \tag*{(31)} N ϵ ( a , b ) := i ∈ A , j ∈ / A max { ( c j ∗ ( f ) − ϵ ) J j ( f j ) − ϵ a , ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ b } . (31)
Then
E f [ T R ] ≤ N ϵ ( a , b ) + ∑ n > N ϵ ( a , b ) P f ( T R > n ) . (32) \mathbb{E}_{\mathbf{f}}[T^R]\leq N_{\epsilon}(a,b)+\sum_{n>N_{\epsilon}(a,b)}\mathsf{P}_{\mathbf{f}}(T^R>n). \tag*{(32)} E f [ T R ] ≤ N ϵ ( a , b ) + n > N ϵ ( a , b ) ∑ P f ( T R > n ) . (32)
To prove the claim, it suffices to show that
∑ n > N ϵ ( a , b ) P f ( T R > n ) < ∞ , (33) \sum_{n>N_{\epsilon}(a,b)}\mathsf{P}_{\mathbf{f}}(T^R>n)<\infty, \tag*{(33)} n > N ϵ ( a , b ) ∑ P f ( T R > n ) < ∞ , (33)
then letting a , b → ∞ a,b\to\infty a , b → ∞ in (32), and using (24), we obtain
E f [ T R ] ≲ max i ∈ A , j ∉ A { ∣ log β ∣ ( c j ∗ ( f ) − ϵ ) J j ( f j ) − ϵ , ∣ log α ∣ ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ } . \mathbb{E}_{\mathbf{f}}[T^R]\lesssim\max_{i\in A,\,j\notin A}\left\{\frac{\lvert\log\beta\rvert}{(c_j^*(\mathbf{f})-\epsilon)J_j(f_j)-\epsilon},\frac{\lvert\log\alpha\rvert}{(c_i^*(\mathbf{f})-\epsilon)I_i(f_i)-\epsilon}\right\}. E f [ T R ] ≲ i ∈ A , j ∈ / A max { ( c j ∗ ( f ) − ϵ ) J j ( f j ) − ϵ ∣ log β ∣ , ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ ∣ log α ∣ } .
Letting ϵ → 0 \epsilon\to0 ϵ → 0 , it yields
E f [ T R ] ≲ ∣ log β ∣ y A ( f ) J A ∗ ( f ) ∼ ∣ log α ∣ x A ( f ) I A ∗ ( f ) , \mathbb{E}_{\mathbf{f}}[T^R]\lesssim\frac{\lvert\log\beta\rvert}{y_A(\mathbf{f})J_A^*(\mathbf{f})}\sim\frac{\lvert\log\alpha\rvert}{x_A(\mathbf{f})I_A^*(\mathbf{f})}, E f [ T R ] ≲ y A ( f ) J A ∗ ( f ) ∣ log β ∣ ∼ x A ( f ) I A ∗ ( f ) ∣ log α ∣ ,
which matches the UALB.
In order to prove (33), we note that on the event { T R > n } \{T^R>n\} { T R > n } , there exists either j ∉ A j\notin A j ∈ / A such that Λ j R ( n ) > − a \Lambda_j^R(n)>-a Λ j R ( n ) > − a , or i ∈ A i\in A i ∈ A such that Λ i R ( n ) < b \Lambda_i^R(n)<b Λ i R ( n ) < b . Hence, by the union bound,
P f ( T R > n ) ≤ ∑ j ∉ A P f ( − Λ j R ( n ) < a ) + ∑ i ∈ A P f ( Λ i R ( n ) < b ) . (34) \mathsf{P}_{\mathbf{f}}(T^R>n)\leq\sum_{j\notin A}\mathsf{P}_{\mathbf{f}}(-\Lambda_j^R(n)<a)+\sum_{i\in A}\mathsf{P}_{\mathbf{f}}(\Lambda_i^R(n)<b). \tag*{(34)} P f ( T R > n ) ≤ j ∈ / A ∑ P f ( − Λ j R ( n ) < a ) + i ∈ A ∑ P f ( Λ i R ( n ) < b ) . (34)
For n > N ϵ ( a , b ) n>N_{\epsilon}(a,b) n > N ϵ ( a , b ) , we have
a < n ( ( c j ∗ ( f ) − ϵ ) J j ( f j ) − ϵ ) , b < n ( ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ ) , a<n\bigl((c_j^*(\mathbf{f})-\epsilon)J_j(f_j)-\epsilon\bigr),\qquad b<n\bigl((c_i^*(\mathbf{f})-\epsilon)I_i(f_i)-\epsilon\bigr), a < n ( ( c j ∗ ( f ) − ϵ ) J j ( f j ) − ϵ ) , b < n ( ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ ) ,
and therefore
∑ n > N ϵ ( a , b ) P f ( T R > n ) ≤ ∑ j ∉ A ∑ n > N ϵ ( a , b ) P f ( − 1 n Λ j R ( n ) < ( c j ∗ ( f ) − ϵ ) J j ( f j ) − ϵ ) + ∑ i ∈ A ∑ n > N ϵ ( a , b ) P f ( 1 n Λ i R ( n ) < ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ ) . (35) \begin{aligned}
\sum_{n>N_{\epsilon}(a,b)}\mathsf{P}_{\mathbf{f}}(T^R>n)
&\leq\sum_{j\notin A}\sum_{n>N_{\epsilon}(a,b)}\mathsf{P}_{\mathbf{f}}\left(-\frac{1}{n}\Lambda_j^R(n)<(c_j^*(\mathbf{f})-\epsilon)J_j(f_j)-\epsilon\right)\\
&\quad+\sum_{i\in A}\sum_{n>N_{\epsilon}(a,b)}\mathsf{P}_{\mathbf{f}}\left(\frac{1}{n}\Lambda_i^R(n)<(c_i^*(\mathbf{f})-\epsilon)I_i(f_i)-\epsilon\right). \tag*{(35)}
\end{aligned} n > N ϵ ( a , b ) ∑ P f ( T R > n ) ≤ j ∈ / A ∑ n > N ϵ ( a , b ) ∑ P f ( − n 1 Λ j R ( n ) < ( c j ∗ ( f ) − ϵ ) J j ( f j ) − ϵ ) + i ∈ A ∑ n > N ϵ ( a , b ) ∑ P f ( n 1 Λ i R ( n ) < ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ ) . (35)
By assumption (30), both series converge, which proves (33). ■
Next, we provide sufficient conditions under which a policy ( R , T R , Δ R ) (R,T^R,\Delta^R) ( R , T R , Δ R ) attains the infimum (8). Since the test statistics Λ i R \Lambda_i^R Λ i R depend on both the sampling rule R R R and the estimator, we decompose the asymptotic optimality condition into three components: a condition on the sampling frequencies, a condition on the asymptotic behavior of the test statistics, and a condition on the convergence of the estimator.
The first condition requires that, in the long run, the sampling rule allocates to each source i ∈ [ M ] i \in[M] i ∈ [ M ] a frequency that is at least c i ∗ ( f ) c_i^*(\mathbf{f}) c i ∗ ( f ) , as defined in (29). The second condition imposes a growth requirement on the test statistics in the spirit of a one-sided law of large numbers. For each source and each ϵ > 0 \epsilon> 0 ϵ > 0 , it requires that the probabilities of meeting the stated sampling-frequency bound while falling below the corresponding lower bound on the rate of evidence accumulation be summable. Thus, by the Borel–Cantelli lemma, such shortfalls occur only finitely often almost surely. The third condition controls the rate at which the estimator f ^ n \hat{f}_n f ^ n converges to the true densities. In particular, for each i ∈ [ M ] i \in[M] i ∈ [ M ] and ϵ > 0 \epsilon> 0 ϵ > 0 , define
V ϵ ( f i ) : = { h ∈ Φ i : ∥ f i − h ∥ 1 < ϵ } (36) V_\epsilon(f_i) := \{h \in\Phi_i : \lVert f_i-h\rVert_1 < \epsilon\} \tag*{(36)} V ϵ ( f i ) := { h ∈ Φ i : ∥ f i − h ∥ 1 < ϵ } (36)
τ i R ( ϵ ) : = inf { n ∈ N : f ^ i , m ∈ V ϵ ( f i ) for all m ≥ n } (37) \tau_i^R(\epsilon) := \inf\left\{n \in\mathbb{N} : \hat{f}_{i,m} \in V_\epsilon(f_i)\ \text{for all }m \ge n\right\} \tag*{(37)} τ i R ( ϵ ) := inf { n ∈ N : f ^ i , m ∈ V ϵ ( f i ) for all m ≥ n } (37)
which is the first time after which the estimator remains within ϵ \epsilon ϵ distance from the true density. The requirement is essentially that E f [ τ i R ( ϵ ) ] < ∞ \mathbb{E}_{\mathbf{f}}[\tau_i^R(\epsilon)] < \infty E f [ τ i R ( ϵ )] < ∞ for every ϵ > 0 \epsilon> 0 ϵ > 0 , which is stronger than almost sure uniform convergence and is commonly referred to as quick convergence; see [39 Subsec. 2.4.3] .
Theorem 7.2. Let A ⊆ [ M ] A \subseteq[M] A ⊆ [ M ] , f ∈ Φ A \mathbf{f} \in\Phi_A f ∈ Φ A , and let R R R be a sampling rule that satisfies (6) with T = T R T = T^R T = T R defined in (22) with thresholds a , b a,b a , b satisfying (24). If
(i) for each i ∈ [ M ] i \in[M] i ∈ [ M ] , the functional c i R c_i^R c i R satisfies
c i R ( g ) ≥ c i ∗ ( g ) , (38) c_i^R(\mathbf{g}) \ge c_i^*(\mathbf{g}), \tag*{(38)} c i R ( g ) ≥ c i ∗ ( g ) , (38) for every D ⊆ [ M ] D \subseteq[M] D ⊆ [ M ] and every g ∈ Φ D \mathbf{g} \in\Phi_D g ∈ Φ D ;
(ii) for every ϵ > 0 \epsilon> 0 ϵ > 0 , the test statistics satisfy
∑ n = 1 ∞ P f ( 1 n Λ i R ( n ) < ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ , π i R ( n ) ≥ c i ∗ ( f ) − ϵ ) < ∞ , ∀ i ∈ A , \sum_{n=1}^{\infty} \mathbb{P}_{\mathbf{f}}\left(\frac{1}{n}\Lambda_i^R(n) < (c_i^*(\mathbf{f})-\epsilon)I_i(f_i)-\epsilon,\ \pi_i^R(n) \ge c_i^*(\mathbf{f})-\epsilon\right) < \infty,\quad\forall i \in A, n = 1 ∑ ∞ P f ( n 1 Λ i R ( n ) < ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ , π i R ( n ) ≥ c i ∗ ( f ) − ϵ ) < ∞ , ∀ i ∈ A , ∑ n = 1 ∞ P f ( 1 n Λ j R ( n ) < ( c j ∗ ( f ) − ϵ ) J j ( f j ) − ϵ , π j R ( n ) ≥ c j ∗ ( f ) − ϵ ) < ∞ , ∀ j ∉ A ; (39) \sum_{n=1}^{\infty} \mathbb{P}_{\mathbf{f}}\left(\frac{1}{n}\Lambda_j^R(n) < (c_j^*(\mathbf{f})-\epsilon)J_j(f_j)-\epsilon,\ \pi_j^R(n) \ge c_j^*(\mathbf{f})-\epsilon\right) < \infty,\quad\forall j \notin A; \tag*{(39)} n = 1 ∑ ∞ P f ( n 1 Λ j R ( n ) < ( c j ∗ ( f ) − ϵ ) J j ( f j ) − ϵ , π j R ( n ) ≥ c j ∗ ( f ) − ϵ ) < ∞ , ∀ j ∈ / A ; (39) (iii) for the estimator { f ^ n : n ∈ N } \{\hat{f}_n : n \in\mathbb{N}\} { f ^ n : n ∈ N } , for every ρ ∈ ( 0 , 1 ] \rho\in(0,1] ρ ∈ ( 0 , 1 ] , ϵ > 0 \epsilon> 0 ϵ > 0 ,
∑ n = 1 ∞ n P f ( f ^ i , n ∉ V ϵ ( f i ) , π i R ( n ) ≥ ρ ) < ∞ , ∀ i ∈ [ M ] , (40) \sum_{n=1}^{\infty} n\mathbb{P}_{\mathbf{f}}\left(\hat{f}_{i,n} \notin V_\epsilon(f_i),\ \pi_i^R(n) \ge\rho\right) < \infty,\quad\forall i \in[M], \tag*{(40)} n = 1 ∑ ∞ n P f ( f ^ i , n ∈ / V ϵ ( f i ) , π i R ( n ) ≥ ρ ) < ∞ , ∀ i ∈ [ M ] , (40) then R R R is asymptotically optimal under P f \mathbb{P}_{\mathbf{f}} P f .
Proof. In view of Theorem 7.1, it suffices to verify (30). We prove the condition for i ∈ A i \in A i ∈ A , as the proof for j ∉ A j \notin A j ∈ / A is analogous. For any ϵ > 0 \epsilon> 0 ϵ > 0 ,
P f ( 1 n Λ i R ( n ) < ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ ) ≤ P f ( 1 n Λ i R ( n ) < ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ , π i R ( n ) ≥ c i ∗ ( f ) − ϵ ) + P f ( π i R ( n ) < c i ∗ ( f ) − ϵ ) . \begin{aligned}
\mathbb{P}_{\mathbf{f}}\left(\frac{1}{n}\Lambda_i^R(n) < (c_i^*(\mathbf{f})-\epsilon)I_i(f_i)-\epsilon\right) \\
&\le\mathbb{P}_{\mathbf{f}}\left(\frac{1}{n}\Lambda_i^R(n) < (c_i^*(\mathbf{f})-\epsilon)I_i(f_i)-\epsilon,\ \pi_i^R(n) \ge c_i^*(\mathbf{f})-\epsilon\right) + \mathbb{P}_{\mathbf{f}}\left(\pi_i^R(n) < c_i^*(\mathbf{f})-\epsilon\right).
\end{aligned} P f ( n 1 Λ i R ( n ) < ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ ) ≤ P f ( n 1 Λ i R ( n ) < ( c i ∗ ( f ) − ϵ ) I i ( f i ) − ϵ , π i R ( n ) ≥ c i ∗ ( f ) − ϵ ) + P f ( π i R ( n ) < c i ∗ ( f ) − ϵ ) .
The first term is summable by (39) . It remains to show that
∑ n = 1 ∞ P f ( π i R ( n ) < c i ∗ ( f ) − ϵ ) < ∞ . (41) \sum_{n=1}^{\infty} \mathbb{P}_{f}\left(\pi_i^R(n)<c_i^*(f)-\epsilon\right)<\infty. \tag*{(41)} n = 1 ∑ ∞ P f ( π i R ( n ) < c i ∗ ( f ) − ϵ ) < ∞. (41)
We write
π i R ( n ) = 1 n ∑ m = 1 n ( R i ( m ) − c i R ( f ^ m − 1 ) ) + 1 n ∑ m = 1 n c i R ( f ^ m − 1 ) ≥ 1 n ∑ m = 1 n ( R i ( m ) − c i R ( f ^ m − 1 ) ) + 1 n ∑ m = 1 n c i ∗ ( f ^ m − 1 ) , \begin{aligned}
\pi_i^R(n)=\frac{1}{n}\sum_{m=1}^{n}\left(R_i(m)-c_i^R(\widehat{f}_{m-1})\right)+\frac{1}{n}\sum_{m=1}^{n}c_i^R(\widehat{f}_{m-1}) \\
&\geq\frac{1}{n}\sum_{m=1}^{n}\left(R_i(m)-c_i^R(\widehat{f}_{m-1})\right)+\frac{1}{n}\sum_{m=1}^{n}c_i^*(\widehat{f}_{m-1}),
\end{aligned} π i R ( n ) = n 1 m = 1 ∑ n ( R i ( m ) − c i R ( f m − 1 ) ) + n 1 m = 1 ∑ n c i R ( f m − 1 ) ≥ n 1 m = 1 ∑ n ( R i ( m ) − c i R ( f m − 1 ) ) + n 1 m = 1 ∑ n c i ∗ ( f m − 1 ) ,
where the inequality follows from (38) . Hence,
P f ( π i R ( n ) < c i ∗ ( f ) − ϵ ) ≤ P f ( 1 n ∑ m = 1 n ( R i ( m ) − c i R ( f ^ m − 1 ) ) < − ϵ 2 ) + P f ( 1 n ∑ m = 1 n c i ∗ ( f ^ m − 1 ) < c i ∗ ( f ) − ϵ 2 ) ≤ P f ( ∑ m = 1 n ( R i ( m ) − c i R ( f ^ m − 1 ) ) < − ϵ 2 n ) + P f ( ∑ m = 1 n ( c i ∗ ( f ^ m − 1 ) − c i ∗ ( f ) ) < − ϵ 2 n ) . \begin{aligned}
\mathbb{P}_{f}\left(\pi_i^R(n)<c_i^*(f)-\epsilon\right) \\
&\leq\mathbb{P}_{f}\left(\frac{1}{n}\sum_{m=1}^{n}\left(R_i(m)-c_i^R(\widehat{f}_{m-1})\right)<-\frac{\epsilon}{2}\right)+\mathbb{P}_{f}\left(\frac{1}{n}\sum_{m=1}^{n}c_i^*(\widehat{f}_{m-1})<c_i^*(f)-\frac{\epsilon}{2}\right) \\
&\leq\mathbb{P}_{f}\left(\sum_{m=1}^{n}\left(R_i(m)-c_i^R(\widehat{f}_{m-1})\right)<-\frac{\epsilon}{2}n\right)+\mathbb{P}_{f}\left(\sum_{m=1}^{n}\left(c_i^*(\widehat{f}_{m-1})-c_i^*(f)\right)<-\frac{\epsilon}{2}n\right).
\end{aligned} P f ( π i R ( n ) < c i ∗ ( f ) − ϵ ) ≤ P f ( n 1 m = 1 ∑ n ( R i ( m ) − c i R ( f m − 1 ) ) < − 2 ϵ ) + P f ( n 1 m = 1 ∑ n c i ∗ ( f m − 1 ) < c i ∗ ( f ) − 2 ϵ ) ≤ P f ( m = 1 ∑ n ( R i ( m ) − c i R ( f m − 1 ) ) < − 2 ϵ n ) + P f ( m = 1 ∑ n ( c i ∗ ( f m − 1 ) − c i ∗ ( f ) ) < − 2 ϵ n ) .
The first probability is exponentially decaying by Azuma-Hoeffding inequality, since by (25)
{ R i ( m ) − c i R ( f ^ m − 1 ) : m ≥ 1 } \left\{R_i(m)-c_i^R(\widehat{f}_{m-1}):m\geq1\right\} { R i ( m ) − c i R ( f m − 1 ) : m ≥ 1 }
is a martingale difference sequence, uniformly bounded by 1. Therefore, it remains to control the second probability. By assumption of continuity of the infimum map and the connection of x A ( f ) x_A(f) x A ( f ) , y A ( f ) y_A(f) y A ( f ) with I i ( f i ) I_i(f_i) I i ( f i ) and J i ( f i ) J_i(f_i) J i ( f i ) , i ∈ [ M ] i\in[M] i ∈ [ M ] , it follows that c i ∗ c_i^* c i ∗ is continuous at f f f , and thus there exists δ > 0 \delta>0 δ > 0 such that for any g ∈ Φ g\in\Phi g ∈ Φ
max j ∈ [ M ] ∥ f j ( x ) − g j ( x ) ∥ 1 < δ ⇒ ∣ c i ∗ ( g ) − c i ∗ ( f ) ∣ < ϵ / 4. \max_{j\in[M]}\left\|f_j(x)-g_j(x)\right\|_1<\delta\quad\Rightarrow\quad\left|c_i^*(g)-c_i^*(f)\right|<\epsilon/4. j ∈ [ M ] max ∥ f j ( x ) − g j ( x ) ∥ 1 < δ ⇒ ∣ c i ∗ ( g ) − c i ∗ ( f ) ∣ < ϵ /4.
Let
τ ~ : = max j ∈ [ M ] τ j R ( δ ) . \widetilde{\tau}:=\max_{j\in[M]}\tau_j^R(\delta). τ := j ∈ [ M ] max τ j R ( δ ) .
Then, for all m > τ ~ m>\widetilde{\tau} m > τ ,
∣ c i ∗ ( f ^ m − 1 ) − c i ∗ ( f ) ∣ < ϵ / 4. \left|c_i^*(\widehat{f}_{m-1})-c_i^*(f)\right|<\epsilon/4. c i ∗ ( f m − 1 ) − c i ∗ ( f ) < ϵ /4.
Since for all m ≤ τ ~ m\leq\widetilde{\tau} m ≤ τ we have ∣ c i ∗ ( f ^ m − 1 ) − c i ∗ ( f ) ∣ ≤ 1 \left|c_i^*(\widehat{f}_{m-1})-c_i^*(f)\right|\leq1 c i ∗ ( f m − 1 ) − c i ∗ ( f ) ≤ 1 , it holds
P f ( ∑ m = 1 n ∣ c i ∗ ( f ^ m − 1 ) − c i ∗ ( f ) ∣ > ϵ 2 n ) ≤ P f ( τ ~ + ϵ 4 n > ϵ 2 n ) . \mathbb{P}_{f}\left(\sum_{m=1}^{n}\left|c_i^*(\widehat{f}_{m-1})-c_i^*(f)\right|>\frac{\epsilon}{2}n\right)\leq\mathbb{P}_{f}\left(\widetilde{\tau}+\frac{\epsilon}{4}n>\frac{\epsilon}{2}n\right). P f ( m = 1 ∑ n c i ∗ ( f m − 1 ) − c i ∗ ( f ) > 2 ϵ n ) ≤ P f ( τ + 4 ϵ n > 2 ϵ n ) .
Thus, it suffices to show that, for every j ∈ [ M ] j\in[M] j ∈ [ M ] ,
∑ n = 1 ∞ P f ( τ j R ( δ ) > ϵ 4 n ) < ∞ , \sum_{n=1}^{\infty}\mathbb{P}_{f}\left(\tau_j^R(\delta)>\frac{\epsilon}{4}n\right)<\infty, n = 1 ∑ ∞ P f ( τ j R ( δ ) > 4 ϵ n ) < ∞ ,
which follows by Proposition 7.1.
An important implication of conditions (38) and (40) of Theorem 7.2 is that the estimator f ^ n \hat{f}_n f ^ n converges quickly to the true density vector f f f , in the sense that the random times τ i R ( ϵ ) \tau_i^R(\epsilon) τ i R ( ϵ ) , i ∈ [ M ] i \in[M] i ∈ [ M ] , defined in (37), have summable tail probabilities and hence finite expectations.
Proposition 7.1. Let A ⊆ [ M ] A \subseteq[M] A ⊆ [ M ] , f ∈ Φ A f \in\Phi_A f ∈ Φ A , and let R R R be a sampling rule. If assumptions (i) and (iii) of Theorem 7.2 hold then
∑ n = 1 ∞ P f ( τ i R ( ϵ ) > c n ) < ∞ , (42) \sum_{n=1}^{\infty} \mathbb{P}_f\left(\tau_i^R(\epsilon)>cn\right)<\infty, \tag*{(42)} n = 1 ∑ ∞ P f ( τ i R ( ϵ ) > c n ) < ∞ , (42) for any ϵ > 0 \epsilon>0 ϵ > 0 and c > 0 c>0 c > 0 .
Proof: The proof is presented in Appendix B.
As a corollary to Theorem 7.2, we specify a particular choice of the functional c i R c_i^R c i R , i ∈ [ M ] i \in[M] i ∈ [ M ] , that yields asymptotic optimality.
Corollary 7.1. Let A ⊆ [ M ] A \subseteq[M] A ⊆ [ M ] , f ∈ Φ A f \in\Phi_A f ∈ Φ A , and let R R R be a sampling rule. If, for each i ∈ [ M ] i \in[M] i ∈ [ M ] ,
c i R ( g ) = c i ∗ ( g ) , c_i^R(g)=c_i^*(g), c i R ( g ) = c i ∗ ( g ) , for every D ⊆ [ M ] D \subseteq[M] D ⊆ [ M ] and every g ∈ Φ D g \in\Phi_D g ∈ Φ D , and if conditions (39) and (40) of Theorem 7.2 are satisfied, then R R R is asymptotically optimal.
Proof: By Theorem 7.2, it suffices to verify that R R R satisfies the sampling constraint (6) with T = T R T=T^R T = T R . By Proposition 6.1, it is enough to show that (26) holds. Evaluating at g = f g=f g = f and using the definition (29), we obtain
∑ i = 1 M c i ∗ ( f ) = x A ( f ) ∑ i ∈ A I A ∗ ( f ) I i ( f i ) + y A ( f ) ∑ i ∉ A J A ∗ ( f ) J i ( f i ) = x A ( f ) K ^ A ( f ) + y A ( f ) K ˘ A ( f ) ≤ K , \begin{aligned}
\sum_{i=1}^{M} c_i^*(f)=x_A(f)\sum_{i\in A}\frac{I_A^*(f)}{I_i(f_i)}+y_A(f)\sum_{i\notin A}\frac{J_A^*(f)}{J_i(f_i)} \\
&=x_A(f)\widehat{K}_A(f)+y_A(f)\breve{K}_A(f)\le K,
\end{aligned} i = 1 ∑ M c i ∗ ( f ) = x A ( f ) i ∈ A ∑ I i ( f i ) I A ∗ ( f ) + y A ( f ) i ∈ / A ∑ J i ( f i ) J A ∗ ( f ) = x A ( f ) K A ( f ) + y A ( f ) K ˘ A ( f ) ≤ K ,
where the last inequality follows from the definitions of x A ( f ) x_A(f) x A ( f ) and y A ( f ) y_A(f) y A ( f ) in Theorem 5.1.
Suitable Test Statistics
In this section, we introduce the generalized log-likelihood ratio (GLLR) and the adaptive log-likelihood ratio (ALLR) and establish conditions under which each statistic satisfies the growth requirement (39) of Theorem 7.2.
To this end, we impose regularity conditions on the density classes Φ i \Phi_i Φ i , i ∈ [ M ] i \in[M] i ∈ [ M ] , similar in spirit to those verified by Pavlov [10 ] for multidimensional exponential families under suitable compactness assumptions. For a regular minimal exponential family restricted to a compact subset of its natural parameter space, Proposition A.1 shows that the map from parameters to densities is a homeomorphism when the density class is equipped with the L 1 L^1 L 1 topology. This correspondence motivates formulating our regularity conditions directly in terms of densities and their L 1 L^1 L 1 neighborhoods. We can therefore impose these conditions on non-parametric density classes Φ i \Phi_i Φ i , i ∈ [ M ] i \in[M] i ∈ [ M ] , without assuming a finite-dimensional parametrization. Proposition A.2 verifies these conditions for the exponential families.
Regularity Conditions
Analogously to the density regularity conditions in [10 ] , we impose the following regularity conditions, which will be used in the analysis of the asymptotic properties of the proposed test statistics. For each i ∈ [ M ] i \in[M] i ∈ [ M ] and each g ∈ Φ i g \in\Phi_i g ∈ Φ i , define
p ϵ + ( g , X i ) : = sup h ∈ V ϵ ( g ) h ( X i ) . p_\epsilon^+(g, X_i) := \sup_{h \in V_\epsilon(g)} h(X_i). p ϵ + ( g , X i ) := h ∈ V ϵ ( g ) sup h ( X i ) .
where
V ϵ ( g ) : = { h ∈ Φ i : ∥ g − h ∥ 1 < ϵ } . V_\epsilon(g) := \{h \in\Phi_i : \lVert g-h \rVert_1 < \epsilon\}. V ϵ ( g ) := { h ∈ Φ i : ∥ g − h ∥ 1 < ϵ } .
We assume that the following conditions hold for every i ∈ [ M ] i \in[M] i ∈ [ M ] :
(a) The class of densities Φ i \Phi_i Φ i is compact with respect to the L 1 L^1 L 1 norm.
(b) Every g ∈ Φ i g \in\Phi_i g ∈ Φ i satisfies g ( x ) > 0 g(x) > 0 g ( x ) > 0 for all x ∈ S x \in\mathcal{S} x ∈ S .
(c) The Kullback–Leibler divergence satisfies
D ( f i , g ) > 0 , ∀ g ∈ Φ i , g ≠ f i . (43) D(f_i,g) > 0,\qquad\forall g \in\Phi_i,\ g \ne f_i. \tag*{(43)} D ( f i , g ) > 0 , ∀ g ∈ Φ i , g = f i . (43)
(d) The mapping g ↦ D ( f i , g ) g \mapsto D(f_i,g) g ↦ D ( f i , g ) is continuous on Φ i \Phi_i Φ i .
(e) There exist q > 2 q > 2 q > 2 , ϵ > 0 \epsilon> 0 ϵ > 0 , and c > 0 c > 0 c > 0 such that
E f i [ sup h ∈ V ϵ ( g ) ∣ log h ( X i ) ∣ q ] ≤ c , ∀ g , f i ∈ Φ i . (44) \mathbb{E}_{f_i}\left[\sup_{h \in V_\epsilon(g)} \lvert\log h(X_i)\rvert^q\right] \le c,\qquad\forall g,f_i \in\Phi_i. \tag*{(44)} E f i [ h ∈ V ϵ ( g ) sup ∣ log h ( X i ) ∣ q ] ≤ c , ∀ g , f i ∈ Φ i . (44)
(f) There exist ϵ > 0 \epsilon> 0 ϵ > 0 and c > 0 c > 0 c > 0 such that
E f i [ p ϵ + ( g , X i ) f i ( X i ) ] = ∫ S p ϵ + ( g , x ) d ν i ( x ) ≤ c , ∀ g , f i ∈ Φ i . (45) \mathbb{E}_{f_i}\left[\frac{p_\epsilon^+(g,X_i)}{f_i(X_i)}\right] = \int_{\mathcal{S}}p_\epsilon^+(g,x)\,d\nu_i(x) \le c,\qquad\forall g,f_i \in\Phi_i. \tag*{(45)} E f i [ f i ( X i ) p ϵ + ( g , X i ) ] = ∫ S p ϵ + ( g , x ) d ν i ( x ) ≤ c , ∀ g , f i ∈ Φ i . (45)
(g) The function
H f i ( g , ϵ ) : = E f i [ sup h ∈ V ϵ ( g ) ∣ log h ( X i ) − log g ( X i ) ∣ ] H_{f_i}(g,\epsilon) := \mathbb{E}_{f_i}\left[\sup_{h \in V_\epsilon(g)}\lvert\log h(X_i)-\log g(X_i)\rvert\right] H f i ( g , ϵ ) := E f i [ h ∈ V ϵ ( g ) sup ∣ log h ( X i ) − log g ( X i )∣ ]
converges to 0 0 0 as ϵ → 0 \epsilon\to0 ϵ → 0 , uniformly over g ∈ Φ i g \in\Phi_i g ∈ Φ i .
Condition (a) ensures that Φ i \Phi_i Φ i is compact and hence totally bounded in the L 1 L^1 L 1 norm. Condition (b) ensures that the log-likelihoods are finite and well defined. Conditions (c) and (d) ensure that the Kullback–Leibler divergence uniquely identifies the true density, up to almost-everywhere equality, and varies continuously over Φ i \Phi_i Φ i . Conditions (e) and (f) provide uniform moment and likelihood-ratio bounds that, together with the other assumptions, are used to derive summable tail bounds for the proposed test statistics. Finally, Condition (g) controls the expected local oscillation of the log-density uniformly over Φ i \Phi_i Φ i , for each fixed true density, and enables likelihood comparisons within small L 1 L^1 L 1 neighborhoods. In Proposition A.2 of Appendix A, we verify these conditions for regular minimal exponential families.
A non-parametric example is obtained by mixing Gaussian densities with fixed variance and means in a compact interval. Each component density is determined by its mean, while the mixing distribution can be any probability measure on the interval. The following proposition shows that regularity conditions (a)–(g) also hold for mixtures of densities from the exponential families.
Proposition 8.1: For each i ∈ [ M ] i \in[M] i ∈ [ M ] , let E i = { g θ : θ ∈ Θ i } \mathcal{E}_i = \{g_\theta: \theta\in\Theta_i\} E i = { g θ : θ ∈ Θ i } be an exponential family with densities of the form
g θ ( x ) = h i ( x ) exp { θ ⊤ T i ( x ) − b i ( θ ) } , θ ∈ Θ i , g_\theta(x) = h_i(x) \exp\{\theta^\top T_i(x) - b_i(\theta)\}, \qquad\theta\in\Theta_i, g θ ( x ) = h i ( x ) exp { θ ⊤ T i ( x ) − b i ( θ )} , θ ∈ Θ i , where Θ i \Theta_i Θ i is a compact subset of the natural parameter space. Let P ( Θ i ) \mathcal{P}(\Theta_i) P ( Θ i ) denote the set of Borel probability measures on Θ i \Theta_i Θ i , and define the class of mixture densities
Φ i : = { q i , P : P ∈ P ( Θ i ) } , q i , P ( x ) : = ∫ Θ i g θ ( x ) P ( d θ ) . \Phi_i := \{q_{i,P} : P \in\mathcal{P}(\Theta_i)\}, \qquad q_{i,P}(x) := \int_{\Theta_i} g_\theta(x)\,P(d\theta). Φ i := { q i , P : P ∈ P ( Θ i )} , q i , P ( x ) := ∫ Θ i g θ ( x ) P ( d θ ) . Then Φ i \Phi_i Φ i satisfies regularity conditions (a)–(g).
Proof: The proof is presented in Appendix A.
Generalized Log-Likelihood Ratio
To define the Generalized Log-Likelihood Ratio (GLLR), we first consider the log-likelihood of source i i i under a candidate density g ∈ Φ i g \in\Phi_i g ∈ Φ i , based on the observations collected up to time n n n . For any sampling rule R R R , define
l i R ( n ; g ) : = l i R ( n − 1 ; g ) + log ( g ( X i ( n ) ) ) R i ( n ) , (46) l_i^R(n;g) := l_i^R(n-1;g) + \log(g(X_i(n)))R_i(n), \tag*{(46)} l i R ( n ; g ) := l i R ( n − 1 ; g ) + log ( g ( X i ( n ))) R i ( n ) , (46)
with l i R ( 0 ; g ) : = 0 l_i^R(0;g) := 0 l i R ( 0 ; g ) := 0 . Here, n n n denotes the global sampling time, and should not be confused with the total number of observations collected from source i i i up to time n n n , which is given by N i R ( n ) N_i^R(n) N i R ( n ) in (5). The GLLR statistic is defined as
Λ i R ( n ) : = l i 1 ( n ) − l i 0 ( n ) , n ∈ N , (47) \Lambda_i^R(n) := l_i^1(n) - l_i^0(n), \qquad n \in\mathbb{N}, \tag*{(47)} Λ i R ( n ) := l i 1 ( n ) − l i 0 ( n ) , n ∈ N , (47)
where
l i 1 ( n ) : = sup g ∈ Φ 1 i l i ( n ; g ) , l i 0 ( n ) : = sup g ∈ Φ 0 i l i ( n ; g ) . (48) l_i^1(n) := \sup_{g \in\Phi_1^i} l_i(n;g), \qquad l_i^0(n) := \sup_{g \in\Phi_0^i} l_i(n;g). \tag*{(48)} l i 1 ( n ) := g ∈ Φ 1 i sup l i ( n ; g ) , l i 0 ( n ) := g ∈ Φ 0 i sup l i ( n ; g ) . (48)
Large positive values of Λ i R ( n ) \Lambda_i^R(n) Λ i R ( n ) indicate that i ∈ A i \in A i ∈ A , while large negative values indicate that i ∉ A i \notin A i ∈ / A . The following result shows that the GLLR satisfies condition (39) of Theorem 7.2.
Theorem 8.1. Let A ⊆ [ M ] A \subseteq[M] A ⊆ [ M ] , f ∈ Φ A f \in\Phi_A f ∈ Φ A , and let R R R be a sampling rule. If { Λ i R ( n ) , i ∈ [ M ] } \{\Lambda_i^R(n), i \in[M]\} { Λ i R ( n ) , i ∈ [ M ]} are the GLLR statistics, then for every ρ ∈ ( 0 , 1 ] \rho\in(0,1] ρ ∈ ( 0 , 1 ] and ϵ > 0 \epsilon> 0 ϵ > 0 , the sequences
P f ( 1 n Λ i R ( n ) < ρ I i ( f i ) − ϵ , π i R ( n ) ≥ ρ ) , ∀ i ∈ A , P f ( − 1 n Λ j R ( n ) < ρ J j ( f j ) − ϵ , π j R ( n ) ≥ ρ ) , ∀ j ∉ A , (49) \begin{aligned}
\mathbb{P}_f\left(\frac{1}{n}\Lambda_i^R(n)<\rho I_i(f_i)-\epsilon,\ \pi_i^R(n)\geq\rho\right),\quad&\forall i\in A,\\
\mathbb{P}_f\left(-\frac{1}{n}\Lambda_j^R(n)<\rho J_j(f_j)-\epsilon,\ \pi_j^R(n)\geq\rho\right),\quad&\forall j\notin A, \tag*{(49)}
\end{aligned} P f ( n 1 Λ i R ( n ) < ρ I i ( f i ) − ϵ , π i R ( n ) ≥ ρ ) , P f ( − n 1 Λ j R ( n ) < ρ J j ( f j ) − ϵ , π j R ( n ) ≥ ρ ) , ∀ i ∈ A , ∀ j ∈ / A , (49) are exponentially decaying.
Proof. We cover the null class Φ i 0 \Phi_i^0 Φ i 0 by finitely many neighborhoods and bound the likelihood over each neighborhood using the corresponding upper density envelope. This reduces the GLLR deviation event to a finite union of deviation events for sums of log ratios between the true density and these envelopes. Lemma B.1 provides exponential bounds for these events, and a union bound yields the desired exponential decay. The full proof is given in Appendix B.
We observe that, for each i ∈ [ M ] i \in[M] i ∈ [ M ] , by choosing ρ = c i ∗ ( f ) − ϵ > 0 \rho=c_i^*(f)-\epsilon>0 ρ = c i ∗ ( f ) − ϵ > 0 for sufficiently small ϵ > 0 \epsilon>0 ϵ > 0 , Theorem 8.1 implies condition (39) of Theorem 7.2 .
Adaptive Log-Likelihood Ratio
For the definition of the Adaptive Log-Likelihood Ratio (ALLR), we consider, in addition to (46)–(48), a log-likelihood constructed using estimated densities. Specifically, for any sampling rule R R R , define
l ^ i ( n ) : = l ^ i ( n − 1 ) + log ( f ^ i , n − 1 ( X i ( n ) ) R i ( n ) ) (50) \hat{l}_i(n):=\hat{l}_i(n-1)+\log\left(\frac{\hat{f}_{i,n-1}(X_i(n))}{R_i(n)}\right) \tag*{(50)} l ^ i ( n ) := l ^ i ( n − 1 ) + log ( R i ( n ) f ^ i , n − 1 ( X i ( n )) ) (50)
with l ^ i ( 0 ) : = 0 \hat{l}_i(0):=0 l ^ i ( 0 ) := 0 , where f ^ i , n − 1 \hat{f}_{i,n-1} f ^ i , n − 1 is the estimator of f i f_i f i based on observations up to time n − 1 n-1 n − 1 . Unlike the GLLR, the ALLR replaces the unknown densities with sequential estimates, adapting gradually to model uncertainty. For each i ∈ [ M ] i \in[M] i ∈ [ M ] , the ALLR statistic is defined as
Λ i R ( n ) : = { l ^ i ( n ) − l i 0 ( n ) , if l i 0 ( n ) ≤ min { l i 1 ( n ) , l ^ i ( n ) } , − ( l ^ i ( n ) − l i 1 ( n ) ) , if l i 1 ( n ) ≤ min { l i 0 ( n ) , l ^ i ( n ) } , 0 , otherwise. (51) \Lambda_i^R(n):=
\begin{cases}
\hat{l}_i(n)-l_i^0(n), & \text{if }l_i^0(n)\leq\min\{l_i^1(n),\hat{l}_i(n)\},\\
-\bigl(\hat{l}_i(n)-l_i^1(n)\bigr), & \text{if }l_i^1(n)\leq\min\{l_i^0(n),\hat{l}_i(n)\},\\
0, & \text{otherwise.}
\end{cases} \tag*{(51)} Λ i R ( n ) := ⎩ ⎨ ⎧ l ^ i ( n ) − l i 0 ( n ) , − ( l ^ i ( n ) − l i 1 ( n ) ) , 0 , if l i 0 ( n ) ≤ min { l i 1 ( n ) , l ^ i ( n )} , if l i 1 ( n ) ≤ min { l i 0 ( n ) , l ^ i ( n )} , otherwise. (51)
Thus, the ALLR compares the estimated log-likelihood with the best alternatives under the regular and anomalous hypotheses. Large positive values of Λ i R ( n ) \Lambda_i^R(n) Λ i R ( n ) provide evidence that i ∈ A i\in A i ∈ A , while large negative values indicate that i ∉ A i\notin A i ∈ / A .
The following theorem establishes condition (39) of Theorem 7.2 for the ALLR statistics. Since these statistics incorporate sequential density estimates, their analysis requires control of the convergence of these estimates. Proposition 7.1 shows that conditions (38) and (40) imply the summability condition (42) for the random times τ i R ( δ ) \tau_i^R(\delta) τ i R ( δ ) defined in (37). We therefore state the theorem directly in terms of this summability condition.
Theorem 8.2. Let A ⊆ [ M ] A \subseteq[M] A ⊆ [ M ] , f ∈ Φ A f \in\Phi_A f ∈ Φ A , and let R R R be a sampling rule. Suppose that:
(i) { Λ i R ( n ) , i ∈ [ M ] } \{\Lambda_i^R(n), i \in[M]\} { Λ i R ( n ) , i ∈ [ M ]} are the ALLR statistics;
(ii) for the estimator { f ^ n : n ∈ N } \{\hat{f}_n : n \in\mathbb{N}\} { f ^ n : n ∈ N } , for any δ > 0 \delta> 0 δ > 0 , and c > 0 c > 0 c > 0 ,
∑ n = 1 ∞ P f ( τ i R ( δ ) > c n ) < ∞ , ∀ i ∈ [ M ] . (52) \sum_{n=1}^{\infty} \mathbb{P}_f\left(\tau_i^R(\delta) > cn\right) < \infty,\quad\forall i \in[M]. \tag*{(52)} n = 1 ∑ ∞ P f ( τ i R ( δ ) > c n ) < ∞ , ∀ i ∈ [ M ] . (52) Then, for every ρ ∈ ( 0 , 1 ] \rho\in(0,1] ρ ∈ ( 0 , 1 ] and ϵ > 0 \epsilon> 0 ϵ > 0 ,
∑ n = 1 ∞ P f ( 1 n Λ i R ( n ) < ρ I i ( f i ) − ϵ , π i R ( n ) ≥ ρ ) < ∞ , ∀ i ∈ A , (53) \sum_{n=1}^{\infty} \mathbb{P}_f\left(\frac{1}{n}\Lambda_i^R(n) < \rho I_i(f_i)-\epsilon,\ \pi_i^R(n) \geq\rho\right) < \infty,\quad\forall i \in A, \tag*{(53)} n = 1 ∑ ∞ P f ( n 1 Λ i R ( n ) < ρ I i ( f i ) − ϵ , π i R ( n ) ≥ ρ ) < ∞ , ∀ i ∈ A , (53) ∑ n = 1 ∞ P f ( − 1 n Λ j R ( n ) < ρ J j ( f j ) − ϵ , π j R ( n ) ≥ ρ ) < ∞ , ∀ j ∉ A . \sum_{n=1}^{\infty} \mathbb{P}_f\left(-\frac{1}{n}\Lambda_j^R(n) < \rho J_j(f_j)-\epsilon,\ \pi_j^R(n) \geq\rho\right) < \infty,\quad\forall j \notin A. n = 1 ∑ ∞ P f ( − n 1 Λ j R ( n ) < ρ J j ( f j ) − ϵ , π j R ( n ) ≥ ρ ) < ∞ , ∀ j ∈ / A .
Proof. Using the definition of the ALLR and the bounds established in the proof of Theorem 8.1, we reduce the argument to controlling the estimation error l i ( n ; f i ) − l ^ i ( n ) l_i(n; f_i)-\hat{l}_i(n) l i ( n ; f i ) − l ^ i ( n ) . This error decomposes into a martingale term and a sum of Kullback–Leibler divergences. Establishing summable tail bounds for these two terms yields (39). The full proof is given in Appendix B. ■
By choosing ρ = c i ∗ ( f ) − ϵ > 0 \rho=c_i^*(f)-\epsilon>0 ρ = c i ∗ ( f ) − ϵ > 0 for each i ∈ [ M ] i \in[M] i ∈ [ M ] , for sufficiently small ϵ > 0 \epsilon>0 ϵ > 0 , Theorem 8.2 implies condition (39) of Theorem 7.2.
Error Control and Threshold Selection
In addition to condition (39) of Theorem 7.2, we must verify that, for a suitable test statistic, the stopping and decision rules introduced in Subsection VI-B satisfy the error constraint (4) and the asymptotic scaling (24).
GLLR
GLLR: For GLLR, this can be established under parametric assumptions on Φ i \Phi_i Φ i , i ∈ [ M ] i \in[M] i ∈ [ M ] . In particular, when each Φ i \Phi_i Φ i is a subset corresponding to a multidimensional exponential family, Chan and Lai [15 Theorem 1 and Subsection 3.1] provide a first-order asymptotic approximation of the error probabilities for GLLR-type tests.
Proposition 8.2. Suppose that, for each i ∈ [ M ] i \in[M] i ∈ [ M ] , the sets Φ i 0 \Phi_i^0 Φ i 0 and Φ i 1 \Phi_i^1 Φ i 1 correspond to parametric subsets of a multidimensional exponential family. Let R R R be an arbitrary sampling rule and { Λ i R ( n ) , i ∈ [ M ] } \{\Lambda_i^R(n), i \in[M]\} { Λ i R ( n ) , i ∈ [ M ]} be the GLLR statistics. Then the policy ( R , T R , Δ R ) (R,T^R,\Delta^R) ( R , T R , Δ R ) satisfies the error constraint (4) asymptotically when the thresholds a , b → ∞ a,b \to\infty a , b → ∞ are chosen so that
C a q / 2 e − a ∼ β , C b q / 2 e − b ∼ α , (54) C a^{q/2}e^{-a}\sim\beta,\qquad C b^{q/2}e^{-b}\sim\alpha, \tag*{(54)} C a q /2 e − a ∼ β , C b q /2 e − b ∼ α , (54)
as α , β → 0 \alpha, \beta\to0 α , β → 0 . In particular, these thresholds satisfy the asymptotic scaling (24).
Although the asymptotic optimality theory developed in this paper is formulated under the regularity conditions of Subsection VIII-A, Proposition A.2 shows that these assumptions are satisfied by regular minimal exponential families. Consequently, combining Proposition 8.2 with Theorem 7.2 establishes the asymptotic optimality of the proposed GLLR policy for multidimensional exponential-family models.
In Proposition 8.2, C C C and q q q are constants. The constant C C C is given in [15 Theorem 1] and depends on the dimension of the parameter, the exponential-family model, and geometric properties of the parameter space Φ \Phi Φ . The constant q q q is the maximum absolute difference between the dimensions of the manifolds Φ i 1 \Phi_i^1 Φ i 1 and Φ i 0 \Phi_i^0 Φ i 0 , i.e.,
q : = max i ∈ [ M ] ∣ dim ( Φ i 1 ) − dim ( Φ i 0 ) ∣ . q := \max_{i\in[M]} \left|\dim(\Phi_i^1)-\dim(\Phi_i^0)\right|. q := i ∈ [ M ] max dim ( Φ i 1 ) − dim ( Φ i 0 ) .
Related asymptotic approximations, with less precise constants, were obtained by Woodroofe [14 Sections 7.2 and 8.2] . In the one-dimensional exponential-family case, the constant C C C can be evaluated explicitly using Lorden’s approximation; see [39 (5.67)] .
ALLR
The following result shows that the ALLR yields nonasymptotic error control with thresholds that also satisfy the asymptotic scaling (24), without imposing a parametric structure on Φ i \Phi_i Φ i . In contrast to the GLLR, this provides guarantees for nonparametric hypothesis classes.
Proposition 8.3: Let R R R be an arbitrary sampling rule, and { Λ i R ( n ) , i ∈ [ M ] } \{\Lambda_i^R(n), i \in[M]\} { Λ i R ( n ) , i ∈ [ M ]} be the ALLR statistics. If the thresholds of the stopping rule are chosen as
a = ∣ log ( β ) ∣ + log M , b = ∣ log ( α ) ∣ + log M , (55) a = |\log(\beta)|+\log M,\qquad b = |\log(\alpha)|+\log M, \tag*{(55)} a = ∣ log ( β ) ∣ + log M , b = ∣ log ( α ) ∣ + log M , (55)
then the policy ( R , T R , Δ R ) (R,T^R,\Delta^R) ( R , T R , Δ R ) satisfies the error constraint (4), and the asymptotic scaling (24).
Proof: We prove the bound for the probability of missed detection, as the bound for false alarm is analogous. A missed detection occurs if there exists i ∈ A i\in A i ∈ A such that
B i : = { Λ i R ( T R ) ≤ − a } . B_i := \{\Lambda_i^R(T^R)\le-a\}. B i := { Λ i R ( T R ) ≤ − a } .
By the union bound,
P f ( ∣ A ∖ Δ R ( T R ) ∣ ≥ 1 ) ≤ ∑ i ∈ A P f ( B i ) . \mathbb{P}_{f}\left(|A\setminus\Delta^R(T^R)|\ge1\right)\le\sum_{i\in A}\mathbb{P}_{f}(B_i). P f ( ∣ A ∖ Δ R ( T R ) ∣ ≥ 1 ) ≤ i ∈ A ∑ P f ( B i ) .
If Λ i R ( T R ) ≥ 0 \Lambda_i^R(T^R)\ge0 Λ i R ( T R ) ≥ 0 , then P f ( B i ) = 0 \mathbb{P}_{f}(B_i)=0 P f ( B i ) = 0 . Otherwise, by (51),
Λ i R ( T R ) = − ( l ^ i ( T R ) − l i 1 ( T R ) ) , \Lambda_i^R(T^R)=-\left(\hat l_i(T^R)-l_i^1(T^R)\right), Λ i R ( T R ) = − ( l ^ i ( T R ) − l i 1 ( T R ) ) ,
and hence
P f ( B i ) ≤ P f ( sup n ∈ N { l ^ i ( n ) − l i 1 ( n ) } ≥ a ) . \mathbb{P}_{f}(B_i)\le\mathbb{P}_{f}\left(\sup_{n\in\mathbb{N}}\{\hat l_i(n)-l_i^1(n)\}\ge a\right). P f ( B i ) ≤ P f ( n ∈ N sup { l ^ i ( n ) − l i 1 ( n )} ≥ a ) .
Since i ∈ A i \in A i ∈ A , (48) implies l ^ i 1 ( n ) ≥ l i ( n ; f i ) \hat{l}_i^1(n) \ge l_i(n; f_i) l ^ i 1 ( n ) ≥ l i ( n ; f i ) for all n n n , and therefore
P f ( B i ) ≤ P f ( sup n ∈ N { l ^ i ( n ) − l i ( n ; f i ) } ≥ a ) P_f(B_i) \le P_f\left(\sup_{n \in\mathbb{N}}\{\hat{l}_i(n)-l_i(n;f_i)\} \ge a\right) P f ( B i ) ≤ P f ( n ∈ N sup { l ^ i ( n ) − l i ( n ; f i )} ≥ a )
We now show that
M i ( n ) : = exp { l ^ i ( n ) − l i ( n ; f i ) } , n ≥ 0 , M_i(n) := \exp\{\hat{l}_i(n)-l_i(n;f_i)\}, \quad n \ge0, M i ( n ) := exp { l ^ i ( n ) − l i ( n ; f i )} , n ≥ 0 ,
is a nonnegative martingale. Indeed,
E f [ M i ( n ) ∣ F n − 1 R ] = M i ( n − 1 ) E f [ ( f ^ i , n − 1 ( X i ( n ) ) f i ( X i ( n ) ) ) R i ( n ) ∣ F n − 1 R ] . \mathbb{E}_f[M_i(n)\mid\mathcal{F}_{n-1}^{R}] = M_i(n-1)\mathbb{E}_f\left[\left.\left(\frac{\hat{f}_{i,n-1}(X_i(n))}{f_i(X_i(n))}\right)^{R_i(n)}\right|\mathcal{F}_{n-1}^{R}\right]. E f [ M i ( n ) ∣ F n − 1 R ] = M i ( n − 1 ) E f ( f i ( X i ( n )) f ^ i , n − 1 ( X i ( n )) ) R i ( n ) F n − 1 R .
Since R i ( n ) R_i(n) R i ( n ) are F n − 1 R \mathcal{F}_{n-1}^{R} F n − 1 R -measurable, we consider two cases. If R i ( n ) = 0 R_i(n)=0 R i ( n ) = 0 , the conditional expectation equals 1 1 1 . If R i ( n ) = 1 R_i(n)=1 R i ( n ) = 1 , then using the independence of X i ( n ) X_i(n) X i ( n ) from F n − 1 R \mathcal{F}_{n-1}^{R} F n − 1 R , and the fact that f ^ i , n − 1 \hat{f}_{i,n-1} f ^ i , n − 1 is F n − 1 R \mathcal{F}_{n-1}^{R} F n − 1 R -measurable
E f [ f ^ i , n − 1 ( X i ( n ) ) f i ( X i ( n ) ) ∣ F n − 1 R ] = ∫ f ^ i , n − 1 ( x ) d x = 1. \mathbb{E}_f\left[\left.\frac{\hat{f}_{i,n-1}(X_i(n))}{f_i(X_i(n))}\right|\mathcal{F}_{n-1}^{R}\right]=\int\hat{f}_{i,n-1}(x)\,\mathrm{d}x=1. E f [ f i ( X i ( n )) f ^ i , n − 1 ( X i ( n )) F n − 1 R ] = ∫ f ^ i , n − 1 ( x ) d x = 1.
Thus, { M i ( n ) : n ≥ 0 } \{M_i(n):n\ge0\} { M i ( n ) : n ≥ 0 } is a nonnegative martingale. By Ville’s inequality,
P f ( sup n ∈ N { l ^ i ( n ) − l i ( n ; f i ) } ≥ a ) ≤ e − a . P_f\left(\sup_{n\in\mathbb{N}}\{\hat{l}_i(n)-l_i(n;f_i)\}\ge a\right)\le e^{-a}. P f ( n ∈ N sup { l ^ i ( n ) − l i ( n ; f i )} ≥ a ) ≤ e − a .
Consequently,
P f ( ∣ A △ Δ R ( T R ) ∣ ≥ 1 ) ≤ ∣ A ∣ e − a ≤ M e − a . P_f\left(|A\mathbin{\triangle}\Delta^{R}(T^{R})|\ge1\right)\le|A|e^{-a}\le Me^{-a}. P f ( ∣ A △ Δ R ( T R ) ∣ ≥ 1 ) ≤ ∣ A ∣ e − a ≤ M e − a .
Choosing a = ∣ log β ∣ + log M a=|\log\beta|+\log M a = ∣ log β ∣ + log M yields
P f ( ∣ A △ Δ R ( T R ) ∣ ≥ 1 ) ≤ β . P_f\left(|A\mathbin{\triangle}\Delta^{R}(T^{R})|\ge1\right)\le\beta. P f ( ∣ A △ Δ R ( T R ) ∣ ≥ 1 ) ≤ β .
■ \blacksquare ■
The above results highlight a distinction between the GLLR and the ALLR at the level of theoretical guarantees. The GLLR can be applied in nonparametric settings; however, rigorous error control guarantees are available only under additional parametric assumptions, typically when Φ i \Phi_i Φ i corresponds to an exponential family. In contrast, the ALLR admits nonasymptotic error control without requiring a specific parametric structure on Φ i \Phi_i Φ i thereby ensuring applicability to a broader class of problems. Also the error bound of the ALLR is independent of the structure of space Φ i \Phi_i Φ i , in contrast to (54) where C C C depends on Φ i \Phi_i Φ i .
Suitable Estimators
In this section, we present two density estimators, the maximum likelihood estimator (MLE) and the kernel density estimator (KDE), and establish conditions under which they satisfy (40) of Theorem 7.2 .
Maximum Likelihood Estimator
The maximum likelihood estimator of f i f_i f i , based on the observations collected up to time n n n , is defined by
f ^ i , n ∈ arg max g ∈ Φ i l i ( n ; g ) \hat{f}_{i,n} \in\arg\max_{g\in\Phi_i} l_i(n;g) f ^ i , n ∈ arg g ∈ Φ i max l i ( n ; g )
where l i ( n ; g ) l_i(n;g) l i ( n ; g ) is the log-likelihood for source i i i under density g g g , defined in (46) . To ensure that a maximizer exists for every realization of the observations, we assume that, for each i ∈ [ M ] i\in[M] i ∈ [ M ] :
(i) the class Φ i \Phi_i Φ i is compact with respect to the L 1 L^1 L 1 norm;
(ii) for every x ∈ S x\in\mathcal{S} x ∈ S , the map
g ↦ log g ( x ) g\mapsto\log g(x) g ↦ log g ( x )
is upper semicontinuous with respect to the L 1 L^1 L 1 norm.
Under these assumptions, g ↦ l i ( n ; g ) g\mapsto l_i(n;g) g ↦ l i ( n ; g ) is upper semicontinuous on the compact set Φ i \Phi_i Φ i . Hence, by the Weierstrass extreme value theorem for upper semicontinuous functions, it attains its maximum. Lemma A.1, implies that the above assumptions are satisfied by every regular minimal exponential family with compact parameter space.
Theorem 9.1: Let A ⊆ [ M ] A\subseteq[M] A ⊆ [ M ] , f ∈ Φ A f\in\Phi_A f ∈ Φ A , and let R R R be a sampling rule. Suppose that the regularity conditions in Subsection VIII-A and the upper semicontinuity assumption (ii) in Subsection IX-A hold. For each i ∈ [ M ] i\in[M] i ∈ [ M ] , let { f ^ i , n : n ∈ N } \{\hat{f}_{i,n}:n\in\mathbb{N}\} { f ^ i , n : n ∈ N } be a sequence of MLEs defined above. Then, for every i ∈ [ M ] i\in[M] i ∈ [ M ] , ϵ > 0 \epsilon>0 ϵ > 0 , and ρ ∈ ( 0 , 1 ] \rho\in(0,1] ρ ∈ ( 0 , 1 ] , the sequence
{ P f ( f ^ i , n ∉ V ϵ ( f i ) , π i R ( n ) ≥ ρ ) : n ∈ N } \left\{\mathbb{P}_f\left(\hat{f}_{i,n}\notin V_\epsilon(f_i),\pi_i^R(n)\geq\rho\right):n\in\mathbb{N}\right\} { P f ( f ^ i , n ∈ / V ϵ ( f i ) , π i R ( n ) ≥ ρ ) : n ∈ N } decays exponentially.
Proof: We cover the compact set Φ i ∖ V ϵ ( f i ) \Phi_i\setminus V_\epsilon(f_i) Φ i ∖ V ϵ ( f i ) by finitely many suitable neighborhoods and bound the log-likelihood over each using its upper density envelope. This reduces the MLE deviation event to a finite union of events whose probabilities decay exponentially. The full proof is given in Appendix C.
Kernel Density Estimator
In this subsection, we assume that S = R \mathcal{S}=\mathbb{R} S = R . A non-parametric estimate of the density f i f_i f i can be obtained via the kernel density estimator (KDE). Let K : R → R K:\mathbb{R}\to\mathbb{R} K : R → R be a probability density, which we denote as the kernel. The kernel density estimator based on the observed samples is defined as
f ^ i , n ( x ) = 1 N i R ( n ) η i , n ∑ k = 1 n R i ( k ) K ( x − X i ( k ) η i , n ) , x ∈ R , \hat{f}_{i,n}(x)=\frac{1}{N_i^R(n)\eta_{i,n}}\sum_{k=1}^{n}R_i(k)K\left(\frac{x-X_i(k)}{\eta_{i,n}}\right),\qquad x\in\mathbb{R}, f ^ i , n ( x ) = N i R ( n ) η i , n 1 k = 1 ∑ n R i ( k ) K ( η i , n x − X i ( k ) ) , x ∈ R ,
where η i , n : = h N i R ( n ) > 0 \eta_{i,n}:=h_{N_i^R(n)}>0 η i , n := h N i R ( n ) > 0 is a function of N i R ( n ) N_i^R(n) N i R ( n ) , where h : N → R + h:\mathbb{N}\to\mathbb{R}_+ h : N → R + . The sequence { η i , n : n ∈ N } \{\eta_{i,n}:n\in\mathbb{N}\} { η i , n : n ∈ N } is assumed to satisfy
η i , n → 0 , N i R ( n ) η i , n → ∞ , as N i R ( n ) → ∞ . (56) \eta_{i,n}\to0,\qquad N_i^R(n)\eta_{i,n}\to\infty,\qquad\text{as }N_i^R(n)\to\infty. \tag*{(56)} η i , n → 0 , N i R ( n ) η i , n → ∞ , as N i R ( n ) → ∞. (56)
We assume that the kernel K K K satisfies the following conditions:
∫ R u K ( u ) d u = 0 \int_{\mathbb{R}} uK(u)\,\mathrm{d}u=0 ∫ R u K ( u ) d u = 0 ,
∫ R u 2 K ( u ) d u < ∞ \int_{\mathbb{R}} u^2K(u)\,\mathrm{d}u<\infty ∫ R u 2 K ( u ) d u < ∞ ,
K ∈ L p K\in L^p K ∈ L p , i.e., ∫ R ( K ( u ) ) p d u < ∞ \int_{\mathbb{R}}(K(u))^p\,\mathrm{d}u<\infty ∫ R ( K ( u ) ) p d u < ∞ , for some p > 15 / 2 p>15/2 p > 15/2 .
For the following theorem, we assume that, for the same p > 15 / 2 p>15/2 p > 15/2 as in condition (3), the true density f i ∈ Φ i f_i\in\Phi_i f i ∈ Φ i satisfies:
(a) f i ∈ L p / 2 f_i\in L^{p/2} f i ∈ L p /2 , i.e., ∫ R f i p / 2 ( x ) d x < ∞ \int_{\mathbb{R}}f_i^{p/2}(x)\,\mathrm{d}x<\infty ∫ R f i p /2 ( x ) d x < ∞ ,
(b) f i f_i f i is twice continuously differentiable on R \mathbb{R} R , and f i ′ ′ ∈ L p f_i''\in L^p f i ′′ ∈ L p , i.e., ∫ R ∣ f i ′ ′ ( x ) ∣ p d x < ∞ \int_{\mathbb{R}}|f_i''(x)|^p\,\mathrm{d}x<\infty ∫ R ∣ f i ′′ ( x ) ∣ p d x < ∞ .
The assumptions (a)–(b) are expressed for the true density f i f_i f i , which is unknown. Hence, they are assumed to hold locally on Φ i \Phi_i Φ i , i.e., in a neighborhood of every f i ∈ Φ i f_i\in\Phi_i f i ∈ Φ i .
Theorem 9.2: Assume that, for each i ∈ [ M ] i\in[M] i ∈ [ M ] , { f ^ i , n : n ∈ N } \{\hat{f}_{i,n}:n\in\mathbb{N}\} { f ^ i , n : n ∈ N } is the sequence of KDEs defined above, with bandwidth sequence { η i , n } \{\eta_{i,n}\} { η i , n } satisfying (56). Suppose further that the kernel assumptions (1)–(3) and the density assumptions (a)–(b) hold. Then for every ρ ∈ ( 0 , 1 ] \rho\in(0,1] ρ ∈ ( 0 , 1 ] , ϵ > 0 \epsilon>0 ϵ > 0 ,
∑ n = 1 ∞ n P f ( f ^ i , n ∉ V ϵ ( f i ) , π i R ( n ) ≥ ρ ) < ∞ , ∀ i ∈ [ M ] . \sum_{n=1}^{\infty} n\,\mathbb{P}_f\left(\hat{f}_{i,n}\notin V_{\epsilon}(f_i),\ \pi_i^R(n)\geq\rho\right)<\infty,\qquad\forall i\in[M]. n = 1 ∑ ∞ n P f ( f ^ i , n ∈ / V ϵ ( f i ) , π i R ( n ) ≥ ρ ) < ∞ , ∀ i ∈ [ M ] .
Proof: The proof follows the arguments in [40 Theorem 24.1] , adapted to the present sequential setting with observation control. We first reduce the L 1 L^1 L 1 deviation to an L p L^p L p deviation. We then partition the event of interest according to the number of samples collected from source i i i and decompose the KDE error into its stochastic and bias components. The bias is controlled using Taylor’s expansion and Minkowski’s inequality, while the stochastic component is bounded using the martingale version of Rosenthal’s inequality. The bandwidth choice, together with p > 15 / 2 p>15/2 p > 15/2 , makes the resulting weighted probability bounds summable. The detailed proof is provided in Appendix C.
Simulation Study - Advantage over agnostic sampling
In this section, we present simulation results that illustrate the asymptotic theory developed in the previous sections. Specifically, we compare the performance of the GLLR and the ALLR combined with each of the proposed estimators (MLE, and KDE), under the asymptotically optimal sampling frequencies suggested by our theory. We also compare their performance with that of an agnostic sampling rule that samples each source uniformly with frequency K / M K/M K / M .
The simulation setup consists of M = 10 M = 10 M = 10 sources, and at each time instant only half of them can be observed, i.e., K = 5 K = 5 K = 5 . We consider both a parametric and a nonparametric framework. In each case, the objective is to identify the subset of anomalous sources as quickly as possible. We assume that the anomalous set consists of the first ∣ A ∣ |A| ∣ A ∣ sources, i.e., A = { 1 , … , ∣ A ∣ } A = \{1,\ldots,|A|\} A = { 1 , … , ∣ A ∣ } , and perform 11 simulation experiments, corresponding to the possible values ∣ A ∣ ∈ { 0 , … , 10 } |A| \in\{0,\ldots,10\} ∣ A ∣ ∈ { 0 , … , 10 } . The thresholds a a a and b b b in the stopping rule (22) are selected via simulation so that both error probabilities in (4) are equal to 10 − 2 10^{-2} 1 0 − 2 . For each estimated expected stopping time, the Monte Carlo standard error is equal to 1. The simulated procedure is summarized in Algorithm 1.
Algorithm 1: Policy for anomaly identification.
Repeat the following steps (1)–(5), until the stopping condition (22) is satisfied.
Based on the already collected measurements, estimate the density f f f by f ^ n \hat{f}_n f ^ n , using a suitable estimator.
Compute the desired minimum long-run sampling frequencies ( c 1 ∗ ( f ^ n ) , … , c M ∗ ( f ^ n ) ) (c_1^*(\hat{f}_n),\ldots,c_M^*(\hat{f}_n)) ( c 1 ∗ ( f ^ n ) , … , c M ∗ ( f ^ n )) according to (29) .
Choose sampling probabilities c i R ( f ^ n ) c_i^R(\hat{f}_n) c i R ( f ^ n ) such that c i R ( f ^ n ) ≥ c i ∗ ( f ^ n ) c_i^R(\hat{f}_n) \ge c_i^*(\hat{f}_n) c i R ( f ^ n ) ≥ c i ∗ ( f ^ n ) , and ∑ i = 1 M c i R ( f ^ n ) ≤ K \sum_{i=1}^{M} c_i^R(\hat{f}_n) \le K ∑ i = 1 M c i R ( f ^ n ) ≤ K .
Perform sampling based on a probabilistic sampling rule presented in Section VI.
Update the LLRs of the sources we have received samples.
Decide as anomalous the sources with positive LLRs.
Parametric framework
In the parametric framework, the measurements from each source i ∈ [ M ] i \in[M] i ∈ [ M ] follow a Gaussian density,
X i ( n ) ∼ N ( μ i , σ i 2 ) , n ∈ N . X_i(n) \sim\mathcal{N}(\mu_i,\sigma_i^2), \qquad n \in\mathbb{N}. X i ( n ) ∼ N ( μ i , σ i 2 ) , n ∈ N .
If μ i ∈ [ 0 , 0.5 ] \mu_i \in[0,0.5] μ i ∈ [ 0 , 0.5 ] and σ i ∈ [ 0 , 0.2 ] \sigma_i \in[0,0.2] σ i ∈ [ 0 , 0.2 ] then source i i i is characterized as regular, whereas if μ i ∈ [ 0.55 , 1 ] \mu_i \in[0.55,1] μ i ∈ [ 0.55 , 1 ] and σ i ∈ [ 0.25 , 0.4 ] \sigma_i \in[0.25,0.4] σ i ∈ [ 0.25 , 0.4 ] the source is characterized as anomalous. The true but unknown values of ( μ i , σ i ) (\mu_i,\sigma_i) ( μ i , σ i ) for each i ∈ [ M ] i \in[M] i ∈ [ M ] , for this simulation, are
( μ i , σ i ) = { ( 0.7 + 0.02 i , 0.28 + 0.01 i ) , i ∈ A , ( 0.2 + 0.02 i , 0.08 + 0.005 i ) , i ∉ A . (\mu_i,\sigma_i) =
\begin{cases}
(0.7 + 0.02i,\ 0.28 + 0.01i), & i \in A,\\
(0.2 + 0.02i,\ 0.08 + 0.005i), & i \notin A.
\end{cases} ( μ i , σ i ) = { ( 0.7 + 0.02 i , 0.28 + 0.01 i ) , ( 0.2 + 0.02 i , 0.08 + 0.005 i ) , i ∈ A , i ∈ / A .
For each source i ∈ [ M ] i \in[M] i ∈ [ M ] , the parameters are estimated by their empirical counterparts based on the observations collected from that source. Specifically,
μ ^ i ( n ) : = 1 N i R ( n ) ∑ m = 1 n X i ( m ) R i ( m ) , σ ^ i ( n ) : = 1 N i R ( n ) ∑ m = 1 n ( X i ( m ) − μ ^ i ( n ) ) 2 R i ( m ) . (57) \hat{\mu}_i(n) := \frac{1}{N_i^R(n)} \sum_{m=1}^{n} X_i(m)R_i(m), \qquad\hat{\sigma}_i(n) := \sqrt{\frac{1}{N_i^R(n)} \sum_{m=1}^{n} (X_i(m)-\hat{\mu}_i(n))^2R_i(m)}. \tag*{(57)} μ ^ i ( n ) := N i R ( n ) 1 m = 1 ∑ n X i ( m ) R i ( m ) , σ ^ i ( n ) := N i R ( n ) 1 m = 1 ∑ n ( X i ( m ) − μ ^ i ( n ) ) 2 R i ( m ) . (57)
When i ∈ A i \in A i ∈ A , the difficulty of the underlying testing problem corresponding to source i i i is determined by the KL divergence of N ( 0.7 + 0.02 i , 0.28 + 0.01 i ) \mathcal{N}(0.7+0.02i,0.28+0.01i) N ( 0.7 + 0.02 i , 0.28 + 0.01 i ) versus N ( 0.5 , 0.2 ) \mathcal{N}(0.5,0.2) N ( 0.5 , 0.2 ) , where the latter is the regular density with the smallest KL distance from N ( 0.7 + 0.02 i , 0.28 + 0.01 i ) \mathcal{N}(0.7+0.02i,0.28+0.01i) N ( 0.7 + 0.02 i , 0.28 + 0.01 i ) , and KL divergence (1) is
I i ( μ i , σ i ) = D ( N ( 0.7 + 0.02 i , 0.28 + 0.01 i ) ∥ N ( 0.5 , 0.2 ) ) . I_i(\mu_i,\sigma_i)=D\bigl(\mathcal{N}(0.7+0.02i,0.28+0.01i)\Vert\mathcal{N}(0.5,0.2)\bigr). I i ( μ i , σ i ) = D ( N ( 0.7 + 0.02 i , 0.28 + 0.01 i ) ∥ N ( 0.5 , 0.2 ) ) .
When i ∉ A i \notin A i ∈ / A , the difficulty of the underlying testing problem corresponding to source i i i is determined by the KL divergence of N ( 0.2 + 0.02 i , 0.08 + 0.005 i ) \mathcal{N}(0.2+0.02i,0.08+0.005i) N ( 0.2 + 0.02 i , 0.08 + 0.005 i ) versus N ( 0.55 , 0.25 ) \mathcal{N}(0.55,0.25) N ( 0.55 , 0.25 ) , where the latter is the anomalous density with the smallest KL distance from N ( 0.2 + 0.02 i , 0.08 + 0.005 i ) \mathcal{N}(0.2+0.02i,0.08+0.005i) N ( 0.2 + 0.02 i , 0.08 + 0.005 i ) , and the KL divergence (2) is
J i ( μ i , σ i ) = D ( N ( 0.2 + 0.02 i , 0.08 + 0.005 i ) ∥ N ( 0.55 , 0.25 ) ) . J_i(\mu_i,\sigma_i)=D\bigl(\mathcal{N}(0.2+0.02i,0.08+0.005i)\Vert\mathcal{N}(0.55,0.25)\bigr). J i ( μ i , σ i ) = D ( N ( 0.2 + 0.02 i , 0.08 + 0.005 i ) ∥ N ( 0.55 , 0.25 ) ) .
Table I summarizes the values of I i I_i I i , J i J_i J i for each i ∈ [ M ] i \in[M] i ∈ [ M ] .
Table I. KL divergences.
When A = { 1 , … , k } A=\{1,\ldots,k\} A = { 1 , … , k } , with k ∈ 0 , … , M k\in0,\ldots,M k ∈ 0 , … , M , the difficulty of the anomaly identification problem is determined by the collection of KL divergences
S k : = { I 1 ( μ 1 , σ 1 ) , … , I k ( μ k , σ k ) , J k + 1 ( μ k + 1 , σ k + 1 ) , … , J M ( μ M , σ M ) } , 0 < k < M , S_k := \{I_1(\mu_1,\sigma_1),\ldots,I_k(\mu_k,\sigma_k),J_{k+1}(\mu_{k+1},\sigma_{k+1}),\ldots,J_M(\mu_M,\sigma_M)\}, \quad0<k<M, S k := { I 1 ( μ 1 , σ 1 ) , … , I k ( μ k , σ k ) , J k + 1 ( μ k + 1 , σ k + 1 ) , … , J M ( μ M , σ M )} , 0 < k < M ,
and S 0 : = { J i ( μ i , σ i ) : i ∈ [ M ] } S_0:=\{J_i(\mu_i,\sigma_i):i\in[M]\} S 0 := { J i ( μ i , σ i ) : i ∈ [ M ]} , and S M : = { I i ( μ i , σ i ) : i ∈ [ M ] } S_M:=\{I_i(\mu_i,\sigma_i):i\in[M]\} S M := { I i ( μ i , σ i ) : i ∈ [ M ]} . Also,
S k : = S k − 1 ∪ { I k ( μ k , σ k ) } ∖ { J k ( μ k , σ k ) } , S_k := S_{k-1}\cup\{I_k(\mu_k,\sigma_k)\}\setminus\{J_k(\mu_k,\sigma_k)\}, S k := S k − 1 ∪ { I k ( μ k , σ k )} ∖ { J k ( μ k , σ k )} ,
and thus as k k k increases from 0 to 3 a larger KL divergence is replaced by a smaller one, whereas from 4 to 10 a smaller KL divergence is replaced by a larger one.
Since the expected stopping time is approximately inversely proportional to the KL divergences that characterize the difficulty of the underlying testing problem, we expect the overall expected stopping time E [ T ] E[T] E [ T ] to increase as k k k increases from 0 to 3, and to decrease as k k k increases from 4 to 10. This behavior follows from the evolution of the sets S k S_k S k , where initially larger KL divergences are replaced by smaller ones, and subsequently smaller KL divergences are replaced by larger ones. The predicted behavior is observed in Figure 1 and Figure 2 .
Figure 1. Expected stopping time vs number of anomalies ∣ A ∣ |A| ∣ A ∣ , GLLR vs GLLR agnostic
Figure 2. Expected stopping time versus the number of anomalies ∣ A ∣ |A| ∣ A ∣ for ALLR+MLE, and ALLR+KDE vs their agnostic counterparts.
Figure 1 compares the performance of the GLLR combined with the asymptotically optimal sampling frequencies prescribed by condition (38) against the corresponding agnostic sampling rule that samples each source with probability K / M K/M K / M . Similarly, Figure 2 compares the performance of the ALLR combined with the MLE, and KDE under the asymptotically optimal sampling frequencies against their agnostic counterparts. For the MLE, the parameter estimates are computed according to (57) . For the KDE, we employ a Gaussian kernel with bandwidth h i , n = ( N i R ( n ) ) − 0.2 h_{i,n} = (N_i^R(n))^{-0.2} h i , n = ( N i R ( n ) ) − 0.2 .
In all cases, the expected stopping time attains its maximum around ∣ A ∣ = 3 |A| = 3 ∣ A ∣ = 3 , corresponding to the most difficult anomaly configuration. Moreover, the sampling rules based on the asymptotically optimal frequencies consistently outperform their agnostic counterparts. Among the adaptive procedures, ALLR+MLE achieves the smallest expected stopping times, followed by ALLR+KDE, reflecting the superior accuracy of the MLE in the Gaussian setting.
In the long run, for each i ∈ [ M ] i \in[M] i ∈ [ M ] , the estimators
μ ^ ( n ) : = ( μ ^ 1 ( n ) , … , μ ^ M ( n ) ) , σ ^ ( n ) : = ( σ ^ 1 ( n ) , … , σ ^ M ( n ) ) , \widehat{\boldsymbol{\mu}}(n) := (\widehat{\mu}_1(n), \ldots, \widehat{\mu}_M(n)), \qquad\widehat{\boldsymbol{\sigma}}(n) := (\widehat{\sigma}_1(n), \ldots, \widehat{\sigma}_M(n)), μ ( n ) := ( μ 1 ( n ) , … , μ M ( n )) , σ ( n ) := ( σ 1 ( n ) , … , σ M ( n )) ,
converge to the actual parameters
μ : = ( μ 1 , … , μ M ) , σ : = ( σ 1 , … , σ M ) , \boldsymbol{\mu} := (\mu_1, \ldots, \mu_M), \qquad\boldsymbol{\sigma} := (\sigma_1, \ldots, \sigma_M), μ := ( μ 1 , … , μ M ) , σ := ( σ 1 , … , σ M ) ,
and the minimum sampling frequencies that guarantee asymptotic optimality are given by
c i ∗ ( μ , σ ) : = { x A ( μ , σ ) I A ∗ ( μ , σ ) I i ( μ i , σ i ) , if i ∈ A , y A ( μ , σ ) J A ∗ ( μ , σ ) J i ( μ i , σ i ) , if i ∉ A , c_i^*(\boldsymbol{\mu},\boldsymbol{\sigma}) := \begin{cases} x_A(\boldsymbol{\mu},\boldsymbol{\sigma}) \dfrac{I_A^*(\boldsymbol{\mu},\boldsymbol{\sigma})}{I_i(\mu_i,\sigma_i)}, & \text{if } i \in A, \\ y_A(\boldsymbol{\mu},\boldsymbol{\sigma}) \dfrac{J_A^*(\boldsymbol{\mu},\boldsymbol{\sigma})}{J_i(\mu_i,\sigma_i)}, & \text{if } i \notin A, \end{cases} c i ∗ ( μ , σ ) := ⎩ ⎨ ⎧ x A ( μ , σ ) I i ( μ i , σ i ) I A ∗ ( μ , σ ) , y A ( μ , σ ) J i ( μ i , σ i ) J A ∗ ( μ , σ ) , if i ∈ A , if i ∈ / A ,
where x A ( μ , σ ) x_A(\boldsymbol{\mu},\boldsymbol{\sigma}) x A ( μ , σ ) , y A ( μ , σ ) y_A(\boldsymbol{\mu},\boldsymbol{\sigma}) y A ( μ , σ ) are given in Theorem 5.1. We assume r = 1 r = 1 r = 1 , and according to Table I the minimum values of the respective KL divergences are I A ∗ ( μ , σ ) = 0.78 I_A^*(\boldsymbol{\mu},\boldsymbol{\sigma}) = 0.78 I A ∗ ( μ , σ ) = 0.78 , J A ∗ ( μ , σ ) = 0.46 J_A^*(\boldsymbol{\mu},\boldsymbol{\sigma}) = 0.46 J A ∗ ( μ , σ ) = 0.46 .
Table 2. Optimal sampling frequencies c i ∗ ( μ , σ ) c_i^*(\boldsymbol{\mu},\boldsymbol{\sigma}) c i ∗ ( μ , σ ) for ∣ A ∣ = 5 |A| = 5 ∣ A ∣ = 5 , A = { 1 , 2 , 3 , 4 , 5 } A = \{1, 2, 3, 4, 5\} A = { 1 , 2 , 3 , 4 , 5 } .
tbl:II reports the asymptotically optimal sampling frequencies for the case ∣ A ∣ = 5 |A| = 5 ∣ A ∣ = 5 , i.e., A = { 1 , 2 , 3 , 4 , 5 } A = \{1, 2, 3, 4, 5\} A = { 1 , 2 , 3 , 4 , 5 } . We observe that sources associated with smaller KL divergences are assigned larger sampling frequencies. This is consistent with the asymptotic theory, according to which more samples should be allocated to the most difficult sources.
Non-Parametric framework
In the nonparametric framework, the observations from source i ∈ [ M ] i \in[M] i ∈ [ M ] are generated according to an unknown density f i f_i f i ,
X i ( n ) ∼ f i , n ∈ N . X_i(n) \sim f_i,\quad n \in\mathbb{N}. X i ( n ) ∼ f i , n ∈ N .
We consider two classes of densities, which define the two hypotheses,
the class Φ i 0 : = S ↑ \Phi_i^0 := \mathcal{S}_{\uparrow} Φ i 0 := S ↑ contains all densities on [ 0 , 1 ] [0,1] [ 0 , 1 ] that are nondecreasing, uniformly bounded by 2, uniformly equicontinuous, and satisfy
g ( 1 ) − g ( 0 ) ≥ 0.01 , g ∈ S ↑ . g(1)-g(0) \ge0.01,\qquad g \in\mathcal{S}_{\uparrow}. g ( 1 ) − g ( 0 ) ≥ 0.01 , g ∈ S ↑ .
the class Φ i 1 : = S ↓ \Phi_i^1 := \mathcal{S}_{\downarrow} Φ i 1 := S ↓ contains all densities on [ 0 , 1 ] [0,1] [ 0 , 1 ] that are nonincreasing, uniformly bounded by 2, uniformly equicontinuous, and satisfy
g ( 0 ) − g ( 1 ) ≥ 0.01 , g ∈ S ↓ . g(0)-g(1) \ge0.01,\qquad g \in\mathcal{S}_{\downarrow}. g ( 0 ) − g ( 1 ) ≥ 0.01 , g ∈ S ↓ .
If f i ∈ S ↑ f_i \in\mathcal{S}_{\uparrow} f i ∈ S ↑ then source i i i is characterized as regular, whereas if f i ∈ S ↓ f_i \in\mathcal{S}_{\downarrow} f i ∈ S ↓ the source is characterized as anomalous. The above classes satisfy the assumptions imposed in Section IX. The monotonicity constraints ensure separation of the two hypotheses. The true but unknown densities f i f_i f i , used for this simulation, are
f i ( x ) = { − 0.8 x + 1.4 , i ∈ A , 1.6 x + 0.2 , i ∉ A . f_i(x)=
\begin{cases}
-0.8x+1.4, & i \in A,\\
1.6x+0.2, & i \notin A.
\end{cases} f i ( x ) = { − 0.8 x + 1.4 , 1.6 x + 0.2 , i ∈ A , i ∈ / A .
For each i ∈ [ M ] i \in[M] i ∈ [ M ] , we estimate f i f_i f i using the Grenander estimator, which is the nonparametric maximum likelihood estimator for monotone densities [41 ] . The Grenander estimator is obtained by computing the least concave majorant (respectively, the greatest convex minorant) of the empirical distribution function and then taking its left derivative. Since the resulting estimator is piecewise constant, we employ a smoothed version of the Grenander estimator in order to reduce discontinuities and improve numerical stability. Figure 3 illustrates a typical realization of the smoothed Grenander estimator for the density f ( x ) = 1.6 x + 0.2 f(x)=1.6x+0.2 f ( x ) = 1.6 x + 0.2 .
Figure 3. Smoothed Grenander estimator for the density f ( x ) = 1.6 x + 0.2 f(x)=1.6x+0.2 f ( x ) = 1.6 x + 0.2 .
When i ∈ A i \in A i ∈ A , the difficulty of the underlying testing problem corresponding to source i i i is determined by the KL divergence of f i ( x ) = − 0.8 x + 1.4 f_i(x) = -0.8x + 1.4 f i ( x ) = − 0.8 x + 1.4 versus g i 1 ( x ) = 0.01 x + 0.995 g_i^1(x) = 0.01x + 0.995 g i 1 ( x ) = 0.01 x + 0.995 , where the latter is the regular density with the smallest KL distance from f i ( x ) f_i(x) f i ( x ) , and the KL divergence (1) is equal to
I i ( f i ) = D ( f i ∥ g i 1 ) = 0.027. I_i(f_i) = \mathrm{D}(f_i\Vert g_i^1) = 0.027. I i ( f i ) = D ( f i ∥ g i 1 ) = 0.027.
When i ∉ A i \notin A i ∈ / A , the difficulty of the underlying testing problem corresponding to source i i i is determined by the KL divergence of f i ( x ) = 1.6 x + 0.2 f_i(x) = 1.6x + 0.2 f i ( x ) = 1.6 x + 0.2 versus g i 0 ( x ) = − 0.01 x + 1.005 g_i^0(x) = -0.01x + 1.005 g i 0 ( x ) = − 0.01 x + 1.005 , where the latter is the anomalous density with the smallest KL distance from f i ( x ) f_i(x) f i ( x ) , and the KL divergence (2) is equal to
J i ( f i ) = D ( f i ∥ g i 0 ) = 0.116. J_i(f_i) = \mathrm{D}(f_i\Vert g_i^0) = 0.116. J i ( f i ) = D ( f i ∥ g i 0 ) = 0.116.
In case A = { 1 , … , k } A = \{1, \ldots, k\} A = { 1 , … , k } , k ∈ { 0 , … , M } k \in\{0, \ldots, M\} k ∈ { 0 , … , M } the overall difficulty of the anomaly detection problem is determined by the KL divergences
S k ≔ { I 1 ( f 1 ) , … , I k ( f k ) , J k + 1 ( f k + 1 ) , … , J M ( f M ) } . S_k \coloneqq\{I_1(f_1), \ldots, I_k(f_k), J_{k+1}(f_{k+1}), \ldots, J_M(f_M)\}. S k : = { I 1 ( f 1 ) , … , I k ( f k ) , J k + 1 ( f k + 1 ) , … , J M ( f M )} .
Since I i ( f i ) < J i ( f i ) I_i(f_i) < J_i(f_i) I i ( f i ) < J i ( f i ) , an anomalous source is harder to classify than a regular source. Consequently, as the number of anomalies k k k increases, the overall difficulty of the anomaly identification problem increases, and we expect the expected stopping time E [ T ] E[T] E [ T ] to increase monotonically. This behavior is clearly observed in Figure 4 and Figure 5 .
Figure 4. Expected stopping time vs number of anomalies ∣ A ∣ |A| ∣ A ∣ , GLLR vs GLLR agnostic
Figure 5. Expected stopping time versus the number of anomalies ∣ A ∣ |A| ∣ A ∣ for ALLR+MLE, ALLR+KDE vs their agnostic counterparts.
Figure 4 compares the performance of the GLLR combined with the asymptotically optimal sampling frequencies prescribed by condition (38) against the corresponding agnostic sampling rule that samples each source with probability K / M K/M K / M . To compute the GLLR, we evaluate the likelihood ratio at the Grenander MLE. Similarly, Figure 5 compares the performance of ALLR combined with the MLE, and KDE under the asymptotically optimal sampling frequencies against their agnostic counterparts. For the ALLR+MLE procedure, the density is estimated by the smoothed Grenander estimator. For the KDE, we employ a Gaussian kernel with bandwidth h i , n = ( N i R ( n ) ) − 0.2 h_{i,n} = (N_i^R(n))^{-0.2} h i , n = ( N i R ( n ) ) − 0.2 and we restrict and normalize the estimate at [ 0 , 1 ] [0, 1] [ 0 , 1 ] .
In all cases, the expected stopping time increases with the number of anomalies, reflecting the fact that anomalous sources are associated with smaller KL divergences. Moreover, the sampling rules based on the asymptotically optimal frequencies consistently outperform their agnostic counterparts. Among the adaptive procedures, ALLR+MLE achieves the smallest expected stopping times, followed by ALLR+KDE. This ordering is expected, since the Grenander estimator is the nonparametric MLE for monotone densities and is therefore well matched to the underlying model.
In the long run, for each i ∈ [ M ] i \in[M] i ∈ [ M ] , the estimator f ^ n \hat{f}_n f ^ n converges to the actual densities f f f . We note that, in the present simulation setup, all anomalous (resp. regular) sources share the same KL divergence. Consequently, I A ∗ ( f ) = I I_A^*(f) = I I A ∗ ( f ) = I , J A ∗ ( f ) = J J_A^*(f) = J J A ∗ ( f ) = J , and
K ^ A ( f ) = ∣ A ∣ , K ˘ A ( f ) = M − ∣ A ∣ , s A ∗ ( f ) = I / J . \hat{K}_A(f) = |A|,\qquad\breve{K}_A(f) = M - |A|,\qquad s_A^*(f) = I/J. K ^ A ( f ) = ∣ A ∣ , K ˘ A ( f ) = M − ∣ A ∣ , s A ∗ ( f ) = I / J .
For K = 5 K = 5 K = 5 , M = 10 M = 10 M = 10 , and assuming r = 1 r = 1 r = 1 , the minimum sampling frequencies that guarantee asymptotic optimality are given by
c i ∗ ( f ) = 0 , ∀ i ∈ A , c j ∗ ( f ) = K M − ∣ A ∣ ∧ 1 , ∀ j ∉ A , c_i^*(f) = 0,\qquad\forall i \in A,\qquad c_j^*(f) = \frac{K}{M-|A|} \wedge1,\qquad\forall j \notin A, c i ∗ ( f ) = 0 , ∀ i ∈ A , c j ∗ ( f ) = M − ∣ A ∣ K ∧ 1 , ∀ j ∈ / A ,
c i ∗ ( f ) = K J ∣ A ∣ J + ( M − ∣ A ∣ ) I ∧ J I ∧ 1 , ∀ i ∈ A , c j ∗ ( f ) = K I ∣ A ∣ J + ( M − ∣ A ∣ ) I ∧ I J ∧ 1 , ∀ j ∉ A , c_i^*(f) = \frac{KJ}{|A|J+(M-|A|)I}\wedge\frac{J}{I}\wedge1,\qquad\forall i \in A,\qquad c_j^*(f) = \frac{KI}{|A|J+(M-|A|)I}\wedge\frac{I}{J}\wedge1,\qquad\forall j \notin A, c i ∗ ( f ) = ∣ A ∣ J + ( M − ∣ A ∣ ) I K J ∧ I J ∧ 1 , ∀ i ∈ A , c j ∗ ( f ) = ∣ A ∣ J + ( M − ∣ A ∣ ) I K I ∧ J I ∧ 1 , ∀ j ∈ / A ,
(iii) if ∣ A ∣ = M |A| = M ∣ A ∣ = M
c i ∗ ( f ) = K ∣ A ∣ ∧ 1 , ∀ i ∈ A , c j ∗ ( f ) = 0 , ∀ j ∉ A . c_i^*(f) = \frac{K}{|A|} \wedge1,\quad\forall i \in A,\qquad c_j^*(f) = 0,\quad\forall j \notin A. c i ∗ ( f ) = ∣ A ∣ K ∧ 1 , ∀ i ∈ A , c j ∗ ( f ) = 0 , ∀ j ∈ / A .
Simulation Study - Robustness in model misspecification
We compare GLLR with ALLR combined with KDE under model misspecification and sampling constraints. When the parametric model is correctly specified, GLLR can benefit from the efficiency of the MLE. Under misspecification, however, the fitted parametric density may remain systematically different from the true density, whereas KDE can approximate it more accurately. This can improve the estimation of KL numbers and lead to a more efficient allocation of samples across sources.
We fix constants B , C ≥ 1 B, C \ge1 B , C ≥ 1 and 0 < c < 1 0 < c < 1 0 < c < 1 , we denote by
H : = { N ( μ , σ 2 ) : − B ≤ μ ≤ B , c ≤ σ ≤ C } \mathcal{H} := \left\{\mathcal{N}(\mu, \sigma^2) : -B \le\mu\le B,\ c \le\sigma\le C\right\} H := { N ( μ , σ 2 ) : − B ≤ μ ≤ B , c ≤ σ ≤ C }
the set of Gaussian densities with means in [ − B , B ] [-B, B] [ − B , B ] and standard deviations in [ c , C ] [c, C] [ c , C ] , and by H ‾ \overline{\mathcal{H}} H the closed convex hull of H \mathcal{H} H in L 1 L^1 L 1 . Since H \mathcal{H} H is compact in L 1 L^1 L 1 , its closed convex hull H ‾ \overline{\mathcal{H}} H is also compact in L 1 L^1 L 1 . For a fixed 0 < ϵ ≤ B 0 < \epsilon\le B 0 < ϵ ≤ B , we consider the classes
H 1 : = { f ∈ H ‾ : ϵ ≤ E f [ X ] ≤ B , c 2 ≤ Var f ( X ) ≤ C 2 } (58) \mathcal{H}_1 := \left\{f \in\overline{\mathcal{H}} : \epsilon\le\mathbb{E}_f[X] \le B,\ c^2 \le\operatorname{Var}_f(X) \le C^2\right\} \tag*{(58)} H 1 := { f ∈ H : ϵ ≤ E f [ X ] ≤ B , c 2 ≤ Var f ( X ) ≤ C 2 } (58)
and
H 0 : = { N ( μ , σ 2 ) : − B ≤ μ ≤ 0 , c ≤ σ ≤ C } . (59) \mathcal{H}_0 := \left\{\mathcal{N}(\mu, \sigma^2) : -B \le\mu\le0,\ c \le\sigma\le C\right\}. \tag*{(59)} H 0 := { N ( μ , σ 2 ) : − B ≤ μ ≤ 0 , c ≤ σ ≤ C } . (59)
Both H 1 , H 0 \mathcal{H}_1, \mathcal{H}_0 H 1 , H 0 are closed subsets of the compact set H ‾ \overline{\mathcal{H}} H and thus themselves compact.
We consider two sources, i.e., M = 2 M = 2 M = 2 , and we impose the following composite hypotheses testing problem on each source,
H i 1 : f i ∈ H 1 , H_i^1 : f_i \in\mathcal{H}_1, H i 1 : f i ∈ H 1 ,
H i 0 : f i ∈ H 0 , (60) H_i^0 : f_i \in\mathcal{H}_0, \tag*{(60)} H i 0 : f i ∈ H 0 , (60)
where i ∈ [ 2 ] i \in[2] i ∈ [ 2 ] . We can sample only one source at a time, i.e., K = 1 K = 1 K = 1 . The underlying true densities are
f i : = 1 2 N ( m i − d , s 2 ) + 1 2 N ( m i + d , s 2 ) , i ∈ [ 2 ] , (61) f_i := \frac{1}{2}\mathcal{N}(m_i-d, s^2) + \frac{1}{2}\mathcal{N}(m_i+d, s^2),\qquad i \in[2], \tag*{(61)} f i := 2 1 N ( m i − d , s 2 ) + 2 1 N ( m i + d , s 2 ) , i ∈ [ 2 ] , (61)
where s ∈ ( 0 , 1 ) s \in(0,1) s ∈ ( 0 , 1 ) and d > 0 d > 0 d > 0 satisfy
c ≤ s , s 2 + d 2 + 1 ≤ C 2 , 1 + d ≤ B . c \le s,\qquad s^2 + d^2 + 1 \le C^2,\qquad1 + d \le B. c ≤ s , s 2 + d 2 + 1 ≤ C 2 , 1 + d ≤ B .
The source means are
m i : = { 1 , i = 1 , ϵ , i = 2. (62) m_i := \begin{cases}
1, & i = 1,\\
\epsilon, & i = 2.
\end{cases} \tag*{(62)} m i := { 1 , ϵ , i = 1 , i = 2. (62)
where ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) . Each density in (61) has mean and variance
E f i [ X ] = m i , Var f i ( X ) = s 2 + d 2 . \mathbb{E}_{f_i}[X] = m_i, \qquad\operatorname{Var}_{f_i}(X) = s^2 + d^2. E f i [ X ] = m i , Var f i ( X ) = s 2 + d 2 .
The parameter restrictions ensure that both Gaussian components belong to H \mathcal{H} H and that the resulting densities satisfy f 1 , f 2 ∈ H 1 f_1,f_2 \in\mathcal{H}_1 f 1 , f 2 ∈ H 1 . Thus, both sources are anomalous.
Misspecified parametric model.
Due to the complexity of the alternative class H 1 \mathcal{H}_1 H 1 , a practitioner may approximate it by the Gaussian subclass H 1 s \mathcal{H}_1^s H 1 s , while retaining the original null class. Thus, the original and surrogate problems have the same Gaussian null, and the model misspecification concerns only the alternative. Specifically, we define
H 1 s : = { N ( μ , σ 2 ) : ϵ ≤ μ ≤ B , c ≤ σ ≤ C } , (63) \mathcal{H}_1^s := \left\{\mathcal{N}(\mu,\sigma^2): \epsilon\le\mu\le B,\ c \le\sigma\le C\right\}, \tag*{(63)} H 1 s := { N ( μ , σ 2 ) : ϵ ≤ μ ≤ B , c ≤ σ ≤ C } , (63)
and the corresponding surrogate hypotheses are
H i 1 , s : f i ∈ H 1 s , (64) \mathcal{H}_i^{1,s}: f_i \in\mathcal{H}_1^s, \tag*{(64)} H i 1 , s : f i ∈ H 1 s , (64)
H i 0 , s : f i ∈ H 0 . \mathcal{H}_i^{0,s}: f_i \in\mathcal{H}_0. H i 0 , s : f i ∈ H 0 .
where i ∈ [ 2 ] i \in[2] i ∈ [ 2 ] . Set H 1 s \mathcal{H}_1^s H 1 s is compact in L 1 L^1 L 1 . The observations are generated from the true mixture densities in (61) . Their means are given by (62) , and their common variance is s 2 + d 2 s^2+d^2 s 2 + d 2 .
We compare the performance of GLLR with that of ALLR combined with KDE, both operating under the simplified Gaussian hypotheses. GLLR uses the MLE to exploit the assumed Gaussian structure, which can yield efficient parameter estimation when the model is correctly specified. ALLR combined with KDE estimates the underlying density without imposing a Gaussian form. Under model misspecification, this flexibility can improve the estimation of the true information numbers used to determine the optimal sampling frequencies and thereby yield a smaller first-order asymptotic approximation to the expected stopping time.
Computation of optimal sampling frequencies
GLLR for the simplified problem
1) GLLR for the simplified problem: GLLR uses the MLE to estimate the Gaussian mean and variance from the observations sampled from each source. The Gaussian MLEs are given by the empirical mean and standard deviation,
μ ^ i ( n ) : = 1 N i R ( n ) ∑ m = 1 n X i ( m ) R i ( m ) , \widehat{\mu}_i(n) := \frac{1}{N_i^R(n)}\sum_{m=1}^{n} X_i(m)R_i(m), μ i ( n ) := N i R ( n ) 1 m = 1 ∑ n X i ( m ) R i ( m ) ,
σ ^ i ( n ) : = [ 1 N i R ( n ) ∑ m = 1 n ( X i ( m ) − μ ^ i ( n ) ) 2 R i ( m ) ] 1 / 2 . \widehat{\sigma}_i(n) := \left[\frac{1}{N_i^R(n)}\sum_{m=1}^{n}\left(X_i(m)-\widehat{\mu}_i(n)\right)^2R_i(m)\right]^{1/2}. σ i ( n ) := [ N i R ( n ) 1 m = 1 ∑ n ( X i ( m ) − μ i ( n ) ) 2 R i ( m ) ] 1/2 .
Almost surely, as n → ∞ n \to\infty n → ∞ , the empirical estimators converge to the true values
( ( μ ^ i ( n ) , σ ^ i ( n ) ) ) ⟶ ( m i , s 2 + d 2 ) (65) \left((\hat{\mu}_i(n),\hat{\sigma}_i(n))\right) \longrightarrow\left(m_i,\sqrt{s^2+d^2}\right) \tag*{(65)} ( ( μ ^ i ( n ) , σ ^ i ( n )) ) ⟶ ( m i , s 2 + d 2 ) (65)
where i ∈ [ 2 ] i \in[2] i ∈ [ 2 ] . Therefore, for any Gaussian density N ( μ , σ 2 ) \mathcal{N}(\mu,\sigma^2) N ( μ , σ 2 ) , the KL divergence from the fitted Gaussian N ( m i , s 2 + d 2 ) \mathcal{N}(m_i,s^2+d^2) N ( m i , s 2 + d 2 ) is
D ( N ( m i , s 2 + d 2 ) ∥ N ( μ , σ 2 ) ) = 1 2 [ log σ 2 s 2 + d 2 + s 2 + d 2 + ( m i − μ ) 2 σ 2 − 1 ] . D\left(\mathcal{N}(m_i,s^2+d^2)\mathbin\Vert\mathcal{N}(\mu,\sigma^2)\right)=\frac{1}{2}\left[\log\frac{\sigma^2}{s^2+d^2}+\frac{s^2+d^2+(m_i-\mu)^2}{\sigma^2}-1\right]. D ( N ( m i , s 2 + d 2 ) ∥ N ( μ , σ 2 ) ) = 2 1 [ log s 2 + d 2 σ 2 + σ 2 s 2 + d 2 + ( m i − μ ) 2 − 1 ] .
The minimizing mean over [ − B , 0 ] [-B,0] [ − B , 0 ] is μ = 0 \mu=0 μ = 0 , and the minimizing variance is s 2 + d 2 + m i 2 s^2+d^2+m_i^2 s 2 + d 2 + m i 2 , and thus
arg min g ∈ H 0 D ( N ( m i , s 2 + d 2 ) ∥ g ) = N ( 0 , s 2 + d 2 + m i 2 ) , i ∈ [ 2 ] . \underset{g\in\mathcal{H}_0}{\arg\min}\ D\left(\mathcal{N}(m_i,s^2+d^2)\mathbin\Vert g\right)=\mathcal{N}(0,s^2+d^2+m_i^2),\qquad i\in[2]. g ∈ H 0 arg min D ( N ( m i , s 2 + d 2 ) ∥ g ) = N ( 0 , s 2 + d 2 + m i 2 ) , i ∈ [ 2 ] .
The corresponding information numbers, under the assumption of Gaussian model, are therefore
I 1 G : = inf g ∈ H 0 D ( N ( 1 , s 2 + d 2 ) ∥ g ) = 1 2 log ( 1 + 1 s 2 + d 2 ) , (66) I_1^G:=\inf_{g\in\mathcal{H}_0}D\left(\mathcal{N}(1,s^2+d^2)\mathbin\Vert g\right)=\frac{1}{2}\log\left(1+\frac{1}{s^2+d^2}\right), \tag*{(66)} I 1 G := g ∈ H 0 inf D ( N ( 1 , s 2 + d 2 ) ∥ g ) = 2 1 log ( 1 + s 2 + d 2 1 ) , (66)
I 2 G ( ϵ ) : = inf g ∈ H 0 D ( N ( ϵ , s 2 + d 2 ) ∥ g ) = 1 2 log ( 1 + ϵ 2 s 2 + d 2 ) . (67) I_2^G(\epsilon):=\inf_{g\in\mathcal{H}_0}D\left(\mathcal{N}(\epsilon,s^2+d^2)\mathbin\Vert g\right)=\frac{1}{2}\log\left(1+\frac{\epsilon^2}{s^2+d^2}\right). \tag*{(67)} I 2 G ( ϵ ) := g ∈ H 0 inf D ( N ( ϵ , s 2 + d 2 ) ∥ g ) = 2 1 log ( 1 + s 2 + d 2 ϵ 2 ) . (67)
Consequently, the sampling frequencies that satisfy
c 1 G ( ϵ ) I 1 G = c 2 G ( ϵ ) I 2 G ( ϵ ) , c 1 G ( ϵ ) + c 2 G ( ϵ ) = 1 , c_1^G(\epsilon)I_1^G=c_2^G(\epsilon)I_2^G(\epsilon),\qquad c_1^G(\epsilon)+c_2^G(\epsilon)=1, c 1 G ( ϵ ) I 1 G = c 2 G ( ϵ ) I 2 G ( ϵ ) , c 1 G ( ϵ ) + c 2 G ( ϵ ) = 1 ,
are
c 1 G ( ϵ ) = I 2 G ( ϵ ) I 1 G + I 2 G ( ϵ ) , c 2 G ( ϵ ) = I 1 G I 1 G + I 2 G ( ϵ ) . (68) c_1^G(\epsilon)=\frac{I_2^G(\epsilon)}{I_1^G+I_2^G(\epsilon)},\qquad c_2^G(\epsilon)=\frac{I_1^G}{I_1^G+I_2^G(\epsilon)}. \tag*{(68)} c 1 G ( ϵ ) = I 1 G + I 2 G ( ϵ ) I 2 G ( ϵ ) , c 2 G ( ϵ ) = I 1 G + I 2 G ( ϵ ) I 1 G . (68)
We note that as ϵ → 0 \epsilon\to0 ϵ → 0 , the sampling frequencies c 1 G ( ϵ ) → 0 c_1^G(\epsilon)\to0 c 1 G ( ϵ ) → 0 , and c 2 G ( ϵ ) → 1 c_2^G(\epsilon)\to1 c 2 G ( ϵ ) → 1 .
ALLR+KDE for the simplified problem
Although the MLE incorporates the assumed parametric structure when estimating the unknown parameters, the KDE estimates the underlying density without imposing a Gaussian form. Under the KDE consistency conditions, and since each source is sampled infinitely often, we have
∥ f ^ i , n − f i ∥ 1 → 0 , a.s. , i ∈ [ 2 ] , (69) \left\|\hat{f}_{i,n}-f_i\right\|_1\to0,\quad\text{a.s.},\qquad i\in[2], \tag*{(69)} f ^ i , n − f i 1 → 0 , a.s. , i ∈ [ 2 ] , (69)
where each f i f_i f i is defined in (61) . For each source, the Gaussian null density minimizing the KL divergence from the true density is
arg min g ∈ H 0 D ( f i ∥ g ) = N ( 0 , ν i 2 ) , i ∈ [ 2 ] , \underset{g\in\mathcal{H}_0}{\arg\min}\ D(f_i\mathbin\Vert g)=\mathcal{N}(0,\nu_i^2),\qquad i\in[2], g ∈ H 0 arg min D ( f i ∥ g ) = N ( 0 , ν i 2 ) , i ∈ [ 2 ] ,
where
ν i 2 : = m i 2 + s 2 + d 2 , i ∈ [ 2 ] . (70) \nu_i^2:=m_i^2+s^2+d^2,\qquad i\in[2]. \tag*{(70)} ν i 2 := m i 2 + s 2 + d 2 , i ∈ [ 2 ] . (70)
To show this, let us consider a g = N ( μ , σ 2 ) g = \mathcal{N}(\mu,\sigma^2) g = N ( μ , σ 2 ) , where μ ∈ [ − B , 0 ] \mu\in[-B,0] μ ∈ [ − B , 0 ] and σ ∈ [ c , C ] \sigma\in[c,C] σ ∈ [ c , C ] . We have
D ( f i , g ) = ∫ R f i ( x ) log f i ( x ) d x − ∫ R f i ( x ) log g ( x ) d x = ∫ R f i ( x ) log f i ( x ) d x + 1 2 log ( 2 π σ 2 ) + E f i [ ( X − μ ) 2 ] 2 σ 2 . \begin{aligned}
D(f_i,g)=\int_{\mathbb{R}} f_i(x)\log f_i(x)\,dx-\int_{\mathbb{R}} f_i(x)\log g(x)\,dx \\
&=\int_{\mathbb{R}} f_i(x)\log f_i(x)\,dx+\frac{1}{2}\log(2\pi\sigma^2)+\frac{\mathbb{E}_{f_i}[(X-\mu)^2]}{2\sigma^2}.
\end{aligned} D ( f i , g ) = ∫ R f i ( x ) log f i ( x ) d x − ∫ R f i ( x ) log g ( x ) d x = ∫ R f i ( x ) log f i ( x ) d x + 2 1 log ( 2 π σ 2 ) + 2 σ 2 E f i [( X − μ ) 2 ] .
Since
E f i [ X ] = m i , Var f i ( X ) = s 2 + d 2 , \mathbb{E}_{f_i}[X]=m_i,\qquad\operatorname{Var}_{f_i}(X)=s^2+d^2, E f i [ X ] = m i , Var f i ( X ) = s 2 + d 2 ,
we obtain
E f i [ ( X − μ ) 2 ] = E f i [ ( X − m i + m i − μ ) 2 ] = s 2 + d 2 + ( m i − μ ) 2 , \begin{aligned}
\mathbb{E}_{f_i}[(X-\mu)^2]=\mathbb{E}_{f_i}[(X-m_i+m_i-\mu)^2] \\
&=s^2+d^2+(m_i-\mu)^2,
\end{aligned} E f i [( X − μ ) 2 ] = E f i [( X − m i + m i − μ ) 2 ] = s 2 + d 2 + ( m i − μ ) 2 ,
which gives
D ( f i , g ) = ∫ R f i ( x ) log f i ( x ) d x + 1 2 log ( 2 π σ 2 ) + s 2 + d 2 + ( m i − μ ) 2 2 σ 2 . D(f_i,g)=\int_{\mathbb{R}} f_i(x)\log f_i(x)\,dx+\frac{1}{2}\log(2\pi\sigma^2)+\frac{s^2+d^2+(m_i-\mu)^2}{2\sigma^2}. D ( f i , g ) = ∫ R f i ( x ) log f i ( x ) d x + 2 1 log ( 2 π σ 2 ) + 2 σ 2 s 2 + d 2 + ( m i − μ ) 2 .
The expression is minimized over μ ∈ [ − B , 0 ] \mu\in[-B,0] μ ∈ [ − B , 0 ] at μ = 0 \mu=0 μ = 0 , and then the minimizing variance is (70) . In order to compute the corresponding information numbers, we set
g i : = N ( 0 , ν i 2 ) , f 0 : = 1 2 N ( − d , s 2 ) + 1 2 N ( d , s 2 ) , g_i:=\mathcal{N}(0,\nu_i^2),\qquad f_0:=\frac{1}{2}\mathcal{N}(-d,s^2)+\frac{1}{2}\mathcal{N}(d,s^2), g i := N ( 0 , ν i 2 ) , f 0 := 2 1 N ( − d , s 2 ) + 2 1 N ( d , s 2 ) ,
and we note that f i ( x ) = f 0 ( x − m i ) f_i(x)=f_0(x-m_i) f i ( x ) = f 0 ( x − m i ) , i ∈ [ 2 ] i\in[2] i ∈ [ 2 ] . Thus,
I i ( ϵ ) : = D ( f i , g ) = ∫ R f 0 ( x − m i ) log f 0 ( x − m i ) g i ( x ) d x = ∫ R f 0 ( y ) log f 0 ( y ) g i ( y + m i ) d y = ∫ R f 0 ( y ) log f 0 ( y ) g i ( y ) d y + ∫ R f 0 ( y ) log g i ( y ) g i ( y + m i ) d y . \begin{aligned}
I_i(\epsilon):=D(f_i,g)=\int_{\mathbb{R}}f_0(x-m_i)\log\frac{f_0(x-m_i)}{g_i(x)}\,dx \\
&=\int_{\mathbb{R}}f_0(y)\log\frac{f_0(y)}{g_i(y+m_i)}\,dy \\
&=\int_{\mathbb{R}}f_0(y)\log\frac{f_0(y)}{g_i(y)}\,dy+\int_{\mathbb{R}}f_0(y)\log\frac{g_i(y)}{g_i(y+m_i)}\,dy.
\end{aligned} I i ( ϵ ) := D ( f i , g ) = ∫ R f 0 ( x − m i ) log g i ( x ) f 0 ( x − m i ) d x = ∫ R f 0 ( y ) log g i ( y + m i ) f 0 ( y ) d y = ∫ R f 0 ( y ) log g i ( y ) f 0 ( y ) d y + ∫ R f 0 ( y ) log g i ( y + m i ) g i ( y ) d y .
For the centered Gaussian density g i g_i g i ,
log g i ( y ) g i ( y + m i ) = m i y ν i 2 + m i 2 2 ν i 2 , \log\frac{g_i(y)}{g_i(y+m_i)}=\frac{m_i y}{\nu_i^2}+\frac{m_i^2}{2\nu_i^2}, log g i ( y + m i ) g i ( y ) = ν i 2 m i y + 2 ν i 2 m i 2 ,
and since E f 0 [ Y ] = 0 \mathbb{E}_{f_0}[Y]=0 E f 0 [ Y ] = 0 , we get
I i ( ϵ ) = D ( f 0 ∥ g i ) + m i 2 2 ν i 2 , i ∈ [ 2 ] . (71) I_i(\epsilon)=D(f_0\Vert g_i)+\frac{m_i^2}{2\nu_i^2},\qquad i\in[2]. \tag*{(71)} I i ( ϵ ) = D ( f 0 ∥ g i ) + 2 ν i 2 m i 2 , i ∈ [ 2 ] . (71)
Consequently, the sampling frequencies that satisfy
c 1 ( ϵ ) I 1 ( ϵ ) = c 2 ( ϵ ) I 2 ( ϵ ) , c 1 ( ϵ ) + c 2 ( ϵ ) = 1 , c_1(\epsilon)I_1(\epsilon)=c_2(\epsilon)I_2(\epsilon),\qquad c_1(\epsilon)+c_2(\epsilon)=1, c 1 ( ϵ ) I 1 ( ϵ ) = c 2 ( ϵ ) I 2 ( ϵ ) , c 1 ( ϵ ) + c 2 ( ϵ ) = 1 ,
are
c 1 ( ϵ ) = I 2 ( ϵ ) I 1 ( ϵ ) + I 2 ( ϵ ) , c 2 ( ϵ ) = I 1 ( ϵ ) I 1 ( ϵ ) + I 2 ( ϵ ) . (72) c_1(\epsilon)=\frac{I_2(\epsilon)}{I_1(\epsilon)+I_2(\epsilon)},\qquad c_2(\epsilon)=\frac{I_1(\epsilon)}{I_1(\epsilon)+I_2(\epsilon)}. \tag*{(72)} c 1 ( ϵ ) = I 1 ( ϵ ) + I 2 ( ϵ ) I 2 ( ϵ ) , c 2 ( ϵ ) = I 1 ( ϵ ) + I 2 ( ϵ ) I 1 ( ϵ ) . (72)
In contrast to (68) , both
lim ϵ → 0 + c 1 A ( ϵ ) , lim ϵ → 0 + c 2 A ( ϵ ) > 0 , (73) \lim_{\epsilon\to0^+}c_1^A(\epsilon),\qquad\lim_{\epsilon\to0^+}c_2^A(\epsilon)>0, \tag*{(73)} ϵ → 0 + lim c 1 A ( ϵ ) , ϵ → 0 + lim c 2 A ( ϵ ) > 0 , (73)
due to the fact that lim ϵ → 0 I i ( ϵ ) ≥ D ( f 0 ∥ N ( 0 , s 2 + d 2 ) ) > 0 \lim_{\epsilon\to0} I_i(\epsilon)\ge D(f_0\Vert\mathcal{N}(0,s^2+d^2))>0 lim ϵ → 0 I i ( ϵ ) ≥ D ( f 0 ∥ N ( 0 , s 2 + d 2 )) > 0 , for both i ∈ [ 2 ] i\in[2] i ∈ [ 2 ] .
Effect on the expected stopping time
We examine how replacing the original problem (60) by the Gaussian surrogate (64) affects the expected stopping time asymptotically. We denote by T G R T_G^R T G R the stopping time obtained using GLLR under the simplified Gaussian model, and by T A K R T_{AK}^R T A K R the stopping time obtained using ALLR combined with KDE. Both procedures use stopping rule (22), with their respective statistics. The quantities I i ( ϵ ) I_i(\epsilon) I i ( ϵ ) govern the limiting growth of the ALLR statistic, whereas I i G ( ϵ ) I_i^G(\epsilon) I i G ( ϵ ) govern the limiting growth of the GLLR statistic under the true densities. Although I i G ( ϵ ) I_i^G(\epsilon) I i G ( ϵ ) are defined in (66) assuming that the fitted Gaussian is the true density, in this example they also equal the limiting GLLR drifts under the true densities because f i f_i f i and N ( m i , s 2 + d 2 ) \mathcal{N}(m_i,s^2+d^2) N ( m i , s 2 + d 2 ) share the same mean and variance. For fixed ϵ > 0 \epsilon>0 ϵ > 0 , we have
E [ 2 ] [ T G R ] ∼ ∣ log α ∣ min i ∈ [ 2 ] c i G ( ϵ ) I i G ( ϵ ) = ∣ log α ∣ c 1 G ( ϵ ) I 1 G ( ϵ ) , α → 0 , (74) \mathbb{E}_{[2]}[T_G^R]\sim\frac{|\log\alpha|}{\min_{i\in[2]}c_i^G(\epsilon)I_i^G(\epsilon)}=\frac{|\log\alpha|}{c_1^G(\epsilon)I_1^G(\epsilon)},\qquad\alpha\to0, \tag*{(74)} E [ 2 ] [ T G R ] ∼ min i ∈ [ 2 ] c i G ( ϵ ) I i G ( ϵ ) ∣ log α ∣ = c 1 G ( ϵ ) I 1 G ( ϵ ) ∣ log α ∣ , α → 0 , (74)
E [ 2 ] [ T A K R ] ∼ ∣ log α ∣ min i ∈ [ 2 ] c i ( ϵ ) I i ( ϵ ) = ∣ log α ∣ c 1 ( ϵ ) I 1 ( ϵ ) , α → 0 , \mathbb{E}_{[2]}[T_{AK}^R]\sim\frac{|\log\alpha|}{\min_{i\in[2]}c_i(\epsilon)I_i(\epsilon)}=\frac{|\log\alpha|}{c_1(\epsilon)I_1(\epsilon)},\qquad\alpha\to0, E [ 2 ] [ T A K R ] ∼ min i ∈ [ 2 ] c i ( ϵ ) I i ( ϵ ) ∣ log α ∣ = c 1 ( ϵ ) I 1 ( ϵ ) ∣ log α ∣ , α → 0 ,
and
R ( ϵ ) : = lim α → 0 E [ 2 ] [ T G R ] E [ 2 ] [ T A K R ] = c 1 ( ϵ ) I 1 ( ϵ ) c 1 G ( ϵ ) I 1 G ( ϵ ) . (75) \mathcal{R}(\epsilon):=\lim_{\alpha\to0}\frac{\mathbb{E}_{[2]}[T_G^R]}{\mathbb{E}_{[2]}[T_{AK}^R]}=\frac{c_1(\epsilon)I_1(\epsilon)}{c_1^G(\epsilon)I_1^G(\epsilon)}. \tag*{(75)} R ( ϵ ) := α → 0 lim E [ 2 ] [ T A K R ] E [ 2 ] [ T G R ] = c 1 G ( ϵ ) I 1 G ( ϵ ) c 1 ( ϵ ) I 1 ( ϵ ) . (75)
Letting ϵ → 0 \epsilon\to0 ϵ → 0 , c 1 ( ϵ ) I 1 ( ϵ ) c_1(\epsilon)I_1(\epsilon) c 1 ( ϵ ) I 1 ( ϵ ) converges to a strictly positive constant, whereas c 1 G ( ϵ ) I 1 G ( ϵ ) → 0 c_1^G(\epsilon)I_1^G(\epsilon)\to0 c 1 G ( ϵ ) I 1 G ( ϵ ) → 0 . Therefore, R ( ϵ ) → ∞ \mathcal{R}(\epsilon)\to\infty R ( ϵ ) → ∞ , which means that for sufficiently small ϵ \epsilon ϵ , the first-order asymptotic expected stopping time of GLLR can be arbitrarily larger than that of ALLR combined with KDE.
Simulation
We simulate two independent sources in which both alternative hypotheses hold. The observations from source i i i have density
f i = 1 2 N ( m i − d , s 2 ) + 1 2 N ( m i + d , s 2 ) , m 1 = 1 , m 2 = ϵ , f_i=\frac{1}{2}\mathcal{N}(m_i-d,s^2)+\frac{1}{2}\mathcal{N}(m_i+d,s^2),\qquad m_1=1,\quad m_2=\epsilon, f i = 2 1 N ( m i − d , s 2 ) + 2 1 N ( m i + d , s 2 ) , m 1 = 1 , m 2 = ϵ ,
where s = 0.25 s=0.25 s = 0.25 and d = 1 − s 2 d=\sqrt{1-s^2} d = 1 − s 2 . Thus, both true variances equal one. We take B = 3 B=3 B = 3 , c = 0.1 c=0.1 c = 0.1 , and C = 2 C=2 C = 2 . The shared Gaussian null has mean in [ − B , 0 ] [-B,0] [ − B , 0 ] and variance in [ c 2 , C 2 ] [c^2,C^2] [ c 2 , C 2 ] , while the Gaussian surrogate alternative has mean in [ ϵ , B ] [\epsilon,B] [ ϵ , B ] and the same variance bounds.
In the following simulations, we compare E [ 2 ] [ T G R ] \mathbb{E}_{[2]}[T_G^R] E [ 2 ] [ T G R ] and E [ 2 ] [ T A K R ] \mathbb{E}_{[2]}[T_{AK}^R] E [ 2 ] [ T A K R ] for fixed, small values of ϵ \epsilon ϵ and decreasing values of α \alpha α . Figure 6 shows the ratio E [ 2 ] [ T G R ] / E [ 2 ] [ T A K R ] \mathbb{E}_{[2]}[T_G^R]/\mathbb{E}_{[2]}[T_{AK}^R] E [ 2 ] [ T G R ] / E [ 2 ] [ T A K R ] at target error probabilities α \alpha α versus ∣ log ( α ) ∣ ∈ { 1 , … , 10 } |\log(\alpha)|\in\{1,\ldots,10\} ∣ log ( α ) ∣ ∈ { 1 , … , 10 } . For each target error, stopping thresholds were selected using simulations so that the estimate of the error probability was as close as possible to α \alpha α . The Monte Carlo standard error of each estimated stopping time ratio is below 0.1, for each value of ϵ \epsilon ϵ . The ratio is larger for smaller ϵ \epsilon ϵ , consistent with R ( ϵ ) → ∞ \mathcal{R}(\epsilon)\to\infty R ( ϵ ) → ∞ as ϵ → 0 \epsilon\to0 ϵ → 0 .
Figure 6. Ratio of expected stopping times versus ∣ log ( α ) ∣ |\log(\alpha)| ∣ log ( α ) ∣ .
In the following simulations, we compare E [ 2 ] [ T G R ] \mathbb{E}_{[2]}[T_G^R] E [ 2 ] [ T G R ] and E [ 2 ] [ T A K R ] \mathbb{E}_{[2]}[T_{AK}^R] E [ 2 ] [ T A K R ] for small, fixed target error probabilities α \alpha α . Figure 7 shows the estimated ratio of the expected stopping time of the surrogate GLLR to that of the ALLR with KDE versus 1 / ϵ 1/\epsilon 1/ ϵ , for ϵ ∈ { 0.05 , 0.1 , 0.2 } \epsilon\in\{0.05, 0.1, 0.2\} ϵ ∈ { 0.05 , 0.1 , 0.2 } . For each fixed ϵ \epsilon ϵ , smaller target error probabilities yield larger ratios. The Monte Carlo standard error of every estimated ratio is below 0.1.
Figure 7. Ratio of expected stopping times versus 1 / ϵ 1/\epsilon 1/ ϵ .
Conclusion
We studied sequential anomaly identification under sampling constraints and composite hypotheses, with particular emphasis on the nonparametric setting. We derived a universal asymptotic lower bound on the expected stopping time and established sufficient conditions for attaining this bound as the error probabilities vanish. These conditions identify the sampling frequencies needed for first-order asymptotic optimality. We also proposed suitable test statistics and density estimators and showed, under appropriate assumptions, that they satisfy the corresponding convergence conditions.
The framework also accommodates generalized error criteria that tolerate a prescribed number of classification errors. In [42 ] , we developed the corresponding asymptotic optimality conditions for simple hypotheses and a more restrictive sampling constraint. The same methodology applies directly to the present composite setting by replacing the Kullback–Leibler divergences between simple hypotheses with the information quantities defined in (1) and (2) . Directions for future research include extending the framework to non-i.i.d. observations from each source [43 ] and incorporating switching costs into the sampling process [44 ] .
Appendix A
In this appendix, we verify that regularity conditions (a)–(g) in Subsection VIII-A hold for regular minimal exponential families restricted to compact parameter sets. We also prove Proposition 8.1, which extends these conditions to mixtures of densities from these families.
For each i ∈ [ M ] i \in[M] i ∈ [ M ] , consider the exponential family
g θ ( x ) = h i ( x ) exp { θ T T i ( x ) − b i ( θ ) } , θ ∈ Θ i , g_{\theta}(x) = h_i(x) \exp\{\theta^{\mathsf{T}} T_i(x) - b_i(\theta)\}, \qquad\theta\in\Theta_i, g θ ( x ) = h i ( x ) exp { θ T T i ( x ) − b i ( θ )} , θ ∈ Θ i ,
where h i : S → ( 0 , ∞ ) h_i : S \to(0,\infty) h i : S → ( 0 , ∞ ) and T i : S → R d i T_i : S \to\mathbb{R}^{d_i} T i : S → R d i are measurable,
b i ( θ ) = log ∫ S h i ( x ) exp { θ ⊤ T i ( x ) } d ν i ( x ) , b_i(\theta) = \log\int_S h_i(x)\exp\{\theta^{\top}T_i(x)\}\,\mathrm{d}\nu_i(x), b i ( θ ) = log ∫ S h i ( x ) exp { θ ⊤ T i ( x )} d ν i ( x ) ,
and
Θ ~ i = { θ ∈ R d i : 0 < ∫ S h i ( x ) exp { θ ⊤ T i ( x ) } d ν i ( x ) < ∞ } \widetilde{\Theta}_i = \left\{\theta\in\mathbb{R}^{d_i} : 0 < \int_S h_i(x)\exp\{\theta^{\top}T_i(x)\}\,\mathrm{d}\nu_i(x) < \infty\right\} Θ i = { θ ∈ R d i : 0 < ∫ S h i ( x ) exp { θ ⊤ T i ( x )} d ν i ( x ) < ∞ }
is the natural parameter space. We impose the following conditions:
The family is regular : the natural parameter space Θ ~ i \widetilde{\Theta}_i Θ i is a nonempty open subset of R d i \mathbb{R}^{d_i} R d i .
The parameter set Θ i \Theta_i Θ i is a nonempty compact subset of Θ ~ i \widetilde{\Theta}_i Θ i , equipped with the Euclidean topology.
The family is minimal : there do not exist a nonzero vector v ∈ R d i v \in\mathbb{R}^{d_i} v ∈ R d i and a constant c ∈ R c \in\mathbb{R} c ∈ R such that
v ⊤ T i ( x ) = c , ν i -almost everywhere . v^{\top}T_i(x) = c,\qquad\nu_i\text{-almost everywhere}. v ⊤ T i ( x ) = c , ν i -almost everywhere .
which implies that the parametrization is identifiable, i.e., for all θ , η ∈ Θ i \theta,\eta\in\Theta_i θ , η ∈ Θ i ,
g θ = g η , ν i -almost everywhere ⟺ θ = η . g_\theta= g_\eta,\qquad\nu_i\text{-almost everywhere} \quad\Longleftrightarrow\quad\theta= \eta. g θ = g η , ν i -almost everywhere ⟺ θ = η .
Proposition A.1: Fix i ∈ [ M ] i \in[M] i ∈ [ M ] . Let Θ i \Theta_i Θ i be a compact subset of the open natural parameter space of a regular minimal exponential family
g θ ( x ) = h i ( x ) exp { θ ⊤ T i ( x ) − b i ( θ ) } , θ ∈ Θ i , g_\theta(x) = h_i(x)\exp\{\theta^{\top}T_i(x)-b_i(\theta)\},\qquad\theta\in\Theta_i, g θ ( x ) = h i ( x ) exp { θ ⊤ T i ( x ) − b i ( θ )} , θ ∈ Θ i , and define
E i : = { g θ : θ ∈ Θ i } . (76) \mathcal{E}_i := \{g_\theta: \theta\in\Theta_i\}. \tag*{(76)} E i := { g θ : θ ∈ Θ i } . (76) Equip Θ i \Theta_i Θ i with the Euclidean norm and E i \mathcal{E}_i E i with the L 1 L^1 L 1 norm. Then the mapping
Ψ i : Θ i → E i , Ψ i ( θ ) = g θ , \Psi_i : \Theta_i \to\mathcal{E}_i,\qquad\Psi_i(\theta) = g_\theta, Ψ i : Θ i → E i , Ψ i ( θ ) = g θ , is a homeomorphism, i.e., it is a bijection and for every sequence { θ n } n ≥ 1 ⊂ Θ i \{\theta_n\}_{n\geq1} \subset\Theta_i { θ n } n ≥ 1 ⊂ Θ i and every θ ∈ Θ i \theta\in\Theta_i θ ∈ Θ i ,
θ n → θ ⟺ ∥ g θ n − g θ ∥ 1 → 0. \theta_n \to\theta\quad\Longleftrightarrow\quad\lVert g_{\theta_n} - g_\theta\rVert_1 \to0. θ n → θ ⟺ ∥ g θ n − g θ ∥ 1 → 0. Moreover, E i \mathcal{E}_i E i is compact in L 1 L^1 L 1 , and both Ψ i \Psi_i Ψ i and Ψ i − 1 \Psi_i^{-1} Ψ i − 1 are uniformly continuous.
Proof. In order to show that Ψ i \Psi_i Ψ i is bijective, it suffices to show that it is surjective and injective. Surjectivity follows from the definition of E i \mathcal{E}_i E i . To prove injectivity, suppose that g θ = g η g_\theta= g_\eta g θ = g η , ν i \nu_i ν i -almost everywhere. Since h i > 0 h_i > 0 h i > 0 , ν i \nu_i ν i -almost everywhere, taking logarithms gives
( θ − η ) ⊤ T i ( x ) = b i ( θ ) − b i ( η ) , ν i -almost everywhere . (\theta-\eta)^{\top}T_i(x) = b_i(\theta)-b_i(\eta),\qquad\nu_i\text{-almost everywhere}. ( θ − η ) ⊤ T i ( x ) = b i ( θ ) − b i ( η ) , ν i -almost everywhere .
The minimality condition implies θ = η \theta= \eta θ = η .
To prove that Ψ i \Psi_i Ψ i is continuous, we consider a convergent sequence θ n → θ \theta_n \to\theta θ n → θ in Θ i \Theta_i Θ i . By continuity of the log-partition function b i b_i b i ,
g θ n ( x ) → g θ ( x ) , ν i -almost everywhere . g_{\theta_n}(x) \to g_\theta(x), \qquad\nu_i\text{-almost everywhere}. g θ n ( x ) → g θ ( x ) , ν i -almost everywhere .
Since these functions are probability densities, Scheffé’s lemma [45 ] yields
∥ g θ n − g θ ∥ 1 → 0. \lVert g_{\theta_n} - g_\theta\rVert_1 \to0. ∥ g θ n − g θ ∥ 1 → 0.
Thus, Ψ i \Psi_i Ψ i is continuous.
Since Ψ i \Psi_i Ψ i is a continuous bijection from the compact space Θ i \Theta_i Θ i onto the Hausdorff space E i \mathcal{E}_i E i , by [46 Theorem 26.6] we get that Ψ i − 1 \Psi_i^{-1} Ψ i − 1 is continuous. Alternatively, we can prove that Ψ i − 1 \Psi_i^{-1} Ψ i − 1 is continuous as follows. Suppose that
g η n → g η in L 1 . (77) g_{\eta_n} \to g_\eta\quad\text{in } L^1. \tag*{(77)} g η n → g η in L 1 . (77)
Since Θ i \Theta_i Θ i is compact, in order to prove that η n → η \eta_n \to\eta η n → η it suffices to show that every convergent subsequence of { η n : n ∈ N } \{\eta_n : n \in\mathbb{N}\} { η n : n ∈ N } has limit η \eta η . Let { η n k : k ∈ N } \{\eta_{n_k} : k \in\mathbb{N}\} { η n k : k ∈ N } be any convergent subsequence, with
η n k → ζ ∈ Θ i . \eta_{n_k} \to\zeta\in\Theta_i. η n k → ζ ∈ Θ i .
By continuity of Ψ i \Psi_i Ψ i ,
g η n k → g ζ in L 1 . g_{\eta_{n_k}} \to g_\zeta\quad\text{in } L^1. g η n k → g ζ in L 1 .
On the other hand, by (76),
g η n k → g η in L 1 . g_{\eta_{n_k}} \to g_\eta\quad\text{in } L^1. g η n k → g η in L 1 .
Since limits in L 1 L^1 L 1 are unique,
g η = g ζ , ν i -almost everywhere . g_\eta= g_\zeta, \qquad\nu_i\text{-almost everywhere}. g η = g ζ , ν i -almost everywhere .
Identifiability therefore implies η = ζ \eta= \zeta η = ζ . Consequently, η n → η \eta_n \to\eta η n → η , proving continuity of Ψ i − 1 \Psi_i^{-1} Ψ i − 1 .
Furthermore, E i \mathcal{E}_i E i is compact as the continuous image of Θ i \Theta_i Θ i . Since both spaces are compact metric spaces, the Heine–Cantor theorem [47 Theorem 4.19] implies that Ψ i \Psi_i Ψ i and Ψ i − 1 \Psi_i^{-1} Ψ i − 1 are uniformly continuous.
Proposition A.2: Suppose that Φ i = E i \Phi_i = \mathcal{E}_i Φ i = E i for every i ∈ [ M ] i \in[M] i ∈ [ M ] , where E i \mathcal{E}_i E i is defined in (75). Then regularity conditions (a)–(g) in Subsection VIII-A are satisfied.
Proof: Fix i ∈ [ M ] i \in[M] i ∈ [ M ] , and let f i = g θ i f_i = g_{\theta_i} f i = g θ i . To simplify the notation, we use the change of measure
d μ i ( x ) = h i ( x ) d ν i ( x ) . d\mu_i(x) = h_i(x)\,d\nu_i(x). d μ i ( x ) = h i ( x ) d ν i ( x ) .
(a) Condition (a) follows by Lemma A.1.
(b) Condition (b) follows from h i ( x ) > 0 h_i(x) > 0 h i ( x ) > 0 for all x ∈ S x \in S x ∈ S and from the exponential form of g θ g_\theta g θ .
(c) Let η ∈ Θ i \eta\in\Theta_i η ∈ Θ i . Then
D ( g θ i , g η ) = E g θ i [ log g θ i ( X i ) g η ( X i ) ] = ( θ i − η ) T ∇ b i ( θ i ) − ( b i ( θ i ) − b i ( η ) ) . (78) \begin{aligned}
D(g_{\theta_i},g_\eta)=\mathbb{E}_{g_{\theta_i}}\left[\log\frac{g_{\theta_i}(X_i)}{g_\eta(X_i)}\right] \\
&=(\theta_i-\eta)^{\mathsf{T}}\nabla b_i(\theta_i)-\left(b_i(\theta_i)-b_i(\eta)\right). \tag*{(78)}
\end{aligned} D ( g θ i , g η ) = E g θ i [ log g η ( X i ) g θ i ( X i ) ] = ( θ i − η ) T ∇ b i ( θ i ) − ( b i ( θ i ) − b i ( η ) ) . (78)
Since the exponential family is regular and minimal, the log-partition function b i b_i b i is strictly convex. Therefore,
D ( g θ i , g η ) > 0 , ∀ η ∈ Θ i , η ≠ θ i . D(g_{\theta_i},g_\eta)>0,\qquad\forall\eta\in\Theta_i,\ \eta\ne\theta_i. D ( g θ i , g η ) > 0 , ∀ η ∈ Θ i , η = θ i .
This proves condition (c).
(d) By Lemma A.1, we have
∥ g θ n − g θ ∥ 1 → 0 ⇒ θ n → θ . \lVert g_{\theta_n}-g_\theta\rVert_1\to0\quad\Rightarrow\quad\theta_n\to\theta. ∥ g θ n − g θ ∥ 1 → 0 ⇒ θ n → θ .
Then the continuity of the mapping
g η ↦ D ( g θ i , g η ) g_\eta\mapsto D(g_{\theta_i},g_\eta) g η ↦ D ( g θ i , g η )
follows from the continuity of b i b_i b i on Θ i \Theta_i Θ i and (77).
(e) Fix q > 2 q>2 q > 2 and ϵ > 0 \epsilon>0 ϵ > 0 . Fix η ∈ Θ i \eta\in\Theta_i η ∈ Θ i . We have
log g η ( X i ) = η T T i ( X i ) − b i ( η ) , \log g_\eta(X_i)=\eta^{\mathsf{T}}T_i(X_i)-b_i(\eta), log g η ( X i ) = η T T i ( X i ) − b i ( η ) ,
and, by convexity,
∣ log g η ( X i ) ∣ q ≤ 2 q − 1 ( ∣ η T T i ( X i ) ∣ q + ∣ b i ( η ) ∣ q ) \left|\log g_\eta(X_i)\right|^q\le2^{q-1}\left(\left|\eta^{\mathsf{T}}T_i(X_i)\right|^q+\left|b_i(\eta)\right|^q\right) ∣ log g η ( X i ) ∣ q ≤ 2 q − 1 ( η T T i ( X i ) q + ∣ b i ( η ) ∣ q )
≤ 2 q − 1 ( ∥ η ∥ q ∥ T i ( X i ) ∥ q + ∣ b i ( η ) ∣ q ) , \le2^{q-1}\left(\lVert\eta\rVert^q\lVert T_i(X_i)\rVert^q+\left|b_i(\eta)\right|^q\right), ≤ 2 q − 1 ( ∥ η ∥ q ∥ T i ( X i ) ∥ q + ∣ b i ( η ) ∣ q ) ,
where the last inequality follows from the Cauchy–Schwarz inequality. Since Θ i \Theta_i Θ i is compact and b i b_i b i is continuous on Θ i \Theta_i Θ i , there exist constants c 1 , c 2 > 0 c_1,c_2>0 c 1 , c 2 > 0 such that
sup η ∈ Θ i ∣ log g η ( X i ) ∣ q ≤ c 1 + c 2 ∑ k = 1 d i ∣ T i , k ( X i ) ∣ q . \sup_{\eta\in\Theta_i}\left|\log g_\eta(X_i)\right|^q\le c_1+c_2\sum_{k=1}^{d_i}\left|T_{i,k}(X_i)\right|^q. η ∈ Θ i sup ∣ log g η ( X i ) ∣ q ≤ c 1 + c 2 k = 1 ∑ d i ∣ T i , k ( X i ) ∣ q .
Consequently, since V ϵ ( g η ) ⊂ Φ i V_\epsilon(g_\eta)\subset\Phi_i V ϵ ( g η ) ⊂ Φ i ,
sup h ∈ V ϵ ( g η ) ∣ log h ( X i ) ∣ q ≤ c 1 + c 2 ∑ k = 1 d i ∣ T i , k ( X i ) ∣ q . (79) \sup_{h\in V_\epsilon(g_\eta)}\left|\log h(X_i)\right|^q\le c_1+c_2\sum_{k=1}^{d_i}\left|T_{i,k}(X_i)\right|^q. \tag*{(79)} h ∈ V ϵ ( g η ) sup ∣ log h ( X i ) ∣ q ≤ c 1 + c 2 k = 1 ∑ d i ∣ T i , k ( X i ) ∣ q . (79)
It therefore suffices to show that all moments of T i ( X i ) T_i(X_i) T i ( X i ) are uniformly bounded over Θ i \Theta_i Θ i . Since Θ i \Theta_i Θ i is a compact subset of the open set Θ ~ i \widetilde{\Theta}_i Θ i , there exists δ > 0 \delta>0 δ > 0 such that
θ i + u ∈ Θ ~ i , \theta_i+u\in\widetilde{\Theta}_i, θ i + u ∈ Θ i ,
for all θ i ∈ Θ i \theta_i\in\Theta_i θ i ∈ Θ i and all u ∈ R d i u\in\mathbb{R}^{d_i} u ∈ R d i with ∥ u ∥ ∞ < δ \lVert u\rVert_\infty<\delta ∥ u ∥ ∞ < δ . Fix δ ′ < δ \delta'<\delta δ ′ < δ . Since
exp { δ ′ ∑ k = 1 d i ∣ T i , k ( X i ) ∣ } ≤ ∑ σ ∈ { − 1 , 1 } d i exp { δ ′ σ T T i ( X i ) } , \exp\left\{\delta'\sum_{k=1}^{d_i}\left|T_{i,k}(X_i)\right|\right\}
\le\sum_{\sigma\in\{-1,1\}^{d_i}}\exp\left\{\delta'\sigma^{\mathsf{T}}T_i(X_i)\right\}, exp { δ ′ k = 1 ∑ d i ∣ T i , k ( X i ) ∣ } ≤ σ ∈ { − 1 , 1 } d i ∑ exp { δ ′ σ T T i ( X i ) } ,
we have
E g θ i [ exp { δ ′ ∑ k = 1 d i ∣ T i , k ( X i ) ∣ } ] ≤ ∑ σ ∈ { − 1 , 1 } d i E g θ i [ exp { δ ′ σ T T i ( X i ) } ] . \mathbb{E}_{g_{\theta_i}}\left[\exp\left\{\delta' \sum_{k=1}^{d_i} \lvert T_{i,k}(X_i)\rvert\right\}\right] \le\sum_{\sigma\in\{-1,1\}^{d_i}} \mathbb{E}_{g_{\theta_i}}\left[\exp\left\{\delta'\sigma^\mathsf{T}T_i(X_i)\right\}\right]. E g θ i [ exp { δ ′ k = 1 ∑ d i ∣ T i , k ( X i )∣ } ] ≤ σ ∈ { − 1 , 1 } d i ∑ E g θ i [ exp { δ ′ σ T T i ( X i ) } ] .
Fix σ ∈ { − 1 , 1 } d i \sigma\in\{-1,1\}^{d_i} σ ∈ { − 1 , 1 } d i . Since ∥ δ ′ σ ∥ ∞ < δ \lVert\delta'\sigma\rVert_\infty<\delta ∥ δ ′ σ ∥ ∞ < δ , it follows that θ i + δ ′ σ ∈ Θ ~ i \theta_i+\delta'\sigma\in\widetilde{\Theta}_i θ i + δ ′ σ ∈ Θ i . Therefore,
E g θ i [ exp { δ ′ σ T T i ( X i ) } ] = ∫ exp { δ ′ σ T T i ( x ) } e θ i T T i ( x ) − b i ( θ i ) d μ i ( x ) = e − b i ( θ i ) ∫ exp { ( θ i + δ ′ σ ) T T i ( x ) } d μ i ( x ) = exp { b i ( θ i + δ ′ σ ) − b i ( θ i ) } < ∞ . \begin{aligned}
\mathbb{E}_{g_{\theta_i}}\left[\exp\left\{\delta'\sigma^\mathsf{T}T_i(X_i)\right\}\right]
&=\int\exp\left\{\delta'\sigma^\mathsf{T}T_i(x)\right\}e^{\theta_i^\mathsf{T}T_i(x)-b_i(\theta_i)}\,d\mu_i(x)\\
&=e^{-b_i(\theta_i)}\int\exp\left\{(\theta_i+\delta'\sigma)^\mathsf{T}T_i(x)\right\}\,d\mu_i(x)\\
&=\exp\left\{b_i(\theta_i+\delta'\sigma)-b_i(\theta_i)\right\}<\infty.
\end{aligned} E g θ i [ exp { δ ′ σ T T i ( X i ) } ] = ∫ exp { δ ′ σ T T i ( x ) } e θ i T T i ( x ) − b i ( θ i ) d μ i ( x ) = e − b i ( θ i ) ∫ exp { ( θ i + δ ′ σ ) T T i ( x ) } d μ i ( x ) = exp { b i ( θ i + δ ′ σ ) − b i ( θ i ) } < ∞.
Since b i ( θ i + δ ′ σ ) − b i ( θ i ) b_i(\theta_i+\delta'\sigma)-b_i(\theta_i) b i ( θ i + δ ′ σ ) − b i ( θ i ) is continuous with respect to θ i \theta_i θ i on the compact set Θ i \Theta_i Θ i , we conclude that
E g θ i [ exp { δ ′ σ T T i ( X i ) } ] ≤ sup ζ ∈ Θ i exp { b i ( ζ + δ ′ σ ) − b i ( ζ ) } < ∞ . (80) \mathbb{E}_{g_{\theta_i}}\left[\exp\left\{\delta'\sigma^\mathsf{T}T_i(X_i)\right\}\right]\le\sup_{\zeta\in\Theta_i}\exp\left\{b_i(\zeta+\delta'\sigma)-b_i(\zeta)\right\}<\infty. \tag*{(80)} E g θ i [ exp { δ ′ σ T T i ( X i ) } ] ≤ ζ ∈ Θ i sup exp { b i ( ζ + δ ′ σ ) − b i ( ζ ) } < ∞. (80)
which proves the claim.
(f) We prove condition (f) by constructing an integrable upper envelope for the entire family. Since Θ i \Theta_i Θ i is compact and Θ ~ i \widetilde{\Theta}_i Θ i is open, each point of Θ i \Theta_i Θ i lies in the interior of a closed cube contained in Θ ~ i \widetilde{\Theta}_i Θ i . The interiors of finitely many such cubes cover Θ i \Theta_i Θ i . Let V i \mathcal{V}_i V i denote the set of all their vertices. These vertices need not belong to Θ i \Theta_i Θ i , but they belong to Θ ~ i \widetilde{\Theta}_i Θ i , so each v ∈ V i v\in\mathcal{V}_i v ∈ V i defines a density g v g_v g v .
Fix θ ∈ Θ i \theta\in\Theta_i θ ∈ Θ i and x ∈ S x\in S x ∈ S , and choose a cube containing θ \theta θ . Over this cube, the linear function u ↦ u T T i ( x ) u\mapsto u^\mathsf{T}T_i(x) u ↦ u T T i ( x ) is maximized at a vertex v v v : in coordinate k k k , choose the upper endpoint if T i , k ( x ) ≥ 0 T_{i,k}(x)\ge0 T i , k ( x ) ≥ 0 and the lower endpoint otherwise. Thus,
θ T T i ( x ) ≤ v T T i ( x ) . \theta^\mathsf{T}T_i(x)\le v^\mathsf{T}T_i(x). θ T T i ( x ) ≤ v T T i ( x ) .
Also, by continuity of b i b_i b i and compactness of Θ i \Theta_i Θ i , the minimum m i : = min θ ∈ Θ i b i ( θ ) m_i:=\min_{\theta\in\Theta_i}b_i(\theta) m i := min θ ∈ Θ i b i ( θ ) is finite. Combining these two bounds gives
g θ ( x ) = h i ( x ) e θ T T i ( x ) − b i ( θ ) ≤ h i ( x ) e v T T i ( x ) − m i = e b i ( v ) − m i g v ( x ) . g_\theta(x)=h_i(x)e^{\theta^\mathsf{T}T_i(x)-b_i(\theta)}\le h_i(x)e^{v^\mathsf{T}T_i(x)-m_i}=e^{b_i(v)-m_i}g_v(x). g θ ( x ) = h i ( x ) e θ T T i ( x ) − b i ( θ ) ≤ h i ( x ) e v T T i ( x ) − m i = e b i ( v ) − m i g v ( x ) .
The selected vertex may depend on θ \theta θ and x x x , but it always belongs to the same finite set V i \mathcal{V}_i V i . Hence, the sum over all vertices bounds g θ ( x ) g_\theta(x) g θ ( x ) for every θ ∈ Θ i \theta\in\Theta_i θ ∈ Θ i . Moreover, since V ϵ ( g η ) ⊆ Φ i V_\epsilon(g_\eta)\subseteq\Phi_i V ϵ ( g η ) ⊆ Φ i , we obtain
p ϵ + ( g η , x ) ≤ sup θ ∈ Θ i g θ ( x ) ≤ ∑ v ∈ V i e b i ( v ) − m i g v ( x ) . p_\epsilon^+(g_\eta,x)\le\sup_{\theta\in\Theta_i}g_\theta(x)\le\sum_{v\in\mathcal{V}_i}e^{b_i(v)-m_i}g_v(x). p ϵ + ( g η , x ) ≤ θ ∈ Θ i sup g θ ( x ) ≤ v ∈ V i ∑ e b i ( v ) − m i g v ( x ) .
Finally, each g v g_v g v integrates to one. Therefore,
E f i [ p ϵ + ( g η , X i ) f i ( X i ) ] = ∫ S p ϵ + ( g η , x ) d ν i ( x ) ≤ ∑ v ∈ V i e b i ( v ) − m i < ∞ . \mathbb{E}_{f_i}\left[\frac{p_\epsilon^+(g_\eta,X_i)}{f_i(X_i)}\right]=\int_S p_\epsilon^+(g_\eta,x)\,d\nu_i(x)\le\sum_{v\in\mathcal{V}_i}e^{b_i(v)-m_i}<\infty. E f i [ f i ( X i ) p ϵ + ( g η , X i ) ] = ∫ S p ϵ + ( g η , x ) d ν i ( x ) ≤ v ∈ V i ∑ e b i ( v ) − m i < ∞.
This bound is independent of η \eta η , f i f_i f i , and ϵ \epsilon ϵ .
(g) For η ∈ Θ i \eta\in\Theta_i η ∈ Θ i , and g ζ ∈ V ϵ ( g η ) g_\zeta\in V_\epsilon(g_\eta) g ζ ∈ V ϵ ( g η ) ,
∣ log g η ( X i ) − log g ζ ( X i ) ∣ = ∣ ( η − ζ ) ⊤ T i ( X i ) − b i ( η ) + b i ( ζ ) ∣ ≤ ∥ η − ζ ∥ 2 ∥ T i ( X i ) ∥ 2 + ∣ b i ( η ) − b i ( ζ ) ∣ . \begin{aligned}
\left|\log g_\eta(X_i)-\log g_\zeta(X_i)\right|=\left|(\eta-\zeta)^\top T_i(X_i)-b_i(\eta)+b_i(\zeta)\right| \\
&\leq\|\eta-\zeta\|_2\|T_i(X_i)\|_2+\left|b_i(\eta)-b_i(\zeta)\right|.
\end{aligned} ∣ log g η ( X i ) − log g ζ ( X i ) ∣ = ( η − ζ ) ⊤ T i ( X i ) − b i ( η ) + b i ( ζ ) ≤ ∥ η − ζ ∥ 2 ∥ T i ( X i ) ∥ 2 + ∣ b i ( η ) − b i ( ζ ) ∣ .
Since b i b_i b i is continuously differentiable on the open natural parameter space and Θ i \Theta_i Θ i is compact, b i b_i b i is Lipschitz on Θ i \Theta_i Θ i . Hence, there exists a constant C > 0 C>0 C > 0 such that
∣ log g η ( X i ) − log g ζ ( X i ) ∣ ≤ ∥ η − ζ ∥ 2 ( ∥ T i ( X i ) ∥ 2 + C ) . \left|\log g_\eta(X_i)-\log g_\zeta(X_i)\right|\leq\|\eta-\zeta\|_2\left(\|T_i(X_i)\|_2+C\right). ∣ log g η ( X i ) − log g ζ ( X i ) ∣ ≤ ∥ η − ζ ∥ 2 ( ∥ T i ( X i ) ∥ 2 + C ) .
Since ∥ η − ζ ∥ 2 → 0 \|\eta-\zeta\|_2\to0 ∥ η − ζ ∥ 2 → 0 as ϵ → 0 \epsilon\to0 ϵ → 0 , and all moments of T i , k ( X i ) T_{i,k}(X_i) T i , k ( X i ) are uniformly bounded over Θ i \Theta_i Θ i by (79), the dominated convergence theorem yields H g θ i ( g , ϵ ) → 0 H_{g_{\theta_i}}(g,\epsilon)\to0 H g θ i ( g , ϵ ) → 0 , as ϵ → 0 \epsilon\to0 ϵ → 0 , uniformly over all θ i ∈ Θ i \theta_i\in\Theta_i θ i ∈ Θ i .
We proceed to the proof of Proposition 8.1.
Proof of Proposition 8.1: Fix i ∈ [ M ] i\in[M] i ∈ [ M ] . By Proposition A.2, conditions (a)--(g) hold for E i \mathcal{E}_i E i . We show that they also hold for the mixture class
Φ i : = { q i , P : P ∈ P ( Θ i ) } . \Phi_i:=\{q_{i,P}:P\in\mathcal{P}(\Theta_i)\}. Φ i := { q i , P : P ∈ P ( Θ i )} .
The proof for condition (d) is provided last.
(a) To prove condition (a) it suffices to show that P ( Θ i ) \mathcal{P}(\Theta_i) P ( Θ i ) equipped with its weak topology is compact, and that the map P ↦ q i , P P\mapsto q_{i,P} P ↦ q i , P is continuous from P ( Θ i ) \mathcal{P}(\Theta_i) P ( Θ i ) into L 1 L^1 L 1 . Then, Φ i \Phi_i Φ i is compact in L 1 L^1 L 1 , as the continuous image of a compact space.
Since Θ i \Theta_i Θ i is compact and P ( Θ i ) = 1 P(\Theta_i)=1 P ( Θ i ) = 1 for every P ∈ P ( Θ i ) P\in\mathcal{P}(\Theta_i) P ∈ P ( Θ i ) , the family P ( Θ i ) \mathcal{P}(\Theta_i) P ( Θ i ) is uniformly tight. By Prokhorov’s theorem [48 Theorem 5.1] , every sequence in P ( Θ i ) \mathcal{P}(\Theta_i) P ( Θ i ) has a weakly convergent subsequence with limit in P ( Θ i ) \mathcal{P}(\Theta_i) P ( Θ i ) . Since the weak topology is metrizable, this establishes compactness of P ( Θ i ) \mathcal{P}(\Theta_i) P ( Θ i ) .
To show the continuity of the map P ↦ q i , P P\mapsto q_{i,P} P ↦ q i , P , it suffices to show that for every sequence P n → w P P_n\xrightarrow{w}P P n w P , where → w \xrightarrow{w} w stands for weak convergence, it holds
q i , P n → q i , P , in L 1 . (81) q_{i,P_n}\to q_{i,P},\quad\text{in }L^1. \tag*{(81)} q i , P n → q i , P , in L 1 . (81)
We note that for each fixed x ∈ S x\in\mathcal{S} x ∈ S , the exponential-family representation
g θ ( x ) = h i ( x ) exp { θ ⊤ T i ( x ) − b i ( θ ) } g_\theta(x)=h_i(x)\exp\{\theta^\top T_i(x)-b_i(\theta)\} g θ ( x ) = h i ( x ) exp { θ ⊤ T i ( x ) − b i ( θ )}
and the continuity of b i b_i b i imply that θ ↦ g θ ( x ) \theta\mapsto g_\theta(x) θ ↦ g θ ( x ) is continuous. Since Θ i \Theta_i Θ i is compact, this map is also bounded on Θ i \Theta_i Θ i , with a bound that may depend on x x x . By definition of weak convergence, the P n → w P P_n\xrightarrow{w}P P n w P implies
q i , P n ( x ) = ∫ Θ i g θ ( x ) P n ( d θ ) → ∫ Θ i g θ ( x ) P ( d θ ) = q i , P ( x ) . (82) q_{i,P_n}(x)=\int_{\Theta_i}g_\theta(x)P_n(\mathrm{d}\theta)\to\int_{\Theta_i}g_\theta(x)P(\mathrm{d}\theta)=q_{i,P}(x). \tag*{(82)} q i , P n ( x ) = ∫ Θ i g θ ( x ) P n ( d θ ) → ∫ Θ i g θ ( x ) P ( d θ ) = q i , P ( x ) . (82)
Since q i , P n q_{i,P_n} q i , P n and q i , P q_{i,P} q i , P are probability densities, the pointwise convergence (81) implies the L 1 L^1 L 1 convergence (80) by Scheffé’s lemma [45 ] .
(b) Each g θ ( x ) g_\theta(x) g θ ( x ) is strictly positive for every x ∈ S x \in\mathcal{S} x ∈ S . Therefore,
q i , P ( x ) = ∫ Θ i g θ ( x ) P ( d θ ) > 0 q_{i,P}(x)=\int_{\Theta_i} g_\theta(x)\,P(\mathrm{d}\theta)>0 q i , P ( x ) = ∫ Θ i g θ ( x ) P ( d θ ) > 0
for every P ∈ P ( Θ i ) P \in\mathcal{P}(\Theta_i) P ∈ P ( Θ i ) and x ∈ S x \in\mathcal{S} x ∈ S .
(c) If f i f_i f i and g g g are distinct as elements of Φ i \Phi_i Φ i with L 1 L^1 L 1 , then D ( f i , g ) > 0 D(f_i,g)>0 D ( f i , g ) > 0 , by Pinsker’s inequality.
(e) In order to prove the claim it suffices to show that for the envelope
L i ( x ) : = sup g ∈ E i ∣ log g ( x ) ∣ , L_i(x):=\sup_{g\in\mathcal{E}_i}|\log g(x)|, L i ( x ) := g ∈ E i sup ∣ log g ( x ) ∣ ,
it holds
sup f i ∈ E i E f i [ L i ( X i ) q ] < ∞ . (83) \sup_{f_i\in\mathcal{E}_i}\mathbb{E}_{f_i}[L_i(X_i)^q]<\infty. \tag*{(83)} f i ∈ E i sup E f i [ L i ( X i ) q ] < ∞. (83)
Indeed, for every P ∈ P ( Θ i ) P \in\mathcal{P}(\Theta_i) P ∈ P ( Θ i ) , Tonelli’s theorem gives
E q i , P [ L i ( X i ) q ] = ∫ Θ i E g θ [ L i ( X i ) q ] P ( d θ ) ≤ sup f i ∈ E i E f i [ L i ( X i ) q ] . \mathbb{E}_{q_{i,P}}[L_i(X_i)^q]
=\int_{\Theta_i}\mathbb{E}_{g_\theta}[L_i(X_i)^q]\,P(\mathrm{d}\theta)
\leq\sup_{f_i\in\mathcal{E}_i}\mathbb{E}_{f_i}[L_i(X_i)^q]. E q i , P [ L i ( X i ) q ] = ∫ Θ i E g θ [ L i ( X i ) q ] P ( d θ ) ≤ f i ∈ E i sup E f i [ L i ( X i ) q ] .
Also, for every mixture q i , P ( x ) q_{i,P}(x) q i , P ( x ) , it holds ∣ log q i , P ( x ) ∣ ≤ L i ( x ) |\log q_{i,P}(x)|\leq L_i(x) ∣ log q i , P ( x ) ∣ ≤ L i ( x ) , and thus for every ϵ > 0 \epsilon>0 ϵ > 0 ,
sup f i , g ∈ Φ i E f i [ sup p ∈ V ϵ ( g ) ∣ log p ( X i ) ∣ q ] ≤ sup f i ∈ Φ i E f i [ L i ( X i ) q ] ≤ sup f i ∈ E i E f i [ L i ( X i ) q ] < ∞ , (84) \begin{aligned}
\sup_{f_i,g\in\Phi_i}\mathbb{E}_{f_i}\left[\sup_{p\in V_\epsilon(g)}|\log p(X_i)|^q\right]
\leq\sup_{f_i\in\Phi_i}\mathbb{E}_{f_i}[L_i(X_i)^q] \\
&\leq\sup_{f_i\in\mathcal{E}_i}\mathbb{E}_{f_i}[L_i(X_i)^q]<\infty, \tag*{(84)}
\end{aligned} f i , g ∈ Φ i sup E f i [ p ∈ V ϵ ( g ) sup ∣ log p ( X i ) ∣ q ] ≤ f i ∈ Φ i sup E f i [ L i ( X i ) q ] ≤ f i ∈ E i sup E f i [ L i ( X i ) q ] < ∞ , (84)
which shows (e) for Φ i \Phi_i Φ i . To prove (82), we apply condition (e) to E i \mathcal{E}_i E i , and we cover E i \mathcal{E}_i E i by finitely many neighborhoods of the radius specified in that condition. Such a finite cover exists by compactness of E i \mathcal{E}_i E i . Then L i ( x ) L_i(x) L i ( x ) is bounded by the sum of the corresponding local upper envelopes, each of which is integrable by Proposition A.2.
(f) In order to prove the claim it suffices to show that for the envelope
U i ( x ) : = sup g ∈ E i g ( x ) , U_i(x):=\sup_{g\in\mathcal{E}_i}g(x), U i ( x ) := g ∈ E i sup g ( x ) ,
it holds
∫ S U i ( x ) d ν i ( x ) < ∞ . (85) \int_{\mathcal{S}}U_i(x)\,\mathrm{d}\nu_i(x)<\infty. \tag*{(85)} ∫ S U i ( x ) d ν i ( x ) < ∞. (85)
Indeed, for every mixture q i , P ( x ) ≤ U i ( x ) q_{i,P}(x)\leq U_i(x) q i , P ( x ) ≤ U i ( x ) , and thus for all f i , g ∈ Φ i f_i,g\in\Phi_i f i , g ∈ Φ i and every ϵ > 0 \epsilon>0 ϵ > 0 ,
E f i [ p ϵ + ( g , X i ) f i ( X i ) ] = ∫ S p ϵ + ( g , x ) d ν i ( x ) ≤ ∫ S U i ( x ) d ν i ( x ) < ∞ , \mathbb{E}_{f_i}\left[\frac{p_\epsilon^+(g,X_i)}{f_i(X_i)}\right]
=\int_{\mathcal{S}}p_\epsilon^+(g,x)\,\mathrm{d}\nu_i(x)
\leq\int_{\mathcal{S}}U_i(x)\,\mathrm{d}\nu_i(x)<\infty, E f i [ f i ( X i ) p ϵ + ( g , X i ) ] = ∫ S p ϵ + ( g , x ) d ν i ( x ) ≤ ∫ S U i ( x ) d ν i ( x ) < ∞ ,
where the equality uses the strict positivity of f i f_i f i . The upper bound is independent of both f i f_i f i and g g g . To show (85) , we apply condition (f) to E i \mathcal{E}_i E i , and we cover E i \mathcal{E}_i E i by finitely many neighborhoods of the radius specified in that condition. Such a finite cover exists by compactness of E i \mathcal{E}_i E i . Then U i ( x ) U_i(x) U i ( x ) is bounded by the sum of the corresponding local upper envelopes, each of which is integrable by Proposition A.2.
(g) Fix ϵ > 0 \epsilon> 0 ϵ > 0 , and define
W i , ϵ ( x ) : = sup p , q ∈ Φ i ∥ p − q ∥ 1 < ϵ ∣ log p ( x ) − log q ( x ) ∣ . (86) W_{i,\epsilon}(x) := \sup_{\substack{p,q\in\Phi_i\\ \lVert p-q\rVert_1<\epsilon}} \lvert\log p(x)-\log q(x)\rvert. \tag*{(86)} W i , ϵ ( x ) := p , q ∈ Φ i ∥ p − q ∥ 1 < ϵ sup ∣ log p ( x ) − log q ( x )∣ . (86)
We observe that for each fixed f i ∈ Φ i f_i\in\Phi_i f i ∈ Φ i ,
sup g ∈ Φ i E f i [ sup p ∈ V ϵ ( g ) ∣ log p ( X i ) − log g ( X i ) ∣ ] ≤ E f i [ W i , ϵ ( X i ) ] . \sup_{g\in\Phi_i}\mathbb{E}_{f_i}\left[\sup_{p\in V_\epsilon(g)}\lvert\log p(X_i)-\log g(X_i)\rvert\right]\leq\mathbb{E}_{f_i}[W_{i,\epsilon}(X_i)]. g ∈ Φ i sup E f i [ p ∈ V ϵ ( g ) sup ∣ log p ( X i ) − log g ( X i )∣ ] ≤ E f i [ W i , ϵ ( X i )] .
Thus, in order to prove the claim it suffices to show that
W i , ϵ ( x ) → 0 as ϵ ↓ 0 , (87) W_{i,\epsilon}(x)\to0\quad\text{as }\epsilon\downarrow0, \tag*{(87)} W i , ϵ ( x ) → 0 as ϵ ↓ 0 , (87)
for ν i \nu_i ν i -almost every x x x . Then by dominated convergence theorem, we deduce that
E f i [ W i , ϵ ( X i ) ] → 0 as ϵ ↓ 0 , (88) \mathbb{E}_{f_i}[W_{i,\epsilon}(X_i)]\to0\quad\text{as }\epsilon\downarrow0, \tag*{(88)} E f i [ W i , ϵ ( X i )] → 0 as ϵ ↓ 0 , (88)
because for every mixture q i , P ( x ) q_{i,P}(x) q i , P ( x ) , it holds ∣ log q i , P ( x ) ∣ ≤ L i ( x ) \lvert\log q_{i,P}(x)\rvert\leq L_i(x) ∣ log q i , P ( x )∣ ≤ L i ( x ) , and thus
W i , ϵ ( x ) ≤ 2 L i ( x ) , ν i almost everywhere , W_{i,\epsilon}(x)\leq2L_i(x),\quad\nu_i\text{ almost everywhere}, W i , ϵ ( x ) ≤ 2 L i ( x ) , ν i almost everywhere ,
where sup f i ∈ Φ i E f i [ L i ( X i ) ] < ∞ \sup_{f_i\in\Phi_i}\mathbb{E}_{f_i}[L_i(X_i)]<\infty sup f i ∈ Φ i E f i [ L i ( X i )] < ∞ by (84) .
To this end, for each P ∈ P ( Θ i ) P\in\mathcal{P}(\Theta_i) P ∈ P ( Θ i ) we write
q i , P ( x ) = h i ( x ) F i , P ( T i ( x ) ) , q_{i,P}(x)=h_i(x)F_{i,P}(T_i(x)), q i , P ( x ) = h i ( x ) F i , P ( T i ( x )) ,
where
F i , P ( t ) : = ∫ Θ i exp { θ ⊤ t − b i ( θ ) } P ( d θ ) , t ∈ R d i . F_{i,P}(t):=\int_{\Theta_i}\exp\{\theta^\top t-b_i(\theta)\}\,P(\mathrm{d}\theta),\quad t\in\mathbb{R}^{d_i}. F i , P ( t ) := ∫ Θ i exp { θ ⊤ t − b i ( θ )} P ( d θ ) , t ∈ R d i .
The function F i , P F_{i,P} F i , P is strictly positive because its integrand is strictly positive. It is also continuous in t t t by dominated convergence, since the integrand is continuous and bounded on Θ i × K \Theta_i\times K Θ i × K for every compact set K ⊆ R d i K\subseteq\mathbb{R}^{d_i} K ⊆ R d i .
Fix θ i , 0 ∈ Θ i \theta_{i,0}\in\Theta_i θ i , 0 ∈ Θ i , and let S T i S_{T_i} S T i denote the support of the distribution of T i ( X i ) T_i(X_i) T i ( X i ) under g θ i , 0 g_{\theta_{i,0}} g θ i , 0 . If q i , P = q i , Q q_{i,P}=q_{i,Q} q i , P = q i , Q ν i \nu_i ν i -almost everywhere, then, since h i h_i h i is strictly positive,
F i , P ( T i ( X i ) ) = F i , Q ( T i ( X i ) ) F_{i,P}(T_i(X_i))=F_{i,Q}(T_i(X_i)) F i , P ( T i ( X i )) = F i , Q ( T i ( X i ))
almost surely under g θ i , 0 g_{\theta_{i,0}} g θ i , 0 . It follows that F i , P ( t ) = F i , Q ( t ) F_{i,P}(t)=F_{i,Q}(t) F i , P ( t ) = F i , Q ( t ) for every t ∈ S T i t\in S_{T_i} t ∈ S T i . Indeed, if the two functions differed at some t ∈ S T i t\in S_{T_i} t ∈ S T i , their continuity would imply that they differ throughout an open neighborhood of t t t . By the definition of support, this neighborhood has positive probability under the law of T i ( X i ) T_i(X_i) T i ( X i ) , contradicting the almost sure equality.
In order to prove (86), it suffices to show that for each t ∈ S T i t\in S_{T_i} t ∈ S T i ,
lim ϵ ↓ 0 sup P , Q ∈ P ( Θ i ) ∥ q i , P − q i , Q ∥ 1 < ϵ ∣ log F i , P ( t ) − log F i , Q ( t ) ∣ = 0. (89) \lim_{\epsilon\mathbin{\downarrow}0}\sup_{\substack{P,Q\in\mathcal{P}(\Theta_i)\\ \lVert q_{i,P}-q_{i,Q}\rVert_1<\epsilon}}\left|\log F_{i,P}(t)-\log F_{i,Q}(t)\right|=0. \tag*{(89)} ϵ ↓ 0 lim P , Q ∈ P ( Θ i ) ∥ q i , P − q i , Q ∥ 1 < ϵ sup ∣ log F i , P ( t ) − log F i , Q ( t ) ∣ = 0. (89)
We proceed by contradiction. Fix t ∈ S T i t\in S_{T_i} t ∈ S T i and suppose that there exist η > 0 \eta>0 η > 0 and sequences P n , Q n ∈ P ( Θ i ) P_n,Q_n\in\mathcal{P}(\Theta_i) P n , Q n ∈ P ( Θ i ) such that
∥ q i , P n − q i , Q n ∥ 1 → 0 , ∣ log F i , P n ( t ) − log F i , Q n ( t ) ∣ ≥ η for every n . \lVert q_{i,P_n}-q_{i,Q_n}\rVert_1\to0,\qquad\left|\log F_{i,P_n}(t)-\log F_{i,Q_n}(t)\right|\geq\eta\quad\text{for every }n. ∥ q i , P n − q i , Q n ∥ 1 → 0 , ∣ log F i , P n ( t ) − log F i , Q n ( t ) ∣ ≥ η for every n .
By compactness of P ( Θ i ) \mathcal{P}(\Theta_i) P ( Θ i ) in the weak topology, there exists a subsequence { P n k } \{P_{n_k}\} { P n k } converging weakly to some P ∈ P ( Θ i ) P\in\mathcal{P}(\Theta_i) P ∈ P ( Θ i ) . Applying compactness again to { Q n k } \{Q_{n_k}\} { Q n k } , we obtain a further subsequence { Q n k ℓ } \{Q_{n_{k_\ell}}\} { Q n k ℓ } converging weakly to some Q ∈ P ( Θ i ) Q\in\mathcal{P}(\Theta_i) Q ∈ P ( Θ i ) . Along these same indices, P n k ℓ P_{n_{k_\ell}} P n k ℓ still converges weakly to P P P . Relabeling this common subsequence by m m m , we thus have P m → w P P_m\xrightarrow{w}P P m w P and Q m → w Q Q_m\xrightarrow{w}Q Q m w Q .
In part (a), we proved the continuity of the map P ↦ q i , P P\mapsto q_{i,P} P ↦ q i , P from P ( Θ i ) \mathcal{P}(\Theta_i) P ( Θ i ) equipped with its weak topology into L 1 L^1 L 1 . Thus,
∥ q i , P m − q i , P ∥ 1 → 0 , ∥ q i , Q m − q i , Q ∥ 1 → 0. \lVert q_{i,P_m}-q_{i,P}\rVert_1\to0,\qquad\lVert q_{i,Q_m}-q_{i,Q}\rVert_1\to0. ∥ q i , P m − q i , P ∥ 1 → 0 , ∥ q i , Q m − q i , Q ∥ 1 → 0.
Since ∥ q i , P m − q i , Q m ∥ 1 → 0 \lVert q_{i,P_m}-q_{i,Q_m}\rVert_1\to0 ∥ q i , P m − q i , Q m ∥ 1 → 0 by construction, the triangle inequality gives
∥ q i , P − q i , Q ∥ 1 ≤ ∥ q i , P − q i , P m ∥ 1 + ∥ q i , P m − q i , Q m ∥ 1 + ∥ q i , Q m − q i , Q ∥ 1 → 0. \lVert q_{i,P}-q_{i,Q}\rVert_1\leq\lVert q_{i,P}-q_{i,P_m}\rVert_1+\lVert q_{i,P_m}-q_{i,Q_m}\rVert_1+\lVert q_{i,Q_m}-q_{i,Q}\rVert_1\to0. ∥ q i , P − q i , Q ∥ 1 ≤ ∥ q i , P − q i , P m ∥ 1 + ∥ q i , P m − q i , Q m ∥ 1 + ∥ q i , Q m − q i , Q ∥ 1 → 0.
Hence, q i , P = q i , Q q_{i,P}=q_{i,Q} q i , P = q i , Q ν i \nu_i ν i -almost everywhere. Since t ∈ S T i t\in S_{T_i} t ∈ S T i , the support argument above yields
F i , P ( t ) = F i , Q ( t ) > 0. F_{i,P}(t)=F_{i,Q}(t)>0. F i , P ( t ) = F i , Q ( t ) > 0.
For this fixed t t t , the map θ ↦ exp { θ ⊤ t − b i ( θ ) } \theta\mapsto\exp\{\theta^\top t-b_i(\theta)\} θ ↦ exp { θ ⊤ t − b i ( θ )} is bounded and continuous on Θ i \Theta_i Θ i . By weak convergence,
F i , P m ( t ) → F i , P ( t ) , F i , Q m ( t ) → F i , Q ( t ) . F_{i,P_m}(t)\to F_{i,P}(t),\qquad F_{i,Q_m}(t)\to F_{i,Q}(t). F i , P m ( t ) → F i , P ( t ) , F i , Q m ( t ) → F i , Q ( t ) .
Since the two limits coincide and are strictly positive, continuity of the logarithm gives
∣ log F i , P m ( t ) − log F i , Q m ( t ) ∣ → 0. \left|\log F_{i,P_m}(t)-\log F_{i,Q_m}(t)\right|\to0. ∣ log F i , P m ( t ) − log F i , Q m ( t ) ∣ → 0.
This contradicts the lower bound η > 0 \eta>0 η > 0 .
(d) The log-moment bound (83) implies that D ( f i , p ) D(f_i,p) D ( f i , p ) is finite for every f i , p ∈ Φ i f_i,p \in\Phi_i f i , p ∈ Φ i . Whenever p , q ∈ Φ i p,q \in\Phi_i p , q ∈ Φ i satisfy ∥ p − q ∥ 1 < ϵ \lVert p-q\rVert_1 < \epsilon ∥ p − q ∥ 1 < ϵ ,
∣ D ( f i , p ) − D ( f i , q ) ∣ = ∣ E f i [ log q ( X i ) − log p ( X i ) ] ∣ ≤ E f i [ W i , ϵ ( X i ) ] . \begin{aligned}
\lvert D(f_i,p)-D(f_i,q)\rvert=\lvert\mathbb{E}_{f_i}[\log q(X_i)-\log p(X_i)]\rvert\\
&\leq\mathbb{E}_{f_i}[W_{i,\epsilon}(X_i)].
\end{aligned} ∣ D ( f i , p ) − D ( f i , q )∣ = ∣ E f i [ log q ( X i ) − log p ( X i )]∣ ≤ E f i [ W i , ϵ ( X i )] .
where W i , ϵ ( x ) W_{i,\epsilon}(x) W i , ϵ ( x ) is defined in (86) . By (88) , the right-hand side tends to zero with ϵ → 0 \epsilon\to0 ϵ → 0 , proving the continuity required in (d).
Appendix B
In Appendix B, we prove Theorems 8.1 and 8.2, establishing the required asymptotic properties of the GLLR and ALLR statistics. The proofs use Lemma B.1, which provides an exponential bound on the probability that the sampled sum defined below is negative while the empirical sampling frequency is at least a fixed positive constant. To state this lemma, we introduce the following notation.
For each i ∈ [ M ] i \in[M] i ∈ [ M ] , g ∈ Φ i g \in\Phi_i g ∈ Φ i , ϵ > 0 \epsilon> 0 ϵ > 0 , and δ ∈ R \delta\in\mathbb{R} δ ∈ R define
z i ( X i ( m ) ) : = log f i ( X i ( m ) ) − log p ϵ + ( g , X i ( m ) ) − δ , m ∈ N , z_i(X_i(m)):=\log f_i(X_i(m))-\log p_\epsilon^+(g,X_i(m))-\delta,\qquad m\in\mathbb{N}, z i ( X i ( m )) := log f i ( X i ( m )) − log p ϵ + ( g , X i ( m )) − δ , m ∈ N ,
and
S i ( n ) : = ∑ m = 1 n z i ( X i ( m ) ) R i ( m ) , n ∈ N . S_i(n):=\sum_{m=1}^{n}z_i(X_i(m))R_i(m),\qquad n\in\mathbb{N}. S i ( n ) := m = 1 ∑ n z i ( X i ( m )) R i ( m ) , n ∈ N .
Lemma B.1. If δ < D ( f i , g ) \delta< D(f_i,g) δ < D ( f i , g ) , then there exists ϵ > 0 \epsilon> 0 ϵ > 0 such that, for every ρ > 0 \rho> 0 ρ > 0 ,
P f i ( S i ( n ) ≤ 0 , π i R ( n ) ≥ ρ ) , n ∈ N , \mathbb{P}_{f_i}\left(S_i(n)\leq0,\pi_i^R(n)\geq\rho\right),\qquad n\in\mathbb{N}, P f i ( S i ( n ) ≤ 0 , π i R ( n ) ≥ ρ ) , n ∈ N , is exponentially decaying.
Proof. To prove the claim, it suffices to show that there exist constants s > 0 s > 0 s > 0 and b > 0 b > 0 b > 0 such that
E f i [ e − s z i ( X i ( 1 ) ) ] ≤ e − b . (90) \mathbb{E}_{f_i}\left[e^{-s z_i(X_i(1))}\right]\leq e^{-b}. \tag*{(90)} E f i [ e − s z i ( X i ( 1 )) ] ≤ e − b . (90)
Indeed, since R i ( n ) ∈ { 0 , 1 } R_i(n)\in\{0,1\} R i ( n ) ∈ { 0 , 1 } is F n − 1 R \mathcal{F}_{n-1}^R F n − 1 R -measurable and X i ( n ) X_i(n) X i ( n ) is independent of F n − 1 R \mathcal{F}_{n-1}^R F n − 1 R , we have
E f i [ e − s z i ( X i ( n ) ) R i ( n ) ∣ F n − 1 R ] = ( E f i [ e − s z i ( X i ( n ) ) ] ) R i ( n ) ≤ e − b R i ( n ) , a.s. (91) \begin{aligned}
\mathbb{E}_{f_i}\left[\left.e^{-s z_i(X_i(n))R_i(n)}\right|\mathcal{F}_{n-1}^R\right]
=\left(\mathbb{E}_{f_i}\left[e^{-s z_i(X_i(n))}\right]\right)^{R_i(n)} \\
&\leq e^{-bR_i(n)},\qquad\text{a.s.} \tag*{(91)}
\end{aligned} E f i [ e − s z i ( X i ( n )) R i ( n ) F n − 1 R ] = ( E f i [ e − s z i ( X i ( n )) ] ) R i ( n ) ≤ e − b R i ( n ) , a.s. (91)
where the last inequality follows from (90) . Consequently, the process
Q ( n ) : = exp { − s S i ( n ) + b N i R ( n ) } , n ∈ N , Q(n):=\exp\left\{-sS_i(n)+bN_i^R(n)\right\},\qquad n\in\mathbb{N}, Q ( n ) := exp { − s S i ( n ) + b N i R ( n ) } , n ∈ N ,
is a nonnegative supermartingale. Therefore,
E f [ Q ( n ) ] ≤ E f [ Q ( 0 ) ] = 1. \mathbb{E}_f[Q(n)] \le\mathbb{E}_f[Q(0)] = 1. E f [ Q ( n )] ≤ E f [ Q ( 0 )] = 1.
On the event
{ S i ( n ) ≤ 0 , N i R ( n ) ≥ ρ n } , \{S_i(n) \le0,\ N_i^{R}(n) \ge\rho n\}, { S i ( n ) ≤ 0 , N i R ( n ) ≥ ρ n } ,
we have
Q ( n ) ≥ e b ρ n . Q(n) \ge e^{b\rho n}. Q ( n ) ≥ e b ρ n .
Hence,
P f ( S i ( n ) ≤ 0 , N i R ( n ) ≥ ρ n ) ≤ P f ( Q ( n ) ≥ e b ρ n ) ≤ e − b ρ n E f [ Q ( n ) ] ≤ e − b ρ n , \mathbb{P}_f\left(S_i(n) \le0,\ N_i^{R}(n) \ge\rho n\right) \le\mathbb{P}_f\left(Q(n) \ge e^{b\rho n}\right)
\le e^{-b\rho n}\mathbb{E}_f[Q(n)]
\le e^{-b\rho n}, P f ( S i ( n ) ≤ 0 , N i R ( n ) ≥ ρ n ) ≤ P f ( Q ( n ) ≥ e b ρ n ) ≤ e − b ρ n E f [ Q ( n )] ≤ e − b ρ n ,
which proves the claim. To prove (89), we consider the function
w ( t ) : = E f [ e − t z i ( X i ( 1 ) ) ] . w(t) := \mathbb{E}_f\left[e^{-tz_i(X_i(1))}\right]. w ( t ) := E f [ e − t z i ( X i ( 1 )) ] .
By the dominated convergence theorem, in order to prove that w ( t ) w(t) w ( t ) is twice differentiable on the interval [ 0 , γ ) [0,\gamma) [ 0 , γ ) , for some γ > 0 \gamma> 0 γ > 0 , it suffices to show that
E f [ z i 2 ( X i ( 1 ) ) e − t z i ( X i ( 1 ) ) ] < ∞ , ∀ t ∈ [ 0 , γ ) , (92) \mathbb{E}_f\left[z_i^2(X_i(1))e^{-tz_i(X_i(1))}\right] < \infty,\qquad\forall t \in[0,\gamma), \tag*{(92)} E f [ z i 2 ( X i ( 1 )) e − t z i ( X i ( 1 )) ] < ∞ , ∀ t ∈ [ 0 , γ ) , (92)
which follows form the fact that for every t ∈ [ 0 , a ] t \in[0,a] t ∈ [ 0 , a ] , a ∈ ( 0 , γ ) a \in(0,\gamma) a ∈ ( 0 , γ ) , and k ∈ { 1 , 2 } k \in\{1,2\} k ∈ { 1 , 2 } , it holds
∣ z i ( X i ( 1 ) ) ∣ k e − t z i ( X i ( 1 ) ) ≤ e a + z i 2 ( X i ( 1 ) ) ( 1 + e − a z i ( X i ( 1 ) ) ) , a.s. \left|z_i(X_i(1))\right|^k e^{-tz_i(X_i(1))} \le e^a + z_i^2(X_i(1))\left(1+e^{-az_i(X_i(1))}\right),\qquad\text{a.s.} ∣ z i ( X i ( 1 )) ∣ k e − t z i ( X i ( 1 )) ≤ e a + z i 2 ( X i ( 1 )) ( 1 + e − a z i ( X i ( 1 )) ) , a.s.
By Hölder’s inequality,
E f [ z i 2 ( X i ( 1 ) ) e − t z i ( X i ( 1 ) ) ] ≤ E f [ ∣ z i ( X i ( 1 ) ) ∣ q ] 2 / q E f [ e − q q − 2 t z i ( X i ( 1 ) ) ] ( q − 2 ) / q . \mathbb{E}_f\left[z_i^2(X_i(1))e^{-tz_i(X_i(1))}\right]
\le\mathbb{E}_f\left[|z_i(X_i(1))|^q\right]^{2/q}
\mathbb{E}_f\left[e^{-\frac{q}{q-2}tz_i(X_i(1))}\right]^{(q-2)/q}. E f [ z i 2 ( X i ( 1 )) e − t z i ( X i ( 1 )) ] ≤ E f [ ∣ z i ( X i ( 1 )) ∣ q ] 2/ q E f [ e − q − 2 q t z i ( X i ( 1 )) ] ( q − 2 ) / q .
Since log \log log is an increasing function, the first term on the right-hand side is bounded by regularity condition (e). For the second term, it suffices to show that there exists a constant C > 0 C > 0 C > 0 such that
E f [ e − u z i ( X i ( 1 ) ) ] ≤ C , ∀ u ∈ [ 0 , 1 ) . (93) \mathbb{E}_f\left[e^{-uz_i(X_i(1))}\right] \le C,\qquad\forall u \in[0,1). \tag*{(93)} E f [ e − u z i ( X i ( 1 )) ] ≤ C , ∀ u ∈ [ 0 , 1 ) . (93)
Indeed, for every u ∈ [ 0 , 1 ) u \in[0,1) u ∈ [ 0 , 1 ) ,
∫ e − u z i ( x ) f i ( x ) d ν i ( x ) = ∫ { z i ( x ) ≥ 0 } e − u z i ( x ) f i ( x ) d ν i ( x ) + ∫ { z i ( x ) < 0 } e − u z i ( x ) f i ( x ) d ν i ( x ) \int e^{-uz_i(x)}f_i(x)\,d\nu_i(x)
=
\int_{\{z_i(x)\ge0\}}e^{-uz_i(x)}f_i(x)\,d\nu_i(x)
+
\int_{\{z_i(x)<0\}}e^{-uz_i(x)}f_i(x)\,d\nu_i(x) ∫ e − u z i ( x ) f i ( x ) d ν i ( x ) = ∫ { z i ( x ) ≥ 0 } e − u z i ( x ) f i ( x ) d ν i ( x ) + ∫ { z i ( x ) < 0 } e − u z i ( x ) f i ( x ) d ν i ( x )
≤ 1 + ∫ { z i ( x ) < 0 } e − z i ( x ) f i ( x ) d ν i ( x ) ≤ E f [ e − z i ( X i ( 1 ) ) ] + 1 = e δ E f [ p ϵ + ( g , X i ( 1 ) ) f i ( X i ( 1 ) ) ] + 1 , \begin{aligned}
\le1+\int_{\{z_i(x)<0\}}e^{-z_i(x)}f_i(x)\,d\nu_i(x) \\
&\le\mathbb{E}_f\left[e^{-z_i(X_i(1))}\right]+1
= e^\delta\mathbb{E}_f\left[\frac{p_\epsilon^+(g,X_i(1))}{f_i(X_i(1))}\right]+1,
\end{aligned} ≤ 1 + ∫ { z i ( x ) < 0 } e − z i ( x ) f i ( x ) d ν i ( x ) ≤ E f [ e − z i ( X i ( 1 )) ] + 1 = e δ E f [ f i ( X i ( 1 )) p ϵ + ( g , X i ( 1 )) ] + 1 ,
and (93) follows from regularity condition (f). Therefore, w ( t ) w(t) w ( t ) is twice differentiable for all t ∈ [ 0 , γ ) t \in[0,\gamma) t ∈ [ 0 , γ ) , where γ = ( q − 2 ) / q \gamma=(q-2)/q γ = ( q − 2 ) / q , and
w ′ ( t ) = E f [ − z i ( X i ( 1 ) ) e − t z i ( X i ( 1 ) ) ] , w ′ ′ ( t ) = E f [ z i 2 ( X i ( 1 ) ) e − t z i ( X i ( 1 ) ) ] . w'(t)=\mathbb{E}_f\left[-z_i(X_i(1))e^{-tz_i(X_i(1))}\right],\qquad w''(t)=\mathbb{E}_f\left[z_i^2(X_i(1))e^{-tz_i(X_i(1))}\right]. w ′ ( t ) = E f [ − z i ( X i ( 1 )) e − t z i ( X i ( 1 )) ] , w ′′ ( t ) = E f [ z i 2 ( X i ( 1 )) e − t z i ( X i ( 1 )) ] .
Therefore, w ( 0 ) = 1 w(0)=1 w ( 0 ) = 1 and
w ′ ( 0 ) = E f [ − z i ( X i ( 1 ) ) ] = E f [ sup h ∈ V ϵ ( g ) log h ( X i ( 1 ) ) − log f i ( X i ( 1 ) ) ] + δ = E f [ sup h ∈ V ϵ ( g ) ( log h ( X i ( 1 ) ) − log g ( X i ( 1 ) ) ) ] + E f [ log g ( X i ( 1 ) ) − log f i ( X i ( 1 ) ) ] + δ ≤ H f ( g , ϵ ) − D ( f i , g ) + δ . \begin{aligned}
w'(0)=\mathbb{E}_f\left[-z_i(X_i(1))\right] \\
&=\mathbb{E}_f\left[\sup_{h\in V_\epsilon(g)}\log h(X_i(1))-\log f_i(X_i(1))\right]+\delta\\
&=\mathbb{E}_f\left[\sup_{h\in V_\epsilon(g)}\left(\log h(X_i(1))-\log g(X_i(1))\right)\right]+\mathbb{E}_f\left[\log g(X_i(1))-\log f_i(X_i(1))\right]+\delta\\
&\le H_f(g,\epsilon)-D(f_i,g)+\delta.
\end{aligned} w ′ ( 0 ) = E f [ − z i ( X i ( 1 )) ] = E f [ h ∈ V ϵ ( g ) sup log h ( X i ( 1 )) − log f i ( X i ( 1 )) ] + δ = E f [ h ∈ V ϵ ( g ) sup ( log h ( X i ( 1 )) − log g ( X i ( 1 )) ) ] + E f [ log g ( X i ( 1 )) − log f i ( X i ( 1 )) ] + δ ≤ H f ( g , ϵ ) − D ( f i , g ) + δ .
By regularity condition (g), there exists ϵ > 0 \epsilon>0 ϵ > 0 sufficiently small such that
H f ( g , ϵ ) < D ( f i , g ) − δ 2 , H_f(g,\epsilon)<\frac{D(f_i,g)-\delta}{2}, H f ( g , ϵ ) < 2 D ( f i , g ) − δ ,
and therefore
w ′ ( 0 ) < − D ( f i , g ) − δ 2 < 0. w'(0)<-\frac{D(f_i,g)-\delta}{2}<0. w ′ ( 0 ) < − 2 D ( f i , g ) − δ < 0.
Moreover,
w ′ ′ ( t ) ≤ D , ∀ t ∈ [ 0 , ( q − 2 ) / q ) , w''(t)\le D,\qquad\forall t\in[0,(q-2)/q), w ′′ ( t ) ≤ D , ∀ t ∈ [ 0 , ( q − 2 ) / q ) ,
where D > 0 D>0 D > 0 is a constant. Hence, w ′ w' w ′ is Lipschitz continuous on [ 0 , ( q − 2 ) / q ) [0,(q-2)/q) [ 0 , ( q − 2 ) / q ) . Since w ( 0 ) = 1 w(0)=1 w ( 0 ) = 1 and the right derivative w + ′ ( 0 ) < 0 w'_+(0)<0 w + ′ ( 0 ) < 0 , there exist constants s > 0 s>0 s > 0 and b > 0 b>0 b > 0 such that
w ( s ) = e − b < 1 , w(s)=e^{-b}<1, w ( s ) = e − b < 1 ,
which proves (90) . ■
Proof of Theorem 8.1: We prove the result for i ∈ A i\in A i ∈ A , as the case i ∉ A i\notin A i ∈ / A follows analogously. Fix i ∈ A i\in A i ∈ A , ρ ∈ ( 0 , 1 ] \rho\in(0,1] ρ ∈ ( 0 , 1 ] , and ϵ ∈ ( 0 , I i ( f i ) ) \epsilon\in(0,I_i(f_i)) ϵ ∈ ( 0 , I i ( f i )) . By the definition of l i 1 ( n ) l_i^1(n) l i 1 ( n ) in (48) , we have
l i 1 ( n ) ≥ l i ( n ; f i ) , ∀ n ∈ N . (94) l_i^1(n)\ge l_i(n;f_i),\qquad\forall n\in\mathbb{N}. \tag*{(94)} l i 1 ( n ) ≥ l i ( n ; f i ) , ∀ n ∈ N . (94)
Since Φ i 0 \Phi_i^0 Φ i 0 is compact, there exists a finite δ \delta δ -cover { g j } j = 1 L ( δ ) ⊂ Φ i 0 \{g_j\}_{j=1}^{L(\delta)}\subset\Phi_i^0 { g j } j = 1 L ( δ ) ⊂ Φ i 0 , and
l i 0 ( n ) ≤ max 1 ≤ j ≤ L ( δ ) sup g ∈ V δ ( g j ) l i ( n ; g ) . l_i^0(n)\le\max_{1\le j\le L(\delta)}\sup_{g\in V_\delta(g_j)}l_i(n;g). l i 0 ( n ) ≤ 1 ≤ j ≤ L ( δ ) max g ∈ V δ ( g j ) sup l i ( n ; g ) .
To prove the claim, it suffices to show that for each j ∈ [ L ( δ ) ] j\in[L(\delta)] j ∈ [ L ( δ )] ,
P f ( l i ( n ; f i ) − sup g ∈ V δ ( g j ) l i ( n ; g ) < ( ρ I i ( f i ) − ϵ 2 ) n , π i R ( n ) ≥ ρ ) \mathbb{P}_f\left(l_i(n;f_i)-\sup_{g\in V_\delta(g_j)}l_i(n;g)<\left(\rho I_i(f_i)-\frac{\epsilon}{2}\right)n,\ \pi_i^R(n)\ge\rho\right) P f ( l i ( n ; f i ) − g ∈ V δ ( g j ) sup l i ( n ; g ) < ( ρ I i ( f i ) − 2 ϵ ) n , π i R ( n ) ≥ ρ )
is exponentially decaying. Fix j ∈ [ L ( δ ) ] j \in[L(\delta)] j ∈ [ L ( δ )] . Since N i R ( n ) ≥ ρ n N_i^R(n) \ge\rho n N i R ( n ) ≥ ρ n , and
sup g ∈ V δ ( g j ) l i ( n ; g ) ≤ ∑ m = 1 n log p δ + ( g j , X i ( m ) ) R i ( m ) , \sup_{g \in V_\delta(g_j)} l_i(n;g) \le\sum_{m=1}^{n} \log p_\delta^+(g_j,X_i(m))R_i(m), g ∈ V δ ( g j ) sup l i ( n ; g ) ≤ m = 1 ∑ n log p δ + ( g j , X i ( m )) R i ( m ) ,
we have
P f ( l i ( n ; f i ) − sup g ∈ V δ ( g j ) l i ( n ; g ) < ( ρ I i ( f i ) − ϵ 2 ) n , π i R ( n ) ≥ ρ ) ≤ P f ( l i ( n ; f i ) − ∑ m = 1 n log p δ + ( g j , X i ( m ) ) R i ( m ) − N i R ( n ) I i ( f i ) < − ϵ 2 n , π i R ( n ) ≥ ρ ) = P f ( ∑ m = 1 n ( log f i ( X i ( m ) ) − log p δ + ( g j , X i ( m ) ) − I i ( f i ) ) R i ( m ) < − ϵ 2 n , π i R ( n ) ≥ ρ ) ≤ P f ( ∑ m = 1 n ( log f i ( X i ( m ) ) − log p δ + ( g j , X i ( m ) ) − I i ( f i ) + ϵ 2 ) R i ( m ) < 0 , π i R ( n ) ≥ ρ ) , \begin{aligned}
\mathbb{P}_f\left(l_i(n;f_i)-\sup_{g \in V_\delta(g_j)}l_i(n;g)<\left(\rho I_i(f_i)-\frac{\epsilon}{2}\right)n,\ \pi_i^R(n)\ge\rho\right) \\
&\le\mathbb{P}_f\left(l_i(n;f_i)-\sum_{m=1}^{n}\log p_\delta^+(g_j,X_i(m))R_i(m)-N_i^R(n)I_i(f_i)<-\frac{\epsilon}{2}n,\ \pi_i^R(n)\ge\rho\right) \\
&=\mathbb{P}_f\left(\sum_{m=1}^{n}\left(\log f_i(X_i(m))-\log p_\delta^+(g_j,X_i(m))-I_i(f_i)\right)R_i(m)<-\frac{\epsilon}{2}n,\ \pi_i^R(n)\ge\rho\right) \\
&\le\mathbb{P}_f\left(\sum_{m=1}^{n}\left(\log f_i(X_i(m))-\log p_\delta^+(g_j,X_i(m))-I_i(f_i)+\frac{\epsilon}{2}\right)R_i(m)<0,\ \pi_i^R(n)\ge\rho\right),
\end{aligned} P f ( l i ( n ; f i ) − g ∈ V δ ( g j ) sup l i ( n ; g ) < ( ρ I i ( f i ) − 2 ϵ ) n , π i R ( n ) ≥ ρ ) ≤ P f ( l i ( n ; f i ) − m = 1 ∑ n log p δ + ( g j , X i ( m )) R i ( m ) − N i R ( n ) I i ( f i ) < − 2 ϵ n , π i R ( n ) ≥ ρ ) = P f ( m = 1 ∑ n ( log f i ( X i ( m )) − log p δ + ( g j , X i ( m )) − I i ( f i ) ) R i ( m ) < − 2 ϵ n , π i R ( n ) ≥ ρ ) ≤ P f ( m = 1 ∑ n ( log f i ( X i ( m )) − log p δ + ( g j , X i ( m )) − I i ( f i ) + 2 ϵ ) R i ( m ) < 0 , π i R ( n ) ≥ ρ ) ,
where, in the last inequality, we used the fact that N i R ( n ) ≤ n N_i^R(n)\le n N i R ( n ) ≤ n . By definition
I i ( f i ) − ϵ 2 < D ( f i , g j ) , I_i(f_i)-\frac{\epsilon}{2}<D(f_i,g_j), I i ( f i ) − 2 ϵ < D ( f i , g j ) ,
and thus for δ = I i ( f i ) − ϵ 2 \delta=I_i(f_i)-\frac{\epsilon}{2} δ = I i ( f i ) − 2 ϵ the claim follows by Lemma B.1.
■
We next prove Proposition 7.1, which will be used in the proof of Theorem 8.2.
Proof of Proposition 7.1: By definition (37) of τ i R ( ϵ ) \tau_i^R(\epsilon) τ i R ( ϵ ) , we have
{ τ i R ( ϵ ) > c n } ⊆ { ∃ m ≥ c n : f ^ i , m ∉ V ϵ ( f i ) } \{\tau_i^R(\epsilon)>cn\}\subseteq\{\exists m\ge cn:\widehat f_{i,m}\notin V_\epsilon(f_i)\} { τ i R ( ϵ ) > c n } ⊆ { ∃ m ≥ c n : f i , m ∈ / V ϵ ( f i )}
and by the union bound,
P f ( τ i R ( ϵ ) > c n ) ≤ ∑ m = ⌈ c n ⌉ ∞ P f ( f ^ i , m ∉ V ϵ ( f i ) ) . \mathbb{P}_f(\tau_i^R(\epsilon)>cn)\le\sum_{m=\lceil cn\rceil}^{\infty}\mathbb{P}_f(\widehat f_{i,m}\notin V_\epsilon(f_i)). P f ( τ i R ( ϵ ) > c n ) ≤ m = ⌈ c n ⌉ ∑ ∞ P f ( f i , m ∈ / V ϵ ( f i )) .
Therefore,
∑ n = 1 ∞ P f ( τ i R ( ϵ ) > c n ) ≤ ∑ n = 1 ∞ ∑ m = ⌈ c n ⌉ ∞ P f ( f ^ i , m ∉ V ϵ ( f i ) ) . \sum_{n=1}^{\infty}\mathbb{P}_f(\tau_i^R(\epsilon)>cn)\le\sum_{n=1}^{\infty}\sum_{m=\lceil cn\rceil}^{\infty}\mathbb{P}_f(\widehat f_{i,m}\notin V_\epsilon(f_i)). n = 1 ∑ ∞ P f ( τ i R ( ϵ ) > c n ) ≤ n = 1 ∑ ∞ m = ⌈ c n ⌉ ∑ ∞ P f ( f i , m ∈ / V ϵ ( f i )) .
For fixed m m m , the number of integers n n n such that ⌈ c n ⌉ = m \lceil cn\rceil=m ⌈ c n ⌉ = m is at most ⌈ 1 / c ⌉ \lceil1/c\rceil ⌈ 1/ c ⌉ . Hence,
∑ n = 1 ∞ P f ( τ i R ( ϵ ) > c n ) ≤ ∑ n = 1 ∞ ∑ m = n ∞ ⌈ 1 / c ⌉ P f ( f ^ i , m ∉ V ϵ ( f i ) ) ≤ ⌈ 1 / c ⌉ ∑ n = 1 ∞ n P f ( f ^ i , n ∉ V ϵ ( f i ) ) . \begin{aligned}
\sum_{n=1}^{\infty}\mathbb{P}_f(\tau_i^R(\epsilon)>cn)\le\sum_{n=1}^{\infty}\sum_{m=n}^{\infty}\lceil1/c\rceil\mathbb{P}_f(\widehat f_{i,m}\notin V_\epsilon(f_i)) \\
&\le\lceil1/c\rceil\sum_{n=1}^{\infty}n\mathbb{P}_f(\widehat f_{i,n}\notin V_\epsilon(f_i)).
\end{aligned} n = 1 ∑ ∞ P f ( τ i R ( ϵ ) > c n ) ≤ n = 1 ∑ ∞ m = n ∑ ∞ ⌈ 1/ c ⌉ P f ( f i , m ∈ / V ϵ ( f i )) ≤ ⌈ 1/ c ⌉ n = 1 ∑ ∞ n P f ( f i , n ∈ / V ϵ ( f i )) .
We now split
∑ n = 1 ∞ n P f ( f ^ i , n ∉ V ϵ ( f i ) ) ≤ ∑ n = 1 ∞ n P f ( f ^ i , n ∉ V ϵ ( f i ) , π i R ( n ) ≥ ρ ) + ∑ n = 1 ∞ n P f ( π i R ( n ) < ρ ) . \sum_{n=1}^{\infty} nP_f\left(\hat{f}_{i,n} \notin V_{\epsilon}(f_i)\right) \le\sum_{n=1}^{\infty} nP_f\left(\hat{f}_{i,n} \notin V_{\epsilon}(f_i),\ \pi_i^R(n) \ge\rho\right) + \sum_{n=1}^{\infty} nP_f\left(\pi_i^R(n) < \rho\right). n = 1 ∑ ∞ n P f ( f ^ i , n ∈ / V ϵ ( f i ) ) ≤ n = 1 ∑ ∞ n P f ( f ^ i , n ∈ / V ϵ ( f i ) , π i R ( n ) ≥ ρ ) + n = 1 ∑ ∞ n P f ( π i R ( n ) < ρ ) .
The first term is summable by condition (40) of Theorem 7.2 . The second term is summable by Lemma 6.1, because by (29) and the fact that x A ( f ) , y A ( f ) > 0 x_A(f), y_A(f) > 0 x A ( f ) , y A ( f ) > 0 it follows that c i ∗ ( g ) > 0 c_i^*(g) > 0 c i ∗ ( g ) > 0 for every i ∈ [ M ] i \in[M] i ∈ [ M ] and g ∈ Φ g \in\Phi g ∈ Φ , and therefore condition (38) implies (27).
Proof of Theorem 8.2. We prove the result for i ∈ A i \in A i ∈ A , as the case i ∈ A c i \in A^c i ∈ A c follows analogously. Fix i ∈ A i \in A i ∈ A , ρ ∈ ( 0 , 1 ] \rho\in(0,1] ρ ∈ ( 0 , 1 ] , and ϵ > 0 \epsilon> 0 ϵ > 0 . We begin with the decomposition
P f ( 1 n Λ i R ( n ) < ρ I i ( f i ) − ϵ , π i R ( n ) ≥ ρ ) ≤ P f ( 1 n Λ i R ( n ) < ρ I i ( f i ) − ϵ , l i 1 ( n ) − l i 0 ( n ) > 0 , l ^ i ( n ) − l i 0 ( n ) > 0 , π i R ( n ) ≥ ρ ) + P f ( l ^ i ( n ) − l i 0 ( n ) ≤ 0 , π i R ( n ) ≥ ρ ) + P f ( l i 1 ( n ) − l i 0 ( n ) ≤ 0 , π i R ( n ) ≥ ρ ) . \begin{aligned}
P_f\left(\frac{1}{n}\Lambda_i^R(n) < \rho I_i(f_i)-\epsilon,\ \pi_i^R(n) \ge\rho\right) \\
&\le P_f\left(\frac{1}{n}\Lambda_i^R(n) < \rho I_i(f_i)-\epsilon,\ l_i^1(n)-l_i^0(n)>0,\ \hat{l}_i(n)-l_i^0(n)>0,\ \pi_i^R(n) \ge\rho\right) \\
&+P_f\left(\hat{l}_i(n)-l_i^0(n)\le0,\ \pi_i^R(n) \ge\rho\right)+P_f\left(l_i^1(n)-l_i^0(n)\le0,\ \pi_i^R(n) \ge\rho\right).
\end{aligned} P f ( n 1 Λ i R ( n ) < ρ I i ( f i ) − ϵ , π i R ( n ) ≥ ρ ) ≤ P f ( n 1 Λ i R ( n ) < ρ I i ( f i ) − ϵ , l i 1 ( n ) − l i 0 ( n ) > 0 , l ^ i ( n ) − l i 0 ( n ) > 0 , π i R ( n ) ≥ ρ ) + P f ( l ^ i ( n ) − l i 0 ( n ) ≤ 0 , π i R ( n ) ≥ ρ ) + P f ( l i 1 ( n ) − l i 0 ( n ) ≤ 0 , π i R ( n ) ≥ ρ ) .
On the event { l i 1 ( n ) − l i 0 ( n ) > 0 , l ^ i ( n ) − l i 0 ( n ) > 0 } \{l_i^1(n)-l_i^0(n)>0,\ \hat{l}_i(n)-l_i^0(n)>0\} { l i 1 ( n ) − l i 0 ( n ) > 0 , l ^ i ( n ) − l i 0 ( n ) > 0 } , we have Λ i R ( n ) = l ^ i ( n ) − l i 0 ( n ) \Lambda_i^R(n)=\hat{l}_i(n)-l_i^0(n) Λ i R ( n ) = l ^ i ( n ) − l i 0 ( n ) by (51). Hence,
P f ( 1 n Λ i R ( n ) < ρ I i ( f i ) − ϵ , π i R ( n ) ≥ ρ ) ≤ 2 P f ( l ^ i ( n ) − l i 0 ( n ) < ( ρ I i ( f i ) − ϵ ) n , π i R ( n ) ≥ ρ ) + P f ( l i 1 ( n ) − l i 0 ( n ) ≤ 0 , π i R ( n ) ≥ ρ ) . \begin{aligned}
P_f\left(\frac{1}{n}\Lambda_i^R(n) < \rho I_i(f_i)-\epsilon,\ \pi_i^R(n) \ge\rho\right) \le2P_f\left(\hat{l}_i(n)-l_i^0(n)<(\rho I_i(f_i)-\epsilon)n,\ \pi_i^R(n) \ge\rho\right) \\
&+P_f\left(l_i^1(n)-l_i^0(n)\le0,\ \pi_i^R(n) \ge\rho\right).
\end{aligned} P f ( n 1 Λ i R ( n ) < ρ I i ( f i ) − ϵ , π i R ( n ) ≥ ρ ) ≤ 2 P f ( l ^ i ( n ) − l i 0 ( n ) < ( ρ I i ( f i ) − ϵ ) n , π i R ( n ) ≥ ρ ) + P f ( l i 1 ( n ) − l i 0 ( n ) ≤ 0 , π i R ( n ) ≥ ρ ) .
By Theorem 8.1, the last term is exponentially decaying. It remains to show that
∑ n = 1 ∞ P f ( l ^ i ( n ) − l i 0 ( n ) < ( ρ I i ( f i ) − ϵ ) n , π i R ( n ) ≥ ρ ) < ∞ . \sum_{n=1}^{\infty}P_f\left(\hat{l}_i(n)-l_i^0(n)<(\rho I_i(f_i)-\epsilon)n,\ \pi_i^R(n) \ge\rho\right)<\infty. n = 1 ∑ ∞ P f ( l ^ i ( n ) − l i 0 ( n ) < ( ρ I i ( f i ) − ϵ ) n , π i R ( n ) ≥ ρ ) < ∞.
Since
l ^ i ( n ) − l i 0 ( n ) < ( ρ I i ( f i ) − ϵ ) n ⟺ l ^ i ( n ) − l i ( n ; f i ) + l i ( n ; f i ) − l i 0 ( n ) < ( ρ I i ( f i ) − ϵ 2 ) n − ϵ 2 n , \hat{l}_i(n)-l_i^0(n)<(\rho I_i(f_i)-\epsilon)n \Longleftrightarrow\hat{l}_i(n)-l_i(n;f_i)+l_i(n;f_i)-l_i^0(n)<\left(\rho I_i(f_i)-\frac{\epsilon}{2}\right)n-\frac{\epsilon}{2}n, l ^ i ( n ) − l i 0 ( n ) < ( ρ I i ( f i ) − ϵ ) n ⟺ l ^ i ( n ) − l i ( n ; f i ) + l i ( n ; f i ) − l i 0 ( n ) < ( ρ I i ( f i ) − 2 ϵ ) n − 2 ϵ n ,
we obtain
P f ( l ^ i ( n ) − l i 0 ( n ) < ( ρ I i ( f i ) − ϵ ) n , π i R ( n ) ≥ ρ ) ≤ P f ( l ^ i ( n ) − l i ( n ; f i ) < − ϵ 2 n , π i R ( n ) ≥ ρ ) + P f ( l i ( n ; f i ) − l i 0 ( n ) < ( ρ I i ( f i ) − ϵ 2 ) n , π i R ( n ) ≥ ρ ) . \begin{aligned}
P_f\left(\hat{l}_i(n)-l_i^0(n)<(\rho I_i(f_i)-\epsilon)n,\ \pi_i^R(n) \ge\rho\right) \\
&\le P_f\left(\hat{l}_i(n)-l_i(n;f_i)<-\frac{\epsilon}{2}n,\ \pi_i^R(n) \ge\rho\right)
+P_f\left(l_i(n;f_i)-l_i^0(n)<\left(\rho I_i(f_i)-\frac{\epsilon}{2}\right)n,\ \pi_i^R(n) \ge\rho\right).
\end{aligned} P f ( l ^ i ( n ) − l i 0 ( n ) < ( ρ I i ( f i ) − ϵ ) n , π i R ( n ) ≥ ρ ) ≤ P f ( l ^ i ( n ) − l i ( n ; f i ) < − 2 ϵ n , π i R ( n ) ≥ ρ ) + P f ( l i ( n ; f i ) − l i 0 ( n ) < ( ρ I i ( f i ) − 2 ϵ ) n , π i R ( n ) ≥ ρ ) .
In the proof of Theorem 8.1, it was shown that the second term is exponentially decaying. Hence, it suffices to show that
∑ n = 1 ∞ P f ( l ^ i ( n ) − l i ( n ; f i ) < − ϵ 2 n , π i R ( n ) ≥ ρ ) < ∞ . (95) \sum_{n=1}^{\infty}P_f\left(\hat{l}_i(n)-l_i(n;f_i)<-\frac{\epsilon}{2}n,\ \pi_i^R(n) \ge\rho\right)<\infty. \tag*{(95)} n = 1 ∑ ∞ P f ( l ^ i ( n ) − l i ( n ; f i ) < − 2 ϵ n , π i R ( n ) ≥ ρ ) < ∞. (95)
We decompose
l i ( n ; f i ) − l ^ i ( n ) = Λ ˉ i R ( n ) + Q ˉ i R ( n ) , l_i(n;f_i)-\hat{l}_i(n)=\bar{\Lambda}_i^R(n)+\bar{Q}_i^R(n), l i ( n ; f i ) − l ^ i ( n ) = Λ ˉ i R ( n ) + Q ˉ i R ( n ) ,
where
Λ ˉ i R ( n ) : = ∑ m = 1 n ( log f i ( X i ( m ) ) f ^ i , m − 1 ( X i ( m ) ) − D ( f i , f ^ i , m − 1 ) ) R i ( m ) , \bar{\Lambda}_i^R(n) := \sum_{m=1}^{n} \left( \log\frac{f_i(X_i(m))}{\hat{f}_{i,m-1}(X_i(m))} - D(f_i,\hat{f}_{i,m-1}) \right) R_i(m), Λ ˉ i R ( n ) := m = 1 ∑ n ( log f ^ i , m − 1 ( X i ( m )) f i ( X i ( m )) − D ( f i , f ^ i , m − 1 ) ) R i ( m ) ,
Q ˉ i R ( n ) : = ∑ m = 1 n D ( f i , f ^ i , m − 1 ) R i ( m ) . \bar{Q}_i^R(n) := \sum_{m=1}^{n} D(f_i,\hat{f}_{i,m-1})R_i(m). Q ˉ i R ( n ) := m = 1 ∑ n D ( f i , f ^ i , m − 1 ) R i ( m ) .
Thus,
P f ( l i ( n ; f i ) − l ^ i ( n ) ≥ ϵ 2 n , π i R ( n ) ≥ ρ ) ≤ P f ( l i ( n ; f i ) − l ^ i ( n ) ≥ ϵ 2 n ) ≤ P f ( Λ ˉ i R ( n ) ≥ ϵ 4 n ) + P f ( Q ˉ i R ( n ) ≥ ϵ 4 n ) . \begin{aligned}
\mathbb{P}_f\left(l_i(n;f_i)-\hat{l}_i(n)\geq\frac{\epsilon}{2}n,\ \pi_i^R(n)\geq\rho\right)
\leq\mathbb{P}_f\left(l_i(n;f_i)-\hat{l}_i(n)\geq\frac{\epsilon}{2}n\right) \\
&\leq\mathbb{P}_f\left(\bar{\Lambda}_i^R(n)\geq\frac{\epsilon}{4}n\right)+\mathbb{P}_f\left(\bar{Q}_i^R(n)\geq\frac{\epsilon}{4}n\right).
\end{aligned} P f ( l i ( n ; f i ) − l ^ i ( n ) ≥ 2 ϵ n , π i R ( n ) ≥ ρ ) ≤ P f ( l i ( n ; f i ) − l ^ i ( n ) ≥ 2 ϵ n ) ≤ P f ( Λ ˉ i R ( n ) ≥ 4 ϵ n ) + P f ( Q ˉ i R ( n ) ≥ 4 ϵ n ) .
Since f ^ i , n − 1 , R i ( n ) ∈ F n − 1 R \hat{f}_{i,n-1},R_i(n)\in\mathcal{F}_{n-1}^R f ^ i , n − 1 , R i ( n ) ∈ F n − 1 R , { Λ ˉ i R ( n ) : n ∈ N } \{\bar{\Lambda}_i^R(n):n\in\mathbb{N}\} { Λ ˉ i R ( n ) : n ∈ N } is a F n R \mathcal{F}_n^R F n R -martingale. By regularity condition (e) and Rosenthal’s inequality [49 Theorem 2.12] , there exists C 0 > 0 C_0>0 C 0 > 0 such that
E f [ ∣ Λ ˉ i R ( n ) ∣ q ] ≤ C 0 n q / 2 . \mathbb{E}_f[|\bar{\Lambda}_i^R(n)|^q]\leq C_0n^{q/2}. E f [ ∣ Λ ˉ i R ( n ) ∣ q ] ≤ C 0 n q /2 .
By Markov’s inequality
P f ( Λ ˉ i R ( n ) ≥ ϵ 4 n ) ≤ C 0 ( ϵ / 4 ) q 1 n q / 2 , \mathbb{P}_f\left(\bar{\Lambda}_i^R(n)\geq\frac{\epsilon}{4}n\right)\leq\frac{C_0}{(\epsilon/4)^q}\frac{1}{n^{q/2}}, P f ( Λ ˉ i R ( n ) ≥ 4 ϵ n ) ≤ ( ϵ /4 ) q C 0 n q /2 1 ,
and since q > 2 q>2 q > 2
∑ n = 1 ∞ P f ( Λ ˉ i R ( n ) ≥ ϵ 4 n ) < ∞ . \sum_{n=1}^{\infty}\mathbb{P}_f\left(\bar{\Lambda}_i^R(n)\geq\frac{\epsilon}{4}n\right)<\infty. n = 1 ∑ ∞ P f ( Λ ˉ i R ( n ) ≥ 4 ϵ n ) < ∞.
For the second term, by regularity condition (d) there exists δ > 0 \delta>0 δ > 0 such that for any h ∈ Φ i h\in\Phi_i h ∈ Φ i with ∥ h − f i ∥ 1 < δ \lVert h-f_i\rVert_1<\delta ∥ h − f i ∥ 1 < δ , it holds that D ( f i , h ) < ϵ / 8 D(f_i,h)<\epsilon/8 D ( f i , h ) < ϵ /8 . By definition of τ i R ( δ ) \tau_i^R(\delta) τ i R ( δ ) in (37), we note that D ( f i , f ^ i , m − 1 ) < ϵ / 8 D(f_i,\hat{f}_{i,m-1})<\epsilon/8 D ( f i , f ^ i , m − 1 ) < ϵ /8 for all m ≥ τ i R ( δ ) m\geq\tau_i^R(\delta) m ≥ τ i R ( δ ) . Therefore,
Q ˉ i R ( n ) ≤ ∑ m = 1 τ i R ( δ ) D ( f i , f ^ i , m − 1 ) + ϵ 8 n . \bar{Q}_i^R(n)\leq\sum_{m=1}^{\tau_i^R(\delta)}D(f_i,\hat{f}_{i,m-1})+\frac{\epsilon}{8}n. Q ˉ i R ( n ) ≤ m = 1 ∑ τ i R ( δ ) D ( f i , f ^ i , m − 1 ) + 8 ϵ n .
Hence,
P f ( Q ˉ i R ( n ) ≥ ϵ 4 n ) ≤ P f ( τ i R ( δ ) ≥ ϵ 8 B n ) , \mathbb{P}_f\left(\bar{Q}_i^R(n)\geq\frac{\epsilon}{4}n\right)\leq\mathbb{P}_f\left(\tau_i^R(\delta)\geq\frac{\epsilon}{8Bn}\right), P f ( Q ˉ i R ( n ) ≥ 4 ϵ n ) ≤ P f ( τ i R ( δ ) ≥ 8 B n ϵ ) ,
where B : = sup g ∈ Φ i D ( f i , g ) B:=\sup_{g\in\Phi_i}D(f_i,g) B := sup g ∈ Φ i D ( f i , g ) and by regularity conditions (a) and (d) it follows that B < ∞ B<\infty B < ∞ . The summability follows from (52).
Appendix C
This appendix provides the proofs of Theorems 9.1 and 9.2, which establish that the MLE and KDE, respectively, satisfy condition (40) of Theorem 7.2 under the stated assumptions.
Proof of Theorem 9.1: We fix i ∈ [ M ] i \in[M] i ∈ [ M ] and ϵ > 0 \epsilon> 0 ϵ > 0 . Let V ϵ ( f i ) V_{\epsilon}(f_i) V ϵ ( f i ) denote the ϵ \epsilon ϵ -neighborhood of f i f_i f i defined in (36). Since Φ i ∖ V ϵ ( f i ) \Phi_i \setminus V_{\epsilon}(f_i) Φ i ∖ V ϵ ( f i ) is compact as a closed subset of the compact set Φ i \Phi_i Φ i , it admits a finite ϵ ′ \epsilon' ϵ ′ -cover { g j } j = 1 L ( ϵ ′ ) \{g_j\}_{j=1}^{L(\epsilon')} { g j } j = 1 L ( ϵ ′ ) for any ϵ ′ > 0 \epsilon' > 0 ϵ ′ > 0 . For each j ∈ [ L ( ϵ ′ ) ] j \in[L(\epsilon')] j ∈ [ L ( ϵ ′ )] , define
S i j ( n ) : = ∑ m = 1 n z i j ( X i ( m ) ) R i ( m ) . S_i^j(n) := \sum_{m=1}^{n} z_i^j(X_i(m))R_i(m). S i j ( n ) := m = 1 ∑ n z i j ( X i ( m )) R i ( m ) .
where
z i j ( X i ( m ) ) : = log f i ( X i ( m ) ) − log p ϵ ′ + ( g j , X i ( m ) ) , m ∈ N . z_i^j(X_i(m)) := \log f_i(X_i(m)) - \log p_{\epsilon'}^+(g_j,X_i(m)), \qquad m \in\mathbb{N}. z i j ( X i ( m )) := log f i ( X i ( m )) − log p ϵ ′ + ( g j , X i ( m )) , m ∈ N .
We note that by definition (46), for each j ∈ [ L ( ϵ ′ ) ] j \in[L(\epsilon')] j ∈ [ L ( ϵ ′ )] ,
l i ( n ; f i ) − l i ( n ; g ) ≥ S i j ( n ) , ∀ g ∈ V ϵ ′ ( g j ) . l_i(n;f_i)-l_i(n;g) \ge S_i^j(n), \qquad\forall g \in V_{\epsilon'}(g_j). l i ( n ; f i ) − l i ( n ; g ) ≥ S i j ( n ) , ∀ g ∈ V ϵ ′ ( g j ) .
Consequently, if S i j ( n ) > 0 S_i^j(n) > 0 S i j ( n ) > 0 for every j ∈ [ L ( ϵ ′ ) ] j \in[L(\epsilon')] j ∈ [ L ( ϵ ′ )] , then every density outside V ϵ ( f i ) V_{\epsilon}(f_i) V ϵ ( f i ) has strictly smaller likelihood than f i f_i f i . Since f ^ i , n \hat f_{i,n} f ^ i , n is a maximizer over Φ i \Phi_i Φ i , this implies f ^ i , n ∈ V ϵ ( f i ) \hat f_{i,n} \in V_{\epsilon}(f_i) f ^ i , n ∈ V ϵ ( f i ) . Thus, by the union bound,
P f ( f ^ i , n ∉ V ϵ ( f i ) , π i R ( n ) ≥ ρ ) ≤ ∑ j = 1 L ( ϵ ′ ) P f ( S i j ( n ) ≤ 0 , π i R ( n ) ≥ ρ ) . \mathbb{P}_f\left(\hat f_{i,n} \notin V_{\epsilon}(f_i),\ \pi_i^R(n) \ge\rho\right)
\le\sum_{j=1}^{L(\epsilon')} \mathbb{P}_f\left(S_i^j(n) \le0,\ \pi_i^R(n) \ge\rho\right). P f ( f ^ i , n ∈ / V ϵ ( f i ) , π i R ( n ) ≥ ρ ) ≤ j = 1 ∑ L ( ϵ ′ ) P f ( S i j ( n ) ≤ 0 , π i R ( n ) ≥ ρ ) .
By regularity condition (c), D ( f i , g j ) > 0 D(f_i,g_j)>0 D ( f i , g j ) > 0 for every j ∈ [ L ( ϵ ′ ) ] j \in[L(\epsilon')] j ∈ [ L ( ϵ ′ )] . Hence, Lemma B.1 implies that each probability on the right-hand side is exponentially decaying. Since the sum is finite, the left-hand side is also exponentially decaying. ■
The following lemma allows us to control L 1 L^1 L 1 estimation errors through bounds on L p L^p L p errors, for p > 1 p>1 p > 1 , and will be used in the proof of Theorem 9.2.
Lemma C.1. Let ( S , S , ν i ) (\mathcal{S},\mathcal{S},\nu_i) ( S , S , ν i ) be a σ \sigma σ -finite measure space, let p > 1 p>1 p > 1 , and let f i f_i f i be a density with respect to ν i \nu_i ν i . For every ϵ > 0 \epsilon>0 ϵ > 0 , there exists a measurable set B ϵ ⊆ S B_{\epsilon}\subseteq\mathcal{S} B ϵ ⊆ S such that
0 < ν i ( B ϵ ) < ∞ 0<\nu_i(B_{\epsilon})<\infty 0 < ν i ( B ϵ ) < ∞ and, for every density g g g with respect to ν i \nu_i ν i ,
∥ g − f i ∥ 1 > ϵ ⇒ ∥ g − f i ∥ p > ϵ 4 ν i ( B ϵ ) 1 − 1 / p . \lVert g-f_i\rVert_1>\epsilon\quad\Rightarrow\quad\lVert g-f_i\rVert_p>\frac{\epsilon}{4\nu_i(B_{\epsilon})^{1-1/p}}. ∥ g − f i ∥ 1 > ϵ ⇒ ∥ g − f i ∥ p > 4 ν i ( B ϵ ) 1 − 1/ p ϵ .
Proof: Since f i f_i f i is a density and ν i \nu_i ν i is σ \sigma σ -finite, there exists a measurable set B ϵ ⊆ S B_{\epsilon}\subseteq\mathcal{S} B ϵ ⊆ S such that
0 < ν i ( B ϵ ) < ∞ and ∫ B ϵ c f i ( x ) ν i ( d x ) < ϵ 4 . 0<\nu_i(B_{\epsilon})<\infty\qquad\text{and}\qquad\int_{B_{\epsilon}^{c}} f_i(x)\,\nu_i(dx)<\frac{\epsilon}{4}. 0 < ν i ( B ϵ ) < ∞ and ∫ B ϵ c f i ( x ) ν i ( d x ) < 4 ϵ .
Let g g g be any density with respect to ν i \nu_i ν i . Since both g g g and f i f_i f i integrate to one,
∫ B ϵ c g ( x ) ν i ( d x ) = 1 − ∫ B ϵ g ( x ) ν i ( d x ) = ∫ B ϵ c f i ( x ) ν i ( d x ) + ∫ B ϵ ( f i ( x ) − g ( x ) ) ν i ( d x ) ≤ ∫ B ϵ c f i ( x ) ν i ( d x ) + ∫ B ϵ ∣ g ( x ) − f i ( x ) ∣ ν i ( d x ) . \begin{aligned}
\int_{B_\epsilon^c} g(x)\,\nu_i(dx) &= 1-\int_{B_\epsilon} g(x)\,\nu_i(dx)\\
&=\int_{B_\epsilon^c} f_i(x)\,\nu_i(dx)+\int_{B_\epsilon}(f_i(x)-g(x))\,\nu_i(dx)\\
&\le\int_{B_\epsilon^c} f_i(x)\,\nu_i(dx)+\int_{B_\epsilon}|g(x)-f_i(x)|\,\nu_i(dx).
\end{aligned} ∫ B ϵ c g ( x ) ν i ( d x ) = 1 − ∫ B ϵ g ( x ) ν i ( d x ) = ∫ B ϵ c f i ( x ) ν i ( d x ) + ∫ B ϵ ( f i ( x ) − g ( x )) ν i ( d x ) ≤ ∫ B ϵ c f i ( x ) ν i ( d x ) + ∫ B ϵ ∣ g ( x ) − f i ( x ) ∣ ν i ( d x ) .
Moreover, because g g g and f i f_i f i are nonnegative,
∥ g − f i ∥ 1 = ∫ B ϵ ∣ g ( x ) − f i ( x ) ∣ ν i ( d x ) + ∫ B ϵ c ∣ g ( x ) − f i ( x ) ∣ ν i ( d x ) ≤ ∫ B ϵ ∣ g ( x ) − f i ( x ) ∣ ν i ( d x ) + ∫ B ϵ c g ( x ) ν i ( d x ) + ∫ B ϵ c f i ( x ) ν i ( d x ) ≤ 2 ∫ B ϵ ∣ g ( x ) − f i ( x ) ∣ ν i ( d x ) + 2 ∫ B ϵ c f i ( x ) ν i ( d x ) . \begin{aligned}
\|g-f_i\|_1
&=\int_{B_\epsilon}|g(x)-f_i(x)|\,\nu_i(dx)+\int_{B_\epsilon^c}|g(x)-f_i(x)|\,\nu_i(dx)\\
&\le\int_{B_\epsilon}|g(x)-f_i(x)|\,\nu_i(dx)+\int_{B_\epsilon^c}g(x)\,\nu_i(dx)+\int_{B_\epsilon^c}f_i(x)\,\nu_i(dx)\\
&\le2\int_{B_\epsilon}|g(x)-f_i(x)|\,\nu_i(dx)+2\int_{B_\epsilon^c}f_i(x)\,\nu_i(dx).
\end{aligned} ∥ g − f i ∥ 1 = ∫ B ϵ ∣ g ( x ) − f i ( x ) ∣ ν i ( d x ) + ∫ B ϵ c ∣ g ( x ) − f i ( x ) ∣ ν i ( d x ) ≤ ∫ B ϵ ∣ g ( x ) − f i ( x ) ∣ ν i ( d x ) + ∫ B ϵ c g ( x ) ν i ( d x ) + ∫ B ϵ c f i ( x ) ν i ( d x ) ≤ 2 ∫ B ϵ ∣ g ( x ) − f i ( x ) ∣ ν i ( d x ) + 2 ∫ B ϵ c f i ( x ) ν i ( d x ) .
Since ν i ( B ϵ ) < ∞ \nu_i(B_\epsilon)<\infty ν i ( B ϵ ) < ∞ , by Holder’s inequality,
∫ B ϵ ∣ g ( x ) − f i ( x ) ∣ ν i ( d x ) ≤ ν i ( B ϵ ) 1 − 1 / p ∥ g − f i ∥ p . \int_{B_\epsilon}|g(x)-f_i(x)|\,\nu_i(dx)\le\nu_i(B_\epsilon)^{1-1/p}\|g-f_i\|_p. ∫ B ϵ ∣ g ( x ) − f i ( x ) ∣ ν i ( d x ) ≤ ν i ( B ϵ ) 1 − 1/ p ∥ g − f i ∥ p .
Therefore,
∥ g − f i ∥ 1 ≤ 2 ν i ( B ϵ ) 1 − 1 / p ∥ g − f i ∥ p + ϵ 2 . \|g-f_i\|_1\le2\nu_i(B_\epsilon)^{1-1/p}\|g-f_i\|_p+\frac{\epsilon}{2}. ∥ g − f i ∥ 1 ≤ 2 ν i ( B ϵ ) 1 − 1/ p ∥ g − f i ∥ p + 2 ϵ .
If ∥ g − f i ∥ 1 > ϵ \|g-f_i\|_1>\epsilon ∥ g − f i ∥ 1 > ϵ , it follows that
2 ν i ( B ϵ ) 1 − 1 / p ∥ g − f i ∥ p > ϵ 2 . 2\nu_i(B_\epsilon)^{1-1/p}\|g-f_i\|_p>\frac{\epsilon}{2}. 2 ν i ( B ϵ ) 1 − 1/ p ∥ g − f i ∥ p > 2 ϵ .
Proof of Theorem 9.2: We fix i ∈ [ M ] i\in[M] i ∈ [ M ] , and ϵ > 0 \epsilon>0 ϵ > 0 . By Lemma C.1, there is ϵ ′ > 0 \epsilon'>0 ϵ ′ > 0 that depends only on ϵ \epsilon ϵ , p p p and f i f_i f i , such that
P f ( ∫ R ∣ f i ( x ) − f ^ i , n ( x ) ∣ d x ≥ ϵ , π i R ( n ) ≥ ρ ) ≤ P f ( ∫ R ∣ f i ( x ) − f ^ i , n ( x ) ∣ p d x ≥ ϵ ′ , π i R ( n ) ≥ ρ ) . \mathbb{P}_f\left(\int_{\mathbb{R}}|f_i(x)-\widehat f_{i,n}(x)|\,dx\ge\epsilon,\ \pi_i^R(n)\ge\rho\right)
\le
\mathbb{P}_f\left(\int_{\mathbb{R}}|f_i(x)-\widehat f_{i,n}(x)|^p\,dx\ge\epsilon',\ \pi_i^R(n)\ge\rho\right). P f ( ∫ R ∣ f i ( x ) − f i , n ( x ) ∣ d x ≥ ϵ , π i R ( n ) ≥ ρ ) ≤ P f ( ∫ R ∣ f i ( x ) − f i , n ( x ) ∣ p d x ≥ ϵ ′ , π i R ( n ) ≥ ρ ) .
Thus, in order to prove the claim it suffices to show that
∑ n = 1 ∞ n P f ( ∫ R ∣ f i ( x ) − f ^ i , n ( x ) ∣ p d x ≥ ϵ ′ , π i R ( n ) ≥ ρ ) < ∞ , ∀ i ∈ [ M ] . (96) \sum_{n=1}^{\infty} n\mathbb{P}_f\left(\int_{\mathbb{R}}|f_i(x)-\widehat f_{i,n}(x)|^p\,dx\ge\epsilon',\ \pi_i^R(n)\ge\rho\right)<\infty,\quad\forall i\in[M]. \tag*{(96)} n = 1 ∑ ∞ n P f ( ∫ R ∣ f i ( x ) − f i , n ( x ) ∣ p d x ≥ ϵ ′ , π i R ( n ) ≥ ρ ) < ∞ , ∀ i ∈ [ M ] . (96)
By union bound
P f ( ∫ R ∣ f i ( x ) − f ^ i , n ( x ) ∣ p d x ≥ ϵ ′ , N i R ( n ) ≥ ρ n ) ≤ ∑ m = ⌈ ρ n ⌉ n P f ( ∫ R ∣ f i ( x ) − f ^ i , n ( x ) ∣ p d x ≥ ϵ ′ , N i R ( n ) = m ) . (97) \mathbb{P}_f\left(\int_{\mathbb{R}}|f_i(x)-\widehat f_{i,n}(x)|^p\,dx\ge\epsilon',\ N_i^R(n)\ge\rho n\right)
\le
\sum_{m=\lceil\rho n\rceil}^{n}\mathbb{P}_f\left(\int_{\mathbb{R}}|f_i(x)-\widehat f_{i,n}(x)|^p\,dx\ge\epsilon',\ N_i^R(n)=m\right). \tag*{(97)} P f ( ∫ R ∣ f i ( x ) − f i , n ( x ) ∣ p d x ≥ ϵ ′ , N i R ( n ) ≥ ρ n ) ≤ m = ⌈ ρ n ⌉ ∑ n P f ( ∫ R ∣ f i ( x ) − f i , n ( x ) ∣ p d x ≥ ϵ ′ , N i R ( n ) = m ) . (97)
and by Markov’s inequality
P f ( ∫ R ∣ f i ( x ) − f ^ i , n ( x ) ∣ p d x ≥ ϵ ′ , N i R ( n ) = m ) ≤ 1 ϵ ′ E f [ ∫ R ∣ f ^ i , n ( x ) − f i ( x ) ∣ p d x ; N i R ( n ) = m ] . \mathbb{P}_f\left(\int_{\mathbb{R}}|f_i(x)-\widehat f_{i,n}(x)|^p\,dx\ge\epsilon',\ N_i^R(n)=m\right)
\le\frac{1}{\epsilon'}\mathbb{E}_f\left[\int_{\mathbb{R}}|\widehat f_{i,n}(x)-f_i(x)|^p\,dx;\ N_i^R(n)=m\right]. P f ( ∫ R ∣ f i ( x ) − f i , n ( x ) ∣ p d x ≥ ϵ ′ , N i R ( n ) = m ) ≤ ϵ ′ 1 E f [ ∫ R ∣ f i , n ( x ) − f i ( x ) ∣ p d x ; N i R ( n ) = m ] .
In order to prove the claim, it suffices to show that there exists a constant D 0 > 0 D_0>0 D 0 > 0 such that
E f [ ∫ R ∣ f ^ i , n ( x ) − f i ( x ) ∣ p d x ; N i R ( n ) = m ] ≤ D 0 ( 1 ( n h m ) p / 2 + h m 2 p ) . (98) \mathbb{E}_f\left[\int_{\mathbb{R}}|\hat{f}_{i,n}(x)-f_i(x)|^p\,dx;N_i^R(n)=m\right]\leq D_0\left(\frac{1}{(nh_m)^{p/2}}+h_m^{2p}\right). \tag*{(98)} E f [ ∫ R ∣ f ^ i , n ( x ) − f i ( x ) ∣ p d x ; N i R ( n ) = m ] ≤ D 0 ( ( n h m ) p /2 1 + h m 2 p ) . (98)
According to [40 Theorem 24.1] we choose the function h h h as h m : = m − 1 / 5 h_m:=m^{-1/5} h m := m − 1/5 . Therefore, η i , n : = ( N i R ( n ) ) − 1 / 5 \eta_{i,n}:=(N_i^R(n))^{-1/5} η i , n := ( N i R ( n ) ) − 1/5 and on the event { N i R ( n ) = m } \{N_i^R(n)=m\} { N i R ( n ) = m } we have η i , n = h m = m − 1 / 5 \eta_{i,n}=h_m=m^{-1/5} η i , n = h m = m − 1/5 . Since ρ n ≤ m ≤ n \rho n\leq m\leq n ρ n ≤ m ≤ n , it holds
n − 1 / 5 ≤ h m = m − 1 / 5 ≤ ( ρ n ) − 1 / 5 . n^{-1/5}\leq h_m=m^{-1/5}\leq(\rho n)^{-1/5}. n − 1/5 ≤ h m = m − 1/5 ≤ ( ρ n ) − 1/5 .
and as a result
( 1 n p / 2 h m p / 2 + h m 2 p ) ≤ ( 1 n 2 p / 5 + 1 ρ 2 p / 5 n 2 p / 5 ) = ( 1 + 1 ρ 2 p / 5 ) 1 n 2 p / 5 . \left(\frac{1}{n^{p/2}h_m^{p/2}}+h_m^{2p}\right)\leq\left(\frac{1}{n^{2p/5}}+\frac{1}{\rho^{2p/5}n^{2p/5}}\right)=\left(1+\frac{1}{\rho^{2p/5}}\right)\frac{1}{n^{2p/5}}. ( n p /2 h m p /2 1 + h m 2 p ) ≤ ( n 2 p /5 1 + ρ 2 p /5 n 2 p /5 1 ) = ( 1 + ρ 2 p /5 1 ) n 2 p /5 1 .
Since the upper bound is independent of m m m , by (97) we deduce that
P f ( ∫ R ∣ f i ( x ) − f ^ i , n ( x ) ∣ p d x ≥ ϵ ′ , N i R ( n ) ≥ ρ n ) ≤ ( 1 + 1 ρ 2 p / 5 ) ( 1 − ρ ) 1 n ( 2 p − 5 ) / 5 , \mathbb{P}_f\left(\int_{\mathbb{R}}|f_i(x)-\hat{f}_{i,n}(x)|^pdx\geq\epsilon',\,N_i^R(n)\geq\rho n\right)\leq\left(1+\frac{1}{\rho^{2p/5}}\right)(1-\rho)\frac{1}{n^{(2p-5)/5}}, P f ( ∫ R ∣ f i ( x ) − f ^ i , n ( x ) ∣ p d x ≥ ϵ ′ , N i R ( n ) ≥ ρ n ) ≤ ( 1 + ρ 2 p /5 1 ) ( 1 − ρ ) n ( 2 p − 5 ) /5 1 ,
and thus for p > 15 / 2 p>15/2 p > 15/2 we prove claim (96) .
In order to prove (97), we write
f ^ i , n ( x ) − f i ( x ) = ( f ^ i , n ( x ) − f ˉ n , m ) + ( f ˉ n , m − f i ( x ) ) , \hat{f}_{i,n}(x)-f_i(x)=(\hat{f}_{i,n}(x)-\bar{f}_{n,m})+(\bar{f}_{n,m}-f_i(x)), f ^ i , n ( x ) − f i ( x ) = ( f ^ i , n ( x ) − f ˉ n , m ) + ( f ˉ n , m − f i ( x )) ,
where
f ˉ n , m : = 1 m h m ∑ k = 1 n R i ( k ) E f [ K ( x − X i ( k ) h m ) ] , \bar{f}_{n,m}:=\frac{1}{mh_m}\sum_{k=1}^{n}R_i(k)\mathbb{E}_f\left[K\left(\frac{x-X_i(k)}{h_m}\right)\right], f ˉ n , m := m h m 1 k = 1 ∑ n R i ( k ) E f [ K ( h m x − X i ( k ) ) ] ,
and by convexity, we have
E f [ ∣ f ^ i , n ( x ) − f i ( x ) ∣ p ; N i R ( n ) = m ] ≤ 2 p − 1 E f [ ∣ f ^ i , n ( x ) − f ˉ n , m ∣ p ; N i R ( n ) = m ] \mathbb{E}_f\left[|\hat{f}_{i,n}(x)-f_i(x)|^p;N_i^R(n)=m\right]\leq2^{p-1}\mathbb{E}_f\left[|\hat{f}_{i,n}(x)-\bar{f}_{n,m}|^p;N_i^R(n)=m\right] E f [ ∣ f ^ i , n ( x ) − f i ( x ) ∣ p ; N i R ( n ) = m ] ≤ 2 p − 1 E f [ ∣ f ^ i , n ( x ) − f ˉ n , m ∣ p ; N i R ( n ) = m ]
+ 2 p − 1 E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] . \qquad+2^{p-1}\mathbb{E}_f\left[|\bar{f}_{n,m}-f_i(x)|^p;N_i^R(n)=m\right]. + 2 p − 1 E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] .
By Tonelli’s theorem
E f [ ∫ R ∣ f ^ i , n ( x ) − f i ( x ) ∣ p d x ; N i R ( n ) = m ] = ∫ R E f [ ∣ f ^ i , n ( x ) − f i ( x ) ∣ p ; N i R ( n ) = m ] d x \mathbb{E}_f\left[\int_{\mathbb{R}}|\hat{f}_{i,n}(x)-f_i(x)|^pdx;N_i^R(n)=m\right]=\int_{\mathbb{R}}\mathbb{E}_f\left[|\hat{f}_{i,n}(x)-f_i(x)|^p;N_i^R(n)=m\right]\,dx E f [ ∫ R ∣ f ^ i , n ( x ) − f i ( x ) ∣ p d x ; N i R ( n ) = m ] = ∫ R E f [ ∣ f ^ i , n ( x ) − f i ( x ) ∣ p ; N i R ( n ) = m ] d x
≤ 2 p − 1 ∫ R E f [ ∣ f ^ i , n ( x ) − f ˉ n , m ∣ p ; N i R ( n ) = m ] d x \leq2^{p-1}\int_{\mathbb{R}}\mathbb{E}_f\left[|\hat{f}_{i,n}(x)-\bar{f}_{n,m}|^p;N_i^R(n)=m\right]\,dx ≤ 2 p − 1 ∫ R E f [ ∣ f ^ i , n ( x ) − f ˉ n , m ∣ p ; N i R ( n ) = m ] d x
+ 2 p − 1 ∫ R E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] d x , \qquad+2^{p-1}\int_{\mathbb{R}}\mathbb{E}_f\left[|\bar{f}_{n,m}-f_i(x)|^p;N_i^R(n)=m\right]\,dx, + 2 p − 1 ∫ R E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] d x ,
where the first term on the right hand side is referred to as the stochastic term , while the second one is the bias term .
We first bound the bias term . Since { X i ( n ) : n ∈ N } \{X_i(n):n\in\mathbb{N}\} { X i ( n ) : n ∈ N } are iid, we get that on the event { N i R ( n ) = m } \{N_i^R(n)=m\} { N i R ( n ) = m } ,
E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] = E f [ ∣ 1 h m E f [ K ( x − X i ( 1 ) h m ) ] − f i ( x ) ∣ p ; N i R ( n ) = m ] ≤ ∣ 1 h m E f [ K ( x − X i ( 1 ) h m ) ] − f i ( x ) ∣ p . \begin{aligned}
\mathbb{E}_f\left[|\bar{f}_{n,m}-f_i(x)|^p;N_i^R(n)=m\right]=\mathbb{E}_f\left[\left|\frac{1}{h_m}\mathbb{E}_f\left[K\left(\frac{x-X_i(1)}{h_m}\right)\right]-f_i(x)\right|^p;N_i^R(n)=m\right] \\
&\leq\left|\frac{1}{h_m}\mathbb{E}_f\left[K\left(\frac{x-X_i(1)}{h_m}\right)\right]-f_i(x)\right|^p.
\end{aligned} E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] = E f [ h m 1 E f [ K ( h m x − X i ( 1 ) ) ] − f i ( x ) p ; N i R ( n ) = m ] ≤ h m 1 E f [ K ( h m x − X i ( 1 ) ) ] − f i ( x ) p .
We have
1 h m E f [ K ( x − X i ( 1 ) h m ) ] = 1 h m ∫ R K ( x − y h m ) f i ( y ) d y = ∫ R K ( u ) f i ( x − h m u ) d u . \frac{1}{h_m}\mathbb{E}_f\left[K\left(\frac{x-X_i(1)}{h_m}\right)\right]=\frac{1}{h_m}\int_{\mathbb{R}}K\left(\frac{x-y}{h_m}\right)f_i(y)\,dy=\int_{\mathbb{R}}K(u)f_i(x-h_m u)\,du. h m 1 E f [ K ( h m x − X i ( 1 ) ) ] = h m 1 ∫ R K ( h m x − y ) f i ( y ) d y = ∫ R K ( u ) f i ( x − h m u ) d u .
Since K K K is a density,
1 h m E f [ K ( x − X i ( 1 ) h m ) ] − f i ( x ) = ∫ R K ( u ) ( f i ( x − h m u ) − f i ( x ) ) d u . \frac{1}{h_m}\mathbb{E}_f\left[K\left(\frac{x-X_i(1)}{h_m}\right)\right]-f_i(x)=\int_{\mathbb{R}}K(u)\left(f_i(x-h_m u)-f_i(x)\right)\,du. h m 1 E f [ K ( h m x − X i ( 1 ) ) ] − f i ( x ) = ∫ R K ( u ) ( f i ( x − h m u ) − f i ( x ) ) d u .
Using Taylor’s expansion with the integral form of the remainder [40 Theorem 24.1] ,
f i ( x − h m u ) − f i ( x ) = − h m u f i ′ ( x ) + h m 2 u 2 ∫ 0 1 f i ′ ′ ( x − t h m u ) ( 1 − t ) d t , f_i(x-h_m u)-f_i(x)=-h_m u f_i'(x)+h_m^2u^2\int_0^1 f_i''(x-th_m u)(1-t)\,dt, f i ( x − h m u ) − f i ( x ) = − h m u f i ′ ( x ) + h m 2 u 2 ∫ 0 1 f i ′′ ( x − t h m u ) ( 1 − t ) d t ,
which implies
1 h m E f [ K ( x − X i ( 1 ) h m ) ] − f i ( x ) = − h m f i ′ ( x ) ∫ R u K ( u ) d u + h m 2 ∫ R u 2 K ( u ) ∫ 0 1 f i ′ ′ ( x − t h m u ) ( 1 − t ) d t d u = h m 2 ∫ R u 2 K ( u ) ∫ 0 1 f i ′ ′ ( x − t h m u ) ( 1 − t ) d t d u , \begin{aligned}
\frac{1}{h_m}\mathbb{E}_f\left[K\left(\frac{x-X_i(1)}{h_m}\right)\right]-f_i(x)
&=-h_m f_i'(x)\int_{\mathbb{R}}uK(u)\,du+h_m^2\int_{\mathbb{R}}u^2K(u)\int_0^1 f_i''(x-th_m u)(1-t)\,dt\,du\\
&=h_m^2\int_{\mathbb{R}}u^2K(u)\int_0^1 f_i''(x-th_m u)(1-t)\,dt\,du,
\end{aligned} h m 1 E f [ K ( h m x − X i ( 1 ) ) ] − f i ( x ) = − h m f i ′ ( x ) ∫ R u K ( u ) d u + h m 2 ∫ R u 2 K ( u ) ∫ 0 1 f i ′′ ( x − t h m u ) ( 1 − t ) d t d u = h m 2 ∫ R u 2 K ( u ) ∫ 0 1 f i ′′ ( x − t h m u ) ( 1 − t ) d t d u ,
where the second equality follows by assumption (1). Considering the L p L^p L p -norm both sides, by Minkowski’s inequality [50 Theorem 202] we have
∥ 1 h m E f [ K ( ⋅ − X i ( 1 ) h m ) ] − f i ∥ p ≤ h m 2 ∫ R u 2 ∣ K ( u ) ∣ ∫ 0 1 ( 1 − t ) ∥ f i ′ ′ ( ⋅ − t h m u ) ∥ p d t d u . \left\|\frac{1}{h_m}\mathbb{E}_f\left[K\left(\frac{\cdot-X_i(1)}{h_m}\right)\right]-f_i\right\|_p\le h_m^2\int_{\mathbb{R}}u^2|K(u)|\int_0^1(1-t)\left\|f_i''(\cdot-th_m u)\right\|_p\,dt\,du. h m 1 E f [ K ( h m ⋅ − X i ( 1 ) ) ] − f i p ≤ h m 2 ∫ R u 2 ∣ K ( u ) ∣ ∫ 0 1 ( 1 − t ) ∥ f i ′′ ( ⋅ − t h m u ) ∥ p d t d u .
By translation invariance of the L p L^p L p norm,
∥ f i ′ ′ ( ⋅ − t h m u ) ∥ p = ∥ f i ′ ′ ∥ p , \left\|f_i''(\cdot-th_m u)\right\|_p=\left\|f_i''\right\|_p, ∥ f i ′′ ( ⋅ − t h m u ) ∥ p = ∥ f i ′′ ∥ p ,
and thus by assumptions (2) and (b) there exists a constant D 1 > 0 D_1>0 D 1 > 0 such that
∫ R E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] d x ≤ ∥ 1 h m E f [ K ( ⋅ − X i ( 1 ) h m ) ] − f i ∥ p p ≤ D 1 h m 2 p . (99) \int_{\mathbb{R}} \mathbb{E}_f\left[|\bar f_{n,m}-f_i(x)|^p;N_i^R(n)=m\right]\,dx\le\left\|\frac{1}{h_m}\mathbb{E}_f\left[K\left(\frac{\cdot-X_i(1)}{h_m}\right)\right]-f_i\right\|_p^p\le D_1h_m^{2p}. \tag*{(99)} ∫ R E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] d x ≤ h m 1 E f [ K ( h m ⋅ − X i ( 1 ) ) ] − f i p p ≤ D 1 h m 2 p . (99)
Next, we bound the stochastic term. For this, we set
Y i ( x , k ) : = 1 h m K ( x − X i ( k ) h m ) − 1 h m E f [ K ( x − X i ( k ) h m ) ] , (100) Y_i(x,k):=\frac{1}{h_m}K\left(\frac{x-X_i(k)}{h_m}\right)-\frac{1}{h_m}\mathbb{E}_f\left[K\left(\frac{x-X_i(k)}{h_m}\right)\right], \tag*{(100)} Y i ( x , k ) := h m 1 K ( h m x − X i ( k ) ) − h m 1 E f [ K ( h m x − X i ( k ) ) ] , (100)
and
S n : = ∑ k = 1 n R i ( k ) Y i ( x , k ) . S_n:=\sum_{k=1}^{n}R_i(k)Y_i(x,k). S n := k = 1 ∑ n R i ( k ) Y i ( x , k ) .
Therefore,
E f [ ∣ f ^ n , i ( x ) − f ˉ n , m ∣ p ; N i R ( n ) = m ] = 1 m p E f [ ∣ S n ∣ p ; N i R ( n ) = m ] ≤ 1 m p E f [ ∣ S n ∣ p ] . \mathbb{E}_f\left[|\hat f_{n,i}(x)-\bar f_{n,m}|^p;N_i^R(n)=m\right]=\frac{1}{m^p}\mathbb{E}_f\left[|S_n|^p;N_i^R(n)=m\right]\le\frac{1}{m^p}\mathbb{E}_f[|S_n|^p]. E f [ ∣ f ^ n , i ( x ) − f ˉ n , m ∣ p ; N i R ( n ) = m ] = m p 1 E f [ ∣ S n ∣ p ; N i R ( n ) = m ] ≤ m p 1 E f [ ∣ S n ∣ p ] .
Since R i ( n ) ∈ F n − 1 R R_i(n)\in\mathcal{F}_{n-1}^R R i ( n ) ∈ F n − 1 R , the { S n : n ∈ N } \{S_n:n\in\mathbb{N}\} { S n : n ∈ N } is a martingale with respect to F n R \mathcal{F}_n^R F n R . By Rosenthal’s inequality [49 Theorem 2.12] , there exists a constant D 2 > 0 D_2>0 D 2 > 0 such that
E f [ ∣ S n ∣ p ] ≤ D 2 { n p / 2 ( E f [ ∣ Y i ( x , 1 ) ∣ 2 ] ) p / 2 + n E f [ ∣ Y i ( x , 1 ) ∣ p ] } . (101) \mathbb{E}_f[|S_n|^p]\le D_2\left\{n^{p/2}\left(\mathbb{E}_f[|Y_i(x,1)|^2]\right)^{p/2}+n\mathbb{E}_f[|Y_i(x,1)|^p]\right\}. \tag*{(101)} E f [ ∣ S n ∣ p ] ≤ D 2 { n p /2 ( E f [ ∣ Y i ( x , 1 ) ∣ 2 ] ) p /2 + n E f [ ∣ Y i ( x , 1 ) ∣ p ] } . (101)
and thus
∫ R E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] d x ≤ D 2 m p [ n p / 2 ∫ R ( E f [ ∣ Y i ( x , 1 ) ∣ 2 ] ) p / 2 d x + n ∫ R E f [ ∣ Y i ( x , 1 ) ∣ p ] d x ] . (102) \int_{\mathbb{R}} \mathbb{E}_{f}\left[|\bar{f}_{n,m}-f_i(x)|^p;N_i^{R}(n)=m\right]\,dx \leq\frac{D_2}{m^p}\left[n^{p/2}\int_{\mathbb{R}}\left(\mathbb{E}_{f}[|Y_i(x,1)|^2]\right)^{p/2}\,dx+n\int_{\mathbb{R}}\mathbb{E}_{f}[|Y_i(x,1)|^p]\,dx\right]. \tag*{(102)} ∫ R E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] d x ≤ m p D 2 [ n p /2 ∫ R ( E f [ ∣ Y i ( x , 1 ) ∣ 2 ] ) p /2 d x + n ∫ R E f [ ∣ Y i ( x , 1 ) ∣ p ] d x ] . (102)
By definition of Y i ( x , 1 ) Y_i(x,1) Y i ( x , 1 ) in (100) , E f [ ∣ Y i ( x , 1 ) ∣ 2 ] \mathbb{E}_{f}[|Y_i(x,1)|^2] E f [ ∣ Y i ( x , 1 ) ∣ 2 ] is the variance of
1 h m K ( x − X i ( 1 ) h m ) , \frac{1}{h_m}K\left(\frac{x-X_i(1)}{h_m}\right), h m 1 K ( h m x − X i ( 1 ) ) ,
and thus
E f [ ∣ Y i ( x , 1 ) ∣ 2 ] ≤ E f [ 1 h m 2 K 2 ( x − X i ( 1 ) h m ) ] = 1 h m ∫ R K 2 ( u ) f i ( x − h m u ) d u . \mathbb{E}_{f}[|Y_i(x,1)|^2]\leq\mathbb{E}_{f}\left[\frac{1}{h_m^2}K^2\left(\frac{x-X_i(1)}{h_m}\right)\right]=\frac{1}{h_m}\int_{\mathbb{R}}K^2(u)f_i(x-h_m u)\,du. E f [ ∣ Y i ( x , 1 ) ∣ 2 ] ≤ E f [ h m 2 1 K 2 ( h m x − X i ( 1 ) ) ] = h m 1 ∫ R K 2 ( u ) f i ( x − h m u ) d u .
Using Minkowski’s inequality,
∥ E f [ ∣ Y i ( x , 1 ) ∣ 2 ] ∥ p / 2 ≤ 1 h m ∫ R K 2 ( u ) ∥ f i ( ⋅ − h m u ) ∥ p / 2 d u , \left\|\mathbb{E}_{f}[|Y_i(x,1)|^2]\right\|_{p/2}\leq\frac{1}{h_m}\int_{\mathbb{R}}K^2(u)\left\|f_i(\cdot-h_m u)\right\|_{p/2}\,du, E f [ ∣ Y i ( x , 1 ) ∣ 2 ] p /2 ≤ h m 1 ∫ R K 2 ( u ) ∥ f i ( ⋅ − h m u ) ∥ p /2 d u ,
and by translation invariance of the L p / 2 L^{p/2} L p /2 norm, and assumptions (3) and (a) we deduce that there exists a constant D 3 > 0 D_3>0 D 3 > 0 such that
∫ R ( E f [ ∣ Y i ( x , 1 ) ∣ 2 ] ) p / 2 d x = ∥ E f [ ∣ Y i ( x , 1 ) ∣ 2 ] ∥ p / 2 p / 2 ≤ D 3 h m p / 2 . \int_{\mathbb{R}}\left(\mathbb{E}_{f}[|Y_i(x,1)|^2]\right)^{p/2}\,dx=\left\|\mathbb{E}_{f}[|Y_i(x,1)|^2]\right\|_{p/2}^{p/2}\leq\frac{D_3}{h_m^{p/2}}. ∫ R ( E f [ ∣ Y i ( x , 1 ) ∣ 2 ] ) p /2 d x = E f [ ∣ Y i ( x , 1 ) ∣ 2 ] p /2 p /2 ≤ h m p /2 D 3 .
For the second integral in (102) , by convexity we have
E f [ ∣ Y i ( x , 1 ) ∣ p ] ≤ 2 p − 1 E f [ ∣ 1 h m p K ( x − X i ( 1 ) h m ) ∣ p ] + 2 p − 1 ∣ E f [ 1 h m K ( x − X i ( 1 ) h m ) ] ∣ p , \mathbb{E}_{f}[|Y_i(x,1)|^p]\leq2^{p-1}\mathbb{E}_{f}\left[\left|\frac{1}{h_m^p}K\left(\frac{x-X_i(1)}{h_m}\right)\right|^p\right]+2^{p-1}\left|\mathbb{E}_{f}\left[\frac{1}{h_m}K\left(\frac{x-X_i(1)}{h_m}\right)\right]\right|^p, E f [ ∣ Y i ( x , 1 ) ∣ p ] ≤ 2 p − 1 E f [ h m p 1 K ( h m x − X i ( 1 ) ) p ] + 2 p − 1 E f [ h m 1 K ( h m x − X i ( 1 ) ) ] p ,
and since p > 1 p>1 p > 1 by Holder’s inequality
∣ E f [ 1 h m K ( x − X i ( 1 ) h m ) ] ∣ p ≤ E f [ 1 h m p ∣ K ( x − X i ( 1 ) h m ) ∣ p ] , \left|\mathbb{E}_{f}\left[\frac{1}{h_m}K\left(\frac{x-X_i(1)}{h_m}\right)\right]\right|^p\leq\mathbb{E}_{f}\left[\frac{1}{h_m^p}\left|K\left(\frac{x-X_i(1)}{h_m}\right)\right|^p\right], E f [ h m 1 K ( h m x − X i ( 1 ) ) ] p ≤ E f [ h m p 1 K ( h m x − X i ( 1 ) ) p ] ,
which gives
E f [ ∣ Y i ( x , 1 ) ∣ p ] ≤ 2 p E f [ 1 h m p ∣ K ( x − X i ( 1 ) h m ) ∣ p ] . \mathbb{E}_{f}[|Y_i(x,1)|^p]\leq2^p\mathbb{E}_{f}\left[\frac{1}{h_m^p}\left|K\left(\frac{x-X_i(1)}{h_m}\right)\right|^p\right]. E f [ ∣ Y i ( x , 1 ) ∣ p ] ≤ 2 p E f [ h m p 1 K ( h m x − X i ( 1 ) ) p ] .
Therefore,
∫ R E f [ ∣ Y i ( x , 1 ) ∣ p ] d x ≤ 2 p E f [ ∫ R 1 h m p ∣ K ( x − X i ( 1 ) h m ) ∣ p d x ] = 2 p ∫ R ∫ R 1 h m p ∣ K ( x − y h m ) ∣ p d x f i ( y ) d y . \begin{aligned}
\int_{\mathbb{R}}\mathbb{E}_{f}[|Y_i(x,1)|^p]\,dx\leq2^p\mathbb{E}_{f}\left[\int_{\mathbb{R}}\frac{1}{h_m^p}\left|K\left(\frac{x-X_i(1)}{h_m}\right)\right|^p\,dx\right] \\
&=2^p\int_{\mathbb{R}}\int_{\mathbb{R}}\frac{1}{h_m^p}\left|K\left(\frac{x-y}{h_m}\right)\right|^p\,dx\,f_i(y)\,dy.
\end{aligned} ∫ R E f [ ∣ Y i ( x , 1 ) ∣ p ] d x ≤ 2 p E f [ ∫ R h m p 1 K ( h m x − X i ( 1 ) ) p d x ] = 2 p ∫ R ∫ R h m p 1 K ( h m x − y ) p d x f i ( y ) d y .
For fixed y y y , we do the change of variables u = ( x − y ) / h m u=(x-y)/h_m u = ( x − y ) / h m , and
∫ R 1 h m p ∣ K ( x − y h m ) ∣ p d x = 1 h m p − 1 ∫ R ∣ K ( u ) ∣ p d u . \int_{\mathbb{R}}\frac{1}{h_m^p}\left|K\left(\frac{x-y}{h_m}\right)\right|^p\,dx=\frac{1}{h_m^{p-1}}\int_{\mathbb{R}}|K(u)|^p\,du. ∫ R h m p 1 K ( h m x − y ) p d x = h m p − 1 1 ∫ R ∣ K ( u ) ∣ p d u .
Hence, by the assumption (3), there exists a constant D 4 > 0 D_4>0 D 4 > 0 such that
∫ R E f [ ∣ Y i ( x , 1 ) ∣ p ] d x ≤ D 4 1 h m p − 1 . \int_{\mathbb{R}}\mathbb{E}_{f}[|Y_i(x,1)|^p]\,dx\leq D_4\frac{1}{h_m^{p-1}}. ∫ R E f [ ∣ Y i ( x , 1 ) ∣ p ] d x ≤ D 4 h m p − 1 1 .
As a result, by (102)
∫ R E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] d x ≤ D 2 m p [ n p / 2 D 3 h m p / 2 + n D 4 h m p − 1 ] ≤ D 2 ρ p [ D 3 n p / 2 h m p / 2 + D 4 n p − 1 h m p − 1 ] . \int_{\mathbb{R}} \mathbb{E}_f\left[|\bar{f}_{n,m}-f_i(x)|^p;N_i^{R}(n)=m\right]\,dx \le\frac{D_2}{m^p}\left[n^{p/2}\frac{D_3}{h_m^{p/2}}+n\frac{D_4}{h_m^{p-1}}\right]
\le\frac{D_2}{\rho^p}\left[\frac{D_3}{n^{p/2}h_m^{p/2}}+\frac{D_4}{n^{p-1}h_m^{p-1}}\right]. ∫ R E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] d x ≤ m p D 2 [ n p /2 h m p /2 D 3 + n h m p − 1 D 4 ] ≤ ρ p D 2 [ n p /2 h m p /2 D 3 + n p − 1 h m p − 1 D 4 ] .
where for the last inequality we used the fact that ρ n ≤ m \rho n \le m ρ n ≤ m . Since p > 2 p > 2 p > 2 and 1 ≤ n h m 1 \le nh_m 1 ≤ n h m ,
1 ( n h m ) p − 1 ≤ 1 ( n h m ) p / 2 . \frac{1}{(nh_m)^{p-1}} \le\frac{1}{(nh_m)^{p/2}}. ( n h m ) p − 1 1 ≤ ( n h m ) p /2 1 .
and thus there exists D 5 > 0 D_5 > 0 D 5 > 0 such that
∫ R E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] d x ≤ D 5 ( n h m ) p / 2 . (103) \int_{\mathbb{R}} \mathbb{E}_f\left[|\bar{f}_{n,m}-f_i(x)|^p;N_i^{R}(n)=m\right]\,dx \le\frac{D_5}{(nh_m)^{p/2}}. \tag*{(103)} ∫ R E f [ ∣ f ˉ n , m − f i ( x ) ∣ p ; N i R ( n ) = m ] d x ≤ ( n h m ) p /2 D 5 . (103)
Combining (99) and (103) , we show (98) .