betaFeedback

WorksMathematics

Risk-optimal active learning with informative abstention

Abstract

We study label acquisition when an oracle may abstain and its response probability shares parameters with the classifier. A query can then provide information through both a returned label and the response outcome. Under a correctly specified parametric model, we derive the classifier information after adjustment for response-model nuisance parameters. Combining this information with the local curvature of multiclass zero-one risk gives an acquisition criterion for minimizing the leading asymptotic excess risk. We characterize the optimal allocation under a fixed query budget and show how query allocation affects response identifiability. The oracle criterion reduces to response-discounted acquisition when response probability does not depend on the classifier parameters. Simulations examine the resulting query allocations and isolate the contribution of the response-outcome term through an ablation. A Consumer Complaints replay retains observed human abstentions and reveals benchmark labels upon response. Response-aware acquisition has lower mean test error than response-discounted acquisition at the two largest budgets, with paired differences significant after Holm correction. Entropy sampling performs better at the smallest budget. The gains depend on the response mechanism and query budget.

Acknowledgements

We are grateful to Cuong V. Nguyen of the Department of Mathematical Sciences at Durham University, UK, for generously taking the time to read the manuscript and for his thoughtful comments. His observations on adaptive sampling helped us clarify the assumptions and proof of the two-stage acquisition result.

Introduction

Obtaining a class label may require a human annotator, expert assessment, diagnostic procedure, or user interaction. A query does not always produce a label: an annotator may decline an ambiguous case, an expert may find the available information insufficient, or a user may not respond. This raises two questions for active learning: which observations should be queried, and how should the possibility and the observed pattern of response affect query selection?

Classical active learning allocates a limited labeling budget by selectively requesting labels ([11, 10, 19]. Uncertainty sampling prioritizes observations with ambiguous class probabilities, whereas information-based methods assess the value of a label for parameter estimation or prediction ([7, 20, 4]. Other approaches target prediction error, predictive information, or downstream decisions ([15, 3, 6, 8].

Statistical approaches to sampling provide related foundations. Wang et al. [22] and Yao and Wang [25] derive optimal subsampling procedures for logistic and softmax regression, respectively. Imberg et al. [9] study sampling schemes for prediction-error reduction in unbiased active learning, while Bach (2006) develops active learning for potentially misspecified generalized linear models. These methods address different inferential and predictive objectives. The problem considered here additionally involves a stochastic oracle response whose distribution shares parameters with the classifier.

Imperfect labeling oracles have received substantial attention. Yan et al. [24] study active learning when a labeler may return an incorrect label or abstain, obtaining query-complexity guarantees under assumptions on the noise and abstention processes. Fang and Zhu [5] consider uncertain labeling knowledge, allowing an oracle to indicate that it does not know a label. Amin et al. [2] study a different form of abstention: examples routed to a human labeler no longer require a prediction from the machine. Their algorithm jointly learns a model and decides when to request labels. In a Bayesian pool-based setting, Nguyen et al. [12] develop acquisition rules that learn both the classifier and an unknown abstention mechanism. Thus, neither allowing an oracle to abstain nor learning its response mechanism alongside the classifier is itself new.

A closely related approach is the biased-nonresponse framework of Robinson et al. [14]. Their expected-utility rule weights an ordinary acquisition utility by the estimated probability of response; an upper-confidence-bound extension accounts for uncertainty in that estimate. This addresses the reduced value of queries that are unlikely to return labels. It does not, however, derive classifier information from the realized response outcome through a shared-parameter likelihood. Likewise, Nguyen et al. [12] use separate classification and abstention models, rather than deriving the nuisance-adjusted Fisher information of a classifier-linked response mechanism and weighting it by local zero–one classification risk.

Likelihood-based work on informative label missingness motivates this distinction. Under Rubin’s framework, conditional independence of missingness and the unobserved class given the features does not by itself justify ignoring a mechanism that shares parameters with the classification model [16]. Ahfock and McLachlan [1] show that uncertainty-dependent label missingness can carry information about a discriminant rule. More recent work develops information decompositions and connects the resulting information to classification performance [23, 17]. These results concern the information available under a given missingness mechanism. Here the investigator also controls which observations are submitted to the oracle.

Our starting point is the classification-risk acquisition framework of Setoudehtazangi and McLachlan [18], in which every requested label is returned. Under its regularity conditions, leading asymptotic excess zero–one risk depends on Fisher information weighted by the local curvature HRH_{\mathrm{R}} of the active Bayes decision boundary. A label is therefore valuable according to whether its information improves estimation in directions that affect classification, rather than according to uncertainty or parameter precision alone. We consider how this criterion changes when the oracle may abstain.

Let YY denote the observed features, ZZ the class label, AA the investigator’s query indicator, and RR the oracle response indicator. Conditional on a query, we model

Pr⁡(R=1∣A=1,Y=y)=r(y;θ,η),\Pr(R = 1 \mid A = 1, Y = y) = r(y; \theta, \eta),

where θ\theta governs the classification model and η\eta contains response-specific nuisance parameters. We assume that a returned label is correct and that

R⊥Z∣(Y,A=1).R \perp Z \mid(Y, A = 1).

The acquisition rule is known and parameter-free. Conditional on the observed features, its realized query decision therefore carries no likelihood information about the model parameters. The subsequent response outcome can carry such information because its probability depends on θ\theta.

For the joint parameter ϕ=(θ⊤,η⊤)⊤\phi= (\theta^{\top}, \eta^{\top})^{\top}, a query at yy contributes

K(y)=r(y)J~(y)+r(y){1−r(y)}q(y)q(y)⊤,K(y) = r(y)\widetilde{J}(y) + r(y)\{1-r(y)\}q(y)q(y)^{\top},

where J~(y)\widetilde{J}(y) is the conditional class-label information augmented by zero nuisance blocks, and

q(y)=∇ϕlogit⁡r(y;θ,η).q(y) = \nabla_{\phi}\operatorname{logit} r(y; \theta, \eta).

The first term is information from a potentially returned label. The second is information from observing whether the oracle responds. When the response-model nuisance information is nonsingular, eliminating η\eta leaves a positive semidefinite response contribution to efficient classifier information. This contribution vanishes precisely when the classifier-related response score is explained by the nuisance response score under the design-weighted measure. Sharing parameters is therefore insufficient by itself: useful response information must survive nuisance adjustment and affect parameter directions relevant to classification.

Combining the query information with local risk curvature gives the marginal acquisition value

ψa(y)=ψaL(y)+ψaR(y),\psi_a(y) = \psi_a^L(y) + \psi_a^R(y),

where aa denotes the acquisition design. The returned-label term ψaL(y)\psi_a^L(y) accounts for the probability of receiving an informative label. The response term ψaR(y)\psi_a^R(y) accounts for the classification value of the response outcome itself. A query can consequently be informative even when no class label is returned. When response probability does not depend on the classifier parameters, the oracle criterion reduces to response-discounted acquisition.

The joint information matrix is affine in the acquisition design. The resulting optimization therefore retains the convex trace-inverse structure of a linear optimal-design criterion [13] and admits a threshold characterization under a fixed query budget. Query allocation must also preserve sufficient variation for nonsingular response-model information.

We derive this rank condition and give a two-stage oracle-value result for a pilot-selected design applied to fresh observations. The risk conclusion requires uniform criterion consistency, uniform asymptotic linearity in mean square, and a higher-moment bound. The proof conditions on the pilot and then averages over it. These conditions are not established for the sequential finite-pool implementation used in the experiments.

The simulations examine oracle and adaptive acquisition under noninformative and informative response mechanisms. An ablation retains the full information matrix and joint estimation while omitting the explicit response-outcome term from query scores. It separates the effect of that term from changes in the information matrix used by response-discounted acquisition. The observed gains are small and depend on the response mechanism and query budget.

We also use the Consumer Complaints annotations collected by Totakura et al. [21]. Each worker is treated as a separate oracle. The replay preserves observed human response and abstention patterns but reveals the benchmark class whenever a worker responds. It therefore evaluates acquisition under human abstention without modeling errors in returned labels. This comparison evaluates the complete acquisition rules; it does not isolate the explicit response-outcome term.

The contribution is a classification-risk acquisition criterion for a shared-parameter query–response model. It values response outcomes through their nuisance-adjusted information and their effect on the classification boundary. The analysis identifies conditions under which this additional information changes query selection, while the experiments examine when those changes improve classification.

§ 3 introduces the query–response model and derives its Fisher information. § 4 characterizes the efficient classifier information after nuisance adjustment. § 5 derives the classification-risk criterion and its marginal query value. Subsequent sections examine optimal allocation, response identifiability, adaptive acquisition, and a binary logistic illustration. Simulations and the Consumer Complaints replay follow, before the discussion of findings and limitations.

Statistical framework and informative oracle response

Let Y∈Y⊆RpY \in\mathcal{Y} \subseteq\mathbb{R}^{p} denote an observed feature vector and Z∈{1,…,g}Z \in\{1,\ldots,g\} its class label. We consider a regular parametric classification model

pθ(y,z)=pθ(y)τz(y;θ),τk(y;θ)=Pr⁡θ(Z=k∣Y=y),p_{\theta}(y,z) = p_{\theta}(y)\tau_{z}(y;\theta), \qquad\tau_{k}(y;\theta) = \Pr_{\theta}(Z=k \mid Y=y),

where θ∈Θ⊆Rd\theta\in\Theta\subseteq\mathbb{R}^{d} is the parameter governing the classification model. The feature vector is observed for every sampling unit. After observing Y=yY=y, the investigator may request its class label. Let

A={1,if a label is requested,0,otherwise,A = \begin{cases} 1, & \text{if a label is requested},\\ 0, & \text{otherwise}, \end{cases}

and suppose

Pr⁡(A=1∣Y=y,Z)=a(y),0≤a(y)≤1.(1)\Pr(A=1 \mid Y=y,Z)=a(y), \qquad0 \leq a(y) \leq1. \tag*{(1)}

The measurable function a:Y→[0,1]a:\mathcal{Y}\to[0,1] is the acquisition design. It is chosen by the investigator and is treated as known and parameter-free. A label request need not result in a returned label. Conditional on A=1A=1, let

R={1,if the oracle returns a label,0,if the oracle abstains.R = \begin{cases} 1, & \text{if the oracle returns a label},\\ 0, & \text{if the oracle abstains}. \end{cases}

We model the response probability as

Pr⁡(R=1∣A=1,Y=y)=r(y;θ,η),0<r(y;θ,η)<1.(2)\Pr(R=1 \mid A=1,Y=y)=r(y;\theta,\eta), \qquad0<r(y;\theta,\eta)<1. \tag*{(2)}

where η∈H⊆Rq\eta\in\mathcal{H} \subseteq\mathbb{R}^{q} contains parameters specific to the response mechanism. The defining feature of (2) is that the oracle response may depend on the same parameter θ\theta that determines the classifier. The realized response or abstention can therefore itself contain information about the classification model. Throughout the main development we assume

R⊥Z∣(Y,A=1),(3)R \perp Z \mid(Y,A=1), \tag*{(3)}

and that a returned label is correct. Thus the response mechanism is feature-dependent but does not depend further on the unobserved class once YY is given. Noisy returned、】【 labels and class-dependent response mechanisms are outside the scope of the present paper.

For one sampling unit, the observed data can be represented by

O=(Y,A,AR,ARZ).O = (Y, A, AR, ARZ).

Up to factors involving the known acquisition design aa, its likelihood is

L(θ,η;O)∝pθ(Y)[r(Y;θ,η)R{1−r(Y;θ,η)}1−RτZ(Y;θ)R]A.(4)L(\theta,\eta;O) \propto p_\theta(Y)\left[r(Y;\theta,\eta)^R\{1-r(Y;\theta,\eta)\}^{1-R}\tau_Z(Y;\theta)^R\right]^A. \tag*{(4)}

Hence

ℓ(θ,η;O)=log⁡pθ(Y)+A[Rlog⁡r(Y;θ,η)+(1−R)log⁡{1−r(Y;θ,η)}+Rlog⁡τZ(Y;θ)].(5)\ell(\theta,\eta;O) = \log p_\theta(Y) + A\left[R\log r(Y;\theta,\eta)+(1-R)\log\{1-r(Y;\theta,\eta)\}+R\log\tau_Z(Y;\theta)\right]. \tag*{(5)}

Define the feature and conditional-label scores by

SY(Y;θ)=∇θlog⁡pθ(Y),SZ∣Y(Y,Z;θ)=∇θlog⁡τZ(Y;θ),S_Y(Y;\theta)=\nabla_\theta\log p_\theta(Y),\qquad S_{Z\mid Y}(Y,Z;\theta)=\nabla_\theta\log\tau_Z(Y;\theta),

and let

qθ(y)=∇θlogit⁡r(y;θ,η),qη(y)=∇ηlogit⁡r(y;θ,η).q_\theta(y)=\nabla_\theta\operatorname{logit}r(y;\theta,\eta),\qquad q_\eta(y)=\nabla_\eta\operatorname{logit}r(y;\theta,\eta).

Differentiation of (5) gives

Sθ=SY+A[{R−r(Y)}qθ(Y)+RSZ∣Y],(6)S_\theta=S_Y+A\left[\{R-r(Y)\}q_\theta(Y)+RS_{Z\mid Y}\right], \tag*{(6)}
Sη=A{R−r(Y)}qη(Y).(7)S_\eta=A\{R-r(Y)\}q_\eta(Y). \tag*{(7)}

Thus a query can generate information through two distinct channels: the class label when it is returned and the response indicator itself.

Let

J(y)=Eθ[SZ∣YSZ∣YT∣Y=y]J(y)=E_\theta\left[S_{Z\mid Y}S_{Z\mid Y}^{\mathsf T}\mid Y=y\right]

denote the conditional Fisher information in a returned class label, and set

w(y)=r(y){1−r(y)},q(y)=(qθ(y)qη(y)),J~(y)=(J(y)000).w(y)=r(y)\{1-r(y)\},\qquad q(y)=\begin{pmatrix}q_\theta(y)\\q_\eta(y)\end{pmatrix},\qquad\widetilde{J}(y)=\begin{pmatrix}J(y)&0\\0&0\end{pmatrix}.

The information generated by a query at feature value yy is

K(y)=r(y)J~(y)+w(y)q(y)q(y)T.(8)K(y)=r(y)\widetilde{J}(y)+w(y)q(y)q(y)^{\mathsf T}. \tag*{(8)}

The first term is the expected information from the returned label, whereas the second is the information carried by the response or abstention outcome.

Let

ϕ=(θT,ηT)T,IY=(IY(θ)000),\phi=(\theta^{\mathsf T},\eta^{\mathsf T})^{\mathsf T},\qquad I_Y=\begin{pmatrix}I_Y(\theta)&0\\0&0\end{pmatrix},

where

IY(θ)=Eθ{SY(Y;θ)SY(Y;θ)T}.I_Y(\theta)=E_\theta\{S_Y(Y;\theta)S_Y(Y;\theta)^{\mathsf T}\}.

Proposition 2.1 Suppose that (1)–(3) hold and that the usual regularity conditions for Fisher-information calculations are satisfied. For a fixed acquisition design aa, the per-observation Fisher information for ϕ\phi is

I(a)=IY+Eθ[a(Y)K(Y)],(9)\mathcal{I}(a)=I_Y+E_\theta[a(Y)K(Y)], \tag*{(9)}

where K(y)K(y) is defined in (8). Equivalently,

I(a)=IY+Eθ[a(Y){r(Y)J(Y)+r(Y){1−r(Y)}q(Y)q(Y)T}].(10)\mathcal{I}(a)=I_Y+E_\theta\left[a(Y)\left\{r(Y)\mathcal{J}(Y)+r(Y)\{1-r(Y)\}q(Y)q(Y)^{\mathsf T}\right\}\right]. \tag*{(10)}

The proof is given in Supplementary Section S1.1.

Proposition 2.1 separates the two sources of information created by a label request. If a requested label is returned with certainty, r(y)≡1r(y)\equiv1, then the response term vanishes and (10) reduces to the usual information for selective label acquisition. More generally, if the response mechanism is parameter-dependent, the statistical value of a query is not determined only by the probability of obtaining its label: the observed response or abstention is itself part of the annotation experiment.

Because η\eta is unknown, however, the second term in (8) cannot be interpreted directly as additional information about the classifier. The component that remains informative for θ\theta after adjustment for the response-model nuisance parameters is derived next.

Efficient classifier information and the value of oracle response

The information decomposition in Proposition 2.1 treats ϕ=(θT,ηT)T\phi=(\theta^{\mathsf T},\eta^{\mathsf T})^{\mathsf T} as the full parameter vector. For classification, however, θ\theta is the parameter of interest, whereas η\eta is a nuisance parameter governing the oracle-response mechanism. Consequently, the response-information term in (10) must be adjusted for uncertainty in η\eta.

Partition the Fisher information according to ϕ=(θT,ηT)T\phi=(\theta^{\mathsf T},\eta^{\mathsf T})^{\mathsf T}:

I(a)=(Iθθ(a)Iθη(a)Iηθ(a)Iηη(a))\mathcal{I}(a)= \begin{pmatrix} I_{\theta\theta}(a) & I_{\theta\eta}(a)\\ I_{\eta\theta}(a) & I_{\eta\eta}(a) \end{pmatrix}

From (10),

Iθθ(a)=IY(θ)+Eθ[a(Y)r(Y)J(Y)]+Cθθ(a),(11)I_{\theta\theta}(a)=I_Y(\theta)+E_\theta[a(Y)r(Y)J(Y)]+C_{\theta\theta}(a), \tag*{(11)}
Iθη(a)=Cθη(a),(12)I_{\theta\eta}(a)=C_{\theta\eta}(a), \tag*{(12)}
Iηη(a)=Cηη(a),(13)I_{\eta\eta}(a)=C_{\eta\eta}(a), \tag*{(13)}

where

Cθθ(a)=Eθ[a(Y)w(Y)qθ(Y)qθ(Y)T],(14)C_{\theta\theta}(a)=E_\theta\left[a(Y)w(Y)q_\theta(Y)q_\theta(Y)^{\mathsf T}\right], \tag*{(14)}
Cθη(a)=Eθ[a(Y)w(Y)qθ(Y)qη(Y)T],(15)C_{\theta\eta}(a)=E_\theta\left[a(Y)w(Y)q_\theta(Y)q_\eta(Y)^{\mathsf T}\right], \tag*{(15)}
Cηη(a)=Eθ[a(Y)w(Y)qη(Y)qη(Y)T].(16)C_{\eta\eta}(a)=E_\theta\left[a(Y)w(Y)q_\eta(Y)q_\eta(Y)^{\mathsf T}\right]. \tag*{(16)}

Assume throughout this section that Cηη(a)C_{\eta\eta}(a) is positive definite. The efficient Fisher information for θ\theta, after adjustment for η\eta, is the Schur complement

Ieff(a)=Iθθ(a)−Iθη(a)Iηη(a)−1Iηθ(a).(17)I_{\mathrm{eff}}(a)=I_{\theta\theta}(a)-I_{\theta\eta}(a)I_{\eta\eta}(a)^{-1}I_{\eta\theta}(a). \tag*{(17)}

Substitution of (11)–(13) yields

Ieff(a)=IY(θ)+Eθ[a(Y)r(Y)J(Y)]+IR,eff(a),(18)I_{\mathrm{eff}}(a)=I_Y(\theta)+E_\theta[a(Y)r(Y)J(Y)]+I_{R,\mathrm{eff}}(a), \tag*{(18)}

where

IR,eff(a)=Cθθ(a)−Cθη(a)Cηη(a)−1Cηθ(a).(19)I_{R,\mathrm{eff}}(a)=C_{\theta\theta}(a)-C_{\theta\eta}(a)C_{\eta\eta}(a)^{-1}C_{\eta\theta}(a). \tag*{(19)}

The first two terms in (18) are respectively the information available from the feature distribution and the expected information from labels that are actually returned. The final term measures the additional classifier information supplied by the response process after removing the component attributable to the nuisance parameter η\eta. The following result gives a direct characterization of this contribution.

Proposition 3.1 Suppose that Cηη(a)C_{\eta\eta}(a) is positive definite and define

Ba=Cθη(a)Cηη(a)−1.(20)B_a=C_{\theta\eta}(a)C_{\eta\eta}(a)^{-1}. \tag*{(20)}

Then

IR,eff(a)=Eθ[a(Y)w(Y){qθ(Y)−Baqη(Y)}{qθ(Y)−Baqη(Y)}T].(21)I_{R,\mathrm{eff}}(a)=E_\theta\left[a(Y)w(Y)\{q_\theta(Y)-B_aq_\eta(Y)\}\{q_\theta(Y)-B_aq_\eta(Y)\}^{\mathsf T}\right]. \tag*{(21)}

Consequently,

IR,eff(a)⪰0.I_{R,\mathrm{eff}}(a)\succeq0.

Moreover,

IR,eff(a)=0⟺qθ(Y)=Baqη(Y)μa-almost surely,(22)I_{R,\mathrm{eff}}(a)=0\quad\Longleftrightarrow\quad q_\theta(Y)=B_aq_\eta(Y)\quad\mu_a\text{-almost surely}, \tag*{(22)}

where μa\mu_a is the measure with density proportional to a(y)w(y)pθ(y)a(y)w(y)p_\theta(y).

The proof is given in Supplementary Section S1.2.

Proposition 3.1 identifies precisely when observing whether the oracle responds provides information about the classifier beyond that contained in returned labels. The vector qθ(y)q_\theta(y) describes the local sensitivity of the response probability to the classifier parameter, while qη(y)q_\eta(y) describes sensitivity to the response-specific nuisance parameter. Equation (21) shows that only the component of qθ(Y)q_\theta(Y) that cannot be represented through the nuisance score contributes efficient information for classification.

This distinction is important. Parameter dependence of the response probability alone does not guarantee additional information about θ\theta after nuisance adjustment. If (22) holds, all classifier-related variation in the response mechanism is statistically confounded with variation in η\eta, and the response indicator contributes no additional efficient information. Otherwise,

IR,eff(a)≠0,I_{R,\mathrm{eff}}(a) \ne0,

and observing response or abstention supplies classifier information that would be discarded by a design criterion based only on the probability of obtaining a label.

The decomposition also identifies two limiting cases. If the response mechanism is independent of the classifier parameter, then qθ(y)≡0q_\theta(y) \equiv0 and hence

IR,eff(a)=0.I_{R,\mathrm{eff}}(a) = 0.

The Bernoulli weight w(y)=r(y){1−r(y)}w(y) = r(y)\{1-r(y)\} approaches zero as the response probability approaches either endpoint. If the corresponding logit-score vectors remain bounded, the pointwise response-information contribution also vanishes.

The positive-definiteness requirement on Cηη(a)C_{\eta\eta}(a) is itself design-dependent. An acquisition rule may therefore be highly informative for classification while failing to identify the response mechanism. We return to this issue when defining the admissible class of acquisition designs.

Classification risk under informative oracle response

The information criterion in § 4 measures precision in estimating the model parameters, but the objective of label acquisition is classification. We therefore evaluate an acquisition design through its leading contribution to the zero-one classification risk.

Let

dθ(y)=arg max⁡1≤k≤gτk(y;θ)d_\theta(y) = \operatorname*{arg\,max}_{1 \le k \le g} \tau_k(y;\theta)

denote the plug-in Bayes rule and define its risk, under the true parameter θ0\theta_0, by

R(θ)=Pr⁡θ0{dθ(Y)≠Z}.R(\theta) = \Pr_{\theta_0}\{d_\theta(Y) \ne Z\}.

Let

R∗=R(θ0)R^* = R(\theta_0)

be the Bayes risk. Under the regular boundary conditions for multiclass classification, the local excess risk admits the quadratic expansion

R(θ0+h)−R∗=12h⊤HRh+o(∥h∥2),h→0,(23)R(\theta_0+h)-R^* = \frac{1}{2}h^\top H_R h + o(\lVert h\rVert^2), \qquad h \to0, \tag*{(23)}

where HR⪰0H_R \succeq0 is the classification-risk curvature matrix at θ0\theta_0. It depends only on parameter directions that perturb active Bayes decision boundaries.

The response parameter η\eta does not enter the classifier. Accordingly, for the full parameter

ϕ=(θ⊤,η⊤)⊤,\phi= (\theta^\top,\eta^\top)^\top,

define the augmented risk-curvature matrix

H~R=(HR000).(24)\widetilde{H}_R = \begin{pmatrix} H_R & 0 \\ 0 & 0 \end{pmatrix}. \tag*{(24)}

Suppose that, under a fixed admissible acquisition design aa, an efficient estimator ϕ^a\hat{\phi}_a satisfies

n(ϕ^a−ϕ0)→dN{0,I(a)−1},(25)\sqrt{n}(\hat{\phi}_a-\phi_0) \xrightarrow{d} N\{0,\mathcal{I}(a)^{-1}\}, \tag*{(25)}

with I(a)\mathcal{I}(a) nonsingular. Assume additionally that, writing en,a=θ^a−θ0e_{n,a}=\hat{\theta}_a-\theta_0,

nE(en,aen,aT)⟶Ieff(a)−1,nE∣R(θ^a)−R∗−12en,aTHRen,a∣⟶0.nE(e_{n,a}e_{n,a}^{\mathsf T}) \longrightarrow I_{\mathrm{eff}}(a)^{-1}, \qquad nE\left|R(\hat{\theta}_a)-R^*-\frac{1}{2}e_{n,a}^{\mathsf T}H_Re_{n,a}\right| \longrightarrow0.

These moment and remainder conditions, together with (23), give the leading expected excess risk

E{R(θ^a)}−R∗=12ntr⁡{H~RI(a)−1}+o(n−1).(26)E\{R(\hat{\theta}_a)\}-R^*=\frac{1}{2n}\operatorname{tr}\left\{\widetilde{H}_R\mathcal{I}(a)^{-1}\right\}+o(n^{-1}). \tag*{(26)}

Equivalently, using the block-inverse identity,

tr⁡{H~RI(a)−1}=tr⁡{HRIeff(a)−1},(27)\operatorname{tr}\left\{\widetilde{H}_R\mathcal{I}(a)^{-1}\right\}=\operatorname{tr}\left\{H_RI_{\mathrm{eff}}(a)^{-1}\right\}, \tag*{(27)}

where Ieff(a)I_{\mathrm{eff}}(a) is the nuisance-adjusted information in (17). This motivates the classification-risk criterion

Φ(a)=tr⁡{H~RI(a)−1}.(28)\Phi(a)=\operatorname{tr}\left\{\widetilde{H}_R\mathcal{I}(a)^{-1}\right\}. \tag*{(28)}

For a labeling budget ρ∈(0,1)\rho\in(0,1), the oracle acquisition problem is therefore

min⁡a∈AρΦ(a),Aρ={a:0≤a(y)≤1, Eθ0{a(Y)}=ρ, I(a)≻0}.(29)\min_{a\in\mathcal{A}_\rho}\Phi(a), \qquad\mathcal{A}_\rho=\{a:0\leq a(y)\leq1,\ E_{\theta_0}\{a(Y)\}=\rho,\ \mathcal{I}(a)\succ0\}. \tag*{(29)}

The nonsingularity condition is substantive in the present setting because the acquisition design determines not only which labels may be observed but also which response outcomes are available for identifying the response mechanism.

Since

I(a)=IY+E{a(Y)K(Y)}\mathcal{I}(a)=\mathcal{I}_Y+E\{a(Y)K(Y)\}

is affine in aa, the criterion in (28) retains the trace-inverse structure of an optimal-design problem despite the nuisance adjustment in Ieff(a)I_{\mathrm{eff}}(a).

The local value of querying a feature vector yy is obtained from the directional derivative of Φ\Phi. Define

ψa(y)=tr⁡{H~RI(a)−1K(y)I(a)−1}.(30)\psi_a(y)=\operatorname{tr}\left\{\widetilde{H}_R\mathcal{I}(a)^{-1}K(y)\mathcal{I}(a)^{-1}\right\}. \tag*{(30)}

Because

K(y)=r(y)J~(y)+w(y)q(y)q(y)T,K(y)=r(y)\widetilde{J}(y)+w(y)q(y)q(y)^{\mathsf{T}},

the query value decomposes as

ψa(y)=ψaL(y)+ψaR(y),(31)\psi_a(y)=\psi_a^L(y)+\psi_a^R(y), \tag*{(31)}

where

ψaL(y)=r(y)tr⁡{H~RI(a)−1J~(y)I(a)−1},(32)\psi_a^L(y)=r(y)\operatorname{tr}\left\{\widetilde{H}_R\mathcal{I}(a)^{-1}\widetilde{J}(y)\mathcal{I}(a)^{-1}\right\}, \tag*{(32)}
ψaR(y)=w(y)q(y)TI(a)−1H~RI(a)−1q(y).(33)\psi_a^R(y)=w(y)q(y)^{\mathsf{T}}\mathcal{I}(a)^{-1}\widetilde{H}_R\mathcal{I}(a)^{-1}q(y). \tag*{(33)}

The term ψaL(y)\psi_a^L(y) is the classification value of the label, discounted by the probability that the oracle returns it. In contrast, ψaR(y)\psi_a^R(y) is the classification value of observing the response outcome itself. In particular,

ψaR(y)≥0.\psi_a^R(y)\ge0.

Thus the value of a query is not, in general, obtained by simply multiplying a label-acquisition score by the probability of response. Such a calculation retains ψaL(y)\psi_a^L(y) but discards ψaR(y)\psi_a^R(y).

Proposition 4.1 Suppose that I(a)≻0\mathcal{I}(a)\succ0, and let hh be a bounded measurable perturbation such that at=a+tha_t=a+th remains admissible for sufficiently small ∣t∣|t|. Then

ddtΦ(at)∣t=0=−Eθ0{h(Y)ψa(Y)},(34)\left.\frac{d}{dt}\Phi(a_t)\right|_{t=0}=-\mathbb{E}_{\theta_0}\{h(Y)\psi_a(Y)\}, \tag*{(34)}

where ψa\psi_a is given by (30). Moreover, ψa\psi_a admits the decomposition (31)–(33).

The proof is given in Supplementary Section S1.3. Proposition 4.1 provides the local principle underlying response-aware acquisition. A candidate can be valuable because its label is likely to be returned and informative for the decision boundary, because its response outcome is itself informative for classifier-relevant parameter directions, or through both channels. Consequently, two candidates can be ordered differently by a response-discounted label criterion and by the full classification-risk criterion. The optimal allocation implied by this marginal value is characterized in the next section.

Risk-optimal acquisition and response identifiability

We now characterize acquisition designs that minimize the leading classification-risk criterion

Φ(a)=tr⁡{H~RI(a)−1}\Phi(a)=\operatorname{tr}\left\{\widetilde{H}_R\mathcal{I}(a)^{-1}\right\}

subject to a fixed query budget. Because the information matrix is affine in the acquisition design, the resulting problem retains a convex optimal-design structure even though the response mechanism contains unknown nuisance parameters.

For a fixed budget ρ∈(0,1)\rho\in(0,1), consider

Aρ∘={a:Y→[0,1]:Eθ0{a(Y)}=ρ, I(a)≻0}.(35)\mathcal{A}^{\circ}_{\rho}=\{a:\mathcal{Y}\to[0,1]:\mathbb{E}_{\theta_0}\{a(Y)\}=\rho,\ \mathcal{I}(a)\succ0\}. \tag*{(35)}

We assume that this class is nonempty and that an optimal design exists.

Proposition 5.1 On any convex subset of Aρ∘\mathcal{A}^{\circ}_{\rho}, the criterion

a⟼Φ(a)a\longmapsto\Phi(a)

is convex.

The proof is given in Supplementary Section S1.4. Convexity, together with the derivative in Proposition 4.1, yields a direct characterization of the oracle acquisition rule.

Theorem 5.2 Let a∗∈Aρ∘a^*\in\mathcal{A}^{\circ}_{\rho} minimize Φ(a)\Phi(a), and suppose that a∗a^* is an interior point of the positive-definite information domain relative to the box and budget constraints. Then there exists λ∈R\lambda\in\mathbb{R} such that, for Pθ0P_{\theta_0}-almost every yy,

{ψa∗(y)>λ⟹a∗(y)=1,ψa∗(y)<λ⟹a∗(y)=0,0<a∗(y)<1⟹ψa∗(y)=λ,(36)\begin{cases} \psi_{a^*}(y)>\lambda&\Longrightarrow a^*(y)=1,\\ \psi_{a^*}(y)<\lambda&\Longrightarrow a^*(y)=0,\\ 0<a^*(y)<1 &\Longrightarrow\psi_{a^*}(y)=\lambda, \tag*{(36)} \end{cases}

where

ψa∗(y)=ψa∗L(y)+ψa∗R(y)\psi_{a^*}(y)=\psi^L_{a^*}(y)+\psi^R_{a^*}(y)

is the response-aware classification value defined in (4.9). Conversely, any feasible design satisfying these conditions is globally optimal.

The proof is given in Supplementary Section S1.5. Theorem 5.2 shows that the optimal design allocates the labeling budget according to the total classification value of a query, rather than according to the value of a potentially returned label alone. Except on level sets where randomization is required to satisfy the budget, the oracle rule therefore has a threshold form.

This distinction can alter the ordering of candidate observations. A response-discounted rule based only on ψaL(y)\psi^L_a(y) favors observations whose returned labels are valuable after accounting for their probability of response. The full rule instead ranks observations according to

ψaL(y)+ψaR(y).\psi^L_a(y)+\psi^R_a(y).

Consequently, an observation with a smaller probability of returning a label may still have greater total acquisition value if its response outcome is sufficiently informative about classifier-relevant parameter directions. The phenomenon does not require response probabilities close to zero: indeed, the Bernoulli information factor w(y)=r(y){1−r(y)}w(y)=r(y)\{1-r(y)\} vanishes as r(y)r(y) approaches either zero or one.

Response identifiability under selective acquisition

The preceding optimality result presumes that the information matrix is nonsingular. In the present problem this requirement depends on the acquisition design itself. A design that concentrates queries on a restricted region of the feature space can provide substantial classification information while failing to identify the response mechanism. The nuisance-information block is

Cηη(a)=Eθ0[a(Y)w(Y)qη(Y)qη(Y)T].(37)C_{\eta\eta}(a)=E_{\theta_0}\left[a(Y)w(Y)q_\eta(Y)q_\eta(Y)^{\mathsf T}\right]. \tag*{(37)}

Proposition 5.3 For any v∈Rqv\in\mathbb{R}^{q},

vTCηη(a)v=Eθ0[a(Y)w(Y){vTqη(Y)}2].(38)v^{\mathsf T}C_{\eta\eta}(a)v=E_{\theta_0}\left[a(Y)w(Y)\{v^{\mathsf T}q_\eta(Y)\}^{2}\right]. \tag*{(38)}

Consequently,

Cηη(a)≻0C_{\eta\eta}(a)\succ0

if and only if there is no nonzero v∈Rqv\in\mathbb{R}^{q} such that

vTqη(Y)=0μa-almost surely,(39)v^{\mathsf T}q_\eta(Y)=0\quad\mu_a\text{-almost surely}, \tag*{(39)}

where μa\mu_a is the weighted measure introduced in Proposition 3.1.

The proof is given in Supplementary § 13.6. Proposition 5.3 gives a design-specific rank condition for estimation of the response nuisance parameter. In particular, positive query probability alone is insufficient: the queried observations must exhibit enough variation in the response-model regressors to distinguish the components of η\eta.

For example, consider the response model

logit⁡r(y;θ,η)=η0+η1U(y;θ),(40)\operatorname{logit} r(y;\theta,\eta)=\eta_0+\eta_1U(y;\theta), \tag*{(40)}

where U(y;θ)U(y;\theta) is a scalar measure of classification uncertainty. Then

qη(y)=(1U(y;θ)).q_\eta(y)= \begin{pmatrix} 1\\ U(y;\theta) \end{pmatrix}.

Provided a(Y)w(Y)a(Y)w(Y) has positive expectation, Cηη(a)≻0C_{\eta\eta}(a)\succ0 if and only if

Var⁡μa{U(Y;θ0)}>0.(41)\operatorname{Var}_{\mu_a}\{U(Y;\theta_0)\}>0. \tag*{(41)}

Thus a design supported only on observations having the same uncertainty level cannot separately identify the intercept and uncertainty effect in the response model. In particular, concentrating all queries on a single decision-boundary uncertainty level can destroy response-mechanism identifiability.

For theoretical and numerical implementation it is therefore useful to work with the uniformly regular design class

Aρ,κ={a:0≤a(y)≤1, Eθ0{a(Y)}=ρ, λmin⁡{I(a)}≥κ},κ>0.(42)\mathcal{A}_{\rho,\kappa}=\{a:0\leq a(y)\leq1,\ E_{\theta_0}\{a(Y)\}=\rho,\ \lambda_{\min}\{\mathcal{I}(a)\}\geq\kappa\},\qquad\kappa>0. \tag*{(42)}

The lower information bound prevents the acquisition procedure from obtaining an apparent classification advantage by approaching a design under which some components of the annotation experiment cease to be identifiable. This constraint is one of the regularity requirements used in the two-stage analysis below; it does not by itself establish the estimator conditions needed for that analysis.

Adaptive response-aware acquisition

The oracle criterion depends on the unknown classification and response parameters and on the risk curvature. We first analyze a two-stage procedure with a pilot-selected design that is held fixed during an independent second stage. We then distinguish this experiment from the sequential finite-pool implementation used in the numerical studies.

Pilot selection and the second-stage experiment

Let mnm_n be the number of pilot sampling units and let nn be the number of new second-stage sampling units. Assume

mn⟶∞,mn/n⟶0.(43)m_n \longrightarrow\infty,\qquad m_n/n \longrightarrow0. \tag*{(43)}

Write F0,n\mathcal{F}_{0,n} for the sigma-field generated by the pilot features, query decisions, responses, and returned labels, together with any auxiliary randomness used to choose the second-stage design. The pilot uses a design that provides nonsingular information for the parameters to be estimated. Let ϕ^0\widehat{\phi}_0 and H^R\widehat{H}_R denote the pilot estimates. The selected acquisition function a^n\widehat{a}_n is F0,n\mathcal{F}_{0,n}-measurable.

Generate new pairs (Yi,Zi)(Y_i,Z_i), i=1,…,ni=1,\ldots,n, independently from pθ0p_{\theta_0} and independently of F0,n\mathcal{F}_{0,n}. Let Ui,ViU_i,V_i be independent uniform random variables on (0,1)(0,1), independent across units and of the pilot and the new pairs. Set

Ai=1{Ui≤a^n(Yi)},AiRi=Ai1{Vi≤r(Yi;ϕ0)}.(44)A_i = 1\{U_i \le\widehat{a}_n(Y_i)\},\qquad A_iR_i = A_i1\{V_i \le r(Y_i;\phi_0)\}. \tag*{(44)}

This construction implements the conditional independence of response and class label assumed in § 3. Define Oi=(Yi,Ai,AiRi,AiRiZi)O_i=(Y_i,A_i,A_iR_i,A_iR_iZ_i), and let Pϕ,aP_{\phi,a} denote the one-unit observed-data law under parameter ϕ\phi and a fixed design aa. Then

L(O1,…,On∣F0,n)=Pϕ0,a^n⊗n.(45)\mathcal{L}(O_1,\ldots,O_n\mid\mathcal{F}_{0,n})=P_{\phi_0,\widehat{a}_n}^{\otimes n}. \tag*{(45)}

Thus the second-stage observations are independent and identically distributed conditional on the pilot. Unconditionally they generally are not independent, because they share the random design a^n\widehat{a}_n. We do not apply an iid argument to the pooled two-stage sample.

For example, for a fixed design aa, the nonzero observed-data densities are

pϕ,a(y,A=0)=pθ(y){1−a(y)},p_{\phi,a}(y,A=0)=p_{\theta}(y)\{1-a(y)\},
pϕ,a(y,A=1,R=0)=pθ(y)a(y){1−r(y;ϕ)},p_{\phi,a}(y,A=1,R=0)=p_{\theta}(y)a(y)\{1-r(y;\phi)\},
pϕ,a(y,A=1,R=1,Z=z)=pθ(y)a(y)r(y;ϕ)τz(y;θ).p_{\phi,a}(y,A=1,R=1,Z=z)=p_{\theta}(y)a(y)r(y;\phi)\tau_z(y;\theta).

Conditional on the observed pilot, the selected function a^n\widehat a_n is held fixed when differentiating the second-stage likelihood. Its information per sampling unit is therefore I(a^n)\mathcal{I}(\widehat a_n) from Proposition 2.1, evaluated at ϕ0\phi_0. The returned-label subset need not follow the population feature law; (45) concerns the complete observed-data units, including unqueried units and queried abstentions.

Design selection and oracle value

Use a fixed, nonempty measurable design class D⊆Aρ,κ\mathcal{D}\subseteq\mathcal{A}_{\rho,\kappa}, where Aρ,κ\mathcal{A}_{\rho,\kappa} is defined in (42). One may take the full regular class if the uniform assumptions below hold; a restricted class may be needed for empirical estimation. All suprema and selected designs below are assumed measurable. Define

I^n(a)=I^Y,n+E^n{a(Y)K^n(Y)},(46)\widehat{\mathcal{I}}_n(a)=\widehat{\mathcal{I}}_{Y,n}+\widehat{E}_n\{a(Y)\widehat{K}_n(Y)\}, \tag*{(46)}

where

K^n(y)=r^n(y)J~n(y)+r^n(y){1−r^n(y)}q^n(y)q^n(y)T,(47)\widehat{K}_n(y)=\widehat r_n(y)\widetilde{\mathcal{J}}_n(y)+\widehat r_n(y)\{1-\widehat r_n(y)\}\widehat q_n(y)\widehat q_n(y)^{\mathsf T}, \tag*{(47)}

and

Φ^n(a)=tr⁡{H^R,nI^n(a)−1},H^R,n=(H^R000).(48)\widehat{\Phi}_n(a)=\operatorname{tr}\{\widehat H_{R,n}\widehat{\mathcal{I}}_n(a)^{-1}\},\qquad \widehat H_{R,n}= \begin{pmatrix} \widehat H_R & 0\\ 0 & 0 \end{pmatrix}. \tag*{(48)}

The criterion and every quantity used to choose a^n\widehat a_n must be F0,n\mathcal{F}_{0,n}-measurable. In this experiment, E^n\widehat E_n may use pilot features, model-based integration, or an independent reference feature sample included in F0,n\mathcal{F}_{0,n}. It does not use the features of the second-stage sample. Positive definiteness of the estimated information is required on the class over which the criterion is optimized.

Choose a measurable approximate minimizer

a^n∈D,Φ^n(a^n)≤inf⁡a∈DΦ^n(a)+εn,0≤εn=op(1).(49)\widehat a_n\in\mathcal{D},\qquad \widehat\Phi_n(\widehat a_n)\leq\inf_{a\in\mathcal{D}}\widehat\Phi_n(a)+\varepsilon_n,\qquad 0\leq\varepsilon_n=o_p(1). \tag*{(49)}

Feasibility in the population class D\mathcal{D} is an assumption. Using estimated budget and information constraints instead requires a separate feasibility-approximation argument.

For deterministic a∈Da\in\mathcal{D}, let ϕ^n,a\widehat\phi_{n,a} be an estimator based only on nn iid observations from Pϕ0,aP_{\phi_0,a}. It is a measurable function of these observations and aa, with no other pilot dependence. Write EaE_a for expectation in this fixed-design experiment and Sa(O;ϕ0)S_a(O;\phi_0) for its joint score, given by (6)–(7). Let θ^n,a∈Θ\widehat\theta_{n,a}\in\Theta be its classification component.

Theorem 6.1 Suppose that a∗∈arg min⁡a∈DΦ(a)a^* \in\operatorname*{arg\,min}_{a\in\mathcal{D}} \Phi(a) exists, (49) holds, and

Δn:=sup⁡a∈D∣Φ^n(a)−Φ(a)∣→p0.(50)\Delta_n := \sup_{a\in\mathcal{D}} \left|\widehat{\Phi}_n(a)-\Phi(a)\right| \xrightarrow{p} 0. \tag*{(50)}

Then

0≤Φ(a^n)−Φ(a∗)≤2Δn+εn→p0.(51)0 \leq\Phi(\widehat{a}_n)-\Phi(a^*) \leq2\Delta_n+\varepsilon_n \xrightarrow{p} 0. \tag*{(51)}

For the expected-risk conclusion, additionally assume the independent second-stage construction (6.2)–(6.3), the local risk expansion (23), and the following uniform fixed-design estimator conditions. First, assume

n(ϕ^n,a−ϕ0)=I(a)−11n∑i=1nSa(Oi;ϕ0)+ζn,a,(52)\sqrt{n}(\widehat{\phi}_{n,a}-\phi_0)=\mathcal{I}(a)^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^{n}S_a(O_i;\phi_0)+\zeta_{n,a}, \tag*{(52)}
sup⁡a∈DEa∥ζn,a∥2⟶0.\sup_{a\in\mathcal{D}} E_a\|\zeta_{n,a}\|^2 \longrightarrow0.

Second, for some δ>0\delta>0, assume

sup⁡n≥n0sup⁡a∈DEa∥n(θ^n,a−θ0)∥2+δ<∞.(53)\sup_{n\geq n_0}\sup_{a\in\mathcal{D}}E_a\left\|\sqrt{n}(\widehat{\theta}_{n,a}-\theta_0)\right\|^{2+\delta}<\infty. \tag*{(53)}

Assume also the usual score identities EaSa=0E_aS_a=0 and Ea(SaSaT)=I(a)E_a(S_aS_a^{\mathsf T})=\mathcal{I}(a). Then the second-stage estimator satisfies

E{R(θ^n,a^n)−R∗∣F0,n}=Φ(a^n)2n+rn,E∣rn∣=o(n−1),(54)E\{R(\widehat{\theta}_{n,\widehat{a}_n})-R^*\mid\mathcal{F}_{0,n}\}=\frac{\Phi(\widehat{a}_n)}{2n}+r_n,\qquad E|r_n|=o(n^{-1}), \tag*{(54)}

and consequently

E{R(θ^n,a^n)}−R∗=Φ(a∗)2n+o(n−1).(55)E\{R(\widehat{\theta}_{n,\widehat{a}_n})\}-R^*=\frac{\Phi(a^*)}{2n}+o(n^{-1}). \tag*{(55)}

The proof is given in Supplementary Section S1.7. It first establishes a uniform expected-risk expansion for deterministic designs and then uses the exact conditional law (45). In particular, pointwise asymptotic normality for each fixed design would not suffice: the selected design varies with the pilot, and convergence in distribution alone does not justify convergence of expected risk. Conditions (52) and (53) supply the required uniform second-moment and remainder control. They are assumptions on the estimator family, not consequences established here for arbitrary likelihood optimizers.

The information bound gives 0≤Φ(a)≤κ−1tr⁡(H~R)0\leq\Phi(a)\leq\kappa^{-1}\operatorname{tr}(\widetilde{H}_{R}) on D\mathcal{D}. This bound allows the conditional conclusion to be averaged over the pilot without treating the two stages as iid. The result does not require a unique oracle design. It is an optimal-value result over D\mathcal{D}, rather than a proof that the selected design itself converges to a unique function.

Uniform criterion consistency must also be justified for the chosen approximation and class. Pilot-parameter consistency alone does not imply (50) for unrestricted measurable designs when expectations are replaced by empirical averages. A suitable uniform law of large numbers, an appropriate restricted design class, or direct uniform control of model-based integration is needed.

The theorem concerns a fixed expected query budget: conditional on the pilot, the second stage uses nρn\rho queries in expectation. Its final estimator uses only the new observations. If Nn=n+mnN_n=n+m_n counts both stages, (43) gives n/Nn→1n/N_n\to1, so the same risk coefficient can be written with denominator 2Nn2N_n. Pilot queries contribute at most mn=o(Nn)m_n=o(N_n), and the overall expected query fraction tends to ρ\rho. These accounting statements do not prove an equivalent result for a pooled estimator. For such an estimator θ~n\tilde{\theta}_n, one sufficient additional requirement is

E∥n(θ~n−θ^n,a^n)∥2⟶0,sup⁡n≥n0E∥n(θ~n−θ0)∥2+δ<∞.\mathbb{E}\left\|\sqrt{n}\left(\tilde{\theta}_n-\hat{\theta}_{n,\hat{a}_n}\right)\right\|^2 \longrightarrow0,\qquad\sup_{n\ge n_0}\mathbb{E}\left\|\sqrt{n}\left(\tilde{\theta}_n-\theta_0\right)\right\|^{2+\delta}<\infty.

The same quadratic-risk argument then transfers the leading coefficient. The condition mn/n→0m_n/n\to0 alone does not establish these requirements.

Sequential finite-pool implementation

For an available pool {y1,…,yN}\{y_1,\ldots,y_N\}, the plug-in marginal value is

ψ^a(yi)=ψ^aL(yi)+ψ^aR(yi).(56)\widehat{\psi}_a(y_i)=\widehat{\psi}_a^L(y_i)+\widehat{\psi}_a^R(y_i). \tag*{(56)}

A practical implementation updates the fitted classifier, response model, and information matrices after each batch and selects high-scoring remaining observations. This uses the query-value decomposition but is a different sampling experiment from the theorem. Selection without replacement, exact batch sizes, reuse of pilot observations in estimation, and repeated model updates generally prevent the conditional product law (45) from holding. Likewise, choosing the design using the features of the eventual estimation pool does not satisfy the fresh-sample construction above. Conditioning additionally on those features leads to a fixed-feature experiment, not the iid population experiment analyzed here. A separate conditional or adaptive sampling argument would be needed for that setting.

Accordingly, Theorem 6.1 does not establish oracle attainment for the sequential finite-pool experiments. Those experiments assess a practical implementation of the criterion. The pilot provides initial information about both the classifier and the response mechanism; a regular pilot design helps avoid beginning acquisition with an unidentified response model.

Binary logistic illustration

We now illustrate the preceding theory in a binary classification model for which the decision boundary, label information, response information, and classification-risk curvature can all be written explicitly.

Let

Z∈{0,1},Pr⁡(Z=1∣Y=y)=τθ(y)=exp⁡(α+βy)1+exp⁡(α+βy).Z\in\{0,1\},\qquad\Pr(Z=1\mid Y=y)=\tau_\theta(y)=\frac{\exp(\alpha+\beta y)}{1+\exp(\alpha+\beta y)}.

with

θ=(α,β)T,β>0.\theta=(\alpha,\beta)^\mathsf{T},\qquad\beta>0.

For simplicity, suppose that the marginal distribution of YY does not depend on θ\theta, with density ff. The Bayes classifier assigns class 1 whenever

α+βy>0,\alpha+\beta y>0,

so the decision boundary is

b(θ)=−αβ.(57)b(\theta)=-\frac{\alpha}{\beta}. \tag*{(57)}

Let

b0=−α0β0b_0=-\frac{\alpha_0}{\beta_0}

denote the true boundary. Under the usual smoothness condition f(b0)>0f(b_0)>0, the local excess risk satisfies

R(θ0+h)−R∗=f(b0)4β0(hα+b0hβ)2+o(∥h∥2).(58)R(\theta_0+h)-R^*=\frac{f(b_0)}{4\beta_0}(h_\alpha+b_0h_\beta)^2+o(\lVert h\rVert^2). \tag*{(58)}

Hence

HR=f(b0)2β0v0v0T,v0=(1b0).(59)H_R=\frac{f(b_0)}{2\beta_0}v_0v_0^\mathsf{T},\qquad v_0=\begin{pmatrix}1\\b_0\end{pmatrix}. \tag*{(59)}

Thus only parameter directions that perturb the decision boundary contribute to first-order classification risk.

Write

x(y)=(1y).x(y)=\begin{pmatrix}1\\y\end{pmatrix}.

The conditional class-label information is

J(y)=τ(y){1−τ(y)}x(y)x(y)T.(60)J(y)=\tau(y)\{1-\tau(y)\}x(y)x(y)^\mathsf{T}. \tag*{(60)}

Because

τ(y){1−τ(y)}\tau(y)\{1-\tau(y)\}

is maximized at τ(y)=1/2\tau(y)=1/2, the scalar Bernoulli variance factor is largest at the Bayes boundary. The full information matrix also depends on x(y)x(y)Tx(y)x(y)^\mathsf{T}, so its magnitude and classification value need not be largest there.

To model informative oracle response, consider

logit⁡r(y;θ,η)=η0+η1U(y;θ),(61)\operatorname{logit}r(y;\theta,\eta)=\eta_0+\eta_1U(y;\theta), \tag*{(61)}

where

U(y;θ)=−τ(y)log⁡τ(y)−{1−τ(y)}log⁡{1−τ(y)}(62)U(y;\theta)=-\tau(y)\log\tau(y)-\{1-\tau(y)\}\log\{1-\tau(y)\} \tag*{(62)}

is the predictive entropy. The case η1<0\eta_1<0 corresponds to an oracle that is less likely to respond at more uncertain observations.

Since

∇θτ(y)=τ(y){1−τ(y)}x(y)\nabla_\theta\tau(y)=\tau(y)\{1-\tau(y)\}x(y)

and

dUdτ=log⁡1−ττ,\frac{dU}{d\tau}=\log\frac{1-\tau}{\tau},

we obtain

∇θU(y;θ)=−{α+βy}τ(y){1−τ(y)}x(y).(63)\nabla_\theta U(y;\theta)=-\{\alpha+\beta y\}\tau(y)\{1-\tau(y)\}x(y). \tag*{(63)}

Therefore

qθ(y)=−η1{α+βy}τ(y){1−τ(y)}x(y),(64)q_\theta(y)=-\eta_1\{\alpha+\beta y\}\tau(y)\{1-\tau(y)\}x(y), \tag*{(64)}

whereas

qη(y)=(1U(y;θ)).(65)q_{\eta}(y)=\begin{pmatrix}1\\U(y;\theta)\end{pmatrix}. \tag*{(65)}

The raw response-information contribution at yy is consequently

KR(y)=r(y){1−r(y)}qθ(y)qθ(y)T=r(y){1−r(y)}η12{α+βy}2τ(y)2{1−τ(y)}2x(y)x(y)T.(66)K_R(y)=r(y)\{1-r(y)\}q_{\theta}(y)q_{\theta}(y)^{\mathsf{T}} =r(y)\{1-r(y)\}\eta_1^2\{\alpha+\beta y\}^2\tau(y)^2\{1-\tau(y)\}^2x(y)x(y)^{\mathsf{T}}. \tag*{(66)}

In contrast, the expected returned-label contribution is

KL(y)=r(y)τ(y){1−τ(y)}x(y)x(y)T.(67)K_L(y)=r(y)\tau(y)\{1-\tau(y)\}x(y)x(y)^{\mathsf{T}}. \tag*{(67)}

These two sources of information have fundamentally different spatial geometry. At the Bayes boundary y=b(θ)y=b(\theta),

α+βb(θ)=0,\alpha+\beta b(\theta)=0,

and hence

qθ{b(θ)}=0,KR{b(θ)}=0.(68)q_{\theta}\{b(\theta)\}=0,\qquad K_R\{b(\theta)\}=0. \tag*{(68)}

Thus, under the entropy-linked response model, the raw classifier information in the response indicator vanishes exactly at the decision boundary, even though the scalar Bernoulli variance factor is maximal there.

At the opposite extreme, as ∣α+βy∣→∞|\alpha+\beta y|\to\infty,

τ(y){1−τ(y)}→0,\tau(y)\{1-\tau(y)\}\to0,

so both the label and response information vanish. Response information is therefore concentrated away from the exact boundary but not in the extreme tails. It is naturally associated with intermediate off-boundary regions.

This distinction can change acquisition rankings. A response-discounted criterion considers only the expected value of the returned label, represented locally by ψaL(y)\psi_a^L(y). The full response-aware criterion instead uses

ψa(y)=ψaL(y)+ψaR(y).\psi_a(y)=\psi_a^L(y)+\psi_a^R(y).

For two candidates y1y_1 and y2y_2, it is therefore possible that

ψaL(y1)>ψaL(y2),\psi_a^L(y_1)>\psi_a^L(y_2),

while

ψa(y1)<ψa(y2).\psi_a(y_1)<\psi_a(y_2).

The reversal occurs when the additional classification value carried by the response outcome at y2y_2 more than compensates for its smaller response-discounted label value.

The example also clarifies why informative response should not be interpreted as a preference for almost-certain abstention. The Bernoulli factor

r(y){1−r(y)}r(y)\{1-r(y)\}

vanishes when r(y)r(y) approaches either zero or one. Response information is largest only where the response probability remains stochastic and its classifier-related sensitivity is substantial. The numerical experiments in § 9 will examine how this geometry translates into finite-sample acquisition decisions and classification performance.

Simulation study

We investigated the finite-sample behavior of the response-aware acquisition criterion in the binary logistic model of § 8. The simulation was designed to separate two questions. First, we examined oracle acquisition, in which the true model parameters are used to evaluate candidate query scores. This isolates the statistical effect of the acquisition criterion itself. Second, we considered an adaptive procedure in which the classifier and response parameters are estimated from the accumulated query-response data and the acquisition scores are updated sequentially.

Throughout,

Y∼N(0,1),Z∣Y=y∼Bernoulli⁡{τ(y)},(69)Y \sim N(0,1), \qquad Z \mid Y=y \sim\operatorname{Bernoulli}\{\tau(y)\}, \tag*{(69)}

with

τ(y)=logit⁡−1(−0.6+2y).\tau(y)=\operatorname{logit}^{-1}(-0.6+2y).

The true Bayes boundary is therefore

b0=0.3.b_0=0.3.

The oracle-response mechanism was

logit⁡r(y)=η0+η1U(y),(70)\operatorname{logit} r(y)=\eta_0+\eta_1U(y), \tag*{(70)}

where U(y)U(y) is the binary predictive entropy defined in (62). Three response regimes were considered:

η1∈{0,−3,−5},\eta_1 \in\{0,-3,-5\},

representing noninformative, moderately informative, and strongly informative response, respectively. For each value of η1\eta_1, the intercept η0\eta_0 was calibrated so that the marginal response probability satisfied

E{r(Y)}=0.40.\mathbb{E}\{r(Y)\}=0.40.

The resulting values were

η0=−0.405,0.920,1.786,\eta_0=-0.405,\qquad0.920,\qquad1.786,

with corresponding response probabilities at the Bayes boundary equal to 0.400, 0.239, and 0.157. Thus the scenarios differ in the informativeness of the response mechanism rather than simply in the expected number of returned labels.

For each Monte Carlo replication, an unlabeled pool of size 3000 was generated. Query budgets of

B∈{300,600,900}B \in\{300, 600, 900\}

were considered. These budgets include the common random pilot of 100 queried observations. Subsequent queries were acquired in batches of 25. The same potential class labels and oracle-response outcomes were used across methods within each replication, yielding paired comparisons.

Six acquisition procedures were compared: random sampling, entropy sampling, ordinary classification-risk-optimal acquisition that treats labels as always available, response-discounted risk-optimal acquisition, the proposed response-aware criterion, and a label-only ablation with full information.

The response-aware criterion uses

ψa(y)=ψaL(y)+ψaR(y).\psi_a(y) = \psi_a^L(y) + \psi_a^R(y).

The ablation retains the full query–response information matrix, including response-model nuisance parameters, but scores candidates using only ψaL(y)\psi_a^L(y). It therefore differs from response-discounted acquisition, which uses a label-based information matrix.

All six procedures used the same full observed-data likelihood after acquisition. The ablation examines the effect of including the explicit response-outcome term in sequential query selection while retaining full-information calculations and joint estimation. Once the query paths diverge, the procedures naturally have different accumulated data and fitted parameters.

For adaptive acquisition, the parameters

(α,β,η0,η1)(\alpha, \beta, \eta_0, \eta_1)

were re-estimated from the accumulated queried data using the likelihood

ℓ=∑i:Ai=1[Rilog⁡ri+(1−Ri)log⁡(1−ri)+Ri{Zilog⁡τi+(1−Zi)log⁡(1−τi)}].\ell= \sum_{i:A_i=1} \left[R_i \log r_i + (1-R_i)\log(1-r_i) + R_i\{Z_i\log\tau_i + (1-Z_i)\log(1-\tau_i)\}\right].

The principal performance measure was excess classification risk relative to the true Bayes classifier. We also recorded boundary-estimation error, the number of returned labels, response rates, and the locations of queried observations. Results below are based on 500 Monte Carlo replications. All fits evaluated at the reported budgets returned optimizer convergence code zero.

Likelihood optimization used box constraints

α∈[−8,8],β∈[0.05,8],η0,η1∈[−12,12].\alpha\in[-8, 8], \qquad\beta\in[0.05, 8], \qquad\eta_0, \eta_1 \in[-12, 12].

At each batch, the accumulated information matrices are sums over all previously queried observations, including the pilot. Ordinary risk sums J(yi)J(y_i), response-discounted risk sums r(yi)J(yi)r(y_i)J(y_i), and the response-aware and ablation rules both sum K(yi)K(y_i). These quantities are evaluated at the parameters used for scoring in that batch. For acquisition calculations, each accumulated information matrix MM was symmetrized and replaced by

M+[10−8+max⁡{0,10−8−λmin⁡(M)}]IM + \left[10^{-8} + \max\{0,10^{-8} - \lambda_{\min}(M)\}\right] I

before inversion. Optimizer convergence codes describe numerical termination and do not establish global likelihood maximization.

Acquisition geometry

Figure 1 illustrates the population geometry in the strongly informative-response regime. The returned-label and response components have markedly different spatial behavior. The raw classifier-related response-information component vanishes at the true decision boundary b0=0.3b_0 = 0.3, as predicted by (68). The nuisance-adjusted marginal response value need not vanish there. Numerically, the ordinary risk score is maximized near y=0.57y = 0.57, whereas the response-discounted and response-aware scores are maximized near y=1.30y = 1.30 and y=1.22y = 1.22, respectively. The response component itself is maximized near y=1.21y = 1.21.

Figure 1. Information and acquisition geometry in the strongly informative-response setting. The vertical dotted line marks the true Bayes boundary b0=0.3b_0 = 0.3. Panel (a) shows the class and response probabilities. Panel (b) shows the decision-relevant information from returned labels and response outcomes. Panel (c) shows the ordinary, response-discounted, and response-aware acquisition scores. Panel (d) shows the response-aware score and its two components. The scores in panels (c) and (d) are evaluated at the constant reference design a(y)≡0.5a(y) \equiv0.5. Each curve in panels (b)–(d) is divided by its own maximum over the plotting grid, so curve heights cannot be used to compare magnitudes across curves.

The figure therefore demonstrates that accounting for the response indicator does not merely rescale an uncertainty-based acquisition score. Instead, it changes the spatial allocation of queries by assigning value to observations whose response outcome is informative for classifier-relevant parameter directions.

Classification performance

Table 1 reports the mean excess classification risk for the adaptive procedures. Under noninformative response, the response-aware procedure provides no systematic advantage, as expected. In the corresponding oracle experiment, the response-aware and response-discounted procedures coincide exactly for every replication and budget, providing a direct numerical check of the theoretical reduction

ResponseBBRandomEntropyOrdinaryDiscountedAblationResponse-aware
Noninformative3002.807 (0.169)1.890 (0.125)1.816 (0.106)2.089 (0.125)1.918 (0.111)2.138 (0.157)
Noninformative6001.495 (0.105)0.827 (0.052)0.800 (0.049)0.778 (0.045)0.825 (0.050)0.896 (0.062)
Noninformative9000.977 (0.066)0.520 (0.034)0.537 (0.035)0.510 (0.032)0.519 (0.033)0.554 (0.035)
Moderate3001.800 (0.114)2.218 (0.146)1.927 (0.132)1.612 (0.093)1.700 (0.115)1.479 (0.106)
Moderate6000.763 (0.047)1.061 (0.065)0.990 (0.064)0.704 (0.043)0.711 (0.053)0.628 (0.045)
Moderate9000.520 (0.032)0.609 (0.037)0.616 (0.038)0.463 (0.030)0.467 (0.033)0.359 (0.023)
Strong3000.926 (0.061)1.966 (0.130)1.463 (0.096)0.656 (0.040)0.609 (0.039)0.578 (0.039)
Strong6000.457 (0.030)1.056 (0.068)0.837 (0.056)0.276 (0.017)0.275 (0.018)0.249 (0.016)
Strong9000.331 (0.024)0.634 (0.042)0.604 (0.038)0.176 (0.010)0.179 (0.011)0.172 (0.010)

Table 1. Mean excess classification risk multiplied by 10310^3 for adaptive acquisition over 500 replications, with Monte Carlo standard errors in parentheses. The ablation retains full information but omits the explicit response-outcome acquisition term. The smallest mean in each row is in bold.

qθ(y)=0⟹ψaR(y)=0.q_{\theta}(y) = 0 \quad\Longrightarrow\quad\psi_a^R(y) = 0.

The picture changes once oracle response is informative. Under moderate informative response, the response-aware procedure has the smallest mean excess risk at all three budgets. At B=900B = 900, for example, its mean excess risk is 0.359×10−30.359 \times10^{-3}, compared with 0.463×10−30.463 \times10^{-3} for the response-discounted rule, 0.520×10−30.520 \times10^{-3} for random acquisition, 0.609×10−30.609 \times10^{-3} for entropy sampling, and 0.616×10−30.616 \times10^{-3} for ordinary risk-optimal acquisition.

The same qualitative pattern appears under strongly informative response. The proposed procedure attains mean excess risks

0.578, 0.249, 0.172 in units of 10−310^{-3} at budgets 300, 600, and 900, respectively. The corresponding values for the response-discounted method are

0.656,0.276,0.176.0.656,\quad0.276,\quad0.176.

The gain relative to the response-discounted rule is therefore most visible at smaller and intermediate budgets and decreases as the available query budget grows.

The response-aware rule does not dominate in every setting. In the oracle study, it performs similarly to response-discounted acquisition in several settings and has slightly higher mean risk in some. The ablation below examines whether including the explicit response-outcome term improves acquisition when full information and joint estimation are retained.

Figure 2 gives the corresponding oracle and adaptive learning curves. The figure includes the label-only, full-information ablation alongside the five original acquisition

Figure 2. Mean excess classification risk as a function of the query budget under oracle and adaptive acquisition. Results are based on 500 Monte Carlo replications. Shaded regions represent approximate 95% Monte Carlo intervals for the mean.

procedures. The adaptive response-aware rule also improves substantially over random acquisition, particularly under strong informative response. At B=900B = 900, its excess risk is approximately 48% lower than that of random acquisition in the strong-response setting and approximately 31% lower in the moderate-response setting.

Contribution of the response-outcome term

The ablation retains the full information matrix and joint likelihood, but omits ψaR\psi_a^R from candidate scores. Supplementary Table S1 reports paired excess-risk differences, defined as the full criterion minus the ablation.

Under moderately informative response and B=900B = 900, the adaptive procedure reduced mean excess risk from 0.4666×10−30.4666 \times10^{-3} to 0.3589×10−30.3589 \times10^{-3}. The paired difference was −0.1077×10−3-0.1077 \times10^{-3}, with a pointwise 95% Monte Carlo tt interval of [−0.1726,−0.0428]×10−3[-0.1726, -0.0428] \times10^{-3} and Holm-adjusted p=0.0107p = 0.0107. The oracle difference was −0.1107×10−3-0.1107 \times10^{-3}, with interval [−0.1862,−0.0352]×10−3[-0.1862, -0.0352] \times10^{-3} and adjusted p=0.0373p = 0.0373. Holm adjustments cover the nine comparisons within each study.

Relative to the ablation’s mean excess risk, both reductions were approximately 23%. In absolute terms, the adaptive reduction was approximately 0.0108 percentage points of classification error. The full adaptive criterion had lower mean excess risk in all six informative-response settings, although the other five intervals included zero.

Under noninformative response, the oracle procedures had identical query paths in every saved trace and identical recorded outcomes across all 500 replications. The adaptive procedures need not coincide: estimation error can give the fitted response model a nonzero classifier-dependent component.

The evidence for an additional benefit is therefore confined to the moderate-response setting at the largest budget. The unresolved comparisons do not establish equivalence, and the results do not support a general advantage from increasing response dependence.

Where the procedures query

The risk differences are accompanied by substantial changes in the locations of the queried observations. Figure 3 shows the empirical distribution of queried feature values in the strongly informative-response regime. Entropy and ordinary-risk acquisition place most of their queries close to the Bayes boundary, where classification uncertainty is largest. In contrast, both response-discounted and response-aware acquisition shift substantial mass to off-boundary regions.

Figure 3. Empirical distribution of queried feature values in the strongly informative-response setting. The dotted vertical line marks the true Bayes boundary. Entropy and ordinary-risk sampling concentrate near the boundary, whereas response-discounted and response-aware acquisition place substantial query mass in informative off-boundary regions.

For the adaptive procedures at B=900B = 900, the mean absolute distance from the true boundary was 0.26, 0.27, 0.83, and 0.84 for entropy, ordinary-risk, response-discounted, and response-aware acquisition, respectively. Thus, unlike uncertainty-based rules, the response-aware design places substantial query mass away from the decision boundary, reflecting the joint value of returned labels and response information. The distinction is also visible in the realized response rates. Under strong informative response and B=900B = 900, entropy and ordinary-risk acquisition obtain responses for only about 20% of queried observations, whereas the response-discounted and response-aware procedures obtain response rates of approximately 40%. The proposed acquisition rule therefore avoids spending most of its budget in regions where the oracle almost always abstains, while retaining the additional information carried by the response outcome itself.

Real-data application

We evaluate the acquisition rules using the Consumer Complaints data from the AL-RCTA collection of Totakura et al. [21], which contains crowd-sourced annotations with explicit annotator abstentions. The analysis uses 2,999 complaints presented to each of ten human annotators, with the response indicator defined by

Rij=I{annotator j returned a nonzero annotation for complaint i}.R_{ij} = \mathbb{I}\{\text{annotator }j\text{ returned a nonzero annotation for complaint }i\}.

Empirical response rates varied substantially across annotators, from approximately 0.75 to 0.97. Response was also related to classification difficulty: in a preliminary out-of-fold analysis, the response rate decreased from 0.972 in the lowest predictive-entropy decile to 0.827 in the highest.

The theoretical model assumes that a returned label is correct, whereas the crowd annotations may themselves contain labeling errors. We therefore retain the observed human response indicator RijR_{ij}, but, when Rij=1R_{ij} = 1, reveal the benchmark product category rather than the annotator’s reported class. This preserves the genuine human abstention pattern while isolating informative response from label noise. Each annotator is treated as a separate oracle.

Complaint narratives were represented by TF–IDF features followed by a 12-dimensional truncated singular-value decomposition. One of the 3,000 shared complaints was excluded because its narrative was missing. TF–IDF, singular-value decomposition, feature scaling, and narrative-length scaling were constructed using all 2,999 narratives before generating the splits. The analysis therefore uses a transductive feature representation: test narratives contribute to unsupervised preprocessing, but test labels are withheld from fitting and acquisition. A multinomial logistic classifier was fitted jointly with

logit⁡{rj(y;θ,ηj)}=η0j+η1jUθ(y)+η2jL(y),\operatorname{logit}\{r_j(y;\theta,\eta_j)\} = \eta_{0j} + \eta_{1j}U_{\theta}(y) + \eta_{2j}L(y),

where Uθ(y)U_{\theta}(y) is normalized predictive entropy and L(y)L(y) is standardized log narrative length. We compared random, entropy, ordinary classification-risk, response-discounted risk, and response-aware acquisition. The classification task comprised six product categories. Across 30 replications, all methods shared a stratified split with approximately 70% of observations allocated to training and 30% to testing, a common pilot sample of 300 training observations, and the same realized oracle responses. Labels were subsequently queried in batches of 50 at additional budgets B∈{150,300,450,600}B \in\{150, 300, 450, 600\}. All methods used the same penalized joint likelihood, with quadratic penalty coefficients 1 for classifier slopes and 0.5 for response-model slopes; intercepts were unpenalized. Risk-based acquisition used an empirical kernel approximation to the active-boundary curvature, weighted by the mean fitted probability of the two leading classes. The Gaussian-kernel bandwidth was the larger of 0.20 and the empirical 25th percentile of their absolute log-probability gaps. The curvature matrix was trace-normalized. Information matrices used diagonal regularization of 10−310^{-3}. Before inversion, each eigenvalue of the symmetrized matrix was replaced by the larger of its original value and max⁡{10−8λmax⁡,10−10}\max\{10^{-8}\lambda_{\max}, 10^{-10}\}, where λmax⁡\lambda_{\max} is the largest eigenvalue. For ordinary-risk and response-discounted acquisition, the accumulated information matrix uses conditional classifier information from labels actually returned. Response-aware acquisition instead uses the fitted full query–response information matrix, including the response-model nuisance parameters. Thus, the response-discounted baseline is not obtained simply by removing ψaR\psi_a^R from the response-aware score while retaining the same information matrix. This experiment compares the complete acquisition rules rather than isolating the contribution of that additive term. Each batch selected the 50 highest-scoring available observations before refitting. This sequential implementation illustrates the criterion; it is not a numerical verification of the two-stage oracle-attainment theorem.

Acquisition rule150300450600
Random0.260 (0.0018)0.254 (0.0017)0.250 (0.0017)0.247 (0.0016)
Entropy0.246 (0.0022)0.241 (0.0017)0.238 (0.0015)0.235 (0.0015)
Ordinary risk0.258 (0.0019)0.244 (0.0015)0.237 (0.0016)0.232 (0.0016)
Response-discounted0.258 (0.0019)0.244 (0.0015)0.237 (0.0016)0.232 (0.0015)
Response-aware0.258 (0.0019)0.243 (0.0016)0.235 (0.0017)0.231 (0.0016)

Table 2. Held-out classification error in the Consumer Complaints application. Entries are means over 30 replications after averaging over the ten human oracles; parentheses contain Monte Carlo standard errors. The query budget excludes the common pilot sample of size 300. Additional query budget BB

Entropy sampling has the lowest mean test error at the two smallest budgets, whereas response-aware acquisition has the lowest mean error at B=450B = 450 and B=600B = 600. The mean paired differences in test error (response-aware minus response-discounted) are

−0.000052,−0.000892,−0.001152,−0.001152-0.000052,\quad-0.000892,\quad-0.001152,\quad-0.001152

at B=150,300,450,600B = 150, 300, 450, 600, respectively. Pointwise 95% Monte Carlo tt intervals exclude zero at the three larger budgets; at B=600B = 600, the interval is [−0.001679,−0.000624][-0.001679, -0.000624]. After Holm correction across the 16 budget–comparator comparisons, the advantages over response-discounted acquisition remain significant at B=450B = 450 and B=600B = 600, with adjusted pp-values 0.0202 and 0.00111, respectively. Response-aware acquisition is worse than entropy sampling at B=150B = 150; its advantage over entropy at B=600B = 600 does not remain significant under this correction. These intervals quantify variability across repeated splits and acquisition experiments for the fixed dataset and ten annotators, rather than uncertainty over a population of datasets or annotators.

Figure 4 shows a negative association between fitted predictive entropy and response probability in the pooled random-query reference cloud. The two risk-based procedures follow distinct, nonmonotone trajectories in their evolving fitted coordinates. Their first-batch mean locations are similar. In the final batch, response-aware acquisition has mean entropy 0.781 and fitted response probability 0.668, compared with 0.691 and 0.690, respectively, for response-discounted acquisition. These trajectories illustrate differences in sequential querying and model updating; they do not by themselves identify which information component causes the performance difference.

Figure 4. Uncertainty–response acquisition geometry for the Consumer Complaints application using worker W8, who had the largest observed abstention fraction. Panel (a) pools random-query locations across 30 replications. Panels (b) and (c) show response-discounted and response-aware acquisition in replication 6. Coordinates are predictive entropy and response probability evaluated under each method’s fitted model at the time of querying. Open circles trace batch-wise mean query locations; the diamond marks the final batch at additional budget B=600B = 600. Replication 6 was selected because its paired response-aware minus response-discounted test-error difference was closest to the median across the 30 W8 replications. The panels describe evolving fitted coordinates, rather than a common fixed-model representation.

The aggregate improvement is heterogeneous across human oracles. At B=600B = 600, the largest mean improvement over response-discounted acquisition occurs for W4, with paired difference −0.00483 and 95% Monte Carlo tt interval [−0.00768, −0.00198]. Only W4 remains significant after Holm correction across the ten worker-specific comparisons, with adjusted p=0.0167p = 0.0167. The other worker-specific comparisons are statistically unresolved after this correction.

Discussion and conclusions

This paper extends classification-risk-optimal label acquisition to an oracle that may abstain and whose response probability shares parameters with the classifier. The response outcome can then supply information even when no label is returned. After adjustment for response-model nuisance parameters, this contribution is

IR,eff(a)=E[a(Y)w(Y){qθ(Y)−Baqη(Y)}{qθ(Y)−Baqη(Y)}T].I_{R,\mathrm{eff}}(a) = E\left[a(Y)w(Y)\{q_{\theta}(Y)-B_aq_{\eta}(Y)\}\{q_{\theta}(Y)-B_aq_{\eta}(Y)\}^{\mathsf{T}}\right].

It is positive semidefinite and vanishes when the classifier-related response score is fully explained by the nuisance response score under the design-weighted measure. Parameter dependence alone is therefore insufficient for response outcomes to improve classification. The remaining information must also act in directions relevant to the decision boundary.

Combining the information from returned labels and response outcomes with the local curvature of zero–one risk gives the marginal query value

ψa(y)=ψaL(y)+ψaR(y).\psi_a(y) = \psi_a^L(y) + \psi_a^R(y).

The associated oracle allocation retains the convex trace-inverse structure of the underlying design problem and admits a threshold characterization under a fixed query budget. Response identifiability places an additional restriction on the allocation: queries must provide enough variation to estimate the response mechanism.

The simulations show how accounting for response changes query selection. In the strongly informative-response setting, response-aware and response-discounted acquisition both place substantial query mass away from the Bayes boundary, where response is more likely. Entropy and ordinary classification-risk acquisition remain more concentrated near the boundary. Under noninformative response, the response-aware and response-discounted oracle procedures coincide.

The ablation examines the additional effect of including the explicit response-outcome term while retaining full information and joint estimation. Under moderately informative response at B=900B = 900, the full criterion reduced mean excess risk in both the oracle and adaptive studies, with significance after the stated within-study Holm corrections. The other comparisons were statistically unresolved. The absolute improvement was small, and the results do not establish a general advantage across response mechanisms and budgets. Stronger response dependence alone does not imply a larger acquisition gain.

The Consumer Complaints replay provides evidence using observed human abstentions. Response-aware acquisition had the lowest mean test error at the two largest budgets, with improvements over response-discounted acquisition remaining significant after Holm correction at B=450B = 450 and B=600B = 600. Entropy sampling performed better at the smallest budget. The improvements were small and varied across annotators. This comparison evaluates complete acquisition rules with different information matrices; it does not isolate the contribution of the explicit response-outcome term.

The theory is local and parametric and relies on correct specification of the classification and response models. An association between response and predictive difficulty can motivate a shared-parameter model, but does not establish that the assumed link is correct. Performance under misspecification remains unresolved. The two-stage result applies to a pilot-selected design held fixed on fresh second-stage observations. It relies on uniform criterion consistency, mean-square asymptotic linearity, and a higher-moment bound. It does not establish these estimator conditions, or oracle attainment, for the sequential finite-pool implementation.

We assume that returned labels are correct and that

R⊥Z∣(Y,A=1).R \perp Z \mid(Y,A=1).

The replay therefore preserves human response patterns while revealing benchmark labels upon response. Using actual worker labels would require accounting for annotation error and, potentially, class-dependent response. Joint allocation across annotators with different costs, response mechanisms, and accuracies would require further development.

The main contribution is a classification-risk acquisition criterion that accounts for information in oracle response outcomes after nuisance adjustment. The numerical studies support its usefulness in some settings, while showing that the gains depend on the response mechanism and query budget.

Statements and Declarations

Code and data availability

Code and reproducibility instructions for the simulation studies, the Consumer Complaints analysis, and figure generation are available in the Risk-optimal active learning with informative abstention repository. The Consumer Complaints data are obtained from the original AL-RCTA source cited in the manuscript.

CRediT authorship contribution statement

F. Setoudehtazangi: Conceptualization, Methodology, Formal analysis, Software, Investigation, Visualization, Writing – original draft. G.J. McLachlan: Methodology, Supervision, Writing – review & editing.

Funding

This research received no specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Competing interests

The authors declare no competing interests.

Ethical approval

This study involved secondary analysis of an existing dataset and did not involve the recruitment of human participants or collection of new human-subject data by the authors.

Proofs

Proof of Proposition 2.1

For one observation, write

r(Y)=r(Y;θ,η).r(Y)=r(Y;\theta,\eta).

From (6)–(7), the score for ϕ=(θT,ηT)T\phi=(\theta^{\mathsf{T}},\eta^{\mathsf{T}})^{\mathsf{T}} can be written as

Sϕ=(SY0)+A[(R−r)q+RS~Z∣Y],(71)S_{\phi}= \begin{pmatrix} S_Y\\ 0 \end{pmatrix} +A\left[(R-r)q+R\widetilde{S}_{Z\mid Y}\right], \tag*{(71)}

where

q=(qθqη),S~Z∣Y=(SZ∣Y0).q= \begin{pmatrix} q_{\theta}\\ q_{\eta} \end{pmatrix}, \qquad \widetilde{S}_{Z\mid Y}= \begin{pmatrix} S_{Z\mid Y}\\ 0 \end{pmatrix}.

Conditional on Y=yY=y, the acquisition indicator satisfies

E(A∣Y=y)=a(y).E(A\mid Y=y)=a(y).

Moreover, by (2),

E(R−r∣Y=y,A=1)=0E(R-r\mid Y=y,A=1)=0

and

E{(R−r)2∣Y=y,A=1}=r(y){1−r(y)}=w(y).E\{(R-r)^2\mid Y=y,A=1\}=r(y)\{1-r(y)\}=w(y).

Since SZ∣YS_{Z\mid Y} is a conditional likelihood score,

E(SZ∣Y∣Y=y)=0.E(S_{Z\mid Y}\mid Y=y)=0.

Assumption (3) further implies that, conditional on Y=yY=y and A=1A=1, the response indicator RR and the class label ZZ are independent. It follows that

E[(R−r)RS~Z∣Y∣Y=y,A=1]=E{(R−r)R∣Y=y,A=1}E(S~Z∣Y∣Y=y)=0.(72)\begin{aligned} E\left[(R-r)R\widetilde{S}_{Z\mid Y}\mid Y=y,A=1\right] &=E\{(R-r)R\mid Y=y,A=1\}E\left(\widetilde{S}_{Z\mid Y}\mid Y=y\right)\\ &=0. \tag*{(72)} \end{aligned}

Thus the score contribution from the response indicator is orthogonal, conditional on YY, to the score contribution from the returned class label. The conditional second moment of the returned-label term is

E[RS~Z∣YS~Z∣YT∣Y=y,A=1]=r(y)E[S~Z∣YS~Z∣YT∣Y=y]=r(y)J~(y),(73)\begin{aligned} E\left[R\widetilde{S}_{Z\mid Y}\widetilde{S}_{Z\mid Y}^{\mathsf{T}}\mid Y=y,A=1\right] =r(y)E\left[\widetilde{S}_{Z\mid Y}\widetilde{S}_{Z\mid Y}^{\mathsf{T}}\mid Y=y\right] \\ &= r(y)\widetilde{J}(y), \tag*{(73)} \end{aligned}

where

J~(y)=(J(y)000).\widetilde{J}(y)= \begin{pmatrix} J(y) & 0 \\ 0 & 0 \end{pmatrix}.

Similarly,

E[(R−r)2qqT∣Y=y,A=1]=w(y)q(y)q(y)T.(74)E\left[(R-r)^2qq^{\mathsf T}\mid Y=y,A=1\right]=w(y)q(y)q(y)^{\mathsf T}. \tag*{(74)}

Combining (S1.2)–(S1.4) gives the conditional Fisher information generated by a query:

K(y)=r(y)J~(y)+w(y)q(y)q(y)T.(75)K(y)=r(y)\widetilde{J}(y)+w(y)q(y)q(y)^{\mathsf T}. \tag*{(75)}

It remains to combine the query contribution with the information in the feature vector. The feature score satisfies

E(SY)=0.E(S_Y)=0.

Conditional on YY, the mean of the query-score contribution in (S1.1) is zero:

E[(R−r)q+RS~Z∣Y∣Y,A=1]=qE(R−r∣Y,A=1)+E(R∣Y,A=1)E(S~Z∣Y∣Y)=0.E\left[(R-r)q+R\widetilde{S}_{Z\mid Y}\mid Y,A=1\right] =qE(R-r\mid Y,A=1)+E(R\mid Y,A=1)E(\widetilde{S}_{Z\mid Y}\mid Y)=0.

Hence the feature score is orthogonal to the query-score contribution. Since AA is generated according to the known parameter-free design (1),

E[AK(Y)∣Y]=a(Y)K(Y).E[AK(Y)\mid Y]=a(Y)K(Y).

Therefore

I(a)=E(SϕSϕT)=(IY(θ)000)+Eθ{a(Y)K(Y)}=IY+Eθ{a(Y)K(Y)},\begin{aligned} \mathcal{I}(a)=E(S_{\phi}S_{\phi}^{\mathsf T}) \\ &=\begin{pmatrix} \mathcal{I}_Y(\theta) & 0 \\ 0 & 0 \end{pmatrix}+E_{\theta}\{a(Y)K(Y)\} \\ &=\mathcal{I}_Y+E_{\theta}\{a(Y)K(Y)\}, \end{aligned}

which is (9). Substitution of (S1.5) yields (10), completing the proof. □

Proof of Proposition 3.1

For notational simplicity, write

a=a(Y),w=w(Y),qθ=qθ(Y),qη=qη(Y).a=a(Y),\qquad w=w(Y),\qquad q_{\theta}=q_{\theta}(Y),\qquad q_{\eta}=q_{\eta}(Y).

By (19),

IR,eff(a)=Cθθ−CθηCηη−1Cηθ.I_{R,\mathrm{eff}}(a)=C_{\theta\theta}-C_{\theta\eta}C_{\eta\eta}^{-1}C_{\eta\theta}.

Let

Ba=CθηCηη−1.B_a=C_{\theta\eta}C_{\eta\eta}^{-1}.

Since CηηC_{\eta\eta} is symmetric and positive definite,

BaCηη=Cθη,CηηBaT=Cηθ.B_a C_{\eta\eta} = C_{\theta\eta}, \qquad C_{\eta\eta} B_a^{\mathsf{T}} = C_{\eta\theta}.

Consider

Eθ[aw(qθ−Baqη)(qθ−Baqη)T].\mathbb{E}_{\theta}\left[aw(q_{\theta}-B_aq_{\eta})(q_{\theta}-B_aq_{\eta})^{\mathsf{T}}\right].

Expanding the outer product and using (3.4)–(3.6) gives

Eθ[aw(qθ−Baqη)(qθ−Baqη)T]=Cθθ−BaCηθ−CθηBaT+BaCηηBaT.\begin{aligned} \mathbb{E}_{\theta}\left[aw(q_{\theta}-B_aq_{\eta})(q_{\theta}-B_aq_{\eta})^{\mathsf{T}}\right] &= C_{\theta\theta}-B_aC_{\eta\theta}-C_{\theta\eta}B_a^{\mathsf{T}}+B_aC_{\eta\eta}B_a^{\mathsf{T}}. \end{aligned}

Because

BaCηθ=CθηCηη−1Cηθ,B_aC_{\eta\theta}=C_{\theta\eta}C_{\eta\eta}^{-1}C_{\eta\theta},

and

BaCηηBaT=CθηCηη−1Cηθ,B_aC_{\eta\eta}B_a^{\mathsf{T}}=C_{\theta\eta}C_{\eta\eta}^{-1}C_{\eta\theta},

the preceding expression reduces to

Cθθ−CθηCηη−1Cηθ=IR,eff(a),C_{\theta\theta}-C_{\theta\eta}C_{\eta\eta}^{-1}C_{\eta\theta}=I_{R,\mathrm{eff}}(a),

which proves (21).

For any v∈Rdv \in\mathbb{R}^{d},

vTIR,eff(a)v=Eθ[aw{vT(qθ−Baqη)}2]≥0.v^{\mathsf{T}}I_{R,\mathrm{eff}}(a)v=\mathbb{E}_{\theta}\left[aw\left\{v^{\mathsf{T}}(q_{\theta}-B_aq_{\eta})\right\}^{2}\right]\geq0.

Hence

IR,eff(a)⪰0.I_{R,\mathrm{eff}}(a)\succeq0.

It remains to characterize when the matrix is zero. If

qθ(Y)=Baqη(Y)q_{\theta}(Y)=B_aq_{\eta}(Y)

μa\mu_a-almost surely, then (21) immediately gives IR,eff(a)=0I_{R,\mathrm{eff}}(a)=0.

Conversely, suppose that IR,eff(a)=0I_{R,\mathrm{eff}}(a)=0. For every v∈Rdv \in\mathbb{R}^{d},

Eθ[a(Y)w(Y){vT[qθ(Y)−Baqη(Y)]}2]=0.\mathbb{E}_{\theta}\left[a(Y)w(Y)\left\{v^{\mathsf{T}}[q_{\theta}(Y)-B_aq_{\eta}(Y)]\right\}^{2}\right]=0.

The integrand is nonnegative, and therefore

vT[qθ(Y)−Baqη(Y)]=0v^{\mathsf{T}}[q_{\theta}(Y)-B_aq_{\eta}(Y)]=0

μa\mu_a-almost surely for every v∈Rdv \in\mathbb{R}^{d}. Taking vv successively to be the elements of the canonical basis of Rd\mathbb{R}^{d} yields

qθ(Y)−Baqη(Y)=0q_{\theta}(Y)-B_aq_{\eta}(Y)=0

μa\mu_a-almost surely. This establishes (22) and completes the proof.

Proof of Proposition 4.1

Let

at(y)=a(y)+th(y)a_t(y) = a(y) + th(y)

for tt in a neighborhood of zero such that ata_t remains admissible. By Proposition 2.1,

I(at)=IY+Eθ0{at(Y)K(Y)},\mathcal{I}(a_t) = \mathcal{I}_Y + E_{\theta_0}\{a_t(Y)K(Y)\},

and hence

ddtI(at)∣t=0=Eθ0{h(Y)K(Y)}.(76)\left.\frac{d}{dt}\mathcal{I}(a_t)\right|_{t=0} = E_{\theta_0}\{h(Y)K(Y)\}. \tag*{(76)}

For any differentiable nonsingular matrix-valued function M(t)M(t),

ddtM(t)−1=−M(t)−1dM(t)dtM(t)−1.\frac{d}{dt}M(t)^{-1} = -M(t)^{-1}\frac{dM(t)}{dt}M(t)^{-1}.

Applying this identity to M(t)=I(at)M(t)=\mathcal{I}(a_t) and using (S1.6) gives

ddtΦ(at)∣t=0=ddttr⁡{H~RI(at)−1}∣t=0=−tr⁡[H~RI(a)−1Eθ0{h(Y)K(Y)}I(a)−1].\left.\frac{d}{dt}\Phi(a_t)\right|_{t=0} = \left.\frac{d}{dt}\operatorname{tr}\left\{\widetilde{H}_R\mathcal{I}(a_t)^{-1}\right\}\right|_{t=0} = -\operatorname{tr}\left[\widetilde{H}_R\mathcal{I}(a)^{-1}E_{\theta_0}\{h(Y)K(Y)\}\mathcal{I}(a)^{-1}\right].

By linearity of expectation and trace,

ddtΦ(at)∣t=0=−Eθ0[h(Y)tr⁡{H~RI(a)−1K(Y)I(a)−1}]=−Eθ0{h(Y)ψa(Y)},\left.\frac{d}{dt}\Phi(a_t)\right|_{t=0} = -E_{\theta_0}\left[h(Y)\operatorname{tr}\left\{\widetilde{H}_R\mathcal{I}(a)^{-1}K(Y)\mathcal{I}(a)^{-1}\right\}\right] = -E_{\theta_0}\{h(Y)\psi_a(Y)\},

which establishes (34).

It remains to obtain the decomposition of ψa(y)\psi_a(y). From (8),

K(y)=r(y)J~(y)+w(y)q(y)q(y)T.K(y) = r(y)\widetilde{J}(y) + w(y)q(y)q(y)^{\mathsf{T}}.

Substituting this expression into (30) and using linearity of the trace yields

ψa(y)=r(y)tr⁡{H~RI(a)−1J~(y)I(a)−1}+w(y)tr⁡{H~RI(a)−1q(y)q(y)TI(a)−1}.\psi_a(y) = r(y)\operatorname{tr}\left\{\widetilde{H}_R\mathcal{I}(a)^{-1}\widetilde{J}(y)\mathcal{I}(a)^{-1}\right\} + w(y)\operatorname{tr}\left\{\widetilde{H}_R\mathcal{I}(a)^{-1}q(y)q(y)^{\mathsf{T}}\mathcal{I}(a)^{-1}\right\}.

Using

tr⁡{MqqTN}=qTNMq\operatorname{tr}\{Mqq^{\mathsf{T}}N\} = q^{\mathsf{T}}NMq

with the symmetry of I(a)−1\mathcal{I}(a)^{-1} and H~R\widetilde{H}_R, the second term becomes

w(y)q(y)TI(a)−1H~RI(a)−1q(y).w(y)q(y)^{\mathsf{T}}\mathcal{I}(a)^{-1}\widetilde{H}_R\mathcal{I}(a)^{-1}q(y).

Therefore

ψa(y)=ψaL(y)+ψaR(y),\psi_a(y)=\psi_a^L(y)+\psi_a^R(y),

with ψaL\psi_a^L and ψaR\psi_a^R given by (32) and (33), respectively. Finally, since w(y)≥0w(y)\ge0, H~R⪰0\widetilde{H}_R\succeq0, and I(a)−1\mathcal{I}(a)^{-1} is symmetric,

ψaR(y)=w(y)∥H~R1/2I(a)−1q(y)∥2≥0.\psi_a^R(y)=w(y)\left\|\widetilde{H}_R^{1/2}\mathcal{I}(a)^{-1}q(y)\right\|^2\ge0.

This completes the proof. □

Proof of Proposition 5.1

Let a0a_0 and a1a_1 be two designs for which the corresponding information matrices are positive definite, and let

at=(1−t)a0+ta1,0≤t≤1.a_t=(1-t)a_0+ta_1,\qquad0\le t\le1.

By Proposition 2.1,

I(at)=(1−t)I(a0)+tI(a1).\mathcal{I}(a_t)=(1-t)\mathcal{I}(a_0)+t\mathcal{I}(a_1).

Set

Mt=I(at),D=I(a1)−I(a0).M_t=\mathcal{I}(a_t),\qquad D=\mathcal{I}(a_1)-\mathcal{I}(a_0).

Then

dMtdt=D.\frac{dM_t}{dt}=D.

Using

ddtMt−1=−Mt−1DMt−1,\frac{d}{dt}M_t^{-1}=-M_t^{-1}DM_t^{-1},

we obtain

ddtΦ(at)=−tr⁡(H~RMt−1DMt−1).\frac{d}{dt}\Phi(a_t)=-\operatorname{tr}\left(\widetilde{H}_R M_t^{-1}DM_t^{-1}\right).

Differentiating once more gives

d2dt2Φ(at)=2tr⁡(H~RMt−1DMt−1DMt−1).\frac{d^2}{dt^2}\Phi(a_t)=2\operatorname{tr}\left(\widetilde{H}_R M_t^{-1}DM_t^{-1}DM_t^{-1}\right).

Let

G=Mt−1/2DMt−1/2,C=Mt−1/2H~RMt−1/2.G=M_t^{-1/2}DM_t^{-1/2},\qquad C=M_t^{-1/2}\widetilde{H}_R M_t^{-1/2}.

Since H~R⪰0\widetilde{H}_R\succeq0, we have C⪰0C\succeq0, while GG is symmetric. Cyclic invariance of the trace yields

d2dt2Φ(at)=2tr⁡(CG2).\frac{d^2}{dt^2}\Phi(a_t)=2\operatorname{tr}(CG^2).

Because G2⪰0G^2\succeq0 and C⪰0C\succeq0,

tr⁡(CG2)=tr⁡(C1/2G2C1/2)≥0.\operatorname{tr}(CG^2)=\operatorname{tr}(C^{1/2}G^2C^{1/2})\ge0.

Hence

d2dt2Φ(at)≥0,\frac{d^2}{dt^2}\Phi(a_t) \ge0,

which proves convexity along every line segment in the positive-definite information domain. Therefore Φ\Phi is convex on any convex subset of that domain. □

Proof of Theorem 5.2

Let a∗a^* be optimal and write ψ=ψa∗\psi=\psi_{a^*}. All probabilities and expectations in this proof are taken under Pθ0P_{\theta_0}. For ϵ>0\epsilon>0, choose measurable sets B0⊆{a∗≤1−ϵ}B_0\subseteq\{a^*\le1-\epsilon\} and B1⊆{a∗≥ϵ}B_1\subseteq\{a^*\ge\epsilon\} of positive probability, and put

h(y)=1B0(y)Pθ0(B0)−1B1(y)Pθ0(B1).h(y)=\frac{\mathbf{1}_{B_0}(y)}{P_{\theta_0}(B_0)}-\frac{\mathbf{1}_{B_1}(y)}{P_{\theta_0}(B_1)}.

Since Eθ0{h(Y)}=0E_{\theta_0}\{h(Y)\}=0, this exchange preserves the budget. The slack on B0B_0 and B1B_1 ensures that a∗+tha^*+th remains in [0,1][0,1] for sufficiently small t>0t>0. Its information matrix remains positive definite by continuity. The right derivative, given by the same differentiation as in Proposition 4.1, therefore satisfies

0≤ddtΦ(a∗+th)∣t=0+=Eθ0{ψ(Y)∣Y∈B1}−Eθ0{ψ(Y)∣Y∈B0}.0\le\left.\frac{d}{dt}\Phi(a^*+th)\right|_{t=0+}=E_{\theta_0}\{\psi(Y)\mid Y\in B_1\}-E_{\theta_0}\{\psi(Y)\mid Y\in B_0\}.

It follows that

ess sup⁡{a∗<1}ψ≤ess inf⁡{a∗>0}ψ.\operatorname*{ess\,sup}_{\{a^*<1\}}\psi\le\operatorname*{ess\,inf}_{\{a^*>0\}}\psi.

Indeed, if this inequality failed, there would be positive-probability subsets of {a∗<1}\{a^*<1\} and {a∗>0}\{a^*>0\} on which the scores were separated in the opposite order. Intersecting these subsets with {a∗≤1−ϵ}\{a^*\le1-\epsilon\} and {a∗≥ϵ}\{a^*\ge\epsilon\}, respectively, preserves positive probability for sufficiently small ϵ\epsilon, contradicting the exchange inequality. Both sets have positive probability because 0<ρ<10<\rho<1. Since the score is nonnegative and integrable, the displayed interval contains a finite λ\lambda. For such a λ\lambda, ψ≤λ\psi\le\lambda almost surely on {a∗<1}\{a^*<1\}, and ψ≥λ\psi\ge\lambda almost surely on {a∗>0}\{a^*>0\}. These inequalities give the stated threshold conditions, including equality wherever 0<a∗<10<a^*<1.

For sufficiency, let aa be any feasible design and interpret DΦ(a∗)[a−a∗]D\Phi(a^*)[a-a^*] as the right derivative along the feasible segment from a∗a^* to aa. By convexity,

Φ(a)≥Φ(a∗)+DΦ(a∗)[a−a∗].\Phi(a)\ge\Phi(a^*)+D\Phi(a^*)[a-a^*].

The derivative term is

DΦ(a∗)[a−a∗]=−Eθ0[{a(Y)−a∗(Y)}ψa∗(Y)].D\Phi(a^*)[a-a^*]=-E_{\theta_0}\left[\{a(Y)-a^*(Y)\}\psi_{a^*}(Y)\right].

Since both designs satisfy the same budget,

Eθ0{a(Y)−a∗(Y)}=0,E_{\theta_0}\{a(Y)-a^*(Y)\}=0,

and therefore

DΦ(a∗)[a−a∗]=Eθ0[{a(Y)−a∗(Y)}{λ−ψa∗(Y)}].D\Phi(a^*)[a-a^*]=E_{\theta_0}\left[\{a(Y)-a^*(Y)\}\{\lambda-\psi_{a^*}(Y)\}\right].

If a∗(y)=0a^*(y)=0, then a(y)−a∗(y)≥0a(y)-a^*(y)\ge0 and λ−ψa∗(y)≥0\lambda-\psi_{a^*}(y)\ge0. If a∗(y)=1a^*(y)=1, then a(y)−a∗(y)≤0a(y)-a^*(y)\le0 and λ−ψa∗(y)≤0\lambda-\psi_{a^*}(y)\le0. If 0<a∗(y)<10<a^*(y)<1, then λ−ψa∗(y)=0\lambda-\psi_{a^*}(y)=0. Thus the integrand is nonnegative almost surely, so

DΦ(a∗)[a−a∗]≥0.D\Phi(a^*)[a-a^*]\ge0.

Consequently,

Φ(a)≥Φ(a∗)\Phi(a)\ge\Phi(a^*)

for every feasible aa, proving global optimality. □\square

Proof of Proposition 5.3

From (37),

Cηη(a)=Eθ0[a(Y)w(Y)qη(Y)qη(Y)⊤].C_{\eta\eta}(a)=E_{\theta_0}\left[a(Y)w(Y)q_\eta(Y)q_\eta(Y)^\top\right].

Therefore, for any v∈Rqv\in\mathbb{R}^q,

v⊤Cηη(a)v=Eθ0[a(Y)w(Y)v⊤qη(Y)qη(Y)⊤v]=Eθ0[a(Y)w(Y){v⊤qη(Y)}2],v^\top C_{\eta\eta}(a)v=E_{\theta_0}\left[a(Y)w(Y)v^\top q_\eta(Y)q_\eta(Y)^\top v\right] =E_{\theta_0}\left[a(Y)w(Y)\{v^\top q_\eta(Y)\}^2\right],

which proves (38).

Suppose first that there exists v≠0v\ne0 such that

v⊤qη(Y)=0v^\top q_\eta(Y)=0

μa\mu_a-almost surely. Then

v⊤Cηη(a)v=0,v^\top C_{\eta\eta}(a)v=0,

so Cηη(a)C_{\eta\eta}(a) cannot be positive definite.

Conversely, if Cηη(a)C_{\eta\eta}(a) is not positive definite, its positive semidefiniteness implies that there exists v≠0v\ne0 for which

v⊤Cηη(a)v=0.v^\top C_{\eta\eta}(a)v=0.

Hence

Eθ0[a(Y)w(Y){v⊤qη(Y)}2]=0.E_{\theta_0}\left[a(Y)w(Y)\{v^\top q_\eta(Y)\}^2\right]=0.

The integrand is nonnegative, and therefore

v⊤qη(Y)=0v^\top q_\eta(Y)=0

μa\mu_a-almost surely. This establishes the equivalence.

For the model

logit⁡r(Y;θ,η)=η0+η1U(Y;θ),(77)\operatorname{logit} r(Y;\theta,\eta)=\eta_0+\eta_1U(Y;\theta), \tag*{(77)}

we have

qη(Y)=(1U(Y;θ)).q_{\eta}(Y)=\begin{pmatrix}1\\ U(Y;\theta)\end{pmatrix}.

Let expectation under the normalized measure μa\mu_a be denoted by EμaE_{\mu_a}. Apart from the positive normalizing constant Eθ0{a(Y)w(Y)}E_{\theta_0}\{a(Y)w(Y)\}, the nuisance-information matrix is

Eμa[(1U)(1 U)]=(1Eμa(U)Eμa(U)Eμa(U2)).E_{\mu_a}\left[\begin{pmatrix}1\\ U\end{pmatrix}(1\ U)\right] = \begin{pmatrix} 1 & E_{\mu_a}(U)\\ E_{\mu_a}(U) & E_{\mu_a}(U^2) \end{pmatrix}.

Its determinant is

Eμa(U2)−Eμa(U)2=Var⁡μa(U).E_{\mu_a}(U^2)-E_{\mu_a}(U)^2=\operatorname{Var}_{\mu_a}(U).

Thus, when Eθ0{a(Y)w(Y)}>0E_{\theta_0}\{a(Y)w(Y)\}>0,

Cηη(a)≻0⟺Var⁡μa{U(Y;θ0)}>0,C_{\eta\eta}(a)\succ0\quad\Longleftrightarrow\quad\operatorname{Var}_{\mu_a}\{U(Y;\theta_0)\}>0,

which proves (41). □

Proof of Theorem 6.1

Oracle criterion value.

The approximate minimization condition and the definition of Δn\Delta_n give

Φ(a^n)≤Φ^n(a^n)+Δn≤Φ^n(a∗)+εn+Δn≤Φ(a∗)+εn+2Δn.\Phi(\widehat a_n)\leq\widehat\Phi_n(\widehat a_n)+\Delta_n\leq\widehat\Phi_n(a^*)+\varepsilon_n+\Delta_n\leq\Phi(a^*)+\varepsilon_n+2\Delta_n.

Since a^n∈D\widehat a_n\in\mathcal{D} and a∗a^* minimizes Φ\Phi there, Φ(a^n)≥Φ(a∗)\Phi(\widehat a_n)\geq\Phi(a^*). This proves (51). This part does not require independence of the observations.

Uniform fixed-design second moments.

For deterministic a∈Da\in\mathcal{D}, put

Tn,a=n(ϕ^n,a−ϕ0),Wn,a=I(a)−11n∑i=1nSa(Oi;ϕ0).T_{n,a}=\sqrt{n}(\widehat\phi_{n,a}-\phi_0),\qquad W_{n,a}=\mathcal{I}(a)^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^{n}S_a(O_i;\phi_0).

The iid score identities imply exactly

EaWn,a=0,Ea(Wn,aWn,aT)=I(a)−1.E_aW_{n,a}=0,\qquad E_a(W_{n,a}W_{n,a}^{\mathsf T})=\mathcal{I}(a)^{-1}.

Writing d+qd+q for the dimension of ϕ\phi, the information lower bound therefore gives

sup⁡a∈DEa∥Wn,a∥2=sup⁡a∈Dtr⁡{I(a)−1}≤d+qκ.\sup_{a\in\mathcal{D}}E_a\lVert W_{n,a}\rVert^2 = \sup_{a\in\mathcal{D}}\operatorname{tr}\{\mathcal{I}(a)^{-1}\} \leq\frac{d+q}{\kappa}.

By (52), Tn,a=Wn,a+ζn,aT_{n,a}=W_{n,a}+\zeta_{n,a}, with the squared remainder tending to zero uniformly in expectation. Cauchy–Schwarz applied to the two cross terms yields

sup⁡a∈D∥Ea(Tn,aTn,aT)−I(a)−1∥⟶0,\sup_{a\in\mathcal{D}}\left\lVert E_a(T_{n,a}T_{n,a}^{\mathsf T})-\mathcal{I}(a)^{-1}\right\rVert\longrightarrow0,

where ∥⋅∥\lVert\cdot\rVert denotes the operator norm for matrices. With en,a=θ^n,a−θ0e_{n,a}=\hat{\theta}_{n,a}-\theta_0, taking classification blocks and using the block-inverse identity gives

sup⁡a∈D∥nEa(en,aen,aT)−Ieff(a)−1∥⟶0.\sup_{a\in\mathcal{D}}\left\lVert nE_a(e_{n,a}e_{n,a}^{\mathsf T})-I_{\mathrm{eff}}(a)^{-1}\right\rVert\longrightarrow0.

Consequently,

sup⁡a∈D∣nEa(en,aTHRen,a)−Φ(a)∣⟶0.\sup_{a\in\mathcal{D}}\left|nE_a(e_{n,a}^{\mathsf T}H_Re_{n,a})-\Phi(a)\right|\longrightarrow0.

Uniform control of the risk remainder.

Define

b(h)=R(θ0+h)−R∗−12hTHRhb(h)=R(\theta_0+h)-R^*-\frac{1}{2}h^{\mathsf T}H_Rh

for parameter perturbations in the domain of the classifier. By (23), for every ϵ>0\epsilon>0 there is a radius s>0s>0 such that ∣b(h)∣≤ϵ∥h∥2|b(h)|\leq\epsilon\lVert h\rVert^2 whenever ∥h∥≤s\lVert h\rVert\leq s. Condition (53) gives a finite constant CC such that, for all sufficiently large nn,

sup⁡a∈DEa∥nen,a∥2+δ≤C.\sup_{a\in\mathcal{D}}E_a\lVert\sqrt{n}e_{n,a}\rVert^{2+\delta}\leq C.

On the complement of the radius-ss ball, boundedness of classification risk implies ∣b(e)∣≤1+12∥HR∥∥e∥2|b(e)|\leq1+\frac{1}{2}\lVert H_R\rVert\lVert e\rVert^2. Markov’s inequality gives

sup⁡a∈DnPa(∥en,a∥>s)≤Cs−(2+δ)n−δ/2,\sup_{a\in\mathcal{D}}nP_a(\lVert e_{n,a}\rVert>s)\leq Cs^{-(2+\delta)}n^{-\delta/2},
sup⁡a∈DnEa{∥en,a∥21(∥en,a∥>s)}≤Cs−δn−δ/2.\sup_{a\in\mathcal{D}}nE_a\{\lVert e_{n,a}\rVert^2 1(\lVert e_{n,a}\rVert>s)\}\leq Cs^{-\delta}n^{-\delta/2}.

Both bounds tend to zero. Inside the ball,

sup⁡a∈DnEa{∣b(en,a)∣1(∥en,a∥≤s)}≤ϵsup⁡a∈DEa∥nen,a∥2.\sup_{a\in\mathcal{D}}nE_a\{|b(e_{n,a})|1(\lVert e_{n,a}\rVert\leq s)\}\leq\epsilon\sup_{a\in\mathcal{D}}E_a\lVert\sqrt{n}e_{n,a}\rVert^2.

The last supremum is bounded, by either the moment condition or the preceding second-moment calculation. Taking the limit superior and then letting ϵ↓0\epsilon\downarrow0 proves

sup⁡a∈DnEa∣b(en,a)∣⟶0.\sup_{a\in\mathcal{D}}nE_a|b(e_{n,a})|\longrightarrow0.

Together with the quadratic term, this establishes the uniform expansion

sup⁡a∈D∣Ea{R(θ^n,a)−R∗}−Φ(a)2n∣=o(n−1).(78)\sup_{a\in\mathcal{D}}\left|E_a\{R(\hat{\theta}_{n,a})-R^*\}-\frac{\Phi(a)}{2n}\right|=o(n^{-1}). \tag*{(78)}

Conditioning on the pilot.

For deterministic aa, let gn(a)=Ea{R(θ^n,a)−R∗}g_n(a)=E_a\{R(\hat{\theta}_{n,a})-R^*\}. The selected design is F0,n\mathcal{F}_{0,n}-measurable, and the estimator uses the pilot only through this design. The conditional product law (45) therefore gives the exact identity

E{R(θ^n,a^n)−R∗∣F0,n}=gn(a^n).E\{R(\widehat{\theta}_{n,\widehat{a}_n})-R^* \mid\mathcal{F}_{0,n}\}=g_n(\widehat{a}_n).

Set rn=gn(a^n)−Φ(a^n)/(2n)r_n=g_n(\widehat{a}_n)-\Phi(\widehat{a}_n)/(2n). Equation (S1.7) implies

E∣rn∣≤sup⁡a∈D∣gn(a)−Φ(a)2n∣=o(n−1),E|r_n|\leq\sup_{a\in\mathcal{D}}\left|g_n(a)-\frac{\Phi(a)}{2n}\right|=o(n^{-1}),

which proves (54). This step is why uniform fixed-design control is needed instead of an expansion only for each individual deterministic design.

Averaging over the pilot.

Taking expectations now gives

E{R(θ^n,a^n)}−R∗=12nE{Φ(a^n)}+o(n−1).E\{R(\widehat{\theta}_{n,\widehat{a}_n})\}-R^*=\frac{1}{2n}E\{\Phi(\widehat{a}_n)\}+o(n^{-1}).

On D\mathcal{D},

0≤Φ(a)≤κ−1tr⁡(H~R).0\leq\Phi(a)\leq\kappa^{-1}\operatorname{tr}(\widetilde{H}_{R}).

The convergence in probability in (51), together with this deterministic bound, implies EΦ(a^n)→Φ(a∗)E\Phi(\widehat{a}_n)\to\Phi(a^*). This proves (55). Finally, Nn=n+mnN_n=n+m_n satisfies Nn/n→1N_n/n\to1, so replacing nn by NnN_n in the leading denominator changes the expression by o(Nn−1)o(N_n^{-1}). This change of denominator concerns sample-size accounting only; the proof has not treated the pooled two-stage observations as iid. □

Derivation of the binary logistic risk expansion

For the binary logistic model,

τθ(y)=exp⁡(α+βy)1+exp⁡(α+βy),β>0,\tau_\theta(y)=\frac{\exp(\alpha+\beta y)}{1+\exp(\alpha+\beta y)},\qquad\beta>0,

the Bayes classifier under parameter θ\theta changes class at

b(θ)=−αβ.b(\theta)=-\frac{\alpha}{\beta}.

Let

b0=b(θ0)=−α0β0.b_0=b(\theta_0)=-\frac{\alpha_0}{\beta_0}.

For θ\theta sufficiently close to θ0\theta_0, the classifier based on θ\theta differs from the Bayes classifier only on the interval between b(θ)b(\theta) and b0b_0. Hence

R(θ)−R∗=∣∫b0b(θ){2τθ0(y)−1}f(y) dy∣.(79)R(\theta)-R^*=\left|\int_{b_0}^{b(\theta)}\{2\tau_{\theta_0}(y)-1\}f(y)\,dy\right|. \tag*{(79)}

Since

τθ0(b0)=12,\tau_{\theta_0}(b_0)=\frac{1}{2},

the integrand vanishes at b0b_0. Moreover,

ddyτθ0(y)=β0τθ0(y){1−τθ0(y)},\frac{d}{dy}\tau_{\theta_0}(y)=\beta_0\tau_{\theta_0}(y)\{1-\tau_{\theta_0}(y)\},

so

ddy{2τθ0(y)−1}∣y=b0=β02.\left.\frac{d}{dy}\{2\tau_{\theta_0}(y)-1\}\right|_{y=b_0}=\frac{\beta_0}{2}.

Therefore,

{2τθ0(y)−1}f(y)=β0f(b0)2(y−b0)+o(∣y−b0∣).\{2\tau_{\theta_0}(y)-1\}f(y)=\frac{\beta_0f(b_0)}{2}(y-b_0)+o(|y-b_0|).

Substitution into (S1.8) gives

R(θ)−R∗=β0f(b0)4{b(θ)−b0}2+o({b(θ)−b0}2).(80)R(\theta)-R^*=\frac{\beta_0f(b_0)}{4}\{b(\theta)-b_0\}^2+o\left(\{b(\theta)-b_0\}^2\right). \tag*{(80)}

Now let

θ=θ0+h,h=(hαhβ).\theta=\theta_0+h,\qquad h=\begin{pmatrix}h_\alpha\\h_\beta\end{pmatrix}.

The gradient of the boundary function is

∇θb(θ)=(−1/βα/β2).\nabla_\theta b(\theta)=\begin{pmatrix}-1/\beta\\\alpha/\beta^2\end{pmatrix}.

At θ0\theta_0, using α0=−β0b0\alpha_0=-\beta_0b_0,

∇θb(θ0)=−1β0(1b0).\nabla_\theta b(\theta_0)=-\frac{1}{\beta_0}\begin{pmatrix}1\\b_0\end{pmatrix}.

Thus

b(θ0+h)−b0=−1β0(hα+b0hβ)+o(∥h∥).b(\theta_0+h)-b_0=-\frac{1}{\beta_0}(h_\alpha+b_0h_\beta)+o(\lVert h\rVert).

Substituting this expansion into (S1.9) yields

R(θ0+h)−R∗=f(b0)4β0(hα+b0hβ)2+o(∥h∥2).R(\theta_0+h)-R^*=\frac{f(b_0)}{4\beta_0}(h_\alpha+b_0h_\beta)^2+o(\lVert h\rVert^2).

Hence

R(θ0+h)−R∗=12hTHRh+o(∥h∥2),R(\theta_0+h)-R^*=\frac{1}{2}h^{\mathsf T}H_Rh+o(\lVert h\rVert^2),

with

HR=f(b0)2β0(1b0)(1b0).H_R=\frac{f(b_0)}{2\beta_0}\begin{pmatrix}1\\b_0\end{pmatrix}\begin{pmatrix}1&b_0\end{pmatrix}.

This proves (58) and (59).

Response-term ablation

The ablation uses the full query–response information matrix and the same joint likelihood as response-aware acquisition, but omits ψaR\psi_a^R from candidate scores. All comparisons use 500 paired replications and total query budgets including the common pilot.

For each replication, the paired difference is the excess risk of the full criterion minus that of the ablation. Intervals are pointwise 95% Monte Carlo tt intervals. Two-sided paired tt-test pp-values are adjusted by Holm’s method over the nine regime–budget comparisons separately within the oracle and adaptive studies. The oracle null comparisons are exactly zero; their intervals are degenerate and their pp-values are recorded as one by convention.

StudyResponseBBDifference95% intervalpHolmp_{\mathrm{Holm}}
OracleNoninformative3000.0000[0.0000, 0.0000]1.0000
6000.0000[0.0000, 0.0000]1.0000
9000.0000[0.0000, 0.0000]1.0000
Moderate300-0.0874[-0.3741, 0.1994]1.0000
600-0.0960[-0.2255, 0.0335]1.0000
900-0.1107[-0.1862, -0.0352]0.0373
Strong300-0.0372[-0.1182, 0.0437]1.0000
6000.0013[-0.0162, 0.0188]1.0000
900-0.0071[-0.0198, 0.0056]1.0000
AdaptiveNoninformative3000.2192[-0.1308, 0.5691]1.0000
6000.0707[-0.0577, 0.1990]1.0000
9000.0354[-0.0181, 0.0889]1.0000
Moderate300-0.2205[-0.4623, 0.0214]0.5912
600-0.0832[-0.2036, 0.0372]1.0000
900-0.1077[-0.1726, -0.0428]0.0107
Strong300-0.0305[-0.1247, 0.0638]1.0000
600-0.0259[-0.0602, 0.0083]0.9648
900-0.0071[-0.0247, 0.0104]1.0000

Table S1. Paired excess-risk differences for response-aware acquisition versus the label-only, full-information ablation. Differences and interval endpoints are multiplied by 10310^3. Negative values favor response-aware acquisition.

Figure S1 displays the paired differences and pointwise intervals. The intervals are not simultaneous confidence intervals.

Figure S1. Excess-risk differences between the full response-aware criterion and the label-only, full-information ablation. Negative differences favor the full criterion. Bars are pointwise 95% Monte Carlo tt intervals over 500 paired replications. Query budgets include the pilot.

References

  1. [1]Ahfock, D. and McLachlan, G. J. (2020). An apparent paradox: a classifier based on a partially classified sample may have smaller expected error rate than that if the sample were completely classified. Statistics and Computing, 30:1779–1790. https://doi.org/10.1007/s11222-020-09971-5.
  2. [2]Amin, K., DeSalvo, G., and Rostamizadeh, A. (2021). Learning with labeling induced abstentions. In Advances in Neural Information Processing Systems, volume 34, pages 12576–12586. Curran Associates, Inc. Bach, F. R. (2006). Active learning for misspecified generalized linear models. In Advances in Neural Information Processing Systems, volume 19, pages 65–72. MIT Press.
  3. [3]Bickford Smith, F., Kirsch, A., Farquhar, S., Gal, Y., Foster, A., and Rainforth, T. (2023). Prediction-oriented Bayesian active learning. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 7331–7348. PMLR. Available at: https://proceedings.mlr.press/v206/bickfordsmith23a.html.arxiv.org/abs/2304.08151
  4. [4]Chen, Y. and Biros, G. (2023). FIRAL: An active learning algorithm for multinomial logistic regression. In Advances in Neural Information Processing Systems, volume 36, pages 65894–65905. Curran Associates, Inc. https://doi.org/10.52202/075280-2877.
  5. [5]Fang, M. and Zhu, X. (2014). Active learning with uncertain labeling knowledge. Pattern Recognition Letters, 43:98–108. https://doi.org/10.1016/j.patrec.2013.10.011.
  6. [6]Filstroff, L., Sundin, I., Mikkola, P., Tuilpin, A., Kylmäoja, J., and Kaski, S. (2024). Targeted active learning for Bayesian decision-making. Transactions on Machine Learning Research. Available at: https://openreview.net/forum?id=KxPju iMgmm.
  7. [7]Hoi, S. C. H., Jin, R., Zhu, J., and Lyu, M. R. (2006). Batch mode active learning and its application to medical image classification. In Proceedings of the 23rd International Conference on Machine Learning, pages 417–424. ACM. https://doi.org/10.1145/1143844.1143897.
  8. [8]Huang, Z., Bickford Smith, F., and Rainforth, T. (2026). Loss-driven Bayesian active learning. In Proceedings of the 29th International Conference on Artificial Intelligence and Statistics, volume 300 of Proceedings of Machine Learning Research, pages 5140–5148. PMLR. Available at: https://proceedings.mlr.press/v300/huang26b.html.arxiv.org/abs/2604.11995
  9. [9]Imberg, H., Jonasson, J., and Axelson-Fisk, M. (2020). Optimal sampling in unbiased active learning. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 559–569. PMLR. Available at: https://proceedings.mlr.press/v108/imberg20a.html.
  10. [10]Lewis, D. D. and Catlett, J. (1994). Heterogeneous uncertainty sampling for supervised learning. In Machine Learning Proceedings 1994, pages 148–156. Morgan Kaufmann. https://doi.org/10.1016/B978-1-55860-335-6.50026-X.
  11. [11]Lewis, D. D. and Gale, W. A. (1994). A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3–12. Springer. https://doi.org/10.1007/978-1-4471-2099-5_1.
  12. [12]Nguyen, C. V., Ho, L. S. T., Xu, H., Dinh, V., and Nguyen, B. T. (2022). Bayesian active learning with abstention feedbacks. Neurocomputing, 471:242–250. https://doi.org/10.1016/j.neucom.2021.11.027.
  13. [13]Pukelsheim, F. (2006). Optimal Design of Experiments. Society for Industrial and Applied Mathematics, Philadelphia, PA. https://doi.org/10.1137/1.9780898719109.
  14. [14]Robinson, T. S., Tax, N., Mudd, R., and Guy, I. (2024). Active learning with biased non-response to label requests. Data Mining and Knowledge Discovery, 38(4):2117–2140. https://doi.org/10.1007/s10618-024-01026-x.
  15. [15]Roy, N. and McCallum, A. (2001). Toward optimal active learning through Monte Carlo estimation of error reduction. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 441–448. Morgan Kaufmann.
  16. [16]Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3):581–592. https://doi.org/10.1093/biomet/63.3.581.
  17. [17]Setoudehtazangi, F. and McLachlan, G. J. (2026a). Informative label missingness in multiclass classification: Information geometry and excess risk. arXiv preprint arXiv:2608.30561. https://doi.org/10.48550/arXiv.2608.30561.
  18. [18]Setoudehtazangi, F. and McLachlan, G. J. (2026b). Classification-risk-optimal label acquisition. arXiv preprint arXiv:2609.06873. https://doi.org/10.48550/arXiv.2609.06873.
  19. [19]Settles, B. (2009). Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison. Available at: https://research.cs.wisc.edu/techreports/2009/TR1648.pdf.
  20. [20]Sourati, J., Akcakaya, M., Leen, T. K., Erdogmus, D., and Dy, J. G. (2017). Asymptotic analysis of objectives based on Fisher information in active learning. Journal of Machine Learning Research, 18(34):1–41. Available at: https://www.jmlr.org/papers/v18/15-104.html.
  21. [21]Totakura, V., Singh, A., Dong, Y., and Chakraborty, S. (2026). An analysis of active learning algorithms using real-world crowd-sourced text annotations. In Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN 2026). Accepted for publication. Preprint available at: https://arxiv.org/abs/2604.23290.
  22. [22]Wang, H., Zhu, R., and Ma, P. (2018). Optimal subsampling for large sample logistic regression. Journal of the American Statistical Association, 113(522):829–844. https://doi.org/10.1080/01621459.2017.1292914.
  23. [23]Wang, Y.-G., Wu, J., and McLachlan, G. J. (2026). Learning from uncertainty-dependent missing labels for semi-supervised classification. arXiv preprint arXiv:2608.23960. https://doi.org/10.48550/arXiv.2608.23960.
  24. [24]Yan, S., Chaudhuri, K., and Javidi, T. (2016). Active learning from imperfect labelers. In Advances in Neural Information Processing Systems, volume 29, pages 2128–2136. Curran Associates, Inc.arxiv.org/abs/1610.09730
  25. [25]Yao, Y. and Wang, H. (2019). Optimal subsampling for softmax regression. Statistical Papers, 60(2):585–599. https://doi.org/10.1007/s00362-018-01068-6.

Paper details

vlnc:W_xtujafzw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents