WorksMathematics
Risk-optimal active learning with informative abstention
Abstract
We study label acquisition when an oracle may abstain and its response probability shares parameters with the classifier. A query can then provide information through both a returned label and the response outcome. Under a correctly specified parametric model, we derive the classifier information after adjustment for response-model nuisance parameters. Combining this information with the local curvature of multiclass zero-one risk gives an acquisition criterion for minimizing the leading asymptotic excess risk. We characterize the optimal allocation under a fixed query budget and show how query allocation affects response identifiability. The oracle criterion reduces to response-discounted acquisition when response probability does not depend on the classifier parameters. Simulations examine the resulting query allocations and isolate the contribution of the response-outcome term through an ablation. A Consumer Complaints replay retains observed human abstentions and reveals benchmark labels upon response. Response-aware acquisition has lower mean test error than response-discounted acquisition at the two largest budgets, with paired differences significant after Holm correction. Entropy sampling performs better at the smallest budget. The gains depend on the response mechanism and query budget.
Acknowledgements
We are grateful to Cuong V. Nguyen of the Department of Mathematical Sciences at Durham University, UK, for generously taking the time to read the manuscript and for his thoughtful comments. His observations on adaptive sampling helped us clarify the assumptions and proof of the two-stage acquisition result.
Introduction
Obtaining a class label may require a human annotator, expert assessment, diagnostic procedure, or user interaction. A query does not always produce a label: an annotator may decline an ambiguous case, an expert may find the available information insufficient, or a user may not respond. This raises two questions for active learning: which observations should be queried, and how should the possibility and the observed pattern of response affect query selection?
Classical active learning allocates a limited labeling budget by selectively requesting labels ([11, 10, 19]. Uncertainty sampling prioritizes observations with ambiguous class probabilities, whereas information-based methods assess the value of a label for parameter estimation or prediction ([7, 20, 4]. Other approaches target prediction error, predictive information, or downstream decisions ([15, 3, 6, 8].
Statistical approaches to sampling provide related foundations. Wang et al. [22] and Yao and Wang [25] derive optimal subsampling procedures for logistic and softmax regression, respectively. Imberg et al. [9] study sampling schemes for prediction-error reduction in unbiased active learning, while Bach (2006) develops active learning for potentially misspecified generalized linear models. These methods address different inferential and predictive objectives. The problem considered here additionally involves a stochastic oracle response whose distribution shares parameters with the classifier.
Imperfect labeling oracles have received substantial attention. Yan et al. [24] study active learning when a labeler may return an incorrect label or abstain, obtaining query-complexity guarantees under assumptions on the noise and abstention processes. Fang and Zhu [5] consider uncertain labeling knowledge, allowing an oracle to indicate that it does not know a label. Amin et al. [2] study a different form of abstention: examples routed to a human labeler no longer require a prediction from the machine. Their algorithm jointly learns a model and decides when to request labels. In a Bayesian pool-based setting, Nguyen et al. [12] develop acquisition rules that learn both the classifier and an unknown abstention mechanism. Thus, neither allowing an oracle to abstain nor learning its response mechanism alongside the classifier is itself new.
A closely related approach is the biased-nonresponse framework of Robinson et al. [14]. Their expected-utility rule weights an ordinary acquisition utility by the estimated probability of response; an upper-confidence-bound extension accounts for uncertainty in that estimate. This addresses the reduced value of queries that are unlikely to return labels. It does not, however, derive classifier information from the realized response outcome through a shared-parameter likelihood. Likewise, Nguyen et al. [12] use separate classification and abstention models, rather than deriving the nuisance-adjusted Fisher information of a classifier-linked response mechanism and weighting it by local zero–one classification risk.
Likelihood-based work on informative label missingness motivates this distinction. Under Rubin’s framework, conditional independence of missingness and the unobserved class given the features does not by itself justify ignoring a mechanism that shares parameters with the classification model [16]. Ahfock and McLachlan [1] show that uncertainty-dependent label missingness can carry information about a discriminant rule. More recent work develops information decompositions and connects the resulting information to classification performance [23, 17]. These results concern the information available under a given missingness mechanism. Here the investigator also controls which observations are submitted to the oracle.
Our starting point is the classification-risk acquisition framework of Setoudehtazangi and McLachlan [18], in which every requested label is returned. Under its regularity conditions, leading asymptotic excess zero–one risk depends on Fisher information weighted by the local curvature of the active Bayes decision boundary. A label is therefore valuable according to whether its information improves estimation in directions that affect classification, rather than according to uncertainty or parameter precision alone. We consider how this criterion changes when the oracle may abstain.
Let denote the observed features, the class label, the investigator’s query indicator, and the oracle response indicator. Conditional on a query, we model
where governs the classification model and contains response-specific nuisance parameters. We assume that a returned label is correct and that
The acquisition rule is known and parameter-free. Conditional on the observed features, its realized query decision therefore carries no likelihood information about the model parameters. The subsequent response outcome can carry such information because its probability depends on .
For the joint parameter , a query at contributes
where is the conditional class-label information augmented by zero nuisance blocks, and
The first term is information from a potentially returned label. The second is information from observing whether the oracle responds. When the response-model nuisance information is nonsingular, eliminating leaves a positive semidefinite response contribution to efficient classifier information. This contribution vanishes precisely when the classifier-related response score is explained by the nuisance response score under the design-weighted measure. Sharing parameters is therefore insufficient by itself: useful response information must survive nuisance adjustment and affect parameter directions relevant to classification.
Combining the query information with local risk curvature gives the marginal acquisition value
where denotes the acquisition design. The returned-label term accounts for the probability of receiving an informative label. The response term accounts for the classification value of the response outcome itself. A query can consequently be informative even when no class label is returned. When response probability does not depend on the classifier parameters, the oracle criterion reduces to response-discounted acquisition.
The joint information matrix is affine in the acquisition design. The resulting optimization therefore retains the convex trace-inverse structure of a linear optimal-design criterion [13] and admits a threshold characterization under a fixed query budget. Query allocation must also preserve sufficient variation for nonsingular response-model information.
We derive this rank condition and give a two-stage oracle-value result for a pilot-selected design applied to fresh observations. The risk conclusion requires uniform criterion consistency, uniform asymptotic linearity in mean square, and a higher-moment bound. The proof conditions on the pilot and then averages over it. These conditions are not established for the sequential finite-pool implementation used in the experiments.
The simulations examine oracle and adaptive acquisition under noninformative and informative response mechanisms. An ablation retains the full information matrix and joint estimation while omitting the explicit response-outcome term from query scores. It separates the effect of that term from changes in the information matrix used by response-discounted acquisition. The observed gains are small and depend on the response mechanism and query budget.
We also use the Consumer Complaints annotations collected by Totakura et al. [21]. Each worker is treated as a separate oracle. The replay preserves observed human response and abstention patterns but reveals the benchmark class whenever a worker responds. It therefore evaluates acquisition under human abstention without modeling errors in returned labels. This comparison evaluates the complete acquisition rules; it does not isolate the explicit response-outcome term.
The contribution is a classification-risk acquisition criterion for a shared-parameter query–response model. It values response outcomes through their nuisance-adjusted information and their effect on the classification boundary. The analysis identifies conditions under which this additional information changes query selection, while the experiments examine when those changes improve classification.
§ 3 introduces the query–response model and derives its Fisher information. § 4 characterizes the efficient classifier information after nuisance adjustment. § 5 derives the classification-risk criterion and its marginal query value. Subsequent sections examine optimal allocation, response identifiability, adaptive acquisition, and a binary logistic illustration. Simulations and the Consumer Complaints replay follow, before the discussion of findings and limitations.
Statistical framework and informative oracle response
Let denote an observed feature vector and its class label. We consider a regular parametric classification model
where is the parameter governing the classification model. The feature vector is observed for every sampling unit. After observing , the investigator may request its class label. Let
and suppose
The measurable function is the acquisition design. It is chosen by the investigator and is treated as known and parameter-free. A label request need not result in a returned label. Conditional on , let
We model the response probability as
where contains parameters specific to the response mechanism. The defining feature of (2) is that the oracle response may depend on the same parameter that determines the classifier. The realized response or abstention can therefore itself contain information about the classification model. Throughout the main development we assume
and that a returned label is correct. Thus the response mechanism is feature-dependent but does not depend further on the unobserved class once is given. Noisy returned、】【 labels and class-dependent response mechanisms are outside the scope of the present paper.
For one sampling unit, the observed data can be represented by
Up to factors involving the known acquisition design , its likelihood is
Hence
Define the feature and conditional-label scores by
and let
Differentiation of (5) gives
Thus a query can generate information through two distinct channels: the class label when it is returned and the response indicator itself.
Let
denote the conditional Fisher information in a returned class label, and set
The information generated by a query at feature value is
The first term is the expected information from the returned label, whereas the second is the information carried by the response or abstention outcome.
Let
where
Proposition 2.1 Suppose that (1)–(3) hold and that the usual regularity conditions for Fisher-information calculations are satisfied. For a fixed acquisition design , the per-observation Fisher information for is
where is defined in (8). Equivalently,
The proof is given in Supplementary Section S1.1.
Proposition 2.1 separates the two sources of information created by a label request. If a requested label is returned with certainty, , then the response term vanishes and (10) reduces to the usual information for selective label acquisition. More generally, if the response mechanism is parameter-dependent, the statistical value of a query is not determined only by the probability of obtaining its label: the observed response or abstention is itself part of the annotation experiment.
Because is unknown, however, the second term in (8) cannot be interpreted directly as additional information about the classifier. The component that remains informative for after adjustment for the response-model nuisance parameters is derived next.
Efficient classifier information and the value of oracle response
The information decomposition in Proposition 2.1 treats as the full parameter vector. For classification, however, is the parameter of interest, whereas is a nuisance parameter governing the oracle-response mechanism. Consequently, the response-information term in (10) must be adjusted for uncertainty in .
Partition the Fisher information according to :
From (10),
where
Assume throughout this section that is positive definite. The efficient Fisher information for , after adjustment for , is the Schur complement
Substitution of (11)–(13) yields
where
The first two terms in (18) are respectively the information available from the feature distribution and the expected information from labels that are actually returned. The final term measures the additional classifier information supplied by the response process after removing the component attributable to the nuisance parameter . The following result gives a direct characterization of this contribution.
Proposition 3.1 Suppose that is positive definite and define
Then
Consequently,
Moreover,
where is the measure with density proportional to .
The proof is given in Supplementary Section S1.2.
Proposition 3.1 identifies precisely when observing whether the oracle responds provides information about the classifier beyond that contained in returned labels. The vector describes the local sensitivity of the response probability to the classifier parameter, while describes sensitivity to the response-specific nuisance parameter. Equation (21) shows that only the component of that cannot be represented through the nuisance score contributes efficient information for classification.
This distinction is important. Parameter dependence of the response probability alone does not guarantee additional information about after nuisance adjustment. If (22) holds, all classifier-related variation in the response mechanism is statistically confounded with variation in , and the response indicator contributes no additional efficient information. Otherwise,
and observing response or abstention supplies classifier information that would be discarded by a design criterion based only on the probability of obtaining a label.
The decomposition also identifies two limiting cases. If the response mechanism is independent of the classifier parameter, then and hence
The Bernoulli weight approaches zero as the response probability approaches either endpoint. If the corresponding logit-score vectors remain bounded, the pointwise response-information contribution also vanishes.
The positive-definiteness requirement on is itself design-dependent. An acquisition rule may therefore be highly informative for classification while failing to identify the response mechanism. We return to this issue when defining the admissible class of acquisition designs.
Classification risk under informative oracle response
The information criterion in § 4 measures precision in estimating the model parameters, but the objective of label acquisition is classification. We therefore evaluate an acquisition design through its leading contribution to the zero-one classification risk.
Let
denote the plug-in Bayes rule and define its risk, under the true parameter , by
Let
be the Bayes risk. Under the regular boundary conditions for multiclass classification, the local excess risk admits the quadratic expansion
where is the classification-risk curvature matrix at . It depends only on parameter directions that perturb active Bayes decision boundaries.
The response parameter does not enter the classifier. Accordingly, for the full parameter
define the augmented risk-curvature matrix
Suppose that, under a fixed admissible acquisition design , an efficient estimator satisfies
with nonsingular. Assume additionally that, writing ,
These moment and remainder conditions, together with (23), give the leading expected excess risk
Equivalently, using the block-inverse identity,
where is the nuisance-adjusted information in (17). This motivates the classification-risk criterion
For a labeling budget , the oracle acquisition problem is therefore
The nonsingularity condition is substantive in the present setting because the acquisition design determines not only which labels may be observed but also which response outcomes are available for identifying the response mechanism.
Since
is affine in , the criterion in (28) retains the trace-inverse structure of an optimal-design problem despite the nuisance adjustment in .
The local value of querying a feature vector is obtained from the directional derivative of . Define
Because
the query value decomposes as
where
The term is the classification value of the label, discounted by the probability that the oracle returns it. In contrast, is the classification value of observing the response outcome itself. In particular,
Thus the value of a query is not, in general, obtained by simply multiplying a label-acquisition score by the probability of response. Such a calculation retains but discards .
Proposition 4.1 Suppose that , and let be a bounded measurable perturbation such that remains admissible for sufficiently small . Then
where is given by (30). Moreover, admits the decomposition (31)–(33).
The proof is given in Supplementary Section S1.3. Proposition 4.1 provides the local principle underlying response-aware acquisition. A candidate can be valuable because its label is likely to be returned and informative for the decision boundary, because its response outcome is itself informative for classifier-relevant parameter directions, or through both channels. Consequently, two candidates can be ordered differently by a response-discounted label criterion and by the full classification-risk criterion. The optimal allocation implied by this marginal value is characterized in the next section.
Risk-optimal acquisition and response identifiability
We now characterize acquisition designs that minimize the leading classification-risk criterion
subject to a fixed query budget. Because the information matrix is affine in the acquisition design, the resulting problem retains a convex optimal-design structure even though the response mechanism contains unknown nuisance parameters.
For a fixed budget , consider
We assume that this class is nonempty and that an optimal design exists.
Proposition 5.1 On any convex subset of , the criterion
is convex.
The proof is given in Supplementary Section S1.4. Convexity, together with the derivative in Proposition 4.1, yields a direct characterization of the oracle acquisition rule.
Theorem 5.2 Let minimize , and suppose that is an interior point of the positive-definite information domain relative to the box and budget constraints. Then there exists such that, for -almost every ,
where
is the response-aware classification value defined in (4.9). Conversely, any feasible design satisfying these conditions is globally optimal.
The proof is given in Supplementary Section S1.5. Theorem 5.2 shows that the optimal design allocates the labeling budget according to the total classification value of a query, rather than according to the value of a potentially returned label alone. Except on level sets where randomization is required to satisfy the budget, the oracle rule therefore has a threshold form.
This distinction can alter the ordering of candidate observations. A response-discounted rule based only on favors observations whose returned labels are valuable after accounting for their probability of response. The full rule instead ranks observations according to
Consequently, an observation with a smaller probability of returning a label may still have greater total acquisition value if its response outcome is sufficiently informative about classifier-relevant parameter directions. The phenomenon does not require response probabilities close to zero: indeed, the Bernoulli information factor vanishes as approaches either zero or one.
Response identifiability under selective acquisition
The preceding optimality result presumes that the information matrix is nonsingular. In the present problem this requirement depends on the acquisition design itself. A design that concentrates queries on a restricted region of the feature space can provide substantial classification information while failing to identify the response mechanism. The nuisance-information block is
Proposition 5.3 For any ,
Consequently,
if and only if there is no nonzero such that
where is the weighted measure introduced in Proposition 3.1.
The proof is given in Supplementary § 13.6. Proposition 5.3 gives a design-specific rank condition for estimation of the response nuisance parameter. In particular, positive query probability alone is insufficient: the queried observations must exhibit enough variation in the response-model regressors to distinguish the components of .
For example, consider the response model
where is a scalar measure of classification uncertainty. Then
Provided has positive expectation, if and only if
Thus a design supported only on observations having the same uncertainty level cannot separately identify the intercept and uncertainty effect in the response model. In particular, concentrating all queries on a single decision-boundary uncertainty level can destroy response-mechanism identifiability.
For theoretical and numerical implementation it is therefore useful to work with the uniformly regular design class
The lower information bound prevents the acquisition procedure from obtaining an apparent classification advantage by approaching a design under which some components of the annotation experiment cease to be identifiable. This constraint is one of the regularity requirements used in the two-stage analysis below; it does not by itself establish the estimator conditions needed for that analysis.
Adaptive response-aware acquisition
The oracle criterion depends on the unknown classification and response parameters and on the risk curvature. We first analyze a two-stage procedure with a pilot-selected design that is held fixed during an independent second stage. We then distinguish this experiment from the sequential finite-pool implementation used in the numerical studies.
Pilot selection and the second-stage experiment
Let be the number of pilot sampling units and let be the number of new second-stage sampling units. Assume
Write for the sigma-field generated by the pilot features, query decisions, responses, and returned labels, together with any auxiliary randomness used to choose the second-stage design. The pilot uses a design that provides nonsingular information for the parameters to be estimated. Let and denote the pilot estimates. The selected acquisition function is -measurable.
Generate new pairs , , independently from and independently of . Let be independent uniform random variables on , independent across units and of the pilot and the new pairs. Set
This construction implements the conditional independence of response and class label assumed in § 3. Define , and let denote the one-unit observed-data law under parameter and a fixed design . Then
Thus the second-stage observations are independent and identically distributed conditional on the pilot. Unconditionally they generally are not independent, because they share the random design . We do not apply an iid argument to the pooled two-stage sample.
For example, for a fixed design , the nonzero observed-data densities are
Conditional on the observed pilot, the selected function is held fixed when differentiating the second-stage likelihood. Its information per sampling unit is therefore from Proposition 2.1, evaluated at . The returned-label subset need not follow the population feature law; (45) concerns the complete observed-data units, including unqueried units and queried abstentions.
Design selection and oracle value
Use a fixed, nonempty measurable design class , where is defined in (42). One may take the full regular class if the uniform assumptions below hold; a restricted class may be needed for empirical estimation. All suprema and selected designs below are assumed measurable. Define
where
and
The criterion and every quantity used to choose must be -measurable. In this experiment, may use pilot features, model-based integration, or an independent reference feature sample included in . It does not use the features of the second-stage sample. Positive definiteness of the estimated information is required on the class over which the criterion is optimized.
Choose a measurable approximate minimizer
Feasibility in the population class is an assumption. Using estimated budget and information constraints instead requires a separate feasibility-approximation argument.
For deterministic , let be an estimator based only on iid observations from . It is a measurable function of these observations and , with no other pilot dependence. Write for expectation in this fixed-design experiment and for its joint score, given by (6)–(7). Let be its classification component.
Theorem 6.1 Suppose that exists, (49) holds, and
Then
For the expected-risk conclusion, additionally assume the independent second-stage construction (6.2)–(6.3), the local risk expansion (23), and the following uniform fixed-design estimator conditions. First, assume
Second, for some , assume
Assume also the usual score identities and . Then the second-stage estimator satisfies
and consequently
The proof is given in Supplementary Section S1.7. It first establishes a uniform expected-risk expansion for deterministic designs and then uses the exact conditional law (45). In particular, pointwise asymptotic normality for each fixed design would not suffice: the selected design varies with the pilot, and convergence in distribution alone does not justify convergence of expected risk. Conditions (52) and (53) supply the required uniform second-moment and remainder control. They are assumptions on the estimator family, not consequences established here for arbitrary likelihood optimizers.
The information bound gives on . This bound allows the conditional conclusion to be averaged over the pilot without treating the two stages as iid. The result does not require a unique oracle design. It is an optimal-value result over , rather than a proof that the selected design itself converges to a unique function.
Uniform criterion consistency must also be justified for the chosen approximation and class. Pilot-parameter consistency alone does not imply (50) for unrestricted measurable designs when expectations are replaced by empirical averages. A suitable uniform law of large numbers, an appropriate restricted design class, or direct uniform control of model-based integration is needed.
The theorem concerns a fixed expected query budget: conditional on the pilot, the second stage uses queries in expectation. Its final estimator uses only the new observations. If counts both stages, (43) gives , so the same risk coefficient can be written with denominator . Pilot queries contribute at most , and the overall expected query fraction tends to . These accounting statements do not prove an equivalent result for a pooled estimator. For such an estimator , one sufficient additional requirement is
The same quadratic-risk argument then transfers the leading coefficient. The condition alone does not establish these requirements.
Sequential finite-pool implementation
For an available pool , the plug-in marginal value is
A practical implementation updates the fitted classifier, response model, and information matrices after each batch and selects high-scoring remaining observations. This uses the query-value decomposition but is a different sampling experiment from the theorem. Selection without replacement, exact batch sizes, reuse of pilot observations in estimation, and repeated model updates generally prevent the conditional product law (45) from holding. Likewise, choosing the design using the features of the eventual estimation pool does not satisfy the fresh-sample construction above. Conditioning additionally on those features leads to a fixed-feature experiment, not the iid population experiment analyzed here. A separate conditional or adaptive sampling argument would be needed for that setting.
Accordingly, Theorem 6.1 does not establish oracle attainment for the sequential finite-pool experiments. Those experiments assess a practical implementation of the criterion. The pilot provides initial information about both the classifier and the response mechanism; a regular pilot design helps avoid beginning acquisition with an unidentified response model.
Binary logistic illustration
We now illustrate the preceding theory in a binary classification model for which the decision boundary, label information, response information, and classification-risk curvature can all be written explicitly.
Let
with
For simplicity, suppose that the marginal distribution of does not depend on , with density . The Bayes classifier assigns class 1 whenever
so the decision boundary is
Let
denote the true boundary. Under the usual smoothness condition , the local excess risk satisfies
Hence
Thus only parameter directions that perturb the decision boundary contribute to first-order classification risk.
Write
The conditional class-label information is
Because
is maximized at , the scalar Bernoulli variance factor is largest at the Bayes boundary. The full information matrix also depends on , so its magnitude and classification value need not be largest there.
To model informative oracle response, consider
where
is the predictive entropy. The case corresponds to an oracle that is less likely to respond at more uncertain observations.
Since
and
we obtain
Therefore
whereas
The raw response-information contribution at is consequently
In contrast, the expected returned-label contribution is
These two sources of information have fundamentally different spatial geometry. At the Bayes boundary ,
and hence
Thus, under the entropy-linked response model, the raw classifier information in the response indicator vanishes exactly at the decision boundary, even though the scalar Bernoulli variance factor is maximal there.
At the opposite extreme, as ,
so both the label and response information vanish. Response information is therefore concentrated away from the exact boundary but not in the extreme tails. It is naturally associated with intermediate off-boundary regions.
This distinction can change acquisition rankings. A response-discounted criterion considers only the expected value of the returned label, represented locally by . The full response-aware criterion instead uses
For two candidates and , it is therefore possible that
while
The reversal occurs when the additional classification value carried by the response outcome at more than compensates for its smaller response-discounted label value.
The example also clarifies why informative response should not be interpreted as a preference for almost-certain abstention. The Bernoulli factor
vanishes when approaches either zero or one. Response information is largest only where the response probability remains stochastic and its classifier-related sensitivity is substantial. The numerical experiments in § 9 will examine how this geometry translates into finite-sample acquisition decisions and classification performance.
Simulation study
We investigated the finite-sample behavior of the response-aware acquisition criterion in the binary logistic model of § 8. The simulation was designed to separate two questions. First, we examined oracle acquisition, in which the true model parameters are used to evaluate candidate query scores. This isolates the statistical effect of the acquisition criterion itself. Second, we considered an adaptive procedure in which the classifier and response parameters are estimated from the accumulated query-response data and the acquisition scores are updated sequentially.
Throughout,
with
The true Bayes boundary is therefore
The oracle-response mechanism was
where is the binary predictive entropy defined in (62). Three response regimes were considered:
representing noninformative, moderately informative, and strongly informative response, respectively. For each value of , the intercept was calibrated so that the marginal response probability satisfied
The resulting values were
with corresponding response probabilities at the Bayes boundary equal to 0.400, 0.239, and 0.157. Thus the scenarios differ in the informativeness of the response mechanism rather than simply in the expected number of returned labels.
For each Monte Carlo replication, an unlabeled pool of size 3000 was generated. Query budgets of
were considered. These budgets include the common random pilot of 100 queried observations. Subsequent queries were acquired in batches of 25. The same potential class labels and oracle-response outcomes were used across methods within each replication, yielding paired comparisons.
Six acquisition procedures were compared: random sampling, entropy sampling, ordinary classification-risk-optimal acquisition that treats labels as always available, response-discounted risk-optimal acquisition, the proposed response-aware criterion, and a label-only ablation with full information.
The response-aware criterion uses
The ablation retains the full query–response information matrix, including response-model nuisance parameters, but scores candidates using only . It therefore differs from response-discounted acquisition, which uses a label-based information matrix.
All six procedures used the same full observed-data likelihood after acquisition. The ablation examines the effect of including the explicit response-outcome term in sequential query selection while retaining full-information calculations and joint estimation. Once the query paths diverge, the procedures naturally have different accumulated data and fitted parameters.
For adaptive acquisition, the parameters
were re-estimated from the accumulated queried data using the likelihood
The principal performance measure was excess classification risk relative to the true Bayes classifier. We also recorded boundary-estimation error, the number of returned labels, response rates, and the locations of queried observations. Results below are based on 500 Monte Carlo replications. All fits evaluated at the reported budgets returned optimizer convergence code zero.
Likelihood optimization used box constraints
At each batch, the accumulated information matrices are sums over all previously queried observations, including the pilot. Ordinary risk sums , response-discounted risk sums , and the response-aware and ablation rules both sum . These quantities are evaluated at the parameters used for scoring in that batch. For acquisition calculations, each accumulated information matrix was symmetrized and replaced by
before inversion. Optimizer convergence codes describe numerical termination and do not establish global likelihood maximization.
Acquisition geometry
Figure 1 illustrates the population geometry in the strongly informative-response regime. The returned-label and response components have markedly different spatial behavior. The raw classifier-related response-information component vanishes at the true decision boundary , as predicted by (68). The nuisance-adjusted marginal response value need not vanish there. Numerically, the ordinary risk score is maximized near , whereas the response-discounted and response-aware scores are maximized near and , respectively. The response component itself is maximized near .
Figure 1. Information and acquisition geometry in the strongly informative-response setting. The vertical dotted line marks the true Bayes boundary . Panel (a) shows the class and response probabilities. Panel (b) shows the decision-relevant information from returned labels and response outcomes. Panel (c) shows the ordinary, response-discounted, and response-aware acquisition scores. Panel (d) shows the response-aware score and its two components. The scores in panels (c) and (d) are evaluated at the constant reference design . Each curve in panels (b)–(d) is divided by its own maximum over the plotting grid, so curve heights cannot be used to compare magnitudes across curves.
The figure therefore demonstrates that accounting for the response indicator does not merely rescale an uncertainty-based acquisition score. Instead, it changes the spatial allocation of queries by assigning value to observations whose response outcome is informative for classifier-relevant parameter directions.
Classification performance
Table 1 reports the mean excess classification risk for the adaptive procedures. Under noninformative response, the response-aware procedure provides no systematic advantage, as expected. In the corresponding oracle experiment, the response-aware and response-discounted procedures coincide exactly for every replication and budget, providing a direct numerical check of the theoretical reduction
| Response | Random | Entropy | Ordinary | Discounted | Ablation | Response-aware | |
| Noninformative | 300 | 2.807 (0.169) | 1.890 (0.125) | 1.816 (0.106) | 2.089 (0.125) | 1.918 (0.111) | 2.138 (0.157) |
| Noninformative | 600 | 1.495 (0.105) | 0.827 (0.052) | 0.800 (0.049) | 0.778 (0.045) | 0.825 (0.050) | 0.896 (0.062) |
| Noninformative | 900 | 0.977 (0.066) | 0.520 (0.034) | 0.537 (0.035) | 0.510 (0.032) | 0.519 (0.033) | 0.554 (0.035) |
| Moderate | 300 | 1.800 (0.114) | 2.218 (0.146) | 1.927 (0.132) | 1.612 (0.093) | 1.700 (0.115) | 1.479 (0.106) |
| Moderate | 600 | 0.763 (0.047) | 1.061 (0.065) | 0.990 (0.064) | 0.704 (0.043) | 0.711 (0.053) | 0.628 (0.045) |
| Moderate | 900 | 0.520 (0.032) | 0.609 (0.037) | 0.616 (0.038) | 0.463 (0.030) | 0.467 (0.033) | 0.359 (0.023) |
| Strong | 300 | 0.926 (0.061) | 1.966 (0.130) | 1.463 (0.096) | 0.656 (0.040) | 0.609 (0.039) | 0.578 (0.039) |
| Strong | 600 | 0.457 (0.030) | 1.056 (0.068) | 0.837 (0.056) | 0.276 (0.017) | 0.275 (0.018) | 0.249 (0.016) |
| Strong | 900 | 0.331 (0.024) | 0.634 (0.042) | 0.604 (0.038) | 0.176 (0.010) | 0.179 (0.011) | 0.172 (0.010) |
Table 1. Mean excess classification risk multiplied by for adaptive acquisition over 500 replications, with Monte Carlo standard errors in parentheses. The ablation retains full information but omits the explicit response-outcome acquisition term. The smallest mean in each row is in bold.
The picture changes once oracle response is informative. Under moderate informative response, the response-aware procedure has the smallest mean excess risk at all three budgets. At , for example, its mean excess risk is , compared with for the response-discounted rule, for random acquisition, for entropy sampling, and for ordinary risk-optimal acquisition.
The same qualitative pattern appears under strongly informative response. The proposed procedure attains mean excess risks
0.578, 0.249, 0.172 in units of at budgets 300, 600, and 900, respectively. The corresponding values for the response-discounted method are
The gain relative to the response-discounted rule is therefore most visible at smaller and intermediate budgets and decreases as the available query budget grows.
The response-aware rule does not dominate in every setting. In the oracle study, it performs similarly to response-discounted acquisition in several settings and has slightly higher mean risk in some. The ablation below examines whether including the explicit response-outcome term improves acquisition when full information and joint estimation are retained.
Figure 2 gives the corresponding oracle and adaptive learning curves. The figure includes the label-only, full-information ablation alongside the five original acquisition
Figure 2. Mean excess classification risk as a function of the query budget under oracle and adaptive acquisition. Results are based on 500 Monte Carlo replications. Shaded regions represent approximate 95% Monte Carlo intervals for the mean.
procedures. The adaptive response-aware rule also improves substantially over random acquisition, particularly under strong informative response. At , its excess risk is approximately 48% lower than that of random acquisition in the strong-response setting and approximately 31% lower in the moderate-response setting.
Contribution of the response-outcome term
The ablation retains the full information matrix and joint likelihood, but omits from candidate scores. Supplementary Table S1 reports paired excess-risk differences, defined as the full criterion minus the ablation.
Under moderately informative response and , the adaptive procedure reduced mean excess risk from to . The paired difference was , with a pointwise 95% Monte Carlo interval of and Holm-adjusted . The oracle difference was , with interval and adjusted . Holm adjustments cover the nine comparisons within each study.
Relative to the ablation’s mean excess risk, both reductions were approximately 23%. In absolute terms, the adaptive reduction was approximately 0.0108 percentage points of classification error. The full adaptive criterion had lower mean excess risk in all six informative-response settings, although the other five intervals included zero.
Under noninformative response, the oracle procedures had identical query paths in every saved trace and identical recorded outcomes across all 500 replications. The adaptive procedures need not coincide: estimation error can give the fitted response model a nonzero classifier-dependent component.
The evidence for an additional benefit is therefore confined to the moderate-response setting at the largest budget. The unresolved comparisons do not establish equivalence, and the results do not support a general advantage from increasing response dependence.
Where the procedures query
The risk differences are accompanied by substantial changes in the locations of the queried observations. Figure 3 shows the empirical distribution of queried feature values in the strongly informative-response regime. Entropy and ordinary-risk acquisition place most of their queries close to the Bayes boundary, where classification uncertainty is largest. In contrast, both response-discounted and response-aware acquisition shift substantial mass to off-boundary regions.
Figure 3. Empirical distribution of queried feature values in the strongly informative-response setting. The dotted vertical line marks the true Bayes boundary. Entropy and ordinary-risk sampling concentrate near the boundary, whereas response-discounted and response-aware acquisition place substantial query mass in informative off-boundary regions.
For the adaptive procedures at , the mean absolute distance from the true boundary was 0.26, 0.27, 0.83, and 0.84 for entropy, ordinary-risk, response-discounted, and response-aware acquisition, respectively. Thus, unlike uncertainty-based rules, the response-aware design places substantial query mass away from the decision boundary, reflecting the joint value of returned labels and response information. The distinction is also visible in the realized response rates. Under strong informative response and , entropy and ordinary-risk acquisition obtain responses for only about 20% of queried observations, whereas the response-discounted and response-aware procedures obtain response rates of approximately 40%. The proposed acquisition rule therefore avoids spending most of its budget in regions where the oracle almost always abstains, while retaining the additional information carried by the response outcome itself.
Real-data application
We evaluate the acquisition rules using the Consumer Complaints data from the AL-RCTA collection of Totakura et al. [21], which contains crowd-sourced annotations with explicit annotator abstentions. The analysis uses 2,999 complaints presented to each of ten human annotators, with the response indicator defined by
Empirical response rates varied substantially across annotators, from approximately 0.75 to 0.97. Response was also related to classification difficulty: in a preliminary out-of-fold analysis, the response rate decreased from 0.972 in the lowest predictive-entropy decile to 0.827 in the highest.
The theoretical model assumes that a returned label is correct, whereas the crowd annotations may themselves contain labeling errors. We therefore retain the observed human response indicator , but, when , reveal the benchmark product category rather than the annotator’s reported class. This preserves the genuine human abstention pattern while isolating informative response from label noise. Each annotator is treated as a separate oracle.
Complaint narratives were represented by TF–IDF features followed by a 12-dimensional truncated singular-value decomposition. One of the 3,000 shared complaints was excluded because its narrative was missing. TF–IDF, singular-value decomposition, feature scaling, and narrative-length scaling were constructed using all 2,999 narratives before generating the splits. The analysis therefore uses a transductive feature representation: test narratives contribute to unsupervised preprocessing, but test labels are withheld from fitting and acquisition. A multinomial logistic classifier was fitted jointly with
where is normalized predictive entropy and is standardized log narrative length. We compared random, entropy, ordinary classification-risk, response-discounted risk, and response-aware acquisition. The classification task comprised six product categories. Across 30 replications, all methods shared a stratified split with approximately 70% of observations allocated to training and 30% to testing, a common pilot sample of 300 training observations, and the same realized oracle responses. Labels were subsequently queried in batches of 50 at additional budgets . All methods used the same penalized joint likelihood, with quadratic penalty coefficients 1 for classifier slopes and 0.5 for response-model slopes; intercepts were unpenalized. Risk-based acquisition used an empirical kernel approximation to the active-boundary curvature, weighted by the mean fitted probability of the two leading classes. The Gaussian-kernel bandwidth was the larger of 0.20 and the empirical 25th percentile of their absolute log-probability gaps. The curvature matrix was trace-normalized. Information matrices used diagonal regularization of . Before inversion, each eigenvalue of the symmetrized matrix was replaced by the larger of its original value and , where is the largest eigenvalue. For ordinary-risk and response-discounted acquisition, the accumulated information matrix uses conditional classifier information from labels actually returned. Response-aware acquisition instead uses the fitted full query–response information matrix, including the response-model nuisance parameters. Thus, the response-discounted baseline is not obtained simply by removing from the response-aware score while retaining the same information matrix. This experiment compares the complete acquisition rules rather than isolating the contribution of that additive term. Each batch selected the 50 highest-scoring available observations before refitting. This sequential implementation illustrates the criterion; it is not a numerical verification of the two-stage oracle-attainment theorem.
| Acquisition rule | 150 | 300 | 450 | 600 |
| Random | 0.260 (0.0018) | 0.254 (0.0017) | 0.250 (0.0017) | 0.247 (0.0016) |
| Entropy | 0.246 (0.0022) | 0.241 (0.0017) | 0.238 (0.0015) | 0.235 (0.0015) |
| Ordinary risk | 0.258 (0.0019) | 0.244 (0.0015) | 0.237 (0.0016) | 0.232 (0.0016) |
| Response-discounted | 0.258 (0.0019) | 0.244 (0.0015) | 0.237 (0.0016) | 0.232 (0.0015) |
| Response-aware | 0.258 (0.0019) | 0.243 (0.0016) | 0.235 (0.0017) | 0.231 (0.0016) |
Table 2. Held-out classification error in the Consumer Complaints application. Entries are means over 30 replications after averaging over the ten human oracles; parentheses contain Monte Carlo standard errors. The query budget excludes the common pilot sample of size 300. Additional query budget
Entropy sampling has the lowest mean test error at the two smallest budgets, whereas response-aware acquisition has the lowest mean error at and . The mean paired differences in test error (response-aware minus response-discounted) are
at , respectively. Pointwise 95% Monte Carlo intervals exclude zero at the three larger budgets; at , the interval is . After Holm correction across the 16 budget–comparator comparisons, the advantages over response-discounted acquisition remain significant at and , with adjusted -values 0.0202 and 0.00111, respectively. Response-aware acquisition is worse than entropy sampling at ; its advantage over entropy at does not remain significant under this correction. These intervals quantify variability across repeated splits and acquisition experiments for the fixed dataset and ten annotators, rather than uncertainty over a population of datasets or annotators.
Figure 4 shows a negative association between fitted predictive entropy and response probability in the pooled random-query reference cloud. The two risk-based procedures follow distinct, nonmonotone trajectories in their evolving fitted coordinates. Their first-batch mean locations are similar. In the final batch, response-aware acquisition has mean entropy 0.781 and fitted response probability 0.668, compared with 0.691 and 0.690, respectively, for response-discounted acquisition. These trajectories illustrate differences in sequential querying and model updating; they do not by themselves identify which information component causes the performance difference.
Figure 4. Uncertainty–response acquisition geometry for the Consumer Complaints application using worker W8, who had the largest observed abstention fraction. Panel (a) pools random-query locations across 30 replications. Panels (b) and (c) show response-discounted and response-aware acquisition in replication 6. Coordinates are predictive entropy and response probability evaluated under each method’s fitted model at the time of querying. Open circles trace batch-wise mean query locations; the diamond marks the final batch at additional budget . Replication 6 was selected because its paired response-aware minus response-discounted test-error difference was closest to the median across the 30 W8 replications. The panels describe evolving fitted coordinates, rather than a common fixed-model representation.
The aggregate improvement is heterogeneous across human oracles. At , the largest mean improvement over response-discounted acquisition occurs for W4, with paired difference −0.00483 and 95% Monte Carlo interval [−0.00768, −0.00198]. Only W4 remains significant after Holm correction across the ten worker-specific comparisons, with adjusted . The other worker-specific comparisons are statistically unresolved after this correction.
Discussion and conclusions
This paper extends classification-risk-optimal label acquisition to an oracle that may abstain and whose response probability shares parameters with the classifier. The response outcome can then supply information even when no label is returned. After adjustment for response-model nuisance parameters, this contribution is
It is positive semidefinite and vanishes when the classifier-related response score is fully explained by the nuisance response score under the design-weighted measure. Parameter dependence alone is therefore insufficient for response outcomes to improve classification. The remaining information must also act in directions relevant to the decision boundary.
Combining the information from returned labels and response outcomes with the local curvature of zero–one risk gives the marginal query value
The associated oracle allocation retains the convex trace-inverse structure of the underlying design problem and admits a threshold characterization under a fixed query budget. Response identifiability places an additional restriction on the allocation: queries must provide enough variation to estimate the response mechanism.
The simulations show how accounting for response changes query selection. In the strongly informative-response setting, response-aware and response-discounted acquisition both place substantial query mass away from the Bayes boundary, where response is more likely. Entropy and ordinary classification-risk acquisition remain more concentrated near the boundary. Under noninformative response, the response-aware and response-discounted oracle procedures coincide.
The ablation examines the additional effect of including the explicit response-outcome term while retaining full information and joint estimation. Under moderately informative response at , the full criterion reduced mean excess risk in both the oracle and adaptive studies, with significance after the stated within-study Holm corrections. The other comparisons were statistically unresolved. The absolute improvement was small, and the results do not establish a general advantage across response mechanisms and budgets. Stronger response dependence alone does not imply a larger acquisition gain.
The Consumer Complaints replay provides evidence using observed human abstentions. Response-aware acquisition had the lowest mean test error at the two largest budgets, with improvements over response-discounted acquisition remaining significant after Holm correction at and . Entropy sampling performed better at the smallest budget. The improvements were small and varied across annotators. This comparison evaluates complete acquisition rules with different information matrices; it does not isolate the contribution of the explicit response-outcome term.
The theory is local and parametric and relies on correct specification of the classification and response models. An association between response and predictive difficulty can motivate a shared-parameter model, but does not establish that the assumed link is correct. Performance under misspecification remains unresolved. The two-stage result applies to a pilot-selected design held fixed on fresh second-stage observations. It relies on uniform criterion consistency, mean-square asymptotic linearity, and a higher-moment bound. It does not establish these estimator conditions, or oracle attainment, for the sequential finite-pool implementation.
We assume that returned labels are correct and that
The replay therefore preserves human response patterns while revealing benchmark labels upon response. Using actual worker labels would require accounting for annotation error and, potentially, class-dependent response. Joint allocation across annotators with different costs, response mechanisms, and accuracies would require further development.
The main contribution is a classification-risk acquisition criterion that accounts for information in oracle response outcomes after nuisance adjustment. The numerical studies support its usefulness in some settings, while showing that the gains depend on the response mechanism and query budget.
Statements and Declarations
Code and data availability
Code and reproducibility instructions for the simulation studies, the Consumer Complaints analysis, and figure generation are available in the Risk-optimal active learning with informative abstention repository. The Consumer Complaints data are obtained from the original AL-RCTA source cited in the manuscript.
CRediT authorship contribution statement
F. Setoudehtazangi: Conceptualization, Methodology, Formal analysis, Software, Investigation, Visualization, Writing – original draft. G.J. McLachlan: Methodology, Supervision, Writing – review & editing.
Funding
This research received no specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Competing interests
The authors declare no competing interests.
Ethical approval
This study involved secondary analysis of an existing dataset and did not involve the recruitment of human participants or collection of new human-subject data by the authors.
Proofs
Proof of Proposition 2.1
For one observation, write
From (6)–(7), the score for can be written as
where
Conditional on , the acquisition indicator satisfies
Moreover, by (2),
and
Since is a conditional likelihood score,
Assumption (3) further implies that, conditional on and , the response indicator and the class label are independent. It follows that
Thus the score contribution from the response indicator is orthogonal, conditional on , to the score contribution from the returned class label. The conditional second moment of the returned-label term is
where
Similarly,
Combining (S1.2)–(S1.4) gives the conditional Fisher information generated by a query:
It remains to combine the query contribution with the information in the feature vector. The feature score satisfies
Conditional on , the mean of the query-score contribution in (S1.1) is zero:
Hence the feature score is orthogonal to the query-score contribution. Since is generated according to the known parameter-free design (1),
Therefore
which is (9). Substitution of (S1.5) yields (10), completing the proof. □
Proof of Proposition 3.1
For notational simplicity, write
By (19),
Let
Since is symmetric and positive definite,
Consider
Expanding the outer product and using (3.4)–(3.6) gives
Because
and
the preceding expression reduces to
which proves (21).
For any ,
Hence
It remains to characterize when the matrix is zero. If
-almost surely, then (21) immediately gives .
Conversely, suppose that . For every ,
The integrand is nonnegative, and therefore
-almost surely for every . Taking successively to be the elements of the canonical basis of yields
-almost surely. This establishes (22) and completes the proof.
Proof of Proposition 4.1
Let
for in a neighborhood of zero such that remains admissible. By Proposition 2.1,
and hence
For any differentiable nonsingular matrix-valued function ,
Applying this identity to and using (S1.6) gives
By linearity of expectation and trace,
which establishes (34).
It remains to obtain the decomposition of . From (8),
Substituting this expression into (30) and using linearity of the trace yields
Using
with the symmetry of and , the second term becomes
Therefore
with and given by (32) and (33), respectively. Finally, since , , and is symmetric,
This completes the proof. □
Proof of Proposition 5.1
Let and be two designs for which the corresponding information matrices are positive definite, and let
By Proposition 2.1,
Set
Then
Using
we obtain
Differentiating once more gives
Let
Since , we have , while is symmetric. Cyclic invariance of the trace yields
Because and ,
Hence
which proves convexity along every line segment in the positive-definite information domain. Therefore is convex on any convex subset of that domain. □
Proof of Theorem 5.2
Let be optimal and write . All probabilities and expectations in this proof are taken under . For , choose measurable sets and of positive probability, and put
Since , this exchange preserves the budget. The slack on and ensures that remains in for sufficiently small . Its information matrix remains positive definite by continuity. The right derivative, given by the same differentiation as in Proposition 4.1, therefore satisfies
It follows that
Indeed, if this inequality failed, there would be positive-probability subsets of and on which the scores were separated in the opposite order. Intersecting these subsets with and , respectively, preserves positive probability for sufficiently small , contradicting the exchange inequality. Both sets have positive probability because . Since the score is nonnegative and integrable, the displayed interval contains a finite . For such a , almost surely on , and almost surely on . These inequalities give the stated threshold conditions, including equality wherever .
For sufficiency, let be any feasible design and interpret as the right derivative along the feasible segment from to . By convexity,
The derivative term is
Since both designs satisfy the same budget,
and therefore
If , then and . If , then and . If , then . Thus the integrand is nonnegative almost surely, so
Consequently,
for every feasible , proving global optimality.
Proof of Proposition 5.3
From (37),
Therefore, for any ,
which proves (38).
Suppose first that there exists such that
-almost surely. Then
so cannot be positive definite.
Conversely, if is not positive definite, its positive semidefiniteness implies that there exists for which
Hence
The integrand is nonnegative, and therefore
-almost surely. This establishes the equivalence.
For the model
we have
Let expectation under the normalized measure be denoted by . Apart from the positive normalizing constant , the nuisance-information matrix is
Its determinant is
Thus, when ,
which proves (41). □
Proof of Theorem 6.1
Oracle criterion value.
The approximate minimization condition and the definition of give
Since and minimizes there, . This proves (51). This part does not require independence of the observations.
Uniform fixed-design second moments.
For deterministic , put
The iid score identities imply exactly
Writing for the dimension of , the information lower bound therefore gives
By (52), , with the squared remainder tending to zero uniformly in expectation. Cauchy–Schwarz applied to the two cross terms yields
where denotes the operator norm for matrices. With , taking classification blocks and using the block-inverse identity gives
Consequently,
Uniform control of the risk remainder.
Define
for parameter perturbations in the domain of the classifier. By (23), for every there is a radius such that whenever . Condition (53) gives a finite constant such that, for all sufficiently large ,
On the complement of the radius- ball, boundedness of classification risk implies . Markov’s inequality gives
Both bounds tend to zero. Inside the ball,
The last supremum is bounded, by either the moment condition or the preceding second-moment calculation. Taking the limit superior and then letting proves
Together with the quadratic term, this establishes the uniform expansion
Conditioning on the pilot.
For deterministic , let . The selected design is -measurable, and the estimator uses the pilot only through this design. The conditional product law (45) therefore gives the exact identity
Set . Equation (S1.7) implies
which proves (54). This step is why uniform fixed-design control is needed instead of an expansion only for each individual deterministic design.
Averaging over the pilot.
Taking expectations now gives
On ,
The convergence in probability in (51), together with this deterministic bound, implies . This proves (55). Finally, satisfies , so replacing by in the leading denominator changes the expression by . This change of denominator concerns sample-size accounting only; the proof has not treated the pooled two-stage observations as iid. □
Derivation of the binary logistic risk expansion
For the binary logistic model,
the Bayes classifier under parameter changes class at
Let
For sufficiently close to , the classifier based on differs from the Bayes classifier only on the interval between and . Hence
Since
the integrand vanishes at . Moreover,
so
Therefore,
Substitution into (S1.8) gives
Now let
The gradient of the boundary function is
At , using ,
Thus
Substituting this expansion into (S1.9) yields
Hence
with
Response-term ablation
The ablation uses the full query–response information matrix and the same joint likelihood as response-aware acquisition, but omits from candidate scores. All comparisons use 500 paired replications and total query budgets including the common pilot.
For each replication, the paired difference is the excess risk of the full criterion minus that of the ablation. Intervals are pointwise 95% Monte Carlo intervals. Two-sided paired -test -values are adjusted by Holm’s method over the nine regime–budget comparisons separately within the oracle and adaptive studies. The oracle null comparisons are exactly zero; their intervals are degenerate and their -values are recorded as one by convention.
| Study | Response | Difference | 95% interval | ||
| Oracle | Noninformative | 300 | 0.0000 | [0.0000, 0.0000] | 1.0000 |
| 600 | 0.0000 | [0.0000, 0.0000] | 1.0000 | ||
| 900 | 0.0000 | [0.0000, 0.0000] | 1.0000 | ||
| Moderate | 300 | -0.0874 | [-0.3741, 0.1994] | 1.0000 | |
| 600 | -0.0960 | [-0.2255, 0.0335] | 1.0000 | ||
| 900 | -0.1107 | [-0.1862, -0.0352] | 0.0373 | ||
| Strong | 300 | -0.0372 | [-0.1182, 0.0437] | 1.0000 | |
| 600 | 0.0013 | [-0.0162, 0.0188] | 1.0000 | ||
| 900 | -0.0071 | [-0.0198, 0.0056] | 1.0000 | ||
| Adaptive | Noninformative | 300 | 0.2192 | [-0.1308, 0.5691] | 1.0000 |
| 600 | 0.0707 | [-0.0577, 0.1990] | 1.0000 | ||
| 900 | 0.0354 | [-0.0181, 0.0889] | 1.0000 | ||
| Moderate | 300 | -0.2205 | [-0.4623, 0.0214] | 0.5912 | |
| 600 | -0.0832 | [-0.2036, 0.0372] | 1.0000 | ||
| 900 | -0.1077 | [-0.1726, -0.0428] | 0.0107 | ||
| Strong | 300 | -0.0305 | [-0.1247, 0.0638] | 1.0000 | |
| 600 | -0.0259 | [-0.0602, 0.0083] | 0.9648 | ||
| 900 | -0.0071 | [-0.0247, 0.0104] | 1.0000 |
Table S1. Paired excess-risk differences for response-aware acquisition versus the label-only, full-information ablation. Differences and interval endpoints are multiplied by . Negative values favor response-aware acquisition.
Figure S1 displays the paired differences and pointwise intervals. The intervals are not simultaneous confidence intervals.
Figure S1. Excess-risk differences between the full response-aware criterion and the label-only, full-information ablation. Negative differences favor the full criterion. Bars are pointwise 95% Monte Carlo intervals over 500 paired replications. Query budgets include the pilot.
References
- [1]Ahfock, D. and McLachlan, G. J. (2020). An apparent paradox: a classifier based on a partially classified sample may have smaller expected error rate than that if the sample were completely classified. Statistics and Computing, 30:1779–1790. https://doi.org/10.1007/s11222-020-09971-5.
- [2]Amin, K., DeSalvo, G., and Rostamizadeh, A. (2021). Learning with labeling induced abstentions. In Advances in Neural Information Processing Systems, volume 34, pages 12576–12586. Curran Associates, Inc. Bach, F. R. (2006). Active learning for misspecified generalized linear models. In Advances in Neural Information Processing Systems, volume 19, pages 65–72. MIT Press.
- [3]Bickford Smith, F., Kirsch, A., Farquhar, S., Gal, Y., Foster, A., and Rainforth, T. (2023). Prediction-oriented Bayesian active learning. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 7331–7348. PMLR. Available at: https://proceedings.mlr.press/v206/bickfordsmith23a.html.arxiv.org/abs/2304.08151
- [4]Chen, Y. and Biros, G. (2023). FIRAL: An active learning algorithm for multinomial logistic regression. In Advances in Neural Information Processing Systems, volume 36, pages 65894–65905. Curran Associates, Inc. https://doi.org/10.52202/075280-2877.
- [5]Fang, M. and Zhu, X. (2014). Active learning with uncertain labeling knowledge. Pattern Recognition Letters, 43:98–108. https://doi.org/10.1016/j.patrec.2013.10.011.
- [6]Filstroff, L., Sundin, I., Mikkola, P., Tuilpin, A., Kylmäoja, J., and Kaski, S. (2024). Targeted active learning for Bayesian decision-making. Transactions on Machine Learning Research. Available at: https://openreview.net/forum?id=KxPju iMgmm.
- [7]Hoi, S. C. H., Jin, R., Zhu, J., and Lyu, M. R. (2006). Batch mode active learning and its application to medical image classification. In Proceedings of the 23rd International Conference on Machine Learning, pages 417–424. ACM. https://doi.org/10.1145/1143844.1143897.
- [8]Huang, Z., Bickford Smith, F., and Rainforth, T. (2026). Loss-driven Bayesian active learning. In Proceedings of the 29th International Conference on Artificial Intelligence and Statistics, volume 300 of Proceedings of Machine Learning Research, pages 5140–5148. PMLR. Available at: https://proceedings.mlr.press/v300/huang26b.html.arxiv.org/abs/2604.11995
- [9]Imberg, H., Jonasson, J., and Axelson-Fisk, M. (2020). Optimal sampling in unbiased active learning. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 559–569. PMLR. Available at: https://proceedings.mlr.press/v108/imberg20a.html.
- [10]Lewis, D. D. and Catlett, J. (1994). Heterogeneous uncertainty sampling for supervised learning. In Machine Learning Proceedings 1994, pages 148–156. Morgan Kaufmann. https://doi.org/10.1016/B978-1-55860-335-6.50026-X.
- [11]Lewis, D. D. and Gale, W. A. (1994). A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3–12. Springer. https://doi.org/10.1007/978-1-4471-2099-5_1.
- [12]Nguyen, C. V., Ho, L. S. T., Xu, H., Dinh, V., and Nguyen, B. T. (2022). Bayesian active learning with abstention feedbacks. Neurocomputing, 471:242–250. https://doi.org/10.1016/j.neucom.2021.11.027.
- [13]Pukelsheim, F. (2006). Optimal Design of Experiments. Society for Industrial and Applied Mathematics, Philadelphia, PA. https://doi.org/10.1137/1.9780898719109.
- [14]Robinson, T. S., Tax, N., Mudd, R., and Guy, I. (2024). Active learning with biased non-response to label requests. Data Mining and Knowledge Discovery, 38(4):2117–2140. https://doi.org/10.1007/s10618-024-01026-x.
- [15]Roy, N. and McCallum, A. (2001). Toward optimal active learning through Monte Carlo estimation of error reduction. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 441–448. Morgan Kaufmann.
- [16]Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3):581–592. https://doi.org/10.1093/biomet/63.3.581.
- [17]Setoudehtazangi, F. and McLachlan, G. J. (2026a). Informative label missingness in multiclass classification: Information geometry and excess risk. arXiv preprint arXiv:2608.30561. https://doi.org/10.48550/arXiv.2608.30561.
- [18]Setoudehtazangi, F. and McLachlan, G. J. (2026b). Classification-risk-optimal label acquisition. arXiv preprint arXiv:2609.06873. https://doi.org/10.48550/arXiv.2609.06873.
- [19]Settles, B. (2009). Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison. Available at: https://research.cs.wisc.edu/techreports/2009/TR1648.pdf.
- [20]Sourati, J., Akcakaya, M., Leen, T. K., Erdogmus, D., and Dy, J. G. (2017). Asymptotic analysis of objectives based on Fisher information in active learning. Journal of Machine Learning Research, 18(34):1–41. Available at: https://www.jmlr.org/papers/v18/15-104.html.
- [21]Totakura, V., Singh, A., Dong, Y., and Chakraborty, S. (2026). An analysis of active learning algorithms using real-world crowd-sourced text annotations. In Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN 2026). Accepted for publication. Preprint available at: https://arxiv.org/abs/2604.23290.
- [22]Wang, H., Zhu, R., and Ma, P. (2018). Optimal subsampling for large sample logistic regression. Journal of the American Statistical Association, 113(522):829–844. https://doi.org/10.1080/01621459.2017.1292914.
- [23]Wang, Y.-G., Wu, J., and McLachlan, G. J. (2026). Learning from uncertainty-dependent missing labels for semi-supervised classification. arXiv preprint arXiv:2608.23960. https://doi.org/10.48550/arXiv.2608.23960.
- [24]Yan, S., Chaudhuri, K., and Javidi, T. (2016). Active learning from imperfect labelers. In Advances in Neural Information Processing Systems, volume 29, pages 2128–2136. Curran Associates, Inc.arxiv.org/abs/1610.09730
- [25]Yao, Y. and Wang, H. (2019). Optimal subsampling for softmax regression. Statistical Papers, 60(2):585–599. https://doi.org/10.1007/s00362-018-01068-6.




