betaFeedback

WorksComputer Science

Learning to Defer under Expert Drift

Abstract

The learning to defer (L2D) framework enables safety in machine learning and AI systems by allocating difficult or critical decisions to a human expert. However, most prior work on L2D assumes fixed expert reliability, overlooking temporal fluctuations in human performance due to factors such as fatigue and cognitive biases. Humans commonly exhibit such substantial, systematic performance shifts across a wide range of domains, from high-stakes settings such as radiology, aviation, and judicial decision-making, to everyday tasks such as crowdsourced image annotation. We propose Expert-Drift-Adapted Learning-to-Defer (EDA-L2D), a novel expert-aware L2D framework that explicitly models these temporal shifts by conditioning the deferral head on recent histories of expert outcomes via a sequence model. This history-aware approach enables adaptive deferral policies that anticipate reliability fluctuations and better allocate decisions between human and machine over time. We provide comprehensive experiments across three diverse tasks – image classification, medical diagnosis, and hate speech detection – including generalization analyses under realistic deferral conditions and irregular shifts in expert reliability. Our results show that EDA-L2D adapts more effectively than prior L2D methods to diverse temporal patterns of expert reliability, resulting in improved system performance and more effective deferral decisions. Code is available at https://github.com/ZiyaoMou/L2D-Expert-Drift.

Introduction

Hybrid intelligent (HI) systems combine human and machine intelligence to achieve goals unattainable by either alone, emphasizing complementarity over replacement [14, 9, 2]. One popular HI paradigm is learning to defer (L2D): the system learns not only to predict a label but also to decide when to hand off the decision to a human expert. The original formulation of L2D shows that accounting for the expert’s strengths and biases can improve overall accuracy and fairness relative to a static rejection framework [20].

However, most existing L2D systems assume that expert behavior is fixed or stationary. In practice, human performance often varies significantly over time due to fatigue [10, 28, 26, 4] and various cognitive biases [19, 31, 32]. These changes can be large enough to matter operationally. For example, in a workshop-based mammography interpretation study involving 197 radiologists, specificity decreased from 82.74% in the early morning to 71.70% in the late morning, a decline of more than 11 percentage points [3]. Even in crowdsourced image annotation, our analysis of CIFAR-10H [27] identifies 9 annotators who passed the dataset’s official attention checks but nevertheless exhibited accuracy declines of more than 10 percentage points between the first and second halves of their annotation sessions (Appendix § 9.§ 1). That expert drift can be observed even in such a simple task, together with the findings above, motivates L2D models that can adapt to expert drift over time.

We address this challenge by proposing a novel framework for L2D: Expert-Drift-Adapted Learning to Defer (EDA-L2D). Our framework explicitly models non-stationarity in expert performance. We modify L2D’s rejector sub-component to condition on short-horizon histories of expert decisions. This allows the system to be aware of the expert’s current behavior, and modify its deferral policy accordingly. We illustrate our approach in Figure 1a. Concretely, we integrate a sequence model into the deferral head to explicitly model expert reliability over time. We validate our approach across diverse tasks including medical image recognition, hate speech detection, and image classification. We demonstrate consistent gains in system accuracy over existing, time-agnostic L2D approaches. In summary, our contributions are as follows:

Figure 1. (a) An example use case of our EDA-L2D framework in a setting with expert performance drift: a radiologist whose performance degrades over a workday. Our model monitors past outcomes to infer current expert reliability in real time, deferring to the expert when reliable and assigning cases to the model when the expert’s performance declines. (b) Overview of our Expert-Drift-Adapted L2D framework. A streaming sequence of inputs is encoded by the backbone into features. At each timestep, the feature vector is concatenated with the expert’s most recently observed correctness, which is available only for inputs previously deferred to the expert, and passed to an RNN module. The classifier head outputs class logits and the defer head outputs a deferral score; a deferral gate then assigns the instance to either the expert or the classifier. Because the expert’s competence drifts over time, this design makes the L2D system time-aware and adaptive to temporal shifts.

  • We propose a novel Expert-Drift-Adapted Learning to Defer (EDA-L2D) that explicitly models variance in human expert performance over time, allowing the system to flexibly adapt to realistic human experts. We propose our approach in § 3, including a new time-aware learning objective (§ 3.2) and model formulation (§ 3.3).

  • We validate EDA-L2D on three real-world tasks that call for human–AI collaboration, each paired with a simulated expert whose accuracy drifts over time: image classification on CIFAR-10 with drift curves taken from real CIFAR-10H annotators (§ 5.1), chest X-ray diagnosis on CheXpert with a radiologist’s drift over a working week (§ 5.2), and hate speech detection with synthetic forms of drift (§ 5.3). Both parameterizations of EDA-L2D consistently outperform standard L2D, and we further test their robustness to expert changes not seen in training and to delayed, noisy or missing feedback.

Background

We first describe the traditional L2D setting [22], in which the expert’s abilities are assumed to be static and time-invariant.

Data & Models

We will exclusively consider the problem of multi-class classification with a feature space X\mathcal{X} and the label space Y={1,…,K}\mathcal{Y} = \{1,\ldots,K\}. Given some input x∈Xx \in\mathcal{X}, we define P(y∣x)\mathbb{P}(y \mid x) as the unknown label-generating distribution and P(m∣x)\mathbb{P}(m \mid x) as the unknown distribution over expert predictions m∈Ym \in\mathcal{Y}. Here, yy denotes the true label, whereas mm denotes the prediction made by a fallible expert and may therefore differ from yy. Their joint conditional distribution determines the expert’s probability of being correct, P(m=y∣x)\mathbb{P}(m = y \mid x), which is compared with the classifier’s probability of being correct when deciding whether to defer. L2D is comprised of two sub-components: a classifier h:X→Yh : \mathcal{X} \to\mathcal{Y} and a rejector r:X→{0,1}r : \mathcal{X} \to\{0,1\}, which decides whether to make a prediction using the classifier (r(x)=0r(x) = 0) or by deferring to the expert (r(x)=1r(x) = 1).

Classifier-Rejector Loss Function

To train our L2D model, we need to fit both the rejector and classifier models. The classifier incurs a loss of zero (correct) or one (incorrect) when it makes a prediction. Similarly, the human expert incurs the same 0-1 loss when making a prediction (i.e. r(x)=1r(x) = 1). Combining these two 0-1 losses using the rejector model results in the combined classifier-rejector loss:

L0−1(h,r)=Ex,y,m[(1−r(x))I[h(x)≠y]+r(x)I[m≠y]](1)L_{0-1}(h,r) = \mathbb{E}_{x,y,m}[(1-r(x))\mathbb{I}[h(x) \ne y] + r(x)\mathbb{I}[m \ne y]] \tag*{(1)}

where I\mathbb{I} denotes an indicator function checking the prediction against the ground-truth label. When minimizing this loss, the resulting Bayes optimal classifier and rejector satisfy:

h∗(x)=arg⁡max⁡y∈Y P(y=y∣x),r∗(x)=I[P(m=y∣x)≥max⁡y∈YP(y=y∣x)](2)h^*(x) = \underset{y \in\mathcal{Y}}{\arg\max}\ \mathbb{P}(y = y \mid x), \qquad r^*(x) = \mathbb{I}\left[\mathbb{P}(m = y \mid x) \ge\underset{y \in\mathcal{Y}}{\max}\mathbb{P}(y = y \mid x)\right] \tag*{(2)}

where P(y∣x)\mathbb{P}(y \mid x) represents the label probability under the data generating process, and P(m=y∣x)\mathbb{P}(m = y \mid x) is the probability of the expert making the correct prediction. The expert may have knowledge not available to the classifier, so they could outperform the Bayes optimal classifier.

Softmax Surrogate Loss

A consistent surrogate loss for the above L0−1L_{0-1} loss can be derived following the formulation from Mozannar & Sontag [22]. First, we consider an augmented label space that includes both the label space Y\mathcal{Y} and the rejection option ⊥\bot: Y⊥:=Y∪{⊥}\mathcal{Y}^{\bot} := \mathcal{Y} \cup\{\bot\}. Secondly, for a class k∈[1,K]k \in[1,K], let gk:X↦Rg_k : \mathcal{X} \mapsto\mathbb{R}, and let g⊥:X↦Rg_{\bot} : \mathcal{X} \mapsto\mathbb{R} denote the rejection option. We can combine these K+1K+1 with a loss resembling the cross-entropy loss for a softmax parameterization:

ϕSM(g1,…,gK,g⊥;x,y,m)=−log⁡(exp⁡{gy(x)}∑y′∈Y⊥exp⁡{gy′(x)})−I[m=y]log⁡(exp⁡{g⊥(x)}∑y′∈Y⊥exp⁡{gy′(x)})(3)\phi_{\mathrm{SM}}(g_1,\ldots,g_K,g_{\bot};x,y,m) = -\log\left(\frac{\exp\{g_y(x)\}}{\sum_{y' \in\mathcal{Y}^{\bot}}\exp\{g_{y'}(x)\}}\right) - \mathbb{I}[m = y]\log\left(\frac{\exp\{g_{\bot}(x)\}}{\sum_{y' \in\mathcal{Y}^{\bot}}\exp\{g_{y'}(x)\}}\right) \tag*{(3)}

Here, the first term maximizes gkg_k for the true label kk. The second term maximizes the rejection function g⊥g_{\bot}, but only when the expert’s prediction is correct. At test time, the classifier takes the maximum over the classes: y^=h(x)=arg⁡max⁡k∈[1,K] gk(x)\hat{y}=h(x)=\underset{k \in[1,K]}{\arg\max}\ g_k(x). Similarly, we formulate the rejection function as r(x)=I[g⊥(x)≥max⁡k∈[1,K]gk(x)]r(x)=\mathbb{I}[g_{\bot}(x) \ge\underset{k \in[1,K]}{\max}g_k(x)]. The minimizers g1∗,…,gK∗,g⊥∗g_1^*,\ldots,g_K^*,g_{\bot}^* of ϕSM\phi_{\mathrm{SM}} also uniquely minimize the 0-1 loss from (1), L0−1(h,r)L_{0-1}(h,r).

Expert-Drift-Adapted Learning to Defer

This section introduces our Expert-Drift-Adapted Learning to Defer (EDA-L2D) framework, which extends L2D to account for non-stationarity in expert performance. Accordingly, we derive the corresponding softmax surrogate loss for this setting. We also propose multiple parameterizations, including one that uses a recurrent neural network to incorporate short-horizon temporal context into the deferral policy.

Setting: Non-Stationary Expert

We consider the same setting as § 2—L2D for multi-class classification—except now we assume the expert is non-stationary. Thus, instead of a fixed distribution P(m∣x)\mathbb{P}(\mathbf{m}\mid x), the expert’s predictions are generated by Pt(mt∣x)\mathbb{P}_{t}(m_{t}\mid x), where t∈T={1,…,T}t\in\mathcal{T}=\{1,\ldots,T\} is a discrete time index: the position of an input in the sequence of inputs the expert receives, such as the tt-th image of an annotation session or the tt-th case of a working day. Note that we are assuming the classification task itself is stationary, i.e. y∼P(y∣x)y\sim\mathbb{P}(y\mid x), which still has no time dependence.

To ground the notation in an example, consider an L2D system for radiology. The problem of forming a diagnosis or other prediction from the medical image itself is a static task, since we assume the distribution of patients and the mechanism that govern their health is not changing over time (or is at least changing so slowly as to be negligible, e.g. drift in hospital patient demographics). Yet the radiologist who is paired with the L2D system will change, at the very least becoming more tired and distracted over the course of a 10+ hour shift. The expert drift may also be non-monotonic: perhaps her performance degrades over the course of a morning, but after lunch and a mid-day walk, the expert feels rejuvenated for a few more hours.

Returning to the technical details, our EDA-L2D system will have a classifier that is defined just as above: h:X→Yh:\mathcal{X}\to\mathcal{Y}. Again, the classifier is time-invariant since the prediction task itself is not time dependent. The difference, however, arises in the rejector: r:X×T↦{0,1}r:\mathcal{X}\times\mathcal{T}\mapsto\{0,1\}, meaning the rejector is a function of both the feature space X\mathcal{X} and time T\mathcal{T}. While in the simplest case T\mathcal{T} would be a scalar time index, we could also formulate T\mathcal{T} as a feature space that describes the current state of the expert (e.g. tabular biomarkers).

Learning Objective and Bayes Solutions

The learning objective is similar to that in § 2 (equation 1), except now it takes the form of a summation over time:

L0−1T(h,r)=∑t∈TEx,y,mt[(1−r(x,t))I[h(x)≠y]+r(x,t)I[mt≠y]].(4)L_{0-1}^{T}(h,r)=\sum_{t\in\mathcal{T}}\mathbb{E}_{x,y,m_t}\left[(1-r(x,t))\mathbb{I}[h(x)\ne y]+r(x,t)\mathbb{I}[m_t\ne y]\right]. \tag*{(4)}

As the distribution of yy does not depend on time, the classifier’s Bayes solution is just as before: h∗(x)=arg⁡max⁡y∈YP(y=y∣x)h^*(x)=\arg\max_{y\in\mathcal{Y}}\mathbb{P}(y=y\mid x). Yet the Bayes optimal rejector does change, becoming time-dependent like so:

r∗(x,t)=I[Pt(mt=y∣x)≥max⁡y∈YP(y=y∣x)].r^*(x,t)=\mathbb{I}\left[\mathbb{P}_{t}(m_t=y\mid x)\ge\max_{y\in\mathcal{Y}}\mathbb{P}(y=y\mid x)\right].

The intuition is the same as above—compare the chance of expert correctness vs classifier confidence for the modal label—yet now the expert correctness term is time dependent, Pt(mt=y∣x)\mathbb{P}_{t}(m_t=y\mid x).

Implementations of the Softmax Surrogate Loss

The derivation for the corresponding softmax surrogate loss follows fairly directly from Mozannar & Sontag [22]’s original derivation. Yet we consider two distinct parameterizations of the underlying model. The first is a ‘per time step’ version that treats the L2D problem independently at each time step. We consider this the most naive extension of traditional L2D that yet still satisfies our motivating setting of expert drift. We then go on to describe how to use a recurrent neural network (RNN) to parameterize the rejector, which allows the expert’s previous predictions to help inform the deferral policy. For all implementations, we assume we have access to a training set D={{(xt,n,yt,n,mt,n)}n=1N}t∈T\mathcal{D}=\left\{\left\{(x_{t,n},y_{t,n},m_{t,n})\right\}_{n=1}^{N}\right\}_{t\in\mathcal{T}}, which consists of NN feature-label-expert-prediction triplets at each of ∣T∣|\mathcal{T}| time steps. Again, we emphasize that only the underlying distribution of mt,nm_{t,n} is time-varying, and the time indices on xt,n,yt,nx_{t,n},y_{t,n} are there merely to ‘synchronize’ the feature-label pairs with the expert predictions.

Per-Time-Step Implementation and Loss The softmax surrogate in equation (3) can be naturally extended to the drifting-expert setting by instantiating a traditional L2D model at each time step. Specifically, we define a model at each time step and train it locally for that time step via the surrogate:

ϕSM(g1,…,gK,g⊥t;x,y,mt)=−log⁡exp⁡{gy(x)}∑y′∈Yexp⁡{gy′(x)}−I[mt=y]log⁡exp⁡{g⊥t(x)}∑y′∈Y⊥exp⁡{gy′(x)}.(5)\phi_{\mathrm{SM}}(g_1,\ldots,g_K,g_{\perp}^{t};x,y,m_t)=-\log\frac{\exp\{g_y(x)\}}{\sum_{y'\in\mathcal{Y}}\exp\{g_{y'}(x)\}}-\mathbb{I}[m_t=y]\log\frac{\exp\{g_{\perp}^{t}(x)\}}{\sum_{y'\in\mathcal{Y}_{\perp}}\exp\{g_{y'}(x)\}}. \tag*{(5)}

Notice that g⊥tg_{\perp}^{t} is specific to the current time step, but g1,…,gKg_1,\ldots,g_K could be shared across all time steps (since they encode the classifier). As mentioned above, we consider this a naive implementation and will primarily use it to benchmark our second approach, described below. Having g⊥t(x)g_{\perp}^{t}(x) only be a function of xx means that it ignores patterns that could be detected by having information about the expert’s behavior at previous time steps. Moreover, the per-step parameterization has no dedicated defer head for time steps beyond the training horizon. Applying it to longer sequences therefore requires an extrapolation rule, such as reusing the final learned head, which we evaluate in Appendix E.

Loss for RNN-Based Model We next consider a more sophisticated implementation that uses an RNN-based parameterization to incorporate information about the expert’s previous predictions. Here we use a surrogate loss defined across all time steps:

ϕSMT(g1,…,gK,g⊥;{xt,yt,mt}t∈T)=∑t∈T−log⁡exp⁡{gyt(xt)}exp⁡{g⊥(xt,t)}+∑y′∈Yexp⁡{gy′(xt)}−I[mt=yt]log⁡exp⁡{g⊥(xt,t)}exp⁡{g⊥(xt,t)}+∑y′∈Yexp⁡{gy′(xt)}.(6)\phi_{\mathrm{SM}}^{T}(g_1,\ldots,g_K,g_{\perp};\{x_t,y_t,m_t\}_{t\in T})=\sum_{t\in T}-\log\frac{\exp\{g_{y_t}(x_t)\}}{\exp\{g_{\perp}(x_t,t)\}+\sum_{y'\in\mathcal{Y}}\exp\{g_{y'}(x_t)\}}-\mathbb{I}[m_t=y_t]\log\frac{\exp\{g_{\perp}(x_t,t)\}}{\exp\{g_{\perp}(x_t,t)\}+\sum_{y'\in\mathcal{Y}}\exp\{g_{y'}(x_t)\}}. \tag*{(6)}

where here g⊥(xt,t)g_{\perp}(x_t,t) is parameterized with an RNN so that it can leverage information from previous time steps. While we have left the time feature tt vague, this will contain the expert’s previous prediction mt−1m_{t-1} when it is available. We next describe the neural architecture in more detail.

RNN Architecture The RNN first takes as input the feature vector xtx_t, producing a hidden representation ft=ψ0(xt)f_t=\psi_0(x_t). For instance, if xtx_t is an image, then ψ0(⋅)\psi_0(\cdot) could be a convolutional NN. The feature representation is then passed into a recurrent module: zt=ψ1(zt−1,ft,m^t−1)z_t=\psi_1(z_{t-1},f_t,\hat{m}_{t-1}). During training we use teacher forcing: m^t−1=I[mt−1=yt−1]\hat{m}_{t-1}=\mathbb{I}[m_{t-1}=y_{t-1}] is the expert’s true correctness at every step. At test time this is not available. The expert is consulted only on the inputs the system defers, so their correctness can only be observed for those inputs, for example when the expert’s decisions are later verified. We therefore evaluate under realistic routing: if input t−1t-1 was deferred, m^t−1\hat{m}_{t-1} is the expert’s observed correctness; otherwise, m^t−1\hat{m}_{t-1} keeps the last observed value, with a fixed initial value before the first deferral. Feedback in practice may also arrive late, be recorded incorrectly, or be missing, and we study these cases in Section 5.1 and Appendix E.1. Lastly, the gg-functions defined in (6) are produced via a final linear transformation: gk(xt)=wkTztg_k(x_t)=w_k^{\mathsf{T}}z_t (k∈[1,K]k\in[1,K]) and g⊥(xt)=w⊥Tztg_{\perp}(x_t)=w_{\perp}^{\mathsf{T}}z_t. The predicted label is chosen via arg max⁡kgk(xt),g⊥(xt)\operatorname*{arg\,max}_k g_k(x_t),g_{\perp}(x_t), which quantifies the preference for deferral, with the system deferring if g⊥(xt)≥max⁡kgk(xt)g_{\perp}(x_t)\ge\max_k g_k(x_t).

Related Work

Learning to Defer (L2D) Early work on classifiers that can choose to reject or abstain instead of making a prediction [7] inspired more recent work on how to model what happens after the classifier abstains [20], with Mozannar & Sontag [22] deriving the first consistent surrogate loss for the problem. This work has been expanded in recent years to address several limitations, including miscalibration [33, 5], underfitting [24], realizable consistency [23], sample complexity [6], data scarcity [11], and deferral to multiple experts [34]. However, these approaches do not consider the ‘messiness’ of modeling human experts—as we do—and could just as well be applied to a black-box API.

L2D for Sequences and Populations L2D has also been extended to the difficult settings of sequential tasks and encountering novel experts. Regarding the former, Sequential Learning-to-Defer [13] incorporates temporal dynamics by framing deferral as a sequential decision-making problem, employing model-based reinforcement learning. This work, however, is primarily concerned with the sequential nature of the prediction task itself and not robustness to expert drift, which is our primary concern. Regarding the latter, Learning to Defer to a Population (Tailor et al., [29]) uses meta-learning to adapt to never-before-seen experts at test-time. Their motivation is to be robust to novel experts (with the novelty not being a function of time), whereas we are concerned about modeling the temporal drift of a known expert.

Temporal Shifts in Human Performance In many high-stakes domains, human performance shift over time (due to fatigue or other cognitive factors) are well-documented and widely studied. Fatigue is extremely common in high-stakes domains and long-horizon tasks, such as radiology (Bruni et al., [4]), automated driving (Figalová et al., [10]), air traffic control (Peukert et al., [28]), and flying planes (Pan et al., [26]). Beyond affecting surface-level performance on these tasks, fatigue physically affects the brain, causing reduced alertness and vigilance that can be measured directly, e.g. using EEG signals (Figalová et al., [10]). We extend L2D to be adaptive to these variations in human performance.

L2D with Time-Varying Experts Recent work has also begun to relax the assumption of a static expert. Zhang et al. [36] model expert accuracy as a parametric function of cumulative workload and learn a deferral policy with constrained reinforcement learning under a coverage budget. Montreuil et al. [21] study online L2D with bandit feedback, where the set of available experts and their reliability may change across rounds. In contrast, EDA-L2D conditions the deferral decision on the expert’s observed history of correctness, allowing it to track drift that follows no prescribed form.

Experiments

We evaluate our proposed EDA-L2D framework through a progression of controlled simulations. We train the two variants of EDA-L2D (per-step and RNN-based) on three benchmarks spanning distinct modalities and annotation regimes: CIFAR-10 (image classification), CheXpert (chest X-ray classification) and Hate Speech (text moderation). Our primary baseline is “Naive L2D”, meaning that we apply a traditional L2D system that has no awareness of time, collapsing all training data and presenting it to the model without an information about time ordering.

CIFAR-10 with Drifting CIFAR-10H Annotators

Task and Data. We study standard image classification on CIFAR-10 (Krizhevsky, [17]). The dataset contains 60,000 color images at 32×3232 \times32 resolution drawn from 10 object categories (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck), with 50,000 training images and 10,000 test images. Given an input image xx, the system either predicts its class label y∈{1,…,10}y \in\{1,\ldots,10\} or defers to the expert (⊥\perp). Images arrive in sequences, and the model can defer each one to a simulated expert whose accuracy drifts over time following accuracy curves of real annotators from CIFAR-10H (Peterson et al., [27]). We train on sequences of training images and evaluate on the test set in three ways: against simulated experts that follow

each annotator’s accuracy curve, against the annotators’ real answers, and against changes to the expert or its feedback that are not seen during training.

Sequence Construction and Evaluation. We cut the 50,000 CIFAR-10 training images into consecutive sequences of length T=50T = 50, so that each image’s time step is its position in its sequence. We then split the resulting 1,000 sequences at random into a training set containing 90% of them and a validation set containing the remaining 10%, so that each sequence belongs entirely to one set. The official 10,000-image test set is never seen during training and is used for two complementary evaluations. In the simulated evaluation, which provides our main results, we generate 20,000 test sequences per trained model, drawing at every time step a test image uniformly at random and the expert’s answer from the annotator’s accuracy curve; this yields low-variance estimates of system accuracy at every step. In the real-answer evaluation, we instead replay each annotator’s 200 actual answers on the test images they labeled, in their original order. Each trial is placed at the time step it was mapped to when constructing the accuracy curve, which splits the 200 trials into four interleaved sequences of 50 steps (Appendix § 12.§ 1). Unlike the simulated expert, whose correctness depends only on the time step, real answers also reflect which images the annotator found difficult, so this evaluation tests the simulator’s central simplification as well as the transfer to real behaviour.

Models and Training. All variants use a WideResNet-28-4 backbone with a (K+1)(K+1)-way output layer, where the extra output is the deferral option, and are trained with the L2D surrogate loss. Naive L2D uses a single time-independent rejector. Per-step param. uses a single shared backbone and classifier head with 50 defer heads, one for each time step. The RNN-parameterized EDA-L2D passes the backbone features and the previous expert correctness through a recurrent network, using either GRU or LSTM cells, followed by a linear output layer, and trains the backbone and the recurrent network jointly. All models are trained for 200 epochs with SGD, Nesterov momentum 0.9, weight decay 5×10−45 \times10^{-4}, a cosine learning-rate schedule, and a batch size of 200, which keeps the time steps of each sequence contiguous within a batch. We use 5 seeds for every reported result.

Results. Table 1 reports system accuracy averaged over the 50 time steps. Both EDA-L2D parameterizations clearly improve on Naive L2D for all three annotators (Per-step: +3.45, +2.01, +0.94 points; RNN: +3.32, +2.15, +0.92 for annotators 454, 1004 and 54), and both exceed the classifier and the expert alone. The two parameterizations perform similarly: Per-step is slightly better for annotator 454, the RNN for annotator 1004, and they tie for annotator 54, with differences of at most 0.14 points. Figure 2 shows where the gains come from: Naive L2D applies the same deferral rule throughout, whereas both EDA-L2D variants lower their coverage when the expert is accurate and keep more images when the expert’s accuracy drops, most visibly for annotator 454, whose accuracy falls sharply after step 30.

MethodAnnotator 454Annotator 1004Annotator 54
Naive L2D90.38 ± 0.0591.51 ± 0.0993.93 ± 0.09
EDA-L2D, Per-step param.93.83 ± 0.0893.52 ± 0.0794.86 ± 0.05
EDA-L2D, RNN param.93.70 ± 0.0393.66 ± 0.0794.84 ± 0.06
delay 10a10^{a}93.49 ± 0.1793.64 ± 0.0794.84 ± 0.06
no feedbackb^{b}93.34 ± 0.2293.62 ± 0.0794.80 ± 0.06

Table 1. System accuracy (%) on CIFAR-10 with drifting CIFAR-10H annotators, averaged over the 50 time steps; mean ± standard error over 5 seeds, each evaluated on 20,000 Monte Carlo test sequences.

Figure 2. System accuracy and coverage (fraction of images not deferred) along the 50 time steps for the three CIFAR-10H annotators. Lines show the mean over 5 seeds and shaded bands ±\pm one standard error. The RNN is evaluated under realistic routing, with the observed outcomes delivered immediately (feedback), every 10 steps (delay 10), or not at all (no feedback).

Real expert answers. The gains carry over from simulated to real expert behaviour. Given Table 6, when the annotators’ actual answers are replayed, EDA-L2D, RNN param. improves on Naive L2D by +3.8, +1.8 and +0.8 points for annotators 454, 1004 and 54, close to the +3.3, +2.2 and +0.9 points obtained in the simulated evaluation. With only 200 trials per annotator, these estimates are noisier: the gains are significant for annotators 454 and 1004 (paired block bootstrap, 95% intervals [1.7, 6.0] and [0.5, 3.3]), whereas the smaller gain for annotator 54 cannot be resolved at this sample size.

MethodAnnotator 454Annotator 1004Annotator 54
Naive L2D87.789.095.3
EDA-L2D, Per-step param.92.2 (+4.5 [2.6, 6.4])90.7 (+1.7 [0.0, 3.4])95.6 (+0.3 [−1.1, 1.7])
EDA-L2D, RNN param.91.5 (+3.8 [1.7, 6.0])90.8 (+1.8 [0.5, 3.3])96.1 (+0.8 [−0.3, 1.9])
delaya^{a}91.1 (+3.4 [1.0, 5.7])91.0 (+2.0 [0.7, 3.5])96.2 (+0.9 [−0.1, 1.9])
no feedbackb^{b}91.2 (+3.5 [1.3, 5.7])91.0 (+2.0 [0.7, 3.5])96.0 (+0.7 [−0.3, 1.7])

Table 6.

Role of Historical Feedback. The RNN can draw on two sources of information about the expert: the temporal pattern of their accuracy, learned during training, and feedback on their past correctness, observed only for deferred images (§ 3.3). When the expert follows the same curve at training and test time, the time step alone already reveals much of their accuracy, so feedback has little to add. Withholding it entirely costs at most 0.36 points, and the RNN without feedback still gains +2.96, +2.11 and +0.87 points over Naive L2D (Table 1). This contribution remains positive when feedback is delayed, noisy or only partly observed (Table 9). Feedback becomes more important when the expert departs from the pattern seen in training, because it is then the only signal of the change. During a temporary accuracy drop, its contribution grows with the depth of the drop, reaching 0.58 points for annotator 454 at a drop of 0.15 (Table 12). In short, the learned temporal pattern accounts for most of the gain when the expert behaves as expected, and feedback lets the model adapt when the expert does not (Appendix § 13).

Feedback conditionAnnotator 454Annotator 1004Annotator 54
Realistic routing (main setting)+0.360 ± 0.213+0.032 ± 0.002+0.047 ± 0.014
Delay k=5k = 5+0.296 ± 0.142+0.022 ± 0.002+0.047 ± 0.013
Delay k=10k = 10+0.150 ± 0.076+0.020 ± 0.002+0.042 ± 0.010
Noise ε=0.1\varepsilon= 0.1+0.333 ± 0.193+0.028 ± 0.002+0.045 ± 0.013
Noise ε=0.2\varepsilon= 0.2+0.303 ± 0.172+0.028 ± 0.001+0.041 ± 0.011
Noise ε=0.3\varepsilon= 0.3+0.272 ± 0.150+0.024 ± 0.002+0.038 ± 0.010
Partial π=0.5\pi= 0.5+0.324 ± 0.177+0.030 ± 0.002+0.045 ± 0.013
Partial π=0.25\pi= 0.25+0.252 ± 0.118+0.033 ± 0.003+0.039 ± 0.009

Table 9. Imperfect feedback on CIFAR-10 (original expert curve). Gain in system accuracy (percentage points) of the RNN under each feedback condition over the same RNN without feedback, over all 50 steps; mean ± s.e. over 5 seeds.

Annotatord=0d = 0d=0.05d = 0.05d=0.10d = 0.10d=0.15d = 0.15
454+0.37±0.21+0.37 \pm0.21+0.43±0.15+0.43 \pm0.15+0.51±0.11+0.51 \pm0.11+0.58±0.10+0.58 \pm0.10
1004+0.03±0.00+0.03 \pm0.00+0.17±0.02+0.17 \pm0.02+0.29±0.04+0.29 \pm0.04+0.39±0.06+0.39 \pm0.06
54+0.05±0.01+0.05 \pm0.01+0.09±0.03+0.09 \pm0.03+0.14±0.06+0.14 \pm0.06+0.20±0.10+0.20 \pm0.10

Table 12. Temporary dip on CIFAR-10: gain in system accuracy (percentage points) of EDA-L2D, RNN param. over the same model without feedback, during the dip; mean ±\pm s.e. over 5 seeds.

CheXpert with Radiologist Workweek Drift

Task and Data. CheXpert (Irvin et al., [12]) contains 224,316 chest radiographs from 65,240 patients at Stanford Hospital. We use the downsampled release and treat each observation as a one-vs-rest binary target. A binary mask excludes targets labeled uncertain. All images are converted to three channels, normalized, and resized to ImageNet-compatible resolution. For training augmentation, we apply random resized crops, horizontal flips, and random rotations of up to 15∘15^\circ. We split the data by patient into disjoint training, validation, and test sets containing 81%, 9%, and 10% of patients, respectively.

Expert. We simulate an expert whose accuracy drifts over a working week. The temporal pattern comes from a radiology fatigue study [35]: resident performance declines over each workday, partially recovers after midday breaks, and recovers less towards the end of the week. We digitize its mean trajectories for a general cohort (25 steps, 5 per day) and an emergency-department cohort (50 steps, 10 per day), rescale them to the range of balanced accuracies of individual CheXpert radiologists [12], and amplify the drift so that it is pronounced enough to distinguish the methods. Appendix § 9.§ 2 gives the full construction and Figure 6 shows the resulting curves.

Expert-accuracy trajectories for the CheXpert experiments

Figure 6. Expert-accuracy trajectories used for the CheXpert experiments, digitized from Figures 4 and 3b of Vosshenrich et al. [35]. The dashed line in each panel shows the trajectory rescaled to [0.695,0.885][0.695, 0.885]; the solid line shows it after amplifying the differences between weekdays and the fluctuations within each day by the factors given below each panel, then clipping to [0.5,1][0.5, 1].

Setting. We consider two settings, and every method is trained under both. In the per-day setting, each method is trained separately for each weekday on day-long sequences, which gives five models per method. Within a day, EDA-L2D, Per-step param. has one defer head per step, i.e., 5 heads in the general cohort and 10 in the emergency-department cohort. In the whole-week setting, each method is a single model trained on week-long sequences that must cover the entire week, and EDA-L2D, Per-step param. has 25 or 50 defer heads, respectively. Because Naive L2D is also trained per day in the per-day setting, it already adapts to differences between weekdays, so the gains of EDA-L2D in this setting reflect its ability to track drift within a day. We present the per-day setting in the main text; results for the whole-week setting and the system AUROC along the week for individual findings are given in Appendix § 12.§ 2.

Models and Training. All variants use a DenseNet-121 initialized with ImageNet weights, with one class output and one defer output per finding, and are fine-tuned with the same L2D loss from a classifier first trained without deferral. They differ only in the defer outputs: Naive L2D has a single one per finding, EDA-L2D, Per-step param. one per step, and EDA-L2D, RNN param. obtains them from a GRU over the image features and the expert’s past correctness on each finding. Training details are given in Appendix § 10.4.2.

Evaluation. For each finding, the system’s score on an image is the expert’s binary answer if the finding is deferred, and the model’s predicted probability of the finding otherwise. We compute the AUROC of these mixed scores separately for each finding and time step, using only labeled cases, and report system AUROC as the average over findings and steps.

Results. Both parameterizations of EDA-L2D clearly outperform Naive L2D in both cohorts (Table 2 and Table 3), raising the average system AUROC from 0.792 to 0.837 and 0.836 in the general cohort and from 0.774 to 0.807 and 0.806 in the emergency-department cohort. The two parameterizations perform alike, and the gains are largest on findings for which the classifier alone is weak, such as Enlarged Cardiomediastinum and Fracture; Figure 8 in Appendix § 12.§ 2 shows the system AUROC of these findings along the week. Figure 3 shows that EDA-L2D adapts its deferral to the expert’s performance: its coverage falls when the expert is accurate and rises as the expert tires, whereas Naive L2D defers at an almost constant rate. As a result, EDA-L2D stays at or above Naive L2D throughout the week.

Figure 3. CheXpert with one model per weekday, for the general cohort (left) and the emergency-department cohort (right). Lines show the mean over 5 seeds and shaded bands ±\pm one standard error.

MethodEnl. CardiomediastinumConsolidationFractureAverage (14 findings)
Naive L2D0.680 ± 0.0040.764 ± 0.0050.736 ± 0.0080.792 ± 0.002
EDA-L2D, Per-step param.0.767 ± 0.0040.820 ± 0.0050.809 ± 0.0090.837 ± 0.002
EDA-L2D, RNN param.0.759 ± 0.0050.819 ± 0.0010.806 ± 0.0060.836 ± 0.001

Table 2. System AUROC on CheXpert, general cohort, one model per weekday, for three findings and averaged over all 14 findings; each value is averaged over the steps of the week, with mean ± standard error over 5 seeds. Bold: the best L2D method and any L2D method within one standard error of it.

MethodEnl. CardiomediastinumConsolidationFractureAverage (14 findings)
Naive L2D0.678 ± 0.0040.744 ± 0.0040.729 ± 0.0070.774 ± 0.001
EDA-L2D, Per-step param.0.727 ± 0.0070.797 ± 0.0030.769 ± 0.0120.807 ± 0.001
EDA-L2D, RNN param.0.729 ± 0.0050.790 ± 0.0060.769 ± 0.0090.806 ± 0.001
MethodLinear declineWarm-up then fatiguePower-law practice
Naive L2D90.70±0.1790.70 \pm0.1791.14±0.1891.14 \pm0.1893.03±0.0693.03 \pm0.06
EDA-L2D, Per-step param.91.62±0.2191.62 \pm0.2192.37±0.2492.37 \pm0.2493.43±0.1593.43 \pm0.15
EDA-L2D, RNN param.92.05±0.1192.05 \pm0.1192.48±0.1992.48 \pm0.1993.90±0.2493.90 \pm0.24

Table 3. System AUROC on CheXpert, emergency-department cohort, one model per weekday; same layout as Table 2.

Figure 8. System AUROC along the week for three findings, CheXpert with one model per weekday, for the general cohort (left) and the emergency-department cohort (right). Lines show the mean over 5 seeds and shaded bands ± one standard error.

Hate Speech with Synthetic Expert Drift

Task and Data. We study detection of abusive content on Twitter posts using the Davidson et al. [8] corpus of 24,783 English tweets annotated into three mutually exclusive classes: hate speech, offensive but not hate, and neither. To obtain a stream on which expert quality can change over time, we group the tweets, in the order of the released file, into consecutive blocks of T=10T = 10 tweets and treat each block as one sequence.

Synthetic experts. The corpus records no annotator’s accuracy over a controlled sequence, so we simulate the expert. At step t∈{1,…,T}t \in\{1, \ldots, T\}, the expert answers a tweet correctly with probability ptp_t, independently of the tweet, and a miss earns no credit. With u=(t−1)/(T−1)∈[0,1]u = (t - 1)/(T - 1) \in[0, 1], we study three curves, each motivated by a mechanism reported for human annotators. The linear decline, pt=pstart−(pstart−pend)up_t = p_{\mathrm{start}} - (p_{\mathrm{start}} - p_{\mathrm{end}})u, is the simplest form of the drift observed in abusive-language annotation, where judgments shift between labelling sessions and depend on the conditions under which they are collected [1, 15]. The warm-up then fatigue curve,

pt={w0+(wpeak−w0)(u/u^)2,u≤u^,wbase+wpeak−wbase1+exp⁡{K(u−uˉ)},u>u^,p_t = \begin{cases} w_0 + (w_{\mathrm{peak}} - w_0)(u/\hat{u})^2, & u \le\hat{u}, \\ w_{\mathrm{base}} + \dfrac{w_{\mathrm{peak}} - w_{\mathrm{base}}}{1 + \exp\{K(u - \bar{u})\}}, & u > \hat{u}, \end{cases}

follows the two-phase performance model of Zhang et al. [36], in which performance first improves as a person settles into the task and then declines as fatigue accumulates. The power-law practice curve, pt=p∞−(p∞−pstart)t−cp_t = p_{\infty} - (p_{\infty} - p_{\mathrm{start}})t^{-c}, follows the power law of practice [25] and drifts in the opposite direction to the linear decline. The curves are stylised test cases rather than fits to any annotator; each crosses the accuracy of the classifier alone at least once, so the better choice between classifier and expert changes over the sequence. Appendix § 9.§ 3 gives their parameters.

Backbone and training. All methods use the same text classifier: a convolutional encoder over 100-dimensional GloVe embeddings, with 300 filters of widths 3, 4 and 5, max pooling and dropout 0.5. EDA-L2D, RNN param. adds a recurrent layer on top of the encoder features for the deferral decision. All methods are trained with the L2D loss using Adam at a constant learning rate of 10−310^{-3}, without a learning-rate schedule, and batches of 64 tweets (64 sequences for EDA-L2D, RNN param.). For each method we keep the epoch

with the highest validation system accuracy and report the mean over five seeds. Further details are given in Appendix § 10.§ 5.

Results. Table 4 reports the system accuracy averaged over the ten time steps, and Figure 4 shows system accuracy and coverage along the steps. Both parameterizations of EDA-L2D outperform Naive L2D on all three curves. EDA-L2D, RNN param. achieves the highest system accuracy, improving on Naive L2D by 1.35 points on the linear decline, 1.33 points on the warm-up-then-fatigue curve and 0.88 points on the power-law practice curve. EDA-L2D, Per-step param. recovers most of this gain (+0.92, +1.22 and +0.40 points).

Figure 4. System accuracy (top) and coverage (bottom) along the 10 time steps on hate speech, for the three synthetic experts. Lines show the mean over 5 seeds and shaded bands ± one standard error.

Conclusion and Discussion

We propose Expert-Drift-Adapted Learning to Defer (EDA-L2D), a framework that models temporal drift in human expert performance, a common phenomenon caused by factors such as fatigue and cognitive biases. Most L2D methods treat the expert as a fixed oracle, yet human reliability routinely changes over the course of work. EDA-L2D closes this gap by learning from the expert’s observed outcomes, so that its deferral decisions track the expert’s current reliability without assuming a particular form of drift. It incorporates time and the expert’s history of correctness into the L2D model, learning time-varying representations of the expert and adapting its deferral decisions accordingly. Across image classification, chest X-ray diagnosis and hate speech detection, EDA-L2D consistently improves human-AI team performance over standard L2D. These gains largely hold under realistic conditions, where the expert’s correctness is observed only for deferred cases and may be delayed, noisy or partial, and they persist to a considerable extent when the expert departs from the behavior seen in training.

Our work complements several lines of L2D research discussed in § 4, including sequential deferral [13], deferral to unseen or changing experts [29, 21], and fatigue-aware deferral [36]. Building on these connections, our framework extends naturally to the multi-expert setting of Verma et al. [34], with a separate temporal state per expert so that each case can be routed to the expert who is currently most reliable, and to populations of experts, where meta-learning drift patterns, as Tailor et al. [29] do for expert characteristics, would let a new expert’s drift be inferred from few observations. Since the expert’s correctness is observed only on deferred cases, a system that rarely defers cannot notice when the expert recovers; deferring occasionally to gather feedback, as the exploration in the bandit setting of Montreuil et al. [21], could address this.

Richer expert models would also help with effects that our current model does not capture. In our experiments, the expert’s accuracy depends only on the time step, whereas fatigue effects may depend on case difficulty [18]. Real experts are also subject to content-dependent biases, such as anchoring [30] and the gambler’s fallacy in sequential decisions [16], which a recurrent model of past correctness alone may not fully capture. Longitudinal data from real working sessions would allow these effects to be modeled and evaluated directly.

More broadly, our results suggest that human-AI collaboration should account for the dynamic nature of human performance: experts are not fixed oracles, and their reliability changes over the course of their work. We hope this work encourages learning-to-defer methods that adapt to these changes, so that human-AI teams remain effective as the people in them change.

Broader Impact Statement

EDA-L2D aims to make human-AI teams more reliable by accounting for changes in expert performance, such as fatigue, which could reduce errors in settings like clinical reading and content moderation. It also carries risks. Records of an expert’s correctness over time could be repurposed to monitor individual workers, so their use should be restricted to the deferral decision. When the model judges the expert to be less reliable, it takes over more cases, which can reduce human oversight if the model itself errs; experts should remain able to override it. Our experiments use simulated experts and have not been validated with real clinicians or moderators, which is necessary before any high-stakes deployment. Finally, the hate speech dataset contains known annotation biases that a deferral system may inherit.

Theoretical Results

Bayes Rule under Expert Drift

We first characterize the Bayes-optimal decision rule when the expert’s reliability changes over time. Let X\mathcal{X} and Y\mathcal{Y} denote the input and label spaces, and assume that the expert also produces a prediction mt∈Ym_t \in\mathcal{Y} at each time step t∈Tt \in\mathcal{T}. The label distribution P(y∣x)\mathbb{P}(y \mid x) is assumed to be stationary, whereas the expert’s behavior may depend on time.

A classifier h:X→Yh : \mathcal{X} \to\mathcal{Y} is paired with a time-dependent rejector r:X×T→{0,1}r : \mathcal{X} \times\mathcal{T} \to\{0,1\}. The resulting system decision is

δ(x,t)={h(x),r(x,t)=0,⊥,r(x,t)=1,\delta(x,t) = \begin{cases} h(x), & r(x,t)=0,\\ \perp, & r(x,t)=1, \end{cases}

where ⊥\perp denotes deferral to the expert.

Learning-to-defer risk. For an example with label yy and expert prediction mtm_t, the system incurs the cost

cL2D(δ(x,t),y,mt)={I[y^≠y],δ(x,t)=y^∈Y,I[mt≠y],δ(x,t)=⊥.(7)c_{\mathrm{L2D}}(\delta(x,t),y,m_t) = \begin{cases} \mathbb{I}[\hat{y} \ne y], & \delta(x,t)=\hat{y}\in\mathcal{Y},\\ \mathbb{I}[m_t \ne y], & \delta(x,t)=\perp. \end{cases} \tag*{(7)}

Thus, a model prediction is penalized when the classifier is incorrect, whereas a deferred prediction is penalized when the expert is incorrect.

For a fixed input xx and time step tt, define

ηy(x):=P(Y=y∣X=x)\eta_y(x) := \mathbb{P}(Y=y \mid X=x)

and

αt(x):=P(Mt=Y∣X=x,t),\alpha_t(x) := \mathbb{P}(M_t=Y \mid X=x,t),

where αt(x)\alpha_t(x) is the expert’s conditional probability of being correct at time tt. Importantly, this definition does not require the expert prediction and the true label to be conditionally independent.

If the system predicts a class y^\hat{y}, its conditional risk is

R(x,t,y^)=P(Y≠y^∣X=x)=1−ηy^(x).(8)\mathcal{R}(x,t,\hat{y}) = \mathbb{P}(Y \ne\hat{y} \mid X=x) = 1-\eta_{\hat{y}}(x). \tag*{(8)}

The best model prediction is therefore

h∗(x)=arg⁡max⁡y∈Y ηy(x),(9)h^*(x) = \underset{y\in\mathcal{Y}}{\arg\max}\,\eta_y(x), \tag*{(9)}

with conditional risk

1−max⁡y∈Yηy(x).1-\max_{y\in\mathcal{Y}}\eta_y(x).

If the system instead defers, its conditional risk is the expert’s probability of error:

R(x,t,⊥)=P(Mt≠Y∣X=x,t)=1−αt(x).(10)\mathcal{R}(x,t,\perp) = \mathbb{P}(M_t \ne Y \mid X=x,t) = 1-\alpha_t(x). \tag*{(10)}

Comparing these two risks gives the Bayes-optimal rejector:

r∗(x,t)=I[αt(x)≥max⁡y∈Yηy(x)].(11)r^*(x,t) = \mathbb{I}\left[\alpha_t(x) \ge\max_{y\in\mathcal{Y}}\eta_y(x)\right]. \tag*{(11)}

Under this tie-breaking convention, the system defers exactly when the expert’s probability of being correct at time tt is at least as large as the classifier’s probability of correctly predicting its most likely class.

This rule extends the stationary result of [22]. In the stationary setting, the comparison is made against a fixed expert reliability. Under expert drift, that quantity becomes αt(x)\alpha_t(x) and may change over time.

Consistency of the Per-Step Softmax Surrogate

We next show that the per-step softmax surrogate in Equation (5) recovers the Bayes rule above.

Let

py(x,t)=exp⁡{gy(x)}exp⁡{g⊥t(x)}+∑y′∈Yexp⁡{gy′(x)},y∈Y,p_y(x,t) = \frac{\exp\{g_y(x)\}}{\exp\{g_\perp^t(x)\} + \sum_{y'\in\mathcal{Y}} \exp\{g_{y'}(x)\}}, \qquad y\in\mathcal{Y},

and

p⊥(x,t)=exp⁡{g⊥t(x)}exp⁡{g⊥t(x)}+∑y′∈Yexp⁡{gy′(x)}.p_\perp(x,t) = \frac{\exp\{g_\perp^t(x)\}}{\exp\{g_\perp^t(x)\} + \sum_{y'\in\mathcal{Y}} \exp\{g_{y'}(x)\}}.

The per-step surrogate can then be written as

ϕSMt=−log⁡py(x,t)−I[mt=y]log⁡p⊥(x,t).(12)\phi_{\mathrm{SM}}^t = -\log p_y(x,t) - \mathbb{I}[m_t=y]\log p_\perp(x,t). \tag*{(12)}

Conditioning on (x,t)(x,t) and taking the expectation over the label and expert prediction gives

Ct(p;x)=−∑y∈Yηy(x)log⁡py(x,t)−αt(x)log⁡p⊥(x,t).(13)C_t(p;x) = -\sum_{y\in\mathcal{Y}}\eta_y(x)\log p_y(x,t) - \alpha_t(x)\log p_\perp(x,t). \tag*{(13)}

This is a weighted cross-entropy over the augmented output space Y∪{⊥}\mathcal{Y}\cup\{\perp\}. Its minimizing probabilities are

py∗(x,t)=ηy(x)1+αt(x),p⊥∗(x,t)=αt(x)1+αt(x).(14)p_y^*(x,t) = \frac{\eta_y(x)}{1+\alpha_t(x)}, \qquad p_\perp^*(x,t) = \frac{\alpha_t(x)}{1+\alpha_t(x)}. \tag*{(14)}

The common denominator does not affect the ordering of the outputs. Consequently,

arg⁡max⁡y∈Y py∗(x,t)=arg⁡max⁡y∈Y ηy(x),\underset{y\in\mathcal{Y}}{\arg\max}\ p_y^*(x,t) = \underset{y\in\mathcal{Y}}{\arg\max}\ \eta_y(x),

and

p⊥∗(x,t)≥max⁡y∈Y py∗(x,t)⟺αt(x)≥max⁡y∈Y ηy(x).p_\perp^*(x,t) \geq\underset{y\in\mathcal{Y}}{\max}\ p_y^*(x,t) \Longleftrightarrow\alpha_t(x) \geq\underset{y\in\mathcal{Y}}{\max}\ \eta_y(x).

The classifier and rejector induced by the softmax surrogate therefore coincide with the Bayes-optimal rules in Equations (9) and (11). Hence, the per-step softmax surrogate is Bayes-consistent for the drifting-expert L2D objective.

Consistency of the RNN-Based Softmax Surrogate

The RNN-based model additionally conditions its deferral decision on information accumulated from previous time steps. Let HtH_t denote the history available to the model before making the decision at time tt, including previously observed expert-correctness feedback. Define

αt(x,Ht):=P(Mt=Yt∣Xt=x,Ht).(15)\alpha_t(x,H_t) := \mathbb{P}(M_t=Y_t\mid X_t=x,H_t). \tag*{(15)}

Under the assumption that the label distribution remains stationary and that YtY_t is conditionally independent of HtH_t given XtX_t, the classifier confidence remains ηy(x):=P(Yt=y∣Xt=x)\eta_y(x) := \mathbb{P}(Y_t=y\mid X_t=x). The Bayes-optimal history-aware rejector is therefore

r∗(x,Ht)=I[αt(x,Ht)≥max⁡y∈Y ηy(x)].(16)r^*(x,H_t) = \mathbb{I}\left[\alpha_t(x,H_t) \geq\underset{y\in\mathcal{Y}}{\max}\ \eta_y(x)\right]. \tag*{(16)}

The sequence-level surrogate is the sum of the per-step losses:

ϕSMT(g)=∑t∈TϕSMt(g;Xt,Yt,Mt,Ht).(17)\phi_{\mathrm{SM}}^{\mathcal{T}}(g) = \sum_{t\in\mathcal{T}}\phi_{\mathrm{SM}}^t(g;X_t,Y_t,M_t,H_t). \tag*{(17)}

By linearity of expectation, its population risk decomposes as

RRNN(g)=∑t∈TEXt,Ht[Ct(g;Xt,Ht)].(18)\mathcal{R}_{\mathrm{RNN}}(g) = \sum_{t\in\mathcal{T}}\mathbb{E}_{X_t,H_t}\left[C_t(g;X_t,H_t)\right]. \tag*{(18)}

where

Ct(g;x,Ht)=E[ϕSMt∣Xt=x,Ht].C_t(g; x, H_t) = \mathbb{E}\left[\phi^{t}_{\mathrm{SM}} \mid X_t = x, H_t\right].

For each (x,Ht)(x, H_t), this conditional risk has the same weighted cross-entropy form as Equation (13), with αt(x)\alpha_t(x) replaced by αt(x,Ht)\alpha_t(x, H_t). Its optimal softmax probabilities are therefore

py∗(x,Ht)=ηy(x)1+αt(x,Ht),p⊥∗(x,Ht)=αt(x,Ht)1+αt(x,Ht).(19)p_y^*(x, H_t) = \frac{\eta_y(x)}{1+\alpha_t(x, H_t)}, \qquad p_\perp^*(x, H_t) = \frac{\alpha_t(x, H_t)}{1+\alpha_t(x, H_t)}. \tag*{(19)}

Assuming that the recurrent model class is rich enough to represent these conditional probabilities, minimizing the sequence-level surrogate recovers the history-aware Bayes rule in Equation (16). The RNN-based softmax surrogate is therefore Bayes-consistent under this realizability assumption.

Temporal Complementarity under Expert Drift

The Bayes rule in Equation (11) shows that the optimal deferral decision depends on a comparison between the classifier’s confidence and the expert’s current reliability. In particular, the system should defer at (x,t)(x,t) when

αt(x)≥max⁡y∈Yηy(x),\alpha_t(x) \ge\max_{y \in\mathcal{Y}} \eta_y(x),

where

ηy(x)=P(Y=y∣X=x)\eta_y(x) = \mathbb{P}(Y = y \mid X = x)

and

αt(x)=P(Mt=Y∣X=x,t).\alpha_t(x) = \mathbb{P}(M_t = Y \mid X = x,t).

Under our assumption that the input and label distribution remains stationary, the classifier confidence for a given input does not change with time. The expert reliability, however, may drift. Consequently, the relative ordering of the two quantities can change over a sequence:

αt1(x)<max⁡y∈Yηy(x)andαt2(x)>max⁡y∈Yηy(x)(20)\alpha_{t_1}(x) < \max_{y \in\mathcal{Y}} \eta_y(x) \qquad\text{and} \qquad\alpha_{t_2}(x) > \max_{y \in\mathcal{Y}} \eta_y(x) \tag*{(20)}

for two time steps t1t_1 and t2t_2. We refer to this change in ordering as a temporal crossing.

A temporal crossing changes the Bayes-optimal routing decision for the same input: the classifier is preferred at t1t_1, whereas deferral to the expert is preferred at t2t_2. A model that treats expert reliability as stationary cannot represent this change using the expert’s temporal state alone. In contrast, a time-aware deferral model can adjust its decision as the expert moves from one side of the classifier-confidence threshold to the other.

The benefit of modeling expert drift therefore arises not merely because the expert’s accuracy changes, but because these changes can alter which component is more reliable. If the expert remains uniformly better or uniformly worse than the classifier throughout the sequence, temporal adaptation provides no additional routing opportunity. When temporal crossings occur, however, tracking the expert’s current reliability allows the system to defer during periods when the expert is more reliable and rely more heavily on the classifier when the expert’s reliability declines.

EDA-L2D is designed to exploit this temporal structure. Its recurrent deferral component uses the history of the sequence and the available expert-correctness feedback to estimate the expert’s current state and adapt the deferral decision accordingly. The model need not observe αt(x)\alpha_t(x) directly; rather, it learns temporal patterns that help it track when the expert is likely to cross the classifier’s reliability threshold.

Expert Simulation

Training an L2D model requires the expert’s answer for every training input. Existing datasets do not provide such data: CIFAR-10H collects human annotations only on the CIFAR-10 test set, and CheXpert and the hate speech dataset contain no answers of individual experts over time. We therefore simulate the expert, following the setting of § 3.1. The task P(y∣x)\mathbb{P}(y \mid x) is stationary, and only the expert’s predictions Pt(mt∣x)\mathbb{P}_{t}(m_{t} \mid x) change with tt. We capture this change through the expert’s accuracy over time and use a surrogate expert whose probability of being correct depends only on the time step:

Pt(mt=y∣x)=pt,\mathbb{P}_{t}(m_{t}=y \mid x)=p_{t},

where ptp_{t} is a drift curve. In step tt, the surrogate expert returns the true label with probability ptp_{t} (a Bernoulli draw) and the incorrect label otherwise. This yields expert answers for the training, validation and test sets, and a known curve that we can modify at test time for the robustness tests (Appendix § 13).

Drifting Annotators in CIFAR-10H

On CIFAR-10, ptp_{t} is estimated from real annotators in CIFAR-10H ([27]). CIFAR-10H provides human labels for all 10,000 images of the CIFAR-10 test set, collected from 2,571 crowd workers, with about 51 labels per image. Each annotator labels 200 images in sequence, interleaved with 10 attention checks. The raw data record each annotator’s answers in presentation order, together with response times. This makes it possible to follow an individual annotator’s accuracy over the course of a session, which is the drift we want to simulate. Human accuracy is high on average (94.9%), but it varies across annotators and, for some annotators, changes considerably within a session.

Annotator selection. We examine whether individual annotators in CIFAR-10H exhibit systematic changes in reliability over the course of an annotation session. For each of the 2,571 annotators, we consider the 200 non-attention-check trials in presentation order and divide them into two consecutive halves of 100 trials. We compute the change in mean accuracy from the first to the second half and identify 18 annotators whose accuracy decreases by more than 10 percentage points. Figure 5 shows all 18 annotators, grouped by whether they pass the 75% attention-check threshold used by Peterson et al. [27]; the three annotators we use as experts are framed. Failing the attention checks is itself a form of drift that we wish to model, so we do not exclude these annotators a priori. However, annotators with very low attention scores remain close to chance accuracy (0.1 for ten classes) for most of their session (e.g. #751, #1730, #2540). This pattern is more consistent with disengaged or random responding than with a change in reliability, and an expert who is almost always wrong offers little to learn for deferral. We therefore select three annotators whose curves combine long periods of high accuracy with distinct, structured forms of drift:

Figure 5. The 18 CIFAR-10H annotators whose accuracy drops by more than 10 percentage points from the first to the second half of their 200 trials. (a) Annotators below the 75% attention-check threshold; (b) annotators above it. Titles give the annotator id and the number of attention checks passed. Curves are running accuracies over 20 trials. The other annotators stay accurate on the same images, so the drops are not caused by harder images.

  • #454 (6/10 attention checks) is nearly perfect for the first ~120 trials and then deteriorates sharply, which is characteristic of an annotator who is attentive early in the session and disengages later.

  • #1004 (9/10) shows a transient drop to about 0.15 around trials 100–120, followed by a recovery to its initial level, consistent with a temporary lapse.

  • #54 (8/10) improves over the first trials, stays near-perfect for most of the session, and declines in the second half before partially recovering.

The latter two pass the attention checks and change more gradually, and so represent the more realistic case of a diligent annotator whose reliability varies over the session; #454 represents a more extreme, abrupt change.

Temporal Variation in Radiologist Performance

We further examine evidence of temporal variation in expert performance in a clinical setting. Vosshenrich et al. [35] retrospectively analyzed 117,402 radiology reports written by residents between September 2017 and March 2020. For each study, they compared the resident’s preliminary report with the corresponding staff-reviewed final report using the Jaccard similarity coefficient.

Figure 4 of Vosshenrich et al. [35] reports mean report similarity over five two-hour intervals for each weekday for the general resident cohort, while Figure 3 reports the corresponding trajectory for the emergency-department cohort at a finer hourly resolution, with ten measurements per weekday. Both cohorts exhibit systematic temporal variation within individual workdays and across the workweek. Report similarity generally decreases over the course of a workday, with temporary midday increases suggesting partial recovery following lunch breaks; this recovery becomes less pronounced toward the end of the workweek.

Because report similarity is not a direct measure of diagnostic accuracy, we use it only to characterize the temporal pattern of expert drift. We digitize the mean report-similarity trajectories from the two figures (25 points for the general cohort, 50 for the emergency-department cohort), double each value’s deviation from the trajectory mean and add small uniform jitter, which makes the within- and across-day fluctuations more pronounced while keeping their structure.

We then rescale each trajectory linearly to the accuracy range [0.695, 0.885],

rt=0.695+(0.885−0.695)st−smin⁡smax⁡−smin⁡.r_t = 0.695 + (0.885 - 0.695) \frac{s_t - s_{\min}}{s_{\max} - s_{\min}}.

This range is the span of the balanced accuracy, (TPR+1−FPR)/2(\mathrm{TPR} + 1 - \mathrm{FPR})/2, of the three individual radiologists on the five competition pathologies in Figure 3 of Irvin et al. [12]. It runs from 0.695 (Consolidation; FPR = 0.09, TPR = 0.48) to 0.885 (Pleural Effusion; FPR = 0.05, TPR = 0.82). We use individual radiologists rather than their majority vote because in deployment each case is read by a single expert, and balanced accuracy because it accounts for both detecting and ruling out a finding.

Within this range the drift is too small to change the deferral decision over the week. We therefore amplify it around the weekly mean rˉ\bar{r}, separately across and within days:

at=clip⁡(rˉ+2(rˉd(t)−rˉ)+KD(rt−rˉd(t)), 0.5, 1).a_t = \operatorname{clip}\left(\bar{r} + 2\left(\bar{r}_{d(t)} - \bar{r}\right) + K_D\left(r_t - \bar{r}_{d(t)}\right),\, 0.5,\, 1\right).

where rˉd\bar{r}_d is the mean over weekday dd. We use KD=6K_D = 6 for the general cohort and KD=3K_D = 3 for the emergency-department cohort, whose within-day deviations are already larger; with KD=6K_D = 6, most of its days would be clipped into a two-level curve. The resulting trajectories have mean 0.82 and range [0.5, 1] and mean 0.78 and range [0.53,1][0.53, 1]. They exceed the accuracies observed for the CheXpert radiologists and should be read as a stress test of how well each method follows a strongly drifting expert, not as an estimate of real radiologist performance. In short, the fatigue study determines the temporal structure of the drift, and the CheXpert radiologists determine its accuracy scale before amplification (Figure 6a and Figure 6b).

Synthetic Expert Curves for Hate Speech

The expert drift used for CheXpert and CIFAR-10 above is grounded in evidence about real annotators. For hate speech, we have no comparable record of a single expert whose accuracy is measured over a controlled sequence. We therefore complement that evidence with three simulated accuracy curves, each motivated by a mechanism reported in the literature:

  • Linear decline. Studies of abusive-language annotation show that the judgments of the same annotator can shift between labeling sessions ([1]) and that annotations are sensitive to the conditions under which they are collected ([15]). A steady decline provides the simplest representation of this type of drift.

  • Warm-up followed by fatigue. This curve is motivated by the workload-dependent performance model of Zhang et al. [36], in which performance initially improves as a person settles into the task and subsequently declines as fatigue accumulates. It is also consistent with the vigilance decrement observed in prolonged monitoring tasks.

  • Practice curve. This curve is motivated by the power law of practice ([25]), under which performance improves with experience but with diminishing gains. It allows us to evaluate drift in the direction opposite to the linear-decline setting.

These curves are stylized test cases and are not fitted to any individual annotator. Their exact forms and parameters are provided below. Formally, at step t∈{1,…,T}t \in\{1,\ldots,T\} the expert answers the tweet correctly with probability ptp_t, drawn independently for every tweet, and a miss earns no credit, so the accuracy of the expert at step tt is ptp_t. We write u=(t−1)/(T−1)∈[0,1]u=(t-1)/(T-1) \in[0,1] with T=10T=10, so that u=0u=0 at the first step and u=1u=1 at the last.

Linear Decline. Accuracy falls at a constant rate from pstartp_{\mathrm{start}} at the first step to pendp_{\mathrm{end}} at the last step,

ptlin=pstart−(pstart−pend)u,(21)p_t^{\mathrm{lin}} = p_{\mathrm{start}} - (p_{\mathrm{start}} - p_{\mathrm{end}})u, \tag*{(21)}

with pstart=1.0p_{\mathrm{start}}=1.0 and pend=0.55p_{\mathrm{end}}=0.55, i.e. ptlin=1−0.45up_t^{\mathrm{lin}}=1-0.45u.

Warm-up then Fatigue. Following the two-phase performance model of Zhang et al. [36], accuracy first rises quadratically from an initial level w0w_0 to a peak wpeakw_{\mathrm{peak}} during the warm-up phase, which ends at u^\hat{u}, and then declines along a logistic curve towards a floor wbasew_{\mathrm{base}},

ptfat={w0+(wpeak−w0)(u/u^)2,u≤u^,wbase+wpeak−wbase1+exp⁡{K(u−uˉ)},u>u^.(22)p_t^{\mathrm{fat}} = \begin{cases} w_0 + (w_{\mathrm{peak}} - w_0) (u/\hat{u})^2, & u \le\hat{u}, \\ w_{\mathrm{base}} + \dfrac{w_{\mathrm{peak}} - w_{\mathrm{base}}}{1 + \exp\{K(u - \bar{u})\}}, & u > \hat{u}. \end{cases} \tag*{(22)}

Here uˉ\bar{u} is the point at which the decline is halfway complete and KK controls how sharp it is. We use w0=0.85w_0 = 0.85, wpeak=0.99w_{\mathrm{peak}} = 0.99, wbase=0.60w_{\mathrm{base}} = 0.60, u^=0.15\hat{u} = 0.15, uˉ=0.55\bar{u} = 0.55, and K=12K = 12. The two phases join with a gap below 0.005 at u=u^u = \hat{u}.

Power-law Practice. Accuracy improves with experience with diminishing gains,

ptpow=p∞−(p∞−pstart)t−c,(23)p_t^{\mathrm{pow}} = p_{\infty} - (p_{\infty} - p_{\mathrm{start}})t^{-c}, \tag*{(23)}

with pstart=0.65p_{\mathrm{start}} = 0.65, p∞=1.0p_{\infty} = 1.0, and c=0.8c = 0.8. This curve is indexed by the step tt rather than by uu; at the first step, t−c=1t^{-c} = 1 and p1pow=pstartp_1^{\mathrm{pow}} = p_{\mathrm{start}}.

All parameters are our own choices for a ten-step window and are not fitted to data. Each curve crosses the accuracy of the classifier alone, which is about 0.88, at least once.

Implementation Details

All three variants are trained with our surrogate loss (Equation (3)), with K+1K + 1 outputs where output K+1K + 1 means defer; the RNN variant uses its sequential form (Equation (6)). We use the same three datasets as Mozannar & Sontag [22] and follow their setup. They introduce a weight α\alpha on the classification term for inputs on which the expert is correct; we set α=1\alpha= 1, which gives the loss exactly as written. All variants face the same simulated expert (Appendix § 9): the same drift curve and, within a seed, the same expert answers. The variants differ only in how time enters the model.

The Variants and Baseline

Naive L2D. A single time-agnostic model is trained on the items of all steps pooled together. It is the standard L2D model and does not receive the time step or any information about the expert’s earlier responses.

EDA-L2D, Per-step Param. The model has the same architecture and loss as Naive L2D, but one defer output per time step instead of a single one. The class outputs and the backbone are shared across steps, and at step tt the model uses the class outputs together with the defer output of step tt. Each defer head can thus follow its own step of the drift, while the classifier is learned from all the data. On hate speech we therefore use separate models.

EDA-L2D, RNN Param. The backbone of Naive L2D encodes each item, and its features give the KK class outputs as in Naive L2D. A recurrent cell (GRU or LSTM) runs over the sequence; its input at step tt is the item feature and the expert’s correctness at step t−1t - 1, and its state gives the defer output. The class outputs are thus the same as in Naive L2D, and the decision to defer depends on the expert’s history.

Common Training Setup

For each dataset, all variants share the backbone, the classifier head, the optimiser and the weight decay, which we take from the original implementation of Mozannar & Sontag [22]. Naive L2D and EDA-L2D, Per-step param. have the same classifier as their L2D model and differ only in the defer output. We therefore start from their settings but still select the learning rate on the validation set for every dataset. EDA-L2D, RNN param. adds a recurrent module that has no counterpart in their setup, so we select both its learning rate and its architecture on the validation set: the recurrent cell, the hidden size and the number of layers.

On every dataset, each batch consists of whole sequences in time order, so all variants see the same number of inputs per batch, and EDA-L2D, RNN param. can run over each sequence within a batch. Naive L2D and EDA-L2D, Per-step param. treat the inputs of a batch independently.

CIFAR-10 Setup

Temporal Rescaling

For each selected annotator (Appendix B.1), the raw CIFAR-10H response sequence consists of the same 200 non-attention-check trials used for the selection, each recording whether the annotator labeled the image correctly. We first compute the running accuracy of the annotator with a sliding window of 20 trials, and then rescale this curve to T=50T = 50 steps by averaging over consecutive blocks of four trials. The resulting value ptp_t is the annotator’s accuracy at step tt.

We do this for three reasons. First, per-trial correctness is binary and noisy, so block averaging yields a smoother estimate of ptp_t that preserves the drift pattern while removing trial-level fluctuations. Second, the 50,000 training images form only 250 sequences at length 200 but 1,000 sequences at length 50, which gives the rejector more independent sequences to learn from. Third, backpropagation through time for the recurrent rejector becomes four times longer at T=200T = 200, which substantially increases training cost across our searches and seeds. The rescaled curve thus fixes the sequence length T=50T = 50 and the expert’s accuracy ptp_t at every step, which we use next to construct the training and test data.

Data Construction

To match the T=50T = 50 steps of the rescaled curve, we cut the 50,000 training images into consecutive sequences of 50 images and split them at the sequence level into training and validation sets (90%/10%), so that each sequence belongs entirely to one set. The image at position tt of a sequence is paired with the simulated expert at step tt. During training, the expert’s answers are redrawn every time a sequence is seen, so over 200 epochs each training sequence is paired with many different realisations of the expert. An incorrect expert answers with a uniformly random class, which can coincide with the label. The official 10,000-image CIFAR-10 test set is never used during training and is reserved for evaluation.

At test time, a single test sequence would give a noisy estimate: the system accuracy at a step depends on which images appear and on whether the expert happens to be correct. We therefore estimate it by Monte Carlo replay. Each replay sequence draws T=50T = 50 images uniformly at random, with replacement, from the test set, and pairs the image at step tt with an expert who is correct with probability ptp_t. We use 20,000 replay sequences for each of the 5 seeds, i.e. 100,000 per annotator, and all variants are evaluated on the same replay sequences. The Monte Carlo standard error of the accuracy at a single step is then at most 0.25/N\sqrt{0.25/N}: about 0.35 percentage points for one seed (N=20,000N = 20{,}000) and 0.16 points when pooled over seeds (N=100,000N = 100{,}000). This is small compared with the differences between variants, so the differences in our results reflect the trained models rather than evaluation noise.

Hyperparameter Tuning

For Naive L2D we tune the learning rate over {0.05,0.1,0.2}\{0.05, 0.1, 0.2\}, training 5 seeds for 200 epochs at each value and selecting by the mean system accuracy on the held-out validation sequences (Appendix C.3.2). lr = 0.1 is best for every annotator, and we use the same value for EDA-L2D, Per-step param., which differs from Naive L2D only in its defer outputs.

For EDA-L2D, RNN param., we search the learning rate jointly with the recurrent architecture: the cell (GRU or LSTM), the hidden size over {256,512,1024}\{256, 512, 1024\}, the number of layers over {1,2}\{1, 2\} and the learning rate over {0.01,0.05,0.1}\{0.01, 0.05, 0.1\}. Every configuration is trained with 3 seeds for 200 epochs and evaluated on the validation images with 20,000 Monte Carlo sequences under realistic routing, as at test time (Table 5). The learning rate matters most, whereas the architectures differ by less than one point at lr = 0.1. For each annotator we select the configuration with the highest validation system accuracy averaged over the seeds: a GRU with 256 units and 2 layers for annotator 454, an LSTM with 1024 units and 1 layer for annotator 1004, and an LSTM with 1024 units and 2 layers for annotator 54, all with lr = 0.1. The selected configuration is then trained with 5 seeds for the reported results.

CellddLLAnnotator 454: lr 0.10.050.01Annotator 1004: lr 0.10.050.01Annotator 54: lr 0.10.050.01
GRU256193.94±0.1193.47±0.0689.67±0.1793.69±0.1293.50±0.1490.02±0.5394.87±0.1394.75±0.0892.53±0.23
256294.19±0.2793.60±0.1589.27±0.3193.92±0.2893.53±0.3390.38±0.3495.00±0.3294.63±0.2092.52±0.22
512193.94±0.0592.92±0.3389.58±0.1593.85±0.0693.68±0.0689.55±0.3094.90±0.0594.59±0.1492.28±0.30
512293.56±0.2892.91±0.3589.53±0.2093.82±0.0293.46±0.1790.18±0.1294.99±0.0793.92±0.2592.28±0.09
1024193.91±0.2192.61±0.1288.71±0.4293.77±0.0593.81±0.1190.92±0.1194.89±0.0594.63±0.0791.70±0.19
1024293.64±0.2292.75±0.5288.60±0.2793.92±0.0293.76±0.0788.85±0.4295.04±0.1894.34±0.0991.98±0.09
LSTM256194.02±0.1093.83±0.0891.12±0.2093.97±0.0893.60±0.1191.03±0.2295.06±0.0794.87±0.1093.00±0.10
256293.94±0.1793.52±0.1990.77±0.1593.82±0.1393.37±0.2190.78±0.0594.99±0.1894.93±0.1692.79±0.06
512193.87±0.1893.61±0.1590.76±0.1893.70±0.1993.53±0.0890.49±0.3395.03±0.1694.84±0.1392.72±0.20
512293.85±0.2393.50±0.2590.49±0.4093.55±0.1993.38±0.1690.69±0.1695.05±0.1594.85±0.1892.55±0.22
1024194.14±0.2093.78±0.2290.65±0.1294.03±0.0893.66±0.1190.76±0.0895.15±0.0595.13±0.0792.77±0.08
1024293.95±0.3093.88±0.3388.29±1.6793.92±0.1793.82±0.2390.75±0.1795.23±0.1695.10±0.1892.14±0.26

Table 5. Search over the recurrent architecture and learning rate of EDA-L2D, RNN param. on CIFAR-10. Validation system accuracy (%) estimated with 20,000 Monte Carlo sequences per run; mean ± standard error over 3 seeds. dd: hidden size; LL: number of layers. Bold: the configuration selected for each annotator.

CheXpert Setup

Data Construction

We use CheXpert-v1.0-small, the official downsampled-resolution release of CheXpert ([12]), and split its patients into 81% training, 9% validation and 10% test. Unmentioned findings count as negative, and uncertain labels are excluded from the loss and the metrics. Within each set, the images are shuffled and cut into sequences, and the image at position tt is paired with the expert at step tt. The expert follows the curve ata_t of Appendix B.2 over a five-day week, with 5 steps per day for the general cohort and 10 for the emergency-department cohort. At step tt, it answers each of the 14 findings correctly with probability ata_t and flips the label otherwise. Its answers are drawn once, so all variants see the same sequences and expert answers.

We consider two settings. In the whole-week setting, one model is trained on week-long sequences of 25 or 50 steps. In the per-day setting, a separate model is trained for each weekday on day-long sequences of 5 or 10 steps, with the expert following that day’s part of the curve; each of these models uses all training images, and at test time their results are averaged over the five weekdays. In both settings, all variants are fine-tuned from the same classifier, pretrained without deferral on whole-week sequences.

Hyperparameter Tuning

All variants use a DenseNet-121 initialised with ImageNet weights, with one class output and one defer output per finding, and are trained with Adam, learning rate 10−410^{-4} and weight decay 10−510^{-5}. The classifier is first pretrained without deferral for 3 epochs, and every variant is then fine-tuned from it for 5 epochs; we use the final model. A batch contains 4 week-long or 20 day-long sequences, so every variant sees the same number of images per batch.

EDA-L2D, Per-step param. has one defer head per step: 25 or 50 in the whole-week setting, 5 or 10 in the per-day setting. For EDA-L2D, RNN param., we search the recurrent cell over GRU and LSTM, the hidden size over {128, 256, 512} and the number of layers over {1, 2, 3}. Each configuration is trained with two seeds on the general cohort and scored by the validation system AUROC after the last fine-tuning epoch. We select a GRU with hidden size 128 and 3 layers and use it for both cohorts and both settings. Its feedback input contains one correctness indicator per finding, set to 0.5 before any outcome is observed.

Hate Speech Setup

Data Construction

We use the dataset of Davidson et al. [8], with 24,783 tweets labelled as hate speech, offensive or neither. The tweets, in file order, form sequences of T=10T = 10, and the complete sequences are split into 60% training, 10% validation and 30% test. The tweet at position tt of a sequence is paired with the expert at step tt, who is correct with probability ptp_t given by one of the three synthetic curves of Appendix § 9.§ 3: linear decline, warm-up then fatigue, and power-law practice. The expert’s answers are drawn once when the data are built, so all variants are trained and evaluated on the same sequences and the same expert answers. EDA-L2D, Per-step param. trains one model per step; model tt is trained on all training tweets, with the expert’s answers drawn from ptp_t.

Hyperparameter Tuning

All variants use a convolutional text encoder on 100-dimensional GloVe embeddings, with 300 filters of widths 3, 4 and 5 and dropout 0.5; the vocabulary consists of the 25,000 most frequent tokens of the training tweets. We train with Adam and batch size 64, and keep the epoch with the highest validation system accuracy. Naive L2D and EDA-L2D, Per-step param. are trained for 10 epochs on batches of tweets. We select their learning rate from {2×10−4,5×10−4,10−3,2×10−3}\{2 \times10^{-4}, 5 \times10^{-4}, 10^{-3}, 2 \times10^{-3}\} and use 10−310^{-3}. EDA-L2D, RNN param. is trained for 20 epochs on batches of sequences. For each expert, we search the recurrent cell over GRU and LSTM, the learning rate over {5×10−4,10−3,2×10−3}\{5 \times10^{-4}, 10^{-3}, 2 \times10^{-3}\}, the hidden size over {128,256,512}\{128, 256, 512\} and the number of layers over {1,2}\{1, 2\}. An LSTM with learning rate 10−310^{-3}, hidden size 512 and 2 layers performs well for all three experts, and we use it throughout. Its feedback input is the expert’s correctness at the previous step, set to 0 at the first step.

Additional Results

CIFAR-10

Table 6: System accuracy (%) on CIFAR-10 when replaying the real answers of the drifting CIFAR-10H annotators: each annotator’s 200 trials are split into 4 interleaved sequences of 50 steps, so that every trial keeps its original time step, and the accuracy is averaged over 5 seeds. In brackets: the gain over Naive L2D with the 95% interval of a paired block bootstrap over trials (blocks of 10 trials, 10,000 resamples); all methods are evaluated on the same trials.

CheXpert

One model for the whole week. In the whole-week setting, a single model is trained on week-long sequences and must cover all five days (Table 7 and Table 8, Figure 7). Both parameterizations of EDA-L2D again outperform Naive L2D in both cohorts, raising the average system AUROC from 0.791 to 0.835 and 0.830 in the general cohort and from 0.770 to 0.804 and 0.795 in the emergency-department cohort. The gains are close to those of the per-day setting for EDA-L2D, Per-step param., while EDA-L2D, RNN param. gains somewhat less, most visibly in the emergency-department cohort, where it must follow a longer sequence of 50 steps.

MethodEnl.CardiomediastinumConsolidationFractureAverage (14 findings)
Naive L2D0.681±0.0070.681 \pm0.0070.781±0.0100.781 \pm0.0100.716±0.0060.716 \pm0.0060.791±0.0020.791 \pm0.002
EDA-L2D, Per-step param.0.755±0.0040.755 \pm0.0040.822±0.0050.822 \pm0.0050.792±0.0090.792 \pm0.0090.835±0.0010.835 \pm0.001
EDA-L2D, RNN param.0.748±0.0100.748 \pm0.0100.823±0.0060.823 \pm0.0060.789±0.0120.789 \pm0.0120.830±0.0030.830 \pm0.003

Table 7. System AUROC on CheXpert, general cohort, one model for the whole week, for three findings and averaged over all 14 findings; each value is averaged over the steps of the week, with mean ± standard error over 5 seeds. EDA-L2D, RNN param. is evaluated under realistic routing.

MethodEnl.CardiomediastinumConsolidationFractureAverage (14 findings)
Naive L2D0.664±0.0030.664 \pm0.0030.738±0.0030.738 \pm0.0030.716±0.0080.716 \pm0.0080.770±0.0020.770 \pm0.002
EDA-L2D, Per-step param.0.720±0.0110.720 \pm0.0110.791±0.0040.791 \pm0.0040.765±0.0090.765 \pm0.0090.804±0.0020.804 \pm0.002
EDA-L2D, RNN param.0.701±0.0030.701 \pm0.0030.783±0.0030.783 \pm0.0030.739±0.0090.739 \pm0.0090.795±0.0020.795 \pm0.002

Table 8. System AUROC on CheXpert, emergency-department cohort, one model for the whole week; same layout as Table 7.

Figure 7. CheXpert with one model for the whole week, for the general cohort (left) and the emergency-department cohort (right). Top: system AUROC, averaged over the 14 findings; bottom: coverage. Vertical lines separate the weekdays. Lines show the mean over 5 seeds and shaded bands ± one standard error.

Individual findings. Figures 8 and 9 show the system AUROC along the week for three findings on which the classifier alone is weak: Enlarged Cardiomediastinum, Consolidation and Fracture. In both settings and both cohorts, EDA-L2D reaches a system AUROC close to 1 at the steps where the expert is most accurate, whereas Naive L2D stays close to the classifier alone, which accounts for the large per-finding gains in Table 2–Table 8.

Robustness Evaluation

In the main experiments, the expert at test time follows the same drift as during training. In practice, however, the expert’s drift at deployment may differ from the drift observed during model training. We therefore evaluate how the trained models behave when the expert departs from the training drift. All models are trained on the original drift, as in the main experiments, and are not retrained. At test time, we change only the expert or the feedback indicating whether the expert’s decisions were correct. Wherever possible, we use the same random draws for the expert so that the perturbed and unperturbed runs, as well as all methods within each run, are paired. In every test, we compare Naive L2D with EDA-L2D using either the Per-step or RNN parameterization.

Imperfect feedback

EDA-L2D, RNN param. is trained with the expert’s correctness at every step, observed immediately and without error. At deployment, this feedback is far more limited. Under realistic routing, the model observes the expert’s correctness only for deferred inputs and otherwise retains the most recent observed outcome. Keeping the expert’s original accuracy curve, we degrade this feedback further in three ways:

  • Delay: observed outcomes reach the model only once every k∈{5,10}k \in\{5, 10\} steps, so the feedback can be up to kk steps old.

  • Noise: each observed outcome is flipped with probability ε∈{0.1,0.2,0.3}\varepsilon\in\{0.1, 0.2, 0.3\}.

  • Partial observation: a deferred input reveals its outcome only with probability π∈{0.5,0.25}\pi\in\{0.5, 0.25\}.

Figure 9. System AUROC along the week for three findings, CheXpert with one model for the whole week, for the general cohort (left) and the emergency-department cohort (right). Lines show the mean over 5 seeds and shaded bands ± one standard error.

We measure the contribution of feedback as the gain in system accuracy over the same model evaluated without feedback.

Table 9 shows that this contribution degrades gracefully, and its mean remains positive under every condition and for every annotator. Delay is the most harmful degradation. For annotator 454, delivering outcomes only every 10 steps reduces the contribution from +0.36 to +0.15 points, whereas flipping 30% of the outcomes or revealing only a quarter of them still retains roughly 70–75% of it (+0.27 and +0.25 points). This ordering is expected: on a drifting expert, a stale outcome misrepresents the expert’s current state, while an occasional noisy or missing outcome is averaged out by the recurrent state. For annotators 1004 and 54, the contribution stays below 0.05 points in every condition.

These contributions are an order of magnitude smaller than the overall gain of EDA-L2D, RNN param. over Naive L2D (+3.32, +2.15 and +0.92 points; Table 1). Most of the advantage on a fixed drift curve therefore comes from the temporal pattern learned during training. This is unsurprising, because when the expert follows the same curve at training and test time, the time step alone is highly informative about their accuracy, leaving little for feedback to add. Feedback becomes more valuable when the expert departs from this pattern: during a temporary accuracy dip, its contribution grows with the depth of the dip (Appendix § 13.§ 2). Taken together, feedback provides a small but consistent additional gain on the original curve, and this gain does not depend on observations being immediate, accurate or complete.

Temporary dip

The expert’s accuracy drops by d∈{0.05,0.10,0.15}d \in\{0.05, 0.10, 0.15\} for 5 consecutive steps and then returns to its curve. The dip starts at step 1, 6, . . . , 46, and we average over these 10 positions. We report system accuracy during the dip and in the 5 steps after it.

During the dip, both EDA-L2D variants stay above Naive L2D at every dip depth for all three annotators (Table 10). Their gain shrinks as the dip deepens, because a model that relies on the expert’s usual accuracy defers too often while the expert is worse (Figure 10). For EDA-L2D, RNN param., feedback reduces this loss, and its contribution grows with the depth of the dip: from +0.37, +0.03 and +0.05 points without a dip to +0.58, +0.39 and +0.20 points at d=0.15d = 0.15 for annotators 454, 1004 and 54 (Table 12). Without feedback, the model falls slightly below Naive L2D for annotator 54 at d=0.15d = 0.15, whereas with feedback it stays above. In the five steps after the dip, the gains of both variants return to their level without a dip, so a temporary drop does not affect later decisions (Table 11). These results also show that feedback helps EDA-L2D, RNN param. when the expert’s accuracy temporarily drops below the level seen in training.

AnnotatorMethodd=0d = 0d=0.05d = 0.05d=0.10d = 0.10d=0.15d = 0.15
454EDA-L2D, Per-step param.+3.44±0.11+3.21±0.13+3.04±0.13+2.80±0.18
EDA-L2D, RNN param.+3.33±0.07+3.02±0.13+2.75±0.17+2.43±0.24
1004EDA-L2D, Per-step param.+2.01±0.13+1.81±0.15+1.59±0.18+1.38±0.22
EDA-L2D, RNN param.+2.15±0.13+1.70±0.14+1.22±0.16+0.74±0.20
54EDA-L2D, Per-step param.+0.93±0.06+0.67±0.05+0.41±0.11+0.15±0.18
EDA-L2D, RNN param.+0.90±0.07+0.64±0.09+0.41±0.15+0.15±0.20

Table 10. Temporary dip of depth dd for 5 steps on CIFAR-10: gain in system accuracy over Naive L2D (percentage points) during the dip, averaged over the 10 dip positions (d=0d = 0: no dip, same steps); mean ± s.e. over 5 seeds.

AnnotatorMethodd=0.05d = 0.05d=0.10d = 0.10d=0.15d = 0.15
454EDA-L2D, Per-step param.+3.54±0.11+3.57±0.12+3.58±0.13
EDA-L2D, RNN param.+3.38±0.07+3.40±0.08+3.39±0.07
1004EDA-L2D, Per-step param.+2.05±0.13+2.07±0.14+2.12±0.13
EDA-L2D, RNN param.+2.21±0.14+2.23±0.13+2.27±0.13
54EDA-L2D, Per-step param.+0.95±0.06+0.93±0.06+0.93±0.07
EDA-L2D, RNN param.+0.94±0.08+0.90±0.08+0.90±0.08

Table 11. Temporary dip on CIFAR-10: gain in system accuracy over Naive L2D (percentage points) in the 5 steps after the dip, averaged over the dip positions; mean ± s.e. over 5 seeds.

Figure 10. Temporary dip on CIFAR-10: gain in system accuracy over Naive L2D (percentage points) during the dip, as a function of the dip depth dd, averaged over the 10 dip positions. Error bars show ±\pm one standard error over 5 seeds; the dashed line marks Naive L2D.

Beyond the training horizon

All models are trained on sequences of 50 steps. To test whether they still work after the last trained step, we extend the sequences to 60 steps: steps 1–50 follow the original curve, and in steps 51–60 the expert keeps its accuracy at step 50, shifted by Δ∈{0,±0.05,±0.10}\Delta\in\{0,\pm0.05,\pm0.10\}. EDA-L2D, Per-step param. has no defer head beyond step 50 and reuses its last one, while EDA-L2D, RNN param. continues its recurrence. We report system accuracy over steps 51–60.

For annotator 454, EDA-L2D, RNN param. remains well above Naive L2D beyond the training horizon, by +5.71 to +8.17 points for every Δ\Delta (Table 13), so the benefit of modelling the expert’s drift carries over to steps it was not trained on. For annotators 1004 and 54, the gain is small and depends on the direction of the change: the model stays above Naive L2D when the expert’s accuracy decreases for annotator 1004 and when it increases for annotator 54, but falls below it when the change goes the other way. EDA-L2D, Per-step param., which reuses its last head, stays above Naive L2D in every setting and is the more robust choice beyond the training horizon.

AnnotatorMethodΔ=−0.10\Delta= -0.10−0.05-0.0500+0.05+0.05+0.10+0.10
454EDA-L2D, Per-step param.+9.35±0.35+9.35 \pm0.35+8.51±0.32+8.51 \pm0.32+7.84±0.36+7.84 \pm0.36+7.17±0.31+7.17 \pm0.31+6.46±0.29+6.46 \pm0.29
EDA-L2D, RNN param.+8.17±0.22+8.17 \pm0.22+7.52±0.19+7.52 \pm0.19+6.90±0.18+6.90 \pm0.18+6.31±0.18+6.31 \pm0.18+5.71±0.16+5.71 \pm0.16
1004EDA-L2D, Per-step param.+0.81±0.14+0.81 \pm0.14+0.81±0.16+0.81 \pm0.16+0.86±0.13+0.86 \pm0.13+0.84±0.14+0.84 \pm0.14+0.83±0.14+0.83 \pm0.14
EDA-L2D, RNN param.+0.52±0.12+0.52 \pm0.12+0.25±0.14+0.25 \pm0.14+0.00±0.13+0.00 \pm0.13−0.28±0.12-0.28 \pm0.12−0.52±0.14-0.52 \pm0.14
54EDA-L2D, Per-step param.+0.20±0.04+0.20 \pm0.04+0.24±0.09+0.24 \pm0.09+0.31±0.16+0.31 \pm0.16+0.26±0.24+0.26 \pm0.24+0.29±0.25+0.29 \pm0.25
EDA-L2D, RNN param.−1.58±0.61-1.58 \pm0.61−0.73±0.37-0.73 \pm0.37+0.10±0.22+0.10 \pm0.22+0.90±0.19+0.90 \pm0.19+0.96±0.23+0.96 \pm0.23

Table 13. Beyond the training horizon on CIFAR-10: in steps 51–60 the expert keeps its accuracy at step 50, shifted by Δ\Delta. Gain in system accuracy over Naive L2D (percentage points) over steps 51–60, mean ±\pm s.e. over 5 seeds.

References

  1. [1]Gavin Abercrombie, Dirk Hovy, and Vinodkumar Prabhakaran. Temporal and second language influence on intra-annotator agreement and stability in hate speech labelling. In Jakob Prange and Annemarie Friedrich (eds.), Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pp. 96–103, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.law-1.10. URL https://aclanthology.org/2023.law-1.10/.
  2. [2]Zeynep Akata, Dan Balliet, Maarten de Rijke, Frank Dignum, Virginia Dignum, A. E. Eiben, Antske Fokkens, Davide Grossi, Koen V. Hindriks, Holger Hoos, Haley Hung, Catholijn Jonker, Christof Monz, Mark Neerincx, Frans Oliehoek, Henry Prakken, Stefan Scholbach, Linda C. van der Gaag, Frank van Harmelen, Herke van Hoof, M. Birna van Riemsdijk, Aimee van Wynsberghe, Rineke Verbrugge, Bart Verheij, Paul Vossen, and Max Welling. A research agenda for hybrid intelligence: Augmenting human intellect with collaborative, adaptive, responsible, and explainable artificial intelligence. Computer, 53(8):18–28, 2020. doi: 10.1109/MC.2020.2996587. URL https://www.computer.org/csdl/magazine/co/2020/08/09153877/1UB5gL2CnS.DOI
  3. [3]Abdulaziz S. Alshabibi, Moayyad E. Suleiman, Kriscia A. Tapia, Robert Heard, and Patrick C. Brennan. Impact of time of day on radiology image interpretations. Clinical Radiology, 75(10):746–756, 2020. doi: 10.1016/j.crad.2020.05.004.DOI
  4. [4]Silvio G Bruni, Eric Bartlett, and Eugene Yu. Factors involved in discrepant preliminary radiology resident interpretations of neuroradiological imaging studies: a retrospective analysis. American Journal of Roentgenology, 198(6):1367–1374, 2012.DOI
  5. [5]Yuzhou Cao, Hussein Mozannar, Lei Feng, Hongxin Wei, and Bo An. In defense of softmax parametrization for calibrated and consistent learning to defer. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 38485–38503. Curran Associates, Inc., 2023. doi: 10.52202/075280-1671.DOI
  6. [6]Mohammad-Amin Charusaie, Hussein Mozannar, David Sontag, and Samira Samadi. Sample efficient learning of predictors that complement humans. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2972–3005. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/charusaie22a.html.arxiv.org/abs/2207.09584
  7. [7]C. K. Chow. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, EC-6(4):247–254, 1957. doi: 10.1109/TEC.1957.5222035.DOI
  8. [8]Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Proceedings of the International AAAI Conference on Web and Social Media, volume 11, pp. 512–515, 2017. doi: 10.1609/icwsm.v11i1.14955.DOI
  9. [9]Dominik Dellermann, Philipp Ebel, Matthias Söllner, and Jan Marco Leimeister. Hybrid intelligence. Business & Information Systems Engineering, 61(5):637–643, 2019. doi: 10.1007/s12599-019-00595-2. URL https://link.springer.com/article/10.1007/s12599-019-00595-2.DOI
  10. [10]Nikol Figalová, Hans Joachim Bieg, Michael Schulz, Jürgen Pichen, Martin Baumann, Lewis Chuang, and Olga Pollatos. Fatigue and mental underload further pronounced in 13 conditionally automated driving: Results from an eeg experiment on a test track. ArXiv preprint arXiv:2405.18114, 2024.
  11. [11]Patrick Hemmer, Lukas Thede, Michael Vössing, Johannes Jakubik, and Niklas Kühl. Learning to defer with limited expert predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 6002–6011, 2023. doi: 10.1609/aaai.v37i5.25742.
  12. [12]Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 590–597, 2019. doi: 10.1609/aaai.v33i01.3301590.DOI
  13. [13]Shalmali Joshi, Sonali Parbhoo, and Finale Doshi-Velez. Learning-to-defer for sequential medical decision-making under uncertainty. Transactions on Machine Learning Research, 2023. URL https://arxiv.org/abs/2109.06312.
  14. [14]Ece Kamar. Directions in hybrid intelligence: Complementing ai systems with human intelligence. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16), 2016. URL https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/HybridIntelligence.pdf.
  15. [15]Christoph Kern, Stephanie Eckman, Jacob Beck, Rob Chew, Bolei Ma, and Frauke Kreuter. Annotation sensitivity: Training data collection methods affect model performance. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14874–14886, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.992. URL https://aclanthology.org/2023.findings-emnlp.992/.DOI
  16. [16]Marko Kovic and Silje Kristiansen. The gambler’s fallacy fallacy (fallacy). Journal of Risk Research, 22(3): 291–302, 2019. doi: 10.1080/13669877.2017.1378248. URL https://doi.org/10.1080/13669877.2017.1378248.
  17. [17]Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.
  18. [18]Elizabeth A. Krupinski, Kevin S. Berbaum, Robert T. Caldwell, Kevin M. Schartz, and John Kim. Long radiology workdays reduce detection and accommodation accuracy. Journal of the American College of Radiology, 7(9):698–704, 2010. doi: 10.1016/j.jacr.2010.03.004.DOI
  19. [19]Eric Legler, Darío Cuevas Rivera, Sarah Schwöbel, Ben J. Wagner, and Stefan Kiebel. Cognitive computational model reveals repetition bias in a sequential decision-making task. Communications Psychology, 3 (1):92, 2025. doi: 10.1038/s44271-025-00271-0.DOI
  20. [20]David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: Improving fairness and accuracy by learning to defer. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.arxiv.org/abs/1711.06664
  21. [21]Yannis Montreuil, Hoang Duy Dang, Maxime Meyer, Lai Xing Ng, Axel Carlier, and Wei Tsang Ooi. Online learning-to-defer with varying experts. In Proceedings of the 29th International Conference on Artificial Intelligence and Statistics, volume 300 of Proceedings of Machine Learning Research, pp. 2701–2709. PMLR, 2026. URL https://proceedings.mlr.press/v300/montreuil26b.html.arxiv.org/abs/2605.12340
  22. [22]Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7076–7087. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/mozannar20b.html.arxiv.org/abs/2006.01862
  23. [23]Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Who should predict? exact algorithms for learning to defer to humans. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent (eds.), Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 10520–10545. PMLR, 2023. URL https://proceedings.mlr.press/v206/mozannar23a.html.arxiv.org/abs/2301.06197
  24. [24]Harikrishna Narasimhan, Wittawat Jitkrittum, Aditya K Menon, Ankit Rawat, and Sanjiv Kumar. Post-hoc estimators for learning to defer to an expert. In S. Koyejо, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 29292–29304. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/bc8f76d9caadd48f77025b1c889d2e2d-Paper-Conference.pdf.DOI
  25. [25]A. Newell and P. S. Rosenbloom. Mechanisms of skill acquisition and the law of practice. In J. R. Anderson (ed.), Cognitive Skills and Their Acquisition, chapter 1, pp. 1–51. Lawrence Erlbaum Associates, Inc., Hillsdale, NJ, 1981.
  26. [26]Ting Pan, Haibo Wang, Haiqing Si, Haibo Liu, and Mengyue Xu. Research on the identification of pilots’ fatigue status based on functional near-infrared spectroscopy. Aerospace, 9(3):173, 2022. doi: 10.3390/aerospace9030173.DOI
  27. [27]Joshua Peterson, Ruairidh Battleday, Thomas Griffiths, and Olga Russakovsky. Human Uncertainty Makes Classification More Robust. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9616–9625, Los Alamitos, CA, USA, November 2019. IEEE Computer Society. doi: 10.1109/ICCV.2019.00971. URL https://doi.ieeecsociety.org/10.1109/ICCV.2019.00971.DOI
  28. [28]M. Peukert et al. Subjective and objective fatigue dynamics in air traffic control. Journal of Sleep Research, 2023.
  29. [29]Dharmesh Tailor, Aditya Patra, Rajeev Verma, Putra Manggala, and Eric Nalisnick. Learning to defer to a population: A meta-learning approach. In Sanjoy Dasgupta, Stephan Mandt, and Yizhen Li (eds.), Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pp. 3475–3483. PMLR, 02–04 May 2024. URL https://proceedings.mlr.press/v238/tailor24a.html.
  30. [30]B.E. Turvey and J.L. Freeman. Jury psychology. In V.S. Ramachandran (ed.), Encyclopedia of Human Behavior (Second Edition), pp. 495–502. Academic Press, San Diego, second edition, 2012. ISBN 978-0-08-096180-4. doi: https://doi.org/10.1016/B978-0-12-375000-6.00216-0. URL https://www.sciencedirect.com/science/article/pii/B9780123750006002160.
  31. [31]Anne E. Urai, Jan Willem de Gee, Konstantinos Tsetsos, and Tobias H. Donner. Choice history biases subsequent evidence accumulation. eLife, 8:e46331, 2019. doi: 10.7554/eLife.46331.DOI
  32. [32]Didrika S. van de Wouw, Ryan T. McKay, and Nicholas Furl. Biased expectations about future choice options predict sequential economic decisions. Communications Psychology, 2(1):119, 2024. doi: 10.1038/s44271-024-00172-8.DOI
  33. [33]Rajeev Verma and Eric Nalisnick. Calibrated learning to defer with one-vs-all classifiers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 22184–22202. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/verma22c.html.arxiv.org/abs/2202.03673
  34. [34]Rajeev Verma, Daniel Barrejon, and Eric Nalisnick. Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, 2023.arxiv.org/abs/2210.16955
  35. [35]Jan Vosshenrich, Philipp Brantner, Joshy Cyriac, Daniel T. Boll, Elmar M. Merkle, and Tobias Heye. Quantifying radiology resident fatigue: Analysis of preliminary reports. Radiology, 298(3):632–639, 2021. doi: 10.1148/radiol.2021203486. URL https://doi.org/10.1148/radiol.2021203486.
  36. [36]Zheng Zhang, Cuong C. Nguyen, David Rosewarne, Kevin Wells, and Gustavo Carneiro. Fatigue-aware learning to defer via constrained optimisation, 2026. URL https://arxiv.org/abs/2604.00904.

Paper details

vlnc:W_ym47qz46
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents