Introduction
Adaptive inference systems allocate computation on a per-sample basis to balance accuracy and inference cost. For each sample, they decide which one of several predictive models with different cost-accuracy profiles should be used for prediction. This framework covers a broad range of methods, including learning to defer [38 ] , model cascades [6 , 56 ] (Zhang et al. 2024) and dynamic neural networks (DNNs) [22 ] , including early-exit systems [64 ] . We study adaptive inference in a classification setting that involves two components: an array of classifiers with varying cost-accuracy profiles, and a router module that decides, for each sample, which classifier makes the prediction.
A central challenge is how to train such systems effectively. Many approaches rely on separate training , where the classifiers are trained independently and routing is determined by a rule chosen separately, such as confidence-score thresholding [47 , 26 ] . This approach is often sample-inefficient: each classifier is trained on the full sample space, even though the trained router may send it samples from only a small subset of the space. This mismatch may lead to suboptimal results [26 ] and hinder the training process [29 ] . In contrast, joint training aligns classifier and router objectives, encouraging each classifier to specialize in samples routed to it. Empirically, joint training substantially improves prediction accuracy at a fixed sample size, including with losses designed to align training with test-time routing [23 , 67 , 44 , 29 ] . These results motivate the design of joint-training losses with consistency guarantees, which are currently scarce.
We address this gap in adaptive inference for classification under two common routing models, as illustrated in Figure 1 , considering T T T classifiers. Each sample has a base feature X X X , available from the start, and additional features Z 1 , … , Z T Z_1,\ldots,Z_T Z 1 , … , Z T , where Z t Z_t Z t is revealed once the t t t -th classifier is reached. In the single-stage (flat) model, the router directly selects a classifier t t t based on X X X alone, and the classifier then predicts from ( X , Z t ) (X,Z_t) ( X , Z t ) . This model covers certain configurations of elastic networks [14 ] , learning to defer to multiple experts [58 ] , and width-adaptive DNNs [22 ] . In the multi-stage (cascade) model, classifiers are invoked sequentially: at each stage t t t , the router uses the accumulated features ( X , Z 1 : t ) (X,Z_{1:t}) ( X , Z 1 : t ) to either exit with the t t t -th classifier’s prediction or defer to the next stage, which uses a more powerful classifier. We assume that each classifier t t t remains lightweight by predicting from ( X , Z t ) (X,Z_t) ( X , Z t ) alone. This model covers cascades [6 , 56 ] (Zhang et al. 2024), early-exit systems [64 ] , and, more generally, depth-adaptive DNNs [22 ] .
Figure 1. Illustration of flat (left) and cascade (right) routing models. A flat router selects a single classifier f t f_t f t based on X X X , while a cascade router sequentially invokes classifiers f 1 , … , f T f_1,\ldots,f_T f 1 , … , f T until the first exit decision is made based on ( X , Z 1 : t ) (X,Z_{1:t}) ( X , Z 1 : t ) , or until the last classifier f T f_T f T is reached.
In our LLM experiments, for example, Z 1 , … , Z T Z_1,\ldots,Z_T Z 1 , … , Z T correspond to embeddings of the answers produced by pretrained LLMs of increasing size, which serve as inputs to the classifiers (see Appendix § 14 for further justification of this modeling choice). Accessing Z t Z_t Z t incurs a known cost κ t \kappa_t κ t , e.g., the cost of running the t t t -th LLM. Such systems are typically designed so that κ t \kappa_t κ t varies with t t t and is non-decreasing in t t t . Due to the heterogeneous costs, jointly training the classifiers and router requires a cost-sensitive loss that balances the classification errors, measured by the zero-one loss, against the costs. We propose two such losses, one for each routing model.
Main Contributions
We propose hinge-based surrogates of the cost-sensitive zero-one loss for jointly training classifiers and routers in both flat and cascade models. In both cases, we prove linear calibration bounds (Theorems 3.1 and 3.2) that control the excess target risk by the excess hinge risk, thereby establishing Bayes consistency of the surrogate losses. This linear rate is sharp [20 ] . These results place joint training on a principled theoretical foundation. We validate our theory and demonstrate its practical utility through experiments on synthetic and real datasets.
A key technical contribution is the calibration bound for the cascade hinge loss. It is challenging to establish because the exit stage is a stopping time induced by a sequence of adaptive deferral decisions. We address this via a dynamic programming approach [4 , 49 ] , applying backward induction to recursively establish stage-wise calibration through a novel risk decomposition that separates excess risks into contributions from suboptimal deferral rules and suboptimal classifiers. A matching lower bound (Proposition C.8) shows that the resulting calibration constant is tight.
Since routers can require more labeled data to train than classifiers, we also propose a novel post-training prediction-powered inference (PPI) procedure that leverages unlabeled data to refine a trained flat router. At each input, PPI recovers the oracle routing decision with high probability, whenever the width of its confidence interval on risk improvement is narrower than the gap between the oracle and the second-best routing decisions (Theorem 4.1 ). This constitutes the first integration of PPI into adaptive inference.
Related Work
Learning to defer allows a model to either predict with a learned classifier or defer to one or more experts with heterogeneous capabilities and costs [33 , 27 , 62 ] . Here, only the classifier and the deferral rule are learned; the experts are fixed. Building on foundational works on Bayes consistency [3 , 54 , 50 , 69 ] , a line of works develops Bayes-consistent surrogate losses with a single expert [38 , 57 ] or with multiple experts [58 ] . For the single-expert setting, later works establish realizable H \mathcal{H} H -consistency, a guarantee relative to a restricted hypothesis class [39 , 35 ] , where Mao et al. [ [35 ] further identify surrogates that are simultaneously realizable H \mathcal{H} H -consistent, Bayes-consistent, and admit H \mathcal{H} H -consistency bounds [2 ] . Closer to our multi-model setting, Mao et al. [ [36 ] and Mao et al. [ [34 ] prove H \mathcal{H} H -consistency bounds in the multiple-expert setting. Mao et al. [ [36 ] consider the standard setting in which the classifier and the deferral rule are learned jointly, while Mao et al. [ [34 ] consider a two-stage setting where the classifier is first trained then fixed, and the deferral rule is learned in a second stage. Finally, Mao et al. [ [37 ] establish realizable H \mathcal{H} H -consistency, H \mathcal{H} H -consistency bounds, and Bayes consistency for learning to defer to multiple experts, in both the standard and two-stage settings.
In contrast to learning to defer, which learns a single classifier, Regol et al. [ [45 ] and Pugnana et al. [ [42 ] recently studied a setting where all classifiers involved are learned: they study joint training of two classifiers with a deferral rule (or router), and their analysis is restricted to this setting. Since adaptive inference systems often involve more than two models [6 , 56 ] (Zhang et al. 2024), we address this limitation by developing principled losses for jointly training multiple classifiers with the router under both flat and cascade models. As our work is primarily theoretical, we defer discussion of the rich literature on practical architectures and empirical performance in adaptive inference to Appendix A.
Notation
We use x i : = [ x ] i x_i := [x]_i x i := [ x ] i to denote the i i i -th entry of vector x x x . We write [ n ] : = { 1 , 2 , … , n } [n] := \{1, 2, \ldots, n\} [ n ] := { 1 , 2 , … , n } , and for i , j ∈ N i,j \in\mathbb{N} i , j ∈ N with i ≤ j i \leq j i ≤ j , we use the shorthand a i : j : = { a i , a i + 1 , … , a j } a_{i:j} := \{a_i, a_{i+1}, \ldots, a_j\} a i : j := { a i , a i + 1 , … , a j } for sequences of variables or functions. We denote the n n n -dimensional probability simplex by Δ n − 1 \Delta^{n-1} Δ n − 1 . We use uppercase letters for random variables, lowercase for their realizations (e.g., X X X and x x x ), and calligraphic letters to denote spaces of realizations (e.g., x ∈ X x \in\mathcal{X} x ∈ X ).
Multi-Model Classification
We study a multi-model classification problem, where the goal is to learn a tuple π = ( f 1 : T , r ) \pi= (f_{1:T}, r) π = ( f 1 : T , r ) , consisting of T ≥ 2 T \geq2 T ≥ 2 classifiers f 1 : T f_{1:T} f 1 : T and a router r r r . Each classifier f t f_t f t takes the form:
f t : X × Z t → [ K ] , for t ∈ [ T ] , (1) f_t : \mathcal{X} \times\mathcal{Z}_t \to[K], \quad\text{for } t \in[T], \tag*{(1)} f t : X × Z t → [ K ] , for t ∈ [ T ] , (1)
where X ⊂ R p \mathcal{X} \subset\mathbb{R}^{p} X ⊂ R p denotes the space of the base features (e.g., a multiple-choice question prompt), [ K ] [K] [ K ] is the label space for K K K classes (e.g., the answer choices), and Z t ⊂ R p t \mathcal{Z}_t \subset\mathbb{R}^{p_t} Z t ⊂ R p t denotes the space of the additional feature (e.g., LLM embeddings) available at f t f_t f t . Accessing Z t \mathcal{Z}_t Z t incurs a known nonnegative cost κ t \kappa_t κ t , with κ t ≥ κ t − 1 \kappa_t \ge\kappa_{t-1} κ t ≥ κ t − 1 , reflecting that later classifiers are more accurate but more costly. The routing rule takes the form
r : H → [ T ] , (2) r : \mathcal{H} \to[T], \tag*{(2)} r : H → [ T ] , (2)
which maps the history information H ⊆ X × ∏ t = 1 T Z t \mathcal{H} \subseteq\mathcal{X} \times\prod_{t=1}^{T} \mathcal{Z}_t H ⊆ X × ∏ t = 1 T Z t available to the router to a classifier index. The space H \mathcal{H} H varies with the router type, as we specify in Section 2.1. We denote by Π \Pi Π the class of all multi-model classifiers π = ( f 1 : T , r ) \pi= (f_{1:T}, r) π = ( f 1 : T , r ) satisfying (1) and (2).
For a given h ∈ H h \in\mathcal{H} h ∈ H , the prediction y ^ \hat{y} y ^ of π = ( f 1 : T , r ) \pi= (f_{1:T}, r) π = ( f 1 : T , r ) is the output of the classifier selected by the router:
y ^ = f r ( h ) ( x , z r ( h ) ) . (3) \hat{y} = f_{r(h)}(x, z_{r(h)}). \tag*{(3)} y ^ = f r ( h ) ( x , z r ( h ) ) . (3)
Training data. To train π ∈ Π \pi\in\Pi π ∈ Π , we use n n n training samples generated i.i.d. according to
X ( i ) , Z 1 : T ( i ) , Y ( i ) ∼ i . i . d . P for i ∈ [ n ] , (4) X^{(i)}, Z_{1:T}^{(i)}, Y^{(i)} \mathrel{\overset{\mathrm{i.i.d.}}{\sim}} P \quad\text{for } i \in[n], \tag*{(4)} X ( i ) , Z 1 : T ( i ) , Y ( i ) ∼ i.i.d. P for i ∈ [ n ] , (4)
where X ( i ) ∈ X X^{(i)} \in\mathcal{X} X ( i ) ∈ X and Z t ( i ) ∈ Z t Z_t^{(i)} \in\mathcal{Z}_t Z t ( i ) ∈ Z t for t ∈ [ T ] t \in[T] t ∈ [ T ] are features, Y ( i ) ∈ [ K ] Y^{(i)} \in[K] Y ( i ) ∈ [ K ] is the label, and P P P is the joint distribution over ( X ( i ) , Z 1 : T ( i ) , Y ( i ) ) (X^{(i)}, Z_{1:T}^{(i)}, Y^{(i)}) ( X ( i ) , Z 1 : T ( i ) , Y ( i ) ) . We drop the superscript ( i ) (i) ( i ) for a generic sample. In some cascade applications, it is natural to assume the conditional independence
Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) , (5) Y \perp Z_{1:t-1} \mid(X, Z_t), \tag*{(5)} Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) , (5)
where ( X , Z t ) (X, Z_t) ( X , Z t ) subsumes all information in Z 1 : t − 1 Z_{1:t-1} Z 1 : t − 1 relevant to predicting Y Y Y . We now formalize two routing models and their corresponding multi-model classifiers.
Single-Stage and Cascade Routing Models
The single-stage router directly selects one of the T T T classifiers using only the base feature, formalized as:
Definition 2.1 (Single-stage (flat) router). With history information H = X \mathcal{H} = \mathcal{X} H = X , a flat router takes the form
r : X → [ T ] . (6) r : \mathcal{X} \to[T]. \tag*{(6)} r : X → [ T ] . (6)
The multi-stage cascade router sequentially decides, at each stage t t t , whether to exit with classifier f t f_t f t or defer to the more powerful classifier f t + 1 f_{t+1} f t + 1 after collecting additional features, as formalized below:
Definition 2.2 (Cascade router). With history information H t = X × ∏ τ = 1 t Z τ \mathcal{H}_t = \mathcal{X} \times\prod_{\tau=1}^{t} \mathcal{Z}_\tau H t = X × ∏ τ = 1 t Z τ available at stage t ∈ [ T ] t \in[T] t ∈ [ T ] , a cascade router consists of stage-wise deferral rules
d t : H t → { 0 , 1 } for t ∈ [ T − 1 ] , (7) d_t : \mathcal{H}_t \to\{0,1\} \quad\text{for } t \in[T-1], \tag*{(7)} d t : H t → { 0 , 1 } for t ∈ [ T − 1 ] , (7)
where d t ( h t ) = 0 d_t(h_t)=0 d t ( h t ) = 0 indicates exit at stage t t t and d t ( h t ) = 1 d_t(h_t)=1 d t ( h t ) = 1 indicates deferral to the next stage. We fix d T ≡ 0 d_T \equiv0 d T ≡ 0 , so the cascade always terminates at the final stage T T T . Writing H = H T \mathcal{H}=\mathcal{H}_T H = H T for the full history, the stopping time induced by the deferral rules is the cascade routing rule
r ( h ) ∈ min { t ∈ [ T ] : d t ( h t ) = 0 } . (8) r(h) \in\min\{t \in[T] : d_t(h_t)=0\}. \tag*{(8)} r ( h ) ∈ min { t ∈ [ T ] : d t ( h t ) = 0 } . (8)
Restricting the class Π \Pi Π of multi-model classifiers to flat or cascade routers (Definitions 2.1 or 2.2) yields the hypothesis classes Π f l a t ⊂ Π \Pi_{\mathrm{flat}} \subset\Pi Π flat ⊂ Π and Π c a s c ⊂ Π \Pi_{\mathrm{casc}} \subset\Pi Π casc ⊂ Π , respectively, which are the focus of this work.
The Target Cost-Sensitive Zero-One Loss
To capture the tradeoff between prediction error and cost, we follow prior work on cost-sensitive classification and learning to defer [5 , 36 ] and augment the standard zero-one loss with the costs. This yields the 0-1-c loss , denoted by ℓ : X × ∏ t = 1 T Z t × [ K ] × Π × R ≥ 0 T → R ≥ 0 \ell:\mathcal{X}\times\prod_{t=1}^{T}\mathcal{Z}_t\times[K]\times\Pi\times\mathbb{R}_{\geq0}^{T}\to\mathbb{R}_{\geq0} ℓ : X × ∏ t = 1 T Z t × [ K ] × Π × R ≥ 0 T → R ≥ 0 . For a multi-model classifier π = ( f 1 : T , r ) ∈ Π \pi=(f_{1:T},r)\in\Pi π = ( f 1 : T , r ) ∈ Π and a cost vector c ∈ R ≥ 0 T c\in\mathbb{R}_{\geq0}^{T} c ∈ R ≥ 0 T associated with f 1 : T f_{1:T} f 1 : T , we define
ℓ ( x , z 1 : T , y ; π , c ) = 1 { f r ( h ) ( x , z r ( h ) ) ≠ y } + c r ( h ) . (9) \ell(x,z_{1:T},y;\pi,c)=\mathbb{1}\{f_{r(h)}(x,z_{r(h)})\ne y\}+c_{r(h)}. \tag*{(9)} ℓ ( x , z 1 : T , y ; π , c ) = 1 { f r ( h ) ( x , z r ( h ) ) = y } + c r ( h ) . (9)
That is, an incorrect prediction incurs a unit loss, while exiting at the t t t -th classifier additionally incurs a cost c t c_t c t . In (9), the semicolon separates the random data arguments from the fixed multi-model classifier π \pi π and cost vector c c c . The corresponding 0-1-c c c population risk is then
R ( π ) ≡ R ( π ; c ) : = E [ ℓ ( X , Z 1 : T , Y ; π , c ) ] . (10) \mathcal{R}(\pi)\equiv\mathcal{R}(\pi;c):=\mathbb{E}\left[\ell(X,Z_{1:T},Y;\pi,c)\right]. \tag*{(10)} R ( π ) ≡ R ( π ; c ) := E [ ℓ ( X , Z 1 : T , Y ; π , c ) ] . (10)
The form of c c c differs between the two routing models. For the single-stage router, c = [ κ 1 , κ 2 , … , κ T ] T c=[\kappa_1,\kappa_2,\ldots,\kappa_T]^{\mathsf{T}} c = [ κ 1 , κ 2 , … , κ T ] T , where recall κ t \kappa_t κ t denotes the cost of executing only the t t t -th classifier. In a cascade, since the classifiers are invoked sequentially, costs accumulate along the execution path, yielding c = [ κ 1 , κ 1 + κ 2 , … , ∑ t = 1 T κ t ] T c=[\kappa_1,\kappa_1+\kappa_2,\ldots,\sum_{t=1}^{T}\kappa_t]^{\mathsf{T}} c = [ κ 1 , κ 1 + κ 2 , … , ∑ t = 1 T κ t ] T .
Main Results
This section develops surrogate losses for the target 0-1-c c c loss (9), one for the flat model and one for the cascade model. Throughout, we use a tilde ⋅ ~ \tilde{\cdot} ⋅ ~ to denote soft (score-based or probabilistic) predictors, reserving untilded notation for their hard, discrete counterparts. We introduce two soft multi-model-classifier classes Π ~ f l a t \widetilde{\Pi}_{\mathrm{flat}} Π flat and Π ~ c a s c \widetilde{\Pi}_{\mathrm{casc}} Π casc in parallel to Π f l a t \Pi_{\mathrm{flat}} Π flat and Π c a s c \Pi_{\mathrm{casc}} Π casc . Each element π ~ \tilde{\pi} π ~ of these classes is composed of a sequence of score functions f ~ 1 : T \tilde{f}_{1:T} f ~ 1 : T and a soft router r ~ \tilde{r} r ~ that outputs a probability distribution over the T T T classifiers. Formally, we define the class Π ~ f l a t \widetilde{\Pi}_{\mathrm{flat}} Π flat as:
Π ~ f l a t = { π ~ = ( f ~ 1 : T , r ~ ) : f ~ t : X × Z t → R K with \widetilde{\Pi}_{\mathrm{flat}}=
\left\{\tilde{\pi}=(\tilde{f}_{1:T},\tilde{r}):\tilde{f}_t:\mathcal{X}\times\mathcal{Z}_t\to\mathbb{R}^{K}\text{ with}
\right. Π flat = { π ~ = ( f ~ 1 : T , r ~ ) : f ~ t : X × Z t → R K with
∑ k ∈ [ K ] [ f ~ t ( x , z t ) ] k = 0 ∀ ( x , z t ) , and r ~ : X → Δ T − 1 } . (11) \left.
\sum_{k\in[K]}[\tilde{f}_t(x,z_t)]_k=0\ \forall(x,z_t),\text{ and }\tilde{r}:\mathcal{X}\to\Delta^{T-1}\right\}. \tag*{(11)} k ∈ [ K ] ∑ [ f ~ t ( x , z t ) ] k = 0 ∀ ( x , z t ) , and r ~ : X → Δ T − 1 ⎭ ⎬ ⎫ . (11)
Each π ~ = ( f ~ 1 : T , r ~ ) ∈ Π ~ f l a t \tilde{\pi}=(\tilde{f}_{1:T},\tilde{r})\in\widetilde{\Pi}_{\mathrm{flat}} π ~ = ( f ~ 1 : T , r ~ ) ∈ Π flat induces a hard multi-model classifier π = ( f 1 : T , r ) = φ f l a t ( π ~ ) ∈ Π f l a t \pi=(f_{1:T},r)=\varphi_{\mathrm{flat}}(\tilde{\pi})\in\Pi_{\mathrm{flat}} π = ( f 1 : T , r ) = φ flat ( π ~ ) ∈ Π flat via a deterministic argmax map φ f l a t : Π ~ f l a t → Π f l a t \varphi_{\mathrm{flat}}:\widetilde{\Pi}_{\mathrm{flat}}\to\Pi_{\mathrm{flat}} φ flat : Π flat → Π flat , defined by the componentwise relations:
f t ( x , z t ) ∈ arg max k ∈ [ K ] [ f ~ t ( x , z t ) ] k , ∀ t ∈ [ T ] , (12) f_t(x,z_t)\in\underset{k\in[K]}{\operatorname{arg\,max}}\,[\tilde{f}_t(x,z_t)]_k,\qquad\forall t\in[T], \tag*{(12)} f t ( x , z t ) ∈ k ∈ [ K ] arg max [ f ~ t ( x , z t ) ] k , ∀ t ∈ [ T ] , (12)
r ( x ) ∈ arg max t ∈ [ T ] r ~ t ( x ) , with r ~ t ( x ) = [ r ~ ( x ) ] t . (13) r(x)\in\underset{t\in[T]}{\operatorname{arg\,max}}\ \tilde{r}_t(x),\qquad\text{with }\tilde{r}_t(x)=[\tilde{r}(x)]_t. \tag*{(13)} r ( x ) ∈ t ∈ [ T ] arg max r ~ t ( x ) , with r ~ t ( x ) = [ r ~ ( x ) ] t . (13)
For the cascade, we define the class Π ~ c a s c \widetilde{\Pi}_{\mathrm{casc}} Π casc and the map φ c a s c : Π ~ c a s c → Π c a s c \varphi_{\mathrm{casc}}:\widetilde{\Pi}_{\mathrm{casc}}\to\Pi_{\mathrm{casc}} φ casc : Π casc → Π casc analogously. The soft router r ~ \widetilde{r} r consists of soft deferral rules d ~ t : H t → [ 0 , 1 ] \widetilde{d}_t:\mathcal{H}_t\to[0,1] d t : H t → [ 0 , 1 ] for t ∈ [ T − 1 ] t\in[T-1] t ∈ [ T − 1 ] with d ~ T ≡ 0 \widetilde{d}_T\equiv0 d T ≡ 0 , and the map φ c a s c \varphi_{\mathrm{casc}} φ casc induces the hard, thresholding router r r r which replaces (13) and mirrors the structure of (8) , taking the form
r ( h ) ∈ min { t ∈ [ T ] : d ~ t ( h t ) < 0.5 } . (14) r(h)\in\min\{t\in[T]:\widetilde{d}_t(h_t)<0.5\}. \tag*{(14)} r ( h ) ∈ min { t ∈ [ T ] : d t ( h t ) < 0.5 } . (14)
Our goal is to design, for each model ∈ { f l a t , c a s c } \in\{\mathrm{flat},\mathrm{casc}\} ∈ { flat , casc } , a surrogate loss defined over the soft class Π ~ m o d e l \widetilde{\Pi}_{\mathrm{model}} Π model such that driving the surrogate risk of π ~ ∈ Π ~ m o d e l \widetilde{\pi}\in\widetilde{\Pi}_{\mathrm{model}} π ∈ Π model to its minimum drives the target 0-1-c risk (10) of the induced hard multi-model classifier to its minimum over the hard class Π m o d e l \Pi_{\mathrm{model}} Π model . In particular, any minimizer π ~ m o d e l ∗ \widetilde{\pi}^{*}_{\mathrm{model}} π model ∗ of the surrogate risk induces a minimizer π m o d e l ∗ = φ m o d e l ( π ~ m o d e l ∗ ) \pi^{*}_{\mathrm{model}}=\varphi_{\mathrm{model}}(\widetilde{\pi}^{*}_{\mathrm{model}}) π model ∗ = φ model ( π model ∗ ) of the target risk. This desired property is Bayes consistency. In the sequel, we drop subscript from φ m o d e l \varphi_{\mathrm{model}} φ model when the model is clear from context.
Cost-sensitive hinge loss. Let [ x ] + : = max ( x , 0 ) [x]_+:=\max(x,0) [ x ] + := max ( x , 0 ) . We propose a cost-sensitive hinge loss ℓ h i n g e : R K × [ K ] × R ≥ 0 → R ≥ 0 \ell^{\mathrm{hinge}}:\mathbb{R}^{K}\times[K]\times\mathbb{R}_{\ge0}\to\mathbb{R}_{\ge0} ℓ hinge : R K × [ K ] × R ≥ 0 → R ≥ 0 , which acts on the score vector v v v output by a given soft classifier (e.g., f ~ t \widetilde{f}_t f t ), the true label y y y , and the cost c c c of the classifier as follows:
ℓ h i n g e ( v , y ; c ) = ∑ k ∈ [ K ] ∖ y [ v k + 1 K − 1 ] + + K K − 1 c . (15) \ell^{\mathrm{hinge}}(v,y;c)=\sum_{k\in[K]\setminus y}\left[v_k+\frac{1}{K-1}\right]_+ + \frac{K}{K-1}c. \tag*{(15)} ℓ hinge ( v , y ; c ) = k ∈ [ K ] ∖ y ∑ [ v k + K − 1 1 ] + + K − 1 K c . (15)
This loss is a cost-sensitive extension of the multiclass hinge loss from [53 ] and is also used for building the surrogate loss in [45 ] for jointly training two classifiers. We use it in § 3.1 and § 3.2 to construct surrogates of the target loss (9) for joint training in Π ~ f l a t \widetilde{\Pi}_{\mathrm{flat}} Π flat and Π ~ c a s c \widetilde{\Pi}_{\mathrm{casc}} Π casc .
Bayes-Consistent Loss for the Flat Model
For training π ~ = ( f ~ 1 : T , r ~ ) ∈ Π ~ f l a t \widetilde{\pi}=(\widetilde{f}_{1:T},\widetilde{r})\in\widetilde{\Pi}_{\mathrm{flat}} π = ( f 1 : T , r ) ∈ Π flat in the flat setting, we construct the surrogate loss ℓ f l a t h i n g e : X × ∏ t = 1 T Z t × [ K ] × Π ~ f l a t × R ≥ 0 T → R ≥ 0 \ell^{\mathrm{hinge}}_{\mathrm{flat}}:\mathcal{X}\times\prod_{t=1}^{T}\mathcal{Z}_t\times[K]\times\widetilde{\Pi}_{\mathrm{flat}}\times\mathbb{R}_{\ge0}^{T}\to\mathbb{R}_{\ge0} ℓ flat hinge : X × ∏ t = 1 T Z t × [ K ] × Π flat × R ≥ 0 T → R ≥ 0 as a weighted sum of the hinge loss ℓ h i n g e \ell^{\mathrm{hinge}} ℓ hinge across all classifiers:
ℓ f l a t h i n g e ( x , z 1 : T , y ; π ~ , c ) = ∑ t = 1 T r ~ t ( x ) ℓ h i n g e ( f ~ t ( x , z t ) , y ; c t ) , (16) \ell^{\mathrm{hinge}}_{\mathrm{flat}}(x,z_{1:T},y;\widetilde{\pi},c)=\sum_{t=1}^{T}\widetilde{r}_t(x)\ell^{\mathrm{hinge}}(\widetilde{f}_t(x,z_t),y;c_t), \tag*{(16)} ℓ flat hinge ( x , z 1 : T , y ; π , c ) = t = 1 ∑ T r t ( x ) ℓ hinge ( f t ( x , z t ) , y ; c t ) , (16)
with the associated risk R f l a t h i n g e ( π ~ ) = R f l a t h i n g e ( π ~ ; c ) : = E [ ℓ f l a t h i n g e ( X , Z 1 : T , Y ; π ~ , c ) ] \mathcal{R}^{\mathrm{hinge}}_{\mathrm{flat}}(\widetilde{\pi})=\mathcal{R}^{\mathrm{hinge}}_{\mathrm{flat}}(\widetilde{\pi};c):=\mathbb{E}\left[\ell^{\mathrm{hinge}}_{\mathrm{flat}}(X,Z_{1:T},Y;\widetilde{\pi},c)\right] R flat hinge ( π ) = R flat hinge ( π ; c ) := E [ ℓ flat hinge ( X , Z 1 : T , Y ; π , c ) ] . The calibration bound below shows that the excess 0-1-c risk is linearly bounded by the excess hinge risk in the flat setting.
Theorem 3.1 (Linear calibration of ℓ f l a t h i n g e \ell^{\mathrm{hinge}}_{\mathrm{flat}} ℓ flat hinge w.r.t. ℓ \ell ℓ ). Define R f l a t ∗ : = min π ∈ Π f l a t R ( π ) \mathcal{R}^{*}_{\mathrm{flat}}:=\min_{\pi\in\Pi_{\mathrm{flat}}}\mathcal{R}(\pi) R flat ∗ := min π ∈ Π flat R ( π ) and R f l a t h i n g e ∗ : = min π ~ ∈ Π ~ f l a t R f l a t h i n g e ( π ~ ) \mathcal{R}^{\mathrm{hinge}*}_{\mathrm{flat}}:=\min_{\widetilde{\pi}\in\widetilde{\Pi}_{\mathrm{flat}}}\mathcal{R}^{\mathrm{hinge}}_{\mathrm{flat}}(\widetilde{\pi}) R flat hinge ∗ := min π ∈ Π flat R flat hinge ( π ) . Then, for any π ~ ∈ Π ~ f l a t \widetilde{\pi}\in\widetilde{\Pi}_{\mathrm{flat}} π ∈ Π flat ,
R ( φ ( π ~ ) ) − R f l a t ∗ ≤ T ( K − 1 ) ( R f l a t h i n g e ( π ~ ) − R f l a t h i n g e ∗ ) . (17) \mathcal{R}(\varphi(\widetilde{\pi}))-\mathcal{R}^{*}_{\mathrm{flat}}\le T(K-1)\left(\mathcal{R}^{\mathrm{hinge}}_{\mathrm{flat}}(\widetilde{\pi})-\mathcal{R}^{\mathrm{hinge}*}_{\mathrm{flat}}\right). \tag*{(17)} R ( φ ( π )) − R flat ∗ ≤ T ( K − 1 ) ( R flat hinge ( π ) − R flat hinge ∗ ) . (17) This implies R f l a t h i n g e ( π ~ ) → R f l a t h i n g e ∗ ⇒ R ( φ ( π ~ ) ) → R f l a t ∗ \mathcal{R}^{\mathrm{hinge}}_{\mathrm{flat}}(\widetilde{\pi})\to\mathcal{R}^{\mathrm{hinge}*}_{\mathrm{flat}}\Rightarrow\mathcal{R}(\varphi(\widetilde{\pi}))\to\mathcal{R}^{*}_{\mathrm{flat}} R flat hinge ( π ) → R flat hinge ∗ ⇒ R ( φ ( π )) → R flat ∗ , establishing Bayes consistency of the surrogate loss ℓ f l a t h i n g e \ell^{\mathrm{hinge}}_{\mathrm{flat}} ℓ flat hinge . We detail the proof in Appendix § 8 .
Bayes-Consistent Loss for the Cascade
For training π ~ = ( f ~ 1 : T , r ~ ) ∈ Π ~ c a s c \tilde{\pi}=(\tilde{f}_{1:T},\tilde{r})\in\tilde{\Pi}_{\mathrm{casc}} π ~ = ( f ~ 1 : T , r ~ ) ∈ Π ~ casc , we build the cascade surrogate hinge loss ℓ c a s c h i n g e : X × ∏ t = 1 T Z t × [ K ] × Π ~ c a s c × R ≥ 0 T → R ≥ 0 \ell_{\mathrm{casc}}^{\mathrm{hinge}}:\mathcal{X}\times\prod_{t=1}^{T}\mathcal{Z}_{t}\times[K]\times\tilde{\Pi}_{\mathrm{casc}}\times\mathbb{R}_{\geq0}^{T}\to\mathbb{R}_{\geq0} ℓ casc hinge : X × ∏ t = 1 T Z t × [ K ] × Π ~ casc × R ≥ 0 T → R ≥ 0 as follows:
ℓ c a s c h i n g e ( x , z 1 : T , y ; π ~ , c ) = ∑ t = 1 T r ~ t ( x , z 1 : t ) ℓ h i n g e ( f ~ t ( x , z t ) , y ; c t ) . (18) \ell_{\mathrm{casc}}^{\mathrm{hinge}}(x,z_{1:T},y;\tilde{\pi},c)=\sum_{t=1}^{T}\tilde{r}_{t}(x,z_{1:t})\ell^{\mathrm{hinge}}(\tilde{f}_{t}(x,z_{t}),y;c_{t}). \tag*{(18)} ℓ casc hinge ( x , z 1 : T , y ; π ~ , c ) = t = 1 ∑ T r ~ t ( x , z 1 : t ) ℓ hinge ( f ~ t ( x , z t ) , y ; c t ) . (18)
where
r ~ t ( x , z 1 : t ) = ( ∏ τ = 1 t − 1 d ~ τ ( x , z 1 : τ ) ) ( 1 − d ~ t ( x , z 1 : t ) ) , (19) \tilde{r}_{t}(x,z_{1:t})=\left(\prod_{\tau=1}^{t-1}\tilde{d}_{\tau}(x,z_{1:\tau})\right)(1-\tilde{d}_{t}(x,z_{1:t})), \tag*{(19)} r ~ t ( x , z 1 : t ) = ( τ = 1 ∏ t − 1 d ~ τ ( x , z 1 : τ ) ) ( 1 − d ~ t ( x , z 1 : t )) , (19)
and d ~ T ≡ 0 \tilde{d}_{T}\equiv0 d ~ T ≡ 0 . The outer sum in (18) runs over exit stages t ∈ [ T ] t\in[T] t ∈ [ T ] . The r ~ t ( x , z 1 : t ) \tilde{r}_{t}(x,z_{1:t}) r ~ t ( x , z 1 : t ) term represents the probability of exiting at stage t t t , and the ℓ h i n g e \ell^{\mathrm{hinge}} ℓ hinge term is the hinge loss incurred upon exiting at that stage. We define the risk associated with the loss function ℓ c a s c h i n g e \ell_{\mathrm{casc}}^{\mathrm{hinge}} ℓ casc hinge as R c a s c h i n g e ( π ~ ) ≡ R c a s c h i n g e ( π ~ ; c ) : = E [ ℓ c a s c h i n g e ( X , Z 1 : T ; Y ; π ~ , c ) ] \mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}}(\tilde{\pi})\equiv\mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}}(\tilde{\pi};c):=\mathbb{E}[\ell_{\mathrm{casc}}^{\mathrm{hinge}}(X,Z_{1:T};Y;\tilde{\pi},c)] R casc hinge ( π ~ ) ≡ R casc hinge ( π ~ ; c ) := E [ ℓ casc hinge ( X , Z 1 : T ; Y ; π ~ , c )] .
The cascade setting is technically more challenging to analyze than the flat setting because adaptive deferral decisions induce the stopping time in (14) , which indexes the classifier ultimately invoked. As a result, the optimal deferral decision at each stage must account for both the immediate loss and the expected future loss. To prove that ℓ c a s c h i n g e \ell_{\mathrm{casc}}^{\mathrm{hinge}} ℓ casc hinge is calibrated in an analogous sense to Theorem 3.1 of the flat setting, we cast the cascade as a finite-horizon dynamic program and use backward induction with stage-wise risk decomposition to recursively propagate single-stage calibration through the cascade, ultimately yielding Theorem 3.2 below.
Theorem 3.2 (Linear calibration of ℓ c a s c h i n g e \ell_{\mathrm{casc}}^{\mathrm{hinge}} ℓ casc hinge w.r.t. ℓ \ell ℓ ). Assume Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) Y\perp Z_{1:t-1}\mid(X,Z_{t}) Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) as in (5). Define R c a s c ∗ : = min π ∈ Π c a s c R ( π ) \mathcal{R}_{\mathrm{casc}}^{*}:=\min_{\pi\in\Pi_{\mathrm{casc}}}\mathcal{R}(\pi) R casc ∗ := min π ∈ Π casc R ( π ) and R c a s c h i n g e ∗ : = min π ~ ∈ Π ~ c a s c R c a s c h i n g e ( π ~ ) \mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}*}:=\min_{\tilde{\pi}\in\tilde{\Pi}_{\mathrm{casc}}}\mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}}(\tilde{\pi}) R casc hinge ∗ := min π ~ ∈ Π ~ casc R casc hinge ( π ~ ) . Then, for any π ~ ∈ Π ~ c a s c \tilde{\pi}\in\tilde{\Pi}_{\mathrm{casc}} π ~ ∈ Π ~ casc ,
R ( ϕ ( π ~ ) ) − R c a s c ∗ ≤ 2 T − 1 ( K − 1 ) ( R c a s c h i n g e ( π ~ ) − R c a s c h i n g e ∗ ) . (20) \mathcal{R}(\phi(\tilde{\pi}))-\mathcal{R}_{\mathrm{casc}}^{*}\leq2^{T-1}(K-1)\left(\mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}}(\tilde{\pi})-\mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}*}\right). \tag*{(20)} R ( ϕ ( π ~ )) − R casc ∗ ≤ 2 T − 1 ( K − 1 ) ( R casc hinge ( π ~ ) − R casc hinge ∗ ) . (20)
This result establishes the Bayes consistency of ℓ c a s c h i n g e \ell_{\mathrm{casc}}^{\mathrm{hinge}} ℓ casc hinge . We detail its proof in Appendix C. Importantly, the calibration constant 2 T − 1 ( K − 1 ) 2^{T-1}(K-1) 2 T − 1 ( K − 1 ) cannot be improved: Proposition C.8 gives a matching lower bound by constructing bad instances for which the smallest valid calibration constant approaches 2 T − 1 ( K − 1 ) 2^{T-1}(K-1) 2 T − 1 ( K − 1 ) . In these instances, every soft deferral decision is a tie between exit and defer, i.e., d ~ t ≡ 0.5 \tilde{d}_{t}\equiv0.5 d ~ t ≡ 0.5 for all t ∈ [ T − 1 ] t\in[T-1] t ∈ [ T − 1 ] . When the soft deferral decisions are more decisive, the constant can be substantially smaller: Proposition C.10 shows that for m ∈ [ 0 , 0.5 ] m\in[0,0.5] m ∈ [ 0 , 0.5 ] , if ∣ d ~ t ( H t ) − 0.5 ∣ ≥ m |\tilde{d}_{t}(H_{t})-0.5|\geq m ∣ d ~ t ( H t ) − 0.5∣ ≥ m almost surely for all t ∈ [ T − 1 ] t\in[T-1] t ∈ [ T − 1 ] , the constant reduces to ( 2 1 + 2 m ) T − 1 ( K − 1 ) \left(\frac{2}{1+2m}\right)^{T-1}(K-1) ( 1 + 2 m 2 ) T − 1 ( K − 1 ) .
Sections 5.1 and 5.3 present experiments that validate Theorems 3.1–3.2, and demonstrate their practical utility.
Improving Routing with Unlabeled Data and PPI
While Section 3 establishes the Bayes consistency of the proposed hinge losses as the sample size n → ∞ n\to\infty n → ∞ , practical training operates in the finite-sample regime. Empirical evidence from prior work [48 , 29 ] , as well as our own experiments (see Figure 7 in Appendix D.3), shows that the classifiers f 1 : T f_{1:T} f 1 : T converge faster than the router r r r . Consequently, r r r can be suboptimal if labeled data is limited, while f 1 : T f_{1:T} f 1 : T may be more reliable. This motivates post-training methods that improve the routing decisions efficiently without retraining, by leveraging unlabeled data.
Figure 7. Classifier and routing-function accuracy during training in the flat router setting with κ = 0.005 \kappa= 0.005 κ = 0.005 . The router is slower to learn compared to the classifiers.
In this regime, the prediction-powered inference (PPI) framework [1 ] provides a natural tool. PPI uses predictions from trained models on abundant unlabeled data to construct confidence intervals (CI) for quantities otherwise hard to estimate reliably from labeled data alone. In our setting, the trained classifiers can provide these predictions, which we use to refine our trained router.
Accordingly, we consider a setting with access to a large unlabeled dataset { X ( i ) , Z 1 : T ( i ) } i = 1 N ∼ i . i . d . P \{X^{(i)}, Z_{1:T}^{(i)}\}_{i=1}^{N} \overset{\mathrm{i.i.d.}}{\sim} P { X ( i ) , Z 1 : T ( i ) } i = 1 N ∼ i.i.d. P in addition to a held-out labeled dataset { X ( i ) , Z 1 : T ( i ) , Y ( i ) } i = 1 l \{X^{(i)}, Z_{1:T}^{(i)}, Y^{(i)}\}_{i=1}^{l} { X ( i ) , Z 1 : T ( i ) , Y ( i ) } i = 1 l where N ≫ l N \gg l N ≫ l , both separate from the data used to train the multi-model classifier. We show how PPI can be used post-training to improve a flat router, and note that the proposed method may be extended to the cascade setting via the recent e-values framework [43 , 21 , 59 , 13 , 28 ] . For notational simplicity, we present our construction and high-probability guarantees pointwise for a fixed input feature x ∈ X x \in\mathcal{X} x ∈ X . In practice, however, test inputs (e.g., prompts or images) rarely coincide with training samples. We therefore implement the method using binning , which groups similar inputs to ensure coverage of previously unseen samples. The pointwise guarantees in the sequel should be interpreted at the bin level.
Oracle router r f l a t † r_{\mathrm{flat}}^\dagger r flat † . Let π † = ( f 1 : T † , r f l a t ) \pi^\dagger= (f_{1:T}^\dagger, r_{\mathrm{flat}}) π † = ( f 1 : T † , r flat ) denote the multi-model classifier after training. Given f 1 : T † f_{1:T}^\dagger f 1 : T † , the Bayes-optimal routing decision for feature x x x is the one that minimizes the conditional 0-1-cost risk given X = x X = x X = x , taking the form
r f l a t † ( x ) ∈ arg min t ∈ [ T ] R t ( x ; f t † ) , (21) r_{\mathrm{flat}}^\dagger(x) \in\arg\min_{t \in[T]} R_t(x; f_t^\dagger), \tag*{(21)} r flat † ( x ) ∈ arg t ∈ [ T ] min R t ( x ; f t † ) , (21)
where, for any classifier f t : X × Z t → [ K ] f_t : \mathcal{X} \times\mathcal{Z}_t \to[K] f t : X × Z t → [ K ] ,
R t ( x ; f t ) : = E [ 1 { f t ( x , Z t ) ≠ Y } ∣ X = x ] + c t . (22) R_t(x; f_t) := \mathbb{E}\left[\mathbf{1}\{f_t(x, Z_t) \ne Y\} \mid X = x\right] + c_t. \tag*{(22)} R t ( x ; f t ) := E [ 1 { f t ( x , Z t ) = Y } ∣ X = x ] + c t . (22)
Since R t ( x ; f t † ) R_t(x; f_t^\dagger) R t ( x ; f t † ) is an oracle population quantity (an expectation over the true data distribution), the routing decision r f l a t † ( x ) r_{\mathrm{flat}}^\dagger(x) r flat † ( x ) is not computable, but provides a natural benchmark. Our goal is not to estimate the oracle directly, but rather to certify whether the trained routing decision can be safely improved using unlabeled data to approach the oracle.
PPI-corrected router r f l a t P P r_{\mathrm{flat}}^{\mathrm{PP}} r flat PP . To this end, we compare each alternative routing decision to the trained baseline by comparing the difference in conditional risks. Let δ t ( x ) : = R r f l a t ( x ) ( x ; f r f l a t ( x ) † ) − R t ( x ; f t † ) \delta_t(x) := R_{r_{\mathrm{flat}}(x)}(x; f_{r_{\mathrm{flat}}(x)}^\dagger) - R_t(x; f_t^\dagger) δ t ( x ) := R r flat ( x ) ( x ; f r flat ( x ) † ) − R t ( x ; f t † ) for t ∈ [ T ] ∖ { r f l a t ( x ) } t \in[T] \setminus\{r_{\mathrm{flat}}(x)\} t ∈ [ T ] ∖ { r flat ( x )} . We leverage PPI to construct the 95% CI around δ t ( x ) \delta_t(x) δ t ( x ) in the form:
P ( ∣ δ t ( x ) − δ t P P ( x ) ∣ ≥ w t , N b , l b ( x ) ) ≤ 0.05 T − 1 , (23) \mathbb{P}\left(\left|\delta_t(x) - \delta_t^{\mathrm{PP}}(x)\right| \ge w_{t,N_b,l_b}(x)\right) \le\frac{0.05}{T-1}, \tag*{(23)} P ( δ t ( x ) − δ t PP ( x ) ≥ w t , N b , l b ( x ) ) ≤ T − 1 0.05 , (23)
where δ t P P ( x ) \delta_t^{\mathrm{PP}}(x) δ t PP ( x ) and w t , N b , l b ( x ) w_{t,N_b,l_b}(x) w t , N b , l b ( x ) have explicit expressions which we detail in Appendix E.1, N b N_b N b and l b l_b l b denote the numbers of unlabeled and labeled points in the bin, and the probability is over the randomness of both the labeled and unlabeled data. Based on these CIs, we identify a set of routing destinations S ( x ) S(x) S ( x ) that are certified, with probability 1 − 0.05 T − 1 1 - \frac{0.05}{T-1} 1 − T − 1 0.05 , to be no worse than the baseline decision r f l a t ( x ) r_{\mathrm{flat}}(x) r flat ( x ) . If S ( x ) S(x) S ( x ) is nonempty, we propose the PPI-corrected routing decision r f l a t P P ( x ) r_{\mathrm{flat}}^{\mathrm{PP}}(x) r flat PP ( x ) by selecting the alternative with the largest certified improvement; otherwise, we retain the baseline.
Formally, we define
S ( x ) : = { t ∈ [ T ] ∖ { r f l a t ( x ) } : δ t P P ( x ) − w t , N b , l b ( x ) ≥ 0 } , S(x) := \left\{t \in[T] \setminus\{r_{\mathrm{flat}}(x)\} : \delta^{\mathrm{PP}}_t(x) - w_{t,N_b,l_b}(x) \ge0\right\}, S ( x ) := { t ∈ [ T ] ∖ { r flat ( x )} : δ t PP ( x ) − w t , N b , l b ( x ) ≥ 0 } ,
r f l a t P P ( x ) : = { arg max t ∈ S ( x ) δ t P P ( x ) − w t , N b , l b ( x ) if S ( x ) ≠ ∅ , r f l a t ( x ) otherwise . (24) r^{\mathrm{PP}}_{\mathrm{flat}}(x) :=
\begin{cases}
\underset{t \in S(x)}{\arg\max}\ \delta^{\mathrm{PP}}_t(x) - w_{t,N_b,l_b}(x) & \text{if } S(x) \ne\varnothing, \\
r_{\mathrm{flat}}(x) & \text{otherwise}.
\end{cases} \tag*{(24)} r flat PP ( x ) := ⎩ ⎨ ⎧ t ∈ S ( x ) arg max δ t PP ( x ) − w t , N b , l b ( x ) r flat ( x ) if S ( x ) = ∅ , otherwise . (24)
Then r f l a t P P r^{\mathrm{PP}}_{\mathrm{flat}} r flat PP is guaranteed to be never worse than the baseline r f l a t r_{\mathrm{flat}} r flat in the sense of (26) below in Theorem 4.1. Under an additional mild separation assumption, where the oracle decision is unique while the gap between the oracle and the next runner-up is strictly positive, i.e.,
γ ( x ) : = δ r f l a t † ( x ) ( x ) − max t ′ ∈ [ T ] ∖ { r f l a t † ( x ) } δ t ′ ( x ) > 0 , (25) \gamma(x) := \delta_{r^{\dagger}_{\mathrm{flat}}(x)}(x) - \max_{t' \in[T]\setminus\{r^{\dagger}_{\mathrm{flat}}(x)\}} \delta_{t'}(x) > 0, \tag*{(25)} γ ( x ) := δ r flat † ( x ) ( x ) − t ′ ∈ [ T ] ∖ { r flat † ( x )} max δ t ′ ( x ) > 0 , (25)
we can obtain the stronger guarantee in (27) . The CI half-width w t , N b , l b ( x ) w_{t,N_b,l_b}(x) w t , N b , l b ( x ) decreases with the accuracy of the classifier used to predict the labels of the N N N unlabeled samples, motivating our use of the most powerful classifier f T † f^\dagger_T f T † among f 1 : T † f^\dagger_{1:T} f 1 : T † in our experiments in § 5.2 . If the imputing classifier f T † f^\dagger_T f T † is accurate, w t , N b , l b ( x ) w_{t,N_b,l_b}(x) w t , N b , l b ( x ) decreases with N N N roughly as 1 / N 1/\sqrt{N} 1/ N , so for large enough N N N , the CI for δ r f l a t † ( x ) ( x ) \delta_{r^\dagger_{\mathrm{flat}}(x)}(x) δ r flat † ( x ) ( x ) becomes narrow enough to separate it from the runner-up, guaranteeing the recovery of the oracle decision: r f l a t P P ( x ) = r f l a t † ( x ) r^{\mathrm{PP}}_{\mathrm{flat}}(x) = r^\dagger_{\mathrm{flat}}(x) r flat PP ( x ) = r flat † ( x ) .
Theorem 4.1 (Certification for PPI-corrected router). For a given x ∈ X x \in\mathcal{X} x ∈ X and a fixed trained multi-model classifier π † = ( f 1 : T † , r f l a t ) \pi^\dagger= (f^\dagger_{1:T}, r_{\mathrm{flat}}) π † = ( f 1 : T † , r flat ) , let r f l a t † ( x ) r^\dagger_{\mathrm{flat}}(x) r flat † ( x ) and r f l a t P P ( x ) r^{\mathrm{PP}}_{\mathrm{flat}}(x) r flat PP ( x ) be as defined in (21) and (24), then
P ( R r f l a t P P ( x ) ( x ; f r f l a t P P ( x ) † ) > R r f l a t ( x ) ( x ; f r f l a t ( x ) † ) ∩ S ( x ) ≠ ∅ ) ≤ 0.05. (26) \mathbb{P}\left(\mathcal{R}_{r^{\mathrm{PP}}_{\mathrm{flat}}(x)}(x;f^\dagger_{r^{\mathrm{PP}}_{\mathrm{flat}}(x)}) > \mathcal{R}_{r_{\mathrm{flat}}(x)}(x;f^\dagger_{r_{\mathrm{flat}}(x)}) \cap S(x) \ne\varnothing\right) \le0.05. \tag*{(26)} P ( R r flat PP ( x ) ( x ; f r flat PP ( x ) † ) > R r flat ( x ) ( x ; f r flat ( x ) † ) ∩ S ( x ) = ∅ ) ≤ 0.05. (26) Moreover, with γ ( x ) \gamma(x) γ ( x ) as defined in (25), then
P ( r f l a t P P ( x ) ≠ r f l a t † ( x ) ∩ S ( x ) ≠ ∅ ∩ w r f l a t † ( x ) , N b , l b ( x ) < γ ( x ) / 2 ) ≤ 0.05. (27) \mathbb{P}\left(r^{\mathrm{PP}}_{\mathrm{flat}}(x) \ne r^\dagger_{\mathrm{flat}}(x) \cap S(x) \ne\varnothing\cap w_{r^\dagger_{\mathrm{flat}}(x),N_b,l_b}(x) < \gamma(x)/2\right) \le0.05. \tag*{(27)} P ( r flat PP ( x ) = r flat † ( x ) ∩ S ( x ) = ∅ ∩ w r flat † ( x ) , N b , l b ( x ) < γ ( x ) /2 ) ≤ 0.05. (27)
The probabilities in (26) and (27) are over the randomness of both the labeled and unlabeled data. The proof is a combination of (23) with a union bound. For sufficiently accurate classifiers f 1 : T † f^\dagger_{1:T} f 1 : T † and sufficiently large N N N , the CIs become sufficiently tight that S ( x ) ≠ ∅ S(x) \ne\varnothing S ( x ) = ∅ with high probability, in which case (26) and (27) effectively reduce to stronger marginal guarantees. We experimentally illustrate the PPI corrections in § 5.2 .
Experiments
Validating the Bayes Consistency of Surrogate Losses on Synthetic Data
To validate Theorems 3.1 and 3.2 in a controlled setting, we study a synthetic classification task with known ground-truth conditional distributions. We compute the Bayes-optimal solutions (as characterized in Lemmas B.1 and C.4) under both the flat and cascade routing models and verify that minimizing our proposed surrogate losses recovers these solutions.
Under both models, we consider T = 3 T=3 T = 3 classifiers for a K = 3 K=3 K = 3 class problem. The data consists of a base feature X ∈ R X \in\mathbb{R} X ∈ R and two additional features Z 2 , Z 3 ∈ R Z_2, Z_3 \in\mathbb{R} Z 2 , Z 3 ∈ R all sampled independently from Uniform [ − 1 , 1 ] \operatorname{Uniform}[-1,1] Uniform [ − 1 , 1 ] , and a label Y ∈ [ K ] Y \in[K] Y ∈ [ K ] generated according to:
Y ∣ ( X = x , Z t = z t ) ∼ Categorical ( θ ( x , z t ) ) , (28) Y \mid(X=x,Z_t=z_t) \sim\operatorname{Categorical}(\theta(x,z_t)), \tag*{(28)} Y ∣ ( X = x , Z t = z t ) ∼ Categorical ( θ ( x , z t )) , (28)
where the full specification of θ \boldsymbol{\theta} θ is provided in Appendix § 10 .§ 1 . We set the cost of f 1 f_{1} f 1 to κ 1 = 0 \kappa_{1}=0 κ 1 = 0 and assign the same cost κ 2 = κ 3 = 0.05 \kappa_{2}=\kappa_{3}=0.05 κ 2 = κ 3 = 0.05 to f 2 f_{2} f 2 and f 3 f_{3} f 3 , yielding cost vector c = [ 0 , 0.05 , 0.05 ] ⊤ \mathbf{c}=[0,0.05,0.05]^{\top} c = [ 0 , 0.05 , 0.05 ] ⊤ in the flat setting and c = [ 0 , 0.05 , 0.1 ] ⊤ \mathbf{c}=[0,0.05,0.1]^{\top} c = [ 0 , 0.05 , 0.1 ] ⊤ in the cascade setting.
The Bayes-optimal routing decisions in both settings are governed by the conditional risk R t ( x ; f t ∗ ) \mathcal{R}_{t}(x;f_{t}^{*}) R t ( x ; f t ∗ ) in (22) ; see Eq. (31) for the flat and (83) for the cascade setting. We choose θ \boldsymbol{\theta} θ so that this conditional risk is minimized at different exit stage t t t over different regions of X \mathcal{X} X ; the solid lines in the top panels of Figure 2 show its Monte-Carlo estimate denoted by R ^ t ( x ; f t ∗ ) \widehat{\mathcal{R}}_{t}(x;f_{t}^{*}) R t ( x ; f t ∗ ) . This induces nontrivial Bayes-optimal routing boundaries, shown by the shaded regions in the top panels. In the flat setting (top left panel), the Bayes-optimal router r ∗ r^{*} r ∗ selects the classifier with the lowest conditional risk, following (31). In the cascade setting (top right panel), the stage-1 Bayes-optimal deferral rule d 1 ∗ d_{1}^{*} d 1 ∗ does not align with the lower envelope of the risk curves, since it must compare the stage-1 conditional risk R ^ 1 ( x ; f 1 ∗ ) \widehat{\mathcal{R}}_{1}(x;f_{1}^{*}) R 1 ( x ; f 1 ∗ ) against the expected cost of optimally continuing the cascade, following (82). Consequently, the left orange-shaded region in the cascade figure is narrower than the green-shaded region in the flat figure, indicating fewer samples are routed to f 3 f_{3} f 3 in the cascade, due to the accumulated cost of deferral.
Figure 2. (Top) Monte-Carlo estimate of the Bayes-optimal conditional risk and Bayes-optimal routing decisions for the flat router r ∗ r^{*} r ∗ (left) and for the first stage of the cascade router d 1 ∗ d_{1}^{*} d 1 ∗ (right). (Bottom) Learned routing decisions on the test set, where points are colored according to the routing probabilities learned by the model.
Having illustrated the Bayes-optimal solutions, we now train flat and cascade multi-model classifiers, and compare the learned functions with their Bayes-optimal counterparts. Each classifier f 1 ( x ) f_{1}(x) f 1 ( x ) , f 2 ( x , z 2 ) f_{2}(x,z_{2}) f 2 ( x , z 2 ) , f 3 ( x , z 3 ) f_{3}(x,z_{3}) f 3 ( x , z 3 ) is implemented as a three-layer multilayer perceptron. For simplicity, the synthetic setup takes Z 1 = ∅ \mathcal{Z}_{1}=\varnothing Z 1 = ∅ , so f 1 f_{1} f 1 and d 1 d_{1} d 1 depend on x x x alone. Routers are implemented as three-layer networks: a multiclass routing network r : X → { 1 , 2 , 3 } r:\mathcal{X}\to\{1,2,3\} r : X → { 1 , 2 , 3 } in the flat setting, and two sequential binary routing networks d 1 : X → { 0 , 1 } d_{1}:\mathcal{X}\to\{0,1\} d 1 : X → { 0 , 1 } and d 2 : X × Z 2 → { 0 , 1 } d_{2}:\mathcal{X}\times\mathcal{Z}_{2}\to\{0,1\} d 2 : X × Z 2 → { 0 , 1 } in the cascade setting. Models are trained end-to-end with Adam using our proposed surrogate losses, i.e., (16) for flat routers and (18) for cascades. Appendix § 10 .§ 1 provides additional architectural and optimization details.
The bottom panels of Figure 2 visualize the learned routing boundaries for both the flat and cascade settings (via the first deferral function d ~ 1 \widetilde{d}_{1} d 1 ), where each test point is colored according to the routing probabilities learned by r ~ \widetilde{r} r or d ~ 1 \widetilde{d}_{1} d 1 . In both cases, the learned regions closely match the Bayes-optimal regions in the top panels, confirming our Bayes consistency results from Theorems 3.1 and 3.2.
Complementing the routing visualizations in Figure 2 , Figure 6 in Appendix § 10 .§ 2 directly compares the empirical 0-1-c c c risk over minibatches against the empirical Bayes-optimal 0-1-c c c risk for both the flat and cascade settings, under two cost configurations. In all cases, minimizing the proposed hinge losses attains the Bayes-optimal risk on the test set, further validating our theory.
Figure 6. Empirical risk of the learned model trained with surrogate loss in the flat case (two left figures) and the cascade case (two right figures), with varying costs per classifier. In all cases, the empirical risk evaluated on the test set of the learned model converges to the empirical risk of the Bayes-optimal solution.
Demonstrating Post-Training PPI-Based Routing Improvement on Synthetic Data
Figure 7 in Appendix § 10 .§ 3 compares the training dynamics of the classifiers and router by plotting their training accuracies over minibatches. The classifiers converge quickly and plateau early, while the router requires much more labeled data to converge. When labeled data is scarce, we now demonstrate how the post-training PPI procedure of § 4 leverages unlabeled data to improve routing decisions at test time.
We use the same synthetic data generating process from § 5.1 with 2000 labeled samples (n = 1000 n = 1000 n = 1000 training samples and l = 1000 l = 1000 l = 1000 PPI labeled samples) and N = 20,000 N = 20{,}000 N = 20 , 000 unlabeled samples, and partition the input space X \mathcal{X} X using K-means clustering (binning) with 40 bins to account for unseen data, as described in § 4 . We consider three classifiers and a flat router, set up identically to those in § 5.1 . The classifiers and router are trained on the training labeled samples only. Post-training, routing decisions at test time are corrected via the PPI procedure of § 4 .
Figure 3 compares three routers: the oracle router r f l a t † r_{\mathrm{flat}}^{\dagger} r flat † , the trained router r f l a t r_{\mathrm{flat}} r flat , and the PPI-corrected router r f l a t P P r_{\mathrm{flat}}^{\mathrm{PP}} r flat PP . While the trained router deviates noticeably from the oracle due to limited labeled data, PPI correction substantially improves it, approaching r f l a t † r_{\mathrm{flat}}^{\dagger} r flat † and fixing 10.65% routing errors at the cost of 1.45% mistakes. Note that Theorem 4.1 guarantees improvement towards the oracle router r f l a t † r_{\mathrm{flat}}^{\dagger} r flat † , not the Bayes-optimal router r f l a t ∗ r_{\mathrm{flat}}^{*} r flat ∗ . However, even if the trained router is poor, the trained classifiers can be closer to their Bayes-optimal counterparts, in which case the oracle router would be near-optimal (r f l a t † ≈ r f l a t ∗ r_{\mathrm{flat}}^{\dagger} \approx r_{\mathrm{flat}}^{*} r flat † ≈ r flat ∗ ). We illustrate this in Appendix § 11 .§ 2 , where we additionally include the Bayes-optimal router and vary the amount of labeled data to show how this affects the behavior of the oracle, the trained router and the PPI correction. Finally, following the theory, the experiment above keeps the training data and the labeled PPI data separate. However, setting labeled data aside from training can be wasteful, so in Appendix § 11 .§ 2 we also show that reusing the training data as the labeled PPI data can still offer improvements, even if it violates the independence assumption of Theorem 4.1 .
Figure 3. Comparison of the oracle router r f l a t † r_{\mathrm{flat}}^{\dagger} r flat † , the router r f l a t r_{\mathrm{flat}} r flat trained on labeled data, and the PPI-corrected router r f l a t P P r_{\mathrm{flat}}^{\mathrm{PP}} r flat PP leveraging unlabeled data. The fixes and errors introduced by PPI are computed with respect to r f l a t † r_{\mathrm{flat}}^{\dagger} r flat † .
Large Language Models Experiment
To showcase the practical effectiveness of our surrogate losses, we conduct experiments in the T = 3 T = 3 T = 3 setting with LLMs from two model families LLaMA 3 1B, 70B, and 405B [15 ] and Qwen 2.5 1.5B, 32B, 72B [66 ] . In this section we show flat router results for LLaMA models. For cascade router results, and Qwen model experiments with flat router and cascade router setups see Appendix G.
Datasets. We use two standard reasoning benchmarks from the literature: CommonsenseQA [52 ] (commonsense logical reasoning) and GSM8K [11 ] (grade-school math reasoning). See Appendix F.1 for more details on the datasets. For all experiments we use a 80-10-10 train-validation-test split.
Setup. We form each input feature x x x by embedding the natural language prompt into a vector. Additional features z t z_t z t are obtained by embedding the LLM outputs in a similar fashion and learn light classifier modules f t f_t f t on top of them. We provide experimental and implementation details in Appendix F.2. Appendix G reports cascade-router results with the LLaMA models and flat-router results with the Qwen models.
Results. Figure 4 presents a flat router experiment with the three LLaMA models. We present three panels, one per cost configuration (in increasing order of costs from left to right with c = [ 0 , 0.01 , 0.02 ] c = [0, 0.01, 0.02] c = [ 0 , 0.01 , 0.02 ] , [ 0 , 0.05 , 0.07 ] [0, 0.05, 0.07] [ 0 , 0.05 , 0.07 ] , [ 0 , 0.1 , 0.2 ] [0, 0.1, 0.2] [ 0 , 0.1 , 0.2 ] ). For each configuration, we display statistics through box-and-whisker plots computed via bootstrap sampling over 10 trials and the median (horizontal line).
Figure 4. Classifier accuracy and routing fractions on GSM8K across three cost configurations (costs increasing from left panel to right panel).
The orange box represents the performance of each model on the entire dataset (independent of the routing decision), the green box represents the performance of each model on the samples that are routed to it, and the blue box (bottom row) represents the fraction of samples that are routed to each model. In configuration 1 (left panel), the gap between the orange and green boxes shows that the router successfully identifies easier samples on which LLaMA 1B is more accurate than average. As costs increase across the three configurations, a larger fraction of questions is routed to the cheapest model, causing this model’s accuracy on the routed subset (green) to decrease as the model receives a larger fraction of harder questions. Meanwhile, LLaMA 70B and 405B achieve accuracies in the nineties on routed samples. Thus, joint classifier-router training under our proposed loss enables cost savings by finding samples where the 1B is accurate. In Appendices G.1–G.4, we provide additional experiments to demonstrate the utility of our proposed training losses in three aspects: (i) they induce consistent cost-accuracy tradeoff behavior, similar to that in Figure 4 , across flat and cascade settings, as well as other datasets and model families (G.1); (ii) they attain the lowest empirical 0-1-c c c risk against popular separate-training and joint-training baselines (G.2 and G.3); and (iii) they scale stably with the number of classifiers T T T (G.4).
Conclusion
We present a principled framework for jointly training classifiers and routers in cost-sensitive multi-model classification. We propose hinge-based surrogate losses and establish their Bayes consistency with respect to the cost-sensitive zero-one loss in both single-stage and cascade routing models. To address label scarcity, we design a post-training PPI procedure that leverages unlabeled data to refine a trained single-stage router. Experiments on synthetic data and LLM benchmarks validate our theory and the utility of the hinge losses.
Our framework has some limitations. In practice, full joint training can be computationally demanding, and the hinge loss can be harder to optimize than a common log-based version. Moreover, our analysis assumes that the distribution of the additional features Z t Z_t Z t is fixed during training, which excludes settings where they are produced by models that are fine-tuned jointly with the classifiers and router. Appendix I discusses these, along with further limitations in more detail.
Acknowledgements and Disclosure of Funding
S. X. Liu and P. Rebeschini were supported by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [grant number EP/Y028333/1]. F. Regol, A. Valkanas and M. Coates were supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) [reference number 260250], and the Fonds de recherche du Québec (FRQNT) [reference number 324302]. Cette recherche a été partiellement financée par le Conseil de recherches en sciences naturelles et en génie du Canada (CRSNG) [numéro de référence 260250] et le Fonds de recherche du Québec (FRQNT) [numéro de référence 324302].
Extended Literature Review
We study multi-model adaptive inference for classification by casting it as a cost-sensitive classification problem with the classifiers carrying varying costs. We propose losses for jointly training the classifiers and their associated routing model. In this section, we survey related work on multi-model classification, both from an empirical and a theoretical perspective.
Section A.1 reviews empirical approaches, which prioritize inference cost savings through the design of architectures and loss functions that tend to lack statistical guarantees. These works can be further categorized into general approaches applicable to any multi-classifier architecture, and domain-specific ones. Since our work is mainly motivated by LLM-related applications, we review LLM domain-specific approaches. Section A.2 covers the theoretical works, which we organize into three main streams: reject-option classification, learning to defer, and joint classifier-router training. Figure 5 summarizes both empirical and theoretical works discussed here.
Figure 5. Key related works on multi-model classification.
Empirical Works
Domain-Agnostic Approaches. Adaptive computation and dynamic networks reduce average test-time inference cost by conditionally allocating computation, e.g., by exiting early in a deep network and escalating to more expensive components only when needed [22 ] . One common approach here is to train classifiers separately from the router and make test-time routing decisions using confidence scores compared against a calibrated threshold [47 , 26 ] .
This is attractive in practice due to its simplicity, as it allows classifiers to be trained independently from the router using standard supervised learning losses.
In contrast to separate training, a growing empirical literature addresses the train-test mismatch caused by this approach, namely that a classifier trained on all samples is applied at test time only to the samples routed to it [67 , 23 , 44 ] . In early-exit networks, where an input leaves at the first exit whose confidence passes a threshold, Yu et al. [ [67 ] train all exits with one summed loss in which each exit is boosted on top of the earlier ones, so that deeper exits concentrate on the samples that earlier exits find hard. Han et al. [ [23 ] keep this exit rule and meta-learn per-sample loss weights for each exit, which encourage each exit to train on the samples it would receive at test time. Kubaty et al. [ [29 ] show that the schedule for training the backbone and the exit heads also changes the cost-accuracy trade-off, and they propose training the backbone before the whole multi-exit network. Other works also learn the routing rule, as Regol et al. [ [44 ] do by training exit gates together with the classifiers. Mixture-of-experts language models with experts of different sizes train a learned gate together with the experts [51 , 60 ] , and Wang et al. [ [60 ] add a loss term that penalizes the use of larger experts.
Two of these procedures alternate between deciding which samples each classifier should handle and retraining the classifiers. Regol et al. [ [44 ] alternate between fitting the gates to the lowest-cost exit of each sample and retraining the classifiers with each loss weighted by the predicted probability of exiting at that classifier, while Han et al. [ [23 ] alternate between assigning samples to exits by simulating confidence-based exiting and updating the classifiers with the resulting weights. Han et al. [ [23 ] also report that training each exit only on the samples allocated to it, without weights, causes a drastic drop in accuracy. However, these works do not provide theoretical guarantees that their joint training procedures yield models achieving the desired cost-accuracy trade-off.
LLM Domain-Specific Approaches. Recent interest in efficient LLM inference has accelerated work on test-time compute allocation, both within a single model and via multi-model orchestration.
Within a single model, works on inference efficiency include early-exit networks and depth-adaptive mechanisms [17 , 68 , 31 , 8 ] . Works on multi-model orchestration include draft-verify strategies (speculative decoding) [63 , 30 ] , one-shot routers that select a model per input [7 , 41 ] , and cascades that escalate through models of increasing cost [6 , 56 ] (Zhang et al. 2024). The closest works to the setting we consider are the latter two categories: LLM routers and cascades.
An LLM router takes as input a query embedding produced by a small, low-cost language model, and uses it to select which LLM to invoke [7 , 41 ] . Cascade approaches [6 , 56 ] (Zhang et al. 2024) prompt LLMs sequentially from cheapest to most expensive, deciding at each step whether to return the response of the current LLM or escalate to a more expensive one. Both the routers and cascades are generally trained on top of fixed, pretrained LLMs. Unlike domain-agnostic approaches, LLM routers and cascades are tailored to LLM-specific signals such as self-consistency scores [40 ] or human feedback [41 ] , which serve as input to these components.
Theoretical Works
While empirical approaches achieve inference cost savings in practice, they often lack rigorous analysis. Alongside the empirical work, a theoretical literature has developed surrogate losses for various cost-sensitive classification problems, providing statistical guarantees such as excess-risk bounds and different notions of surrogate consistency. This literature can be separated into three main threads, each corresponding to a different problem formulation. The first two, reject-option classification and learning to defer, are closely related to the multi-model classification problem and are the most intensively studied. The third thread, more recently developed, addresses the multi-model classification problem itself, so far only with T = 2 T = 2 T = 2 classifiers. Since our work sits in the third thread and the main text develops it in full, we focus on the first two here.
Reject-Option Classification. A reject-option classifier, i.e., a classifier with a reject option, may abstain from predicting at a cost lower than that of a misclassification [10 , 9 ] . The Bayes-optimal reject decision was characterized by Chow [1957, 1970]. Subsequent work developed the consistency theory for it, establishing excess-risk convergence rates for plug-in and empirical risk minimization (ERM) estimators in the binary classification case [25 ] and estimation error bounds that accommodate arbitrary classification-calibrated losses [5 ] . The same abstention mechanism is also studied under the name selective classification [16 , 61 ] and rejection learning [12 , 24 ] .
Learning to Defer (L2D). Learning to defer generalizes abstention: instead of declining prediction outright, the classifier may defer to one or more experts of varying capabilities and costs [33 , 27 , 62 ] . This literature is close in spirit to multi-model classification problem, since a deferral decision routes the input to an alternative predictor. Mozannar and Sontag [2020] proposed a consistent surrogate for multiclass L2D with a single fixed expert, obtained by a reduction to cost-sensitive learning, which yields a softmax cross-entropy loss over the label space augmented with a deferral option. Verma and Nalisnick [2022] proposed an alternative one-vs-all surrogate for the same single-expert setting and proved that it is a consistent surrogate for the deferral loss. Verma et al. [2023] extended both the softmax and the one-vs-all surrogates to multiple experts.
All of these works established Bayes consistency, a guarantee about minimizers over the class of all measurable functions. This guarantee can be less informative when, in practice, the surrogate is minimized over a restricted hypothesis class H \mathcal{H} H (for instance, linear functions). Long and Servedio [2013] illustrated this with linearly separable distributions, on which a Bayes-consistent surrogate minimized over linear functions keeps a positive classification error, while a surrogate that is not Bayes-consistent drives the error to zero. This motivated them to study consistency relative to a hypothesis class: H \mathcal{H} H -consistency and realizable H \mathcal{H} H -consistency. A surrogate is H \mathcal{H} H -consistent if, for every distribution, minimizing the surrogate risk over H \mathcal{H} H yields an error approaching the smallest error attainable within H \mathcal{H} H . This requirement also includes settings in which no function in H \mathcal{H} H attains zero error, either because the underlying distribution is noisy or because H \mathcal{H} H is too restricted. Long and Servedio [2013] therefore introduced the weaker notion of realizable H \mathcal{H} H -consistency, which imposes the same conditions, but only on distributions for which some function in H \mathcal{H} H attains zero error, so that minimizing the surrogate over H \mathcal{H} H must drive the classification error to zero. Awasthi et al. [ [2 ] later introduced H \mathcal{H} H -consistency bounds, which upper-bound the target estimation error (i.e., the gap between the target risk and the smallest target risk attainable within H \mathcal{H} H ) by a function of the surrogate estimation error for a given hypothesis set, and derived explicit bounds for binary classification. These guarantees were stated for classification with the 0 0 0 -1 1 1 loss; the following works on learning to defer adapt them by replacing the classification error with a deferral loss.
In learning to defer with a single expert, Mozannar et al. [ [39 ] showed that prior Bayes-consistent surrogates can fail to find a good linear classifier and deferral function even when a pair with zero error exists, and they proposed a surrogate that is realizable H \mathcal{H} H -consistent for model classes closed under scaling. Mao et al. [ [35 ] showed that a family of surrogates that includes the one of Mozannar et al. [ [39 ] up to a constant is realizable H \mathcal{H} H -consistent under mild conditions and that suitable members admit H \mathcal{H} H -consistency bounds. This also shows that the surrogate of Mozannar et al. [ [39 ] is Bayes-consistent when the expert cost is its classification error, a question that was left open. For multiple pre-defined experts, Mao et al. [ [36 ] proved H \mathcal{H} H -consistency bounds when the predictor and the deferral function are learned simultaneously, and Mao et al. [ [34 ] proved such bounds for a two-stage variant that keeps a pre-trained predictor fixed while the deferral function is learned. In both works the cost of deferring to an expert can add an inference cost to that expert’s error. Mao et al. [ [36 ] also instantiated their surrogate with a constrained hinge loss whose scores sum to zero and obtained an H \mathcal{H} H -consistency bound that is linear in the surrogate estimation error for distributions on which the best surrogate risk within H \mathcal{H} H equals the Bayes surrogate risk. Finally, Mao et al. [ [37 ] proposed surrogates for learning to defer to multiple experts that are realizable H \mathcal{H} H -consistent, admit H \mathcal{H} H -consistency bounds, and are Bayes-consistent, in both the standard and the two-stage settings. To our knowledge, this is the most general consistency analysis of learning to defer.
Joint classifier-router training. L2D assumes the expert(s) are fixed and learns a single classifier together with a deferral rule. Closest to our setting, Regol et al. [ [45 ] and Pugnana et al. [ [42 ] have studied joint training of two classifiers with a cost-sensitive routing rule, but neither extends to the general multi-model setting. Regol et al. [ [45 ] prove Bayes consistency, while Pugnana et al. [ [42 ] prove realizable H \mathcal{H} H -consistency. In contrast, our framework allows an arbitrary number of classifiers with both single-stage and cascade routing, and like Regol et al. [ [45 ] , we establish Bayes consistency.
Proofs for the Flat Router Model
In this section, we prove the calibration result in Theorem 3.1 for the flat-router hinge loss ℓ f l a t h i n g e \ell_{\mathrm{flat}}^{\mathrm{hinge}} ℓ flat hinge . For this, we first prove some auxiliary results in Sections B.1–B.2, and based on these, we prove calibration in Section B.3.
Notation. We use LHS (resp. RHS) as short for left (resp. right) hand side. We write p ( ⋅ ∣ ⋅ ) p(\cdot\mid\cdot) p ( ⋅ ∣ ⋅ ) for the conditional probability mass function of Y Y Y , e.g., p ( y ∣ x ) ≡ Pr ( Y = y ∣ X = x ) p(y \mid x) \equiv\Pr(Y = y \mid X = x) p ( y ∣ x ) ≡ Pr ( Y = y ∣ X = x ) and p ( y ∣ x , z t ) ≡ Pr ( Y = y ∣ X = x , Z t = z t ) p(y \mid x,z_t) \equiv\Pr(Y = y \mid X = x, Z_t = z_t) p ( y ∣ x , z t ) ≡ Pr ( Y = y ∣ X = x , Z t = z t ) .
Bayes-Optimal Solutions
In this section, we characterize the Bayes-optimal solution π f l a t ∗ ∈ Π f l a t \pi_{\mathrm{flat}}^* \in\Pi_{\mathrm{flat}} π flat ∗ ∈ Π flat under the 0-1-c c c loss (Lemma B.1) and the Bayes-optimal solution π ~ f l a t ∗ ∈ Π ~ f l a t \tilde{\pi}_{\mathrm{flat}}^* \in\widetilde{\Pi}_{\mathrm{flat}} π ~ flat ∗ ∈ Π flat under the hinge loss (Lemma B.2). We first characterize the Bayes-optimal solution within the hypothesis class Π f l a t \Pi_{\mathrm{flat}} Π flat under the 0-1-c c c loss associated with the population risk R ( π ) = E [ ℓ ( X , Z 1 : T , Y ; π , c ) ] \mathcal{R}(\pi) = \mathbb{E}[\ell(X,Z_{1:T},Y;\pi,c)] R ( π ) = E [ ℓ ( X , Z 1 : T , Y ; π , c )] .
Lemma B.1 (Bayes-optimal solution in Π f l a t \Pi_{\mathrm{flat}} Π flat for 0-1-c c c loss). For the 0-1-c c c risk R \mathcal{R} R defined in (10) with cost vector c c c , there exists a Bayes-optimal solution
π f l a t ∗ = ( f 1 : T ∗ , r f l a t ∗ ) ∈ arg min π ∈ Π f l a t R ( π ) (29) \pi_{\mathrm{flat}}^* = (f_{1:T}^*,r_{\mathrm{flat}}^*) \in\underset{\pi\in\Pi_{\mathrm{flat}}}{\arg\min}\mathcal{R}(\pi) \tag*{(29)} π flat ∗ = ( f 1 : T ∗ , r flat ∗ ) ∈ π ∈ Π flat arg min R ( π ) (29) whose inducing components ( f 1 : T ∗ , r f l a t ∗ ) (f_{1:T}^*,r_{\mathrm{flat}}^*) ( f 1 : T ∗ , r flat ∗ ) satisfy:
f t ∗ ( x , z t ) ∈ arg max y ∈ [ K ] p ( y ∣ x , z t ) ∀ t ∈ [ T ] , (30) f_t^*(x,z_t) \in\underset{y \in[K]}{\arg\max}\ p(y \mid x,z_t) \quad\forall\ t \in[T], \tag*{(30)} f t ∗ ( x , z t ) ∈ y ∈ [ K ] arg max p ( y ∣ x , z t ) ∀ t ∈ [ T ] , (30) r f l a t ∗ ( x ) ∈ arg min t ∈ [ T ] R t ( x ; f t ∗ ) , (31) r_{\mathrm{flat}}^*(x) \in\underset{t \in[T]}{\arg\min}\ \mathcal{R}_t(x;f_t^*), \tag*{(31)} r flat ∗ ( x ) ∈ t ∈ [ T ] arg min R t ( x ; f t ∗ ) , (31) where R t ( x ; f t ∗ ) \mathcal{R}_t(x;f_t^*) R t ( x ; f t ∗ ) denotes the flat stage-wise Bayes risk of f t ∗ f_t^* f t ∗ conditioned on X = x X=x X = x :
R t ( x ; f t ∗ ) = 1 − E [ max y ∈ [ K ] p ( y ∣ x , Z t ) ∣ X = x ] + c t ∀ t ∈ [ T ] . (32) \mathcal{R}_t(x;f_t^*) = 1 - \mathbb{E}\left[\underset{y \in[K]}{\max}\ p(y \mid x,Z_t) \mathrel{\big|} X=x\right] + c_t \quad\forall\ t \in[T]. \tag*{(32)} R t ( x ; f t ∗ ) = 1 − E [ y ∈ [ K ] max p ( y ∣ x , Z t ) X = x ] + c t ∀ t ∈ [ T ] . (32)
Proof. We first prove the Bayes-optimal solution for the classifiers f 1 : T ∗ f_{1:T}^* f 1 : T ∗ then the routing function r f l a t ∗ r_{\mathrm{flat}}^* r flat ∗ .
Part 1: Proof for the Bayes-optimal f 1 : T ∗ f_{1:T}^* f 1 : T ∗ . We first minimize R ( π ) ≡ R ( π ; c ) \mathcal{R}(\pi) \equiv\mathcal{R}(\pi;c) R ( π ) ≡ R ( π ; c ) with respect to f 1 : T f_{1:T} f 1 : T for an arbitrary routing rule r f l a t : X → [ T ] r_{\mathrm{flat}}:\mathcal{X}\to[T] r flat : X → [ T ] . The 0-1-c c c risk can be written as
R ( π ) = E [ ℓ ( X , Z 1 : T , Y ; π , c ) ] = E [ 1 { f r f l a t ( X ) ( X , Z r f l a t ( X ) ) ≠ Y } + c r f l a t ( X ) ] = ∑ t = 1 T E [ 1 { r f l a t ( X ) = t } ⋅ ( 1 { f t ( X , Z t ) ≠ Y } + c t ) ] = ∑ t = 1 T E [ 1 { r f l a t ( X ) = t } ⋅ ( E [ 1 { f t ( X , Z t ) ≠ Y } ∣ X ] + c t ) ] , (33) \begin{aligned}
\mathcal{R}(\pi)
&= \mathbb{E}\left[\ell(X,Z_{1:T},Y;\pi,c)\right] \\
&= \mathbb{E}\left[\mathbb{1}\{f_{r_{\mathrm{flat}}(X)}(X,Z_{r_{\mathrm{flat}}(X)})\ne Y\}+c_{r_{\mathrm{flat}}(X)}\right] \\
&= \sum_{t=1}^{T}\mathbb{E}\left[\mathbb{1}\{r_{\mathrm{flat}}(X)=t\}\cdot\left(\mathbb{1}\{f_t(X,Z_t)\ne Y\}+c_t\right)\right] \\
&= \sum_{t=1}^{T}\mathbb{E}\left[\mathbb{1}\{r_{\mathrm{flat}}(X)=t\}\cdot\left(\mathbb{E}\left[\mathbb{1}\{f_t(X,Z_t)\ne Y\}\mid X\right]+c_t\right)\right], \tag*{(33)}
\end{aligned} R ( π ) = E [ ℓ ( X , Z 1 : T , Y ; π , c ) ] = E [ 1 { f r flat ( X ) ( X , Z r flat ( X ) ) = Y } + c r flat ( X ) ] = t = 1 ∑ T E [ 1 { r flat ( X ) = t } ⋅ ( 1 { f t ( X , Z t ) = Y } + c t ) ] = t = 1 ∑ T E [ 1 { r flat ( X ) = t } ⋅ ( E [ 1 { f t ( X , Z t ) = Y } ∣ X ] + c t ) ] , (33)
where the last step applies the tower property of conditional expectation, and R t ( X ; f t ) \mathcal{R}_t(X; f_t) R t ( X ; f t ) denotes the conditional risk of the t t t -th classifier given X X X ; the semicolon here emphasizes that X X X is a variable and f t f_t f t is treated as a parameter.
Since c t c_t c t is a constant that does not depend on f 1 : T f_{1:T} f 1 : T , minimizing R ( π ) \mathcal{R}(\pi) R ( π ) with respect to f t f_t f t reduces to minimizing R t ( X ; f t ) \mathcal{R}_t(X; f_t) R t ( X ; f t ) pointwise in X X X . Fixing x ∈ X x \in\mathcal{X} x ∈ X , by the tower property, we have
R t ( x ; f t ) : = E [ 1 { f t ( X , Z t ) ≠ Y } ∣ X = x ] + c t = E [ E [ 1 { f t ( X , Z t ) ≠ Y } ∣ X = x , Z t ] ∣ X = x ] + c t = E [ P ( Y ≠ f t ( x , Z t ) ∣ X = x , Z t ) ∣ X = x ] + c t = 1 − E [ p ( f t ( x , Z t ) ∣ x , Z t ) ∣ X = x ] + c t , (34) \begin{aligned}
\mathcal{R}_t(x; f_t) &:= \mathbb{E}\left[\mathbf{1}\{f_t(X,Z_t) \ne Y\}\mid X=x\right] + c_t \\
&= \mathbb{E}\left[\mathbb{E}\left[\mathbf{1}\{f_t(X,Z_t) \ne Y\}\mid X=x,Z_t\right]\mid X=x\right] + c_t \\
&= \mathbb{E}\left[\mathbb{P}(Y \ne f_t(x,Z_t)\mid X=x,Z_t)\mid X=x\right] + c_t \\
&= 1-\mathbb{E}\left[p(f_t(x,Z_t)\mid x,Z_t)\mid X=x\right] + c_t, \tag*{(34)}
\end{aligned} R t ( x ; f t ) := E [ 1 { f t ( X , Z t ) = Y } ∣ X = x ] + c t = E [ E [ 1 { f t ( X , Z t ) = Y } ∣ X = x , Z t ] ∣ X = x ] + c t = E [ P ( Y = f t ( x , Z t ) ∣ X = x , Z t ) ∣ X = x ] + c t = 1 − E [ p ( f t ( x , Z t ) ∣ x , Z t ) ∣ X = x ] + c t , (34)
where p ( ⋅ ∣ x , Z t ) p(\cdot\mid x,Z_t) p ( ⋅ ∣ x , Z t ) denotes the conditional probability of Y Y Y given X = x X=x X = x and Z t Z_t Z t .
Therefore, minimizing R t ( x ; f t ) \mathcal{R}_t(x; f_t) R t ( x ; f t ) with respect to f t f_t f t reduces to maximizing p ( f t ( x , z t ) ∣ x , z t ) p(f_t(x,z_t)\mid x,z_t) p ( f t ( x , z t ) ∣ x , z t ) for each ( x , z t ) ∈ X × Z t (x,z_t)\in\mathcal{X}\times\mathcal{Z}_t ( x , z t ) ∈ X × Z t , so pointwise in ( x , z t ) ∈ X × Z t (x,z_t)\in\mathcal{X}\times\mathcal{Z}_t ( x , z t ) ∈ X × Z t , we have
f t ∗ ( x , z t ) ∈ arg max y ∈ [ K ] p ( y ∣ x , z t ) , (35) f_t^*(x,z_t)\in\underset{y\in[K]}{\arg\max}\ p(y\mid x,z_t), \tag*{(35)} f t ∗ ( x , z t ) ∈ y ∈ [ K ] arg max p ( y ∣ x , z t ) , (35)
which proves (30).
Part 2: Proof for the Bayes-optimal r f l a t ∗ r_{\mathrm{flat}}^* r flat ∗ . We now minimize R ( π ) ≡ R ( π ; c ) \mathcal{R}(\pi)\equiv\mathcal{R}(\pi;c) R ( π ) ≡ R ( π ; c ) with respect to r f l a t r_{\mathrm{flat}} r flat holding f 1 : T = f 1 : T ∗ f_{1:T}=f_{1:T}^* f 1 : T = f 1 : T ∗ fixed. Substituting (35) into (34), the conditional Bayes risk of the t t t -th classifier is
R t ( x ; f t ∗ ) : = E [ 1 { f t ∗ ( X , Z t ) ≠ Y } ∣ X = x ] + c t = 1 − E [ max y ∈ [ K ] p ( y ∣ x , Z t ) ∣ X = x ] + c t , (36) \begin{aligned}
\mathcal{R}_t(x;f_t^*) &:= \mathbb{E}\left[\mathbf{1}\{f_t^*(X,Z_t)\ne Y\}\mid X=x\right]+c_t \\
&=1-\mathbb{E}\left[\max_{y\in[K]}p(y\mid x,Z_t)\mid X=x\right]+c_t, \tag*{(36)}
\end{aligned} R t ( x ; f t ∗ ) := E [ 1 { f t ∗ ( X , Z t ) = Y } ∣ X = x ] + c t = 1 − E [ y ∈ [ K ] max p ( y ∣ x , Z t ) ∣ X = x ] + c t , (36)
yielding (32). Since π = ( f 1 : T ∗ , r f l a t ) \pi=(f_{1:T}^*,r_{\mathrm{flat}}) π = ( f 1 : T ∗ , r flat ) here, we write R ( f 1 : T ∗ , r f l a t ) \mathcal{R}(f_{1:T}^*,r_{\mathrm{flat}}) R ( f 1 : T ∗ , r flat ) to mean R ( π ) \mathcal{R}(\pi) R ( π ) . From (33), the unconditional risk is R ( f 1 : T ∗ , r f l a t ) = ∑ t = 1 T E [ 1 { r f l a t ( X ) = t } ⋅ R t ( X ; f t ∗ ) ] \mathcal{R}(f_{1:T}^*,r_{\mathrm{flat}})=\sum_{t=1}^{T}\mathbb{E}[\mathbf{1}\{r_{\mathrm{flat}}(X)=t\}\cdot\mathcal{R}_t(X;f_t^*)] R ( f 1 : T ∗ , r flat ) = ∑ t = 1 T E [ 1 { r flat ( X ) = t } ⋅ R t ( X ; f t ∗ )] , so minimizing R ( f 1 : T ∗ , r f l a t ) \mathcal{R}(f_{1:T}^*,r_{\mathrm{flat}}) R ( f 1 : T ∗ , r flat ) with respect to r f l a t r_{\mathrm{flat}} r flat reduces to a pointwise problem in X X X . For each x ∈ X x\in\mathcal{X} x ∈ X , the optimal routing function selects the classifier with the minimum conditional risk, that is,
r f l a t ∗ ( x ) ∈ arg min t ∈ [ T ] R t ( x ; f t ∗ ) , (37) r_{\mathrm{flat}}^*(x)\in\underset{t\in[T]}{\arg\min}\ \mathcal{R}_t(x;f_t^*), \tag*{(37)} r flat ∗ ( x ) ∈ t ∈ [ T ] arg min R t ( x ; f t ∗ ) , (37)
which proves (31).
We now characterize the Bayes-optimal solution within the hypothesis class Π ~ f l a t \widetilde{\Pi}_{\mathrm{flat}} Π flat under the surrogate hinge loss ℓ f l a t h i n g e \ell_{\mathrm{flat}}^{\mathrm{hinge}} ℓ flat hinge associated with the population risk R f l a t h i n g e ( π ~ ) = E [ ℓ f l a t h i n g e ( X , Z 1 : T , Y ; π ~ , c ) ] \mathcal{R}_{\mathrm{flat}}^{\mathrm{hinge}}(\widetilde{\pi})=\mathbb{E}[\ell_{\mathrm{flat}}^{\mathrm{hinge}}(X,Z_{1:T},Y;\widetilde{\pi},c)] R flat hinge ( π ) = E [ ℓ flat hinge ( X , Z 1 : T , Y ; π , c )] .
Lemma B.2 (Bayes-optimal solution in Π ~ f l a t \widetilde{\Pi}_{\mathrm{flat}} Π flat for hinge loss). There exists a Bayes-optimal solution
π ~ f l a t ∗ = ( f ~ 1 : T ∗ , r ~ f l a t ∗ ) ∈ arg min π ~ ∈ Π ~ f l a t R f l a t h i n g e ( π ~ ) , \widetilde{\pi}_{\mathrm{flat}}^*=(\widetilde{f}_{1:T}^*,\widetilde{r}_{\mathrm{flat}}^*)\in\underset{\widetilde{\pi}\in\widetilde{\Pi}_{\mathrm{flat}}}{\arg\min}\ \mathcal{R}_{\mathrm{flat}}^{\mathrm{hinge}}(\widetilde{\pi}), π flat ∗ = ( f 1 : T ∗ , r flat ∗ ) ∈ π ∈ Π flat arg min R flat hinge ( π ) ,
where, with f t ∗ f_t^* f t ∗ defined as in (30),
[ f ~ t ∗ ( x , z t ) ] k = { 1 for k = f t ∗ ( x , z t ) − 1 K − 1 otherwise ∀ t ∈ [ T ] , (38) \left[\tilde{f}_t^*(x,z_t)\right]_k =
\begin{cases}
1 & \text{for } k=f_t^*(x,z_t)\\
-\frac{1}{K-1} & \text{otherwise}
\end{cases}
\qquad\forall t\in[T], \tag*{(38)} [ f ~ t ∗ ( x , z t ) ] k = { 1 − K − 1 1 for k = f t ∗ ( x , z t ) otherwise ∀ t ∈ [ T ] , (38)
and
r ~ f l a t ∗ ( x ) ∈ { v ∈ Δ T − 1 : v k = 0 ∀ k ∉ arg min t ∈ [ T ] R t ( x ; f t ∗ ) } . (39) \tilde{r}_{\mathrm{flat}}^*(x)\in\left\{v\in\Delta^{T-1}:v_k=0\quad\forall k\notin\underset{t\in[T]}{\operatorname{arg\,min}}\mathcal{R}_t(x;f_t^*)\right\}. \tag*{(39)} r ~ flat ∗ ( x ) ∈ { v ∈ Δ T − 1 : v k = 0 ∀ k ∈ / t ∈ [ T ] arg min R t ( x ; f t ∗ ) } . (39)
Proof. We prove (38) and (39) one at a time.
Part 1: Proof for the Bayes-optimal f ~ 1 : T ∗ \tilde{f}_{1:T}^* f ~ 1 : T ∗ . The result in (38) directly follows from [53 Lemma 1] .
Part 2: Proof for the Bayes-optimal r ~ f l a t ∗ \tilde{r}_{\mathrm{flat}}^* r ~ flat ∗ . We now minimize R f l a t h i n g e ( π ~ ) \mathcal{R}_{\mathrm{flat}}^{\mathrm{hinge}}(\tilde{\pi}) R flat hinge ( π ~ ) w.r.t. r ~ f l a t \tilde{r}_{\mathrm{flat}} r ~ flat holding f ~ 1 : T = f ~ 1 : T ∗ \tilde{f}_{1:T}=\tilde{f}_{1:T}^* f ~ 1 : T = f ~ 1 : T ∗ fixed. By definition the overall flat-router hinge risk is
R f l a t h i n g e ( π ~ ) = E [ ∑ t = 1 T [ r ~ f l a t ( X ) ] t ⋅ ℓ h i n g e ( f ~ t ( X , Z t ) , Y ; c t ) ] . \mathcal{R}_{\mathrm{flat}}^{\mathrm{hinge}}(\tilde{\pi})=\mathbb{E}\left[\sum_{t=1}^{T}[\tilde{r}_{\mathrm{flat}}(X)]_t\cdot\ell^{\mathrm{hinge}}(\tilde{f}_t(X,Z_t),Y;c_t)\right]. R flat hinge ( π ~ ) = E [ t = 1 ∑ T [ r ~ flat ( X ) ] t ⋅ ℓ hinge ( f ~ t ( X , Z t ) , Y ; c t ) ] .
Using f ~ 1 : T = f ~ 1 : T ∗ \tilde{f}_{1:T}=\tilde{f}_{1:T}^* f ~ 1 : T = f ~ 1 : T ∗ and writing π ~ = ( f ~ 1 : T ∗ , r ~ f l a t ) \tilde{\pi}=(\tilde{f}_{1:T}^*,\tilde{r}_{\mathrm{flat}}) π ~ = ( f ~ 1 : T ∗ , r ~ flat ) , we have via Lemma B.3 that
R t h i n g e ( x , z t ; f ~ t ∗ ) : = E [ ℓ h i n g e ( f ~ t ∗ ( X , Z t ) , Y ; c t ) ∣ X = x , Z t = z t ] = K K − 1 R t ( x , z t ; f t ∗ ) , (40) \mathcal{R}_t^{\mathrm{hinge}}(x,z_t;\tilde{f}_t^*):=\mathbb{E}\left[\ell^{\mathrm{hinge}}(\tilde{f}_t^*(X,Z_t),Y;c_t)\mid X=x,Z_t=z_t\right]
=\frac{K}{K-1}\mathcal{R}_t(x,z_t;f_t^*), \tag*{(40)} R t hinge ( x , z t ; f ~ t ∗ ) := E [ ℓ hinge ( f ~ t ∗ ( X , Z t ) , Y ; c t ) ∣ X = x , Z t = z t ] = K − 1 K R t ( x , z t ; f t ∗ ) , (40)
which implies that the overall hinge risk conditioned on X = x X=x X = x can be written as
R h i n g e ( x ; π ~ ) : = ∑ t = 1 T [ r ~ f l a t ( x ) ] t ⋅ E [ R t h i n g e ( x , Z t ; f ~ t ∗ ) ∣ X = x ] = K K − 1 ∑ t = 1 T [ r ~ f l a t ( x ) ] t ⋅ E [ R t ( x , Z t ; f t ∗ ) ∣ X = x ] = K K − 1 ∑ t = 1 T [ r ~ f l a t ( x ) ] t ⋅ R t ( x ; f t ∗ ) , (41) \begin{aligned}
\mathcal{R}^{\mathrm{hinge}}(x;\tilde{\pi})
&:=\sum_{t=1}^{T}[\tilde{r}_{\mathrm{flat}}(x)]_t\cdot\mathbb{E}\left[\mathcal{R}_t^{\mathrm{hinge}}(x,Z_t;\tilde{f}_t^*)\mid X=x\right]\\
&=\frac{K}{K-1}\sum_{t=1}^{T}[\tilde{r}_{\mathrm{flat}}(x)]_t\cdot\mathbb{E}\left[\mathcal{R}_t(x,Z_t;f_t^*)\mid X=x\right]\\
&=\frac{K}{K-1}\sum_{t=1}^{T}[\tilde{r}_{\mathrm{flat}}(x)]_t\cdot\mathcal{R}_t(x;f_t^*), \tag*{(41)}
\end{aligned} R hinge ( x ; π ~ ) := t = 1 ∑ T [ r ~ flat ( x ) ] t ⋅ E [ R t hinge ( x , Z t ; f ~ t ∗ ) ∣ X = x ] = K − 1 K t = 1 ∑ T [ r ~ flat ( x ) ] t ⋅ E [ R t ( x , Z t ; f t ∗ ) ∣ X = x ] = K − 1 K t = 1 ∑ T [ r ~ flat ( x ) ] t ⋅ R t ( x ; f t ∗ ) , (41)
where the last step holds by definition of R t ( x ; f t ∗ ) \mathcal{R}_t(x;f_t^*) R t ( x ; f t ∗ ) in (32). Therefore any minimizer r ~ f l a t ∗ ( x ) ∈ Δ T − 1 \tilde{r}_{\mathrm{flat}}^*(x)\in\Delta^{T-1} r ~ flat ∗ ( x ) ∈ Δ T − 1 of R h i n g e ( x ; π ~ ) \mathcal{R}^{\mathrm{hinge}}(x;\tilde{\pi}) R hinge ( x ; π ~ ) must be supported on the set arg min t ∈ [ T ] R t ( x ; f t ∗ ) \operatorname{arg\,min}_{t\in[T]}\mathcal{R}_t(x;f_t^*) arg min t ∈ [ T ] R t ( x ; f t ∗ ) , establishing (39). In particular, any one-hot vector concentrated on any element of this set is a valid choice. □ \square □
Relationship Between 0-1-c and Hinge Conditional Risks
Using the characterization of the Bayes-optimal solutions from Lemmas B.1–B.2, the next two lemmas connect the hinge and 0-1-c conditional Bayes risks, and the hinge and 0-1-c excess risks, respectively, which will be handy for our calibration proof in the next section.
Lemma B.3 (Hinge and 0-1-c c c conditional Bayes risks differ by a factor of K K − 1 \frac{K}{K-1} K − 1 K ). For t ∈ [ T ] t \in[T] t ∈ [ T ] , let f t ∗ f_t^* f t ∗ and f ~ t ∗ \tilde{f}_t^* f ~ t ∗ denote the Bayes-optimal classifiers from Lemmas B.1–B.2 under the 0-1-c c c loss and the hinge loss, respectively. Define the conditional Bayes risks
R t ( x , z t ; f t ∗ ) : = E [ 1 { f t ∗ ( X , Z t ) ≠ Y } + c t ∣ X = x , Z t = z t ] , (42) \mathcal{R}_t(x,z_t;f_t^*) := \mathbb{E}\left[\mathbb{1}\{f_t^*(X,Z_t) \ne Y\}+c_t \mid X=x,Z_t=z_t\right], \tag*{(42)} R t ( x , z t ; f t ∗ ) := E [ 1 { f t ∗ ( X , Z t ) = Y } + c t ∣ X = x , Z t = z t ] , (42) R t h i n g e ( x , z t ; f ~ t ∗ ) : = E [ ℓ h i n g e ( f ~ t ∗ ( X , Z t ) , Y ; c t ) ∣ X = x , Z t = z t ] , (43) \mathcal{R}_t^{\mathrm{hinge}}(x,z_t;\tilde{f}_t^*) := \mathbb{E}\left[\ell^{\mathrm{hinge}}(\tilde{f}_t^*(X,Z_t),Y;c_t) \mid X=x,Z_t=z_t\right], \tag*{(43)} R t hinge ( x , z t ; f ~ t ∗ ) := E [ ℓ hinge ( f ~ t ∗ ( X , Z t ) , Y ; c t ) ∣ X = x , Z t = z t ] , (43) then
R t h i n g e ( x , z t ; f ~ t ∗ ) = K K − 1 R t ( x , z t ; f t ∗ ) . (44) \mathcal{R}_t^{\mathrm{hinge}}(x,z_t;\tilde{f}_t^*)=\frac{K}{K-1}\mathcal{R}_t(x,z_t;f_t^*). \tag*{(44)} R t hinge ( x , z t ; f ~ t ∗ ) = K − 1 K R t ( x , z t ; f t ∗ ) . (44) Proof. We can write
R t h i n g e ( x , z t ; f ~ t ∗ ) : = E [ ℓ h i n g e ( f ~ t ∗ ( X , Z t ) , Y ; c t ) ∣ X = x , Z t = z t ] = ∑ y ∈ [ K ] p ( y ∣ x , z t ) ℓ h i n g e ( f ~ t ∗ ( x , z t ) , y ; c t ) = p ( f t ∗ ( x , z t ) ∣ x , z t ) K K − 1 c t + ( 1 − p ( f t ∗ ( x , z t ) ∣ x , z t ) ) K K − 1 ( 1 + c t ) = K K − 1 ( 1 − p ( f t ∗ ( x , z t ) ∣ x , z t ) + c t ) = K K − 1 ( 1 − max y ∈ [ K ] p ( y ∣ x , z t ) + c t ) = K K − 1 R t ( x , z t ; f t ∗ ) . (45) \begin{aligned}
\mathcal{R}_t^{\mathrm{hinge}}(x,z_t;\tilde{f}_t^*) &:= \mathbb{E}\left[\ell^{\mathrm{hinge}}(\tilde{f}_t^*(X,Z_t),Y;c_t)\mid X=x,Z_t=z_t\right] \\
&= \sum_{y\in[K]}p(y\mid x,z_t)\ell^{\mathrm{hinge}}(\tilde{f}_t^*(x,z_t),y;c_t) \\
&=p(f_t^*(x,z_t)\mid x,z_t)\frac{K}{K-1}c_t+\left(1-p(f_t^*(x,z_t)\mid x,z_t)\right)\frac{K}{K-1}(1+c_t) \\
&=\frac{K}{K-1}\left(1-p(f_t^*(x,z_t)\mid x,z_t)+c_t\right) \\
&=\frac{K}{K-1}\left(1-\max_{y\in[K]}p(y\mid x,z_t)+c_t\right)=\frac{K}{K-1}\mathcal{R}_t(x,z_t;f_t^*). \tag*{(45)}
\end{aligned} R t hinge ( x , z t ; f ~ t ∗ ) := E [ ℓ hinge ( f ~ t ∗ ( X , Z t ) , Y ; c t ) ∣ X = x , Z t = z t ] = y ∈ [ K ] ∑ p ( y ∣ x , z t ) ℓ hinge ( f ~ t ∗ ( x , z t ) , y ; c t ) = p ( f t ∗ ( x , z t ) ∣ x , z t ) K − 1 K c t + ( 1 − p ( f t ∗ ( x , z t ) ∣ x , z t ) ) K − 1 K ( 1 + c t ) = K − 1 K ( 1 − p ( f t ∗ ( x , z t ) ∣ x , z t ) + c t ) = K − 1 K ( 1 − y ∈ [ K ] max p ( y ∣ x , z t ) + c t ) = K − 1 K R t ( x , z t ; f t ∗ ) . (45)
Lemma B.4 (Calibration of the stage-wise cost-sensitive hinge loss ℓ h i n g e \ell^{\mathrm{hinge}} ℓ hinge ). Let f ~ t \tilde{f}_t f ~ t be a soft classifier as defined in (11) and f t f_t f t its induced hard classifier in (12). Define the conditional risks R t ( x , z t ; f t ) \mathcal{R}_t(x,z_t;f_t) R t ( x , z t ; f t ) and R t h i n g e ( x , z t ; f ~ t ) \mathcal{R}_t^{\mathrm{hinge}}(x,z_t;\tilde{f}_t) R t hinge ( x , z t ; f ~ t ) analogously to (42)–(43). Then, for all t ∈ [ T ] t\in[T] t ∈ [ T ] and ( x , z t ) ∈ X × Z t (x,z_t)\in\mathcal{X}\times\mathcal{Z}_t ( x , z t ) ∈ X × Z t ,
R t ( x , z t ; f t ) − R t ( x , z t ; f t ∗ ) ≤ ( K − 1 ) ( R t h i n g e ( x , z t ; f ~ t ) − R t h i n g e ( x , z t ; f ~ t ∗ ) ) . (46) \mathcal{R}_t(x,z_t;f_t)-\mathcal{R}_t(x,z_t;f_t^*)\leq(K-1)\left(\mathcal{R}_t^{\mathrm{hinge}}(x,z_t;\tilde{f}_t)-\mathcal{R}_t^{\mathrm{hinge}}(x,z_t;\tilde{f}_t^*)\right). \tag*{(46)} R t ( x , z t ; f t ) − R t ( x , z t ; f t ∗ ) ≤ ( K − 1 ) ( R t hinge ( x , z t ; f ~ t ) − R t hinge ( x , z t ; f ~ t ∗ ) ) . (46)
As an immediate remark, since (46) holds for every z t z_t z t , we can trivially obtain the same bound for the risk conditioned on X = x X=x X = x only. Likewise, in our proofs for the cascade model in Appendix § 9 , (46) also holds with ( x , z t ) (x,z_t) ( x , z t ) replaced by h t h_t h t , since p ( y ∣ h t ) = p ( y ∣ x , z t ) p(y\mid h_t)=p(y\mid x,z_t) p ( y ∣ h t ) = p ( y ∣ x , z t ) under the conditional independence assumption (5) and both f t f_t f t and f ~ t \tilde{f}_t f ~ t depend on h t h_t h t only through ( x , z t ) (x,z_t) ( x , z t ) .
Proof. We begin by setting up some notation. Fix ( x , z t ) (x,z_t) ( x , z t ) and let p y : = p ( y ∣ x , z t ) p_y:=p(y\mid x,z_t) p y := p ( y ∣ x , z t ) . Let v : = f ~ t ( x , z t ) ∈ R K v:=\tilde{f}_t(x,z_t)\in\mathbb{R}^K v := f ~ t ( x , z t ) ∈ R K be the score vector that f ~ t \tilde{f}_t f ~ t outputs and j : = f t ( x , z t ) ∈ [ K ] j:=f_t(x,z_t)\in[K] j := f t ( x , z t ) ∈ [ K ] the class index that f t f_t f t outputs. By the argmax map in (12), we have j ∈ arg max k ∈ [ K ] v k j\in\arg\max_{k\in[K]}v_k j ∈ arg max k ∈ [ K ] v k . Using the definition of the hinge loss ℓ h i n g e \ell^{\mathrm{hinge}} ℓ hinge in (15), we define its cost-free variant ℓ ˉ h i n g e \bar{\ell}^{\mathrm{hinge}} ℓ ˉ hinge as
ℓ ˉ h i n g e ( v , y ) : = ℓ h i n g e ( v , y ; 0 ) = ∑ k ∈ [ K ] ∖ y [ v k + 1 K − 1 ] + . (47) \bar{\ell}^{\mathrm{hinge}}(v,y):=\ell^{\mathrm{hinge}}(v,y;0)=\sum_{k\in[K]\setminus y}\left[v_k+\frac{1}{K-1}\right]_+. \tag*{(47)} ℓ ˉ hinge ( v , y ) := ℓ hinge ( v , y ; 0 ) = k ∈ [ K ] ∖ y ∑ [ v k + K − 1 1 ] + . (47)
We further define two useful shorthand: L ( v ) L(v) L ( v ) as the associated conditional risk, and U U U as the total penalty:
L ( v ) : = E [ ℓ ˉ h i n g e ( v , Y ) ∣ X = x , Z t = z t ] = ∑ y ∈ [ K ] p y ℓ ˉ h i n g e ( v , y ) , (48) L(v):=\mathbb{E}\left[\bar{\ell}^{\mathrm{hinge}}(v,Y)\mid X=x,Z_t=z_t\right]=\sum_{y\in[K]}p_y\bar{\ell}^{\mathrm{hinge}}(v,y), \tag*{(48)} L ( v ) := E [ ℓ ˉ hinge ( v , Y ) ∣ X = x , Z t = z t ] = y ∈ [ K ] ∑ p y ℓ ˉ hinge ( v , y ) , (48)
U : = ∑ k ∈ [ K ] [ v k + 1 K − 1 ] + . (49) U:=\sum_{k\in[K]}\left[v_k+\frac{1}{K-1}\right]_+. \tag*{(49)} U := k ∈ [ K ] ∑ [ v k + K − 1 1 ] + . (49)
We first express both excess risks in (46) in terms of p y p_y p y and L ( v ) L(v) L ( v ) , then derive a lower bound for the conditional expectation L ( v ) L(v) L ( v ) to obtain the final inequality.
Part 1. Rewriting the stage-wise excess risks: By definition, we can write
R t h i n g e ( x , z t ; f ~ t ) : = E [ ℓ h i n g e ( f ~ t ( X , Z t ) , Y ; c t ) ∣ X = x , Z t = z t ] = ∑ y ∈ [ K ] p y ℓ h i n g e ( v , y ; c t ) = L ( v ) + K K − 1 c t . (50) \begin{aligned}
\mathcal{R}^{\mathrm{hinge}}_t(x,z_t;\tilde{f}_t) := \mathbb{E}\left[\ell^{\mathrm{hinge}}(\tilde{f}_t(X,Z_t),Y;c_t)\mid X=x,Z_t=z_t\right] \\
&= \sum_{y\in[K]}p_y\ell^{\mathrm{hinge}}(v,y;c_t)=L(v)+\frac{K}{K-1}c_t. \tag*{(50)}
\end{aligned} R t hinge ( x , z t ; f ~ t ) := E [ ℓ hinge ( f ~ t ( X , Z t ) , Y ; c t ) ∣ X = x , Z t = z t ] = y ∈ [ K ] ∑ p y ℓ hinge ( v , y ; c t ) = L ( v ) + K − 1 K c t . (50)
Combined with R t h i n g e ( x , z t ; f t ∗ ) = K K − 1 ( 1 − max y ∈ [ K ] p y ) + K K − 1 c t \mathcal{R}^{\mathrm{hinge}}_t(x,z_t;f_t^*)=\frac{K}{K-1}\left(1-\max_{y\in[K]}p_y\right)+\frac{K}{K-1}c_t R t hinge ( x , z t ; f t ∗ ) = K − 1 K ( 1 − max y ∈ [ K ] p y ) + K − 1 K c t from (45) , the cost terms cancel and the excess stage-wise hinge risk can be written into
R t h i n g e ( x , z t ; f ~ t ) − R t h i n g e ( x , z t ; f t ∗ ) = L ( v ) − K K − 1 ( 1 − max y ∈ [ K ] p y ) . (51) \mathcal{R}^{\mathrm{hinge}}_t(x,z_t;\tilde{f}_t)-\mathcal{R}^{\mathrm{hinge}}_t(x,z_t;f_t^*)=L(v)-\frac{K}{K-1}\left(1-\max_{y\in[K]}p_y\right). \tag*{(51)} R t hinge ( x , z t ; f ~ t ) − R t hinge ( x , z t ; f t ∗ ) = L ( v ) − K − 1 K ( 1 − y ∈ [ K ] max p y ) . (51)
Similarly, for the stage-wise 0-1-c c c risk, since f t f_t f t predicts label j j j , we can write
R t ( x , z t ; f t ) = E [ 1 { f t ( X , Z t ) ≠ Y } + c t ∣ X = x , Z t = z t ] = 1 − p j + c t . (52) R_t(x,z_t;f_t)=\mathbb{E}\left[\mathbb{1}\{f_t(X,Z_t)\ne Y\}+c_t\mid X=x,Z_t=z_t\right]=1-p_j+c_t. \tag*{(52)} R t ( x , z t ; f t ) = E [ 1 { f t ( X , Z t ) = Y } + c t ∣ X = x , Z t = z t ] = 1 − p j + c t . (52)
Combined with R t ( x , z t ; f t ∗ ) = 1 − max y ∈ [ K ] p y + c t R_t(x,z_t;f_t^*)=1-\max_{y\in[K]}p_y+c_t R t ( x , z t ; f t ∗ ) = 1 − max y ∈ [ K ] p y + c t since f t ∗ f_t^* f t ∗ predicts the most likely label, we can write the excess 0-1-c c c stage-wise risk into
R t ( x , z t ; f t ) − R t ( x , z t ; f t ∗ ) = max y ∈ [ K ] p y − p j . (53) R_t(x,z_t;f_t)-R_t(x,z_t;f_t^*)=\max_{y\in[K]}p_y-p_j. \tag*{(53)} R t ( x , z t ; f t ) − R t ( x , z t ; f t ∗ ) = y ∈ [ K ] max p y − p j . (53)
In the rest of the proof, we provide a lower bound for L ( v ) L(v) L ( v ) , which will be integrated into (51) and combined with (53) to prove the final calibration inequality (46) .
Part 2. Lower bounding L ( v ) L(v) L ( v ) : Using the definition of L ( v ) L(v) L ( v ) in (48) , we can rewrite it as follows
L ( v ) = ∑ y ∈ [ K ] p y ∑ k ∈ [ K ] ∖ y [ v k + 1 K − 1 ] + = ( i ) ∑ k ∈ [ K ] ( 1 − p k ) [ v k + 1 K − 1 ] + = ( i i ) U − ∑ k ∈ [ K ] p k [ v k + 1 K − 1 ] + = ( i i i ) U − p j [ v j + 1 K − 1 ] + − ∑ k ∈ [ K ] ∖ j p k [ v k + 1 K − 1 ] + . (54) \begin{aligned}
L(v)
&=\sum_{y\in[K]}p_y\sum_{k\in[K]\setminus y}\left[v_k+\frac{1}{K-1}\right]_+ \\
&\overset{\mathrm{(i)}}{=}\sum_{k\in[K]}(1-p_k)\left[v_k+\frac{1}{K-1}\right]_+ \\
&\overset{\mathrm{(ii)}}{=}U-\sum_{k\in[K]}p_k\left[v_k+\frac{1}{K-1}\right]_+ \\
&\overset{\mathrm{(iii)}}{=}U-p_j\left[v_j+\frac{1}{K-1}\right]_+-\sum_{k\in[K]\setminus j}p_k\left[v_k+\frac{1}{K-1}\right]_+. \tag*{(54)}
\end{aligned} L ( v ) = y ∈ [ K ] ∑ p y k ∈ [ K ] ∖ y ∑ [ v k + K − 1 1 ] + = ( i ) k ∈ [ K ] ∑ ( 1 − p k ) [ v k + K − 1 1 ] + = ( ii ) U − k ∈ [ K ] ∑ p k [ v k + K − 1 1 ] + = ( iii ) U − p j [ v j + K − 1 1 ] + − k ∈ [ K ] ∖ j ∑ p k [ v k + K − 1 1 ] + . (54)
where step (i) follows from exchanging the two summations, step (ii) applies the definition of U U U from (49) , and step (iii) simply splits the j j j -th term from the summation. Using this, we now lower bound L ( v ) L(v) L ( v ) :
L ( v ) = U − p j [ v j + 1 K − 1 ] + − ∑ k ∈ [ K ] ∖ j p k [ v k + 1 K − 1 ] + ≥ U − p j [ v j + 1 K − 1 ] + − max y ∈ [ K ] p y ( U − [ v j + 1 K − 1 ] + ) = U ( 1 − max y ∈ [ K ] p y ) + [ v j + 1 K − 1 ] + ( max y ∈ [ K ] p y − p j ) . (55) \begin{aligned}
L(v)
&=U-p_j\left[v_j+\frac{1}{K-1}\right]_+-\sum_{k\in[K]\setminus j}p_k\left[v_k+\frac{1}{K-1}\right]_+ \\
&\ge U-p_j\left[v_j+\frac{1}{K-1}\right]_+-\max_{y\in[K]}p_y\left(U-\left[v_j+\frac{1}{K-1}\right]_+\right) \\
&=U\left(1-\max_{y\in[K]}p_y\right)+\left[v_j+\frac{1}{K-1}\right]_+\left(\max_{y\in[K]}p_y-p_j\right). \tag*{(55)}
\end{aligned} L ( v ) = U − p j [ v j + K − 1 1 ] + − k ∈ [ K ] ∖ j ∑ p k [ v k + K − 1 1 ] + ≥ U − p j [ v j + K − 1 1 ] + − y ∈ [ K ] max p y ( U − [ v j + K − 1 1 ] + ) = U ( 1 − y ∈ [ K ] max p y ) + [ v j + K − 1 1 ] + ( y ∈ [ K ] max p y − p j ) . (55)
where the inequality holds by applying p k ≤ max y ∈ [ K ] p y p_k\le\max_{y\in[K]}p_y p k ≤ max y ∈ [ K ] p y and the definition of U U U from (49) . Part 3. Combining all components: To combine all components of the proof, we begin with two auxiliary inequalities about U U U and the hinge terms. First, U U U is lower bounded by K K − 1 \frac{K}{K-1} K − 1 K because
U = ∑ k ∈ [ K ] [ v k + 1 K − 1 ] + ≥ ∑ k ∈ [ K ] ( v k + 1 K − 1 ) = ( i i ) K K − 1 , (56) U=\sum_{k\in[K]}\left[v_k+\frac{1}{K-1}\right]_+\geq\sum_{k\in[K]}\left(v_k+\frac{1}{K-1}\right)\overset{\mathrm{(ii)}}{=}\frac{K}{K-1}, \tag*{(56)} U = k ∈ [ K ] ∑ [ v k + K − 1 1 ] + ≥ k ∈ [ K ] ∑ ( v k + K − 1 1 ) = ( ii ) K − 1 K , (56)
where step (i) uses [ x ] + ≥ x [x]_+\geq x [ x ] + ≥ x , and step (ii) uses ∑ k ∈ [ K ] v k = 0 \sum_{k\in[K]}v_k=0 ∑ k ∈ [ K ] v k = 0 . Second, since the predicted label j ∈ arg max k ∈ [ K ] v k j\in\arg\max_{k\in[K]}v_k j ∈ arg max k ∈ [ K ] v k , we have v j ≥ v k v_j\geq v_k v j ≥ v k for all k ∈ [ K ] k\in[K] k ∈ [ K ] , and so
[ v j + 1 K − 1 ] + = 1 K ∑ k ∈ [ K ] [ v j + 1 K − 1 ] + ≥ 1 K ∑ k ∈ [ K ] [ v k + 1 K − 1 ] + = 1 K ⋅ U ≥ 1 K − 1 , (57) \left[v_j+\frac{1}{K-1}\right]_+=\frac{1}{K}\sum_{k\in[K]}\left[v_j+\frac{1}{K-1}\right]_+\geq\frac{1}{K}\sum_{k\in[K]}\left[v_k+\frac{1}{K-1}\right]_+=\frac{1}{K}\cdot U\geq\frac{1}{K-1}, \tag*{(57)} [ v j + K − 1 1 ] + = K 1 k ∈ [ K ] ∑ [ v j + K − 1 1 ] + ≥ K 1 k ∈ [ K ] ∑ [ v k + K − 1 1 ] + = K 1 ⋅ U ≥ K − 1 1 , (57)
where the last inequality applies the previous inequality (56).
Equipped with the lower bound in (55) for L ( v ) L(v) L ( v ) and the auxiliary inequalities (56)–(57), we can now lower bound the excess stage-wise hinge risk from (51) as follows:
R t h i n g e ( x , z t ; f ~ t ) − R t h i n g e ( x , z t ; f ~ t ∗ ) = L ( v ) − K K − 1 ( 1 − max y ∈ [ K ] p y ) ≥ ( i ) U ( 1 − max y ∈ [ K ] p y ) + [ v j + 1 K − 1 ] + ( max y ∈ [ K ] p y − p j ) − K K − 1 ( 1 − max y ∈ [ K ] p y ) = ( U − K K − 1 ) ( 1 − max y ∈ [ K ] p y ) + [ v j + 1 K − 1 ] + ( max y ∈ [ K ] p y − p j ) ≥ ( i i ) [ v j + 1 K − 1 ] + ( max y ∈ [ K ] p y − p j ) ≥ ( i i i ) 1 K − 1 ( max y ∈ [ K ] p y − p j ) = 1 K − 1 ( R t ( x , z t ; f t ) − R t ( x , z t ; f t ∗ ) ) . (58) \begin{aligned}
R_t^{\mathrm{hinge}}(x,z_t;\tilde f_t)-R_t^{\mathrm{hinge}}(x,z_t;\tilde f_t^*)
&=L(v)-\frac{K}{K-1}\left(1-\max_{y\in[K]}p_y\right)\\
&\overset{\mathrm{(i)}}{\geq}U\left(1-\max_{y\in[K]}p_y\right)+\left[v_j+\frac{1}{K-1}\right]_+\left(\max_{y\in[K]}p_y-p_j\right)\\
&\qquad-\frac{K}{K-1}\left(1-\max_{y\in[K]}p_y\right)\\
&=\left(U-\frac{K}{K-1}\right)\left(1-\max_{y\in[K]}p_y\right)+\left[v_j+\frac{1}{K-1}\right]_+\left(\max_{y\in[K]}p_y-p_j\right)\\
&\overset{\mathrm{(ii)}}{\geq}\left[v_j+\frac{1}{K-1}\right]_+\left(\max_{y\in[K]}p_y-p_j\right)\\
&\overset{\mathrm{(iii)}}{\geq}\frac{1}{K-1}\left(\max_{y\in[K]}p_y-p_j\right)\\
&=\frac{1}{K-1}\left(R_t(x,z_t;f_t)-R_t(x,z_t;f_t^*)\right). \tag*{(58)}
\end{aligned} R t hinge ( x , z t ; f ~ t ) − R t hinge ( x , z t ; f ~ t ∗ ) = L ( v ) − K − 1 K ( 1 − y ∈ [ K ] max p y ) ≥ ( i ) U ( 1 − y ∈ [ K ] max p y ) + [ v j + K − 1 1 ] + ( y ∈ [ K ] max p y − p j ) − K − 1 K ( 1 − y ∈ [ K ] max p y ) = ( U − K − 1 K ) ( 1 − y ∈ [ K ] max p y ) + [ v j + K − 1 1 ] + ( y ∈ [ K ] max p y − p j ) ≥ ( ii ) [ v j + K − 1 1 ] + ( y ∈ [ K ] max p y − p j ) ≥ ( iii ) K − 1 1 ( y ∈ [ K ] max p y − p j ) = K − 1 1 ( R t ( x , z t ; f t ) − R t ( x , z t ; f t ∗ ) ) . (58)
where (i) applies (55), (ii) holds by dropping the non-negative term ( U − K K − 1 ) ( 1 − max y ∈ [ K ] p y ) \left(U-\frac{K}{K-1}\right)\left(1-\max_{y\in[K]}p_y\right) ( U − K − 1 K ) ( 1 − max y ∈ [ K ] p y ) , (iii) uses (57), and the last step uses the expression of the excess 0-1-c risk from (53). Multiplying both sides by K − 1 K-1 K − 1 proves our final calibration inequality (46). □
Proof of Theorem 3.1
Using the results in Sections B.1–B.2, we prove our main calibration result. We first restate the theorem.
Theorem 3.1 (Linear calibration of ℓ f l a t h i n g e \ell_{\mathrm{flat}}^{\mathrm{hinge}} ℓ flat hinge w.r.t. ℓ \ell ℓ ). Define R f l a t ∗ : = min π ∈ Π f l a t R ( π ) R_{\mathrm{flat}}^*:=\min_{\pi\in\Pi_{\mathrm{flat}}}R(\pi) R flat ∗ := min π ∈ Π flat R ( π ) and R f l a t h i n g e ∗ : = min π ~ ∈ Π ~ f l a t R f l a t h i n g e ( π ~ ) R_{\mathrm{flat}}^{\mathrm{hinge}*}:=\min_{\tilde\pi\in\widetilde{\Pi}_{\mathrm{flat}}}R_{\mathrm{flat}}^{\mathrm{hinge}}(\tilde\pi) R flat hinge ∗ := min π ~ ∈ Π flat R flat hinge ( π ~ ) . Then, for any π ~ ∈ Π ~ f l a t \tilde\pi\in\widetilde{\Pi}_{\mathrm{flat}} π ~ ∈ Π flat ,
R ( φ ( π ~ ) ) − R f l a t ∗ ≤ T ( K − 1 ) ( R f l a t h i n g e ( π ~ ) − R f l a t h i n g e ∗ ) . (59) R(\varphi(\tilde\pi))-R_{\mathrm{flat}}^*\leq T(K-1)\left(R_{\mathrm{flat}}^{\mathrm{hinge}}(\tilde\pi)-R_{\mathrm{flat}}^{\mathrm{hinge}*}\right). \tag*{(59)} R ( φ ( π ~ )) − R flat ∗ ≤ T ( K − 1 ) ( R flat hinge ( π ~ ) − R flat hinge ∗ ) . (59)
Proof. First we set up some notation. Let
π ~ f l a t = ( f ~ 1 : T , r ~ f l a t ) ∈ Π ~ f l a t and π f l a t = ( f 1 : T , r f l a t ) = φ ( π ~ f l a t ) ∈ Π f l a t \tilde\pi_{\mathrm{flat}}=(\tilde f_{1:T},\tilde r_{\mathrm{flat}})\in\widetilde{\Pi}_{\mathrm{flat}}
\qquad\text{and}\qquad
\pi_{\mathrm{flat}}=(f_{1:T},r_{\mathrm{flat}})=\varphi(\tilde\pi_{\mathrm{flat}})\in\Pi_{\mathrm{flat}} π ~ flat = ( f ~ 1 : T , r ~ flat ) ∈ Π flat and π flat = ( f 1 : T , r flat ) = φ ( π ~ flat ) ∈ Π flat
denote an arbitrary soft predictor and its hard counterpart. Let
π ~ f l a t ∗ = ( f ~ 1 : T ∗ , r ~ f l a t ∗ ) ∈ arg min π ~ ∈ Π ~ f l a t R f l a t h i n g e ( π ~ ) and π f l a t ∗ = ( f 1 : T ∗ , r f l a t ∗ ) ∈ arg min π ∈ Π f l a t R ( π ) \widetilde{\pi}_{\mathrm{flat}}^* = (\widetilde{f}_{1:T}^*, \widetilde{r}_{\mathrm{flat}}^*) \in\underset{\widetilde{\pi} \in\widetilde{\Pi}_{\mathrm{flat}}}{\arg\min}\mathcal{R}_{\mathrm{flat}}^{\mathrm{hinge}}(\widetilde{\pi})
\qquad\text{and}\qquad
\pi_{\mathrm{flat}}^* = (f_{1:T}^*, r_{\mathrm{flat}}^*) \in\underset{\pi\in\Pi_{\mathrm{flat}}}{\arg\min}\mathcal{R}(\pi) π flat ∗ = ( f 1 : T ∗ , r flat ∗ ) ∈ π ∈ Π flat arg min R flat hinge ( π ) and π flat ∗ = ( f 1 : T ∗ , r flat ∗ ) ∈ π ∈ Π flat arg min R ( π )
be the hinge risk minimizer of Lemma B.2, and the 0-1-c c c risk minimizer of Lemma B.1. For a given input x ∈ X x \in\mathcal{X} x ∈ X , we use
t : = r f l a t ( x ) ∈ arg max τ ∈ [ T ] [ r ~ f l a t ( x ) ] τ and t ∗ : = r f l a t ∗ ( x ) ∈ arg min τ ∈ [ T ] R τ ( x ; f τ ∗ ) (60) t := r_{\mathrm{flat}}(x) \in\underset{\tau\in[T]}{\arg\max}[\widetilde{r}_{\mathrm{flat}}(x)]_{\tau}
\qquad\text{and}\qquad
t^* := r_{\mathrm{flat}}^*(x) \in\underset{\tau\in[T]}{\arg\min}\mathcal{R}_{\tau}(x; f_{\tau}^*) \tag*{(60)} t := r flat ( x ) ∈ τ ∈ [ T ] arg max [ r flat ( x ) ] τ and t ∗ := r flat ∗ ( x ) ∈ τ ∈ [ T ] arg min R τ ( x ; f τ ∗ ) (60)
to denote the index of the classifier selected by the hard predictor π f l a t \pi_{\mathrm{flat}} π flat and the Bayes-optimal predictor π f l a t ∗ \pi_{\mathrm{flat}}^* π flat ∗ , respectively.
Hence, to prove (17), it suffices to prove pointwise in x ∈ X x \in\mathcal{X} x ∈ X that
R t ( x ; f t ) − R t ∗ ( x ; f t ∗ ∗ ) ≤ T ( K − 1 ) ( R f l a t h i n g e ( x ; π ~ ) − R f l a t h i n g e ( x ; π ~ ∗ ) ) (61) \mathcal{R}_{t}(x; f_t) - \mathcal{R}_{t^*}(x; f_{t^*}^*) \le T(K-1)\left(\mathcal{R}_{\mathrm{flat}}^{\mathrm{hinge}}(x; \widetilde{\pi}) - \mathcal{R}_{\mathrm{flat}}^{\mathrm{hinge}}(x; \widetilde{\pi}^*)\right) \tag*{(61)} R t ( x ; f t ) − R t ∗ ( x ; f t ∗ ∗ ) ≤ T ( K − 1 ) ( R flat hinge ( x ; π ) − R flat hinge ( x ; π ∗ ) ) (61)
where each term here denotes conditional risks, conditioned on X = x X=x X = x , since taking total expectation on both sides yields the desired result. We prove (61) in the rest of this proof.
Part 1: Decomposition of the conditional excess 0-1-c c c risk. We now provide general definitions that we used to define the terms on the LHS of (61) . For τ ∈ [ T ] \tau\in[T] τ ∈ [ T ] , define in line with (22) ,
R τ ( x ; f τ ) : = E [ 1 { f τ ( x , Z τ ) ≠ Y } ∣ X = x ] + κ τ , (62) \mathcal{R}_{\tau}(x; f_{\tau}) := \mathbb{E}[\mathbb{1}\{f_{\tau}(x, Z_{\tau}) \ne Y\} \mid X=x] + \kappa_{\tau}, \tag*{(62)} R τ ( x ; f τ ) := E [ 1 { f τ ( x , Z τ ) = Y } ∣ X = x ] + κ τ , (62)
R τ ( x ; f τ ∗ ) : = E [ 1 { f τ ∗ ( x , Z τ ) ≠ Y } ∣ X = x ] + κ τ \mathcal{R}_{\tau}(x; f_{\tau}^*) := \mathbb{E}[\mathbb{1}\{f_{\tau}^*(x, Z_{\tau}) \ne Y\} \mid X=x] + \kappa_{\tau} R τ ( x ; f τ ∗ ) := E [ 1 { f τ ∗ ( x , Z τ ) = Y } ∣ X = x ] + κ τ
= ( i ) 1 − E [ max y ∈ [ K ] p ( y ∣ x , Z τ ) ∣ X = x ] + κ τ , (63) \overset{\mathrm{(i)}}{=} 1 - \mathbb{E}\left[\max_{y \in[K]} p(y \mid x, Z_{\tau}) \mid X=x\right] + \kappa_{\tau}, \tag*{(63)} = ( i ) 1 − E [ y ∈ [ K ] max p ( y ∣ x , Z τ ) ∣ X = x ] + κ τ , (63)
where (i) applies the expression for f τ ∗ f_{\tau}^* f τ ∗ from (30) . Combining these, we can write
R t ( x ; f t ) − R t ∗ ( x ; f t ∗ ∗ ) = R t ( x ; f t ) − R t ( x ; f t ∗ ) ⏟ = : Δ i n t e r n a l ( x ) + R t ( x ; f t ∗ ) − R t ∗ ( x ; f t ∗ ∗ ) ⏟ = : Δ s e l e c t ( x ) = Δ i n t e r n a l ( x ) + Δ s e l e c t ( x ) . (64) \mathcal{R}_{t}(x; f_t) - \mathcal{R}_{t^*}(x; f_{t^*}^*) =
\underbrace{\mathcal{R}_{t}(x; f_t) - \mathcal{R}_{t}(x; f_t^*)}_{=:\Delta_{\mathrm{internal}}(x)}
+
\underbrace{\mathcal{R}_{t}(x; f_t^*) - \mathcal{R}_{t^*}(x; f_{t^*}^*)}_{=:\Delta_{\mathrm{select}}(x)}
= \Delta_{\mathrm{internal}}(x) + \Delta_{\mathrm{select}}(x). \tag*{(64)} R t ( x ; f t ) − R t ∗ ( x ; f t ∗ ∗ ) = =: Δ internal ( x ) R t ( x ; f t ) − R t ( x ; f t ∗ ) + =: Δ select ( x ) R t ( x ; f t ∗ ) − R t ∗ ( x ; f t ∗ ∗ ) = Δ internal ( x ) + Δ select ( x ) . (64)
Intuitively, Δ i n t e r n a l ( x ) \Delta_{\mathrm{internal}}(x) Δ internal ( x ) denotes the excess risk caused by using a suboptimal classifier as the t t t -th classifier, so it is an excess risk internal to the t t t -th stage. Note Δ i n t e r n a l ( x ) ≥ 0 \Delta_{\mathrm{internal}}(x) \ge0 Δ internal ( x ) ≥ 0 because f t ∗ f_t^* f t ∗ is the Bayes-optimal t t t -th classifier; see the explanation around (34) –(35) . The Δ s e l e c t ( x ) \Delta_{\mathrm{select}}(x) Δ select ( x ) term is the excess risk caused by selecting a suboptimal exit stage t t t , and Δ s e l e c t ( x ) ≥ 0 \Delta_{\mathrm{select}}(x) \ge0 Δ select ( x ) ≥ 0 by the definition of t ∗ t^* t ∗ in (60) .
Part 2: Decomposition of the conditional excess hinge risk. We can similarly decompose the conditional excess hinge risk. To begin, define the stage-τ \tau τ conditional hinge risk given X = x X=x X = x as
R τ h i n g e ( x ; f ~ τ ) : = E [ ℓ h i n g e ( f ~ τ ( x , Z τ ) , Y ; κ τ ) ∣ X = x ] for τ ∈ [ T ] . (65) \mathcal{R}_{\tau}^{\mathrm{hinge}}(x; \widetilde{f}_{\tau}) := \mathbb{E}\left[\ell^{\mathrm{hinge}}(\widetilde{f}_{\tau}(x, Z_{\tau}), Y; \kappa_{\tau}) \mid X=x\right]
\qquad\text{for }\tau\in[T]. \tag*{(65)} R τ hinge ( x ; f τ ) := E [ ℓ hinge ( f τ ( x , Z τ ) , Y ; κ τ ) ∣ X = x ] for τ ∈ [ T ] . (65)
Evaluated at the Bayes-optimal solution, this is linked to the stage-τ \tau τ conditional 0-1-c c c risk via
R τ h i n g e ( x ; f ~ τ ∗ ) = ( i ) E [ R τ h i n g e ( x , Z τ ; f τ ∗ ) ∣ X = x ] = ( i i ) K K − 1 E [ R τ ( x , Z τ ; f τ ∗ ) ∣ X = x ] = ( i i i ) K K − 1 R τ ( x ; f τ ∗ ) ∀ τ ∈ [ T ] . (66) \mathcal{R}_{\tau}^{\mathrm{hinge}}(x; \widetilde{f}_{\tau}^*) \overset{\mathrm{(i)}}{=} \mathbb{E}\left[\mathcal{R}_{\tau}^{\mathrm{hinge}}(x, Z_{\tau}; f_{\tau}^*) \mid X=x\right]
\overset{\mathrm{(ii)}}{=} \frac{K}{K-1}\mathbb{E}\left[\mathcal{R}_{\tau}(x, Z_{\tau}; f_{\tau}^*) \mid X=x\right]
\overset{\mathrm{(iii)}}{=} \frac{K}{K-1}\mathcal{R}_{\tau}(x; f_{\tau}^*) \qquad\forall\tau\in[T]. \tag*{(66)} R τ hinge ( x ; f τ ∗ ) = ( i ) E [ R τ hinge ( x , Z τ ; f τ ∗ ) ∣ X = x ] = ( ii ) K − 1 K E [ R τ ( x , Z τ ; f τ ∗ ) ∣ X = x ] = ( iii ) K − 1 K R τ ( x ; f τ ∗ ) ∀ τ ∈ [ T ] . (66)
where (i) and (iii) apply the tower property with respect to P Z τ ∣ X ( ⋅ ∣ x ) P_{Z_{\tau}\mid X}(\cdot\mid x) P Z τ ∣ X ( ⋅ ∣ x ) , and (ii) applies (44). This result will become useful shortly.
Note that Lemma B.2 proves that the support of any Bayes-optimal r ~ f l a t ∗ ( x ) \widetilde{r}_{\mathrm{flat}}^*(x) r flat ∗ ( x ) is contained in arg min t ∈ [ T ] R t ( x ; f t ∗ ) \arg\min_{t\in[T]} \mathcal{R}_t(x;f_t^*) arg min t ∈ [ T ] R t ( x ; f t ∗ ) , so the discrete router r f l a t ∗ ( x ) r_{\mathrm{flat}}^*(x) r flat ∗ ( x ) from Lemma B.1 satisfies
t ∗ : = r f l a t ∗ ( x ) ∈ arg min t ∈ [ T ] R t ( x ; f t ∗ ) ⊇ arg max t ∈ [ T ] [ r ~ f l a t ∗ ( x ) ] t . (67) t^* := r_{\mathrm{flat}}^*(x) \in\arg\min_{t\in[T]} \mathcal{R}_t(x;f_t^*) \supseteq\arg\max_{t\in[T]} [\widetilde{r}_{\mathrm{flat}}^*(x)]_t . \tag*{(67)} t ∗ := r flat ∗ ( x ) ∈ arg t ∈ [ T ] min R t ( x ; f t ∗ ) ⊇ arg t ∈ [ T ] max [ r flat ∗ ( x ) ] t . (67)
Hence, using (64) and (66) and letting R f l a t h i n g e ( x ; π ~ ∗ ) \mathcal{R}_{\mathrm{flat}}^{\mathrm{hinge}}(x;\widetilde{\pi}^*) R flat hinge ( x ; π ∗ ) denote the conditional flat-router hinge risk, conditional on X = x X=x X = x , we can write
R f l a t h i n g e ( x ; π ~ ∗ ) : = ∑ τ = 1 T [ r ~ f l a t ∗ ( x ) ] τ R τ h i n g e ( x ; f ~ τ ) \mathcal{R}_{\mathrm{flat}}^{\mathrm{hinge}}(x;\widetilde{\pi}^*) := \sum_{\tau=1}^{T} [\widetilde{r}_{\mathrm{flat}}^*(x)]_{\tau}\mathcal{R}_{\tau}^{\mathrm{hinge}}(x;\widetilde{f}_{\tau}) R flat hinge ( x ; π ∗ ) := τ = 1 ∑ T [ r flat ∗ ( x ) ] τ R τ hinge ( x ; f τ )
= (i) ∑ τ = 1 T [ r ~ f l a t ∗ ( x ) ] τ K K − 1 R τ ( x ; f τ ∗ ) = (ii) K K − 1 R t ∗ ( x ; f t ∗ ∗ ) = (iii) R t ∗ h i n g e ( x ; f ~ t ∗ ∗ ) , (68) \mathrel{\mathop{=}^{\text{(i)}}} \sum_{\tau=1}^{T} [\widetilde{r}_{\mathrm{flat}}^*(x)]_{\tau}\frac{K}{K-1}\mathcal{R}_{\tau}(x;f_{\tau}^*) \mathrel{\mathop{=}^{\text{(ii)}}} \frac{K}{K-1}\mathcal{R}_{t^*}(x;f_{t^*}^*) \mathrel{\mathop{=}^{\text{(iii)}}} \mathcal{R}_{t^*}^{\mathrm{hinge}}(x;\widetilde{f}_{t^*}^*), \tag*{(68)} = (i) τ = 1 ∑ T [ r flat ∗ ( x ) ] τ K − 1 K R τ ( x ; f τ ∗ ) = (ii) K − 1 K R t ∗ ( x ; f t ∗ ∗ ) = (iii) R t ∗ hinge ( x ; f t ∗ ∗ ) , (68)
where steps (i) and (iii) hold by (65), and step (ii) uses the fact that R τ ( x ; f τ ∗ ) = R t ∗ ( x ; f t ∗ ∗ ) \mathcal{R}_{\tau}(x;f_{\tau}^*)=\mathcal{R}_{t^*}(x;f_{t^*}^*) R τ ( x ; f τ ∗ ) = R t ∗ ( x ; f t ∗ ∗ ) for all τ ∈ arg min t ∈ [ T ] R t ( x ; f t ∗ ) \tau\in\arg\min_{t\in[T]}\mathcal{R}_t(x;f_t^*) τ ∈ arg min t ∈ [ T ] R t ( x ; f t ∗ ) . Using (64) and (67), we can therefore write
R f l a t h i n g e ( x ; π ~ ) − R f l a t h i n g e ( x ; π ~ ∗ ) = ∑ τ = 1 T [ r ~ f l a t ( x ) ] τ R τ h i n g e ( x ; f ~ τ ) − R t ∗ h i n g e ( x ; f ~ t ∗ ∗ ) = ∑ τ = 1 T [ r ~ f l a t ( x ) ] τ ( R τ h i n g e ( x ; f ~ τ ) − R τ h i n g e ( x ; f ~ τ ∗ ) + R τ h i n g e ( x ; f ~ τ ∗ ) − R t ∗ h i n g e ( x ; f ~ t ∗ ∗ ) ) = ∑ τ = 1 T [ r ~ f l a t ( x ) ] τ ( R τ h i n g e ( x ; f ~ τ ) − R τ h i n g e ( x ; f ~ τ ∗ ) ) ⏟ = : Δ i n t e r n a l h i n g e ( x , τ ) + ∑ τ = 1 T [ r ~ f l a t ( x ) ] τ ( R τ h i n g e ( x ; f ~ τ ∗ ) − R t ∗ h i n g e ( x ; f ~ t ∗ ∗ ) ) ⏟ = : Δ s e l e c t h i n g e ( x , τ ) = ∑ τ = 1 T [ r ~ f l a t ( x ) ] τ Δ i n t e r n a l h i n g e ( x , τ ) + ∑ τ = 1 T [ r ~ f l a t ( x ) ] τ Δ s e l e c t h i n g e ( x , τ ) , (69) \begin{aligned}
\mathcal{R}_{\mathrm{flat}}^{\mathrm{hinge}}(x;\widetilde{\pi})-\mathcal{R}_{\mathrm{flat}}^{\mathrm{hinge}}(x;\widetilde{\pi}^*) \\
&= \sum_{\tau=1}^{T}[\widetilde{r}_{\mathrm{flat}}(x)]_{\tau}\mathcal{R}_{\tau}^{\mathrm{hinge}}(x;\widetilde{f}_{\tau})-\mathcal{R}_{t^*}^{\mathrm{hinge}}(x;\widetilde{f}_{t^*}^*) \\
&= \sum_{\tau=1}^{T}[\widetilde{r}_{\mathrm{flat}}(x)]_{\tau}\left(\mathcal{R}_{\tau}^{\mathrm{hinge}}(x;\widetilde{f}_{\tau})-\mathcal{R}_{\tau}^{\mathrm{hinge}}(x;\widetilde{f}_{\tau}^*)+\mathcal{R}_{\tau}^{\mathrm{hinge}}(x;\widetilde{f}_{\tau}^*)-\mathcal{R}_{t^*}^{\mathrm{hinge}}(x;\widetilde{f}_{t^*}^*)\right) \\
&= \sum_{\tau=1}^{T}[\widetilde{r}_{\mathrm{flat}}(x)]_{\tau}\underbrace{\left(\mathcal{R}_{\tau}^{\mathrm{hinge}}(x;\widetilde{f}_{\tau})-\mathcal{R}_{\tau}^{\mathrm{hinge}}(x;\widetilde{f}_{\tau}^*)\right)}_{=:\Delta_{\mathrm{internal}}^{\mathrm{hinge}}(x,\tau)}
+\sum_{\tau=1}^{T}[\widetilde{r}_{\mathrm{flat}}(x)]_{\tau}\underbrace{\left(\mathcal{R}_{\tau}^{\mathrm{hinge}}(x;\widetilde{f}_{\tau}^*)-\mathcal{R}_{t^*}^{\mathrm{hinge}}(x;\widetilde{f}_{t^*}^*)\right)}_{=:\Delta_{\mathrm{select}}^{\mathrm{hinge}}(x,\tau)} \\
&= \sum_{\tau=1}^{T}[\widetilde{r}_{\mathrm{flat}}(x)]_{\tau}\Delta_{\mathrm{internal}}^{\mathrm{hinge}}(x,\tau)+\sum_{\tau=1}^{T}[\widetilde{r}_{\mathrm{flat}}(x)]_{\tau}\Delta_{\mathrm{select}}^{\mathrm{hinge}}(x,\tau), \tag*{(69)}
\end{aligned} R flat hinge ( x ; π ) − R flat hinge ( x ; π ∗ ) = τ = 1 ∑ T [ r flat ( x ) ] τ R τ hinge ( x ; f τ ) − R t ∗ hinge ( x ; f t ∗ ∗ ) = τ = 1 ∑ T [ r flat ( x ) ] τ ( R τ hinge ( x ; f τ ) − R τ hinge ( x ; f τ ∗ ) + R τ hinge ( x ; f τ ∗ ) − R t ∗ hinge ( x ; f t ∗ ∗ ) ) = τ = 1 ∑ T [ r flat ( x ) ] τ =: Δ internal hinge ( x , τ ) ( R τ hinge ( x ; f τ ) − R τ hinge ( x ; f τ ∗ ) ) + τ = 1 ∑ T [ r flat ( x ) ] τ =: Δ select hinge ( x , τ ) ( R τ hinge ( x ; f τ ∗ ) − R t ∗ hinge ( x ; f t ∗ ∗ ) ) = τ = 1 ∑ T [ r flat ( x ) ] τ Δ internal hinge ( x , τ ) + τ = 1 ∑ T [ r flat ( x ) ] τ Δ select hinge ( x , τ ) , (69)
where we have Δ i n t e r n a l h i n g e ( x , τ ) ≥ 0 \Delta_{\mathrm{internal}}^{\mathrm{hinge}}(x,\tau)\ge0 Δ internal hinge ( x , τ ) ≥ 0 by applying [53 Lemma 1] to the Bayes-optimal soft classifier f ~ τ ∗ \widetilde{f}_{\tau}^* f τ ∗ , and Δ s e l e c t h i n g e ( x , τ ) = K K − 1 ( R τ ( x , f τ ∗ ) − R t ∗ ( x , f t ∗ ∗ ) ) ≥ 0 \Delta_{\mathrm{select}}^{\mathrm{hinge}}(x,\tau)=\frac{K}{K-1}\left(\mathcal{R}_{\tau}(x,f_{\tau}^*)-\mathcal{R}_{t^*}(x,f_{t^*}^*)\right)\ge0 Δ select hinge ( x , τ ) = K − 1 K ( R τ ( x , f τ ∗ ) − R t ∗ ( x , f t ∗ ∗ ) ) ≥ 0 by (65) and the definition of t ∗ t^* t ∗ in (59).
Part 3: Upper bounding Δ i n t e r n a l ( x ) \Delta_{\mathrm{internal}}(x) Δ internal ( x ) and Δ s e l e c t ( x ) \Delta_{\mathrm{select}}(x) Δ select ( x ) by Δ i n t e r n a l h i n g e ( x , τ ) \Delta_{\mathrm{internal}}^{\mathrm{hinge}}(x,\tau) Δ internal hinge ( x , τ ) and Δ s e l e c t h i n g e ( x , τ ) \Delta_{\mathrm{select}}^{\mathrm{hinge}}(x,\tau) Δ select hinge ( x , τ ) . By Lemma B.4 (given only X = x X=x X = x ), which links Δ i n t e r n a l \Delta_{\mathrm{internal}} Δ internal and Δ i n t e r n a l h i n g e \Delta_{\mathrm{internal}}^{\mathrm{hinge}} Δ internal hinge up to a factor of K − 1 K-1 K − 1 , we can write
Δ i n t e r n a l ( x ) ≤ ( K − 1 ) Δ i n t e r n a l h i n g e ( x , t ) ≤ ( K − 1 ) 1 [ r ~ f l a t ( x ) ] t ∑ τ = 1 T [ r ~ f l a t ( x ) ] τ Δ i n t e r n a l h i n g e ( x , τ ) ≤ T ( K − 1 ) ∑ τ = 1 T [ r ~ f l a t ( x ) ] τ Δ i n t e r n a l h i n g e ( x , τ ) , (70) \begin{aligned}
\Delta_{\mathrm{internal}}(x)\le(K-1)\Delta_{\mathrm{internal}}^{\mathrm{hinge}}(x,t) \\
&\le(K-1)\frac{1}{[\widetilde{r}_{\mathrm{flat}}(x)]_t}\sum_{\tau=1}^{T}[\widetilde{r}_{\mathrm{flat}}(x)]_{\tau}\Delta_{\mathrm{internal}}^{\mathrm{hinge}}(x,\tau) \\
&\le T(K-1)\sum_{\tau=1}^{T}[\widetilde{r}_{\mathrm{flat}}(x)]_{\tau}\Delta_{\mathrm{internal}}^{\mathrm{hinge}}(x,\tau), \tag*{(70)}
\end{aligned} Δ internal ( x ) ≤ ( K − 1 ) Δ internal hinge ( x , t ) ≤ ( K − 1 ) [ r flat ( x ) ] t 1 τ = 1 ∑ T [ r flat ( x ) ] τ Δ internal hinge ( x , τ ) ≤ T ( K − 1 ) τ = 1 ∑ T [ r flat ( x ) ] τ Δ internal hinge ( x , τ ) , (70)
where the last inequality holds because [ r ~ f l a t ( x ) ] τ ≥ 0 [\tilde{r}_{\mathrm{flat}}(x)]_{\tau} \ge0 [ r ~ flat ( x ) ] τ ≥ 0 , Δ i n t e r n a l h i n g e ( x , τ ) ≥ 0 \Delta^{\mathrm{hinge}}_{\mathrm{internal}}(x,\tau) \ge0 Δ internal hinge ( x , τ ) ≥ 0 for all τ ∈ [ T ] \tau\in[T] τ ∈ [ T ] and, by the definition of t t t in (59), [ r ~ f l a t ( x ) ] t ≥ 1 T [\tilde{r}_{\mathrm{flat}}(x)]_{t} \ge\frac{1}{T} [ r ~ flat ( x ) ] t ≥ T 1 at the chosen classifier index t t t . Similarly, we have
∑ τ = 1 T [ r ~ f l a t ( x ) ] τ Δ s e l e c t h i n g e ( x , τ ) = ∑ τ = 1 T [ r ~ f l a t ( x ) ] τ ( R τ h i n g e ( x ; f τ ∗ ) − R t ∗ h i n g e ( x ; f t ∗ ∗ ) ) ≥ ( i ) ∑ τ = 1 T [ r ~ f l a t ( x ) ] τ ⋅ K K − 1 ( R τ ( x ; f τ ∗ ) − R t ∗ ( x ; f t ∗ ∗ ) ) ≥ ( i i ) [ r ~ f l a t ( x ) ] t ⋅ K K − 1 ( R t ( x ; f t ∗ ) − R t ∗ ( x ; f t ∗ ∗ ) ) ≥ ( i i i ) K T ( K − 1 ) Δ s e l e c t ( x ) , \begin{aligned}
\sum_{\tau=1}^{T}[\tilde{r}_{\mathrm{flat}}(x)]_{\tau}\Delta^{\mathrm{hinge}}_{\mathrm{select}}(x,\tau)
&= \sum_{\tau=1}^{T}[\tilde{r}_{\mathrm{flat}}(x)]_{\tau}\left(\mathcal{R}^{\mathrm{hinge}}_{\tau}(x;f_{\tau}^{*})-\mathcal{R}^{\mathrm{hinge}}_{t^{*}}(x;f_{t^{*}}^{*})\right) \\
&\underset{(i)}{\ge} \sum_{\tau=1}^{T}[\tilde{r}_{\mathrm{flat}}(x)]_{\tau}\cdot\frac{K}{K-1}\left(\mathcal{R}_{\tau}(x;f_{\tau}^{*})-\mathcal{R}_{t^{*}}(x;f_{t^{*}}^{*})\right) \\
&\underset{(ii)}{\ge} [\tilde{r}_{\mathrm{flat}}(x)]_{t}\cdot\frac{K}{K-1}\left(\mathcal{R}_{t}(x;f_{t}^{*})-\mathcal{R}_{t^{*}}(x;f_{t^{*}}^{*})\right) \\
&\underset{(iii)}{\ge} \frac{K}{T(K-1)}\Delta_{\mathrm{select}}(x),
\end{aligned} τ = 1 ∑ T [ r ~ flat ( x ) ] τ Δ select hinge ( x , τ ) = τ = 1 ∑ T [ r ~ flat ( x ) ] τ ( R τ hinge ( x ; f τ ∗ ) − R t ∗ hinge ( x ; f t ∗ ∗ ) ) ( i ) ≥ τ = 1 ∑ T [ r ~ flat ( x ) ] τ ⋅ K − 1 K ( R τ ( x ; f τ ∗ ) − R t ∗ ( x ; f t ∗ ∗ ) ) ( ii ) ≥ [ r ~ flat ( x ) ] t ⋅ K − 1 K ( R t ( x ; f t ∗ ) − R t ∗ ( x ; f t ∗ ∗ ) ) ( iii ) ≥ T ( K − 1 ) K Δ select ( x ) ,
⟹ Δ s e l e c t ( x ) ≤ T ( K − 1 ) K ∑ τ = 1 T [ r ~ f l a t ( x ) ] τ Δ s e l e c t h i n g e ( x , τ ) . (71) \Longrightarrow\quad\Delta_{\mathrm{select}}(x) \le\frac{T(K-1)}{K}\sum_{\tau=1}^{T}[\tilde{r}_{\mathrm{flat}}(x)]_{\tau}\Delta^{\mathrm{hinge}}_{\mathrm{select}}(x,\tau). \tag*{(71)} ⟹ Δ select ( x ) ≤ K T ( K − 1 ) τ = 1 ∑ T [ r ~ flat ( x ) ] τ Δ select hinge ( x , τ ) . (71)
where step (i) applies (65), step (ii) holds because [ r ~ f l a t ( x ) ] τ ≥ 0 [\tilde{r}_{\mathrm{flat}}(x)]_{\tau} \ge0 [ r ~ flat ( x ) ] τ ≥ 0 for all τ ∈ [ T ] \tau\in[T] τ ∈ [ T ] and, by the definition of t ∗ t^{*} t ∗ in (59), R τ ( x ; f τ ∗ ) − R t ∗ ( x ; f t ∗ ∗ ) ≥ 0 \mathcal{R}_{\tau}(x;f_{\tau}^{*})-\mathcal{R}_{t^{*}}(x;f_{t^{*}}^{*}) \ge0 R τ ( x ; f τ ∗ ) − R t ∗ ( x ; f t ∗ ∗ ) ≥ 0 for all τ ∈ [ T ] \tau\in[T] τ ∈ [ T ] , and step (iii) holds because [ r ~ f l a t ( x ) ] t ≥ 1 T [\tilde{r}_{\mathrm{flat}}(x)]_{t} \ge\frac{1}{T} [ r ~ flat ( x ) ] t ≥ T 1 and Δ s e l e c t ( x ) = R t ( x ; f t ∗ ) − R t ∗ ( x ; f t ∗ ∗ ) ≥ 0 \Delta_{\mathrm{select}}(x)=\mathcal{R}_{t}(x;f_{t}^{*})-\mathcal{R}_{t^{*}}(x;f_{t^{*}}^{*}) \ge0 Δ select ( x ) = R t ( x ; f t ∗ ) − R t ∗ ( x ; f t ∗ ∗ ) ≥ 0 .
Part 4: Combining all components. We have so far bounded the selection and internal excess 0-1-c risks by their excess hinge risk counterparts, respectively. The final step is to combine these components (Eq. (70) and (71) ) to obtain
R t ( x ; f t ) − R t ∗ ( x ; f t ∗ ∗ ) = Δ i n t e r n a l ( x ) + Δ s e l e c t ( x ) ≤ T ( K − 1 ) ∑ τ = 1 T [ r ~ f l a t ( x ) ] τ ( Δ i n t e r n a l h i n g e ( x , τ ) + Δ s e l e c t h i n g e ( x , τ ) ) = T ( K − 1 ) ( R f l a t h i n g e ( x ; π ~ ) − R f l a t h i n g e ( x ; π ~ ∗ ) ) . (72) \begin{aligned}
\mathcal{R}_{t}(x;f_{t})-\mathcal{R}_{t^{*}}(x;f_{t^{*}}^{*})=\Delta_{\mathrm{internal}}(x)+\Delta_{\mathrm{select}}(x) \\
&\le T(K-1)\sum_{\tau=1}^{T}[\tilde{r}_{\mathrm{flat}}(x)]_{\tau}\left(\Delta^{\mathrm{hinge}}_{\mathrm{internal}}(x,\tau)+\Delta^{\mathrm{hinge}}_{\mathrm{select}}(x,\tau)\right) \\
&= T(K-1)\left(\mathcal{R}^{\mathrm{hinge}}_{\mathrm{flat}}(x;\tilde{\pi})-\mathcal{R}^{\mathrm{hinge}}_{\mathrm{flat}}(x;\tilde{\pi}^{*})\right). \tag*{(72)}
\end{aligned} R t ( x ; f t ) − R t ∗ ( x ; f t ∗ ∗ ) = Δ internal ( x ) + Δ select ( x ) ≤ T ( K − 1 ) τ = 1 ∑ T [ r ~ flat ( x ) ] τ ( Δ internal hinge ( x , τ ) + Δ select hinge ( x , τ ) ) = T ( K − 1 ) ( R flat hinge ( x ; π ~ ) − R flat hinge ( x ; π ~ ∗ ) ) . (72)
This concludes the proof of (60).
Proofs for the Cascade Model
In this section, we prove the calibration result in Theorem 3.2 for the cascade hinge loss ℓ c a s c h i n g e \ell^{\mathrm{hinge}}_{\mathrm{casc}} ℓ casc hinge . Similar to (42)–(43) of the flat case, we first define conditional 0-1-c and hinge risks as
R t ( h t ; f t ) ≔ E [ 1 { f t ( X , Z t ) ≠ Y } + c t ∣ H t = h t ] , (73) \mathcal{R}_t(h_t; f_t) \coloneqq\mathbb{E}\left[\mathbb{1}\{f_t(X,Z_t) \ne Y\} + c_t \mid H_t = h_t\right], \tag*{(73)} R t ( h t ; f t ) : = E [ 1 { f t ( X , Z t ) = Y } + c t ∣ H t = h t ] , (73)
R t h i n g e ( h t ; f ~ t ) ≔ E [ ℓ h i n g e ( f ~ t ( X , Z t ) , Y ; c t ) ∣ H t = h t ] . (74) \mathcal{R}^{\mathrm{hinge}}_t(h_t; \tilde{f}_t) \coloneqq\mathbb{E}\left[\ell^{\mathrm{hinge}}(\tilde{f}_t(X,Z_t),Y;c_t) \mid H_t = h_t\right]. \tag*{(74)} R t hinge ( h t ; f ~ t ) : = E [ ℓ hinge ( f ~ t ( X , Z t ) , Y ; c t ) ∣ H t = h t ] . (74)
As elsewhere in the paper, we suppress the ‘0-1-c’ superscript and use the ‘hinge’ superscript only to distinguish hinge quantities. Thus, R t \mathcal{R}_t R t means R t 0 - 1 - c \mathcal{R}^{0\text{-}1\text{-}c}_t R t 0 - 1 - c , and the same naming convention applies to other quantities throughout. The calibration proof relies on value functions , a concept standard in the dynamic programming literature which we restate below for completeness.
Definition C.1 (Value function). For a cascade classifier π = ( f 1 : T , d 1 : T ) \pi=(f_{1:T},d_{1:T}) π = ( f 1 : T , d 1 : T ) , a loss ∈ { 0 - 1 - c , h i n g e } \in\{0\text{-}1\text{-}c,\mathrm{hinge}\} ∈ { 0 - 1 - c , hinge } , and the stage index t ∈ [ T ] t\in[T] t ∈ [ T ] , we define the stage-t t t value function V t π , l o s s ( h t ) V_t^{\pi,\mathrm{loss}}(h_t) V t π , loss ( h t ) as the expected cost-to-go from stage t t t onward, given the current history h t h_t h t . Formally, define
V t π , l o s s ( h t ) ≔ ( 1 − d t ( h t ) ) R t l o s s ( h t ; f t ) + d t ( h t ) E [ V t + 1 π , l o s s ( H t + 1 ) ∣ H t = h t ] , for t < T (75) V_t^{\pi,\mathrm{loss}}(h_t) \coloneqq(1-d_t(h_t))\mathcal{R}^{\mathrm{loss}}_t(h_t;f_t)+d_t(h_t)\mathbb{E}\left[V_{t+1}^{\pi,\mathrm{loss}}(H_{t+1})\mid H_t=h_t\right], \qquad\text{for }t<T \tag*{(75)} V t π , loss ( h t ) : = ( 1 − d t ( h t )) R t loss ( h t ; f t ) + d t ( h t ) E [ V t + 1 π , loss ( H t + 1 ) ∣ H t = h t ] , for t < T (75) and
V T π , l o s s ( h T ) ≔ R T l o s s ( h T ; f T ) . (76) V_T^{\pi,\mathrm{loss}}(h_T) \coloneqq\mathcal{R}^{\mathrm{loss}}_T(h_T;f_T). \tag*{(76)} V T π , loss ( h T ) : = R T loss ( h T ; f T ) . (76) The recursion in (75) reflects the two possible actions at stage t t t given current history h t h_t h t : if the deferral function exits (d t ( h t ) = 0 d_t(h_t)=0 d t ( h t ) = 0 ), it incurs the immediate conditional risk R t l o s s ( h t ; f t ) \mathcal{R}^{\mathrm{loss}}_t(h_t;f_t) R t loss ( h t ; f t ) ; if it defers (d t ( h t ) = 1 d_t(h_t)=1 d t ( h t ) = 1 ), it incurs the expected future cost starting from stage t + 1 t+1 t + 1 . At the terminal stage, there is no further stage to defer to, and the value function reduces to the immediate risk, yielding (76) . Thus, V t π , l o s s ( h t ) V_t^{\pi,\mathrm{loss}}(h_t) V t π , loss ( h t ) represents the total expected risk incurred by following π = ( f 1 : T , d 1 : T ) \pi=(f_{1:T},d_{1:T}) π = ( f 1 : T , d 1 : T ) from stage t t t onward, conditional on h t h_t h t .
Notation: Since the 0-1-c loss is only used with hard policies (e.g., π \pi π and π c a s c ∗ \pi^*_{\mathrm{casc}} π casc ∗ ) and the hinge loss only with soft policies (e.g., π ~ \tilde{\pi} π ~ and π ~ c a s c ∗ \tilde{\pi}^*_{\mathrm{casc}} π ~ casc ∗ ), we drop the superscript ‘loss’ from V t π , l o s s ( h t ) V_t^{\pi,\mathrm{loss}}(h_t) V t π , loss ( h t ) for brevity, as the associated loss will be clear from context. As in Appendix B, we use LHS (resp. RHS) as short for left (resp. right) hand side; we write p ( ⋅ ∣ ⋅ ) p(\cdot\mid\cdot) p ( ⋅ ∣ ⋅ ) for the conditional probability mass function of Y Y Y , e.g., p ( y ∣ x ) ≡ Pr ( Y = y ∣ X = x ) p(y\mid x)\equiv\Pr(Y=y\mid X=x) p ( y ∣ x ) ≡ Pr ( Y = y ∣ X = x ) and p ( y ∣ x , z t ) ≡ Pr ( Y = y ∣ X = x , Z t = z t ) p(y\mid x,z_t)\equiv\Pr(Y=y\mid X=x,Z_t=z_t) p ( y ∣ x , z t ) ≡ Pr ( Y = y ∣ X = x , Z t = z t ) .
Proof sketch: This section is organized as follows. We first show that the 0-1-c and hinge risks for cascades can be expressed in terms of the stage-1 value functions (Section C.1). Building on this connection, we characterize the Bayes-optimal solutions under both the 0-1-c loss and cascade hinge loss (Section C.2). Next, we establish two useful results linking hinge and 0-1-c quantities (Section C.3). Using this, we complete the proof of the calibration result in Theorem 3.2 (Section C.4).
Connection of Risks to Expected Value Functions
In this section, we establish an essential connection between the global risk of a cascade classifier π = ( f 1 : T , d 1 : T ) \pi=(f_{1:T},d_{1:T}) π = ( f 1 : T , d 1 : T ) and its associated value functions. Specifically, Lemma C.2 shows that both the 0-1-c risk R ( π ) \mathcal{R}(\pi) R ( π ) and hinge risk R c a s c h i n g e ( π ~ ) \mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi}) R casc hinge ( π ~ ) can be expressed as expected value functions at the initial stage for arbitrary π ∈ Π c a s c \pi\in\Pi_{\mathrm{casc}} π ∈ Π casc and π ~ ∈ Π ~ c a s c \tilde{\pi}\in\widetilde{\Pi}_{\mathrm{casc}} π ~ ∈ Π casc , and Corollary C.3 specializes this result to their Bayes-optimal counterparts π c a s c ∗ ∈ Π c a s c \pi^{*}_{\mathrm{casc}} \in\Pi_{\mathrm{casc}} π casc ∗ ∈ Π casc and π ~ c a s c ∗ ∈ Π ~ c a s c \widetilde{\pi}^{*}_{\mathrm{casc}} \in\widetilde{\Pi}_{\mathrm{casc}} π casc ∗ ∈ Π casc . These connections allow us to analyze the risks through the recursive structure of the cascade, and serves as the foundation for characterizing Bayes-optimal solutions in the next section and, ultimately, for proving calibration.
Lemma C.2 (Connection of risks to value functions for arbitrary π \pi π and π ~ \widetilde{\pi} π ). For any hard cascade classifier π ∈ Π c a s c \pi\in\Pi_{\mathrm{casc}} π ∈ Π casc , we have
R ( π ) = E [ V 1 π ( H 1 ) ] , (77) \mathcal{R}(\pi) = \mathbb{E}\left[V_{1}^{\pi}(H_{1})\right], \tag*{(77)} R ( π ) = E [ V 1 π ( H 1 ) ] , (77) and similarly, for any soft cascade classifier π ~ ∈ Π ~ c a s c \widetilde{\pi} \in\widetilde{\Pi}_{\mathrm{casc}} π ∈ Π casc , we have
R c a s c h i n g e ( π ~ ) = E [ V 1 π ~ ( H 1 ) ] . (78) \mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}}(\widetilde{\pi}) = \mathbb{E}\left[V_{1}^{\widetilde{\pi}}(H_{1})\right]. \tag*{(78)} R casc hinge ( π ) = E [ V 1 π ( H 1 ) ] . (78)
Proof. Recall we write H t = ( X , Z 1 : t ) H_t = (X, Z_{1:t}) H t = ( X , Z 1 : t ) and H ≡ H T = ( X , Z 1 : T ) H \equiv H_T = (X, Z_{1:T}) H ≡ H T = ( X , Z 1 : T ) . Let π = ( f 1 : T , r c a s c ) \pi= (f_{1:T}, r_{\mathrm{casc}}) π = ( f 1 : T , r casc ) , where the stopping rule is defined as r c a s c ( h ) = min { t ∈ [ T ] : d t ( h t ) = 0 } r_{\mathrm{casc}}(h) = \min\{t \in[T] : d_t(h_t) = 0\} r casc ( h ) = min { t ∈ [ T ] : d t ( h t ) = 0 } . By the definition of the 0-1-c c c risk R ( π ) \mathcal{R}(\pi) R ( π ) in (9)–(10) and the tower property, we can write
R ( π ) = E [ 1 { f r c a s c ( H ) ( X , Z r c a s c ( H ) ) ≠ Y } + c r c a s c ( H ) ] = ∑ t = 1 T E [ 1 { r c a s c ( H ) = t } ⋅ ( 1 { f t ( X , Z t ) ≠ Y } + c t ) ] = ∑ t = 1 T E [ E [ 1 { r c a s c ( H ) = t } ⋅ ( 1 { f t ( X , Z t ) ≠ Y } + c t ) ∣ H t ] ] = ∑ t = 1 T E [ ( ∏ τ = 1 t − 1 d τ ( H τ ) ) ( 1 − d t ( H t ) ) ⋅ E [ 1 { f t ( X , Z t ) ≠ Y } + c t ∣ H t ] ] = ∑ t = 1 T E [ ( ∏ τ = 1 t − 1 d τ ( H τ ) ) ( 1 − d t ( H t ) ) ⋅ R t ( H t ; f t ) ] = E [ ∑ t = 1 T ( ∏ τ = 1 t − 1 d τ ( H τ ) ) ( 1 − d t ( H t ) ) ⋅ R t ( H t ; f t ) ] . \begin{aligned}
\mathcal{R}(\pi)
&= \mathbb{E}\left[\mathbf{1}\{f_{r_{\mathrm{casc}}(H)}(X,Z_{r_{\mathrm{casc}}(H)}) \ne Y\} + c_{r_{\mathrm{casc}}(H)}\right] \\
&= \sum_{t=1}^{T} \mathbb{E}\left[\mathbf{1}\{r_{\mathrm{casc}}(H)=t\}\cdot\left(\mathbf{1}\{f_t(X,Z_t)\ne Y\}+c_t\right)\right] \\
&= \sum_{t=1}^{T} \mathbb{E}\left[\mathbb{E}\left[\mathbf{1}\{r_{\mathrm{casc}}(H)=t\}\cdot\left(\mathbf{1}\{f_t(X,Z_t)\ne Y\}+c_t\right)\mid H_t\right]\right] \\
&= \sum_{t=1}^{T} \mathbb{E}\left[\left(\prod_{\tau=1}^{t-1}d_{\tau}(H_{\tau})\right)(1-d_t(H_t))\cdot\mathbb{E}\left[\mathbf{1}\{f_t(X,Z_t)\ne Y\}+c_t\mid H_t\right]\right] \\
&= \sum_{t=1}^{T} \mathbb{E}\left[\left(\prod_{\tau=1}^{t-1}d_{\tau}(H_{\tau})\right)(1-d_t(H_t))\cdot\mathcal{R}_t(H_t;f_t)\right] \\
&= \mathbb{E}\left[\sum_{t=1}^{T}\left(\prod_{\tau=1}^{t-1}d_{\tau}(H_{\tau})\right)(1-d_t(H_t))\cdot\mathcal{R}_t(H_t;f_t)\right].
\end{aligned} R ( π ) = E [ 1 { f r casc ( H ) ( X , Z r casc ( H ) ) = Y } + c r casc ( H ) ] = t = 1 ∑ T E [ 1 { r casc ( H ) = t } ⋅ ( 1 { f t ( X , Z t ) = Y } + c t ) ] = t = 1 ∑ T E [ E [ 1 { r casc ( H ) = t } ⋅ ( 1 { f t ( X , Z t ) = Y } + c t ) ∣ H t ] ] = t = 1 ∑ T E [ ( τ = 1 ∏ t − 1 d τ ( H τ ) ) ( 1 − d t ( H t )) ⋅ E [ 1 { f t ( X , Z t ) = Y } + c t ∣ H t ] ] = t = 1 ∑ T E [ ( τ = 1 ∏ t − 1 d τ ( H τ ) ) ( 1 − d t ( H t )) ⋅ R t ( H t ; f t ) ] = E [ t = 1 ∑ T ( τ = 1 ∏ t − 1 d τ ( H τ ) ) ( 1 − d t ( H t )) ⋅ R t ( H t ; f t ) ] .
Therefore, to prove (76), it suffices to show that
E [ ∑ t = 1 T ( ∏ τ = 1 t − 1 d τ ( H τ ) ) ( 1 − d t ( H t ) ) ⋅ R t ( H t ; f t ) ] = E [ V 1 π ( H 1 ) ] , (79) \mathbb{E}\left[\sum_{t=1}^{T}\left(\prod_{\tau=1}^{t-1}d_{\tau}(H_{\tau})\right)(1-d_t(H_t))\cdot\mathcal{R}_t(H_t;f_t)\right] = \mathbb{E}\left[V_{1}^{\pi}(H_{1})\right], \tag*{(79)} E [ t = 1 ∑ T ( τ = 1 ∏ t − 1 d τ ( H τ ) ) ( 1 − d t ( H t )) ⋅ R t ( H t ; f t ) ] = E [ V 1 π ( H 1 ) ] , (79)
which in turn implies that, it suffices to show that for all i ∈ [ T ] i \in[T] i ∈ [ T ] ,
E [ ∑ t = i T ( ∏ τ = i t − 1 d τ ( H τ ) ) ( 1 − d t ( H t ) ) ⋅ R t ( H t ; f t ) ∣ H i ] = V i π ( H i ) , (80) \mathbb{E}\left[\left.\sum_{t=i}^{T}\left(\prod_{\tau=i}^{t-1}d_{\tau}(H_{\tau})\right)(1-d_t(H_t))\cdot\mathcal{R}_t(H_t;f_t)\right|H_i\right] = V_i^{\pi}(H_i), \tag*{(80)} E [ t = i ∑ T ( τ = i ∏ t − 1 d τ ( H τ ) ) ( 1 − d t ( H t )) ⋅ R t ( H t ; f t ) H i ] = V i π ( H i ) , (80)
We prove (79) using backward induction.
Base case i = T i=T i = T : To show (79) holds for i = T i=T i = T , note that by the definition of the stage-T T T value function in (75),
V T π ( H T ) = R T ( H T ; f T ) = E [ ∑ t = T T ( ∏ τ = T t − 1 d τ ( H τ ) ) ( 1 − d t ( H t ) ) ⋅ R t ( H t ; f t ) ∣ H T ] , V_T^{\pi}(H_T) = \mathcal{R}_T(H_T;f_T) = \mathbb{E}\left[\left.\sum_{t=T}^{T}\left(\prod_{\tau=T}^{t-1}d_{\tau}(H_{\tau})\right)(1-d_t(H_t))\cdot\mathcal{R}_t(H_t;f_t)\right|H_T\right], V T π ( H T ) = R T ( H T ; f T ) = E [ t = T ∑ T ( τ = T ∏ t − 1 d τ ( H τ ) ) ( 1 − d t ( H t )) ⋅ R t ( H t ; f t ) H T ] ,
where we used the fact that for a > b a>b a > b , ∏ τ = a b d τ ( H τ ) = 1 \prod_{\tau=a}^{b}d_{\tau}(H_{\tau})=1 ∏ τ = a b d τ ( H τ ) = 1 .
Backward induction: Now, suppose (80) holds for i = s + 1 i=s+1 i = s + 1 , we next prove it holds for i = s i=s i = s . Using the recursive Bellman definition of the value function (75) , we can write
V s π ( H s ) = ( 1 − d s ( H s ) ) R s ( H s ; f s ) + d s ( H s ) E [ V s + 1 π ( H s + 1 ) ∣ H s ] = ( 1 − d s ( H s ) ) R s ( H s ; f s ) + d s ( H s ) E [ E [ ∑ t = s + 1 T ( ∏ τ = s + 1 t − 1 d τ ( H τ ) ) ( 1 − d t ( H t ) ) ⋅ R t ( H t ; f t ) ∣ H s + 1 ] ∣ H s ] = ( 1 − d s ( H s ) ) R s ( H s ; f s ) + d s ( H s ) E [ ∑ t = s + 1 T ( ∏ τ = s + 1 t − 1 d τ ( H τ ) ) ( 1 − d t ( H t ) ) ⋅ R t ( H t ; f t ) ∣ H s ] = E [ ∑ t = s T ( ∏ τ = s t − 1 d τ ( H τ ) ) ( 1 − d t ( H t ) ) ⋅ R t ( H t ; f t ) ∣ H s ] , \begin{aligned}
V_s^\pi(H_s) &= (1-d_s(H_s))R_s(H_s;f_s)+d_s(H_s)\mathbb{E}\left[V_{s+1}^\pi(H_{s+1})\mid H_s\right]\\
&=(1-d_s(H_s))R_s(H_s;f_s)+\\
&\quad d_s(H_s)\mathbb{E}\left[\mathbb{E}\left[\sum_{t=s+1}^{T}\left(\prod_{\tau=s+1}^{t-1}d_\tau(H_\tau)\right)(1-d_t(H_t))\cdot R_t(H_t;f_t)\mid H_{s+1}\right]\mid H_s\right]\\
&=(1-d_s(H_s))R_s(H_s;f_s)+\\
&\quad d_s(H_s)\mathbb{E}\left[\sum_{t=s+1}^{T}\left(\prod_{\tau=s+1}^{t-1}d_\tau(H_\tau)\right)(1-d_t(H_t))\cdot R_t(H_t;f_t)\mid H_s\right]\\
&=\mathbb{E}\left[\sum_{t=s}^{T}\left(\prod_{\tau=s}^{t-1}d_\tau(H_\tau)\right)(1-d_t(H_t))\cdot R_t(H_t;f_t)\mid H_s\right],
\end{aligned} V s π ( H s ) = ( 1 − d s ( H s )) R s ( H s ; f s ) + d s ( H s ) E [ V s + 1 π ( H s + 1 ) ∣ H s ] = ( 1 − d s ( H s )) R s ( H s ; f s ) + d s ( H s ) E [ E [ t = s + 1 ∑ T ( τ = s + 1 ∏ t − 1 d τ ( H τ ) ) ( 1 − d t ( H t )) ⋅ R t ( H t ; f t ) ∣ H s + 1 ] ∣ H s ] = ( 1 − d s ( H s )) R s ( H s ; f s ) + d s ( H s ) E [ t = s + 1 ∑ T ( τ = s + 1 ∏ t − 1 d τ ( H τ ) ) ( 1 − d t ( H t )) ⋅ R t ( H t ; f t ) ∣ H s ] = E [ t = s ∑ T ( τ = s ∏ t − 1 d τ ( H τ ) ) ( 1 − d t ( H t )) ⋅ R t ( H t ; f t ) ∣ H s ] ,
where the last step absorbs the t = s t=s t = s term into the summation, yielding (80) for i = s i=s i = s .
Conclusion: Applying this backward induction argument down from i = T − 1 i=T-1 i = T − 1 to i = 1 i=1 i = 1 yields (80) for all i ∈ [ T ] i\in[T] i ∈ [ T ] . Setting i = 1 i=1 i = 1 in (80) and taking total expectation over H 1 H_1 H 1 proves (79) , which gives (77) .
In the remainder of the proof, we establish (78) . Let π ~ = ( f ~ 1 : T , d ~ 1 : T ) \tilde{\pi}=(\tilde{f}_{1:T},\tilde{d}_{1:T}) π ~ = ( f ~ 1 : T , d ~ 1 : T ) . By definition of R c a s c h i n g e ( π ~ ) R_{\mathrm{casc}}^{\mathrm{hinge}}(\tilde{\pi}) R casc hinge ( π ~ ) in (18) and tower property,
R c a s c h i n g e ( π ~ ) = E [ ℓ c a s c h i n g e ( X , Z 1 : T , Y ; π ~ , c ) ] = ∑ t = 1 T E [ ( ∏ τ = 1 t − 1 d ~ τ ( H τ ) ) ( 1 − d ~ t ( H t ) ) E [ ℓ h i n g e ( f ~ t ( X , Z t ) , Y ; c t ) ∣ H t ] ] = E [ ∑ t = 1 T ( ∏ τ = 1 t − 1 d ~ τ ( H τ ) ) ( 1 − d ~ t ( H t ) ) ⋅ R t h i n g e ( H t ; f ~ t ) ] , \begin{aligned}
R_{\mathrm{casc}}^{\mathrm{hinge}}(\tilde{\pi})
&=\mathbb{E}\left[\ell_{\mathrm{casc}}^{\mathrm{hinge}}(X,Z_{1:T},Y;\tilde{\pi},c)\right]\\
&=\sum_{t=1}^{T}\mathbb{E}\left[\left(\prod_{\tau=1}^{t-1}\tilde{d}_{\tau}(H_{\tau})\right)(1-\tilde{d}_{t}(H_t))\mathbb{E}\left[\ell^{\mathrm{hinge}}(\tilde{f}_{t}(X,Z_t),Y;c_t)\mid H_t\right]\right]\\
&=\mathbb{E}\left[\sum_{t=1}^{T}\left(\prod_{\tau=1}^{t-1}\tilde{d}_{\tau}(H_{\tau})\right)(1-\tilde{d}_{t}(H_t))\cdot R_t^{\mathrm{hinge}}(H_t;\tilde{f}_t)\right],
\end{aligned} R casc hinge ( π ~ ) = E [ ℓ casc hinge ( X , Z 1 : T , Y ; π ~ , c ) ] = t = 1 ∑ T E [ ( τ = 1 ∏ t − 1 d ~ τ ( H τ ) ) ( 1 − d ~ t ( H t )) E [ ℓ hinge ( f ~ t ( X , Z t ) , Y ; c t ) ∣ H t ] ] = E [ t = 1 ∑ T ( τ = 1 ∏ t − 1 d ~ τ ( H τ ) ) ( 1 − d ~ t ( H t )) ⋅ R t hinge ( H t ; f ~ t ) ] ,
so to prove (78) , it suffices to prove
E [ ∑ t = 1 T ( ∏ τ = 1 t − 1 d ~ τ ( H τ ) ) ( 1 − d ~ t ( H t ) ) ⋅ R t h i n g e ( H t ; f ~ t ) ] = E [ V 1 π ~ ( H 1 ) ] , \mathbb{E}\left[\sum_{t=1}^{T}\left(\prod_{\tau=1}^{t-1}\tilde{d}_{\tau}(H_{\tau})\right)(1-\tilde{d}_{t}(H_t))\cdot R_t^{\mathrm{hinge}}(H_t;\tilde{f}_t)\right]=\mathbb{E}\left[V_1^{\tilde{\pi}}(H_1)\right], E [ t = 1 ∑ T ( τ = 1 ∏ t − 1 d ~ τ ( H τ ) ) ( 1 − d ~ t ( H t )) ⋅ R t hinge ( H t ; f ~ t ) ] = E [ V 1 π ~ ( H 1 ) ] ,
which in turn implies it suffices to prove for all i ∈ [ T ] i\in[T] i ∈ [ T ] ,
E [ ∑ t = i T ( ∏ τ = i t − 1 d ~ τ ( H τ ) ) ( 1 − d ~ t ( H t ) ) ⋅ R t h i n g e ( H t ; f ~ t ) ∣ H i ] = V i π ~ ( H i ) . (81) \mathbb{E}\left[\sum_{t=i}^{T}\left(\prod_{\tau=i}^{t-1}\tilde{d}_{\tau}(H_{\tau})\right)(1-\tilde{d}_{t}(H_t))\cdot R_t^{\mathrm{hinge}}(H_t;\tilde{f}_t)\mid H_i\right]=V_i^{\tilde{\pi}}(H_i). \tag*{(81)} E [ t = i ∑ T ( τ = i ∏ t − 1 d ~ τ ( H τ ) ) ( 1 − d ~ t ( H t )) ⋅ R t hinge ( H t ; f ~ t ) ∣ H i ] = V i π ~ ( H i ) . (81)
This is analogous to (80) . The proof works verbatim because the Bellman recursion and tower rule do not depend on whether d ~ t \tilde{d}_t d ~ t lies in { 0 , 1 } \{0,1\} { 0 , 1 } or [ 0 , 1 ] [0,1] [ 0 , 1 ] . □
Lemma C.2 connects the risks to the value functions for arbitrary cascade classifiers π \pi π and π ~ \tilde{\pi} π ~ . Specializing this result to the optimal cascade classifiers π c a s c ∗ \pi_{\mathrm{casc}}^* π casc ∗ and π ~ c a s c ∗ \tilde{\pi}_{\mathrm{casc}}^* π ~ casc ∗ yields a corresponding connection between the Bayes risks and the value functions of the optimal cascade classifiers. This is stated in Corollary C.3.
Corollary C.3 (Connection of risks to value functions for Bayes-optimal π c a s c ∗ \pi^\ast_{\mathrm{casc}} π casc ∗ and π ~ c a s c ∗ \widetilde{\pi}^\ast_{\mathrm{casc}} π casc ∗ ). Let π c a s c ∗ = ( f 1 : T ∗ , d 1 : T ∗ ) ∈ arg min π ∈ Π c a s c R ( π ) \pi^\ast_{\mathrm{casc}}=(f^\ast_{1:T},d^\ast_{1:T})\in\operatorname*{arg\,min}_{\pi\in\Pi_{\mathrm{casc}}}\mathcal{R}(\pi) π casc ∗ = ( f 1 : T ∗ , d 1 : T ∗ ) ∈ arg min π ∈ Π casc R ( π ) be as defined in Lemma C.4, then
R c a s c ∗ = E [ V 1 π c a s c ∗ ( H 1 ) ] . \mathcal{R}^\ast_{\mathrm{casc}}=\mathbb{E}\left[V^{\pi^\ast_{\mathrm{casc}}}_1(H_1)\right]. R casc ∗ = E [ V 1 π casc ∗ ( H 1 ) ] . Similarly, let π ~ c a s c ∗ = ( f ~ 1 : T ∗ , d ~ 1 : T ∗ ) ∈ arg min π ~ ∈ Π ~ c a s c R c a s c h i n g e ( π ~ ) \widetilde{\pi}^\ast_{\mathrm{casc}}=(\widetilde{f}^\ast_{1:T},\widetilde{d}^\ast_{1:T})\in\operatorname*{arg\,min}_{\widetilde{\pi}\in\widetilde{\Pi}_{\mathrm{casc}}}\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\widetilde{\pi}) π casc ∗ = ( f 1 : T ∗ , d 1 : T ∗ ) ∈ arg min π ∈ Π casc R casc hinge ( π ) be as defined in Lemma C.5, then
R c a s c h i n g e ∗ = E [ V 1 π ~ c a s c ∗ ( H 1 ) ] . \mathcal{R}^{\mathrm{hinge}\ast}_{\mathrm{casc}}=\mathbb{E}\left[V^{\widetilde{\pi}^\ast_{\mathrm{casc}}}_1(H_1)\right]. R casc hinge ∗ = E [ V 1 π casc ∗ ( H 1 ) ] . Proof. By (77) , which relates R \mathcal{R} R to V 1 π V^\pi_1 V 1 π for an arbitrary hard cascade classifier π \pi π , we can write
R c a s c ∗ : = min π ∈ Π c a s c R ( π ) = R ( π c a s c ∗ ) = E [ V 1 π c a s c ∗ ( H 1 ) ] . \mathcal{R}^\ast_{\mathrm{casc}}:=\min_{\pi\in\Pi_{\mathrm{casc}}}\mathcal{R}(\pi)=\mathcal{R}(\pi^\ast_{\mathrm{casc}})=\mathbb{E}\left[V^{\pi^\ast_{\mathrm{casc}}}_1(H_1)\right]. R casc ∗ := π ∈ Π casc min R ( π ) = R ( π casc ∗ ) = E [ V 1 π casc ∗ ( H 1 ) ] . By (78) , which relates R c a s c h i n g e \mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}} R casc hinge to V 1 π ~ V^{\widetilde{\pi}}_1 V 1 π for an arbitrary soft cascade classifier π ~ \widetilde{\pi} π , we can write
R c a s c h i n g e ∗ : = min π ~ ∈ Π ~ c a s c R c a s c h i n g e ( π ~ ) = R c a s c h i n g e ( π ~ c a s c ∗ ) = E [ V 1 π ~ c a s c ∗ ( H 1 ) ] . \mathcal{R}^{\mathrm{hinge}\ast}_{\mathrm{casc}}:=\min_{\widetilde{\pi}\in\widetilde{\Pi}_{\mathrm{casc}}}\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\widetilde{\pi})=\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\widetilde{\pi}^\ast_{\mathrm{casc}})=\mathbb{E}\left[V^{\widetilde{\pi}^\ast_{\mathrm{casc}}}_1(H_1)\right]. R casc hinge ∗ := π ∈ Π casc min R casc hinge ( π ) = R casc hinge ( π casc ∗ ) = E [ V 1 π casc ∗ ( H 1 ) ] .
Bayes-Optimal Solutions
Using the connections of risks to expected value functions proven in the previous section, we now characterize the Bayes-optimal solutions under the 0-1-c c c loss and cascade hinge loss, respectively.
Lemma C.4 (Bayes-optimal solution in Π c a s c \Pi_{\mathrm{casc}} Π casc for 0-1-c c c loss). Assume Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) Y\perp Z_{1:t-1}\mid(X,Z_t) Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) as in (5) . For the 0-1-c c c risk R \mathcal{R} R with cost vector c = [ κ 1 , κ 1 + κ 2 , … , ∑ t = 1 T κ t ] c=[\kappa_1,\kappa_1+\kappa_2,\ldots,\sum_{t=1}^{T}\kappa_t] c = [ κ 1 , κ 1 + κ 2 , … , ∑ t = 1 T κ t ] , there exists a Bayes-optimal cascade classifier
π c a s c ∗ = ( f 1 : T ∗ , r c a s c ∗ ) ∈ arg min π ∈ Π c a s c R ( π ) , (82) \pi^\ast_{\mathrm{casc}}=(f^\ast_{1:T},r^\ast_{\mathrm{casc}})\in\operatorname*{arg\,min}_{\pi\in\Pi_{\mathrm{casc}}}\mathcal{R}(\pi), \tag*{(82)} π casc ∗ = ( f 1 : T ∗ , r casc ∗ ) ∈ π ∈ Π casc arg min R ( π ) , (82) where f 1 : T ∗ f^\ast_{1:T} f 1 : T ∗ satisfies (30) , and r c a s c ∗ ( h ) : = min { t ∈ [ T ] : d t ∗ ( h t ) = 0 } r^\ast_{\mathrm{casc}}(h):=\min\{t\in[T]:d^\ast_t(h_t)=0\} r casc ∗ ( h ) := min { t ∈ [ T ] : d t ∗ ( h t ) = 0 } is induced by the deferral rules d 1 : T ∗ d^\ast_{1:T} d 1 : T ∗ given by
d t ∗ ( h t ) = 1 { R t ( h t ; f t ∗ ) > E [ V t + 1 π c a s c ∗ ( H t + 1 ) ∣ H t = h t ] } ∀ t < T , and d T ∗ ( h T ) = 0 , (83) d^\ast_t(h_t)=\mathbb{1}\left\{\mathcal{R}_t(h_t;f^\ast_t)>\mathbb{E}\left[V^{\pi^\ast_{\mathrm{casc}}}_{t+1}(H_{t+1})\mid H_t=h_t\right]\right\}\quad\forall t<T,\qquad\text{and}\qquad d^\ast_T(h_T)=0, \tag*{(83)} d t ∗ ( h t ) = 1 { R t ( h t ; f t ∗ ) > E [ V t + 1 π casc ∗ ( H t + 1 ) ∣ H t = h t ] } ∀ t < T , and d T ∗ ( h T ) = 0 , (83) where
R t ( h t ; f t ∗ ) = 1 − max y ∈ [ K ] p ( y ∣ x , z t ) + c t , ∀ t ∈ [ T ] (84) \mathcal{R}_t(h_t;f^\ast_t)=1-\max_{y\in[K]}p(y\mid x,z_t)+c_t,\qquad\forall t\in[T] \tag*{(84)} R t ( h t ; f t ∗ ) = 1 − y ∈ [ K ] max p ( y ∣ x , z t ) + c t , ∀ t ∈ [ T ] (84) is the Bayes risk of exiting at stage t t t conditional on H t = h t H_t=h_t H t = h t .
Proof. Throughout, we write H t = ( X , Z 1 : t ) H_t=(X,Z_{1:t}) H t = ( X , Z 1 : t ) and H ≡ H T = ( X , Z 1 : T ) H\equiv H_T=(X,Z_{1:T}) H ≡ H T = ( X , Z 1 : T ) . We first optimize f 1 : T f_{1:T} f 1 : T with d 1 : T d_{1:T} d 1 : T fixed and then optimize d 1 : T d_{1:T} d 1 : T with f 1 : T f_{1:T} f 1 : T fixed.
Part 1: Proof for the Bayes-optimal f 1 : T ∗ f^*_{1:T} f 1 : T ∗ . We first minimize R ( π ) R(\pi) R ( π ) with respect to f 1 : T f_{1:T} f 1 : T for an arbitrary routing function r c a s c r_{\mathrm{casc}} r casc , or equivalently, an arbitrary d 1 : T d_{1:T} d 1 : T . Fix d 1 : T d_{1:T} d 1 : T ; we can expand the 0-1-c risk as follows:
R ( π ) = E [ ℓ ( X , Z 1 : T , Y ; π , c ) ] = E [ 1 { f r c a s c ( H ) ( X , Z r c a s c ( H ) ) ≠ Y } + c r c a s c ( H ) ] = ∑ t = 1 T E [ 1 { r c a s c ( H ) = t } ⋅ ( 1 { f t ( X , Z t ) ≠ Y } + c t ) ] = ( i ) ∑ t = 1 T E [ 1 { r c a s c ( H ) = t } ⋅ ( E [ 1 { f t ( X , Z t ) ≠ Y } ∣ H t ] + c t ) ] . \begin{aligned}
R(\pi) &= \mathbb{E}[\ell(X,Z_{1:T},Y;\pi,c)] \\
&= \mathbb{E}\left[\mathbb{1}\{f_{r_{\mathrm{casc}}(H)}(X,Z_{r_{\mathrm{casc}}(H)})\ne Y\}+c_{r_{\mathrm{casc}}(H)}\right] \\
&= \sum_{t=1}^{T}\mathbb{E}\left[\mathbb{1}\{r_{\mathrm{casc}}(H)=t\}\cdot\left(\mathbb{1}\{f_t(X,Z_t)\ne Y\}+c_t\right)\right] \\
&\mathrel{\overset{(i)}{=}} \sum_{t=1}^{T}\mathbb{E}\left[\mathbb{1}\{r_{\mathrm{casc}}(H)=t\}\cdot\left(\mathbb{E}\left[\mathbb{1}\{f_t(X,Z_t)\ne Y\}\mid H_t\right]+c_t\right)\right].
\end{aligned} R ( π ) = E [ ℓ ( X , Z 1 : T , Y ; π , c )] = E [ 1 { f r casc ( H ) ( X , Z r casc ( H ) ) = Y } + c r casc ( H ) ] = t = 1 ∑ T E [ 1 { r casc ( H ) = t } ⋅ ( 1 { f t ( X , Z t ) = Y } + c t ) ] = ( i ) t = 1 ∑ T E [ 1 { r casc ( H ) = t } ⋅ ( E [ 1 { f t ( X , Z t ) = Y } ∣ H t ] + c t ) ] .
where R t ( H t ; f t ) R_t(H_t;f_t) R t ( H t ; f t ) is the conditional risk of exiting at stage t t t given H t H_t H t , as defined in (73) . Note step (i) holds because 1 { r c a s c ( H ) = t } \mathbb{1}\{r_{\mathrm{casc}}(H)=t\} 1 { r casc ( H ) = t } is measurable with respect to H t H_t H t . Minimizing R ( π ) R(\pi) R ( π ) with respect to f t f_t f t reduces to minimizing R t ( H t ; f t ) R_t(H_t;f_t) R t ( H t ; f t ) pointwise in H t H_t H t . Fix h t ∈ H t h_t\in\mathcal{H}_t h t ∈ H t , using Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) Y\perp Z_{1:t-1}\mid(X,Z_t) Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) from (5) , we have
R t ( h t ; f t ) = E [ 1 { f t ( X , Z t ) ≠ Y } ∣ H t = h t ] + c t = ( i ) E [ 1 { Y ≠ f t ( x , z t ) } ∣ X = x , Z t = z t ] + c t = 1 − p ( f t ( x , z t ) ∣ x , z t ) + c t (85) \begin{aligned}
R_t(h_t;f_t) &= \mathbb{E}\left[\mathbb{1}\{f_t(X,Z_t)\ne Y\}\mid H_t=h_t\right]+c_t \\
&\mathrel{\overset{(i)}{=}} \mathbb{E}\left[\mathbb{1}\{Y\ne f_t(x,z_t)\}\mid X=x,Z_t=z_t\right]+c_t \\
&=1-p(f_t(x,z_t)\mid x,z_t)+c_t \tag*{(85)}
\end{aligned} R t ( h t ; f t ) = E [ 1 { f t ( X , Z t ) = Y } ∣ H t = h t ] + c t = ( i ) E [ 1 { Y = f t ( x , z t )} ∣ X = x , Z t = z t ] + c t = 1 − p ( f t ( x , z t ) ∣ x , z t ) + c t (85)
where recall p ( ⋅ ∣ x , z t ) p(\cdot\mid x,z_t) p ( ⋅ ∣ x , z t ) denotes the conditional probability of Y Y Y given ( X , Z t ) = ( x , z t ) (X,Z_t)=(x,z_t) ( X , Z t ) = ( x , z t ) , and step (i) here holds because Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) Y\perp Z_{1:t-1}\mid(X,Z_t) Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) by the assumption in (5) . Hence, minimizing R t ( h t ; f t ) R_t(h_t;f_t) R t ( h t ; f t ) pointwise in h t h_t h t reduces to minimizing (84) pointwise in ( x , z t ) (x,z_t) ( x , z t ) , that is,
f t ∗ ( x , z t ) ∈ arg min y ∈ [ K ] 1 − p ( y ∣ x , z t ) + c t = arg max y ∈ [ K ] p ( y ∣ x , z t ) . (86) f_t^*(x,z_t)\in\underset{y\in[K]}{\arg\min}\;1-p(y\mid x,z_t)+c_t
=\underset{y\in[K]}{\arg\max}\;p(y\mid x,z_t). \tag*{(86)} f t ∗ ( x , z t ) ∈ y ∈ [ K ] arg min 1 − p ( y ∣ x , z t ) + c t = y ∈ [ K ] arg max p ( y ∣ x , z t ) . (86)
This proves that f 1 : T ∗ f^*_{1:T} f 1 : T ∗ satisfy (30). Substituting (85) into (84) yields
R t ( h t ; f t ∗ ) = 1 − max y ∈ [ K ] p ( y ∣ x , z t ) + c t , R_t(h_t;f_t^*)=1-\underset{y\in[K]}{\max}\;p(y\mid x,z_t)+c_t, R t ( h t ; f t ∗ ) = 1 − y ∈ [ K ] max p ( y ∣ x , z t ) + c t ,
as stated in (83).
Note the result in (85) holds uniformly for any routing policy r c a s c r_{\mathrm{casc}} r casc , or equivalently, any d 1 : T d_{1:T} d 1 : T . In other words, it characterizes the Bayes-optimal classifiers f 1 : T ∗ f^*_{1:T} f 1 : T ∗ independently of how d 1 : T d_{1:T} d 1 : T is chosen. Therefore, it remains only to characterize the Bayes-optimal deferral rules d 1 : T ∗ d^*_{1:T} d 1 : T ∗ with the classifiers fixed at f 1 : T = f 1 : T ∗ f_{1:T}=f^*_{1:T} f 1 : T = f 1 : T ∗ .
Part 2: Proof for the Bayes-optimal d 1 : T ∗ d^*_{1:T} d 1 : T ∗ . We now minimize R ( π ) R(\pi) R ( π ) with respect to d 1 : T d_{1:T} d 1 : T with f 1 : T = f 1 : T ∗ f_{1:T}=f^*_{1:T} f 1 : T = f 1 : T ∗ held fixed. Since π = ( f 1 : T , d 1 : T ) \pi=(f_{1:T},d_{1:T}) π = ( f 1 : T , d 1 : T ) , we write R ( f 1 : T , d 1 : T ) R(f_{1:T},d_{1:T}) R ( f 1 : T , d 1 : T ) to mean R ( π ) R(\pi) R ( π ) and V t f 1 : T , d 1 : T ( H t ) V_t^{f_{1:T},d_{1:T}}(H_t) V t f 1 : T , d 1 : T ( H t ) to mean V t π ( H t ) V_t^\pi(H_t) V t π ( H t ) . By (76) which links R R R to V 1 V_1 V 1 , we have
d 1 : T ∗ ∈ arg min d 1 : T R ( f 1 : T ∗ , d 1 : T ) = arg min d 1 : T E [ V 1 f 1 : T ∗ , d 1 : T ( H 1 ) ] . (87) d^*_{1:T}\in\underset{d_{1:T}}{\arg\min}\;R(f^*_{1:T},d_{1:T})
=\underset{d_{1:T}}{\arg\min}\;\mathbb{E}\left[V_1^{f^*_{1:T},d_{1:T}}(H_1)\right]. \tag*{(87)} d 1 : T ∗ ∈ d 1 : T arg min R ( f 1 : T ∗ , d 1 : T ) = d 1 : T arg min E [ V 1 f 1 : T ∗ , d 1 : T ( H 1 ) ] . (87)
where the minimum is over d t : H t → { 0 , 1 } d_t:\mathcal{H}_t\to\{0,1\} d t : H t → { 0 , 1 } for t ∈ [ T ] t\in[T] t ∈ [ T ] . By the recursive definition (74) of the value function, we can unroll (86) to optimize d t d_t d t one at a time, treating the earlier functions d 1 : t − 1 d_{1:t-1} d 1 : t − 1 as arbitrary. Consequently, the optimal d t ∗ d_t^* d t ∗ is pointwise optimal for every trajectory d 1 : t − 1 d_{1:t-1} d 1 : t − 1 , not just for the optimal trajectory d 1 : t − 1 ∗ d_{1:t-1}^* d 1 : t − 1 ∗ , and composing the pointwise minimizers d t ∗ d_t^* d t ∗ across t ∈ [ T ] t \in[T] t ∈ [ T ] yields d 1 : T ∗ d_{1:T}^* d 1 : T ∗ .
In the rest of the proof, we characterize d t ∗ d_t^* d t ∗ using backward induction starting with t = T t = T t = T .
Base case t = T t = T t = T : Since T T T is the terminal stage, d T ∗ ( H T ) d_T^*(H_T) d T ∗ ( H T ) is fixed to 0 0 0 (exit), as stated in (82). By definition (75) of the terminal-stage value function, V T f 1 : T ∗ , d 1 : T − 1 , d T ∗ ( H T ) = R T ( H T ; f T ∗ ) V_T^{f_{1:T}^*,d_{1:T-1},d_T^*}(H_T) = \mathcal{R}_T(H_T;f_T^*) V T f 1 : T ∗ , d 1 : T − 1 , d T ∗ ( H T ) = R T ( H T ; f T ∗ ) for any arbitrary d 1 : T − 1 d_{1:T-1} d 1 : T − 1 , so we can write
V T f 1 : T ∗ , d 1 : T − 1 , d T ∗ ( H T ) = V T π c a s c ∗ ( H T ) where π c a s c ∗ = ( f 1 : T ∗ , d 1 : T ∗ ) . (88) V_T^{f_{1:T}^*,d_{1:T-1},d_T^*}(H_T) = V_T^{\pi_{\mathrm{casc}}^*}(H_T) \quad\text{where} \quad\pi_{\mathrm{casc}}^* = (f_{1:T}^*,d_{1:T}^*). \tag*{(88)} V T f 1 : T ∗ , d 1 : T − 1 , d T ∗ ( H T ) = V T π casc ∗ ( H T ) where π casc ∗ = ( f 1 : T ∗ , d 1 : T ∗ ) . (88)
Backward induction: Using (87), we now prove the induction step. Suppose that for some t ∈ { 1 , … , T − 1 } t \in\{1,\ldots,T-1\} t ∈ { 1 , … , T − 1 } ,
V t + 1 f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t + 1 ) = V t + 1 π c a s c ∗ ( H t + 1 ) (89) V_{t+1}^{f_{1:T}^*,d_{1:t},d_{t+1:T}^*}(H_{t+1}) = V_{t+1}^{\pi_{\mathrm{casc}}^*}(H_{t+1}) \tag*{(89)} V t + 1 f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t + 1 ) = V t + 1 π casc ∗ ( H t + 1 ) (89)
for any arbitrary d 1 : t d_{1:t} d 1 : t , then
d t ∗ ( h t ) = 1 { R t ( h t ; f t ∗ ) > E [ V t + 1 π c a s c ∗ ( H t + 1 ) ∣ H t = h t ] } . (90) d_t^*(h_t) = \mathbb{1}\left\{\mathcal{R}_t(h_t;f_t^*) > \mathbb{E}\left[V_{t+1}^{\pi_{\mathrm{casc}}^*}(H_{t+1}) \mid H_t = h_t\right]\right\}. \tag*{(90)} d t ∗ ( h t ) = 1 { R t ( h t ; f t ∗ ) > E [ V t + 1 π casc ∗ ( H t + 1 ) ∣ H t = h t ] } . (90)
We fix d t + 1 : T = d t + 1 : T ∗ d_{t+1:T} = d_{t+1:T}^* d t + 1 : T = d t + 1 : T ∗ and unroll the value function V 1 V_1 V 1 in the objective (86) to optimize over d t d_t d t . This gives
E [ V 1 f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H 1 ) ] = E [ ( 1 − d 1 ( H 1 ) ) R 1 ( H 1 ; f 1 ∗ ) + d 1 ( H 1 ) E [ ⋯ + E [ ⋯ + d t − 1 ( H t − 1 ) \mathbb{E}\left[V_1^{f_{1:T}^*,d_{1:t},d_{t+1:T}^*}(H_1)\right] = \mathbb{E}\left[(1-d_1(H_1))\mathcal{R}_1(H_1;f_1^*) + d_1(H_1)\mathbb{E}\left[\cdots+\mathbb{E}\left[\cdots+d_{t-1}(H_{t-1})
\right.\right.\right. E [ V 1 f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H 1 ) ] = E [ ( 1 − d 1 ( H 1 )) R 1 ( H 1 ; f 1 ∗ ) + d 1 ( H 1 ) E [ ⋯ + E [ ⋯ + d t − 1 ( H t − 1 )
E [ V t f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t ) ∣ H t − 1 ] ∣ H t − 2 ] ∣ ⋯ ∣ H 1 ] ] . (91) \left.\left.\left.
\mathbb{E}\left[V_t^{f_{1:T}^*,d_{1:t},d_{t+1:T}^*}(H_t)\mid H_{t-1}\right]\mid H_{t-2}\right]\mid\cdots\mid H_1\right]\right]. \tag*{(91)} E [ V t f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t ) ∣ H t − 1 ] ∣ H t − 2 ] ∣ ⋯ ∣ H 1 ] ] . (91)
To clarify, Term I collects the nested expectations taken successively over H t − 1 ∣ H t − 2 H_{t-1}\mid H_{t-2} H t − 1 ∣ H t − 2 , H t − 2 ∣ H t − 3 H_{t-2}\mid H_{t-3} H t − 2 ∣ H t − 3 , … \ldots … down to H 2 ∣ H 1 H_2\mid H_1 H 2 ∣ H 1 , and finally over H 1 H_1 H 1 through the outermost expectation, along with terms depending on d 1 , … , d t − 1 d_1,\ldots,d_{t-1} d 1 , … , d t − 1 . Term II collects the corresponding chain of conditionings from H t − 1 H_{t-1} H t − 1 down to H 1 H_1 H 1 . Hence, the part of (90) that depends on d t d_t d t is
E [ d 1 ( H 1 ) E [ d 2 ( H 2 ) ⋯ E [ d t − 1 ( H t − 1 ) E [ V t f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t ) ∣ H t − 1 ] ∣ H t − 2 ] ⋯ ∣ H 1 ] ] = E [ ∏ τ = 1 t − 1 d τ ( H τ ) V t f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t ) ] , (92) \begin{aligned}
\mathbb{E}\left[d_1(H_1)\mathbb{E}\left[d_2(H_2)\cdots\mathbb{E}\left[d_{t-1}(H_{t-1})\mathbb{E}\left[V_t^{f_{1:T}^*,d_{1:t},d_{t+1:T}^*}(H_t)\mid H_{t-1}\right]\mid H_{t-2}\right]\cdots\mid H_1\right]\right] \\
&= \mathbb{E}\left[\prod_{\tau=1}^{t-1}d_\tau(H_\tau)V_t^{f_{1:T}^*,d_{1:t},d_{t+1:T}^*}(H_t)\right], \tag*{(92)}
\end{aligned} E [ d 1 ( H 1 ) E [ d 2 ( H 2 ) ⋯ E [ d t − 1 ( H t − 1 ) E [ V t f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t ) ∣ H t − 1 ] ∣ H t − 2 ] ⋯ ∣ H 1 ] ] = E [ τ = 1 ∏ t − 1 d τ ( H τ ) V t f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t ) ] , (92)
where the equality uses the tower rule of conditional expectation, and V t f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t ) V_t^{f_{1:T}^*,d_{1:t},d_{t+1:T}^*}(H_t) V t f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t ) is the innermost term of (90), which by definition, is
V t f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t ) = ( 1 − d t ( H t ) ) R t ( H t ; f t ∗ ) + d t ( H t ) E [ V t + 1 f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t + 1 ) ∣ H t ] V_t^{f_{1:T}^*,d_{1:t},d_{t+1:T}^*}(H_t) = (1-d_t(H_t))\mathcal{R}_t(H_t;f_t^*) + d_t(H_t)\mathbb{E}\left[V_{t+1}^{f_{1:T}^*,d_{1:t},d_{t+1:T}^*}(H_{t+1})\mid H_t\right] V t f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t ) = ( 1 − d t ( H t )) R t ( H t ; f t ∗ ) + d t ( H t ) E [ V t + 1 f 1 : T ∗ , d 1 : t , d t + 1 : T ∗ ( H t + 1 ) ∣ H t ]
= ( i ) ( 1 − d t ( H t ) ) R t ( H t ; f t ∗ ) + d t ( H t ) E [ V t + 1 π c a s c ∗ ( H t + 1 ) ∣ H t ] , (93) \overset{(\mathrm{i})}{=} (1-d_t(H_t))\mathcal{R}_t(H_t;f_t^*) + d_t(H_t)\mathbb{E}\left[V_{t+1}^{\pi_{\mathrm{casc}}^*}(H_{t+1})\mid H_t\right], \tag*{(93)} = ( i ) ( 1 − d t ( H t )) R t ( H t ; f t ∗ ) + d t ( H t ) E [ V t + 1 π casc ∗ ( H t + 1 ) ∣ H t ] , (93)
where (i) holds by the induction hypothesis (88). Hence, (90)–(91) imply that the Bayes-optimal deferral function d t ∗ d_t^* d t ∗ should satisfy
d t ∗ ∈ arg min d t E [ V 1 : T f ∗ ; d 1 : t ; d t + 1 : T ∗ ( H 1 ) ] = arg min d t E [ ∏ τ = 1 t − 1 d τ ( H τ ) V t f ∗ ; d 1 : t ; d t + 1 : T ∗ ( H t ) ] . (94) d_t^* \in\arg\min_{d_t} \mathbb{E}\left[V_{1:T}^{f^*;d_{1:t};d_{t+1:T}^*}(H_1)\right] = \arg\min_{d_t} \mathbb{E}\left[\prod_{\tau=1}^{t-1}d_\tau(H_\tau)V_t^{f^*;d_{1:t};d_{t+1:T}^*}(H_t)\right]. \tag*{(94)} d t ∗ ∈ arg d t min E [ V 1 : T f ∗ ; d 1 : t ; d t + 1 : T ∗ ( H 1 ) ] = arg d t min E [ τ = 1 ∏ t − 1 d τ ( H τ ) V t f ∗ ; d 1 : t ; d t + 1 : T ∗ ( H t ) ] . (94)
Since the ∏ τ = 1 t − 1 d τ ( H τ ) \prod_{\tau=1}^{t-1}d_\tau(H_\tau) ∏ τ = 1 t − 1 d τ ( H τ ) term is independent of d t d_t d t and always non-negative, (93) reduces pointwise in h t ∈ H t h_t \in\mathcal{H}_t h t ∈ H t to
d t ∗ ( h t ) ∈ arg min d ∈ { 0 , 1 } V t f 1 : T ∗ ; d 1 : t ; d t + 1 : T ∗ ( h t ) = arg min d ∈ { 0 , 1 } ( 1 − d ) R t ( h t ; f t ∗ ) + d E [ V t + 1 π c a s c ∗ ( H t + 1 ) ∣ H t = h t ] = 1 { R t ( h t ; f t ∗ ) > E [ V t + 1 π c a s c ∗ ( H t + 1 ) ∣ H t = h t ] } . (95) \begin{aligned}
d_t^*(h_t) \in\arg\min_{d\in\{0,1\}} V_t^{f^*_{1:T};d_{1:t};d_{t+1:T}^*}(h_t) \\
&= \arg\min_{d\in\{0,1\}} (1-d)\mathcal{R}_t(h_t;f_t^*) + d\mathbb{E}\left[V_{t+1}^{\pi_{\mathrm{casc}}^*}(H_{t+1}) \mid H_t=h_t\right] \\
&= \mathbb{1}\left\{\mathcal{R}_t(h_t;f_t^*) > \mathbb{E}\left[V_{t+1}^{\pi_{\mathrm{casc}}^*}(H_{t+1}) \mid H_t=h_t\right]\right\}. \tag*{(95)}
\end{aligned} d t ∗ ( h t ) ∈ arg d ∈ { 0 , 1 } min V t f 1 : T ∗ ; d 1 : t ; d t + 1 : T ∗ ( h t ) = arg d ∈ { 0 , 1 } min ( 1 − d ) R t ( h t ; f t ∗ ) + d E [ V t + 1 π casc ∗ ( H t + 1 ) ∣ H t = h t ] = 1 { R t ( h t ; f t ∗ ) > E [ V t + 1 π casc ∗ ( H t + 1 ) ∣ H t = h t ] } . (95)
This proves (89), that is, (82) for stage t t t . Substituting this Bayes-optimal d t ∗ d_t^* d t ∗ back into the recursion (92) yields:
V t f 1 : T ∗ ; d 1 : t − 1 ; d t : T ∗ ( H t ) = ( 1 − d t ∗ ( H t ) ) R t ( H t ; f t ∗ ) + d t ∗ ( H t ) E [ V t + 1 π c a s c ∗ ( H t + 1 ) ∣ H t ] = (i) V t π c a s c ∗ ( H t ) , (96) V_t^{f^*_{1:T};d_{1:t-1};d_{t:T}^*}(H_t) = (1-d_t^*(H_t))\mathcal{R}_t(H_t;f_t^*) + d_t^*(H_t)\mathbb{E}\left[V_{t+1}^{\pi_{\mathrm{casc}}^*}(H_{t+1})\mid H_t\right] \overset{\text{(i)}}{=} V_t^{\pi_{\mathrm{casc}}^*}(H_t), \tag*{(96)} V t f 1 : T ∗ ; d 1 : t − 1 ; d t : T ∗ ( H t ) = ( 1 − d t ∗ ( H t )) R t ( H t ; f t ∗ ) + d t ∗ ( H t ) E [ V t + 1 π casc ∗ ( H t + 1 ) ∣ H t ] = (i) V t π casc ∗ ( H t ) , (96)
for any arbitrary d 1 : t − 1 d_{1:t-1} d 1 : t − 1 , where (i) holds by definition of V t π c a s c ∗ V_t^{\pi_{\mathrm{casc}}^*} V t π casc ∗ . Thus, this establishes (88) for stage t t t , completing the induction step.
Conclusion: By backward induction, we obtain (82) for all t ∈ [ T ] t \in[T] t ∈ [ T ] .
Connection of d 1 : T ∗ d_{1:T}^* d 1 : T ∗ to optimal stopping: Identify the filtration σ ( H t ) \sigma(H_t) σ ( H t ) , the adapted reward process { G t } t = 1 T \{G_t\}_{t=1}^T { G t } t = 1 T with G t = − R t ( H t ; f t ∗ ) G_t=-\mathcal{R}_t(H_t;f_t^*) G t = − R t ( H t ; f t ∗ ) , and let T = { τ : τ ∈ [ T ] , { τ = t } ∈ σ ( H t ) } \mathcal{T}=\{\tau:\tau\in[T],\{\tau=t\}\in\sigma(H_t)\} T = { τ : τ ∈ [ T ] , { τ = t } ∈ σ ( H t )} denote the set of stopping times. Then the deferral rule d 1 : T ∗ d_{1:T}^* d 1 : T ∗ in (82), which minimizes the risk R ( f 1 : T ∗ ; d 1 : T ) \mathcal{R}(f_{1:T}^*;d_{1:T}) R ( f 1 : T ∗ ; d 1 : T ) , coincides with the optimal stopping rule for the finite-horizon problem sup τ ∈ T E [ G τ ] \sup_{\tau\in\mathcal{T}}\mathbb{E}[G_\tau] sup τ ∈ T E [ G τ ] [49 ] , which maximizes the reward, and our value function is its dynamic programming recursion.
While Lemma C.4 characterizes the Bayes-optimal solution under the target 0-1-c loss, the next lemma is in parallel and characterizes the Bayes-optimal solution under the cascade hinge loss.
Lemma C.5 (Bayes-optimal solution in Π ~ c a s c \widetilde{\Pi}_{\mathrm{casc}} Π casc for hinge loss). Assume Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) Y \perp Z_{1:t-1} \mid(X,Z_t) Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) as in (5). There exists a Bayes-optimal cascade classifier
π ~ c a s c ∗ = ( f ~ 1 : T ∗ , d ~ 1 : T ∗ ) ∈ arg min π ~ ∈ Π ~ c a s c R c a s c h i n g e ( π ~ ) , \widetilde{\pi}_{\mathrm{casc}}^*=(\widetilde{f}_{1:T}^*,\widetilde{d}_{1:T}^*)\in\arg\min_{\widetilde{\pi}\in\widetilde{\Pi}_{\mathrm{casc}}}\mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}}(\widetilde{\pi}), π casc ∗ = ( f 1 : T ∗ , d 1 : T ∗ ) ∈ arg π ∈ Π casc min R casc hinge ( π ) , where, with f t ∗ f_t^* f t ∗ defined as in (30),
[ f ~ t ∗ ( x , z t ) ] k = { 1 for k = f t ∗ ( x , z t ) , − 1 K − 1 otherwise ∀ t ∈ [ T ] , (97) \left[\widetilde{f}_t^*(x,z_t)\right]_k=
\begin{cases}
1 & \text{for } k=f_t^*(x,z_t),\\
-\dfrac{1}{K-1} & \text{otherwise}
\end{cases}
\qquad\forall t\in[T], \tag*{(97)} [ f t ∗ ( x , z t ) ] k = ⎩ ⎨ ⎧ 1 − K − 1 1 for k = f t ∗ ( x , z t ) , otherwise ∀ t ∈ [ T ] , (97) and, with R t h i n g e ( h t ; f ~ t ) \mathcal{R}_t^{\mathrm{hinge}}(h_t;\widetilde{f}_t) R t hinge ( h t ; f t ) defined as in (73),
d ~ t ∗ ( h t ) = 1 { R t h i n g e ( h t ; f ~ t ∗ ) > E [ V ~ t + 1 π ~ c a s c ∗ ( H t + 1 ) ∣ H t = h t ] } ∀ t < T and d ~ T ∗ ≡ 0. (98) \widetilde{d}_t^*(h_t)=\mathbb{1}\left\{\mathcal{R}_t^{\mathrm{hinge}}(h_t;\widetilde{f}_t^*)>\mathbb{E}\left[\widetilde{V}_{t+1}^{\widetilde{\pi}_{\mathrm{casc}}^*}(H_{t+1})\mid H_t=h_t\right]\right\}\quad\forall t<T\quad\text{and}\quad\widetilde{d}_T^*\equiv0. \tag*{(98)} d t ∗ ( h t ) = 1 { R t hinge ( h t ; f t ∗ ) > E [ V t + 1 π casc ∗ ( H t + 1 ) ∣ H t = h t ] } ∀ t < T and d T ∗ ≡ 0. (98)
Proof. We minimize R c a s c h i n g e ( π ~ ) \mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}}(\widetilde{\pi}) R casc hinge ( π ) in two stages: first over f ~ 1 : T \widetilde{f}_{1:T} f 1 : T with d ~ 1 : T \widetilde{d}_{1:T} d 1 : T held fixed, then over d ~ 1 : T \widetilde{d}_{1:T} d 1 : T with f ~ 1 : T = f ~ 1 : T ∗ \widetilde{f}_{1:T}=\widetilde{f}_{1:T}^* f 1 : T = f 1 : T ∗ fixed.
Part 1: Proof for the Bayes-optimal f ~ 1 : T ∗ \tilde{f}^{*}_{1:T} f ~ 1 : T ∗ . Since p ( y ∣ x , z 1 : t ) = p ( y ∣ x , z t ) p(y \mid x, z_{1:t}) = p(y \mid x, z_t) p ( y ∣ x , z 1 : t ) = p ( y ∣ x , z t ) , the result in (97) follows from [53 Lemma 1] .
Part 2: Proof for the Bayes-optimal d ~ 1 : T ∗ \tilde{d}^{*}_{1:T} d ~ 1 : T ∗ . We now prove (98) . Since π ~ = ( f ~ 1 : T , d ~ 1 : T ) \tilde{\pi} = (\tilde{f}_{1:T}, \tilde{d}_{1:T}) π ~ = ( f ~ 1 : T , d ~ 1 : T ) , we write R c a s c h i n g e ( f ~ 1 : T , d ~ 1 : T ) \mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{f}_{1:T}, \tilde{d}_{1:T}) R casc hinge ( f ~ 1 : T , d ~ 1 : T ) to mean R c a s c h i n g e ( π ~ ) \mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi}) R casc hinge ( π ~ ) , and V ~ t f ~ 1 : T , d ~ 1 : T ( H t ) \tilde{V}^{\tilde{f}_{1:T},\tilde{d}_{1:T}}_t(H_t) V ~ t f ~ 1 : T , d ~ 1 : T ( H t ) to mean V ~ t π ~ ( H t ) \tilde{V}^{\tilde{\pi}}_t(H_t) V ~ t π ~ ( H t ) . By definition of π ~ c a s c ∗ \tilde{\pi}^{*}_{\mathrm{casc}} π ~ casc ∗ and using (77) which links R c a s c h i n g e \mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}} R casc hinge and V ~ 1 π ~ \tilde{V}^{\tilde{\pi}}_1 V ~ 1 π ~ , we can write
d ~ 1 : T ∗ ∈ arg min d ~ 1 : T R c a s c h i n g e ( f ~ 1 : T ∗ , d ~ 1 : T ) = arg min d 1 : T E [ V ~ 1 f ~ 1 : T ∗ , d ~ 1 : T ∗ ( H 1 ) ] , (99) \tilde{d}^{*}_{1:T} \in\underset{\tilde{d}_{1:T}}{\arg\min}\ \mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{f}^{*}_{1:T},\tilde{d}_{1:T}) = \underset{d_{1:T}}{\arg\min}\ \mathbb{E}\left[\tilde{V}^{\tilde{f}^{*}_{1:T},\tilde{d}^{*}_{1:T}}_1(H_1)\right], \tag*{(99)} d ~ 1 : T ∗ ∈ d ~ 1 : T arg min R casc hinge ( f ~ 1 : T ∗ , d ~ 1 : T ) = d 1 : T arg min E [ V ~ 1 f ~ 1 : T ∗ , d ~ 1 : T ∗ ( H 1 ) ] , (99)
where each minimum is taken over d ~ t : H t → [ 0 , 1 ] \tilde{d}_t:\mathcal{H}_t \to[0,1] d ~ t : H t → [ 0 , 1 ] for t ∈ [ T ] t \in[T] t ∈ [ T ] . The proof proceeds by backward induction, following the same steps as those in Part 2 of the proof of Lemma C.4 , see (86)–(95). For all t < T t < T t < T , this yields
d ~ t ∗ ( h t ) ∈ arg min d ~ ∈ [ 0 , 1 ] ( 1 − d ~ ) R t h i n g e ( h t ; f ~ t ∗ ) + d ~ E [ V ~ t + 1 f ~ 1 : T ∗ , d ~ t + 1 : T ∗ ( H t + 1 ) ∣ H t = h t ] = arg min d ~ ∈ [ 0 , 1 ] ( 1 − d ~ ) R t h i n g e ( h t ; f ~ t ∗ ) + d ~ E [ V ~ t + 1 π ~ c a s c ∗ ( H t + 1 ) ∣ H t = h t ] . \begin{aligned}
\tilde{d}^{*}_t(h_t) \in\underset{\tilde{d}\in[0,1]}{\arg\min}\ (1-\tilde{d})\mathcal{R}^{\mathrm{hinge}}_t(h_t;\tilde{f}^{*}_t)+\tilde{d}\mathbb{E}\left[\tilde{V}^{\tilde{f}^{*}_{1:T},\tilde{d}^{*}_{t+1:T}}_{t+1}(H_{t+1})\mid H_t=h_t\right] \\
&= \underset{\tilde{d}\in[0,1]}{\arg\min}\ (1-\tilde{d})\mathcal{R}^{\mathrm{hinge}}_t(h_t;\tilde{f}^{*}_t)+\tilde{d}\mathbb{E}\left[\tilde{V}^{\tilde{\pi}^{*}_{\mathrm{casc}}}_{t+1}(H_{t+1})\mid H_t=h_t\right].
\end{aligned} d ~ t ∗ ( h t ) ∈ d ~ ∈ [ 0 , 1 ] arg min ( 1 − d ~ ) R t hinge ( h t ; f ~ t ∗ ) + d ~ E [ V ~ t + 1 f ~ 1 : T ∗ , d ~ t + 1 : T ∗ ( H t + 1 ) ∣ H t = h t ] = d ~ ∈ [ 0 , 1 ] arg min ( 1 − d ~ ) R t hinge ( h t ; f ~ t ∗ ) + d ~ E [ V ~ t + 1 π ~ casc ∗ ( H t + 1 ) ∣ H t = h t ] .
Since this objective is linear in d ~ ∈ [ 0 , 1 ] \tilde{d}\in[0,1] d ~ ∈ [ 0 , 1 ] , the minimum is attained at an endpoint, that is,
d ~ t ∗ ( h t ) = 1 { R t h i n g e ( h t ; f ~ t ∗ ) > E [ V ~ t + 1 π ~ c a s c ∗ ( H t + 1 ) ∣ H t = h t ] } , \tilde{d}^{*}_t(h_t)=\mathbb{1}\left\{\mathcal{R}^{\mathrm{hinge}}_t(h_t;\tilde{f}^{*}_t)>\mathbb{E}\left[\tilde{V}^{\tilde{\pi}^{*}_{\mathrm{casc}}}_{t+1}(H_{t+1})\mid H_t=h_t\right]\right\}, d ~ t ∗ ( h t ) = 1 { R t hinge ( h t ; f ~ t ∗ ) > E [ V ~ t + 1 π ~ casc ∗ ( H t + 1 ) ∣ H t = h t ] } ,
yielding (98) .
Connections Between 0-1-c and Hinge Quantities
In this section we establish two useful results linking hinge and 0-1-c quantities, which will help prove calibration in the next section.
Lemma C.6 (Hinge and 0-1-c Bayes quantities differ by a factor of K K − 1 \frac{K}{K-1} K − 1 K ). Assume Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) Y \perp Z_{1:t-1}\mid(X,Z_t) Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) as in (5). Let π c a s c ∗ = ( f 1 : T ∗ ; d 1 : T ∗ ) \pi^{*}_{\mathrm{casc}}=(f^{*}_{1:T};d^{*}_{1:T}) π casc ∗ = ( f 1 : T ∗ ; d 1 : T ∗ ) and π ~ c a s c ∗ = ( f ~ 1 : T ∗ ; d ~ 1 : T ∗ ) \tilde{\pi}^{*}_{\mathrm{casc}}=(\tilde{f}^{*}_{1:T};\tilde{d}^{*}_{1:T}) π ~ casc ∗ = ( f ~ 1 : T ∗ ; d ~ 1 : T ∗ ) denote the Bayes-optimal cascade classifiers from Lemmas C.4 –C.5 under the 0-1-c loss and the hinge loss, respectively. Define the conditional Bayes risks
R t ( h t ; f t ∗ ) : = E [ 1 { f t ∗ ( X , Z t ) ≠ Y } + c t ∣ H t = h t ] , (100) \mathcal{R}_t(h_t;f^{*}_t):=\mathbb{E}\left[\mathbb{1}\{f^{*}_t(X,Z_t)\ne Y\}+c_t\mid H_t=h_t\right], \tag*{(100)} R t ( h t ; f t ∗ ) := E [ 1 { f t ∗ ( X , Z t ) = Y } + c t ∣ H t = h t ] , (100) R t h i n g e ( h t ; f ~ t ∗ ) : = E [ ℓ h i n g e ( f ~ t ∗ ( X , Z t ) , Y ; c t ) ∣ H t = h t ] , (101) \mathcal{R}^{\mathrm{hinge}}_t(h_t;\tilde{f}^{*}_t):=\mathbb{E}\left[\ell^{\mathrm{hinge}}(\tilde{f}^{*}_t(X,Z_t),Y;c_t)\mid H_t=h_t\right], \tag*{(101)} R t hinge ( h t ; f ~ t ∗ ) := E [ ℓ hinge ( f ~ t ∗ ( X , Z t ) , Y ; c t ) ∣ H t = h t ] , (101) then
R t h i n g e ( h t ; f ~ t ∗ ) = K K − 1 R t ( h t ; f t ∗ ) ∀ t ∈ [ T ] . (102) \mathcal{R}^{\mathrm{hinge}}_t(h_t;\tilde{f}^{*}_t)=\frac{K}{K-1}\mathcal{R}_t(h_t;f^{*}_t)\qquad\forall t\in[T]. \tag*{(102)} R t hinge ( h t ; f ~ t ∗ ) = K − 1 K R t ( h t ; f t ∗ ) ∀ t ∈ [ T ] . (102) Moreover, the stage-t t t value functions of π ~ c a s c ∗ \tilde{\pi}^{*}_{\mathrm{casc}} π ~ casc ∗ and π c a s c ∗ \pi^{*}_{\mathrm{casc}} π casc ∗ satisfy
V ~ t π ~ c a s c ∗ ( h t ) = K K − 1 V t π c a s c ∗ ( h t ) ∀ t ∈ [ T ] . (103) \tilde{V}^{\tilde{\pi}^{*}_{\mathrm{casc}}}_t(h_t)=\frac{K}{K-1}V^{\pi^{*}_{\mathrm{casc}}}_t(h_t)\qquad\forall t\in[T]. \tag*{(103)} V ~ t π ~ casc ∗ ( h t ) = K − 1 K V t π casc ∗ ( h t ) ∀ t ∈ [ T ] . (103)
Proof. By Lemmas C.4 –C.5 together with Lemma B.3 , we have
R t ( h t ; f t ∗ ) = 1 − max y ∈ [ K ] p ( y ∣ x , z t ) + c t , \mathcal{R}_t(h_t; f_t^*) = 1 - \max_{y \in[K]} p(y \mid x,z_t) + c_t, R t ( h t ; f t ∗ ) = 1 − y ∈ [ K ] max p ( y ∣ x , z t ) + c t ,
R t h i n g e ( h t ; f ~ t ∗ ) = K K − 1 ( 1 − max y ∈ [ K ] p ( y ∣ x , z t ) + c t ) . \mathcal{R}_t^{\mathrm{hinge}}(h_t; \tilde{f}_t^*) = \frac{K}{K-1}\left(1-\max_{y \in[K]}p(y \mid x,z_t)+c_t\right). R t hinge ( h t ; f ~ t ∗ ) = K − 1 K ( 1 − y ∈ [ K ] max p ( y ∣ x , z t ) + c t ) .
which proves (102) . To prove (103) , we use backward induction.
Base case t = T t=T t = T : we have
V ~ T π ∗ c a s c ( h T ) = R T h i n g e ( h T ; f ~ T ∗ ) and V T π ∗ c a s c ( h T ) = R T ( h T ; f T ∗ ) , \widetilde{V}_T^{\pi^*\mathrm{casc}}(h_T)=\mathcal{R}_T^{\mathrm{hinge}}(h_T;\tilde{f}_T^*) \qquad\text{and}\qquad V_T^{\pi^*\mathrm{casc}}(h_T)=\mathcal{R}_T(h_T;f_T^*), V T π ∗ casc ( h T ) = R T hinge ( h T ; f ~ T ∗ ) and V T π ∗ casc ( h T ) = R T ( h T ; f T ∗ ) ,
so applying (102) gives (103) for t = T t=T t = T .
Backward induction: Assume (103) holds for t + 1 t+1 t + 1 , we prove it for t t t . First, by Lemmas C.4 –C.5 , since both Bayes-optimal deferral rules d 1 : T ∗ d^*_{1:T} d 1 : T ∗ and d ~ 1 : T ∗ \tilde{d}^*_{1:T} d ~ 1 : T ∗ are { 0 , 1 } \{0,1\} { 0 , 1 } -valued (see (83) and (98) ), we have
V t π ∗ c a s c ( h t ) = min { R t ( h t ; f t ∗ ) , E [ V t + 1 π ∗ c a s c ( H t + 1 ) ∣ H t = h t ] } . (104) V_t^{\pi^*\mathrm{casc}}(h_t)=\min\left\{\mathcal{R}_t(h_t;f_t^*),\mathbb{E}\left[V_{t+1}^{\pi^*\mathrm{casc}}(H_{t+1})\mid H_t=h_t\right]\right\}. \tag*{(104)} V t π ∗ casc ( h t ) = min { R t ( h t ; f t ∗ ) , E [ V t + 1 π ∗ casc ( H t + 1 ) ∣ H t = h t ] } . (104)
V ~ t π ~ ∗ c a s c ( h t ) = min { R t h i n g e ( h t ; f ~ t ∗ ) , E [ V ~ t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] } . (105) \widetilde{V}_t^{\tilde{\pi}^*\mathrm{casc}}(h_t)=\min\left\{\mathcal{R}_t^{\mathrm{hinge}}(h_t;\tilde{f}_t^*),\mathbb{E}\left[\widetilde{V}_{t+1}^{\tilde{\pi}^*\mathrm{casc}}(H_{t+1})\mid H_t=h_t\right]\right\}. \tag*{(105)} V t π ~ ∗ casc ( h t ) = min { R t hinge ( h t ; f ~ t ∗ ) , E [ V t + 1 π ~ ∗ casc ( H t + 1 ) ∣ H t = h t ] } . (105)
Note by (105) , (102) and (104) , we can write
V ~ t π ~ ∗ c a s c ( h t ) = ( i ) min { R t h i n g e ( h t ; f ~ t ∗ ) , E [ V ~ t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] } \widetilde{V}_t^{\tilde{\pi}^*\mathrm{casc}}(h_t)
\mathrel{\overset{\mathrm{(i)}}{=}}
\min\left\{\mathcal{R}_t^{\mathrm{hinge}}(h_t;\tilde{f}_t^*),\mathbb{E}\left[\widetilde{V}_{t+1}^{\tilde{\pi}^*\mathrm{casc}}(H_{t+1})\mid H_t=h_t\right]\right\} V t π ~ ∗ casc ( h t ) = ( i ) min { R t hinge ( h t ; f ~ t ∗ ) , E [ V t + 1 π ~ ∗ casc ( H t + 1 ) ∣ H t = h t ] }
= ( i i ) min { K K − 1 R t ( h t ; f t ∗ ) , E [ K K − 1 V ~ t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] } = K K − 1 min { R t ( h t ; f t ∗ ) , E [ V ~ t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] } \begin{aligned}
\mathrel{\overset{\mathrm{(ii)}}{=}}
\min\left\{\frac{K}{K-1}\mathcal{R}_t(h_t;f_t^*),\mathbb{E}\left[\frac{K}{K-1}\widetilde{V}_{t+1}^{\tilde{\pi}^*\mathrm{casc}}(H_{t+1})\mid H_t=h_t\right]\right\} \\
&=\frac{K}{K-1}\min\left\{\mathcal{R}_t(h_t;f_t^*),\mathbb{E}\left[\widetilde{V}_{t+1}^{\tilde{\pi}^*\mathrm{casc}}(H_{t+1})\mid H_t=h_t\right]\right\}
\end{aligned} = ( ii ) min { K − 1 K R t ( h t ; f t ∗ ) , E [ K − 1 K V t + 1 π ~ ∗ casc ( H t + 1 ) ∣ H t = h t ] } = K − 1 K min { R t ( h t ; f t ∗ ) , E [ V t + 1 π ~ ∗ casc ( H t + 1 ) ∣ H t = h t ] }
= ( i i i ) K K − 1 V ~ t π ~ ∗ c a s c ( h t ) , \mathrel{\overset{\mathrm{(iii)}}{=}}\frac{K}{K-1}\widetilde{V}_t^{\tilde{\pi}^*\mathrm{casc}}(h_t), = ( iii ) K − 1 K V t π ~ ∗ casc ( h t ) ,
where step (i) follows from (105) , step (ii) follows from (102) and the induction hypothesis, and step (iii) applies (104) . This completes the induction.
Conclusion: By backward induction, we establish (103) for all t ∈ [ T ] t \in[T] t ∈ [ T ] . □ \square □
The next lemma is an algebraic inequality linking the hard and soft deferral decisions, which will be useful in the proof of the main result (Theorem 3.2) in Section C.4 to show that the suboptimality gap under a hard deferral rule is at most twice that under its soft counterpart.
Lemma C.7 (Inequality linking soft and hard deferral decisions). For d ~ ∈ [ 0 , 1 ] \tilde{d}\in[0,1] d ~ ∈ [ 0 , 1 ] , let d : = 1 { d ~ ≥ 0.5 } d:=\mathbb{1}\{\tilde{d}\ge0.5\} d := 1 { d ~ ≥ 0.5 } . For all R , V ‾ ∈ R \mathcal{R},\overline{V}\in\mathbb{R} R , V ∈ R , it holds that
( 1 − d ) R + d V ‾ − min ( R , V ‾ ) ≤ 2 ( ( 1 − d ~ ) R + d ~ V ‾ − min ( R , V ‾ ) ) , (106) (1-d)\mathcal{R}+d\overline{V}-\min(\mathcal{R},\overline{V})\le2\left((1-\tilde{d})\mathcal{R}+\tilde{d}\overline{V}-\min(\mathcal{R},\overline{V})\right), \tag*{(106)} ( 1 − d ) R + d V − min ( R , V ) ≤ 2 ( ( 1 − d ~ ) R + d ~ V − min ( R , V ) ) , (106) where the 2 on the RHS is the smallest valid constant.
Proof. We first establish two inequalities linking d d d and d ~ \tilde{d} d ~ . When d ~ ≥ 0.5 \tilde{d}\ge0.5 d ~ ≥ 0.5 , we have d = 1 d=1 d = 1 and 2 d ~ ≥ 1 2\tilde{d}\ge1 2 d ~ ≥ 1 , so d ≤ 2 d ~ d\le2\tilde{d} d ≤ 2 d ~ ; when d ~ < 0.5 \tilde{d}<0.5 d ~ < 0.5 , d = 0 d=0 d = 0 so d ≤ 2 d ~ d\le2\tilde{d} d ≤ 2 d ~ is immediate. The same argument applied to 1 − d ~ 1-\tilde{d} 1 − d ~ , using 1 − d = 1 { 1 − d ~ > 0.5 } 1-d=\mathbb{1}\{1-\tilde{d}>0.5\} 1 − d = 1 { 1 − d ~ > 0.5 } , gives 1 − d ≤ 2 ( 1 − d ~ ) 1-d\le2(1-\tilde{d}) 1 − d ≤ 2 ( 1 − d ~ ) . Now we prove (106) by considering two cases.
Case 1. When R ≤ V ‾ \mathcal{R}\le\overline{V} R ≤ V : we have
( 1 − d ) R + d V ‾ − min ( R , V ‾ ) = ( 1 − d ) R + d V ‾ − R = d ( V ‾ − R ) (1-d)\mathcal{R}+d\overline{V}-\min(\mathcal{R},\overline{V})=(1-d)\mathcal{R}+d\overline{V}-\mathcal{R}=d(\overline{V}-\mathcal{R}) ( 1 − d ) R + d V − min ( R , V ) = ( 1 − d ) R + d V − R = d ( V − R ) ≤ ( i ) 2 d ~ ( V ‾ − R ) = 2 ( d ~ V ‾ + ( 1 − d ~ ) R − R ) \mathrel{\overset{\mathrm{(i)}}{\le}}2\tilde{d}(\overline{V}-\mathcal{R})=2\left(\tilde{d}\overline{V}+(1-\tilde{d})\mathcal{R}-\mathcal{R}\right) ≤ ( i ) 2 d ~ ( V − R ) = 2 ( d ~ V + ( 1 − d ~ ) R − R ) = 2 ( ( 1 − d ~ ) R + d ~ V ‾ − min ( R , V ‾ ) ) . =2\left((1-\tilde{d})\mathcal{R}+\tilde{d}\overline{V}-\min(\mathcal{R},\overline{V})\right). = 2 ( ( 1 − d ~ ) R + d ~ V − min ( R , V ) ) .
where step (i) applies d ≤ 2 d ~ d \le2\tilde{d} d ≤ 2 d ~ .
Case 2. When R > V ˉ \mathcal{R} > \bar{V} R > V ˉ : we have
( 1 − d ) R + d V ˉ − min ( R , V ˉ ) = ( 1 − d ) R + d V ˉ − V ˉ = ( 1 − d ) ( R − V ˉ ) (1-d)\mathcal{R}+d\bar{V}-\min(\mathcal{R},\bar{V})=(1-d)\mathcal{R}+d\bar{V}-\bar{V}=(1-d)(\mathcal{R}-\bar{V}) ( 1 − d ) R + d V ˉ − min ( R , V ˉ ) = ( 1 − d ) R + d V ˉ − V ˉ = ( 1 − d ) ( R − V ˉ )
≤ 2 ( 1 − d ~ ) ( R − V ˉ ) = 2 ( ( 1 − d ~ ) R − V ˉ + d ~ V ˉ ) = 2 ( ( 1 − d ~ ) R + d ~ V ˉ − min ( R , V ˉ ) ) \begin{aligned}
\le2(1-\tilde{d})(\mathcal{R}-\bar{V})=2\left((1-\tilde{d})\mathcal{R}-\bar{V}+\tilde{d}\bar{V}\right) \\
&=2\left((1-\tilde{d})\mathcal{R}+\tilde{d}\bar{V}-\min(\mathcal{R},\bar{V})\right)
\end{aligned} ≤ 2 ( 1 − d ~ ) ( R − V ˉ ) = 2 ( ( 1 − d ~ ) R − V ˉ + d ~ V ˉ ) = 2 ( ( 1 − d ~ ) R + d ~ V ˉ − min ( R , V ˉ ) )
where step (i) applies 1 − d ≤ 2 ( 1 − d ~ ) 1-d \le2(1-\tilde{d}) 1 − d ≤ 2 ( 1 − d ~ ) .
Tightness of (105): Finally, the factor of 2 in (105) is tight and cannot be improved: setting R = 0 \mathcal{R}=0 R = 0 , V ˉ = 1 \bar{V}=1 V ˉ = 1 , and d ~ = 0.5 \tilde{d}=0.5 d ~ = 0.5 (so d = 1 d=1 d = 1 ) gives LHS = 1 =1 = 1 and RHS = 1 =1 = 1 .
Proof of Theorem 3.2
In this section, we prove our main calibration result of Theorem 3.2. Like our convention in the main text, we will consistently denote 0-1-c risk and its related quantities without superscripts and hinge risks and their related quantities with ‘hinge’ superscripts, e.g., R c a s c \mathcal{R}_{\mathrm{casc}} R casc , Δ t \Delta_t Δ t and R c a s c h i n g e \mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}} R casc hinge , Δ t h i n g e \Delta^{\mathrm{hinge}}_t Δ t hinge , respectively.
Theorem 3.2 (Linear calibration of ℓ c a s c h i n g e \ell^{\mathrm{hinge}}_{\mathrm{casc}} ℓ casc hinge w.r.t. ℓ \ell ℓ ). Assume Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) Y \perp Z_{1:t-1} \mid(X,Z_t) Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) as in (5). Define R c a s c ∗ : = min π ∈ Π c a s c R ( π ) \mathcal{R}^*_{\mathrm{casc}}:=\min_{\pi\in\Pi_{\mathrm{casc}}}\mathcal{R}(\pi) R casc ∗ := min π ∈ Π casc R ( π ) and R c a s c h i n g e ∗ : = min π ~ ∈ Π ~ c a s c R c a s c h i n g e ( π ~ ) \mathcal{R}^{\mathrm{hinge}*}_{\mathrm{casc}}:=\min_{\tilde{\pi}\in\widetilde{\Pi}_{\mathrm{casc}}}\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi}) R casc hinge ∗ := min π ~ ∈ Π casc R casc hinge ( π ~ ) . Then, for any π ~ ∈ Π ~ c a s c \tilde{\pi}\in\widetilde{\Pi}_{\mathrm{casc}} π ~ ∈ Π casc ,
R ( φ ( π ~ ) ) − R c a s c ≤ 2 T − 1 ( K − 1 ) ( R c a s c h i n g e ( π ~ ) − R c a s c h i n g e ∗ ) . ∗ (107) \mathcal{R}(\varphi(\tilde{\pi}))-\mathcal{R}^*_{\mathrm{casc}\le2^{T-1}(K-1)\left(\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi})-\mathcal{R}^{\mathrm{hinge}*}_{\mathrm{casc}}\right).} \tag*{(107)} R ( φ ( π ~ )) − R casc ≤ 2 T − 1 ( K − 1 ) ( R casc hinge ( π ~ ) − R casc hinge ∗ ) . ∗ (107)
Proof. In this proof, we write φ ( π ~ ) = π = ( f 1 : T , d 1 : T ) \varphi(\tilde{\pi})=\pi=(f_{1:T},d_{1:T}) φ ( π ~ ) = π = ( f 1 : T , d 1 : T ) . Let π c a s c ∗ \pi^*_{\mathrm{casc}} π casc ∗ and π ~ c a s c ∗ \tilde{\pi}^*_{\mathrm{casc}} π ~ casc ∗ denote the Bayes solutions under the target 0-1-c loss and the cascade hinge loss, respectively. That is,
π c a s c ∗ = ( f 1 : T ∗ , d 1 : T ∗ ) ∈ arg min π ∈ Π c a s c R ( π ) , \pi^*_{\mathrm{casc}}=(f^*_{1:T},d^*_{1:T})\in\arg\min_{\pi\in\Pi_{\mathrm{casc}}}\mathcal{R}(\pi), π casc ∗ = ( f 1 : T ∗ , d 1 : T ∗ ) ∈ arg π ∈ Π casc min R ( π ) ,
π ~ c a s c ∗ = ( f ~ 1 : T ∗ , d ~ 1 : T ∗ ) ∈ arg min π ~ ∈ Π ~ c a s c R c a s c h i n g e ( π ~ ) . \tilde{\pi}^*_{\mathrm{casc}}=(\tilde{f}^*_{1:T},\tilde{d}^*_{1:T})\in\arg\min_{\tilde{\pi}\in\widetilde{\Pi}_{\mathrm{casc}}}\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi}). π ~ casc ∗ = ( f ~ 1 : T ∗ , d ~ 1 : T ∗ ) ∈ arg π ~ ∈ Π casc min R casc hinge ( π ~ ) .
Then our end result (20) can be rewritten into
R ( π ) − R ( π c a s c ∗ ) ≤ C 1 ( R c a s c h i n g e ( π ~ ) − R c a s c h i n g e ( π ~ c a s c ∗ ) ) with C 1 = 2 T − 1 ( K − 1 ) . (108) \mathcal{R}(\pi)-\mathcal{R}(\pi^*_{\mathrm{casc}})\le C_1\left(\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi})-\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi}^*_{\mathrm{casc}})\right)\quad\text{with}\quad C_1=2^{T-1}(K-1). \tag*{(108)} R ( π ) − R ( π casc ∗ ) ≤ C 1 ( R casc hinge ( π ~ ) − R casc hinge ( π ~ casc ∗ ) ) with C 1 = 2 T − 1 ( K − 1 ) . (108)
By Lemma C.2 and Corollary C.3, the risks on both sides of (106) equal their expected stage-1 value functions, further reducing (106) to
E [ V 1 π ( H 1 ) ] − E [ V 1 π c a s c ∗ ( H 1 ) ] ≤ C 1 ( E [ V 1 π ~ ( H 1 ) ] − E [ V 1 π ~ c a s c ∗ ( H 1 ) ] ) . (109) \mathbb{E}\left[V^\pi_1(H_1)\right]-\mathbb{E}\left[V^{\pi^*_{\mathrm{casc}}}_1(H_1)\right]\le C_1\left(\mathbb{E}\left[V^{\tilde{\pi}}_1(H_1)\right]-\mathbb{E}\left[V^{\tilde{\pi}^*_{\mathrm{casc}}}_1(H_1)\right]\right). \tag*{(109)} E [ V 1 π ( H 1 ) ] − E [ V 1 π casc ∗ ( H 1 ) ] ≤ C 1 ( E [ V 1 π ~ ( H 1 ) ] − E [ V 1 π ~ casc ∗ ( H 1 ) ] ) . (109)
To prove (107), it suffices in turn to prove for every t ∈ [ T ] t\in[T] t ∈ [ T ] and pointwise in h t ∈ H t h_t\in\mathcal{H}_t h t ∈ H t ,
V t π ( h t ) − V t π c a s c ∗ ( h t ) ⏟ = : Δ t ( h t ) ≤ C t ( V t π ~ ( h t ) − V t π ~ c a s c ∗ ( h t ) ) ⏟ = : Δ t h i n g e ( h t ) with C t = 2 T − t ( K − 1 ) . (110) \underbrace{V^\pi_t(h_t)-V^{\pi^*_{\mathrm{casc}}}_t(h_t)}_{=:\Delta_t(h_t)}
\le C_t\underbrace{\left(V^{\tilde{\pi}}_t(h_t)-V^{\tilde{\pi}^*_{\mathrm{casc}}}_t(h_t)\right)}_{=:\Delta^{\mathrm{hinge}}_t(h_t)}
\quad\text{with}\quad C_t=2^{T-t}(K-1). \tag*{(110)} =: Δ t ( h t ) V t π ( h t ) − V t π casc ∗ ( h t ) ≤ C t =: Δ t hinge ( h t ) ( V t π ~ ( h t ) − V t π ~ casc ∗ ( h t ) ) with C t = 2 T − t ( K − 1 ) . (110)
Here, Δ t ( h t ) \Delta_t(h_t) Δ t ( h t ) and Δ t h i n g e ( h t ) \Delta^{\mathrm{hinge}}_t(h_t) Δ t hinge ( h t ) denote the suboptimality gaps in the stage-t t t value functions associated with the 0-1-c risk and hinge risk, respectively. The form of C t C_t C t arises from unrolling the recursion backward from the terminal stage, where each step of the induction introduces an additional factor of 2, giving C T = K − 1 C_T=K-1 C T = K − 1 up to C 1 = 2 T − 1 ( K − 1 ) C_1=2^{T-1}(K-1) C 1 = 2 T − 1 ( K − 1 ) . In the remainder of the proof, we prove (108) using backward induction.
**Step 1. Base case where t = T t = T t = T : ** By the definition (75) of stage-T T T value functions, (108) specialized to t = T t = T t = T reduces to
R T ( h T ; f T ) − R T ( h T ; f T ∗ ) ⏟ = : Δ T ( h T ) ≤ C T ( R T h i n g e ( h T ; f ~ T ) − R T h i n g e ( h T ; f ~ T ∗ ) ) ⏟ = : Δ T h i n g e ( h T ) with C T = K − 1 , (111) \underbrace{R_T(h_T; f_T) - R_T(h_T; f_T^*)}_{=:\Delta_T(h_T)}
\le C_T \underbrace{\left(R_T^{\mathrm{hinge}}(h_T; \widetilde{f}_T) - R_T^{\mathrm{hinge}}(h_T; \widetilde{f}_T^*)\right)}_{=:\Delta_T^{\mathrm{hinge}}(h_T)}
\quad\text{with } C_T = K - 1, \tag*{(111)} =: Δ T ( h T ) R T ( h T ; f T ) − R T ( h T ; f T ∗ ) ≤ C T =: Δ T hinge ( h T ) ( R T hinge ( h T ; f T ) − R T hinge ( h T ; f T ∗ ) ) with C T = K − 1 , (111)
where Δ T ( h T ) ≥ 0 \Delta_T(h_T) \ge0 Δ T ( h T ) ≥ 0 , Δ T h i n g e ( h T ) ≥ 0 \Delta_T^{\mathrm{hinge}}(h_T) \ge0 Δ T hinge ( h T ) ≥ 0 for all h T ∈ H T h_T \in\mathcal{H}_T h T ∈ H T .
Eq. (109) holds by the single-stage calibration bound of Lemma B.4 with ( x , z t ) replaced by h t . \text{Eq. (109) holds by the single-stage calibration bound of Lemma B.4 with } (x,z_t) \text{ replaced by } h_t. Eq. (109) holds by the single-stage calibration bound of Lemma B.4 with ( x , z t ) replaced by h t .
In (110), Δ T ( h T ) ≥ 0 \Delta_T(h_T) \ge0 Δ T ( h T ) ≥ 0 follows from the Bayes optimality of f T ∗ f_T^* f T ∗ for the 0-1-c c c loss (see explanation above (84)), and Δ T h i n g e ( h T ) ≥ 0 \Delta_T^{\mathrm{hinge}}(h_T) \ge0 Δ T hinge ( h T ) ≥ 0 follows from the Bayes optimality of f ~ T ∗ \widetilde{f}_T^* f T ∗ for the cascade hinge loss. More precisely, Δ T h i n g e ( h T ) ≥ 0 \Delta_T^{\mathrm{hinge}}(h_T) \ge0 Δ T hinge ( h T ) ≥ 0 because of the proof of [53 Lemma 1] and the conditional independence assumption (5).
Step 2. Backward induction: Next, for t ≤ T − 1 t \le T - 1 t ≤ T − 1 , we show that if the suboptimality gaps at stage t + 1 t + 1 t + 1 satisfy:
V t + 1 π ( h t + 1 ) − V t + 1 π ∗ c a s c ( h t + 1 ) ⏟ = : Δ t + 1 ( h t + 1 ) ≤ C t + 1 ( V t + 1 π ~ ( h t + 1 ) − V t + 1 π ~ ∗ c a s c ( h t + 1 ) ) ⏟ = : Δ t + 1 h i n g e ( h t + 1 ) with C t + 1 = 2 T − t − 1 ( K − 1 ) , (112) \underbrace{V_{t+1}^{\pi}(h_{t+1}) - V_{t+1}^{\pi^*\mathrm{casc}}(h_{t+1})}_{=:\Delta_{t+1}(h_{t+1})}
\le C_{t+1}\underbrace{\left(V_{t+1}^{\widetilde{\pi}}(h_{t+1}) - V_{t+1}^{\widetilde{\pi}^*\mathrm{casc}}(h_{t+1})\right)}_{=:\Delta_{t+1}^{\mathrm{hinge}}(h_{t+1})}
\quad\text{with } C_{t+1} = 2^{T-t-1}(K-1), \tag*{(112)} =: Δ t + 1 ( h t + 1 ) V t + 1 π ( h t + 1 ) − V t + 1 π ∗ casc ( h t + 1 ) ≤ C t + 1 =: Δ t + 1 hinge ( h t + 1 ) ( V t + 1 π ( h t + 1 ) − V t + 1 π ∗ casc ( h t + 1 ) ) with C t + 1 = 2 T − t − 1 ( K − 1 ) , (112)
where Δ t + 1 ( h t + 1 ) ≥ 0 \Delta_{t+1}(h_{t+1}) \ge0 Δ t + 1 ( h t + 1 ) ≥ 0 , Δ t + 1 h i n g e ( h t + 1 ) ≥ 0 \Delta_{t+1}^{\mathrm{hinge}}(h_{t+1}) \ge0 Δ t + 1 hinge ( h t + 1 ) ≥ 0 for all h t + 1 ∈ H t + 1 h_{t+1} \in\mathcal{H}_{t+1} h t + 1 ∈ H t + 1 ,
then the corresponding suboptimality gaps at stage t t t satisfy
V t π ( h t ) − V t π ∗ c a s c ( h t ) ⏟ = : Δ t ( h t ) ≤ C t ( V t π ~ ( h t ) − V t π ~ ∗ c a s c ( h t ) ) ⏟ = : Δ t h i n g e ( h t ) with C t = 2 C t + 1 , (113) \underbrace{V_t^{\pi}(h_t) - V_t^{\pi^*\mathrm{casc}}(h_t)}_{=:\Delta_t(h_t)}
\le C_t\underbrace{\left(V_t^{\widetilde{\pi}}(h_t) - V_t^{\widetilde{\pi}^*\mathrm{casc}}(h_t)\right)}_{=:\Delta_t^{\mathrm{hinge}}(h_t)}
\quad\text{with } C_t = 2C_{t+1}, \tag*{(113)} =: Δ t ( h t ) V t π ( h t ) − V t π ∗ casc ( h t ) ≤ C t =: Δ t hinge ( h t ) ( V t π ( h t ) − V t π ∗ casc ( h t ) ) with C t = 2 C t + 1 , (113)
where Δ t ( h t ) ≥ 0 \Delta_t(h_t) \ge0 Δ t ( h t ) ≥ 0 , Δ t h i n g e ( h t ) ≥ 0 \Delta_t^{\mathrm{hinge}}(h_t) \ge0 Δ t hinge ( h t ) ≥ 0 for all h t ∈ H t h_t \in\mathcal{H}_t h t ∈ H t .
Step 2.1. Decomposing the suboptimality gap under the 0-1-c c c loss: Let m m m be the cost achieved by an oracle deferral rule that picks the cheaper routing option at stage t t t , that is,
m : = min { R t ( h t ; f t ) , E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] } . (114) m := \min\left\{R_t(h_t; f_t),\ \mathbb{E}\left[V_{t+1}^{\pi}(H_{t+1}) \mid H_t = h_t\right]\right\}. \tag*{(114)} m := min { R t ( h t ; f t ) , E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] } . (114)
For the induction step, we first expand the Δ t ( h t ) \Delta_t(h_t) Δ t ( h t ) term on the LHS of (113) by adding and subtracting m m m to obtain
Δ t ( h t ) = V t π ( h t ) − V t π ∗ c a s c ( h t ) = V t π ( h t ) − m ⏟ = : Δ t , s e l e c t ( h t ) + m − V t π ∗ c a s c ( h t ) ⏟ = : Δ t , i n t e r n a l ( h t ) . (115) \Delta_t(h_t) = V_t^{\pi}(h_t) - V_t^{\pi^*\mathrm{casc}}(h_t) = \underbrace{V_t^{\pi}(h_t) - m}_{=:\Delta_{t,\mathrm{select}}(h_t)} + \underbrace{m - V_t^{\pi^*\mathrm{casc}}(h_t)}_{=:\Delta_{t,\mathrm{internal}}(h_t)}. \tag*{(115)} Δ t ( h t ) = V t π ( h t ) − V t π ∗ casc ( h t ) = =: Δ t , select ( h t ) V t π ( h t ) − m + =: Δ t , internal ( h t ) m − V t π ∗ casc ( h t ) . (115)
Using the definition of m m m in (114) and the recursive definition of the value function in (75) , we can rewrite the two terms on the RHS of (115) as follows, making their non-negativity explicit:
Δ t , s e l e c t ( h t ) = ( 1 − d t ( h t ) ) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] − min { R t ( h t ; f t ) , E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] } ≥ 0 , (116) \begin{aligned}
\Delta_{t,\mathrm{select}}(h_t) &= (1-d_t(h_t))\mathcal{R}_t(h_t;f_t)+d_t(h_t)\mathbb{E}\left[V^\pi_{t+1}(H_{t+1})\mid H_t=h_t\right]\\
&\quad-\min\left\{\mathcal{R}_t(h_t;f_t),\mathbb{E}\left[V^\pi_{t+1}(H_{t+1})\mid H_t=h_t\right]\right\}\ge0, \tag*{(116)}
\end{aligned} Δ t , select ( h t ) = ( 1 − d t ( h t )) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] − min { R t ( h t ; f t ) , E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] } ≥ 0 , (116)
Δ t , i n t e r n a l ( h t ) = min { R t ( h t ; f t ) , E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] } − min { R t ( h t ; f t ⋆ ) , E [ V t + 1 π c a s c ⋆ ( H t + 1 ) ∣ H t = h t ] } ≥ 0. (117) \begin{aligned}
\Delta_{t,\mathrm{internal}}(h_t) &= \min\left\{\mathcal{R}_t(h_t;f_t),\mathbb{E}\left[V^\pi_{t+1}(H_{t+1})\mid H_t=h_t\right]\right\}\\
&\quad-\min\left\{\mathcal{R}_t(h_t;f_t^\star),\mathbb{E}\left[V^{\pi^\star_{\mathrm{casc}}}_{t+1}(H_{t+1})\mid H_t=h_t\right]\right\}\ge0. \tag*{(117)}
\end{aligned} Δ t , internal ( h t ) = min { R t ( h t ; f t ) , E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] } − min { R t ( h t ; f t ⋆ ) , E [ V t + 1 π casc ⋆ ( H t + 1 ) ∣ H t = h t ] } ≥ 0. (117)
Here, Δ t ( h t ) \Delta_t(h_t) Δ t ( h t ) is decomposed into two terms: Δ t , s e l e c t ( h t ) \Delta_{t,\mathrm{select}}(h_t) Δ t , select ( h t ) measures the suboptimality gap of d t ( h t ) d_t(h_t) d t ( h t ) relative to the oracle deferral rule, while Δ t , i n t e r n a l ( h t ) \Delta_{t,\mathrm{internal}}(h_t) Δ t , internal ( h t ) measures the suboptimality gap of using π \pi π rather than π c a s c ⋆ \pi^\star_{\mathrm{casc}} π casc ⋆ for the downstream stages, holding the deferral choice at t t t as the oracle. We have Δ t , s e l e c t ( h t ) ≥ 0 \Delta_{t,\mathrm{select}}(h_t)\ge0 Δ t , select ( h t ) ≥ 0 because min { a , b } ≤ ( 1 − λ ) a + λ b \min\{a,b\}\le(1-\lambda)a+\lambda b min { a , b } ≤ ( 1 − λ ) a + λb for every λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] . Moreover, Δ t , i n t e r n a l ( h t ) ≥ 0 \Delta_{t,\mathrm{internal}}(h_t)\ge0 Δ t , internal ( h t ) ≥ 0 follows from the Bayes optimality of f t ⋆ f_t^\star f t ⋆ , which gives R t ( h t ; f t ) ≥ R t ( h t ; f t ⋆ ) \mathcal{R}_t(h_t;f_t)\ge\mathcal{R}_t(h_t;f_t^\star) R t ( h t ; f t ) ≥ R t ( h t ; f t ⋆ ) , and the induction hypothesis (112), which assumes Δ t + 1 ( h t + 1 ) = V t + 1 π ( h t + 1 ) − V t + 1 π c a s c ⋆ ( h t + 1 ) ≥ 0 \Delta_{t+1}(h_{t+1})=V^\pi_{t+1}(h_{t+1})-V^{\pi^\star_{\mathrm{casc}}}_{t+1}(h_{t+1})\ge0 Δ t + 1 ( h t + 1 ) = V t + 1 π ( h t + 1 ) − V t + 1 π casc ⋆ ( h t + 1 ) ≥ 0 for all h t + 1 ∈ H t + 1 h_{t+1}\in\mathcal{H}_{t+1} h t + 1 ∈ H t + 1 . Since the sum of two non-negative terms is non-negative, we obtain
Δ t ( h t ) ≥ 0 , (118) \Delta_t(h_t)\ge0, \tag*{(118)} Δ t ( h t ) ≥ 0 , (118)
which is the first part of the conclusion (114) of the induction hypothesis.
Step 2.2. Replacing hard with soft deferral rules: Using (115)–(118) and applying Lemma C.7, we can replace the hard deferral decision d t ( h t ) d_t(h_t) d t ( h t ) in Δ t , s e l e c t ( h t ) \Delta_{t,\mathrm{select}}(h_t) Δ t , select ( h t ) with its soft counterpart d ~ t ( h t ) \widetilde{d}_t(h_t) d t ( h t ) at the cost of a factor of 2, yielding
Δ t ( h t ) = Δ t , s e l e c t ( h t ) + Δ t , i n t e r n a l ( h t ) ≤ 2 ( ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] − m ) + ( m − V t π c a s c ⋆ ( h t ) ) , (119) \begin{aligned}
\Delta_t(h_t)&=\Delta_{t,\mathrm{select}}(h_t)+\Delta_{t,\mathrm{internal}}(h_t)\\
&\le2\left((1-\widetilde{d}_t(h_t))\mathcal{R}_t(h_t;f_t)+\widetilde{d}_t(h_t)\mathbb{E}\left[V^\pi_{t+1}(H_{t+1})\mid H_t=h_t\right]-m\right)+\left(m-V^{\pi^\star_{\mathrm{casc}}}_t(h_t)\right), \tag*{(119)}
\end{aligned} Δ t ( h t ) = Δ t , select ( h t ) + Δ t , internal ( h t ) ≤ 2 ( ( 1 − d t ( h t )) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] − m ) + ( m − V t π casc ⋆ ( h t ) ) , (119)
= 2 ( ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] ) − m − V t π c a s c ⋆ ( h t ) = 2 ( ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] ) − Δ t , i n t e r n a l ( h t ) − 2 V t π c a s c ⋆ ( h t ) ≤ 2 ( ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] − V t π c a s c ⋆ ( h t ) ) , (120) \begin{aligned}
&=2\left((1-\widetilde{d}_t(h_t))\mathcal{R}_t(h_t;f_t)+\widetilde{d}_t(h_t)\mathbb{E}\left[V^\pi_{t+1}(H_{t+1})\mid H_t=h_t\right]\right)-m-V^{\pi^\star_{\mathrm{casc}}}_t(h_t)\\
&=2\left((1-\widetilde{d}_t(h_t))\mathcal{R}_t(h_t;f_t)+\widetilde{d}_t(h_t)\mathbb{E}\left[V^\pi_{t+1}(H_{t+1})\mid H_t=h_t\right]\right)-\Delta_{t,\mathrm{internal}}(h_t)-2V^{\pi^\star_{\mathrm{casc}}}_t(h_t)\\
&\le2\left((1-\widetilde{d}_t(h_t))\mathcal{R}_t(h_t;f_t)+\widetilde{d}_t(h_t)\mathbb{E}\left[V^\pi_{t+1}(H_{t+1})\mid H_t=h_t\right]-V^{\pi^\star_{\mathrm{casc}}}_t(h_t)\right), \tag*{(120)}
\end{aligned} = 2 ( ( 1 − d t ( h t )) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] ) − m − V t π casc ⋆ ( h t ) = 2 ( ( 1 − d t ( h t )) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] ) − Δ t , internal ( h t ) − 2 V t π casc ⋆ ( h t ) ≤ 2 ( ( 1 − d t ( h t )) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] − V t π casc ⋆ ( h t ) ) , (120)
where the equalities follow by rearranging terms and the final inequality follows from Δ t , i n t e r n a l ( h t ) ≥ 0 \Delta_{t,\mathrm{internal}}(h_t)\ge0 Δ t , internal ( h t ) ≥ 0 .
Step 2.3. Connecting Δ t ( h t ) \Delta_t(h_t) Δ t ( h t ) to Δ t h i n g e ( h t ) \Delta_t^{\mathrm{hinge}}(h_t) Δ t hinge ( h t ) : To help connect the 0-1-c suboptimality gap Δ t ( h t ) \Delta_t(h_t) Δ t ( h t ) to its hinge counterpart Δ t h i n g e ( h t ) \Delta_t^{\mathrm{hinge}}(h_t) Δ t hinge ( h t ) , we rewrite the RHS of (120) without the factor of 2, by adding and subtracting the same terms to obtain
1 2 Δ t ( h t ) ≤ ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ~ ( H t + 1 ) ∣ H t = h t ] − V t π ~ ∗ c a s c ( h t ) = ( 1 − d ~ t ( h t ) ) ( R t ( h t ; f t ) − R t ( h t ; f t ∗ ) ) + ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ∗ ) + d ~ t ( h t ) ( E [ V t + 1 π ~ ( H t + 1 ) ∣ H t = h t ] − E [ V t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] ) + d ~ t ( h t ) E [ V t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] − V t π ~ ∗ c a s c ( h t ) = A + B + ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ∗ ) + d ~ t ( h t ) E [ V t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] − V t π ~ ∗ c a s c ( h t ) = A + B + C . (121) \begin{aligned}
\frac{1}{2}\Delta_t(h_t) &\le(1-\widetilde d_t(h_t))\mathcal{R}_t(h_t;f_t)+\widetilde d_t(h_t)\mathbb{E}\left[V^{\widetilde\pi}_{t+1}(H_{t+1})\mid H_t=h_t\right]-V_t^{\widetilde\pi^*\mathrm{casc}}(h_t)\\
&=(1-\widetilde d_t(h_t))\left(\mathcal{R}_t(h_t;f_t)-\mathcal{R}_t(h_t;f_t^*)\right)+(1-\widetilde d_t(h_t))\mathcal{R}_t(h_t;f_t^*)\\
&\qquad+\widetilde d_t(h_t)\left(\mathbb{E}\left[V^{\widetilde\pi}_{t+1}(H_{t+1})\mid H_t=h_t\right]-\mathbb{E}\left[V^{\widetilde\pi^*\mathrm{casc}}_{t+1}(H_{t+1})\mid H_t=h_t\right]\right)\\
&\qquad+\widetilde d_t(h_t)\mathbb{E}\left[V^{\widetilde\pi^*\mathrm{casc}}_{t+1}(H_{t+1})\mid H_t=h_t\right]-V_t^{\widetilde\pi^*\mathrm{casc}}(h_t)\\
&=A+B+(1-\widetilde d_t(h_t))\mathcal{R}_t(h_t;f_t^*)+\widetilde d_t(h_t)\mathbb{E}\left[V^{\widetilde\pi^*\mathrm{casc}}_{t+1}(H_{t+1})\mid H_t=h_t\right]-V_t^{\widetilde\pi^*\mathrm{casc}}(h_t)\\
&=A+B+C. \tag*{(121)}
\end{aligned} 2 1 Δ t ( h t ) ≤ ( 1 − d t ( h t )) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] − V t π ∗ casc ( h t ) = ( 1 − d t ( h t )) ( R t ( h t ; f t ) − R t ( h t ; f t ∗ ) ) + ( 1 − d t ( h t )) R t ( h t ; f t ∗ ) + d t ( h t ) ( E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] − E [ V t + 1 π ∗ casc ( H t + 1 ) ∣ H t = h t ] ) + d t ( h t ) E [ V t + 1 π ∗ casc ( H t + 1 ) ∣ H t = h t ] − V t π ∗ casc ( h t ) = A + B + ( 1 − d t ( h t )) R t ( h t ; f t ∗ ) + d t ( h t ) E [ V t + 1 π ∗ casc ( H t + 1 ) ∣ H t = h t ] − V t π ∗ casc ( h t ) = A + B + C . (121)
Here, A ≥ 0 A\ge0 A ≥ 0 because f t ∗ f_t^* f t ∗ is Bayes-optimal. Term B ≥ 0 B\ge0 B ≥ 0 because the induction hypothesis (112) assumes Δ t + 1 ( h t + 1 ) = V t + 1 π ~ ( h t + 1 ) − V t + 1 π ~ ∗ c a s c ( h t + 1 ) ≥ 0 \Delta_{t+1}(h_{t+1})=V^{\widetilde\pi}_{t+1}(h_{t+1})-V^{\widetilde\pi^*\mathrm{casc}}_{t+1}(h_{t+1})\ge0 Δ t + 1 ( h t + 1 ) = V t + 1 π ( h t + 1 ) − V t + 1 π ∗ casc ( h t + 1 ) ≥ 0 for all h t + 1 ∈ H t + 1 h_{t+1}\in\mathcal{H}_{t+1} h t + 1 ∈ H t + 1 . Term C ≥ 0 C\ge0 C ≥ 0 because, using (104) and the fact that min { a , b } ≤ ( 1 − λ ) a + λ b \min\{a,b\}\le(1-\lambda)a+\lambda b min { a , b } ≤ ( 1 − λ ) a + λb for every λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , we have
V t π ~ ∗ c a s c ( h t ) = min { R t ( h t ; f t ∗ ) , E [ V t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] } ≤ ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ∗ ) + d ~ t ( h t ) E [ V t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] . (122) \begin{aligned}
V_t^{\widetilde\pi^*\mathrm{casc}}(h_t)=\min\left\{\mathcal{R}_t(h_t;f_t^*),\mathbb{E}\left[V^{\widetilde\pi^*\mathrm{casc}}_{t+1}(H_{t+1})\mid H_t=h_t\right]\right\} \\
&\le(1-\widetilde d_t(h_t))\mathcal{R}_t(h_t;f_t^*)+\widetilde d_t(h_t)\mathbb{E}\left[V^{\widetilde\pi^*\mathrm{casc}}_{t+1}(H_{t+1})\mid H_t=h_t\right]. \tag*{(122)}
\end{aligned} V t π ∗ casc ( h t ) = min { R t ( h t ; f t ∗ ) , E [ V t + 1 π ∗ casc ( H t + 1 ) ∣ H t = h t ] } ≤ ( 1 − d t ( h t )) R t ( h t ; f t ∗ ) + d t ( h t ) E [ V t + 1 π ∗ casc ( H t + 1 ) ∣ H t = h t ] . (122)
We upper bound terms A A A , B B B , C C C separately with their hinge counterparts. By single-stage calibration (Lemma B.4 , with h t h_t h t in place of x x x , z t z_t z t ), similar to (111) , we have
A ≤ ( K − 1 ) ( 1 − d ~ t ( h t ) ) ( R t h i n g e ( h t ; f ~ t ) − R t h i n g e ( h t ; f ~ t ∗ ) ) = ( K − 1 ) A ~ . (123) A\le(K-1)(1-\widetilde d_t(h_t))\left(\mathcal{R}^{\mathrm{hinge}}_t(h_t;\widetilde f_t)-\mathcal{R}^{\mathrm{hinge}}_t(h_t;\widetilde f_t^*)\right)=(K-1)\widetilde A. \tag*{(123)} A ≤ ( K − 1 ) ( 1 − d t ( h t )) ( R t hinge ( h t ; f t ) − R t hinge ( h t ; f t ∗ ) ) = ( K − 1 ) A . (123)
Since f ~ t ∗ \widetilde f_t^* f t ∗ is Bayes-optimal for the cascade hinge loss, R t h i n g e ( h t ; f ~ t ) − R t h i n g e ( h t ; f ~ t ∗ ) ≥ 0 \mathcal{R}^{\mathrm{hinge}}_t(h_t;\widetilde f_t)-\mathcal{R}^{\mathrm{hinge}}_t(h_t;\widetilde f_t^*)\ge0 R t hinge ( h t ; f t ) − R t hinge ( h t ; f t ∗ ) ≥ 0 , and so A ~ ≥ 0 \widetilde A\ge0 A ≥ 0 . By the induction hypothesis (112) , we can write
B = d ~ t ( h t ) E [ V t + 1 π ~ ( H t + 1 ) − V t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] ≤ d ~ t ( h t ) C t + 1 E [ V t + 1 π ~ ( H t + 1 ) − V t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] = C t + 1 d ~ t ( h t ) E [ V t + 1 π ~ ( H t + 1 ) − V t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] = C t + 1 B ~ . (124) \begin{aligned}
B&=\widetilde d_t(h_t)\mathbb{E}\left[V^{\widetilde\pi}_{t+1}(H_{t+1})-V^{\widetilde\pi^*\mathrm{casc}}_{t+1}(H_{t+1})\mid H_t=h_t\right]\\
&\le\widetilde d_t(h_t)C_{t+1}\mathbb{E}\left[V^{\widetilde\pi}_{t+1}(H_{t+1})-V^{\widetilde\pi^*\mathrm{casc}}_{t+1}(H_{t+1})\mid H_t=h_t\right]\\
&=C_{t+1}\widetilde d_t(h_t)\mathbb{E}\left[V^{\widetilde\pi}_{t+1}(H_{t+1})-V^{\widetilde\pi^*\mathrm{casc}}_{t+1}(H_{t+1})\mid H_t=h_t\right]=C_{t+1}\widetilde B. \tag*{(124)}
\end{aligned} B = d t ( h t ) E [ V t + 1 π ( H t + 1 ) − V t + 1 π ∗ casc ( H t + 1 ) ∣ H t = h t ] ≤ d t ( h t ) C t + 1 E [ V t + 1 π ( H t + 1 ) − V t + 1 π ∗ casc ( H t + 1 ) ∣ H t = h t ] = C t + 1 d t ( h t ) E [ V t + 1 π ( H t + 1 ) − V t + 1 π ∗ casc ( H t + 1 ) ∣ H t = h t ] = C t + 1 B . (124)
where B ~ ≥ 0 \widetilde B\ge0 B ≥ 0 because the induction hypothesis (112) assumes Δ t + 1 h i n g e ( h t + 1 ) = V t + 1 π ~ ( h t + 1 ) − V t + 1 π ~ ∗ c a s c ( h t + 1 ) ≥ 0 \Delta^{\mathrm{hinge}}_{t+1}(h_{t+1})=V^{\widetilde\pi}_{t+1}(h_{t+1})-V^{\widetilde\pi^*\mathrm{casc}}_{t+1}(h_{t+1})\ge0 Δ t + 1 hinge ( h t + 1 ) = V t + 1 π ( h t + 1 ) − V t + 1 π ∗ casc ( h t + 1 ) ≥ 0 for all h t + 1 ∈ H t + 1 h_{t+1}\in\mathcal{H}_{t+1} h t + 1 ∈ H t + 1 .
Finally, since R t ( h t ; f t ∗ ) = K − 1 K R t h i n g e ( h t ; f ~ t ∗ ) \mathcal{R}_t(h_t;f_t^*)=\frac{K-1}{K}\mathcal{R}^{\mathrm{hinge}}_t(h_t;\widetilde f_t^*) R t ( h t ; f t ∗ ) = K K − 1 R t hinge ( h t ; f t ∗ ) and V t π ~ ∗ c a s c ( h t ) = K − 1 K V t π ~ ∗ c a s c ( h t ) V_t^{\widetilde\pi^*\mathrm{casc}}(h_t)=\frac{K-1}{K}V_t^{\widetilde\pi^*\mathrm{casc}}(h_t) V t π ∗ casc ( h t ) = K K − 1 V t π ∗ casc ( h t ) by Lemma C.6 for all t ∈ [ T ] t\in[T] t ∈ [ T ] , we can write
C = ( 1 − d ~ t ( h t ) ) K − 1 K R t h i n g e ( h t ; f ~ t ∗ ) + d ~ t ( h t ) K − 1 K E [ V t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] − K − 1 K V t π ~ ∗ c a s c ( h t ) = K − 1 K ( ( 1 − d ~ t ( h t ) ) R t h i n g e ( h t ; f ~ t ∗ ) + d ~ t ( h t ) E [ V t + 1 π ~ ∗ c a s c ( H t + 1 ) ∣ H t = h t ] − V t π ~ ∗ c a s c ( h t ) ) = K − 1 K C ~ . (125) \begin{aligned}
C&=(1-\widetilde d_t(h_t))\frac{K-1}{K}\mathcal{R}^{\mathrm{hinge}}_t(h_t;\widetilde f_t^*)+\widetilde d_t(h_t)\frac{K-1}{K}\mathbb{E}\left[V^{\widetilde\pi^*\mathrm{casc}}_{t+1}(H_{t+1})\mid H_t=h_t\right]-\frac{K-1}{K}V_t^{\widetilde\pi^*\mathrm{casc}}(h_t)\\
&=\frac{K-1}{K}\left((1-\widetilde d_t(h_t))\mathcal{R}^{\mathrm{hinge}}_t(h_t;\widetilde f_t^*)+\widetilde d_t(h_t)\mathbb{E}\left[V^{\widetilde\pi^*\mathrm{casc}}_{t+1}(H_{t+1})\mid H_t=h_t\right]-V_t^{\widetilde\pi^*\mathrm{casc}}(h_t)\right)=\frac{K-1}{K}\widetilde C. \tag*{(125)}
\end{aligned} C = ( 1 − d t ( h t )) K K − 1 R t hinge ( h t ; f t ∗ ) + d t ( h t ) K K − 1 E [ V t + 1 π ∗ casc ( H t + 1 ) ∣ H t = h t ] − K K − 1 V t π ∗ casc ( h t ) = K K − 1 ( ( 1 − d t ( h t )) R t hinge ( h t ; f t ∗ ) + d t ( h t ) E [ V t + 1 π ∗ casc ( H t + 1 ) ∣ H t = h t ] − V t π ∗ casc ( h t ) ) = K K − 1 C . (125)
where C ~ ≥ 0 \widetilde{C} \ge0 C ≥ 0 following the same argument as that in (122) for C ≥ 0 C \ge0 C ≥ 0 , using (105) instead of (104) .
Step 2.4. Assembling all terms: Finally, recall from (112) that C t + 1 = 2 T − t − 1 ( K − 1 ) C_{t+1} = 2^{T-t-1}(K-1) C t + 1 = 2 T − t − 1 ( K − 1 ) . Since t ≤ T − 1 t \le T-1 t ≤ T − 1 implies 2 T − t − 1 ≥ 1 2^{T-t-1} \ge1 2 T − t − 1 ≥ 1 , the pre-multiplicative factors K − 1 K-1 K − 1 , C t + 1 C_{t+1} C t + 1 , and K − 1 K \frac{K-1}{K} K K − 1 in (123) , (124) , and (125) , respectively, are all dominated by C t + 1 C_{t+1} C t + 1 . Together with A ~ , B ~ , C ~ ≥ 0 \widetilde{A},\widetilde{B},\widetilde{C} \ge0 A , B , C ≥ 0 , we have A ≤ C t + 1 A ~ A \le C_{t+1}\widetilde{A} A ≤ C t + 1 A , B ≤ C t + 1 B ~ B \le C_{t+1}\widetilde{B} B ≤ C t + 1 B , and C ≤ C t + 1 C ~ C \le C_{t+1}\widetilde{C} C ≤ C t + 1 C .
Substituting this into (121) and rearranging yields
Δ t ( h t ) ≤ 2 ( A + B + C ) ≤ 2 ⋅ C t + 1 ( A ~ + B ~ + C ~ ) = C t ( A ~ + B ~ + C ~ ) . (126) \Delta_t(h_t) \le2(A+B+C) \le2 \cdot C_{t+1}(\widetilde{A}+\widetilde{B}+\widetilde{C}) = C_t(\widetilde{A}+\widetilde{B}+\widetilde{C}). \tag*{(126)} Δ t ( h t ) ≤ 2 ( A + B + C ) ≤ 2 ⋅ C t + 1 ( A + B + C ) = C t ( A + B + C ) . (126)
By the definition of A ~ , B ~ \widetilde{A},\widetilde{B} A , B , and C ~ \widetilde{C} C from (123) , (124) , and (125) , we can write
A ~ + B ~ + C ~ = ( 1 − d ~ t ( h t ) ) ( R t h i n g e ( h t ; f ~ t ) − R t h i n g e ( h t ; f ~ t ∗ ) ) + d ~ t ( h t ) E [ V t + 1 π ~ ( H t + 1 ) − V t + 1 π ~ c a s c ∗ ( H t + 1 ) ∣ H t = h t ] + ( ( 1 − d ~ t ( h t ) ) R t h i n g e ( h t ; f ~ t ∗ ) + d ~ t ( h t ) E [ V t + 1 π ~ c a s c ∗ ( H t + 1 ) ∣ H t = h t ] − V t π ~ c a s c ∗ ( h t ) ) = ( 1 − d ~ t ( h t ) ) R t h i n g e ( h t ; f ~ t ) + d ~ t ( h t ) E [ V t + 1 π ~ ( H t + 1 ) ∣ H t = h t ] − V t π ~ c a s c ∗ ( h t ) = ( i ) V t π ~ ( h t ) − V t π ~ c a s c ∗ ( h t ) = ( i i ) Δ t h i n g e ( h t ) . (127) \begin{aligned}
\widetilde{A}+\widetilde{B}+\widetilde{C}
={}&(1-\widetilde{d}_t(h_t))\left(\mathcal{R}^{\mathrm{hinge}}_t(h_t;\widetilde{f}_t)-\mathcal{R}^{\mathrm{hinge}}_t(h_t;\widetilde{f}_t^*)\right)\\
&+\widetilde{d}_t(h_t)\mathbb{E}\left[V^{\widetilde{\pi}}_{t+1}(H_{t+1})-V^{\widetilde{\pi}^*_{\mathrm{casc}}}_{t+1}(H_{t+1})\mid H_t=h_t\right]\\
&+\left((1-\widetilde{d}_t(h_t))\mathcal{R}^{\mathrm{hinge}}_t(h_t;\widetilde{f}_t^*)+\widetilde{d}_t(h_t)\mathbb{E}\left[V^{\widetilde{\pi}^*_{\mathrm{casc}}}_{t+1}(H_{t+1})\mid H_t=h_t\right]-V^{\widetilde{\pi}^*_{\mathrm{casc}}}_t(h_t)\right)\\
={}&(1-\widetilde{d}_t(h_t))\mathcal{R}^{\mathrm{hinge}}_t(h_t;\widetilde{f}_t)+\widetilde{d}_t(h_t)\mathbb{E}\left[V^{\widetilde{\pi}}_{t+1}(H_{t+1})\mid H_t=h_t\right]-V^{\widetilde{\pi}^*_{\mathrm{casc}}}_t(h_t)\\
&\mathrel{\overset{\mathrm{(i)}}{=}} V^{\widetilde{\pi}}_t(h_t)-V^{\widetilde{\pi}^*_{\mathrm{casc}}}_t(h_t)
\mathrel{\overset{\mathrm{(ii)}}{=}} \Delta^{\mathrm{hinge}}_t(h_t). \tag*{(127)}
\end{aligned} A + B + C = = ( 1 − d t ( h t )) ( R t hinge ( h t ; f t ) − R t hinge ( h t ; f t ∗ ) ) + d t ( h t ) E [ V t + 1 π ( H t + 1 ) − V t + 1 π casc ∗ ( H t + 1 ) ∣ H t = h t ] + ( ( 1 − d t ( h t )) R t hinge ( h t ; f t ∗ ) + d t ( h t ) E [ V t + 1 π casc ∗ ( H t + 1 ) ∣ H t = h t ] − V t π casc ∗ ( h t ) ) ( 1 − d t ( h t )) R t hinge ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] − V t π casc ∗ ( h t ) = ( i ) V t π ( h t ) − V t π casc ∗ ( h t ) = ( ii ) Δ t hinge ( h t ) . (127)
where step (i) applies the definition of V t π ~ ( h t ) V^{\widetilde{\pi}}_t(h_t) V t π ( h t ) (see (75) ), and step (ii) uses the definition of Δ t h i n g e ( h t ) \Delta^{\mathrm{hinge}}_t(h_t) Δ t hinge ( h t ) in (113) . Combining (126) and (127) proves (113) . Together with the result Δ t ( h t ) ≥ 0 \Delta_t(h_t) \ge0 Δ t ( h t ) ≥ 0 established in (118) , this implies Δ t h i n g e ( h t ) ≥ 0 \Delta^{\mathrm{hinge}}_t(h_t) \ge0 Δ t hinge ( h t ) ≥ 0 , proving the second part of (114). Taken together, we have completed the backward induction step.
Step 3. Conclusion: Applying this argument inductively down to t = 1 t=1 t = 1 yields (110) for all t ∈ [ T ] t \in[T] t ∈ [ T ] . Setting t = 1 t=1 t = 1 in (110) and taking total expectation over H 1 H_1 H 1 proves (108) , which in turn proves (20). □
Lower Bound
We prove the following lower bound to show that 2 T − 1 ( K − 1 ) 2^{T-1}(K-1) 2 T − 1 ( K − 1 ) is the smallest constant for which the upper bound in Theorem 3.2 holds uniformly over all data distributions P P P in (4) satisfying (5) , cost vectors c = [ κ 1 , κ 1 + κ 2 , … , ∑ t = 1 T κ t ] T c = [\kappa_1,\kappa_1+\kappa_2,\ldots,\sum_{t=1}^{T}\kappa_t]^\mathsf{T} c = [ κ 1 , κ 1 + κ 2 , … , ∑ t = 1 T κ t ] T with κ 1 = 0 \kappa_1=0 κ 1 = 0 and κ t ≥ κ t − 1 \kappa_t \ge\kappa_{t-1} κ t ≥ κ t − 1 , and multi-model classifiers π ~ ∈ Π ~ c a s c \widetilde{\pi} \in\widetilde{\Pi}_{\mathrm{casc}} π ∈ Π casc .
Proposition C.8 (Lower bound). Let K ≥ 2 K \ge2 K ≥ 2 and T ≥ 2 T \ge2 T ≥ 2 . For every C < 2 T − 1 ( K − 1 ) C < 2^{T-1}(K-1) C < 2 T − 1 ( K − 1 ) , there exists a distribution P P P satisfying (5) , an admissible cost vector c c c , and π ~ ∈ Π ~ c a s c \widetilde{\pi} \in\widetilde{\Pi}_{\mathrm{casc}} π ∈ Π casc such that
R ( φ ( π ~ ) ) − R c a s c ∗ > C ( R c a s c h i n g e ( π ~ ) − R c a s c h i n g e ∗ ) . (128) \mathcal{R}(\varphi(\widetilde{\pi}))-\mathcal{R}^*_{\mathrm{casc}} > C\left(\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\widetilde{\pi})-\mathcal{R}^{\mathrm{hinge}*}_{\mathrm{casc}}\right). \tag*{(128)} R ( φ ( π )) − R casc ∗ > C ( R casc hinge ( π ) − R casc hinge ∗ ) . (128)
Proof. We construct a bad instance of P P P , c c c and π ~ \widetilde{\pi} π , and show that the excess risks of the bad instance have a ratio that approaches 2 T − 1 ( K − 1 ) 2^{T-1}(K-1) 2 T − 1 ( K − 1 ) .
Step 1. Choose P P P and c c c for the bad instance: Set all incremental costs to zero, i.e., κ t = 0 \kappa_t=0 κ t = 0 for all t ∈ [ T ] t \in[T] t ∈ [ T ] , giving the all-zero cost vector c = 0 T c=0_T c = 0 T . Let X X X have any distribution on X \mathcal{X} X , and let Z 1 , Z 2 , … , Z T Z_1,Z_2,\ldots,Z_T Z 1 , Z 2 , … , Z T be constant. Let Y Y Y be independent of X X X with
p y : = Pr ( Y = y ) = { 1 K − 1 for y ∈ [ K − 1 ] , 0 for y = K . (129) p_y := \Pr(Y=y) =
\begin{cases}
\frac{1}{K-1} & \text{for } y \in[K-1],\\
0 & \text{for } y=K.
\end{cases} \tag*{(129)} p y := Pr ( Y = y ) = { K − 1 1 0 for y ∈ [ K − 1 ] , for y = K . (129)
This ensures Y Y Y to be independent of the full history H T = ( X , Z 1 : T ) H_T=(X,Z_{1:T}) H T = ( X , Z 1 : T ) . In particular, (5) holds and p ( y ∣ x , z 1 : t ) = p ( y ∣ x , z t ) = p y p(y\mid x,z_{1:t})=p(y\mid x,z_t)=p_y p ( y ∣ x , z 1 : t ) = p ( y ∣ x , z t ) = p y for all t ∈ [ T ] t\in[T] t ∈ [ T ] .
Using the definition of ℓ h i n g e \ell^{\mathrm{hinge}} ℓ hinge from (15) , for a score vector v ∈ R K v\in\mathbb{R}^K v ∈ R K with ∑ k ∈ [ K ] v k = 0 \sum_{k\in[K]}v_k=0 ∑ k ∈ [ K ] v k = 0 , we can write that
ℓ h i n g e ( v , y ; 0 ) = ∑ k ∈ [ K ] ∖ y [ v k + 1 K − 1 ] + + K K − 1 ⋅ 0 = ∑ k ∈ [ K ] ∖ y a k = : ℓ ˉ h i n g e ( v , y ) . (130) \ell^{\mathrm{hinge}}(v,y;0)=\sum_{k\in[K]\setminus y}\left[v_k+\frac{1}{K-1}\right]_+ + \frac{K}{K-1}\cdot0=\sum_{k\in[K]\setminus y}a_k=:\bar{\ell}^{\mathrm{hinge}}(v,y). \tag*{(130)} ℓ hinge ( v , y ; 0 ) = k ∈ [ K ] ∖ y ∑ [ v k + K − 1 1 ] + + K − 1 K ⋅ 0 = k ∈ [ K ] ∖ y ∑ a k =: ℓ ˉ hinge ( v , y ) . (130)
Define L ( v ) L(v) L ( v ) as follows, and expanding and rearranging gives
L ( v ) : = ∑ y ∈ [ K ] p y ℓ h i n g e ( v , y ) = ∑ y ∈ [ K ] p y ∑ k ∈ [ K ] ∖ y a k = (i) ∑ k ∈ [ K ] a k ∑ y ∈ [ K ] ∖ k p y = ∑ k ∈ [ K ] a k ( 1 − p k ) = (ii) ∑ k = 1 K − 1 a k ⋅ K − 1 K − 1 + a K . (131) \begin{aligned}
L(v)&:=\sum_{y\in[K]}p_y\ell^{\mathrm{hinge}}(v,y)=\sum_{y\in[K]}p_y\sum_{k\in[K]\setminus y}a_k \overset{\text{(i)}}{=}\sum_{k\in[K]}a_k\sum_{y\in[K]\setminus k}p_y\\
&=\sum_{k\in[K]}a_k(1-p_k)\overset{\text{(ii)}}{=}\sum_{k=1}^{K-1}a_k\cdot\frac{K-1}{K-1}+a_K. \tag*{(131)}
\end{aligned} L ( v ) := y ∈ [ K ] ∑ p y ℓ hinge ( v , y ) = y ∈ [ K ] ∑ p y k ∈ [ K ] ∖ y ∑ a k = (i) k ∈ [ K ] ∑ a k y ∈ [ K ] ∖ k ∑ p y = k ∈ [ K ] ∑ a k ( 1 − p k ) = (ii) k = 1 ∑ K − 1 a k ⋅ K − 1 K − 1 + a K . (131)
where step (i) exchanges summations, and step (ii) applies (129) . Moreover, we can lower bound L ( v ) L(v) L ( v ) by
L ( v ) ≥ (i) K − 2 K − 1 ∑ k ∈ [ K ] a k ≥ (ii) K − 2 K − 1 ∑ k ∈ [ K ] [ v k + 1 K − 1 ] = (iii) K − 2 K − 1 ⋅ K ⋅ 1 K − 1 = K ( K − 2 ) ( K − 1 ) 2 = : L ∗ . (132) L(v)\overset{\text{(i)}}{\ge}\frac{K-2}{K-1}\sum_{k\in[K]}a_k\overset{\text{(ii)}}{\ge}\frac{K-2}{K-1}\sum_{k\in[K]}\left[v_k+\frac{1}{K-1}\right]\overset{\text{(iii)}}{=}\frac{K-2}{K-1}\cdot K\cdot\frac{1}{K-1}=\frac{K(K-2)}{(K-1)^2}=:L^*. \tag*{(132)} L ( v ) ≥ (i) K − 1 K − 2 k ∈ [ K ] ∑ a k ≥ (ii) K − 1 K − 2 k ∈ [ K ] ∑ [ v k + K − 1 1 ] = (iii) K − 1 K − 2 ⋅ K ⋅ K − 1 1 = ( K − 1 ) 2 K ( K − 2 ) =: L ∗ . (132)
where step (i) uses a k ≥ 0 a_k\ge0 a k ≥ 0 for all k ∈ [ K ] k\in[K] k ∈ [ K ] , step (ii) uses [ x ] + ≥ x [x]_+\ge x [ x ] + ≥ x for any x ∈ R x\in\mathbb{R} x ∈ R , and step (iii) applies ∑ k ∈ [ K ] v k = 0 \sum_{k\in[K]}v_k=0 ∑ k ∈ [ K ] v k = 0 . Define a score vector v ∗ ∈ R K v^*\in\mathbb{R}^K v ∗ ∈ R K by v 1 ∗ = 1 v_1^*=1 v 1 ∗ = 1 and v k ∗ = − 1 K − 1 v_k^*=-\frac{1}{K-1} v k ∗ = − K − 1 1 for all 2 ≤ k ≤ K 2\le k\le K 2 ≤ k ≤ K , which satisfies ∑ k ∈ [ K ] v k ∗ = 0 \sum_{k\in[K]}v_k^*=0 ∑ k ∈ [ K ] v k ∗ = 0 . Plugging v = v ∗ v=v^* v = v ∗ into (131) and applying p 1 = 1 K − 1 p_1=\frac{1}{K-1} p 1 = K − 1 1 from (129) obtains
L ( v ∗ ) = K K − 1 K − 2 K − 1 + 0 = K ( K − 2 ) ( K − 1 ) 2 = L ∗ . (133) L(v^*)=\frac{K}{K-1}\frac{K-2}{K-1}+0=\frac{K(K-2)}{(K-1)^2}=L^*. \tag*{(133)} L ( v ∗ ) = K − 1 K K − 1 K − 2 + 0 = ( K − 1 ) 2 K ( K − 2 ) = L ∗ . (133)
Step 2. Compute Bayes risks R ∗ \mathcal{R}^* R ∗ and R c a s c h i n g e ∗ \mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}*} R casc hinge ∗ : We now show that in the bad instance that we have set up, the Bayes 0-1-c c c risk and the Bayes cascade hinge risk are R ∗ = K − 2 K − 1 \mathcal{R}^*=\frac{K-2}{K-1} R ∗ = K − 1 K − 2 and R c a s c h i n g e ∗ = L ∗ \mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}*}=L^* R casc hinge ∗ = L ∗ , respectively.
First, using the definition of the 0-1-c c c risk R ( π ) \mathcal{R}(\pi) R ( π ) in (9) –(10) , we can express R ( π ) \mathcal{R}(\pi) R ( π ) in the bad instance as
R ( π ) = E [ 1 { f r ( H ) ( X , Z r ( H ) ) ≠ Y } + c r ( H ) ] = Pr ( f r ( H ) ( X , Z r ( H ) ) ≠ Y ) = 1 − Pr ( Y = f r ( H ) ( X , Z r ( H ) ) ) . (134) \begin{aligned}
\mathcal{R}(\pi)&=\mathbb{E}\left[\mathbb{1}\{f_{r(H)}(X,Z_{r(H)})\ne Y\}+c_{r(H)}\right]\\
&=\Pr\left(f_{r(H)}(X,Z_{r(H)})\ne Y\right)=1-\Pr\left(Y=f_{r(H)}(X,Z_{r(H)})\right). \tag*{(134)}
\end{aligned} R ( π ) = E [ 1 { f r ( H ) ( X , Z r ( H ) ) = Y } + c r ( H ) ] = Pr ( f r ( H ) ( X , Z r ( H ) ) = Y ) = 1 − Pr ( Y = f r ( H ) ( X , Z r ( H ) ) ) . (134)
and, using (129) , we can lower bound it by
R ( π ) ≥ 1 − max y ∈ [ K ] p y = 1 − 1 K − 1 = K − 2 K − 1 . (135) \mathcal{R}(\pi)\ge1-\max_{y\in[K]}p_y=1-\frac{1}{K-1}=\frac{K-2}{K-1}. \tag*{(135)} R ( π ) ≥ 1 − y ∈ [ K ] max p y = 1 − K − 1 1 = K − 1 K − 2 . (135)
Note that equality holds for f t ≡ 1 f_t\equiv1 f t ≡ 1 for all t ∈ [ T ] t\in[T] t ∈ [ T ] and any router r r r , attaining the lower bound K − 2 K − 1 \frac{K-2}{K-1} K − 1 K − 2 . Hence, the Bayes 0-1-c c c risk is
R c a s c ∗ = K − 2 K − 1 . (136) \mathcal{R}_{\mathrm{casc}}^*=\frac{K-2}{K-1}. \tag*{(136)} R casc ∗ = K − 1 K − 2 . (136)
Second, using the definition of the hinge risk R c a s c h i n g e ( π ~ ) \mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}}(\widetilde{\pi}) R casc hinge ( π ) below (18)–(19), we can express R c a s c h i n g e ( π ~ ) \mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}}(\widetilde{\pi}) R casc hinge ( π ) in the bad instance as
R c a s c h i n g e ( π ~ ) = E [ ∑ t = 1 T r ~ t ( H t ) ℓ h i n g e ( f ~ t ( X , Z t ) , Y ; c t ) ] = ( i ) E [ ∑ t = 1 T r ~ t ( H t ) ℓ ˉ h i n g e ( f ~ t ( X , Z t ) , Y ) ] = ∑ t = 1 T E [ r ~ t ( H t ) E [ ℓ ˉ h i n g e ( f ~ t ( X , Z t ) , Y ) ∣ H t ] ] = ( i i ) ∑ t = 1 T E [ r ~ t ( H t ) ∑ y ∈ [ K ] p y ℓ ˉ h i n g e ( f ~ t ( X , Z t ) , y ) ] = ( i i i ) ∑ t = 1 T E [ r ~ t ( H t ) L ( f ~ t ( X , Z t ) ) ] , (137) \begin{aligned}
\mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}}(\widetilde{\pi})
&= \mathbb{E}\left[\sum_{t=1}^{T}\widetilde{r}_t(H_t)\ell^{\mathrm{hinge}}\left(\widetilde{f}_t(X,Z_t),Y;c_t\right)\right]\\
&\overset{\mathrm{(i)}}{=} \mathbb{E}\left[\sum_{t=1}^{T}\widetilde{r}_t(H_t)\bar{\ell}^{\mathrm{hinge}}\left(\widetilde{f}_t(X,Z_t),Y\right)\right]\\
&= \sum_{t=1}^{T}\mathbb{E}\left[\widetilde{r}_t(H_t)\mathbb{E}\left[\bar{\ell}^{\mathrm{hinge}}\left(\widetilde{f}_t(X,Z_t),Y\right)\mid H_t\right]\right]\\
&\overset{\mathrm{(ii)}}{=} \sum_{t=1}^{T}\mathbb{E}\left[\widetilde{r}_t(H_t)\sum_{y\in[K]}p_y\bar{\ell}^{\mathrm{hinge}}\left(\widetilde{f}_t(X,Z_t),y\right)\right]\\
&\overset{\mathrm{(iii)}}{=} \sum_{t=1}^{T}\mathbb{E}\left[\widetilde{r}_t(H_t)L\left(\widetilde{f}_t(X,Z_t)\right)\right], \tag*{(137)}
\end{aligned} R casc hinge ( π ) = E [ t = 1 ∑ T r t ( H t ) ℓ hinge ( f t ( X , Z t ) , Y ; c t ) ] = ( i ) E [ t = 1 ∑ T r t ( H t ) ℓ ˉ hinge ( f t ( X , Z t ) , Y ) ] = t = 1 ∑ T E [ r t ( H t ) E [ ℓ ˉ hinge ( f t ( X , Z t ) , Y ) ∣ H t ] ] = ( ii ) t = 1 ∑ T E r t ( H t ) y ∈ [ K ] ∑ p y ℓ ˉ hinge ( f t ( X , Z t ) , y ) = ( iii ) t = 1 ∑ T E [ r t ( H t ) L ( f t ( X , Z t ) ) ] , (137)
where step (i) applies the definition of ℓ ˉ h i n g e \bar{\ell}^{\mathrm{hinge}} ℓ ˉ hinge in (130) , step (ii) uses Pr ( Y = y ∣ H t ) = p y \Pr(Y=y\mid H_t)=p_y Pr ( Y = y ∣ H t ) = p y , and step (iii) uses the definition of L L L in (131) . Using (132) and ∑ t = 1 T r ~ t ( H t ) = 1 \sum_{t=1}^{T}\widetilde{r}_t(H_t)=1 ∑ t = 1 T r t ( H t ) = 1 , we can further lower bound R c a s c h i n g e ( π ~ ) \mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}}(\widetilde{\pi}) R casc hinge ( π ) by
R c a s c h i n g e ( π ~ ) ≥ ∑ t = 1 T E [ r ~ t ( H t ) L ∗ ] = E [ ∑ t = 1 T r ~ t ( H t ) ] ⋅ L ∗ = L ∗ . (138) \mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}}(\widetilde{\pi})
\geq\sum_{t=1}^{T}\mathbb{E}\left[\widetilde{r}_t(H_t)L^*\right]
= \mathbb{E}\left[\sum_{t=1}^{T}\widetilde{r}_t(H_t)\right]\cdot L^*
= L^*. \tag*{(138)} R casc hinge ( π ) ≥ t = 1 ∑ T E [ r t ( H t ) L ∗ ] = E [ t = 1 ∑ T r t ( H t ) ] ⋅ L ∗ = L ∗ . (138)
Note that (133) implies that equality holds for f ~ t = v ∗ \widetilde{f}_t=v^* f t = v ∗ for all t ∈ [ T ] t\in[T] t ∈ [ T ] and for any soft router r ~ 1 , … , r ~ T \widetilde{r}_1,\ldots,\widetilde{r}_T r 1 , … , r T . Hence, the Bayes cascade hinge risk is
R c a s c h i n g e ∗ = L ∗ . (139) \mathcal{R}_{\mathrm{casc}}^{\mathrm{hinge}*}=L^*. \tag*{(139)} R casc hinge ∗ = L ∗ . (139)
Step 3. Choose π ~ ∈ Π ~ c a s c \widetilde{\pi}\in\widetilde{\Pi}_{\mathrm{casc}} π ∈ Π casc for the bad instance: Recall that v ∗ ∈ R K v^*\in\mathbb{R}^K v ∗ ∈ R K is a score vector with v 1 ∗ = 1 v_1^*=1 v 1 ∗ = 1 and v k ∗ = − 1 K − 1 v_k^*=-\frac{1}{K-1} v k ∗ = − K − 1 1 for all 2 ≤ k ≤ K 2\leq k\leq K 2 ≤ k ≤ K . Fix ϵ ∈ [ 0 , 1 2 ] \epsilon\in[0,\frac{1}{2}] ϵ ∈ [ 0 , 2 1 ] and s ∈ ( 0 , 1 K − 1 ] s\in(0,\frac{1}{K-1}] s ∈ ( 0 , K − 1 1 ] . Define a score vector v ( s ) ∈ R K v^{(s)}\in\mathbb{R}^K v ( s ) ∈ R K by v k ( s ) : = − s v_k^{(s)}:=-s v k ( s ) := − s for k ∈ [ K − 1 ] k\in[K-1] k ∈ [ K − 1 ] and v K ( s ) : = ( K − 1 ) s v_K^{(s)}:=(K-1)s v K ( s ) := ( K − 1 ) s , satisfying ∑ k = 1 K v k ( s ) = 0 \sum_{k=1}^{K}v_k^{(s)}=0 ∑ k = 1 K v k ( s ) = 0 . Then, define π ~ = ( f ~ 1 : T , d ~ 1 : T ) \widetilde{\pi}=(\widetilde{f}_{1:T},\widetilde{d}_{1:T}) π = ( f 1 : T , d 1 : T ) as
f ~ t = v ∗ for t ≤ T − 1 , and f ~ T = v ( s ) , (140) \widetilde{f}_t=v^* \quad\text{for }t\leq T-1,\quad\text{and}\quad\widetilde{f}_T=v^{(s)}, \tag*{(140)} f t = v ∗ for t ≤ T − 1 , and f T = v ( s ) , (140)
and
d ~ t = 1 2 + ϵ for t ≤ T − 1 , and d ~ T = 0. (141) \widetilde{d}_t=\frac{1}{2}+\epsilon\quad\text{for }t\leq T-1,\quad\text{and}\quad\widetilde{d}_T=0. \tag*{(141)} d t = 2 1 + ϵ for t ≤ T − 1 , and d T = 0. (141)
This constructed π ~ \widetilde{\pi} π belongs to Π ~ c a s c \widetilde{\Pi}_{\mathrm{casc}} Π casc by definition of Π ~ c a s c \widetilde{\Pi}_{\mathrm{casc}} Π casc . The probability of exiting at stage T T T is thus
r ~ T ≡ ( 1 2 + ϵ ) T − 1 . (142) \widetilde{r}_T\equiv\left(\frac{1}{2}+\epsilon\right)^{T-1}. \tag*{(142)} r T ≡ ( 2 1 + ϵ ) T − 1 . (142)
and, by the definition of L ( v ) L(v) L ( v ) in (131) , we have
L ( v ( s ) ) = ∑ k = 1 K − 1 [ v k ( s ) + 1 K − 1 ] ⋅ K − 2 K − 1 + [ v K ( s ) + 1 K − 1 ] + = ∑ k = 1 K − 1 [ − s + 1 K − 1 ] ⋅ K − 2 K − 1 + [ ( K − 1 ) s + 1 K − 1 ] + = ( i ) ∑ k = 1 K − 1 [ − s + 1 K − 1 ] ⋅ K − 2 K − 1 + [ ( K − 1 ) s + 1 K − 1 ] = K − 2 K − 1 − s ( K − 2 ) + s ( K − 1 ) + 1 K − 1 = s + 1 , (143) \begin{aligned}
L(v^{(s)}) &= \sum_{k=1}^{K-1}\left[v_k^{(s)}+\frac{1}{K-1}\right]\cdot\frac{K-2}{K-1}+\left[v_K^{(s)}+\frac{1}{K-1}\right]+\\
&= \sum_{k=1}^{K-1}\left[-s+\frac{1}{K-1}\right]\cdot\frac{K-2}{K-1}+\left[(K-1)s+\frac{1}{K-1}\right]+\\
&\overset{(i)}{=} \sum_{k=1}^{K-1}\left[-s+\frac{1}{K-1}\right]\cdot\frac{K-2}{K-1}+\left[(K-1)s+\frac{1}{K-1}\right]\\
&=\frac{K-2}{K-1}-s(K-2)+s(K-1)+\frac{1}{K-1}=s+1, \tag*{(143)}
\end{aligned} L ( v ( s ) ) = k = 1 ∑ K − 1 [ v k ( s ) + K − 1 1 ] ⋅ K − 1 K − 2 + [ v K ( s ) + K − 1 1 ] + = k = 1 ∑ K − 1 [ − s + K − 1 1 ] ⋅ K − 1 K − 2 + [ ( K − 1 ) s + K − 1 1 ] + = ( i ) k = 1 ∑ K − 1 [ − s + K − 1 1 ] ⋅ K − 1 K − 2 + [ ( K − 1 ) s + K − 1 1 ] = K − 1 K − 2 − s ( K − 2 ) + s ( K − 1 ) + K − 1 1 = s + 1 , (143)
where step (i) holds because s ∈ ( 0 , 1 K − 1 ] s \in\left(0,\frac{1}{K-1}\right] s ∈ ( 0 , K − 1 1 ] by construction.
Step 4. Compute the risks R ( φ ( π ~ ) ) \mathcal{R}(\varphi(\tilde{\pi})) R ( φ ( π ~ )) and R c a s c h i n g e ( π ~ ) \mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi}) R casc hinge ( π ~ ) : We are now ready to derive the cascade hinge risk of the constructed π ~ \tilde{\pi} π ~ and the 0-1-c risk of its hard prediction φ ( π ~ ) \varphi(\tilde{\pi}) φ ( π ~ ) . First, together with the expression for R c a s c h i n g e ( π ~ ) \mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi}) R casc hinge ( π ~ ) in (137) , we can write
R c a s c h i n g e ( π ~ ) = ∑ t = 1 T E [ r ~ t ( H t ) L ( f ~ t ( X , Z t ) ) ] = ∑ t = 1 T − 1 E [ r ~ t ( H t ) L ( v ∗ ) ] + E [ r ~ T ( H T ) L ( v ( s ) ) ] = ( i ) ( 1 − E [ r ~ T ( H T ) ] ) L ∗ + E [ r ~ T ( H T ) ] ( s + 1 ) = L ∗ + E [ r ~ T ( H T ) ] ( s + 1 − L ∗ ) = ( i i ) L ∗ + ( 1 2 + ϵ ) T − 1 ( s + 1 − L ∗ ) , (144) \begin{aligned}
\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi})
&=\sum_{t=1}^{T}\mathbb{E}\left[\tilde{r}_t(H_t)L(\tilde{f}_t(X,Z_t))\right]\\
&=\sum_{t=1}^{T-1}\mathbb{E}\left[\tilde{r}_t(H_t)L(v^*)\right]+\mathbb{E}\left[\tilde{r}_T(H_T)L(v^{(s)})\right]\\
&\overset{(i)}{=}\left(1-\mathbb{E}\left[\tilde{r}_T(H_T)\right]\right)L^*+\mathbb{E}\left[\tilde{r}_T(H_T)\right](s+1)\\
&=L^*+\mathbb{E}\left[\tilde{r}_T(H_T)\right](s+1-L^*)\overset{(ii)}{=}L^*+\left(\frac{1}{2}+\epsilon\right)^{T-1}(s+1-L^*), \tag*{(144)}
\end{aligned} R casc hinge ( π ~ ) = t = 1 ∑ T E [ r ~ t ( H t ) L ( f ~ t ( X , Z t )) ] = t = 1 ∑ T − 1 E [ r ~ t ( H t ) L ( v ∗ ) ] + E [ r ~ T ( H T ) L ( v ( s ) ) ] = ( i ) ( 1 − E [ r ~ T ( H T ) ] ) L ∗ + E [ r ~ T ( H T ) ] ( s + 1 ) = L ∗ + E [ r ~ T ( H T ) ] ( s + 1 − L ∗ ) = ( ii ) L ∗ + ( 2 1 + ϵ ) T − 1 ( s + 1 − L ∗ ) , (144)
where step (i) uses L ( v ∗ ) = L ∗ L(v^*)=L^* L ( v ∗ ) = L ∗ from (133) and L ( v ( s ) ) = s + 1 L(v^{(s)})=s+1 L ( v ( s ) ) = s + 1 from (143) , and step (ii) applies (142) .
Second, under the soft-to-hard map φ \varphi φ , as defined in (12) and (14) , the hard classifiers are f t ≡ 1 f_t\equiv1 f t ≡ 1 for t ≤ T − 1 t\leq T-1 t ≤ T − 1 and f T ≡ K f_T\equiv K f T ≡ K and, induced by the deferral rules, the hard routing rule is r ( h ) = T r(h)=T r ( h ) = T for all h ∈ H h\in\mathcal{H} h ∈ H . This implies that
R ( φ ( π ~ ) ) = Pr ( f r ( H ) ( X , Z r ( H ) ) ≠ Y ) = Pr ( f T ( X , Z T ) ≠ Y ) = Pr ( Y ≠ K ) = 1 , (145) \mathcal{R}(\varphi(\tilde{\pi}))=\Pr\left(f_{r(H)}(X,Z_{r(H)})\neq Y\right)=\Pr\left(f_T(X,Z_T)\neq Y\right)=\Pr(Y\neq K)=1, \tag*{(145)} R ( φ ( π ~ )) = Pr ( f r ( H ) ( X , Z r ( H ) ) = Y ) = Pr ( f T ( X , Z T ) = Y ) = Pr ( Y = K ) = 1 , (145)
where the last step uses (129) .
Step 5. Compute excess risks and prove lower bound: The final step computes and links the excess risks. Combining (136) and (145) gives the excess 0-1-c risk:
R ( φ ( π ~ ) ) − R c a s c ∗ = 1 − K − 2 K − 1 = 1 K − 1 . (146) \mathcal{R}(\varphi(\tilde{\pi}))-\mathcal{R}_{\mathrm{casc}}^*=1-\frac{K-2}{K-1}=\frac{1}{K-1}. \tag*{(146)} R ( φ ( π ~ )) − R casc ∗ = 1 − K − 1 K − 2 = K − 1 1 . (146)
Combining (139) and (144) gives the excess cascade hinge risk:
R c a s c h i n g e ( π ~ ) − R c a s c h i n g e ∗ = L ∗ + ( 1 2 + ϵ ) T − 1 ( s + 1 − L ∗ ) − L ∗ = ( 1 2 + ϵ ) T − 1 ( s + 1 − L ∗ ) = ( 1 2 + ϵ ) T − 1 [ s + 1 − K ( K − 2 ) ( K − 1 ) 2 ] = ( 1 2 + ϵ ) T − 1 [ s + 1 ( K − 1 ) 2 ] , (147) \begin{aligned}
\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi})-\mathcal{R}^{\mathrm{hinge}*}_{\mathrm{casc}}
&=L^*+\left(\frac{1}{2}+\epsilon\right)^{T-1}(s+1-L^*)-L^*\\
&=\left(\frac{1}{2}+\epsilon\right)^{T-1}(s+1-L^*)\\
&=\left(\frac{1}{2}+\epsilon\right)^{T-1}\left[s+1-\frac{K(K-2)}{(K-1)^2}\right]\\
&=\left(\frac{1}{2}+\epsilon\right)^{T-1}\left[s+\frac{1}{(K-1)^2}\right], \tag*{(147)}
\end{aligned} R casc hinge ( π ~ ) − R casc hinge ∗ = L ∗ + ( 2 1 + ϵ ) T − 1 ( s + 1 − L ∗ ) − L ∗ = ( 2 1 + ϵ ) T − 1 ( s + 1 − L ∗ ) = ( 2 1 + ϵ ) T − 1 [ s + 1 − ( K − 1 ) 2 K ( K − 2 ) ] = ( 2 1 + ϵ ) T − 1 [ s + ( K − 1 ) 2 1 ] , (147)
where the last step applies (132) . Finally, taking ratio of (146) and (147) gives
R ( φ ( π ~ ) ) − R c a s c ∗ R c a s c h i n g e ( π ~ ) − R c a s c h i n g e ∗ = 1 K − 1 ( 1 2 + ϵ ) T − 1 [ s + 1 ( K − 1 ) 2 ] = 2 T − 1 ( K − 1 ) ( 1 + 2 ϵ ) T − 1 [ s ( K − 1 ) 2 + 1 ] = : R . (148) \frac{\mathcal{R}(\varphi(\tilde{\pi}))-\mathcal{R}_{\mathrm{casc}}^*}{\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi})-\mathcal{R}^{\mathrm{hinge}*}_{\mathrm{casc}}}
=\frac{\frac{1}{K-1}}{\left(\frac{1}{2}+\epsilon\right)^{T-1}\left[s+\frac{1}{(K-1)^2}\right]}
=\frac{2^{T-1}(K-1)}{(1+2\epsilon)^{T-1}\left[s(K-1)^2+1\right]}=:R. \tag*{(148)} R casc hinge ( π ~ ) − R casc hinge ∗ R ( φ ( π ~ )) − R casc ∗ = ( 2 1 + ϵ ) T − 1 [ s + ( K − 1 ) 2 1 ] K − 1 1 = ( 1 + 2 ϵ ) T − 1 [ s ( K − 1 ) 2 + 1 ] 2 T − 1 ( K − 1 ) =: R . (148)
As ϵ , s → 0 \epsilon, s \to0 ϵ , s → 0 , the denominator tends to 1, and so R → 2 T − 1 ( K − 1 ) R \to2^{T-1}(K-1) R → 2 T − 1 ( K − 1 ) , implying that 2 T − 1 ( K − 1 ) 2^{T-1}(K-1) 2 T − 1 ( K − 1 ) is the constant in the lower bound. More concretely, for every C < 2 T − 1 ( K − 1 ) C < 2^{T-1}(K-1) C < 2 T − 1 ( K − 1 ) , we can choose ϵ \epsilon ϵ and s s s small enough that R > C R > C R > C . For example, setting
ϵ = 0 and s = min { 1 K − 1 , 2 T − 1 ( K − 1 ) / C − 1 2 ( K − 1 ) 2 } > 0 , \epsilon= 0 \quad\text{and}\quad s = \min\left\{\frac{1}{K-1},\frac{2^{T-1}(K-1)/C-1}{2(K-1)^2}\right\} > 0, ϵ = 0 and s = min { K − 1 1 , 2 ( K − 1 ) 2 2 T − 1 ( K − 1 ) / C − 1 } > 0 ,
and using the shorthand r : = 2 T − 1 ( K − 1 ) / C > 1 r := 2^{T-1}(K-1)/C > 1 r := 2 T − 1 ( K − 1 ) / C > 1 gives
s ( K − 1 ) 2 + 1 ≤ r − 1 2 + 1 = r + 1 2 < r + r 2 = r . s(K-1)^2+1 \le\frac{r-1}{2}+1 = \frac{r+1}{2} < \frac{r+r}{2}=r. s ( K − 1 ) 2 + 1 ≤ 2 r − 1 + 1 = 2 r + 1 < 2 r + r = r .
This implies that
R = 2 T − 1 ( K − 1 ) 1 ⋅ [ s ( K − 1 ) 2 + 1 ] > 2 T − 1 ( K − 1 ) r = C . (149) R=\frac{2^{T-1}(K-1)}{1\cdot\left[s(K-1)^2+1\right]}>\frac{2^{T-1}(K-1)}{r}=C. \tag*{(149)} R = 1 ⋅ [ s ( K − 1 ) 2 + 1 ] 2 T − 1 ( K − 1 ) > r 2 T − 1 ( K − 1 ) = C . (149)
This proves the proposition.
An Upper Bound in Favorable Instances
Proposition C.8 shows that the constant in Theorem 3.2 cannot be improved in bad problem instances, where every soft deferral decision is maximally indecisive, that is, d ~ t ≡ 0.5 \tilde d_t \equiv0.5 d ~ t ≡ 0.5 . We now show that the constant in Theorem 3.2 is much smaller in more favorable problem instances, where the soft deferral decisions stay away from 0.5. To begin, we prove a refined, margin-dependent version of Lemma C.7 to connect soft and hard deferral decisions.
Lemma C.9 (Margin-dependent inequality linking soft and hard deferral decisions). For d ~ ∈ [ 0 , 1 ] \tilde d \in[0,1] d ~ ∈ [ 0 , 1 ] , let d : = 1 { d ~ ≥ 0.5 } d := \mathbb{1}\{\tilde d \ge0.5\} d := 1 { d ~ ≥ 0.5 } and m : = ∣ d ~ − 0.5 ∣ m := |\tilde d-0.5| m := ∣ d ~ − 0.5∣ . For all R , V ˉ ∈ R \mathcal{R}, \bar V \in\mathbb{R} R , V ˉ ∈ R ,
( 1 − d ) R + d V ˉ − min ( R , V ˉ ) ≤ 2 1 + 2 m [ ( 1 − d ~ ) R + d ~ V ˉ − min ( R , V ˉ ) ] , (150) (1-d)\mathcal{R}+d\bar V-\min(\mathcal{R},\bar V)
\le\frac{2}{1+2m}\left[(1-\tilde d)\mathcal{R}+\tilde d\bar V-\min(\mathcal{R},\bar V)\right], \tag*{(150)} ( 1 − d ) R + d V ˉ − min ( R , V ˉ ) ≤ 1 + 2 m 2 [ ( 1 − d ~ ) R + d ~ V ˉ − min ( R , V ˉ ) ] , (150) where 2 1 + 2 m ∈ [ 1 , 2 ] \frac{2}{1+2m}\in[1,2] 1 + 2 m 2 ∈ [ 1 , 2 ] is the smallest constant valid at the given m m m .
Proof. Let [ x ] + : = max ( x , 0 ) [x]_+ := \max(x,0) [ x ] + := max ( x , 0 ) . Using m : = ∣ d ~ − 0.5 ∣ m := |\tilde d-0.5| m := ∣ d ~ − 0.5∣ , we can define and write
q : = max ( d ~ , 1 − d ~ ) = 1 2 + m , 1 q = 2 1 + 2 m . (151) q:=\max(\tilde d,1-\tilde d)=\frac{1}{2}+m,\qquad\frac{1}{q}=\frac{2}{1+2m}. \tag*{(151)} q := max ( d ~ , 1 − d ~ ) = 2 1 + m , q 1 = 1 + 2 m 2 . (151)
For brevity, we use shorthand H : = ( 1 − d ) R + d V ˉ − min ( R , V ˉ ) H := (1-d)\mathcal{R}+d\bar V-\min(\mathcal{R},\bar V) H := ( 1 − d ) R + d V ˉ − min ( R , V ˉ ) and S : = ( 1 − d ~ ) R + d ~ V ˉ − min ( R , V ˉ ) S := (1-\tilde d)\mathcal{R}+\tilde d\bar V-\min(\mathcal{R},\bar V) S := ( 1 − d ~ ) R + d ~ V ˉ − min ( R , V ˉ ) , so the desired bound in (150) can be equivalently written as
H ≤ 1 q S , (152) H\le\frac{1}{q}S, \tag*{(152)} H ≤ q 1 S , (152)
which we prove below. To establish tightness of (151), it therefore suffices to construct a case in which (153) holds with equality.
Step 1: We first rewrite H H H and S S S as follows, replacing the hard-to-handle min ( R , V ˉ ) \min(\mathcal{R},\bar V) min ( R , V ˉ ) terms with terms involving [ ⋅ ] + [\cdot]_+ [ ⋅ ] + . To rewrite H H H , we consider two cases. If R ≤ V ˉ \mathcal{R}\le\bar V R ≤ V ˉ , then min ( R , V ˉ ) = R \min(\mathcal{R},\bar V)=\mathcal{R} min ( R , V ˉ ) = R , which implies that
( 1 − d ) ( R − V ˉ ) + + d ( V ˉ − R ) + = ( 1 − d ) ⋅ 0 + d ( V ˉ − R ) = ( 1 − d ) R + d V ˉ − min ( R , V ˉ ) = H . (153) \begin{aligned}
(1-d)(\mathcal{R}-\bar V)_+ + d(\bar V-\mathcal{R})_+
= (1-d)\cdot0+d(\bar V-\mathcal{R}) \\
&= (1-d)\mathcal{R}+d\bar V-\min(\mathcal{R},\bar V)=H. \tag*{(153)}
\end{aligned} ( 1 − d ) ( R − V ˉ ) + + d ( V ˉ − R ) + = ( 1 − d ) ⋅ 0 + d ( V ˉ − R ) = ( 1 − d ) R + d V ˉ − min ( R , V ˉ ) = H . (153)
Similarly, if R > V ˉ \mathcal{R} > \bar{V} R > V ˉ , then min ( R , V ˉ ) = V ˉ \min(\mathcal{R},\bar{V}) = \bar{V} min ( R , V ˉ ) = V ˉ , which implies that
( 1 − d ) ( R − V ˉ ) + + d ( V ˉ − R ) + = ( 1 − d ) ( R − V ˉ ) + d ⋅ 0 = ( 1 − d ) R + d V ˉ − min ( R , V ˉ ) = H . (154) (1-d)(\mathcal{R}-\bar{V})_{+}+d(\bar{V}-\mathcal{R})_{+}=(1-d)(\mathcal{R}-\bar{V})+d\cdot0
=(1-d)\mathcal{R}+d\bar{V}-\min(\mathcal{R},\bar{V})=H . \tag*{(154)} ( 1 − d ) ( R − V ˉ ) + + d ( V ˉ − R ) + = ( 1 − d ) ( R − V ˉ ) + d ⋅ 0 = ( 1 − d ) R + d V ˉ − min ( R , V ˉ ) = H . (154)
Together, we have shown
( 1 − d ) ( R − V ˉ ) + + d ( V ˉ − R ) + = H . (155) (1-d)(\mathcal{R}-\bar{V})_{+}+d(\bar{V}-\mathcal{R})_{+}=H . \tag*{(155)} ( 1 − d ) ( R − V ˉ ) + + d ( V ˉ − R ) + = H . (155)
Applying the same argument to S S S obtains
( 1 − d ~ ) ( R − V ˉ ) + + d ~ ( V ˉ − R ) + = S . (156) (1-\tilde{d})(\mathcal{R}-\bar{V})_{+}+\tilde{d}(\bar{V}-\mathcal{R})_{+}=S . \tag*{(156)} ( 1 − d ~ ) ( R − V ˉ ) + + d ~ ( V ˉ − R ) + = S . (156)
Step 2: Using (156) and (157), we can now prove (153). Again, consider two cases: If d ~ < 0.5 \tilde{d}<0.5 d ~ < 0.5 , then d = 0 d=0 d = 0 and q = max ( d ~ , 1 − d ~ ) = 1 − d ~ q=\max(\tilde{d},1-\tilde{d})=1-\tilde{d} q = max ( d ~ , 1 − d ~ ) = 1 − d ~ . Hence, substituting this into (157) and using the fact that [ ⋅ ] + ≥ 0 [\cdot]_{+}\ge0 [ ⋅ ] + ≥ 0 gives
S = ( 1 − d ~ ) ( R − V ˉ ) + + d ~ ( V ˉ − R ) + = q ( R − V ˉ ) + + d ~ ( V ˉ − R ) + ≥ q ( R − V ˉ ) + = q ( ( 1 − d ) ( R − V ˉ ) + + d ( V ˉ − R ) + ) = q H , (157) \begin{aligned}
S&=(1-\tilde{d})(\mathcal{R}-\bar{V})_{+}+\tilde{d}(\bar{V}-\mathcal{R})_{+}=q(\mathcal{R}-\bar{V})_{+}+\tilde{d}(\bar{V}-\mathcal{R})_{+}\\
&\ge q(\mathcal{R}-\bar{V})_{+}\\
&=q\left((1-d)(\mathcal{R}-\bar{V})_{+}+d(\bar{V}-\mathcal{R})_{+}\right)=qH , \tag*{(157)}
\end{aligned} S = ( 1 − d ~ ) ( R − V ˉ ) + + d ~ ( V ˉ − R ) + = q ( R − V ˉ ) + + d ~ ( V ˉ − R ) + ≥ q ( R − V ˉ ) + = q ( ( 1 − d ) ( R − V ˉ ) + + d ( V ˉ − R ) + ) = q H , (157)
where the last step applies (156). When R > V ˉ \mathcal{R}>\bar{V} R > V ˉ , (158) holds with equality.
Analogously, if d ~ ≥ 0.5 \tilde{d}\ge0.5 d ~ ≥ 0.5 , then d = 1 d=1 d = 1 and q = d ~ q=\tilde{d} q = d ~ , so substituting this into (157) and using [ ⋅ ] + ≥ 0 [\cdot]_{+}\ge0 [ ⋅ ] + ≥ 0 gives
S = ( 1 − d ~ ) ( R − V ˉ ) + + d ~ ( V ˉ − R ) + = ( 1 − d ~ ) ( R − V ˉ ) + + q ( V ˉ − R ) + ≥ q ( V ˉ − R ) + = q ( d ( V ˉ − R ) + + ( 1 − d ) ( R − V ˉ ) + ) = q H , (158) \begin{aligned}
S&=(1-\tilde{d})(\mathcal{R}-\bar{V})_{+}+\tilde{d}(\bar{V}-\mathcal{R})_{+}=(1-\tilde{d})(\mathcal{R}-\bar{V})_{+}+q(\bar{V}-\mathcal{R})_{+}\\
&\ge q(\bar{V}-\mathcal{R})_{+}\\
&=q\left(d(\bar{V}-\mathcal{R})_{+}+(1-d)(\mathcal{R}-\bar{V})_{+}\right)=qH , \tag*{(158)}
\end{aligned} S = ( 1 − d ~ ) ( R − V ˉ ) + + d ~ ( V ˉ − R ) + = ( 1 − d ~ ) ( R − V ˉ ) + + q ( V ˉ − R ) + ≥ q ( V ˉ − R ) + = q ( d ( V ˉ − R ) + + ( 1 − d ) ( R − V ˉ ) + ) = q H , (158)
where the last step applies (156). When R < V ˉ \mathcal{R}<\bar{V} R < V ˉ , (159) holds with equality.
Taken together, (158) and (159) establish (153) and show that the inequality is tight. This concludes the proof.
Note that setting m = 0 m=0 m = 0 in (151) recovers (105) in Lemma C.7.
Proposition C.10 (Calibration for decisive soft deferral rules). Assume Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) Y\perp Z_{1:t-1}\mid(X,Z_t) Y ⊥ Z 1 : t − 1 ∣ ( X , Z t ) as in (5), and let R c a s c ∗ \mathcal{R}^{*}_{\mathrm{casc}} R casc ∗ and R c a s c h i n g e ∗ \mathcal{R}^{\mathrm{hinge}*}_{\mathrm{casc}} R casc hinge ∗ be as in Theorem 3.2. Let m ∈ [ 0 , 0.5 ] m\in[0,0.5] m ∈ [ 0 , 0.5 ] . Then, for any π ~ ∈ Π c a s c e \tilde{\pi}\in\Pi_{\mathrm{casc}}^{e} π ~ ∈ Π casc e whose deferral rules satisfy ∣ d ~ t ( H t ) − 0.5 ∣ ≥ m |\tilde{d}_t(H_t)-0.5|\ge m ∣ d ~ t ( H t ) − 0.5∣ ≥ m almost surely for all t ∈ [ T − 1 ] t\in[T-1] t ∈ [ T − 1 ] ,
R ( φ ( π ~ ) ) − R c a s c ∗ ≤ ( 2 1 + 2 m ) T − 1 ( K − 1 ) ( R c a s c h i n g e ( π ~ ) − R c a s c h i n g e ∗ ) . (159) \mathcal{R}(\varphi(\tilde{\pi}))-\mathcal{R}^{*}_{\mathrm{casc}}
\le\left(\frac{2}{1+2m}\right)^{T-1}(K-1)\left(\mathcal{R}^{\mathrm{hinge}}_{\mathrm{casc}}(\tilde{\pi})-\mathcal{R}^{\mathrm{hinge}*}_{\mathrm{casc}}\right). \tag*{(159)} R ( φ ( π ~ )) − R casc ∗ ≤ ( 1 + 2 m 2 ) T − 1 ( K − 1 ) ( R casc hinge ( π ~ ) − R casc hinge ∗ ) . (159) Proof. Let ρ : = 2 1 + 2 m ∈ [ 1 , 2 ] \rho:=\frac{2}{1+2m}\in[1,2] ρ := 1 + 2 m 2 ∈ [ 1 , 2 ] . The proof follows that of Theorem 3.2 in Appendix C.4, with the constants C t + 1 C_{t+1} C t + 1 and C t C_t C t in (111) and (113) changed into C t + 1 = ρ T − t − 1 ( K − 1 ) C_{t+1}=\rho^{T-t-1}(K-1) C t + 1 = ρ T − t − 1 ( K − 1 ) and C t = ρ C t + 1 C_t=\rho C_{t+1} C t = ρ C t + 1 , and one subsequent change in Step 2.2 . The rest of the proof stays unchanged.
In Step 2.2 , we apply Lemma C.9 in place of Lemma C.7. By Lemma C.9, the factor of 2 in (120) becomes 2 / ( 1 + 2 ∣ d ~ t ( h t ) − 0.5 ∣ ) 2/(1+2|\tilde{d}_t(h_t)-0.5|) 2/ ( 1 + 2∣ d ~ t ( h t ) − 0.5∣ ) , giving the following counterpart of (120):
Δ t ( h t ) ≤ 2 1 + 2 ∣ d ~ t ( h t ) − 0.5 ∣ ( ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] − m ) ⏟ = : S t ( h t ) + ( m − V t π c a s c ∗ ( h t ) ) , (160) \Delta_t(h_t)\le
\frac{2}{1+2|\tilde{d}_t(h_t)-0.5|}
\underbrace{\left((1-\tilde{d}_t(h_t))\mathcal{R}_t(h_t;f_t)+\tilde{d}_t(h_t)\mathbb{E}\left[V^\pi_{t+1}(H_{t+1})\mid H_t=h_t\right]-m\right)}_{=:S_t(h_t)}
+\left(m-V^{\pi^*_{\mathrm{casc}}}_t(h_t)\right), \tag*{(160)} Δ t ( h t ) ≤ 1 + 2∣ d ~ t ( h t ) − 0.5∣ 2 =: S t ( h t ) ( ( 1 − d ~ t ( h t )) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ( H t + 1 ) ∣ H t = h t ] − m ) + ( m − V t π casc ∗ ( h t ) ) , (160)
where recall from (114) that m : = min { R t ( h t ; f t ) , E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] } m := \min\left\{\mathcal{R}_t(h_t; f_t), \mathbb{E}\left[V_{t+1}^{\pi^\star}(H_{t+1}) \mid H_t = h_t\right]\right\} m := min { R t ( h t ; f t ) , E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] } and from (115) and (117) that Δ t , i n t e r n a l ( h t ) = m − V t π c a s c ⋆ ( h t ) ≥ 0 \Delta_{t,\mathrm{internal}}(h_t) = m - V_t^{\pi^\star_{\mathrm{casc}}}(h_t) \ge0 Δ t , internal ( h t ) = m − V t π casc ⋆ ( h t ) ≥ 0 . Note that S t ( h t ) ≥ 0 S_t(h_t) \ge0 S t ( h t ) ≥ 0 since min { a , b } ≤ ( 1 − λ ) a + λ b \min\{a,b\} \le(1-\lambda)a+\lambda b min { a , b } ≤ ( 1 − λ ) a + λb for every λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] . Thus, together with 2 / ( 1 + 2 ∣ d ~ t ( h t ) − 0.5 ∣ ) ≤ ρ 2/(1+2|\widetilde{d}_t(h_t)-0.5|) \le\rho 2/ ( 1 + 2∣ d t ( h t ) − 0.5∣ ) ≤ ρ for almost every h t h_t h t by assumption, we can write
Δ t ( h t ) ≤ ρ ( ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] − m ) + ( m − V t π c a s c ⋆ ( h t ) ) = ρ ( ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] ) − ( ρ − 1 ) m − V t π c a s c ⋆ ( h t ) = ρ ( ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] ) − ( ρ − 1 ) ( Δ t , i n t e r n a l ( h t ) + V t π c a s c ⋆ ( h t ) ) − V t π c a s c ⋆ ( h t ) = ρ ( ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] ) − ( ρ − 1 ) Δ t , i n t e r n a l ( h t ) − ρ V t π c a s c ⋆ ( h t ) ≤ ( i ) ρ ( ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] ) − ρ V t π c a s c ⋆ ( h t ) = ρ ( ( 1 − d ~ t ( h t ) ) R t ( h t ; f t ) + d ~ t ( h t ) E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] − V t π c a s c ⋆ ( h t ) ) , (161) \begin{aligned}
\Delta_t(h_t) &\le\rho\left((1-\widetilde{d}_t(h_t))\mathcal{R}_t(h_t;f_t)+\widetilde{d}_t(h_t)\mathbb{E}\left[V_{t+1}^{\pi^\star}(H_{t+1})\mid H_t=h_t\right]-m\right)+\left(m-V_t^{\pi^\star_{\mathrm{casc}}}(h_t)\right)\\
&=\rho\left((1-\widetilde{d}_t(h_t))\mathcal{R}_t(h_t;f_t)+\widetilde{d}_t(h_t)\mathbb{E}\left[V_{t+1}^{\pi^\star}(H_{t+1})\mid H_t=h_t\right]\right)-(\rho-1)m-V_t^{\pi^\star_{\mathrm{casc}}}(h_t)\\
&=\rho\left((1-\widetilde{d}_t(h_t))\mathcal{R}_t(h_t;f_t)+\widetilde{d}_t(h_t)\mathbb{E}\left[V_{t+1}^{\pi^\star}(H_{t+1})\mid H_t=h_t\right]\right)\\
&\quad-(\rho-1)\left(\Delta_{t,\mathrm{internal}}(h_t)+V_t^{\pi^\star_{\mathrm{casc}}}(h_t)\right)-V_t^{\pi^\star_{\mathrm{casc}}}(h_t)\\
&=\rho\left((1-\widetilde{d}_t(h_t))\mathcal{R}_t(h_t;f_t)+\widetilde{d}_t(h_t)\mathbb{E}\left[V_{t+1}^{\pi^\star}(H_{t+1})\mid H_t=h_t\right]\right)-(\rho-1)\Delta_{t,\mathrm{internal}}(h_t)\\
&\quad-\rho V_t^{\pi^\star_{\mathrm{casc}}}(h_t)\\
&\mathrel{\overset{\mathrm{(i)}}{\le}}\rho\left((1-\widetilde{d}_t(h_t))\mathcal{R}_t(h_t;f_t)+\widetilde{d}_t(h_t)\mathbb{E}\left[V_{t+1}^{\pi^\star}(H_{t+1})\mid H_t=h_t\right]\right)-\rho V_t^{\pi^\star_{\mathrm{casc}}}(h_t)\\
&=\rho\left((1-\widetilde{d}_t(h_t))\mathcal{R}_t(h_t;f_t)+\widetilde{d}_t(h_t)\mathbb{E}\left[V_{t+1}^{\pi^\star}(H_{t+1})\mid H_t=h_t\right]-V_t^{\pi^\star_{\mathrm{casc}}}(h_t)\right), \tag*{(161)}
\end{aligned} Δ t ( h t ) ≤ ρ ( ( 1 − d t ( h t )) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] − m ) + ( m − V t π casc ⋆ ( h t ) ) = ρ ( ( 1 − d t ( h t )) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] ) − ( ρ − 1 ) m − V t π casc ⋆ ( h t ) = ρ ( ( 1 − d t ( h t )) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] ) − ( ρ − 1 ) ( Δ t , internal ( h t ) + V t π casc ⋆ ( h t ) ) − V t π casc ⋆ ( h t ) = ρ ( ( 1 − d t ( h t )) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] ) − ( ρ − 1 ) Δ t , internal ( h t ) − ρ V t π casc ⋆ ( h t ) ≤ ( i ) ρ ( ( 1 − d t ( h t )) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] ) − ρ V t π casc ⋆ ( h t ) = ρ ( ( 1 − d t ( h t )) R t ( h t ; f t ) + d t ( h t ) E [ V t + 1 π ⋆ ( H t + 1 ) ∣ H t = h t ] − V t π casc ⋆ ( h t ) ) , (161)
where the inequality (i) follows from ρ − 1 ≥ 0 \rho-1\ge0 ρ − 1 ≥ 0 and Δ t , i n t e r n a l ( h t ) ≥ 0 \Delta_{t,\mathrm{internal}}(h_t)\ge0 Δ t , internal ( h t ) ≥ 0 . The RHS of (161) matches the RHS of (120) with ρ \rho ρ in place of 2. Step 2.3 and Step 2.4 then go through with ρ \rho ρ in place of 2. The domination argument in Step 2.4 still holds: since ρ ≥ 1 \rho\ge1 ρ ≥ 1 implies ρ T − t − 1 ≥ 1 \rho^{T-t-1}\ge1 ρ T − t − 1 ≥ 1 for t ≤ T − 1 t\le T-1 t ≤ T − 1 , the constants K − 1 K-1 K − 1 , C t + 1 = ρ T − t − 1 ( K − 1 ) C_{t+1}=\rho^{T-t-1}(K-1) C t + 1 = ρ T − t − 1 ( K − 1 ) and K − 1 K \frac{K-1}{K} K K − 1 are all dominated by C t + 1 C_{t+1} C t + 1 . This completes the backward induction in Step 2 with stage-wise calibration constants C t + 1 = ρ T − t − 1 ( K − 1 ) C_{t+1}=\rho^{T-t-1}(K-1) C t + 1 = ρ T − t − 1 ( K − 1 ) and C t = ρ C t + 1 C_t=\rho C_{t+1} C t = ρ C t + 1 .
The rest of the proof follows, yielding C 1 = ρ T − 1 ( K − 1 ) C_1=\rho^{T-1}(K-1) C 1 = ρ T − 1 ( K − 1 ) which proves (159) .
Synthetic Experiments
Details for the Synthetic Experiments
The class-probability functions θ ( x , z t ) \boldsymbol{\theta}(x, z_t) θ ( x , z t ) are defined piecewise over the input space X \mathcal{X} X . The goal is to induce regions where deferring to later classifiers improves accuracy enough to offset the routing cost. The parameters of the posterior distribution P Y ∣ X , Z 2 , Z 3 = Categorical ( θ ( x , z 2 , z 3 ) ) P_{Y\mid X,Z_2,Z_3} = \operatorname{Categorical}(\boldsymbol{\theta}(x,z_2,z_3)) P Y ∣ X , Z 2 , Z 3 = Categorical ( θ ( x , z 2 , z 3 )) are given by;
θ y ( x , z 2 , z 3 ) = exp ( ℓ y ( x , z 2 , z 3 ) ) ∑ k = 1 3 exp ( ℓ k ( x , z 2 , z 3 ) ) (162) \theta_y(x,z_2,z_3) = \frac{\exp(\ell_y(x,z_2,z_3))}{\sum_{k=1}^{3}\exp(\ell_k(x,z_2,z_3))} \tag*{(162)} θ y ( x , z 2 , z 3 ) = ∑ k = 1 3 exp ( ℓ k ( x , z 2 , z 3 )) exp ( ℓ y ( x , z 2 , z 3 )) (162)
ℓ 1 ( x , z 2 , z 3 ) = 1.2 − 4 ∣ x ∣ , (163) \ell_1(x,z_2,z_3) = 1.2 - 4|x|, \tag*{(163)} ℓ 1 ( x , z 2 , z 3 ) = 1.2 − 4∣ x ∣ , (163)
ℓ 2 ( x , z 2 , z 3 ) = − 0.2 + 2.2 x + 3 g ( x ) z 2 , (164) \ell_2(x,z_2,z_3) = -0.2 + 2.2x + 3g(x)z_2, \tag*{(164)} ℓ 2 ( x , z 2 , z 3 ) = − 0.2 + 2.2 x + 3 g ( x ) z 2 , (164)
ℓ 3 ( x , z 2 , z 3 ) = − 0.2 − 2.2 x + 3 g ( x ) z 3 , (165) \ell_3(x,z_2,z_3) = -0.2 - 2.2x + 3g(x)z_3, \tag*{(165)} ℓ 3 ( x , z 2 , z 3 ) = − 0.2 − 2.2 x + 3 g ( x ) z 3 , (165)
g ( x ) = σ ( 10 ( ∣ x ∣ − 0.15 ) ) , (166) g(x) = \sigma(10(|x| - 0.15)), \tag*{(166)} g ( x ) = σ ( 10 ( ∣ x ∣ − 0.15 )) , (166)
σ ( u ) = 1 1 + e − u . (167) \sigma(u) = \frac{1}{1 + e^{-u}}. \tag*{(167)} σ ( u ) = 1 + e − u 1 . (167)
The posterior distributions P Y ∣ X P_{Y\mid X} P Y ∣ X , P Y ∣ X , Z 2 P_{Y\mid X,Z_2} P Y ∣ X , Z 2 and P Y ∣ X , Z 3 P_{Y\mid X,Z_3} P Y ∣ X , Z 3 needed to compute the Bayes-optimal risks are estimated using 5000 Monte Carlo samples.
Model architecture. All models ( f 1 ( x ) , f 2 ( x , z 2 ) , f 3 ( x , z 3 ) , r ( x ) , d 1 ( x ) , d 2 ( x , z 2 ) ) (f_1(x), f_2(x,z_2), f_3(x,z_3), r(x), d_1(x), d_2(x,z_2)) ( f 1 ( x ) , f 2 ( x , z 2 ) , f 3 ( x , z 3 ) , r ( x ) , d 1 ( x ) , d 2 ( x , z 2 )) are multilayer perceptrons with ReLU activations, 3 hidden layers of width 128, and batch normalization. We observe that including batch normalization speeds up training but does not change performance, so its use is optional. Each classifier outputs K − 1 K-1 K − 1 scores, with the final class score determined by the constraint that the scores sum to zero, as required by the multiclass hinge formulation.
Training. We use 100,000 training, 1000 validation, and 2000 test samples. Models are trained using the AdamW optimizer with a learning rate of 10 − 3 10^{-3} 1 0 − 3 , a batch size of 512, and a maximum of 50 epochs, with early stopping based on validation performance with a patience of two epochs.
Empirical Risk Minimization in Synthetic Dataset for Various Costs
We report the empirical risk of the learned models alongside the optimal empirical risk throughout training in Figure 6 , for the synthetic experiment presented in § 5.1 . We present both the cascade and the flat setting, for costs κ 1 = 0 , κ 2 = κ 3 = κ = 0.03 \kappa_1 = 0, \kappa_2 = \kappa_3 = \kappa= 0.03 κ 1 = 0 , κ 2 = κ 3 = κ = 0.03 and κ 1 = 0 , κ 2 = κ 3 = κ = 0.06 \kappa_1 = 0, \kappa_2 = \kappa_3 = \kappa= 0.06 κ 1 = 0 , κ 2 = κ 3 = κ = 0.06 . We observe that training with the surrogate losses in both cases attains the optimal risk on the test set.
Accuracy of Classifiers and Router in Synthetic Dataset
In this experiment, we examine how the accuracy of the router and classifiers evolves during training in the synthetic experiment with flat routers presented in § 5.1 . As shown in Figure 7 , the accuracy of the classifiers quickly rises and plateaus, while the routing accuracy takes longer to converge.
Post-Training PPI Correction
Constructing Confidence Intervals Using Unlabeled Data with PPI
In this section, we detail the CI construction in (23) and discuss the result in (27) . For a fixed x ∈ X x \in\mathcal{X} x ∈ X , the CI for δ t ( x ) \delta_t(x) δ t ( x ) in (23) is centered around δ t P P ( x ) \delta_t^{\mathrm{PP}}(x) δ t PP ( x ) with half-width w t , N b , l b ( x ) w_{t,N_b,l_b}(x) w t , N b , l b ( x ) , where N b N_b N b and l b l_b l b refer to the number of unlabeled and labeled samples w.r.t. the considered bin, respectively. We construct δ t P P ( x ) \delta_t^{\mathrm{PP}}(x) δ t PP ( x ) and w t , N b , l b ( x ) w_{t,N_b,l_b}(x) w t , N b , l b ( x ) as follows. Let
Δ t ( x , z t , z r f l a t ( x ) , y ) = 1 { f r n a t ( x ) † ( x , z r f l a t ( x ) ) ≠ y } − 1 { f t † ( x , z t ) ≠ y } + c r f l a t ( x ) − c t , (168) \Delta_t(x,z_t,z_{r_{\mathrm{flat}}(x)},y)=\mathbb{1}\{f^\dagger_{\mathrm{rnat}(x)}(x,z_{r_{\mathrm{flat}}(x)})\ne y\}-\mathbb{1}\{f^\dagger_t(x,z_t)\ne y\}+c_{r_{\mathrm{flat}}(x)}-c_t, \tag*{(168)} Δ t ( x , z t , z r flat ( x ) , y ) = 1 { f rnat ( x ) † ( x , z r flat ( x ) ) = y } − 1 { f t † ( x , z t ) = y } + c r flat ( x ) − c t , (168)
then we define
δ t P P ( x ) : = 1 N b ∑ i = 1 N b Δ t ( x , z t ( i ) , z r f l a t ( x ) ( i ) , f T † ( x , z T ( i ) ) ) − 1 l b ∑ j = 1 l b [ Δ t ( x , z t ( j ) , z r f l a t ( x ) ( j ) , f T † ( x , z T ( j ) ) ) − Δ t ( x , z t ( j ) , z r f l a t ( x ) ( j ) , y ( j ) ) ] . (169) \begin{aligned}
\delta_t^{\mathrm{PP}}(x) := {}& \frac{1}{N_b}\sum_{i=1}^{N_b}\Delta_t\left(x,z_t^{(i)},z_{r_{\mathrm{flat}}(x)}^{(i)},f_T^\dagger\left(x,z_T^{(i)}\right)\right) \\
&-\frac{1}{l_b}\sum_{j=1}^{l_b}\left[\Delta_t\left(x,z_t^{(j)},z_{r_{\mathrm{flat}}(x)}^{(j)},f_T^\dagger\left(x,z_T^{(j)}\right)\right)-\Delta_t\left(x,z_t^{(j)},z_{r_{\mathrm{flat}}(x)}^{(j)},y^{(j)}\right)\right]. \tag*{(169)}
\end{aligned} δ t PP ( x ) := N b 1 i = 1 ∑ N b Δ t ( x , z t ( i ) , z r flat ( x ) ( i ) , f T † ( x , z T ( i ) ) ) − l b 1 j = 1 ∑ l b [ Δ t ( x , z t ( j ) , z r flat ( x ) ( j ) , f T † ( x , z T ( j ) ) ) − Δ t ( x , z t ( j ) , z r flat ( x ) ( j ) , y ( j ) ) ] . (169)
where the sum over Term 1 is evaluated using unlabeled data, while the sum over Term 2, called the rectifier [1 ] , is evaluated using labeled data. We use the most expensive classifier f T † f_T^\dagger f T † to generate predictions on the unlabeled samples, since by design it is generally the most powerful classifier among f 1 : T f_{1:T} f 1 : T .
The resulting CI has half-width w t , N b , l b ( x ) : = 1.96 σ ^ f 2 / N b + σ ^ f − Y 2 / l b w_{t,N_b,l_b}(x):=1.96\sqrt{\hat{\sigma}_f^2/N_b+\hat{\sigma}_{f-Y}^2/l_b} w t , N b , l b ( x ) := 1.96 σ ^ f 2 / N b + σ ^ f − Y 2 / l b , where σ ^ f 2 \hat{\sigma}_f^2 σ ^ f 2 and σ ^ f − Y 2 \hat{\sigma}_{f-Y}^2 σ ^ f − Y 2 are the empirical variances of the summands of Term 1 and Term 2, respectively. When the imputing classifier f T † f_T^\dagger f T † is accurate, the σ ^ f − Y 2 / l b \hat{\sigma}_{f-Y}^2/l_b σ ^ f − Y 2 / l b term is small. The unlabeled-data term σ ^ f 2 / N b \hat{\sigma}_f^2/N_b σ ^ f 2 / N b decreases as the number of unlabeled samples N b N_b N b grows. In this case, taken together, the CIs are typically narrow and do not straddle zero, yielding the guarantee in (26) which says r r f l a t P P ( x ) r_{r_{\mathrm{flat}}^{\mathrm{PP}}}(x) r r flat PP ( x ) yields a lower 0-1-c risk than the baseline r f l a t ( x ) r_{\mathrm{flat}}(x) r flat ( x ) does. When, in addition, the CI for δ r f l a t P P ( x ) ( x ) \delta_{r_{\mathrm{flat}}^{\mathrm{PP}}(x)}(x) δ r flat PP ( x ) ( x ) is sufficiently small to separate it from the next runner-up, we obtain the guarantee in (27) which says r f l a t P P ( x ) = r f l a t † ( x ) r_{\mathrm{flat}}^{\mathrm{PP}}(x)=r_{\mathrm{flat}}^\dagger(x) r flat PP ( x ) = r flat † ( x ) .
Additional PPI Experiments: Impact of Labeled Data Budget on Performance
We provide additional figures illustrating the behavior of the PPI algorithm for the setting presented in § 5.2 as the amount of training and PPI labeled data varies, ranging from n , l = 100 n,l=100 n , l = 100 to n , l = 3000 n,l=3000 n , l = 3000 samples (Figure 8 from (a) to (d)), while the amount of unlabeled data remains fixed. As expected, with fewer labeled samples (100 and 200 shown in Figure 8 (a) and (b), respectively), neither the trained routing model nor the classifiers can reach Bayes-optimality. The trained routing model r f l a t r_{\mathrm{flat}} r flat exhibits large errors w.r.t. r f l a t † r_{\mathrm{flat}}^\dagger r flat † , as indicated by the larger gray shaded regions. The error of the trained classifiers is reflected in the associated oracle router r f l a t † r_{\mathrm{flat}}^\dagger r flat † , which is also far from the Bayes-optimal router r f l a t ∗ r_{\mathrm{flat}}^* r flat ∗ , as indicated by the blue shaded regions (20.6% and 12.5% disagreement). In this regime, although the PPI correction yields larger relative improvements (w.r.t. r f l a t † r_{\mathrm{flat}}^\dagger r flat † ), the overall routing remains farther from being Bayes-optimal due to the poor quality of the initial classifiers.
Figure 8. Comparison of the Bayes-optimal routing function r f l a t ∗ r^{*}_{\mathrm{flat}} r flat ∗ , the router learned on labeled data r f l a t r_{\mathrm{flat}} r flat , and the PPI-corrected router r f l a t P P r^{\mathrm{PP}}_{\mathrm{flat}} r flat PP , for increasing numbers n = l n = l n = l of labeled training and PPI samples. The error of the oracle w.r.t. the Bayes-optimal r f l a t ∗ r^{*}_{\mathrm{flat}} r flat ∗ is shown in blue shaded regions, the error of the trained router w.r.t. the oracle router is shown in gray shaded regions, and the fixes and errors introduced by PPI are computed w.r.t. the oracle router and shown by the green and red shaded regions. Note that these are computed pointwise for every input x x x , whereas the PPI correction is applied at the bin level, so the errors and fixes also reflect the effect of binning.
As the number of labeled samples increases (1000 and 3000 shown in Figure 8 (c) and (d), respectively), the oracle router more closely approximates the Bayes-optimal router, and the trained routing model also more closely approximates the oracle. At n = 1000 n = 1000 n = 1000 , the trained router still fails to capture the region around x = − 0.60 x = -0.60 x = − 0.60 , where the oracle routes samples to the first classifier. With the PPI correction applied, part of this region is now correctly routed to f 1 f_1 f 1 . At n = 3000 n = 3000 n = 3000 , there is sufficient data to learn both the classifiers and the router; the oracle router is very close to the Bayes-optimal router, the trained router is also very close to the oracle, and the PPI correction leaves the routing unchanged.
We also repeat the experiment but reuse the training data as labeled PPI data, instead of separating the labeled set into two independent partitions (Figure 9 ). In that setting, the theoretical guarantee of Theorem 4.1 no longer holds as the classifiers are trained on the same data used for the correction, and are therefore not independent anymore. In practice, however, this violation has little impact, and performance is only slightly lower than with separate labeled sets (Figure 8 ).
Figure 9. Comparison of the Bayes-optimal routing function r f l a t ∗ r^*_{\mathrm{flat}} r flat ∗ , the router learned on labeled data r f l a t r_{\mathrm{flat}} r flat , and the PPI-corrected router r f l a t P P r^{\mathrm{PP}}_{\mathrm{flat}} r flat PP , for increasing numbers n n n of labeled samples, when the same labeled data is used both for training and for the PPI correction.
In summary, the PPI correction improves routing when the trained router is suboptimal, and its ability to recover the Bayes-optimal routing model depends on the quality of the underlying trained classifiers. The same trend holds when the training data is reused for the PPI correction. When sufficient labeled data is available to train near-optimal classifiers, the correction more closely approaches the Bayes-optimal routing behavior.
Details for LLM Experiments
Datasets
We evaluate the utility of our proposed training losses on two multiple-choice question-answering benchmarks that span commonsense reasoning and mathematical problem solving.
CommonsenseQA. CommonsenseQA [52 ] is a dataset designed to test common sense knowledge and reasoning. Each question is a 5-way multiple choice problem (K = 5 K = 5 K = 5 ), where the goal is to identify semantically related concepts. The dataset contains 12,102 questions covering diverse common sense relations such as causes, effects, and properties of everyday concepts. Questions require reasoning beyond surface-level pattern matching, as distractors are carefully designed to be semantically similar to the correct answer.
GSM8K. GSM8K [11 ] is a dataset of 8,792 grade school math word problems designed to evaluate multi-step mathematical reasoning with final answers being numerical values. Since GSM8K is originally a free-form numerical answer dataset, we convert it to a 4-way multiple choice format (K = 4 K = 4 K = 4 ) to match our classification framework. For each question, we generate distractors using the following procedure:
Distractor generation from weak model: We sample 20 chain-of-thought responses from a weak model (Qwen-1.5B) for each question. We collect all incorrect numerical answers produced by the model and normalize them (handling commas, decimal formatting, currency symbols).
Frequency-based distractor selection: We select the 3 most frequently occurring wrong answers as distractors, ensuring each distractor differs from the correct answer and other distractors to avoid duplicates.
Synthetic fallback: If the weak model does not produce enough distinct wrong answers, we supplement the answer pool with synthetic distractors based on common error patterns (e.g., off-by-one errors, order-of-magnitude mistakes, and arithmetic sign errors).
Random ordering: The correct answer position is randomized across the four labels A to D with a deterministic seed per question to ensure reproducibility.
This conversion procedure yields realistic distractors that reflect actual model errors rather than arbitrary numerical alternatives, making the resulting multiple-choice problems more representative of model failure modes.
Data splitting. For both datasets, we partition the data into training, validation, and test sets using an 80-10-10 split with a fixed random seed for reproducibility. The validation set is used for hyperparameter selection and early stopping, while the test set is held out for final evaluation.
Model and Training Details
Setup and feature extraction. We mine LLM input-output pairs from natural language prompts and their natural language responses from a community repository [46 ] covering our multiple choice datasets (this repository also contains ground truth answers). To convert the natural language prompts and responses to fixed size vectors we use a sentence embedding language model.6 Specifically, each sample is represented by a prompt embedding x ∈ R 768 x \in\mathbb{R}^{768} x ∈ R 768 , obtained from the sentence embedding of the prompt text and answer embeddings z t ∈ R 768 z_t \in\mathbb{R}^{768} z t ∈ R 768 for t ∈ [ T ] t \in[T] t ∈ [ T ] . Note that this means that the LLMs as well as the sentence embedding model are used to generate features ( x , z t ) (x,z_t) ( x , z t ) for our classifiers f ~ t \tilde{f}_t f ~ t which we describe in the next paragraph (the LLMs are not the classifiers themselves). For t = 1 t=1 t = 1 , z 1 z_1 z 1 is the embedding of the answer produced by the first LLM, so z 1 z_1 z 1 carries information in these experiments. The prompt embedding x x x is shared by all classifiers and each z t z_t z t is specific to the t t t -th LLM, so ( x , z t ) (x,z_t) ( x , z t ) for t > 1 t>1 t > 1 does not contain z 1 z_1 z 1 . The flat router observes only x x x .
Architecture of classifier f ~ t \tilde{f}_t f ~ t . We describe the architecture of the classifiers used in the LLM experiments in Eq. (170) . The classifier is a feedforward neural network that takes as inputs x , z t x,z_t x , z t and outputs K K K scores that sum to zero:
f ~ t ( x , z t ) = Θ 3 ( t ) ReLU ( BN ( Θ 2 ( t ) ReLU ( BN ( Θ 1 ( t ) [ x ; z t ] ) ) ) ) . (170) \tilde{f}_t(x,z_t)=\Theta_3^{(t)}\operatorname{ReLU}\left(\operatorname{BN}\left(\Theta_2^{(t)}\operatorname{ReLU}\left(\operatorname{BN}\left(\Theta_1^{(t)}[x;z_t]\right)\right)\right)\right). \tag*{(170)} f ~ t ( x , z t ) = Θ 3 ( t ) ReLU ( BN ( Θ 2 ( t ) ReLU ( BN ( Θ 1 ( t ) [ x ; z t ] ) ) ) ) . (170)
Here, BN denotes batch normalization, and [ ⋅ ; ⋅ ] [\cdot;\cdot] [ ⋅ ; ⋅ ] denotes concatenation. The superscript ( t ) (t) ( t ) denotes that parameter matrices Θ k ( t ) \Theta_k^{(t)} Θ k ( t ) belong to the classifier network f ~ t \tilde{f}_t f ~ t . The classifier parameters have dimensions Θ 1 ( t ) ∈ R 128 × 1536 \Theta_1^{(t)} \in\mathbb{R}^{128 \times1536} Θ 1 ( t ) ∈ R 128 × 1536 , Θ 2 ( t ) ∈ R 128 × 128 \Theta_2^{(t)} \in\mathbb{R}^{128 \times128} Θ 2 ( t ) ∈ R 128 × 128 , and Θ 3 ( t ) ∈ R ( K − 1 ) × 128 \Theta_3^{(t)} \in\mathbb{R}^{(K-1) \times128} Θ 3 ( t ) ∈ R ( K − 1 ) × 128 . The classifier outputs K − 1 K-1 K − 1 scores, with the final class score determined by the constraint that the scores sum to zero. During training, batch normalization normalizes each hidden layer with the mean and variance of the current mini-batch, so the output of a network for one sample also depends on the other samples in its mini-batch (during training only, not inference). While this slightly deviates from Π ~ f l a t \widetilde{\Pi}_{\mathrm{flat}} Π flat and Π ~ c a s c \widetilde{\Pi}_{\mathrm{casc}} Π casc definition as additional components such as normalization or regularization were not included in the optimization that our theory considers, it is important to verify that our proposed approach works in a standard neural network setting with the usual components such as batch normalization.
Architecture of flat router. The soft router r ~ ( x ) \tilde{r}(x) r ~ ( x ) outputs a probability distribution over the classifiers. It is implemented as a feedforward neural network:
r ~ ( x ) = Softmax ( BN ( Θ 2 ( r ) ReLU ( BN ( Θ 1 ( r ) x ) ) ) ) . (171) \tilde{r}(x)=\operatorname{Softmax}\left(\operatorname{BN}\left(\Theta_2^{(r)}\operatorname{ReLU}\left(\operatorname{BN}\left(\Theta_1^{(r)}x\right)\right)\right)\right). \tag*{(171)} r ~ ( x ) = Softmax ( BN ( Θ 2 ( r ) ReLU ( BN ( Θ 1 ( r ) x ) ) ) ) . (171)
The superscript ( r ) (r) ( r ) in Θ k ( r ) \Theta_k^{(r)} Θ k ( r ) denotes that we refer to the parameters of the router. The router parameters have dimensions Θ 1 ( r ) ∈ R 128 × 768 \Theta_1^{(r)} \in\mathbb{R}^{128 \times768} Θ 1 ( r ) ∈ R 128 × 768 and Θ 2 ( r ) ∈ R T × 128 \Theta_2^{(r)} \in\mathbb{R}^{T \times128} Θ 2 ( r ) ∈ R T × 128 .
Architecture of cascade router. The cascade consists of similar stage-specific classifiers but with deferral routing modules implemented as feedforward neural networks. The soft deferral rule implemented in the deferral routing modules d ~ t ( x , z 1 : t ) \tilde{d}_t(x,z_{1:t}) d ~ t ( x , z 1 : t ) takes the full history available up to stage t t t , namely the question embedding and all answer embeddings observed so far, and outputs a deferral probability. A deferral module is defined as
d ~ t ( x , z 1 : t ) = σ ( BN ( Φ 2 ( t ) ReLU ( BN ( Φ 1 ( t ) [ x ; z 1 : t ] ) ) ) ) , (172) \tilde{d}_t(x,z_{1:t})=\sigma\left(\operatorname{BN}\left(\Phi_2^{(t)}\operatorname{ReLU}\left(\operatorname{BN}\left(\Phi_1^{(t)}[x;z_{1:t}]\right)\right)\right)\right), \tag*{(172)} d ~ t ( x , z 1 : t ) = σ ( BN ( Φ 2 ( t ) ReLU ( BN ( Φ 1 ( t ) [ x ; z 1 : t ] ) ) ) ) , (172)
where σ \sigma σ denotes the sigmoid function. The deferral module at stage t t t has input dimension 768 + 768 t = 768 ( 1 + t ) 768+768t=768(1+t) 768 + 768 t = 768 ( 1 + t ) , so it is parameterized by Φ 1 ( t ) ∈ R 128 × 768 ( 1 + t ) \Phi_1^{(t)} \in\mathbb{R}^{128 \times768(1+t)} Φ 1 ( t ) ∈ R 128 × 768 ( 1 + t ) and Φ 2 ( t ) ∈ R 1 × 128 \Phi_2^{(t)} \in\mathbb{R}^{1 \times128} Φ 2 ( t ) ∈ R 1 × 128 . There are only T − 1 T-1 T − 1 deferral modules, since the final stage always exits if reached.
Batch Normalization. Batch normalization is optional in our setup. As in the synthetic experiments, we observe that it only speeds up training and does not change performance significantly.
Training. Both the flat and the cascade architectures are trained using the AdamW optimizer with learning rate 10 − 3 10^{-3} 1 0 − 3 and mini-batch size 32. Training runs for a maximum of 1000 epochs for the cascade model and 100 epochs for the router model, with early stopping using a patience of 20 epochs based on validation loss. We employ the proposed hinge-based surrogate loss that balances classification accuracy with routing costs. Training takes approximately 3 hours per dataset with the majority of the time spent on generating embeddings from model answers using an Nvidia GeForce RTX 3060 12GB card.
Evaluation protocol. We report performance using 10 bootstrap trials on the held-out test set. For each trial, we resample the test set with replacement and compute the accuracy of each classifier on all test samples and on the samples routed to it, which for the cascade are the samples that exit at it. We also compute the fraction of samples routed to each classifier, or deferred past each stage of the cascade. This bootstrap procedure provides confidence intervals that account for finite sample variability.
Algorithm implementation. We implement our method in PyTorch and make our code available: https://github.com/networkslab/Consistent-Classifier-Routing.
Additional LLM Experiments
LLM Experiments with Additional Models and Datasets
In this appendix we extend the experiments of § 5.3 in two directions. We first train a cascade router with the LLaMA classifiers on GSM8K, and we then repeat the flat-router experiment of § 5.3 with Qwen classifiers on CommonsenseQA, which changes both the model family and the dataset.
Cascade router with LLaMA classifiers. We use the three cost configurations of § 5.3 , so the per-classifier costs range from [0, 0.01, 0.02] in Configuration 1 to [0, 0.1, 0.2] in Configuration 3. A prompt pays for every classifier it passes through, so an answer from LLaMA 405B costs κ 1 + κ 2 + κ 3 \kappa_1 + \kappa_2 + \kappa_3 κ 1 + κ 2 + κ 3 , from 0.03 in Configuration 1 to 0.3 in Configuration 3. Figure 10 shows how the trained cascade spends this budget. In its bottom row, the fraction of prompts deferred past LLaMA 1B falls from about 66% in Configuration 1 to about 36% in Configuration 3, so the share answered by LLaMA 1B grows from about one third to about two thirds. The fraction deferred past LLaMA 70B, which is the share answered by LLaMA 405B, falls from about 60% to about 10%. The top row shows that LLaMA 70B and LLaMA 405B answer at least 90% of the prompts they receive correctly in every configuration, close to their accuracy on all prompts. For LLaMA 1B, the median accuracy on the prompts it answers stays within 4 percentage points of its accuracy on all prompts, so the first deferral module does not single out prompts that LLaMA 1B answers better than average. In contrast, the flat router of Figure 4 sends only about 9% of the prompts to LLaMA 1B in Configuration 1, and LLaMA 1B answers about 66% of them correctly against about 49% over all prompts.
Figure 10. Classifier accuracy and deferral fractions of the cascade router with LLaMA classifiers on GSM8K across three cost configurations (costs increasing from left panel to right panel). The top row shows the accuracy of each classifier on all test prompts and on the prompts it answers, and the bottom row shows the fraction of prompts deferred at each classifier.
Flat router with Qwen classifiers. We repeat the flat-router experiment of § 5.3 with Qwen classifiers on CommonsenseQA and the same three cost configurations, which changes both the model family and the dataset. Figure 11 shows the results. In the bottom row, the share of prompts routed to Qwen 1.5B stays near 30% in Configurations 1 and 2 and rises to 74% in Configuration 3, while the share routed to Qwen 32B falls from 38% to 9%. In the top row, Qwen 32B is the most accurate classifier on all prompts at 91%, while Qwen 72B reaches 83%. In Configuration 1, the prompts routed to Qwen 1.5B are answered correctly 83% of the time against 76% over all prompts, so the router keeps easier prompts for the cheapest classifier, as the flat router of Figure 4 does for LLaMA 1B. In Configuration 3, Qwen 1.5B answers most prompts, and its accuracy on them falls to about its accuracy on all prompts. These results suggest that the behavior of our training loss is not specific to one model family.
Figure 11. Classifier accuracy and routing fractions of the flat router with Qwen classifiers on CommonsenseQA across three cost configurations (costs increasing from left panel to right panel). The top row shows the accuracy of each classifier on all test prompts and on the prompts routed to it, and the bottom row shows the fraction of prompts routed to each classifier.
Comparison to Supervised LLM Cascade Baselines
Our work is primarily theoretical, but in LLM multiple-choice question answering it can be compared directly with applied cascade methods. In the cascade router setting, we compare it with FrugalGPT [6 ] and TREACLE [Zhang et al., 2024] in one configuration, with Qwen classifiers on CommonsenseQA and per-classifier costs κ = [ 0 , 0.05 , 0.06 ] \kappa= [0, 0.05, 0.06] κ = [ 0 , 0.05 , 0.06 ] in order of model size. We choose these two methods because both learn when to escalate along a cascade of fixed LLMs under a cost budget, which makes them the separate-training counterpart of our setting. FrugalGPT uses a cascade of LLM APIs within a budget constraint [6 ] , and TREACLE casts the choice of LLM and prompt as budget-constrained policy optimization [Zhang et al., 2024]. Other LLM cascade routing methods do not fit this setup at all for structural reasons. C3PO [56 ] does not admit direct comparison as it is an unsupervised technique, whereas the methods we compare to all require direct supervision. RouteLLM [41 ] learns from human preference data to route between exactly two LLMs, so it is a two-model router rather than a cascade of classifiers. RouterDC [7 ] trains a query-only flat router to select one LLM per query (so it does not support a cascade router setup). It keeps the LLMs fixed and does not train classifiers on their outputs or optimize a 0-1-c c c objective, so it addresses a different problem. Our method and Joint CE use the Qwen cascade router, which is replicated for FrugalGPT and TREACLE cascades that query the Qwen models from the cheapest to the most expensive using the same costs. Both FrugalGPT and TREACLE keep the classifiers fixed and train only the routing rule, while our method trains the classifiers and the router jointly. Note that the asymmetry in optimization objectives is deliberate, as this indicates the value added by joint training versus frozen classifiers. Joint classifier - router training methods are expected to perform better and our results confirm this.
Table 1 reports three quantities on the test set for each method. The error rate is the fraction of test prompts answered incorrectly, and the average cost is the mean over the test set of the cost term c r ( h ) c_r(h) c r ( h ) in (9) . The test empirical risk is the sum of the error rate and average cost, which is the empirical 0-1-c c c risk of § 2.2 , the risk our guarantees concern. Joint Cross Entropy (CE) trains the classifiers and the router jointly with the ordinary cross-entropy loss, as described in Appendix § 13 .§ 3 . All methods use the same train-test split. Our method attains the lowest test empirical risk.
Table 1. Error rate, average cost and test empirical risk of each method with Qwen classifiers on the test set of CommonsenseQA. FrugalGPT and TREACLE are run with a target cost budget of 0.03, which acts as a soft constraint, so their average cost can exceed it slightly. Our method attains the lowest test empirical risk. Bold indicates statistically significant difference between the two best models via pairwise Wilcoxon test with 10 trials (our method achieves statistical significance at p < 0.05 p < 0.05 p < 0.05 level).
The results show that joint model-router training under our proposed hinge-based loss provides meaningful improvements in empirical risk, while incurring similar cost to the other methods.
Comparison to Adaptive Inference Baselines
Besides the empirical works which we compared to in Appendix G.2, one might wonder how some naive baselines (e.g. joint training with cross entropy) or theoretical techniques from the broader literature would perform in the cascade router setting. In this appendix, we set up a simple experiment and show that our proposed loss leads to better convergence and lower overall empirical risk.
We compare our proposed loss against adaptive-inference baselines on CommonsenseQA using the Qwen family. Similarly to the other experiments, the LLM weights are frozen and we learn a classifier based on prompt embeddings x i x_i x i and answer embeddings z i z_i z i . The learned classifier associated with stage t t t is f t ( x , z t ) f_t(x,z_t) f t ( x , z t ) . For baselines that use probabilistic predictions, we write
p ^ t ( k ∣ x , z t ) , k ∈ [ K ] , (173) \hat{p}_t(k \mid x,z_t), \qquad k \in[K], \tag*{(173)} p ^ t ( k ∣ x , z t ) , k ∈ [ K ] , (173)
for the class probabilities produced by the corresponding cross-entropy-trained classifier at stage t t t . The baselines are grouped into two categories: separate-training and joint-training baselines.
Separate-training baselines. In these baselines, the classifiers f 1 , f 2 , f 3 f_1, f_2, f_3 f 1 , f 2 , f 3 are trained independently from the routing or deferral rule.
max k ∈ [ K ] p ^ t ( k ∣ x , z t ) ≥ τ t , (174) \max_{k \in[K]} \hat{p}_t(k \mid x,z_t) \ge\tau_t, \tag*{(174)} k ∈ [ K ] max p ^ t ( k ∣ x , z t ) ≥ τ t , (174)
and otherwise defers to the next stage.
1 − max k ∈ [ K ] p ^ t ( k ∣ x , z t ) , (175) 1-\max_{k \in[K]} \hat{p}_t(k \mid x,z_t), \tag*{(175)} 1 − k ∈ [ K ] max p ^ t ( k ∣ x , z t ) , (175)
so lower-confidence predictions are more likely to be deferred.
[38 ] : The standard learning to defer baseline, which learns a classifier and a deferral rule that decides whether to predict or defer to a fixed expert. Its classifier and deferral rule are trained together, while the expert, here the classifier of the most expensive LLM, is trained separately.
Joint-training baselines. In these baselines, classifiers and routing decisions are trained jointly.
Joint cross-entropy training (Joint CE): Joint training of the classifiers and the deferral modules with the loss ∑ t = 1 T r ~ t ( x , z 1 : t ) ( CE ( f ~ t ( x , z t ) , y ) + ∑ t ′ ≤ t κ t ′ ) \sum_{t=1}^{T} \tilde{r}_t(x,z_{1:t})\left(\operatorname{CE}(\tilde{f}_t(x,z_t),y)+\sum_{t'\leq t}\kappa_{t'}\right) ∑ t = 1 T r ~ t ( x , z 1 : t ) ( CE ( f ~ t ( x , z t ) , y ) + ∑ t ′ ≤ t κ t ′ ) , where r ~ t ( x , z 1 : t ) \tilde{r}_t(x,z_{1:t}) r ~ t ( x , z 1 : t ) is the probability of exiting at stage t t t and ∑ t ′ ≤ t κ t ′ \sum_{t'\leq t}\kappa_{t'} ∑ t ′ ≤ t κ t ′ is the cumulative cost of reaching stage t t t , as in (18). Here CE is the cross-entropy of the softmax of the scores f ~ t ( x , z t ) \tilde{f}_t(x,z_t) f ~ t ( x , z t ) . This loss replaces each hinge term of (18) by the cross-entropy and keeps the cost term, so the deferral modules are penalized for deferring to expensive classifiers.
Since Mozannar and Sontag [ [38 ] and Regol et al. [ [45 ] are formulated for two classifiers, we instantiate them using the cheapest and most expensive LLMs. This difference in the number of classifiers reflects the setting our contribution targets as neither of these two methods extends to more than two classifiers, which is the gap our proposed loss fills. Running our method with T = 2 T = 2 T = 2 would not give a more informative comparison, since for two classifiers the loss of Regol et al. [45 ] is a special case of our hinge loss and the two would produce identical results. A difference can therefore only appear for T > 2 T > 2 T > 2 , and Appendix § 13 .§ 4 shows that our loss continues to reduce the risk as T T T grows to 5. All other baselines use three LLMs, same as ours. The router is a cascade router, so a sample that exits at stage t t t pays the cumulative cost κ 1 + ⋯ + κ t \kappa_1 + \cdots+ \kappa_t κ 1 + ⋯ + κ t . The per-classifier costs are κ = [ 0 , 0.05 , 0.06 ] \kappa= [0, 0.05, 0.06] κ = [ 0 , 0.05 , 0.06 ] in order of model size, and every method is evaluated by its test 0-1-c risk under these costs. Figure 12 shows that training with our proposed loss achieves the lowest test-set risk among all baselines. A pairwise Wilcoxon test with 10 trials of our method with the best baseline (at convergence) shows that the difference is statistically significant at the p < 0.05 p < 0.05 p < 0.05 level. Additionally, the joint training methods (second column of legend of Figure 12 ) outperform the separate-training methods, showing the value of training the classifiers together with the router.
Figure 12. Test 0-1-c risk during training. Our method is compared with the separate-training baselines (CT, CT-c, Soft Deferral and L2D from Mozannar and Sontag [38 ] ) and the joint-training baselines (Joint CE and Regol et al. [45 ] ), and it reaches the lowest risk at the end of training.
Scaling with the Number of Classifiers
In our experiments, the number of classifiers has so far been fixed at T = 3 T = 3 T = 3 . To evaluate whether our approach can scale beyond this setting, we consider up to T = 5 T = 5 T = 5 classifiers in the CommonsenseQA experiment (see Figure 13 ). The experiment starts from a single-model setting (T = 1 T = 1 T = 1 ), where no routing rule is required, and progressively increases the number of models to five. We use a cascade router, and the per-classifier costs increase with model size, with κ \kappa κ equal to 0 for Qwen-1.5B, 0.02 for LLaMA-8B, 0.05 for Qwen-32B, 0.06 for Qwen-72B and 0.15 for LLaMA-405B. The results show a consistent reduction in risk as additional classifiers are added, with diminishing returns beyond three classifiers. This aligns with the intuition that adding more classifiers provides additional operating points along the cost-accuracy trade-off curve.
Figure 13. Test 0-1-c empirical risk on CommonsenseQA as a function of the number of classifiers T T T , jointly-trained using our surrogate hinge loss. Starting from a naive single-model setting with the Qwen-1.5B model (T = 1 T = 1 T = 1 ), where no routing rule needs to be learned, we incrementally add more models: Qwen-32B (T = 2 T = 2 T = 2 ), Qwen-72B (T = 3 T = 3 T = 3 ), LLaMA-8B (T = 4 T = 4 T = 4 ), and LLaMA-405B (T = 5 T = 5 T = 5 ) alongside a trainable routing rule. Risk decreases monotonically from T = 1 T = 1 T = 1 to T = 5 T = 5 T = 5 (with diminishing returns beyond T = 3 T = 3 T = 3 ), indicating that the proposed framework scales well with the number of models and consistently benefits from additional routing options.
Additional Feature Variables to Represent Varying Model Capacity
In the main text, we note that the additional feature variables Z t Z_t Z t can represent hidden representations extracted by different architectures with varying modeling capacity. In general, hidden representations produced by larger models are more computationally expensive to obtain, but they are also often more informative. We therefore view these representations as extra “predictive signal” available to the classifier f t f_t f t and to the subsequent cascade routing rules.
This framing can be related to the recently introduced concepts of computationally constrained information [65 ] and epiplexity [19 ] . These formulations provide a statistical abstraction of computational constraints, in which increased model capacity enables access to additional structured information in the data.
The standard approach to model classifiers of varying performance is to consider different hypothesis classes F t , F t + 1 \mathcal{F}_t, \mathcal{F}_{t+1} F t , F t + 1 , where a smaller model is constrained to F t \mathcal{F}_t F t and a larger model to F t + 1 \mathcal{F}_{t+1} F t + 1 . This formulation typically relies on the implicit assumption that the optimal function in F t \mathcal{F}_t F t is worse than that in F t + 1 \mathcal{F}_{t+1} F t + 1 , w.r.t. some expected loss L \mathcal{L} L :
L ( f t ∗ ) ≥ L ( f t + 1 ∗ ) , (176) \mathcal{L}(f_t^*) \ge\mathcal{L}(f_{t+1}^*), \tag*{(176)} L ( f t ∗ ) ≥ L ( f t + 1 ∗ ) , (176)
where f t ∗ ∈ arg min f ∈ F t L ( f ) f_t^* \in\arg\min_{f \in\mathcal{F}_t} \mathcal{L}(f) f t ∗ ∈ arg min f ∈ F t L ( f ) and f t + 1 ∗ ∈ arg min f ∈ F t + 1 L ( f ) f_{t+1}^* \in\arg\min_{f \in\mathcal{F}_{t+1}} \mathcal{L}(f) f t + 1 ∗ ∈ arg min f ∈ F t + 1 L ( f ) .
f t ∗ ∈ arg min f ∈ F t L ( f ) and f t + 1 ∗ ∈ arg min f ∈ F t + 1 L ( f ) . (177) f_t^* \in\arg\min_{f \in\mathcal{F}_t} \mathcal{L}(f) \quad\text{and} \quad f_{t+1}^* \in\arg\min_{f \in\mathcal{F}_{t+1}} \mathcal{L}(f). \tag*{(177)} f t ∗ ∈ arg f ∈ F t min L ( f ) and f t + 1 ∗ ∈ arg f ∈ F t + 1 min L ( f ) . (177)
Instead, we adopt a computationally constrained information-based perspective: we introduce random variables X X X and Z 1 : T Z_{1:T} Z 1 : T , where X X X represents the base information accessible to every model, and ( X , Z t ) (X, Z_t) ( X , Z t ) represents the richer information available to more t t t -th model, which is richer for more powerful ones. As such, we model the random variables Z 1 : T Z_{1:T} Z 1 : T to be dependent. This modeling choice allows us to abstract away from the particular implementation of the architectures generating these hidden representations. However, for Z 1 : T Z_{1:T} Z 1 : T to be interpreted as the hidden representations, the generating architectures must remain fixed during training, since the distribution of Z 1 : T Z_{1:T} Z 1 : T is assumed to remain unchanged.
Limitations and Future Work
In this section we discuss the limitations of our work in more depth. First, while our empirical results suggest that the proposed losses are effective on real datasets, the main contribution of this work remains theoretical. Our primary goal is to provide a theoretical formulation of joint training and to propose a practical loss that is Bayes-consistent with respect to the targeted cost-sensitive zero-one loss. In practice, computational constraints can make full joint training infeasible (for instance, because it requires fitting all classifiers into memory), and simpler training schemes could still be preferable. Nevertheless even in these settings, our analysis can serve as a principled starting point for developing more scalable methods.
A second limitation of our approach is its reliance on the hinge loss. While this loss can be optimized using standard gradient-based techniques, it is less commonly used than smooth surrogate losses such as the logistic loss or the exponential loss. The hinge loss exhibits different optimization properties, such as non-differentiability at zero and flat regions, which can make optimization more challenging in practice. We use the hinge loss because it yields a sharp linear transfer rate from the excess surrogate risk to the excess target risk, as our calibration bounds establish with Theorems 3.1 and 3.2. A linear transfer rate is the strongest transfer rate one can hope for [20 ] . The most natural family of functions that would preserve this behavior is the class of polyhedral losses, which are piecewise-linear convex losses, of which the multiclass hinge loss is one instance. These losses are known to admit linear transfer rates in standard classification settings [20 , 18 ] . We conjecture that our results extend to this broader family.
Smooth surrogates could also be used and might make optimization easier in practice. The tradeoff is that they generally yield weaker guarantees. It is known that for sufficiently non-polyhedral surrogates the square-root transfer rate is unavoidable [20 ] , so replacing our hinge version with a smooth loss would in general forfeit the linear rate our analysis provides. Difficulties in developing a Bayes-consistent cross-entropy surrogate loss for our setting have been discussed by Regol et al. [ [45 ] , but further work in this direction could improve the practical applicability of the proposed surrogate losses.
Finally, our formulation relies on several assumptions that can be restrictive in some settings. First, we assume that the costs can be known before evaluating the classifiers. This is incompatible with autoregressive applications where the output length can vary, which would generally result in a varying cost that we cannot know in advance. Second, our analysis relies on defining a sequence of random variables Z t Z_t Z t that are progressively made accessible to the classifiers, which explains why costlier classifiers can achieve higher predictive performance. In the main text, we note that the Z t Z_t Z t can represent hidden representations computed by progressively larger-capacity models. This abstraction enables our statistical analysis, but it assumes that the distribution of those Z t Z_t Z t remains fixed during training. As a result, our framework directly applies to adaptive inference settings in which backbone networks generating the hidden representations are fixed during optimization, and only the intermediate classifiers are updated. It does not directly extend to settings where the backbone networks are also fine-tuned, since in that case the Z t Z_t Z t could no longer correspond to the hidden representations, as their distribution would evolve during training. This is a limitation of our current setup, as it does not cover the general adaptive inference setting. Extending the framework to this more general setting is an interesting direction for future work.