Quantifying the Value of Privileged Information Using a PAC-Bayesian Approach

Introduction

When collecting data to train machine learning models, we often find ourselves in a position of informational asymmetry. During training, we might have access to high-fidelity or expensive auxiliary features that are completely unavailable when the model is deployed in production. For example, autonomous agents can be trained using ground-truth simulation parameters that real-world onboard sensors cannot observe at runtime [4, 3]. Also, clinical medical diagnostics can utilize comprehensive laboratory biopsies to train an image classifier, yet the deployed model must perform rapid screening using only standard non-invasive diagnostic tools [6]. The Learning Under Privileged Information (LUPI) paradigm provides a formal framework for this setting, aiming to exploit these transient training-only features, termed Privileged Information (PI), to guide the training process toward a superior test-time predictor. However, outside of a few specific domains, we rarely know whether models use this extra information efficiently, or if a given auxiliary feature will genuinely help the learning.

Since its introduction [16], all LUPI theoretical guarantees have been setting-specific. Each proof heavily exploited the choice of a particular learning model (e.g. Support Vector Machines [15, 16] or custom regularized deep networks [18, 14]) rather than the presence of PI itself. Moreover, these frameworks almost entirely evaluate the utility of PI through the lens of computational or sample-complexity advantages, proving, for example, that PI can accelerate optimization convergence rates [11, 15] or demonstrating computational separations when auxiliary features are derived from quantum hardware [1].

What is missing is an algorithm-agnostic account that isolates how the presence of PI reshapes the underlying learning problem from a pure, information-theoretic standpoint. Previous works conflate two questions: whether PI makes the learning problem easier, and whether a chosen deployment rule preserves that advantage when PI is unavailable at inference. We separate and answer both within the PAC-Bayes framework. Starting from Catoni’s PAC bound [2], we introduce two idealized endpoint posteriors: the never-PI Gibbs measure (PI never available) and an oracle, or always-PI, measure on LUPI’s hypothesis space (PI available in training and inference). Their true-risk gap quantifies the potential of PI; since PI is absent at inference, a LUPI learner must push the oracle posterior through a data-independent deployment map $T$, which quantifies its realization. We use Gibbs measures for both endpoints because each uniquely minimizes and simplifies its Catoni upper bound, giving the tightest certificate from which we can gauge the true-risk gap. This lets us define a utility indicator for PI — the log-partition gap between the never-PI and oracle posteriors, which can be evaluated from training quantities as prior integrals without sampling either posterior. We further characterize when this potential can be realized through a deployment map $T$, providing a deployment-certificate criterion. Our contributions are:

• We develop an algorithm-agnostic PAC-Bayes formulation to certify the potential of PI, in which we propose to measure the potential with the log-partition function difference between two idealized Gibbs endpoint posteriors, independently of the learning algorithm. We also provide the condition at which the deployment map can realize the potential.

• We extend the scope of our theoretical results from canonical Gibbs posteriors to an important subclass of exponential-family posteriors, covering exact Bayesian posteriors for linear, generalized-linear, and Gaussian-process models.

• We validate the theoretical results in a Gaussian mixture model (GMM) with Bayesian estimation and Gaussian process classification (GPC) with privileged noise, where the log-partition function difference metric confidently tracks the realized test-time gain as PI quality varies.

Background

Let $\mathcal{X}$ and $\mathcal{Y}$ be topological spaces with their Borel $\sigma$-algebras. Let $\Omega:= \mathcal{X} \times\mathcal{Y}$, $\mathcal{B}(\Omega) := \mathcal{B}(\mathcal{X}) \otimes\mathcal{B}(\mathcal{Y})$, and denote by $\mathcal{D} \in\mathcal{P}(\Omega)$ the probability measures on $\Omega$. The sample $S = ((X_i,Y_i))_{i=1}^{N} \sim\mathcal{D}^{\otimes N}$ takes values in $(\Omega^{N},\mathcal{B}(\Omega)^{\otimes N})$; write $s = ((x_i,y_i))_{i=1}^{N}$ for a realization. Let $(\mathcal{H},d_{\mathcal{H}})$ be a Polish space of Borel-measurable maps $h:\mathcal{X} \to\mathbb{R}$, with Borel $\sigma$-algebra equal to the evaluation $\sigma$-algebra $\mathcal{F} = \sigma(\{\operatorname{ev}_{x}^{-1}(A):x \in\mathcal{X}, A \in\mathcal{B}(\mathbb{R})\})$, where $\operatorname{ev}_{x}(h) = h(x)$. For any $h_0 \in\mathcal{H}$, we consider $\mathcal{M}_{1}^{+}(\mathcal{H}) = \{\rho\in\mathcal{P}(\mathcal{H}): \int d_{\mathcal{H}}(h,h_0)\rho(dh) < \infty\}$. Finally, let $\ell:\mathbb{R} \times\mathcal{Y} \to[0,1]$ be Borel measurable and assume $(h,x,y) \mapsto\ell(h(x),y)$ is jointly measurable.

Learning Using Privileged Information

The Learning Under Privileged Information (LUPI) framework [16, 15] assumes that, during training, the learner observes an additional PI variable $x^{*}$ taking values in a PI space $\mathcal{X}^{*}$. This information is unavailable at inference, so the final predictor must still be a function $h:\mathcal{X}\to\mathbb{R}$ of the ordinary input alone. For a topological PI space $\mathcal{X}^{*}$, we set $\Omega^{*}:=\mathcal{X}\times\mathcal{X}^{*}\times\mathcal{Y}$ with product Borel $\sigma$-algebra $\mathcal{B}(\mathcal{X})\otimes\mathcal{B}(\mathcal{X}^{*})\otimes\mathcal{B}(\mathcal{Y})$, and let $D^{*}$ be a probability measure whose $(X,Y)$-marginal is $D$. Then $S^{*}=((X_i,X_i^{*},Y_i))_{i=1}^{N}\sim(D^{*})^{\otimes N}$.

PI can reveal latent structure, example difficulty, or measurement quality, and thereby guide training towards a better predictor; it changes how $h$ is learned, but not what information $h$ may use after deployment (see Sec. 3.1). The canonical example is SVM+ [16, 15], in which the slack variable $\xi_i$ of each training point $x_i$ is modelled as a correcting function of the privileged variable $x_i^{*}$, i.e., $\xi_i=\langle w^{*},\phi^{*}(x_i^{*})\rangle+b^{*}$, where $\xi_i$ measures how difficult the $i$-th example is to classify — the margin violation of a training point — a quantity that simply does not exist at prediction time, so the correcting function has no role to play once training finishes (also see Sec. B).

Throughout, we treat PI as an intrinsic property of the data rather than of the learner.

Assumption 1 (PI is a property of the data). The privileged variable is generated independently of the model: for any hypothesis/weights $w$,

Equivalently, the conditional law of $X^{*}$ given $X$ does not depend on the learner. This makes the conditional kernel $x\mapsto P(X^{*}\in\cdot\mid X=x)$ a fixed, data-independent object that we may use when constructing the deployment map (Sec. 3.1).

Subsequent work has extended LUPI beyond explicit slack correction. MIML-FCN+ [19] introduced a deep analogue of SVM+, using a privileged network to predict the loss of the ordinary-input network. Generalized distillation [12] instead trains a teacher on privileged inputs and transfers its soft predictions to a student defined on $\mathcal{X}$. A complementary line uses PI to model training labels’ uncertainty: GPC+ represents privileged information through input-dependent label noise [8], while heteroscedastic dropout [10] makes the variance of Gaussian dropout a learned function of $x^{*}$. More recently, TRAM [5] transfers PI through a shared representation with privileged and ordinary prediction heads, training the ordinary head to approximate the prediction obtained after marginalizing the unavailable information.

PAC-Bayes Framework

A prior $\pi\in\mathcal{M}_{1}^{+}(\mathcal{H})$ is fixed independently of the training samples $S$. A randomized learning rule is a Markov kernel $\mathcal{A}:\Omega^{N}\times\mathcal{F}\to[0,1]$, such that for every $s\in\Omega^{N}$, the map $B\mapsto\mathcal{A}(s,B)$ is a probability measure on $(\mathcal{H},\mathcal{F})$, and for every $B\in\mathcal{F}$, the map $s\mapsto\mathcal{A}(s,B)$ is $\mathcal{B}(\Omega)^{\otimes N}$-measurable. For random sample $S$, $\rho_{S}:=\mathcal{A}(S,\cdot)$ is a random measure on $(\mathcal{H},\mathcal{F})$.

We define the true risk and the empirical risk:

Thus $r_{S}$ is a random functional and $r_{s}$ is the corresponding realization. We write $r$ when the sample/realization is clear from the context. For integrable $f$, denote by $\langle f\rangle_{\rho}=\int f(h)\rho(\mathrm{d}h)$ the average function value. In particular, $\langle R\rangle_\rho= \mathbb{E}_{h\sim\rho}[R(h)]$ and $\langle r\rangle_\rho= \mathbb{E}_{h\sim\rho}[r(h)]$ are the true and empirical risks of $h\sim\rho$. When $\rho=\rho_S$, both are random because the posterior depends on the sample. Catoni’s theorem controls this sample-dependence.

Theorem 1 (2). Let $\pi\in\mathcal{P}(\mathcal{H})$ be independent of the sample. For any fixed inverse-temperature $\lambda>0$ and error rate $\varepsilon\in(0,1)$, with probability at least $1-\varepsilon$ over $S\sim D^{\otimes N}$, simultaneously for every $\rho\in\mathcal{P}(\mathcal{H})$ satisfying $\rho\ll\pi$,

where $\Phi_a^{-1}(x)=(1-\exp(-ax))/(1-\exp(-a))$ is the inverse Catoni function and $KL$ the Kullback–Leibler divergence. Also, the bound applies to every posterior selected by a Markov kernel, including $\rho_S=\mathcal{A}(S,\cdot)$.

For $\lambda>0$, the Gibbs learning rule returns the posterior $\rho_\lambda$ defined by, for a realization $s$,

Joint measurability makes the Gibbs learner $\mathcal{A}_\lambda(s,B):=\rho_\lambda(B)$, $B\in\mathcal{F}$, a Markov kernel. Importantly, $\rho_\lambda$ uniquely minimizes the right-hand side of (3).

Lemma 1 (Gibbs posterior minimizes the Catoni bound). Fix $\lambda>0$ and $\varepsilon\in(0,1)$, and define the Catoni certificate

Then, among all posteriors $\rho\ll\pi$, the Gibbs posterior $\rho_\lambda$ of (4) is the unique minimizer of $B_\lambda$, and $B_\lambda(\rho_\lambda)=\Phi_{\lambda/N}^{-1}\left(\frac{-\log Z_\lambda+\log\varepsilon^{-1}}{\lambda}\right)$.

Proof. See Sec. A.1.

Remark 1 (Gibbs identities). Two identities used repeatedly below follow directly from $\mathrm{d}\rho_\lambda/\mathrm{d}\pi=e^{-\lambda r}/Z_\lambda$:

The second identity is what makes the log-partition function a computable stand-in for the (empirical-risk + KL) budget of a Gibbs posterior.

PAC-Bayes Analysis of Privileged Information

Our analysis follows a potential–realization–transfer storyline. We first introduce two idealized endpoint Gibbs measures: the always-PI endpoint, which retains privileged information at prediction time, and the never-PI endpoint, which never observes it; their separation quantifies the potential of PI. Since PI is unavailable at inference, we then deploy the always-PI measure to the hypothesis space and ask how much of this potential can be realized. Finally, we transfer the theory from idealized Gibbs measures to a broad fixed-feature class of posteriors via a data-independent change of prior. Thm. 2 and 3 address the two levels of the argument: comparison of PAC-Bayes certificates and separation of the true endpoint risks, respectively.

Basic assumptions

Let $(\mathcal{H}^{*},d_{\mathcal{H}^{*}})$ be a complete separable metric space of Borel-measurable PI hypotheses $h^{*}\colon\mathcal{X}\times\mathcal{X}^{*}\longrightarrow\mathbb{R}$. We define the Borel $\sigma$-algebra $\mathcal{F}^{*}$ on $\mathcal{H}^{*}$ and collection of measures $\mathcal{M}_{1}^{+}(\mathcal{H}^{*})$ in the same way as $(\mathcal{H},\mathcal{F})$. For a realization $s^{*}=((x_{i},x_{i}^{*},y_{i}))_{i=1}^{N}$, we define

Accordingly, $r_{S^{*}}^{*}$ is a random functional and $r_{s^{*}}^{*}$ is its realization. For simplicity, we write $r^{*}$, and likewise $r$, whenever the relevant sample is unambiguous.

Since $x^{*}$ is unavailable at test time, a learned hypothesis in $\mathcal{H}^{*}$ cannot be deployed directly. We therefore assume a Borel-measurable deployment map

which is part of the learning architecture and independent of the training sample. For each $\rho^{*}\in\mathcal{M}_{1}^{+}(\mathcal{H}^{*})$, we denote by $[T_{\sharp}\rho^{*}](A):=\rho^{*}(T^{-1}(A))$, $A\in\mathcal{F}$, the pushforward measure. We also assume the linear-growth condition $d_{\mathcal{H}}(T(h^{*}),h_{0})\leq C_{T}(1+d_{\mathcal{H}^{*}}(h^{*},h_{0}^{*}))$, $h_{0}=T(h_{0}^{*})$, for some $C_{T}>0$ (e.g., $T$ is Lipschitz-continuous), which guarantees that $T_{\sharp}\rho^{*}\in\mathcal{M}_{1}^{+}(\mathcal{H})$ whenever $\rho^{*}\in\mathcal{M}_{1}^{+}(\mathcal{H}^{*})$. Let $\pi^{*}\in\mathcal{M}_{1}^{+}(\mathcal{H}^{*})$ be a data-independent prior. To compare the PI and non-PI constructions under the same prior belief, we take $\pi:=T_{\sharp}\pi^{*}$; since $T$ and $\pi^{*}$ are data-independent, $\pi$ is a valid PAC-Bayes prior for the Catoni bound in Thm. 1.

Example 1 (Marginalization construction for $T$). Assume $\mathcal{X},\mathcal{X}^{*}$ are standard Borel and let $\mu\colon\mathcal{X}\times\mathcal{F}^{*}\longrightarrow[0,1]$ be a Markov kernel. Denote by $\mu_{x}:=\mu(x,\cdot)$ a measure on $(\mathcal{H}^{*},\mathcal{F}^{*})$. A canonical choice is a regular conditional probability distribution of $X^{*}$ given $X=x$, i.e., for each $B\in\mathcal{F}^{*}$, $\mu_{x}(B)=\Pr(X^{*}\in B\mid X=x)$; under Assumption 1 this kernel is independent of the learner. We define $T\colon\mathcal{H}^{*}\longrightarrow\mathcal{H}$ by

If this integral is absolutely finite for every $x$ and the result belongs to $\mathcal{H}$, kernel integration makes $(h^{*},x)\mapsto(Th^{*})(x)$ jointly measurable. Note that, we assume the conditional probability distribution $\mu_{x}$ is independent from the sample $S$ or $S^{*}$; otherwise, the pushforward of the prior $\pi:=T_{\sharp}\pi^{*}$ will be sample-dependent, which fails the requirements of the Catoni bound.

Endpoint posteriors and the Gibbs envelope

Fix the inverse temperature $\lambda$, a realization $s^{*}$ and its projection $s$. The always-PI and never-PI Gibbs posteriors are defined by

where $Z_{\lambda}^{*}(s^{*}):=\mathbb{E}_{h^{*}\sim\pi^{*}}\exp(-\lambda r^{*}(h^{*}))$ and $Z_{\lambda}(s):=\mathbb{E}_{h\sim\pi}\exp(-\lambda r(h))$. We refer to $\rho_{\mathrm{always}}$ and $\rho_{\mathrm{never}}$ jointly as the endpoint measures. The never-PI endpoint is the ordinary baseline. The always-PI endpoint is an oracle benchmark: it is allowed to retain $x^{*}$ at prediction time and, for fixed $(\lambda,\pi^{*})$, uniquely minimizes the PI Catoni certificate over all $\rho^{*}\ll\pi^{*}$. The always-PI $\rho_{\mathrm{always}} \in\mathcal{M}_{+}^{1}(\mathcal{H}^{*})$ and its true risk should be measured with functional $R^{*}$. In this precise PAC-Bayes sense, it is the best Gibbs posterior available with PI. It is not, however, a deployable LUPI posterior.

The deployment map $T$ pushes forward the oracle endpoint $\rho_{\mathrm{always}}$ to $\mathcal{H}$, resulting in the LUPI posterior $\rho_{\mathrm{LUPI}} \in\mathcal{M}_{+}^{1}(\mathcal{H})$,

The endpoint posteriors depend only on their respective empirical risks and priors, whereas $\rho_{\mathrm{LUPI}}$ also depends on how the deployment map $T$ transfers PI to $\mathcal{H}$. Deployment can lose (part of) the PI. For instance, nothing in our construction forbids a degenerate choice of deployment, i.e., $T(h^{*}) \equiv h_{0}$ for a fixed $h_{0}$, which discards the hypothesis learned with PI entirely. We call the signed endpoint gap

the Gibbs envelope. A positive $\mathcal{G}_{\lambda}$ means that PI opens a genuine gap between the two idealized endpoints. If, in addition, deployment places $\rho_{\mathrm{LUPI}}$ between them, i.e.,

then $\mathcal{G}_{\lambda}$ upper-bounds the realized deployment gain, and a wide envelope is necessary for LUPI to be beneficial but not sufficient. We do not assume (11) to always hold: the endpoint separation and the deployment gain are certified separately in the next subsection. This distinction separates two questions: (1) Potential, whether PI creates a positive Gibbs envelope, and (2) Realization, whether the chosen $T$ converts that potential into an LUPI posterior better than the baseline $\rho_{\mathrm{never}}$.

Partition-function criteria for PI utility

To answer the Potential question, we first ask whether PI improves the optimal upper certificate at the oracle endpoint. Let $B_{\lambda}^{*}(\rho^{*})$ denote the PI analogue of $B_{\lambda}(\rho)$ in Lemma 1, formed with $(r^{*},\pi^{*})$. We use the abbreviations $B_{\lambda}^{*} := B_{\lambda}^{*}(\rho_{\mathrm{always}})$ and $B_{\lambda} := B_{\lambda}(\rho_{\mathrm{never}})$. For the observed pair $(s^{*},s)$, we define the log-partition gap

Throughout, $\log$ denotes the natural logarithm. Because the endpoint posteriors minimize their respective empirical PAC-Bayes objectives, this gap can be evaluated from prior integrals before either posterior is normalized. The next theorem shows that the sign of $\Delta Z$ orders the two endpoint upper certificates.

For a Gibbs measure, the KL term and empirical risk combine into the log-partition function (Remark 1), an identity that makes the comparison both exact and computable; its extension beyond the canonical Gibbs form is discussed in Sec. 3.4.

Theorem 2 (Criterion for the tighter endpoint certificate). Fix $\lambda> 0$ and $\varepsilon\in(0,1/2)$. With probability at least $1-2\varepsilon$ over $S^{*} \sim(D^{*})^{\otimes N}$ and its projection $S$, both endpoints’ upper certificates are valid:

Conditionally on the observed sample, the following equivalences hold:

where $\Delta_{\mathrm{KL}} = \mathrm{KL}(\rho_{\mathrm{always}}\Vert\pi^{*}) - \mathrm{KL}(\rho_{\mathrm{never}}\Vert\pi)$ and $\Delta_{r} = \langle r\rangle_{\rho_{\mathrm{never}}} - \langle r^{*}\rangle_{\rho_{\mathrm{always}}}$. Proof. See Sec. A.2.

Proof idea. Both certificates come from Catoni’s bound plus a union bound. Since $\Phi^{-1}_{\lambda/N}$ is strictly increasing, $B_{\lambda}^{*}<B_{\lambda}$ is equivalent to an ordering of their arguments, in which the confidence terms $\log\varepsilon^{-1}/\lambda$ cancel and leave $\Delta_r>\Delta_{\mathrm{KL}}/\lambda$. Substituting the Gibbs identities (6) collapses this quantity exactly to $\Delta_Z/\lambda$, so the whole condition reduces to $\Delta_Z>0$. Intuitively, $\Delta_r$ measures how much PI lowers the fitted (empirical) risk, while $\Delta_{\mathrm{KL}}/\lambda$ is the extra model-complexity price paid on the richer PI space; PI helps the certificate exactly when the former outweighs the latter.

Remark 2. $\Delta_Z$ is, therefore, a pre-training criterion, rather than a risk certificate itself. It can be estimated from the empirical risks and prior integrals without fitting a normalized endpoint posterior. A positive value certifies that the always-PI endpoint has the tighter upper certificate; it does not yet imply that the Gibbs envelope in (10) is positive.

Thm. 2 isolates the potential of PI visible at the oracle endpoint. To study the Realization question, we must account for what changes when that endpoint measure is transported through $T$. We define the empirical deployment gap and the relative-entropy contraction by

Here $(r\circ T)(h^{*})=r(T(h^{*}))$. We present an upper certificate for the LUPI true risk as follows.

Corollary 1 (Deployment-certificate criterion). For any fixed $\lambda>0$ and $\varepsilon\in(0,1)$, with probability at least $1-\varepsilon$ over $S^{*}\sim(D^{*})^{\otimes N}$,

Conditionally on the observed sample $s^{*}$, we have

Moreover, $\kappa_T(s^{*})\ge0$. Hence, the posterior-free condition

is sufficient for the LUPI upper certificate to be tighter than the never-PI certificate. If $\kappa_T(s^{*})=0$, it is also necessary.

Proof. See Sec. A.3.

Proof idea. Applying Catoni on the augmented sample space to the $x$-only loss $\ell(h(x),y)$ covers both $\rho_{\mathrm{LUPI}}$ and $\rho_{\mathrm{never}}$ simultaneously, so no union bound is needed. Comparing the two certificate arguments and inserting the Gibbs identities together with the definitions of $D_T$ and $\kappa_T$ gives the stated equivalence. The sign $\kappa_T\ge0$ is the data-processing inequality for KL under the pushforward $T$: transporting both $\rho_{\mathrm{always}}$ and $\pi^{*}$ through the same $T$ can only contract their divergence.

Remark 3. The sufficient criterion (18) does not require a separate posterior-fitting procedure. We have $D_T(s^{*})=Z_{\lambda}^{*}(s^{*})^{-1}\int_{\mathcal{H}_{*}}[r(T(h^{*}))-r^{*}(h^{*})]\exp(-\lambda r^{*}(h^{*}))\pi^{*}(dh^{*})$. Thus $\Delta_Z$ and $D_T$ can be estimated from samples $h^{*}\sim\pi^{*}$ whenever $T(h^{*})$, $r(T(h^{*}))$, and $r^{*}(h^{*})$ are accessible.

Both Thm. 2 and Corollary 1 compare upper certificates, not the true risks themselves. In particular, $B_{\lambda}(\rho_{\mathrm{LUPI}}) < B_{\lambda}(\rho_{\mathrm{never}})$ does not imply $\langle R\rangle_{\rho_{\mathrm{LUPI}}} < \langle R\rangle_{\rho_{\mathrm{never}}}$. A true-risk separation requires an upper certificate for one posterior and a lower certificate for the other. The latter is obtained by applying Catoni’s bound to the complementary loss.

Lemma 2 (Catoni lower certificate). Let $\pi$ be a data-independent prior over $\mathcal{H}$, $\ell$ a bounded loss, and $\lambda> 0$. For every posterior $\rho$, with probability at least $1-\varepsilon$ over the draw of $S$,

Proof. See Sec. A.4.

Proof idea. Apply the Catoni upper bound to the complementary loss $\ell' = 1-\ell\in[0,1]$, whose posterior-averaged true and empirical risks are $1-\langle R\rangle_{\rho}$ and $1-\langle r\rangle_{\rho}$; rearranging turns the upper bound on $1-\langle R\rangle_{\rho}$ into a lower bound on $\langle R\rangle_{\rho}$.

The lower certificate turns the potential question into a two-sided test. The next result gives an exact criterion for separating the certificates and a sufficient criterion for proving that the Gibbs envelope is positive.

Theorem 3 (Two-sided certificate for a positive Gibbs envelope). Fix $\lambda> 0$ and $\varepsilon\in(0,1/2)$. For the always-PI and never-PI Gibbs posteriors, define $a^* := \langle r^*\rangle_{\rho_{\mathrm{always}}} + \left(\operatorname{KL}(\rho_{\mathrm{always}}\Vert\pi^*)+\log\varepsilon^{-1}\right)/\lambda$ and $b := 1-\langle r\rangle_{\rho_{\mathrm{never}}}+\left(\operatorname{KL}(\rho_{\mathrm{never}}\Vert\pi)+\log\varepsilon^{-1}\right)/\lambda$. With probability at least $1-2\varepsilon$ over $S^* \sim(\mathcal{D}^*)^{\otimes N}$ and its projection $S$,

Conditionally on a realization $s^*$, separation of the two certificates is characterized exactly by

On the event in (19), either side of (20) implies

Moreover, the following condition is sufficient for (20):

where $C_{\lambda}=1-\frac{N(1-e^{-\lambda/N})}{\lambda}$. Since $\rho_{\mathrm{always}},\rho_{\mathrm{never}}$ are Gibbs measures, the above condition is equivalent to

Proof. See Sec. A.5.

Proof idea. Pair the upper certificate $B_{\lambda}^*$ for the always-PI endpoint (Thm. 2) with the lower certificate $L_{\lambda}(\rho_{\mathrm{never}})$ (Lemma 2) via a union bound. Writing both in terms of $\Phi_{\lambda/N}^{-1}$ and substituting its closed form turns $B_{\lambda}^* < L_{\lambda}(\rho_{\mathrm{never}})$ into the exponential inequality (20). The linear upper bound $\Phi_c^{-1}(x)\leq cx/(1-e^{-c})$ yields the simpler sufficient condition on $\Delta_r$, which the Gibbs identities rewrite as the product-of-partitions condition (23). Note this controls the product $Z_{\lambda}Z_{\lambda}^*$: $\Delta Z>0$ (i.e. $Z_{\lambda}^*>Z_{\lambda}$) is necessary but not sufficient for a true-risk separation.

Corollary 2 (Deployment true-risk separation). In the setting of Thm. 3, define $u := \langle r\rangle_{\rho_{\mathrm{LUPI}}} + \frac{\operatorname{KL}(\rho_{\mathrm{LUPI}}\Vert\pi)+\log\varepsilon^{-1}}{\lambda}$. With probability at least $1-2\varepsilon$ over $S^* \sim(D^*)^{\otimes N}$,

Conditionally on $s^*$, we have

A sufficient condition is

Proof. See Sec. A.6.

From endpoint Gibbs posteriors to practical algorithms

The preceding results are stated for the canonical Gibbs form $\rho\propto\pi e^{-\lambda r}$, which has a limited scope. We now extend them to the fixed-feature subclass of the exponential family (17). We identify an exact extension: any data-independent term in the log-density of this class can be absorbed into a reshaped prior, resulting in an ordinary Gibbs measure on the empirical risk, to which we can directly transfer our theory of Sec. 3.3.

Definition 1 (Fixed-feature subclass of exponential family). A posterior $\rho\ll\pi$ is called fixed-feature if there exist a data-independent measurable map $\tilde{\phi}: \mathcal{H} \to\mathbb{R}^{k}$, a fixed vector $\mu\in\mathbb{R}^{k}$, and $\lambda> 0$ such that

where $\psi(\lambda,\mu)$ is the log-normalizing constant. Thus, the random sample $S$ enters the density only through $r$. The feature $\tilde{\phi}$ and its coefficient $\mu$ describe a fixed shape.

This class contains exact Bayesian posteriors for linear, generalized-linear, and Gaussian-process models. Quadratic Gibbs members have means and modes that recover kernel ridge, frozen last-layer, and NTK-linearized estimators, and include Gaussian variational laws only under an explicit fixed-natural-parameter condition. Each claim and its limitations are verified separately in Sec. C.

Lemma 3 (Reduction to a Gibbs posterior). Define the reshaped prior $\pi_{\mu} \propto\pi e^{-\langle\mu,\tilde{\phi}\rangle}$. Every fixed-feature posterior is the Gibbs posterior on $r$ at temperature $\lambda$ relative to $\pi_{\mu}$:

Proof. See Sec. A.7.

The reduction is exact, but its data-independence condition is essential. If $(\tilde{\phi},\mu)$ is fixed before observing the sample, then $\pi_{\mu}$ is a valid PAC-Bayes prior, and the endpoint results in Thm. 2 and 3 apply with $\pi$ replaced by $\pi_{\mu}$ and $Z_{\lambda}$ by $\mathbb{E}_{\pi_{\mu}}[e^{-\lambda r}]$, with the analogous substitutions on $\mathcal{H}^*$.

Empirical Validation

We evaluate the log-partition function gap (12) between two endpoint posteriors motivated by Thm. 2, i.e., $\Delta Z > 0$, and whether its magnitude tracks the realized gain from PI. In both studies, $T$ is identified with an identity map on the parameter space shared by $\mathcal{H}$ and $\mathcal{H}^{*}$, hence eliminating the need to consider the additional deployment distortion of PI. Non-identity examples of $T$ are given in Sec. B.

Bayesian estimation of a Gaussian mixture

Our first case study is Bayesian parameter estimation for a two-component isotropic Gaussian mixture model (GMM):

where $\mathcal{N}(x;\mu,\sigma^{2}I_{n})$ denotes the probability density at point $x \in\mathbb{R}^{n}$ of a Gaussian r.v. with parameter $(\mu,\sigma^{2})$. Denoting by $\Theta:= [0,1] \times\mathbb{R}^{n} \times\mathbb{R}^{n} \times\mathbb{R}_{>0} \times\mathbb{R}_{>0}$ the parameter space of GMM, the hypothesis space thereof is

In this study, we set the privileged information to be, for each $x_i$, the label $z_i \in\{1,2\}$ of the Gaussian cluster from which $x_i$ is sampled. Therefore, the hypothesis of the LUPI case is the joint density function of $(x,z)$,

where we let $w_1 = w$ and $w_2 = 1-w$. Thus, both hypothesis spaces are parameterized/identified by the same parameter space $\Theta$. However, $\theta\mapsto p_{\theta}$ is not a bijective identification, e.g., swapping the label of Gaussian clusters leads to the same $p_{\theta}$. We consequently formulate Bayesian inference on $\Theta$ and understand the corresponding measures on $\mathcal{H}_{\mathrm{GMM}}$ and $\mathcal{H}_{\mathrm{GMM}}^{*}$ as pushforwards under $\theta\mapsto p_{\theta}$ and $\theta\mapsto p_{\theta}^{*}$, respectively. Deployment marginalizes the training-only label $z$,

Equivalently, $T$ can be identified with the identity map on $\Theta$. We, therefore, place one proper, data-independent prior $\pi$ on $\Theta$ and use its pushforwards as the priors on both hypothesis spaces. We take a Beta prior for $w$ and independent Normal–Inverse-Gamma (NIG) priors for $(\mu_z,\sigma_z^2)$, $z \in\{1,2\}$; see Sec. D for details. Given a PI-augmented data point $s^{*} = ((x_i,z_i))_{i=1}^{N}$ and its projection $s$, we took the negative log-likelihood for the empirical risk functional:

We note that the risks are unbounded, so the Catoni certificates do not apply literally; therefore, we use $\Delta Z$ diagnostically and verify it tracks the realized gain, leaving a bounded-surrogate or unbounded-loss guarantee to future work. Because these are average negative log-likelihoods, we have, for $\lambda= N$, the Gibbs potential satisfies $\exp(-\lambda r(\theta)) = \prod_{i=1}^{N}p_{\theta}(x_i)$ and $\exp(-\lambda r^{*}(\theta)) = \prod_{i=1}^{N}p_{\theta}^{*}(x_i,z_i)$. Hence, the Bayesian posteriors coincide exactly with the two endpoint Gibbs measures, e.g., for $\rho_{\mathrm{never}}$ and $B \in\mathcal{B}(\Theta)$,

where $Z_{\lambda}(s)$ is precisely the Bayesian marginal likelihood. Here, no prior reshaping through Lemma 3 is required. For $\lambda\ne N$, the factor $\exp(-\lambda r(\theta)) = \prod_i p_{\theta}(x_i)^{\lambda/N}$ instead defines a tempered posterior; we therefore take $\lambda= N$ throughout this example. Both endpoint Gibbs measures are computable.

Without PI, the likelihood is a product of mixtures and is not conjugate to the Beta–NIG prior; we sample $\rho_{\mathrm{never}}$ using a Metropolis–Hastings algorithm. The PI likelihood then factorizes into a Beta–Binomial term and two NIG terms, allowing exact ancestral sampling for $\rho_{\mathrm{always}}$. Taking samples from both endpoint measures, $\Delta Z$ can be estimated with Monte Carlo integration. Full posterior updates and computational details are given in Sec. D.

At inference time, we compare three predictors: (1) Bayesian predictive distribution w/o PI (No PI (Full Bayes)), i.e., $\bar{p}_{\mathrm{never}}(x) := \int_{\Theta} p_{\theta}(x)\rho_{\mathrm{never}}(\mathrm{d}\theta)$; (2) Bayesian predictive distribution w/ PI (With PI (Full Bayes)), i.e., $\bar{p}_{\mathrm{LUPI}}(x) := \int_{\Theta} p_{\theta}(x)\rho_{\mathrm{always}}(\mathrm{d}\theta)$; and (3) maximum a posteriori estimation (MAP) w/o PI (MAP Plug-in (No PI)), included as a point-estimation baseline.

Figure 1 illustrates these predictives for a very low number of training samples $N = 8$. The privileged labels reduce component ambiguity and yield a posterior predictive that is more concentrated around the generating components. In Figure 2, we vary the component overlap and average over 14 datasets per setting. The strong positive association between $\Delta Z$ and the testing accuracy gain (Pearson $r = 0.911$) shows that the partition gap successfully predicts the realized gain from PI across overlap regimes.

Gaussian Process Classification with Privileged Noise (GPC+)

We extend our analysis to Gaussian Process Classification (GPC) [13] as a nonparametric Bayesian method for binary classification. We study whether the partition-function gap $\Delta Z$ can retain its diagnostic power in a supervised scenario as well.

Assume $k: \mathcal{X} \times\mathcal{X} \to\mathbb{R}$, a real-valued positive definite function (a.k.a. kernel). Let $\mathcal{H}_k$ denote the reproducing kernel Hilbert space (RKHS) induced by kernel $k$. Consider the Banach space $C(\mathcal{X})$ consisting of continuous latent functions $f: \mathcal{X} \to\mathbb{R}$, equipped with uniform norm $\lVert\cdot\rVert_\infty$. We endow a data-independent prior $\pi= \mathcal{GP}(0, k)$ on $C(\mathcal{X})$, meaning that for any finite set $X = \{x_i\}_{i=1}^{N} \subset\mathcal{X}$, the prior $\pi$ stipulates $\mathbf{f} := (f(x_1), \ldots, f(x_N)) \sim\mathcal{N}(0, K)$ with $K \in S_{++}^{N}$, the cone of positive definite matrices, and $K_{ij} = k(x_i, x_j)$. Then, the latent-function hypothesis space is $\mathcal{H}_{\mathrm{GPC}} := \operatorname{supp}(\pi) = \overline{\mathcal{H}_k}^{\lVert\cdot\rVert_\infty} \subseteq C(\mathcal{X})$. For the true labels $\mathbf{y} = (y_1, \ldots, y_N)$ corresponding to $X$ with $y_i \in\{-1, 1\}$, the model assumes a noisy threshold variable $z_i = f(x_i) + u_i$ with $u_i \sim\mathcal{N}(0, 1)$ and $y_i = \operatorname{sign}(z_i)$, resulting in the probit likelihood

where $\Phi(x) = \frac{1}{2}\left(1 + \operatorname{erf}(x/\sqrt{2})\right)$ is the standard normal CDF. Inference amounts to computing the posterior $p(f \mid X, \mathbf{y}) \propto p(\mathbf{y} \mid f)\pi(f)$, which — unlike in the conjugate GMM case — is non-Gaussian and must be approximated.

PI indicates the reliability of each training label, and it enters the decision threshold via an extra noise term $\epsilon_i \sim\mathcal{N}(0, \sigma_i^2)$ that is additive, independent, and localized [8]. In our case, we use a pre-defined mapping for the privileged noise and the corresponding per-point standard deviation, whose role is shown in (31):

where $x_i^*$ is the PI variable w.r.t. $x_i$, indicating the reliability of the label $y_i$. Combining the noise yields $u_i+\epsilon_i \sim\mathcal{N}(0,1+\sigma_i^2)$, which expands the probit variance:

The LUPI hypothesis space (GPC+) is $h_f^*(x,x^*) := f(x)/q(x^*)$, $\mathcal{H}_{\mathrm{GPC}}^* := \{h_f^*: f \in\mathcal{H}_{\mathrm{GPC}}\}$. Both hypothesis spaces are therefore indexed by the same latent function $f$. Since $q$ is fixed and strictly positive, this indexing is one-to-one, hence the deployment map is $T h_f^* := f$; so $T$ is represented by the identity map on the common $f$-parameterization. For a realization $s^* = ((x_i,x_i^*,y_i))_{i=1}^N$ and its projection $s$, we took the following two empirical risk functionals: $r(f) = -\frac{1}{N}\sum_{i=1}^N \log\Phi(y_i f(x_i))$, $r^*(h_f^*) = -\frac{1}{N}\sum_{i=1}^N \log\Phi\left(\frac{y_i f(x_i)}{q(x_i^*)}\right)$. As in Sec. 4.1, the probit log-loss is unbounded, so $\Delta Z$ is again used diagnostically. The PI scale flattens the likelihood of examples with $\lVert x_i^* \rVert$ close to zero and hence reduces their influence. As in Sec. 4.1, the Bayesian posterior coincides with the endpoint Gibbs measures precisely at $\lambda=N$, i.e., for any Borel set $B$ in $\mathcal{H}$, $\rho_{\mathrm{never}}(B) = \Pr(f \in B \mid X,y)$ and $\rho_{\mathrm{always}}(B) = \Pr(f \in B \mid X,X^*,y)$. The partition functions are marginal likelihoods

where $\pi(\mathrm{d}f) = \mathcal{N}(f;0,K)\,\mathrm{d}f$. Neither integral is available in closed form. We approximate both posteriors and both log evidences by expectation propagation (EP), using the same GP prior and convergence rule in the two endpoints, and compute $\Delta Z = \log Z_N^* - \log Z_N$. At test time, PI is absent and both models predict from the deployed latent function $f$ using $x$ alone.

We evaluate this construction on a noisy circular-boundary task in $\mathbb{R}^2$. With margin $m_i = x_{i,1}^2+x_{i,2}^2-r_0^2$, the clean label is $y_i^{\mathrm{true}} = \operatorname{sign}(m_i)$, while labels close to the boundary are corrupted during training. The privileged feature is a noisy observation $x_i^* = \lvert m_i\rvert+\xi_i^*$ of boundary proximity, where $\xi_i^*$ has noise scale $\sigma^*$; the per-point variance $q(x_i^*)$ is then computed with (30). In this example, we set $N=50$.

We compare the following two models: (1) GPC+ (Privileged Model): $\sigma_i^2 = \sigma_0 e^{-\lVert x_i^* \rVert/\tau}$; and (2) Baseline: $\sigma_i^2 \equiv0$. Both models are trained with the Gaussian kernel $k(x,y) = \sigma_f^2 \exp(-\lVert x-y\rVert^2/2\ell^2)$, where $\ell$ is the lengthscale (a.k.a. bandwidth) hyperparameter. Reliable PI assigns a large variance $\sigma_i^2(x_i^*)$ to examples near the noisy boundary and flattens their probit sites; increasing $\sigma^*$ progressively destroys this alignment. We compare the evidence gap $\Delta Z$ with the clean-test accuracy gain $\Delta_{\mathrm{acc}} := \mathrm{Acc}_{\mathrm{LUPI}}-\mathrm{Acc}_{\mathrm{never}}$.

Figure 3(a) shows that $\Delta Z$ and $\Delta_{\mathrm{acc}}$ decrease together and approach zero as the PI channel becomes uninformative for large noise levels. Both are also highly correlated as depicted in Figure fig:3b. Hence, the same training-time diagnostic $\Delta Z$ that predicts PI utility in the GMM also tracks its realized gain in a supervised, non-conjugate, function-space model.

Conclusion

We introduced an algorithm-agnostic PAC-Bayesian account of privileged information (PI) that separates its potential from its realization at deployment. The endpoint Gibbs measures turn the former into an exact partition-function comparison, while the deployment correction quantifies how much of this advantage survives the removal of PI. Two-sided certificates further provide sufficient conditions for a genuine separation of true risks, and the reduction to a fixed-feature subclass of the exponential family extends the analysis beyond canonical Gibbs posteriors. The GMM and privileged-noise GPC experiments show that the same training-time partition gap tracks realized gains in generative and discriminative settings. We highlight that characterizing the slope of this relationship turns the diagnostic from an ordering into a calibrated, quantitative predictor of the achievable gain per unit and remains an open direction for future work.

The present theory assumes data-independent priors and deployment maps, and the stated Catoni certificates require bounded losses; our log-likelihood experiments therefore use the partition gap diagnostically. Computing partition functions may also be difficult in large models, and favorable endpoint potential alone cannot guarantee a good deployment. Future work should develop unbounded-loss and data-dependent-prior extensions, scalable evidence estimators, and learning principles for deployment maps that retain PI-derived information. These directions would turn the proposed criterion from a model-selection diagnostic into a practical design tool for LUPI.

Proofs

Gibbs Posterior Minimization

Proof of Lemma 1. Since $\Phi_{\lambda/N}^{-1}$ is strictly increasing, minimizing $B_\lambda(\rho)$ is equivalent to minimizing $\lambda\langle r\rangle_\rho+\operatorname{KL}(\rho\Vert\pi)$. From $\frac{\mathrm{d}\rho_\lambda}{\mathrm{d}\pi}=e^{-\lambda r}/Z_\lambda$,

Therefore,

The right-hand side is uniquely minimized at $\rho=\rho_\lambda$, with value $-\log Z_\lambda$. Substitution into $B_\lambda$ proves the claim.

Endpoint Certificate Criterion

Proof of Thm. 2. The Catoni bound (3) applied to $\rho_{\mathrm{never}}$ with $r$ gives $\langle R\rangle_{\rho_{\mathrm{never}}}\le B_\lambda$ with probability $1-\epsilon$. Applied to $\rho_{\mathrm{always}}$ with $r^*$, it bounds $\langle R\rangle_{\rho_{\mathrm{always}}}$; hence $\langle R\rangle_{\rho_{\mathrm{always}}}\le B_\lambda^*$ with probability $1-\epsilon$. A union bound gives both simultaneously with probability $1-2\epsilon$.

Since $\Phi_{\lambda/N}^{-1}$ is strictly increasing, $B_\lambda^*<B_\lambda$ holds if and only if their arguments are ordered; the $\ln(1/\epsilon)/\lambda$ terms cancel, leaving $\Delta_r>\Delta_{\mathrm{KL}}/\lambda$. Substituting the Gibbs identities $\operatorname{KL}(\rho_{\mathrm{never}}\Vert\pi)=-\lambda\langle r\rangle_{\rho_{\mathrm{never}}}-\log Z_\lambda$ and $\operatorname{KL}(\rho_{\mathrm{always}}\Vert\pi^*)=-\lambda\langle r^*\rangle_{\rho_{\mathrm{always}}}-\log Z_\lambda^*$ yields

so the condition reduces to $Z_\lambda^*>Z_\lambda$ since $\lambda>0$.

Deployment-Certificate Criterion

Proof of Corollary 1. Since $\rho_{\mathrm{always}}\ll\pi^*$ and $\pi=T_{\sharp}\pi^*$, we have $\rho_{\mathrm{LUPI}}=T_{\sharp}\rho_{\mathrm{always}}\ll\pi$. Apply Thm. 1 on the privileged sample space with loss $(h,x,x^*,y)\mapsto\ell(h(x),y)$. Its empirical and population risks are $r$ and $R$, respectively. The PAC-Bayes event holds simultaneously for every posterior on $\mathcal{H}$, and therefore yields both inequalities in (17) without a union bound. Since $\Phi_{\lambda/N}^{-1}$ is strictly increasing, it suffices to compare the arguments of the two certificates; their confidence terms cancel. The Gibbs identities and (15) and (16) give

which proves the main claim. Finally, we apply the argument from the data-processing inequality to prove $\operatorname{KL}(\rho_{\mathrm{LUPI}}\Vert\pi)\le\operatorname{KL}(\rho_{\mathrm{always}}\Vert\pi^*)$. Let $\varphi(t)=t\log t$ be defined for $t>0$, which is a convex function. Considering $\pi=T_{\sharp}\pi^*$, $\rho_{\mathrm{LUPI}}=T_{\sharp}\rho_{\mathrm{always}}$ and the fact that

we have

Therefore, $\kappa_T \geq0$, which proves Eq. (18) and the equality case.

Catoni Lower Certificate

Proof of Lemma 2. Apply the Catoni upper bound to the complementary loss $\ell' = 1-\ell\in[0,1]$, whose posterior-averaged true and empirical risks are $1-\langle R\rangle_\rho$ and $1-\langle r\rangle_\rho$. With the same prior $\pi$, temperature $\lambda$, and confidence $\varepsilon$, this gives with probability $\geq1-\varepsilon$

and rearranging yields the claim.

Positive Gibbs-Envelope Certificate

Proof of Thm. 3. Apply Thm. 1 on $(\mathcal{H}^*,\mathcal{F}^*)$ to the PI loss, and apply Lemma 2 to the ordinary loss on $(\mathcal{H},\mathcal{F})$. A union bound proves Eq. (19). By definition, $B^*_\lambda=\Phi^{-1}_{\lambda/N}(a^*)$ and $L_\lambda(\rho_{\mathrm{never}})=1-\Phi^{-1}_{\lambda/N}(b)$. Hence, $B^*_\lambda<L_\lambda(\rho_{\mathrm{never}})$ iff

Substitution of the definition $\Phi^{-1}_{\lambda/N}(x)=(1-e^{-x\lambda/N})/(1-e^{-\lambda/N})$ proves Eq. (20). Combining the two valid certificates in Thm. 2 and Lemma 2 gives Eq. (21). For $x\geq0$, the inequality $1-e^{-cx}\leq cx$, $c>0$, implies $\Phi^{-1}_{\lambda/N}(x)\leq\frac{\lambda x}{N(1-e^{-\lambda/N})}$. Therefore, the condition $a^*+b<N(1-e^{-\lambda/N})/\lambda$ implies

Expanding $a^*+b$ proves Eq. (22).

Deployment True-Risk Separation

Proof of Corollary 2. As above, $\rho_{\mathrm{LUPI}}\ll\pi$. Apply Thm. 1 on the augmented sample space to the ordinary loss, and apply Lemma 2 there to the complementary loss. Both losses ignore $x^*$ and have population risks $R$ and $1 - R$ under the $(X,Y)$-marginal $\mathcal{D}$. The two PAC-Bayes events are simultaneous over the posterior, so they include $\rho_{\mathrm{LUPI}}$ and $\rho_{\mathrm{never}}$ even though the former depends on the full sample $S^*$. A union bound shows that both inequalities in the corollary hold simultaneously with probability at least $1 - 2\epsilon$.

By the definitions of $u$ and $b$, $B_\lambda(\rho_{\mathrm{LUPI}}) = \Phi^{-1}_{\lambda/N}(u)$ and $L_\lambda(\rho_{\mathrm{never}}) = 1 - \Phi^{-1}_{\lambda/N}(b)$. The same substitution used for the endpoint measures yields the equivalence stated in the corollary. On the valid event, certificate separation gives

Finally, the sufficient condition in the corollary is equivalent to $u + b < (1 - e^{-\lambda/N})/(\lambda/N)$. The inequality $\Phi_c^{-1}(x) \le cx/(1 - e^{-c})$ then proves the stated sufficient criterion.

Fixed-Feature Reduction

Proof of Lemma 3. Relative to $\pi$, the right-hand side of Eq. (27) has density proportional to

which is proportional to $\mathrm{d}\rho/\mathrm{d}\pi$ by Eq. (26). Both sides are probability measures, so their normalizing constants agree, and the measures are equal.

More Examples of the Deployment Map

In both validation studies in Sec. 4, the deployment map $T$ becomes the identity after hypotheses are identified by the shared parameter $\theta$ (GMM) or latent function $f$ (GPC+). Nontrivial maps arise when the PI model has degrees of freedom that must be removed or averaged out at deployment.

Many LUPI methods jointly learn a deployable branch $f_\theta:\mathcal{X}\to\mathbb{R}$ and a correcting or teacher branch $g_\eta:\mathcal{X}^*\to\mathbb{R}$ that accesses $x^*$ during training [16, 12, 9, 7], giving rise to the privileged hypothesis

Identifying the hypothesis with the parameter pair $(\theta,\eta)$, the deployment map is the projection

SVM+ (16). Given the realization $s^*=(x_i,x_i^*,y_i)$, SVM+ jointly learns the deployable classifier $f_{w,b}(x)=\langle w,x\rangle+b$ and a privileged correcting function $g_{w^*,b^*}(x^*)=\langle w^*,x^*\rangle+b^*$ by solving

The privileged branch predicts training-example difficulty but is unavailable at inference. Deployment is therefore the many-to-one projection

The PI branch is discarded after training, making $T$ generally many-to-one, which can strictly contract the KL divergence (see Eq. (16)), i.e., $\operatorname{KL}(\rho_{\mathrm{LUPI}}\Vert\pi)=\operatorname{KL}(T_{\sharp}\rho_{\mathrm{always}}\Vert T_{\sharp}\pi^*)<\operatorname{KL}(\rho_{\mathrm{always}}\Vert\pi^*)$.

Generalized distillation [12]. Consider $c$-class classification with $y_i \in\Delta^c$, and let $\sigma: \mathbb{R}^c \to\Delta^c$ denote the softmax map. Generalized distillation (GD) works with logit-valued hypotheses, $f_t$ and $f_s$ — known as the teacher and the student, $\mathcal{F}_t \subseteq\{f_t: \mathcal{U}^* \to\mathbb{R}^c\}$, $\mathcal{F}_s \subseteq\{f_s: \mathcal{X} \to\mathbb{R}^c\}$, where $\mathcal{U}^* = \mathcal{X}^*$ in the original formulation and may be $\mathcal{U}^* = \mathcal{X} \times\mathcal{X}^*$ in multimodal variants [7]. The corresponding predictive distributions are $\sigma\circ f_t$ and $\sigma\circ f_s$. Writing $x_i^* \in\mathcal{U}^*$ for the PI, the teacher is first learned from the privileged examples, $\widehat{f}_t \in\arg\min_{f \in\mathcal{F}_t} \frac{1}{N}\sum_{i=1}^{N}\ell(y_i,\sigma\circ f(x_i^*))+\Omega(\lVert f\rVert)$, where $\Omega$ is a regularizer. Define the teacher’s soft labels $s_i := \sigma(\widehat{f}_t(x_i^*)/\tau) \in\Delta^c$. The student $\widehat{f}_s$ is then learned on the ordinary inputs by balancing the hard labels with the teacher’s soft labels, i.e., $\widehat{f}_s \in\arg\min_{f \in\mathcal{F}_s} \frac{1}{N}\sum_{i=1}^{N}(1-w)\ell(y_i,\sigma\circ f(x_i))+w\ell(s_i,\sigma\circ f(x_i))$, where $w \in[0,1]$. To fit GD into our framework, we set

Accordingly, we treat the learning rule of GD as

Note that the learning rule $\mathcal{A}^*$ here is a frequentist/point estimator, rather than the Bayesian learner we define in Sec. 2.2. To analyze generalized distillation with our results, we need to implement Bayesian inference/estimation for it. Moreover, the deployment map is the data-independent projection

Modality-hallucination methods instantiate the same function-space viewpoint: the privileged network produces depth-derived logits or feature maps, while an RGB-input hallucination network is trained to reproduce them and is retained at test time [9, 7].

Transfer and marginalize (TRAM) [5]. Taking the same setup as in generalized distillation, TRAM jointly learns an $x$-only prediction head and a privileged prediction head over a shared feature space $\mathcal{Z}$. Let $\phi:\mathcal{X}\to\mathcal{Z}$ be the shared feature map, $\psi:\mathcal{Z}\times\mathcal{X}^*\to\mathcal{Z}^*$ the PI feature map, and define the logit-valued hypotheses

Their predictive distributions are respectively $\sigma\circ f_{\phi,\theta}$ and $\sigma\circ g_{\phi,\psi,\eta}$, where $\sigma$ is the softmax map. The corresponding PI hypothesis is therefore

Given the realization $s^* = ((x_i,x_i^*,y_i))_{i=1}^{N}$, TRAM jointly learns these two heads by solving

where $\operatorname{sg}$ is the stop-gradient operator: it is the identity during the forward pass but has zero derivative during backpropagation. Consequently, the first loss $\ell$ trains the deployable head $\theta$ without modifying $\phi$, whereas the privileged loss $\ell'$ trains $(\phi,\psi,\eta)$. Knowledge from PI is therefore transferred to deployment through the shared representation $\phi$, rather than through teacher soft labels as in generalized distillation. Identifying the learned PI hypothesis with $(\phi,\psi,\theta,\eta)$, deployment retains the shared features and the ordinary head:

Thus, $T$ discards the PI-specific parameters $(\psi,\eta)$ while retaining the PI-trained feature map $\phi$.

Examples of Fixed-Feature Distributions

We provide more examples of Definition 1 and state the conditions under which the representation is exact. Throughout, the priors and posteriors are assumed proper and to belong to $\mathcal{M}_{1}^{+}(\mathcal{H})$ under the metric fixed in Sec. 2.

Common likelihood identity. Let $\theta\in\mathcal{H}$ have prior $\pi$ and likelihood $p_{\theta}(y \mid x) > 0$. With $r_{S}(\theta) = -N^{-1}\sum_i \log p_{\theta}(y_i \mid x_i)$, Bayes’ rule gives

Thus every proper Bayesian posterior based on an i.i.d. likelihood has the fixed-feature density with $\lambda= N$ and $\mu= 0$, provided its marginal likelihood is finite and nonzero. The following paragraphs verify the named families and specify what the hypothesis is in each case.

Bayesian linear regression. Let $h_{\beta}(x) = x^{\mathsf{T}}\beta$, let $y \mid\beta\sim\mathcal{N}(X\beta,\sigma^{2}I_{N})$ with fixed $\sigma^{2} > 0$, and take the data-independent prior $\pi= \mathcal{N}(m_{0},V_{0})$. Up to an additive constant independent of $\beta$,

Completing the square gives $V_{S}^{-1} = V_{0}^{-1} + X^{\mathsf{T}}X/\sigma^{2}$ and $m_{S} = V_{S}(V_{0}^{-1}m_{0} + X^{\mathsf{T}}y/\sigma^{2})$, so $\rho= \mathcal{N}(m_{S},V_{S})$. Hence the exact posterior, not merely its mean, has the required density. Unknown noise or hierarchical parameters can be included in $\theta$ and $\pi$; the common likelihood identity remains valid.

Generalized linear models. For a canonical GLM, $p_{\beta}(y_i \mid x_i) = \exp(y_i x_i^{\mathsf{T}}\beta- b(x_i^{\mathsf{T}}\beta) + c(y_i))$. Consequently,

No conjugacy is needed: logistic, Poisson, and other Bayesian GLMs satisfy the density identity whenever the posterior is proper. A noncanonical link is covered by defining $r$ from its actual negative log-likelihood.

Gaussian-process regression. Let $\mathcal{H}$ be a Polish path space carrying the data-independent Radon law $\pi= \mathcal{GP}(m,k)$ (for example, $C(K)$ under the usual sample-continuity conditions on a compact input domain $K$), and let $Y_i \mid f \sim\mathcal{N}(f(x_i),\sigma^{2})$ independently. Then

The likelihood depends on finitely many measurable evaluations of $f$, so the Radon–Nikodym derivative is well defined; Gaussian conditioning gives the usual GP posterior. Its posterior mean agrees with kernel ridge after the standard identification of the noise-to-prior variance ratio with the ridge parameter.

Gaussian-process classification. With the same GP prior and a Bernoulli likelihood $p(y_i \mid f) = g(y_i f(x_i))$, where $g$ is, for example, the logistic CDF or $\Phi$, set $r(f) = -N^{-1} \sum_i \log g(y_i f(x_i))$. The exact posterior again satisfies $\mathrm{d}\rho/ \mathrm{d}\pi\propto e^{-N r_s}$. It is generally non-Gaussian, which affects computation but not the density identity. Laplace or expectation-propagation approximations are separate Gaussian approximations; they are not automatically exact fixed-feature posteriors.

Common quadratic calculation. Let $h_w(x) = \phi(x)^\top w$, $w \in\mathbb{R}^d$, let $F$ have rows $\phi(x_i)^\top$, and set $r(w) = \lVert y - Fw \rVert^2/(2N)$. For fixed $Q \succ0$ and $\eta> 0$, choose $\pi_Q = \mathcal{N}(0,(\lambda\eta Q)^{-1})$. Put $G = F^\top F/N + \eta Q$ and $m = G^{-1}F^\top y/N$. Then

The mean and mode both minimize $r(w) + \eta w^\top Qw/2$. This calculation is the common finite-dimensional Gibbs law underlying the next three model families.

Kernel ridge regression. Define the fixed kernel $k_Q(x,x') = \phi(x)^\top Q^{-1}\phi(x')$ and its training Gram matrix $K = FQ^{-1}F^\top$. The Gaussian mean above satisfies

which is exactly the vector of kernel-ridge fitted values. Thus kernel ridge is not itself a probability measure; it is the mean and mode of this quadratic Gibbs posterior. The construction is a valid use of Lemma 3 when $\phi,Q,\eta$, and the Gaussian prior are fixed independently of the PAC-Bayes sample. General fixed kernels are equivalently represented through their GP priors on functions.

Fixed-shape Gaussian variational posteriors. Let $r(w) = w^\top A_s w/2 - b_s^\top w + c_s$, let $\pi= \mathcal{N}(0,P_0^{-1})$, and let $q_s = \mathcal{N}(m_s,\Sigma_s)$. Direct comparison of Gaussian log-densities shows that $q_s$ is fixed-feature exactly when

for data-independent $M$ and $a$; these residual natural parameters form the fixed feature (quadratic and linear sufficient statistics, respectively). These identities are necessary and sufficient in the nonsingular Gaussian case. In particular, a prescribed covariance $\Sigma_s \equiv\Sigma$ is covered when $A_s$ is fixed and the mean natural parameter obeys $\Sigma^{-1}m_s - \lambda b_s = a$. An arbitrary covariance optimized from the same sample is not covered unless its variation is exactly the contribution $\lambda A_s$ already present in the empirical risk.

Frozen last-layer networks. Let a feature extractor $\phi_\omega: \mathcal{X} \to\mathbb{R}^d$ be fixed independently of the PAC-Bayes sample and train only $h_w(x) = w^\top\phi_\omega(x)$ with squared loss and a Gaussian prior on $w$. Taking $F_{i\cdot} = \phi_\omega(x_i)^\top$ and $Q = I$ in the common quadratic calculation yields a Gaussian Gibbs posterior whose mean and mode are the ridge-trained last layer. If $\omega$ was trained on the same sample and then treated as fixed, the required data independence fails; sample splitting or a data-dependent-prior theorem is then needed.

NTK-linearized networks. Fix an initialization $\vartheta_0$ independently of the sample and linearize $f_\vartheta$ as $f_{\vartheta_0}(x) + J_{\vartheta_0}(x)^\top u$, where $u = \vartheta- \vartheta_0$. After subtracting the fixed offset, this becomes a linear model with design rows $J_{\vartheta_0}(x_i)^\top$. A Gaussian prior on $u$ and squared loss therefore give the preceding Gaussian Gibbs law, while the Gram matrix $JJ^\top$ is the empirical neural tangent kernel. Its mean and mode coincide with NTK kernel ridge. The claim does not extend to a changing Jacobian or to a linearization point fitted on the same sample without an additional data-dependence argument.

Gaussian Mixture Details

We give the construction underlying Sec. 4.1. For $x \in\mathbb{R}^{d}$, the data-generating density is

with $\theta= (w, \mu_{1}, \mu_{2}, \sigma_{1}^{2}, \sigma_{2}^{2}) \in\Theta$. The goal is posterior inference for $\theta$ and evaluation of the deployed posterior predictive, not prediction of the latent assignments themselves. PI consists of the generating assignments $z_i \in\{1,2\}$ observed during training. We write $w_{1} = w, w_{2} = 1-w$.

In both cases we use independent conjugate priors over the mixing weight and the component parameters,

with

where

with mean $\mathbb{E}[\sigma^{2}] = \beta/(\alpha-1)$ for $\alpha> 1$. The $\alpha$ parameter controls the concentration and $\beta$ sets the scale. Coupling $\mu_{k}$ to $\sigma_{k}^{2}$ through the shared factor $\sigma_{k}^{2}/\kappa$ — rather than giving $\mu_{k}$ an independent prior — is what makes $\pi(\mu_{k},\sigma_{k}^{2})$ conjugate to the Gaussian likelihood: the posterior is again Normal-Inverse-Gamma with closed-form updates.

Case 1: without privileged information. Given a realization of the random sample $s = (x_{i})_{i=1}^{N}$, the posterior is

Expanding the product of mixtures yields an exact sum over $2^{N}$ assignments, which is computationally prohibitive as $N$ grows. We therefore sample the never-PI posterior by Metropolis–Hastings: in each step, a proposal $\theta'$ is accepted with probability $\min\left(1,\frac{\pi(\theta')\Pr(S=s\mid\theta')}{\pi(\theta)\Pr(S=s\mid\theta)}\right)$. The chain has stationary distribution $p(\theta\mid S=s)$, and by the ergodic theorem the empirical distribution converges to the posterior.

Case 2: with privileged information. Revealing $z_i$ gives $s^{*} = \{(x_i,z_i)\}_{i=1}^{N}$ and collapses the mixture into a conditional likelihood,

Each point now contributes to exactly one component, so the likelihood is a single product rather than a product of sums. With $n_k = \#\{i : z_i = k\}$, the posterior separates into three independent parts:

The posterior factorizes completely, enabling exact ancestral sampling:

The updated hyperparameters $(\kappa'_k,m'_k,\alpha'_k,\beta'_k)$ follow in closed form by completing the square in the Gaussian exponent against the NIG prior, after decomposing $\sum_{i:z_i=k}\lVert x_i-\mu_k\rVert^2 = S_k+n_k\lVert\mu_k-\bar{x}_k\rVert^2$ about the component sample mean $\bar{x}_k$. This yields exact i.i.d. posterior draws without MCMC.

Both configurations are represented on the common parameter space $\Theta$; PI enters only through the loss, so the deployment map is the identity, $T=\operatorname{id}_{\Theta}$, and consequently $\kappa_T=0$ in Corollary 1. The pointwise negative log-likelihood loss functions are

where $w_1=w$, $w_2=1-w$, and the empirical risks $r,r^*$ are the sample averages of $\ell$ and $\ell^*$, respectively. The partition functions are evaluated by Monte Carlo over the prior,

and analogously for $Z_\lambda^*$ with $r^*$, giving $\Delta Z=\log Z_\lambda^*-\log Z_\lambda$. Because the loss function is a negative log-likelihood and hence unbounded, the Catoni certificate applies to a bounded surrogate; we only use $\Delta Z$ here as a diagnostic quantity, as discussed in Sec. 4.1.

Given the posteriors $\rho_{\mathrm{never}}$ and $\rho_{\mathrm{LUPI}}$, we take the Bayesian predictive distribution for the probability density at $x$:

We compare the predictive distribution — an average over the posterior — to a baseline of the point-estimate maximum a posteriori (MAP). A MAP estimate $\hat{\theta}_{\mathrm{MAP}}$ is a point estimator, obtained by maximizing the unnormalized posterior measure:

A MAP estimate $\hat{\theta}_{\mathrm{MAP}}$ predicts with the plug-in density $p_{\hat{\theta}_{\mathrm{MAP}}}(x)$, i.e., the most probable parameter value from the posterior. However, for small $N$, the posterior is often multi-modal, and a MAP only picks a single mode thereof, which leads to an overconfident prediction.