betaFeedback

WorksComputer Science

Fisher-Glass Learning: Classification Bounds and Subpopulation Audits

Abstract

Accurate classifiers can fail when background cues change without altering the correct labels. Predictive performance alone does not quantify how much task information remains after accounting for such nuisance variation. Here we extend the Fisher-glass framework to connect nuisance-adjusted information with local classification difficulty and subpopulation auditing. We derive lower bounds on local minimax excess classification risk under explicit distinguishability and decision-separation conditions, alongside statistical audits accounting for estimation error and regularization. On Waterbirds, a feature-based diagnostic of task-background alignment selects an accurately classified group, with selection frozen before held-out evaluation. Across five training seeds, background replacements preserving the extracted bird foreground and labels yield 93.2% accuracy when background type is preserved and 40.6% when it changes, a controlled difference of 52.6 percentage points. Separately, search using task labels and a fixed nuisance direction identifies an independently evaluated population with 98.8% membership-weighted accuracy and low estimated projected predictive Fisher information. Training and acquisition experiments show that information gains and predictive gains can diverge. These findings support joint assessment of predictive performance, information under declared nuisance models and controlled sensitivity when diagnosing vulnerability and evaluating model improvements.

Introduction

A classifier can predict observed labels accurately while the data weakly distinguish a specified task effect from admissible nuisance variation. This ambiguity matters for classification when statistically similar alternatives prescribe different labels. Autonomous bird monitoring provides a practical motivation: species classifiers operate under variable illumination and occlusion [1]. We ask when it creates a learning obstacle, and how to audit the remaining task information on predefined and learned populations. Group-robust learning and error-based reweighting emphasize subpopulation performance [2, 3]; our objective is to establish which information claims the data support and assess their relevance through controlled nuisance interventions (Figure 1).

Figure 1. Background sensitivity despite high control accuracy. (A,C) A matched Waterbirds image pair preserves the extracted bird foreground and label while changing background type. For the selected waterbird/water group, mean accuracy across five training seeds falls from 93.2% with same-type backgrounds to 40.6% with opposite-type backgrounds, a drop of 52.6 percentage points. (B) Schematic score geometry: projection removes the nuisance-aligned component of the task score, leaving a small residual.

Efficient-score projection removes task variation that nuisance scores can reproduce [4, 5]. The Fisher-glass framework of Mansouri and Arai [6] connects this geometry to inverse-information tails and a population certificate. We develop its classification consequence: decision separation turns limited distinguishability into excess-risk bounds. A worked regression family shows that constant raw information at a reference parameter can conceal an effective sample size set by the mass near the decision boundary. Statistical corrections then determine which population information statements survive estimation error, regularization, and adaptive selection.

On Waterbirds, a frozen feature-based rule selects a group with 92.0% source accuracy and a 52.6-point controlled drop between same-type and opposite-type background replacements. The information, accuracy, and negative log-likelihood (NLL) selectors identify the same eligible group, whose controlled accuracy drop quantifies background sensitivity. Within that group, nuisance-derived coordinates improve species-held-out prediction of intervention failures beyond a specified confidence baseline. A separate search discovers an accurate population with low predictive information; its sensitivity to interventions remains untested.

Our contributions are:

  • A classification consequence. We derive local excess-risk bounds with explicit decision separation and radius, and exhibit a classification family with matching minimax rates and an interpretable loss of effective sample size.

  • Population information audits. We account for estimation error, ridge bias, membership normalization, and support correction, and evaluate frozen learned slices independently. The audit distinguishes supported information floors from uncertain or adaptively selected estimates.

  • Behavioral evaluation. Controlled interventions and confidence-baseline comparisons test the relevance of nuisance-derived measurements. Training and acquisition evaluate whether information gains correspond to predictive gains, which need not occur together.

Task–Nuisance Geometry and Information Collapse

Let ee index an observed group or latent subpopulation, and let Π\Pi be the declared distribution over these environments. The audit fixes the task contrast, admissible nuisance variations, and common task-coordinate normalization. Information describes the task distinction remaining after nuisance adjustment.

Task Information After Nuisance Profiling

Consider a local model Pθ,μ,eP_{\theta,\mu,e} with task parameter θ\theta and nuisance parameter μ\mu. A likelihood score measures the local change in log likelihood as a parameter varies. At Pe=P0,0,eP_e = P_{0,0,e}, assume differentiability in quadratic mean, with centered, square-integrable task-score vector sT,es_{T,e} and nuisance-score vector sN,es_{N,e}. Let TN,e\mathcal{T}_{N,e} be the closed span of admissible nuisance scores in L2(Pe)L^2(P_e), with inner product ⟨f,g⟩e=EPe[fg]\langle f,g\rangle_e=\mathbb{E}_{P_e}[fg], and let ΠTN,e\Pi_{TN,e} denote orthogonal projection onto that span. The efficient task score and its information are

sT,eeff=sT,e−ΠTN,esT,e,s^{\mathrm{eff}}_{T,e}=s_{T,e}-\Pi_{TN,e}s_{T,e},
Fe:=FT∣N,eeff=EPe[sT,eeff(sT,eeff)⊤]=JTT,e−JTN,eJNN,e−1JNT,e.(1)F_e:=F^{\mathrm{eff}}_{T|N,e}=\mathbb{E}_{P_e}\left[s^{\mathrm{eff}}_{T,e}(s^{\mathrm{eff}}_{T,e})^\top\right]=J_{TT,e}-J_{TN,e}J_{NN,e}^{-1}J_{NT,e}. \tag*{(1)}

The projection acts componentwise. The Schur expression applies when finite nuisance scores span TN,e\mathcal{T}_{N,e} and JNN,e≻0J_{NN,e}\succ0, with JAB,e=EPe[sA,esB,e⊤]J_{AB,e}=\mathbb{E}_{P_e}[s_{A,e}s_{B,e}^{\top}] [4, 5, 7]. Write λe=λmin⁡(Fe)\lambda_e=\lambda_{\min}(F_e) for the weakest task information. A zero eigenvalue identifies a task direction contained in the nuisance span.

For scalar task and nuisance scores with positive raw information, their Fisher correlation ρe=JTN,e/JTT,eJNN,e\rho_e=J_{TN,e}/\sqrt{J_{TT,e}J_{NN,e}} gives

Fe=JTT,e(1−ρe2),FeJTT,e=1−ρe2.(2)F_e=J_{TT,e}(1-\rho_e^2),\qquad\frac{F_e}{J_{TT,e}}=1-\rho_e^2. \tag*{(2)}

Absolute information measures remaining curvature in the declared coordinates. The fraction measures the proportion retained, lies in [0,1][0,1], and is invariant to nonzero scalar rescaling of either score. It compares alignment across slices with different raw information; the prospective selection statistic uses a ridge-regularized feature–Schur analogue with fixed coordinate scales. The Fisher-Glass certificate summarizes absolute information across environments.

Task–Nuisance Alignment and Loss–Information Separation

Let be,reb_e,r_e be centered, orthogonal, unit-norm scores, with nuisance span span⁡{be}\operatorname{span}\{b_e\}. For δe≥0\delta_e\ge0, define the task score ae=be+δerea_e=b_e+\delta_er_e. Then

ae=be+δere,JTT,e=1+δe2,Fe=δe2,FeJTT,e=δe21+δe2.(3)a_e=b_e+\delta_er_e,\qquad J_{TT,e}=1+\delta_e^2,\qquad F_e=\delta_e^2,\qquad\frac{F_e}{J_{TT,e}}=\frac{\delta_e^2}{1+\delta_e^2}. \tag*{(3)}

Projection removes beb_e, leaving δere\delta_er_e; raw information stays at least one as efficient information vanishes. § 5.2 illustrates low observed loss alongside the corresponding feature-moment geometry. A classification cost additionally requires different optimal decisions, which the next section makes explicit. The inherited tail mechanism is illustrated in Appendix Figure A.1.

Classification Difficulty and Population Information

We study the statistical consequences of nuisance projection at two scales. Within an environment, local classification bounds and a worked family connect weak task information to learning difficulty under explicit distinguishability and decision-separation conditions. Across environments, geometry and probability mass near Fisher-null sets govern inverse-information tails, motivating a certificate of population information coverage.

Efficient Information and Local Classification Difficulty

Parameter ambiguity need not imply classification difficulty: probabilities 0.90 and 0.95 prescribe the same label. A lower bound requires statistically similar alternatives whose optimal decisions disagree on enough probability mass. Define classification risk RP(g)=P{g(X)≠Y}R_P(g)=P\{g(X)\ne Y\}, Bayes risk RP⋆=inf⁡gRP(g)R_P^\star=\inf_g R_P(g), and excess risk εP(g)=RP(g)−RP⋆\varepsilon_P(g)=R_P(g)-R_P^\star. For a fixed local family, let Re(n)\mathfrak{R}_e(n) denote minimax expected excess classification risk based on nn independent observations from environment ee. Write KL⁡(P∥Q)\operatorname{KL}(P\Vert Q) for the Kullback–Leibler divergence.

Theorem 1 (Local classification difficulty). Fix λe>0\lambda_e>0 and a unit weakest efficient-information direction ueu_e. For every 0<t≤t0,e0<t\le t_{0,e}, suppose the local family contains Pe,+t,Pe,−tP_{e,+t},P_{e,-t} with task coordinates ±tue\pm tu_e and nuisance coordinates allowed to vary. Assume, for constants Ke,ce,t0,e>0K_e,c_e,t_{0,e}>0 and q>0q>0,

KL⁡(Pe,+t∥Pe,−t)≤Ket2λe,inf⁡g{εPe,+t(g)+εPe,−t(g)}≥cetq.(4)\operatorname{KL}(P_{e,+t}\Vert P_{e,-t})\le K_et^2\lambda_e,\qquad\inf_g\{\varepsilon_{P_{e,+t}}(g)+\varepsilon_{P_{e,-t}}(g)\}\ge c_et^q. \tag*{(4)}

Then, for a universal constant c0>0c_0>0,

Re(n)≥c0cemin⁡{t0,eq,(nKeλe)−q/2}.(5)\mathfrak{R}_e(n)\ge c_0c_e\min\left\{t_{0,e}^q,(nK_e\lambda_e)^{-q/2}\right\}. \tag*{(5)}

The KL bound limits distinguishability; decision separation supplies the classification cost. Both must hold over radius t0,et_{0,e}. With quadratic separation, the interior rate is ce/(nKeλe)c_e/(nK_e\lambda_e), capped by the local decision scale. Appendix § 11.§ 3 proves the testing reduction; the next example calculates both conditions and a matching upper bound.

A Worked Classification Mechanism

Let X∼Unif⁡[−1,1]X \sim\operatorname{Unif}[-1,1], 0<δ≤1/20 < \delta\le1/2, ∣θ∣,∣μ∣≤δ/8|\theta|,|\mu| \le\delta/8, and

ηθ,μ(x):=Pθ,μ(Y=1∣X=x)=12+x4+θ+μ1{∣x∣>δ}.(6)\eta_{\theta,\mu}(x) := P_{\theta,\mu}(Y=1 \mid X=x) = \frac{1}{2}+\frac{x}{4}+\theta+\mu1\{|x|>\delta\}. \tag*{(6)}

Here δ\delta is boundary-region half-width and probability mass. The nuisance reproduces a task shift outside that region; the Bayes rule is gθ∗(x)=1{x≥−4θ}g_\theta^*(x)=1\{x\ge-4\theta\} throughout the family. At (θ,μ)=(0,0)(\theta,\mu)=(0,0),

JTT=4log⁡3,Fδ=4log⁡2+δ2−δ≍δ.(7)J_{TT}=4\log3,\qquad F_\delta=4\log\frac{2+\delta}{2-\delta}\asymp\delta. \tag*{(7)}

References

  1. [1]El Mustapha Mansouri. Autonomous AI bird feeder for backyard biodiversity monitoring, 2025. URL https://arxiv.org/abs/2508.09398.
  2. [2]Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=ryxGuJrFvS.arxiv.org/abs/1911.08731
  3. [3]Evan Z. Liu, Behzad Haghgoo, Annie S. Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6781–6792. PMLR, 2021. URL https://proceedings.mlr.press/v139/liu21f.html.arxiv.org/abs/2107.09044
  4. [4]Peter J. Bickel, Chris A. J. Klaassen, Ya’acov Ritov, and Jon A. Wellner. Efficient and Adaptive Estimation for Semiparametric Models. Johns Hopkins University Press, 1993.
  5. [5]Anastasios A. Tsiatis. Semiparametric Theory and Missing Data. Springer Series in Statistics. Springer, 2006. doi: 10.1007/0-387-37345-4.DOI
  6. [6]El Mustapha Mansouri and Keigo Arai. Fisher Glasses: Tail-certified quantum metrology in quenched environments. arXiv preprint arXiv:2607.01085v1, 2026. URL https://arxiv.org/abs/2607.01085v1.
  7. [7]D. R. Cox and N. Reid. Parameter orthogonality and approximate conditional inference. Journal of the Royal Statistical Society: Series B (Methodological), 49(1):1–18, 1987. doi: 10.1111/j.2517-6161.1987.tb01422.x.DOI
  8. [8]R. Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. The Journal of Risk, 2(3):21–41, 2000. doi: 10.21314/JOR.2000.038.DOI
  9. [9]R. Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distributions. Journal of Banking & Finance, 26(7):1443–1471, 2002. doi: 10.1016/S0378-4266(02)00271-6.DOI
  10. [10]Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for MobileNetV3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1314–1324, 2019. doi: 10.1109/ICCV.2019.00140. URL https://openaccess.thecvf.com/content_ICCV_2019/html/Howard_Searching_for_MobileNetV3_ICCV_2019_paper.html.DOI
  11. [11]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 2021. URL https://proceedings.mlr.press/v139/radford21a.html.arxiv.org/abs/2103.00020
  12. [12]Whitney K. Newey. Semiparametric efficiency bounds. Journal of Applied Econometrics, 5(2):99–135, 1990. doi: 10.1002/jae.3950050202.DOI
  13. [13]Lucien Le Cam. Asymptotic Methods in Statistical Decision Theory. Springer Series in Statistics. Springer, 1986. doi: 10.1007/978-1-4612-4946-7.
  14. [14]Alexandre B. Tsybakov. Introduction to Nonparametric Estimation. Springer Series in Statistics. Springer, 2009. doi: 10.1007/b13794.DOI
  15. [15]Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019. doi: 10.48550/arXiv.1907.02893.DOI
  16. [16]Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 18347–18377. PMLR, 2022. URL https://proceedings.mlr.press/v162/rame22a.html.arxiv.org/abs/2109.02934
  17. [17]Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM Conference on Health, Inference, and Learning, pages 151–159. ACM, 2020. doi: 10.1145/3368555.3384468.
  18. [18]Nimit S. Sohoni, Jared A. Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. In Advances in Neural Information Processing Systems, volume 33, pages 19339–19352. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/e0688d13958a19e087e123148555e4b4-Abstract.html.arxiv.org/abs/2011.12945
  19. [19]Bhargavi Paranjape, Pradeep Dasigi, Vivek Srikumar, Luke Zettlemoyer, and Hannaneh Hajishirzi. AGRO: Adversarial discovery of error-prone groups for robust optimization. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=IrzkT99fDJH.arxiv.org/abs/2212.00921
  20. [20]Friedrich Pukelsheim. Optimal Design of Experiments, volume 50 of Classics in Applied Mathematics. SIAM, 2006. doi: 10.1137/1.9780898719109.DOI
  21. [21]Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham M. Kakade. Gone fishing: Neural active learning with Fisher embeddings. In Advances in Neural Information Processing Systems, volume 34, pages 8927–8939. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/hash/4afe044911ed2c247005912512ace23b-Abstract.html.arxiv.org/abs/2106.09675
  22. [22]Jean-Yves Audibert and Alexandre B. Tsybakov. Fast learning rates for plug-in classifiers. The Annals of Statistics, 35(2):608–633, 2007. doi: 10.1214/009053606000001217.DOI
  23. [23]Joel A. Tropp. User-friendly tail bounds for sums of random matrices. Foundations of Computational Mathematics, 12(4):389–434, 2012. doi: 10.1007/s10208-011-9099-z.DOI
  24. [24]Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115:211–252, 2015. doi: 10.1007/s11263-015-0816-y.DOI
  25. [25]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330. PMLR, 2017. URL https://proceedings.mlr.press/v70/guo17a.html.arxiv.org/abs/1706.04599
  26. [26]Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. URL https://www.vision.caltech.edu/datasets/cub_200_2011/.
  27. [27]Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452–1464, 2018. doi: 10.1109/TPAMI.2017.2723009. URL https://doi.org/10.1109/TPAMI.2017.2723009.
  28. [28]Alexandru Telea. An image inpainting technique based on the fast marching method. Journal of Graphics Tools, 9(1):25–36, 2004. doi: 10.1080/10867651.2004.10487596. URL https://doi.org/10.1080/10867651.2004.10487596.
  29. [29]Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, pages 491–500. ACM, 2019. doi: 10.1145/3308560.3317593.
  30. [30]Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5637–5664. PMLR, 2021. URL https://proceedings.mlr.press/v139/koh21a.html.arxiv.org/abs/2012.07421

Paper details

vlnc:W_ph8hrzxn
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents