betaFeedback

WorksComputer Science

AI-Assisted Classification under Correlation Neglect and Trust

Abstract

When does an AI assistant improve human decision-making, and when does it hurt? We argue that the answer depends not just on AI accuracy but also on how much its evidence overlaps with the human's and how the human combines the two. A worker makes a binary decision by observing her own signal and receiving a recommendation from an AI. We develop a human behavioral model with two primitives: her awareness of the overlap and her trust in the AI. Our analysis gives a closed-form phase diagram over AI capability and overlap, with regions of complementarity, impairment, and automation. A worker who neglects the overlap (correlation neglect) can be made less accurate by a more capable AI, double-counting the shared evidence. Trust reshapes the regions - undertrust may compensate for correlation neglect, so that the trust level maximizing accuracy depends on the deployment parameters. Empirically, we identify the two model parameters by pairing the same workers with different AIs, and break the accuracy loss into a correlation-neglect cost and an undertrust cost, one tied to reducing overlap on the AI side, the other to calibrating trust on the human side. On ChatBench, AI assistance raises accuracy by 21 points, yet workers apply only about a quarter of the Bayes-rational weight to AI information; GPT-4o falls in the automation region and Llama-3.1-8b stays at the complementarity boundary; and calibrating trust recovers about 2.6 points for GPT-4o but almost nothing for Llama-3.1-8b.

Introduction

Empirical evaluations of human-AI teams reach very different conclusions. Some studies report large accuracy gains from AI assistance [18, 17], while many find no improvement over the unaided human, or outright degradation: a more capable AI can hurt team performance [5], AI explanations need not help [6], AI predictions may not improve expert radiologists [1], and a meta-analysis across tasks and systems finds that human-AI teams on average perform worse [19]. We argue that whether assistance helps turns not on the AI’s accuracy alone, but also on how much its evidence overlaps with the human’s and how the human combines it with her own. For example, when a worker treats overlapping evidence as if it were independent, she counts the shared part twice, a bias documented as correlation neglect in belief formation [10, 11], and the team can end up less accurate than the worker alone. This paper constructs a model built on these factors, uses it to characterize when AI assistance helps and when it hurts, and develops an empirical estimation strategy that exploits variation of outputs obtained by pairing the same workers with different AIs.

We study a binary classification problem in which the human and the AI each hold evidence about the state, summarized by their log-likelihood ratios ℓH\ell_H and ℓA\ell_A, and the worker decides by the sign of a weighted sum wHℓH+wAℓAw_H\ell_H + w_A\ell_A. The model has two parts. In the observation model, each agent’s evidence is a collection of cues, and the key primitive is the overlap coefficient λ\lambda, the share of the AI’s cues the human already holds. In the behavioral model, the worker combines the two signals through correlation awareness α^∈[0,1]\hat{\alpha} \in[0,1], representing how well she accounts for that overlap, and trust β^>0\hat{\beta} > 0, how much weight she puts on the AI information. Two benchmarks sit inside this family. The Bayes-rational worker (α^=1\hat{\alpha} = 1, β^=1\hat{\beta} = 1) de-correlates and weights optimally, while the correlation-neglect worker (α^=0\hat{\alpha} = 0) treats the two signals as independent. Overlap is a property of the information, awareness and trust are properties of the worker, and together they determine the team’s accuracy.

Our first results compare team performance against human and AI alone. Under the Bayes-rational benchmark the team is at least as accurate as either source alone, and strictly more accurate unless the AI’s evidence already contains the human’s. The impairment and automation outcomes seen in the literature, outside that case, must then come from the worker’s behavior rather than from the information structure. Away from that benchmark, we obtain a closed-form phase diagram over AI capability τA\tau_A and overlap λ\lambda with three regions, complementarity where the team is most accurate, impairment where the unaided human is, and automation where the AI alone is. Two of its features run against intuition. The first is that a more capable AI can make the team less accurate. When overlap is high (λ>1−β^/2\lambda> 1-\hat{\beta}/2), a stronger AI only gives a correlation-neglecting worker more evidence to double-count, and that can lower accuracy. The second is that undertrust is not simply an error to be corrected. Reducing trust shrinks the impairment region but brings automation on at lower capability, so the trust level that maximizes team accuracy is not always β^=1\hat{\beta} = 1 but depends on the deployment parameters.

Our second set of results turns the model into a measurement tool. A Bayes-rational worker’s weight on the AI changes with overlap, while a correlation-neglecting worker’s does not, and this difference lets us estimate the parameters by pairing the same workers with two AIs of different overlap. Trust β^\hat{\beta} is sharply identified, while awareness α^\hat{\alpha} is only weakly identified because the overlap coefficient in our experiment is relatively small. We then split the worker’s accuracy loss against the Bayes-rational benchmark into two parts that add up exactly at any level of awareness, a correlation-neglect cost from misperceiving the overlap and an undertrust cost from underweighting the AI. Each is tied to a distinct lever on opposite sides of the information-behavior divide: reducing overlap on the AI side by training it to supply what the human lacks, and calibrating trust on the human side.

We examine the framework in the context of ChatBench [8], where 643 workers answer multiple-choice questions paired with either GPT-4o or Llama-3.1-8b. AI assistance raises accuracy by 21 percentage points, yet workers put only about a quarter of the Bayes-rational weight on the AI (β^=0.285\hat{\beta} = 0.285). GPT-4o is more capable and overlaps the human less, so it lies in the automation regime, where the AI alone beats the team; Llama-3.1-8b is weaker and overlaps more, so it sits at the boundary between complementarity and automation. The worker’s behavior is nearly the same in both, so it is the difference in AI capability between the two deployments that shifts the deployment from one regime to the other, as the framework predicts. The cost of undertrust is therefore uneven. We simulate a trust-calibrated decision rule, which recovers about 2.6 points of accuracy for the strong AI and moves the team toward the boundary, but almost nothing for the weak AI.

The two sides of the information-behavior divide provide two design levers. On the human side, trust should be set to the deployment’s accuracy-maximizing level, not reflexively to β^=1\hat{\beta} = 1, since some undertrust is protective in the impairment region. On the AI side, reducing overlap by training the AI to supply what the human lacks rather than what she already knows can raise team accuracy even without raising the AI’s own accuracy. The phase diagram also says when to show the AI at all: withhold its signal when the AI is too weak, provide it in the complementarity band, and automate once the AI is strong enough.

Our work sits among several literatures but differs in what it treats as endogenous. Studies of optimal human-AI collaboration through sufficient statistics [2] leave out the conditionally dependent signals that are central here. Work on reducing overreliance [7] and on testing whether AI adds to human expertise [3, 4] takes the intervention or the test as given, whereas we characterize the best deployment as a function of capability, overlap, and behavior. Closest in spirit, Guo et al. [ [12] define over- and under-reliance from the human’s adoption or switch rates against a Bayesian benchmark; our two parameters give a micro-foundation for their two sources of loss, trust for their reliance loss and correlation awareness for their discrimination loss, with a model of a worker who combines a continuous AI signal with her own rather than choosing per instance between the two recommendations. Other work on combining expert information [9, 16] and learning-to-defer [14, 15, 20, 21] takes the human’s decision rule as exogenous, whereas we derive it from behavioral primitives. Section 2 introduces the model, Section 3 develops the phase diagram and welfare decomposition, and Section 4 presents the empirical framework and results.

Model

A worker forms a tentative answer to a question, sees an AI assistant’s recommendation, and decides whether to revise. The evidence available about a question is a collection of cues, atomic units of evidence about the underlying answer that an agent might or might not access. For a question on classical mechanics, the cues are the relevant physical principles, the diagnostic features of the answer choices, and the patterns from prior similar problems. If a human and an AI both lean on the standard textbook derivation, there will be substantial overlap in cues whereas if they each approach the question through different reasoning paths there will be little overlap.

Formally, a binary state Y∈{0,1}Y \in\{0, 1\} is drawn from the symmetric prior P(Y=1)=P(Y=0)=1/2P(Y = 1) = P(Y = 0) = 1/2.1 There is a finite collection of NN cues {X1,…,XN}\{X_1, \ldots, X_N\}, conditionally independent given YY:

Xk∣Y=1∼N(μ1,σ2),Xk∣Y=0∼N(μ0,σ2),μ1≠μ0,X_k \mid Y = 1 \sim\mathcal{N}(\mu_1, \sigma^2), \qquad X_k \mid Y = 0 \sim\mathcal{N}(\mu_0, \sigma^2), \qquad\mu_1 \ne\mu_0,

where μ1,μ0∈R\mu_1, \mu_0 \in\mathbb{R} are the state-conditional means and σ2>0\sigma^2 > 0 is the common conditional variance; the assumption μ1≠μ0\mu_1 \ne\mu_0 ensures each cue is informative about YY. We denote the per-cue capability by θ:=(μ1−μ0)2/(2σ2)\theta:= (\mu_1 - \mu_0)^2/(2\sigma^2), the signal-to-noise ratio of a single cue.2

The human observes the cues in XH⊆{1,…,N}\mathcal{X}_H \subseteq\{1, \ldots, N\} with cardinality nHn_H; the AI observes the cues in XA\mathcal{X}_A with cardinality nAn_A; their shared cues have cardinality n∩:=∣XH∩XA∣n_{\cap} := \lvert\mathcal{X}_H \cap\mathcal{X}_A\rvert. Each agent forms a log-likelihood ratio summarizing her evidence,

ℓH:=∑k∈XHLLR⁡(Xk),ℓA:=∑k∈XALLR⁡(Xk),\ell_H := \sum_{k \in\mathcal{X}_H} \operatorname{LLR}(X_k), \qquad\ell_A := \sum_{k \in\mathcal{X}_A} \operatorname{LLR}(X_k),

where LLR⁡(Xk):=log⁡[ϕ(Xk;μ1,σ2)/ϕ(Xk;μ0,σ2)]\operatorname{LLR}(X_k) := \log[\phi(X_k; \mu_1, \sigma^2)/\phi(X_k; \mu_0, \sigma^2)] is the standard Gaussian log-likelihood ratio, evaluating to (μ1−μ0)(Xk−(μ1+μ0)/2)/σ2(\mu_1 - \mu_0)(X_k - (\mu_1 + \mu_0)/2)/\sigma^2. Here, ϕ(⋅;μ,σ2)\phi(\cdot; \mu, \sigma^2) denotes the Gaussian density with mean μ\mu and variance σ2\sigma^2. Since posterior log-odds equal prior log-odds plus the log-likelihood ratio, and the prior is symmetric (log-prior-odds equal 0), for either agent B∈{H,A}B \in\{H, A\}, ℓB\ell_B is exactly BB’s posterior log-odds for Y=1Y = 1 given her cues.

The overlap coefficient

λ:=n∩nA∈[0,1]\lambda:= \frac{n_{\cap}}{n_A} \in[0,1]

measures redundancy of the AI’s evidence from the human’s standpoint: if λ=0.5\lambda= 0.5, half of the AI’s cues are also possessed by the human, so only half of the AI’s evidence is new to the human. Equivalently, λ\lambda is the population regression coefficient of ℓH\ell_H on ℓA\ell_A given YY.

Lemma 2.1 (Joint distribution of log-likelihood ratios). *Conditional on YY,

(ℓHℓA)∣Y∼N((2Y−1)(τHτA),Σ),Σ=(2τH2τAλ2τAλ2τA),\begin{pmatrix} \ell_H \\ \ell_A \end{pmatrix} \mid Y \sim\mathcal{N}\left((2Y-1)\begin{pmatrix} \tau_H \\ \tau_A \end{pmatrix}, \Sigma\right), \qquad\Sigma= \begin{pmatrix} 2\tau_H & 2\tau_A\lambda\\ 2\tau_A\lambda& 2\tau_A \end{pmatrix},

where τH:=θnH\tau_H := \theta n_H and τA:=θnA\tau_A := \theta n_A are the agent capabilities.

The capability τB\tau_B measures how informative agent BB’s evidence is. By Lemma 2.1, ℓB\ell_B has conditional mean of magnitude τB\tau_B and conditional standard deviation 2τB\sqrt{2\tau_B}, so its signal-to-noise ratio (SNR) given YY is τB/2τB=τB/2\tau_B/\sqrt{2\tau_B}=\sqrt{\tau_B/2}. An agent who classifies via 1{ℓB>0}1\{\ell_B > 0\} is correct iff ℓB\ell_B shares the sign of (2Y−1)(2Y-1); by symmetry, this happens with probability Φ(τB/2)\Phi\left(\sqrt{\tau_B/2}\right), where Φ\Phi is the standard normal CDF. Standalone classification accuracy is thus Φ\Phi evaluated at the SNR, and is strictly increasing in τB\tau_B.

We refer to the cue pool {1,…,N}\{1,\ldots,N\} as the domain (a property of the question type), and to XH\mathcal{X}_H as the human’s access within that domain. The following shows that, under uniform sampling, overlap depends on the domain and the human’s access but not on the AI’s capability.

Proposition 2.2 (Overlap as a domain property). Fix XH\mathcal{X}_H with ∣XH∣=nH|\mathcal{X}_H|=n_H. Suppose the AI’s cue access XA\mathcal{X}_A is a uniformly random subset of size nAn_A drawn without replacement from {1,…,N}\{1,\ldots,N\}. Then E[λ]=nH/N\mathbb{E}[\lambda]=n_H/N, regardless of nAn_A.

If an AI scales by uniformly sampling more cues from a fixed pool, its overlap with a given human equals the human’s coverage fraction nH/Nn_H/N in expectation (the share of all cues the human accesses), regardless of the AI’s capability. Overlap is therefore not tunable through AI scale alone. The invariance is a statement in expectation under uniform random sampling of XA\mathcal{X}_A, not a claim that all AI systems in a domain share the same realized λ\lambda: two AI systems whose cue access is drawn from different sampling distributions (different training corpora, training objectives, or architectures) can yield different realized λ\lambda even within the same domain.

In the analysis that follows we consider (τH,τA,λ)(\tau_H,\tau_A,\lambda) as separate deployment parameters, because different humans paired with different AIs in different domains generate different parameter combinations. Two feasibility constraints on (τH,τA,λ)(\tau_H,\tau_A,\lambda) deserve note. The model-agnostic constraint, from positive semidefiniteness of the covariance matrix in Lemma 2.1, is τA≤τH/λ2\tau_A \le\tau_H/\lambda^2. The cue-pool generative model imposes the tighter bound τA≤τH/λ\tau_A \le\tau_H/\lambda (since n∩≤nHn_{\cap}\le n_H); at this boundary, the AI’s evidence is Blackwell-equivalent to the human-AI team’s. All quantitative results in our analysis live within the cue-pool feasible region (which is contained in the broader PSD region).

The Worker’s Decision Rule

After processing its cues, the AI communicates a sufficient statistic for ℓA\ell_A to the human, such as a recommended answer, a probability report, or a textual explanation. We assume this communication is informationally equivalent to ℓA\ell_A (the below analysis is agnostic to the format of the information). We refer to the human decision-maker who has communicated with AI and has access to (ℓH,ℓA)(\ell_H,\ell_A) as the worker. The worker forms a posterior probability for Y=1Y=1 and provides a binary decision. We model the worker via two parameters that capture distinct primitives: one a belief about the joint distribution of signals, the other a behavioral discount on the AI’s contribution.

  • Correlation awareness α^∈[0,1]\hat{\alpha}\in[0,1] (a belief). The worker’s perceived overlap is α^⋅λ\hat{\alpha}\cdot\lambda. At α^=1\hat{\alpha}=1 she perceives overlap correctly; at α^=0\hat{\alpha}=0 she perceives the signals as conditionally independent (correlation neglect); intermediate values represent partial awareness.

  • AI trust β^>0\hat{\beta}>0 (a behavioral discount). The worker applies a multiplicative weight β^\hat{\beta} to the AI signal ℓA\ell_A in her decision statistic. At β^=1\hat{\beta}=1 she applies no behavioral discount; β^<1\hat{\beta}<1 corresponds to AI undertrust; β^>1\hat{\beta}>1 corresponds to overtrust.

These parameters are jointly identified from cross-AI variation in λ\lambda (Appendix § 13).

References

  1. [1]Nikhil Agarwal, Alex Moehring, Pranav Rajpurkar, and Tobias Salz. Combining human expertise with artificial intelligence: Experimental evidence from radiology. NBER Working Paper 31422, National Bureau of Economic Research, 2023.DOI
  2. [2]Nikhil Agarwal, Alex Moehring, and Alexander Wolitzky. Designing human-ai collaboration: A sufficient-statistic approach. Technical report, National Bureau of Economic Research, 2025.DOI
  3. [3]Rohan Alur, Loren Laine, Darrick K. Li, Manish Raghavan, Devavrāt Shah, and Dennis Shung. Auditing for human expertise. In Advances in Neural Information Processing Systems 37 (NeurIPS), 2023.arxiv.org/abs/2306.01646
  4. [4]Rohan Alur, Manish Raghavan, and Devavrāt Shah. Human expertise in algorithmic prediction. In Advances in Neural Information Processing Systems 38 (NeurIPS), 2024.arxiv.org/abs/2402.00793
  5. [5]Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S. Weld, Walter S. Lasecki, and Eric Horvitz. Updates in human-AI teams: Understanding and addressing the performance/compatibility tradeoff. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2429–2437, 2019.DOI
  6. [6]Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel S. Weld. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI), 2021. doi: 10.1145/3411764.3445717.DOI
  7. [7]Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z. Gajos. To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1–21, 2021. doi: 10.1145/3449287.DOI
  8. [8]Serina Chang, Ashton Anderson, and Jake M Hofman. Chatbench: From static benchmarks to human-ai evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26009–26038, 2025.
  9. [9]Kate Donahue, Alexandra Chouldechova, and Krishnaram Kenthapadi. Human-algorithm collaboration: Achieving complementarity and avoiding unfairness. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 1639–1656, 2022. doi: 10.1145/3531146.3533221.DOI
  10. [10]Benjamin Enke and Florian Zimmermann. Correlation neglect in belief formation. The review of economic studies, 86(1):313–332, 2019.
  11. [11]Erik Eyster and Georg Weizsacker. Correlation neglect in financial decision-making. 2010.DOI
  12. [12]Ziyang Guo, Yifan Wu, Jason D. Hartline, and Jessica Hullman. A decision theoretic framework for measuring ai reliance. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’24, page 221–236, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704505. doi: 10.1145/3630106.3658901. URL https://doi.org/10.1145/3630106.3658901.
  13. [13]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  14. [14]David Madras, Tonian Pitassi, and Richard Zemel. Predict responsibly: Improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2018.arxiv.org/abs/1711.06664
  15. [15]Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pages 7076–7087. PMLR, 2020.arxiv.org/abs/2006.01862
  16. [16]Kenny Peng, Nikhil Garg, and Jon Kleinberg. A no free lunch theorem for human-ai collaboration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 14369–14376, 2025.arxiv.org/abs/2411.15230
  17. [17]Mark Steyvers, Heliodoro Tejeda, Gavin Kerrigan, and Padhraic Smyth. Bayesian modeling of human–ai complementarity. Proceedings of the National Academy of Sciences, 119(11):e2111547119, 2022.
  18. [18]Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan Halpern, Monika Janda, Aimilios Lallas, Caterina Longo, Josep Malvehy, John Paoli, Susana Puig, Cliff Rosendahl, H. Peter Soyer, Iris Zalaudek, and Harald Kittler. Human–computer collaboration for skin cancer recognition. Nature Medicine, 26(8):1229–1234, 2020. doi: 10.1038/s41591-020-0942-0.DOI
  19. [19]Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone. When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8(12):2293–2303, 2024.DOI
  20. [20]Rajeev Verma, Daniel Barrejon, and Eric Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 206 of Proceedings of Machine Learning Research, pages 11415–11434. PMLR, 2023.arxiv.org/abs/2210.16955
  21. [21]Bryan Wilder, Eric Horvitz, and Ece Kamar. Learning to complement humans. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI), pages 1526–1533, 2020.arxiv.org/abs/2005.00582

Paper details

vlnc:W_j25e2xfy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents