Introduction
Empirical evaluations of human-AI teams reach very different conclusions. Some studies report large accuracy gains from AI assistance [18 , 17 ] , while many find no improvement over the unaided human, or outright degradation: a more capable AI can hurt team performance [5 ] , AI explanations need not help [6 ] , AI predictions may not improve expert radiologists [1 ] , and a meta-analysis across tasks and systems finds that human-AI teams on average perform worse [19 ] . We argue that whether assistance helps turns not on the AI’s accuracy alone, but also on how much its evidence overlaps with the human’s and how the human combines it with her own. For example, when a worker treats overlapping evidence as if it were independent, she counts the shared part twice, a bias documented as correlation neglect in belief formation [10 , 11 ] , and the team can end up less accurate than the worker alone. This paper constructs a model built on these factors, uses it to characterize when AI assistance helps and when it hurts, and develops an empirical estimation strategy that exploits variation of outputs obtained by pairing the same workers with different AIs.
We study a binary classification problem in which the human and the AI each hold evidence about the state, summarized by their log-likelihood ratios ℓ H \ell_H ℓ H and ℓ A \ell_A ℓ A , and the worker decides by the sign of a weighted sum w H ℓ H + w A ℓ A w_H\ell_H + w_A\ell_A w H ℓ H + w A ℓ A . The model has two parts. In the observation model , each agent’s evidence is a collection of cues, and the key primitive is the overlap coefficient λ \lambda λ , the share of the AI’s cues the human already holds. In the behavioral model , the worker combines the two signals through correlation awareness α ^ ∈ [ 0 , 1 ] \hat{\alpha} \in[0,1] α ^ ∈ [ 0 , 1 ] , representing how well she accounts for that overlap, and trust β ^ > 0 \hat{\beta} > 0 β ^ > 0 , how much weight she puts on the AI information. Two benchmarks sit inside this family. The Bayes-rational worker (α ^ = 1 \hat{\alpha} = 1 α ^ = 1 , β ^ = 1 \hat{\beta} = 1 β ^ = 1 ) de-correlates and weights optimally, while the correlation-neglect worker (α ^ = 0 \hat{\alpha} = 0 α ^ = 0 ) treats the two signals as independent. Overlap is a property of the information, awareness and trust are properties of the worker, and together they determine the team’s accuracy.
Our first results compare team performance against human and AI alone. Under the Bayes-rational benchmark the team is at least as accurate as either source alone, and strictly more accurate unless the AI’s evidence already contains the human’s. The impairment and automation outcomes seen in the literature, outside that case, must then come from the worker’s behavior rather than from the information structure. Away from that benchmark, we obtain a closed-form phase diagram over AI capability τ A \tau_A τ A and overlap λ \lambda λ with three regions, complementarity where the team is most accurate, impairment where the unaided human is, and automation where the AI alone is. Two of its features run against intuition. The first is that a more capable AI can make the team less accurate. When overlap is high (λ > 1 − β ^ / 2 \lambda> 1-\hat{\beta}/2 λ > 1 − β ^ /2 ), a stronger AI only gives a correlation-neglecting worker more evidence to double-count, and that can lower accuracy. The second is that undertrust is not simply an error to be corrected. Reducing trust shrinks the impairment region but brings automation on at lower capability, so the trust level that maximizes team accuracy is not always β ^ = 1 \hat{\beta} = 1 β ^ = 1 but depends on the deployment parameters.
Our second set of results turns the model into a measurement tool. A Bayes-rational worker’s weight on the AI changes with overlap, while a correlation-neglecting worker’s does not, and this difference lets us estimate the parameters by pairing the same workers with two AIs of different overlap. Trust β ^ \hat{\beta} β ^ is sharply identified, while awareness α ^ \hat{\alpha} α ^ is only weakly identified because the overlap coefficient in our experiment is relatively small. We then split the worker’s accuracy loss against the Bayes-rational benchmark into two parts that add up exactly at any level of awareness, a correlation-neglect cost from misperceiving the overlap and an undertrust cost from underweighting the AI. Each is tied to a distinct lever on opposite sides of the information-behavior divide: reducing overlap on the AI side by training it to supply what the human lacks, and calibrating trust on the human side.
We examine the framework in the context of ChatBench [8 ] , where 643 workers answer multiple-choice questions paired with either GPT-4o or Llama-3.1-8b. AI assistance raises accuracy by 21 percentage points, yet workers put only about a quarter of the Bayes-rational weight on the AI (β ^ = 0.285 \hat{\beta} = 0.285 β ^ = 0.285 ). GPT-4o is more capable and overlaps the human less, so it lies in the automation regime, where the AI alone beats the team; Llama-3.1-8b is weaker and overlaps more, so it sits at the boundary between complementarity and automation. The worker’s behavior is nearly the same in both, so it is the difference in AI capability between the two deployments that shifts the deployment from one regime to the other, as the framework predicts. The cost of undertrust is therefore uneven. We simulate a trust-calibrated decision rule, which recovers about 2.6 points of accuracy for the strong AI and moves the team toward the boundary, but almost nothing for the weak AI.
The two sides of the information-behavior divide provide two design levers. On the human side, trust should be set to the deployment’s accuracy-maximizing level, not reflexively to β ^ = 1 \hat{\beta} = 1 β ^ = 1 , since some undertrust is protective in the impairment region. On the AI side, reducing overlap by training the AI to supply what the human lacks rather than what she already knows can raise team accuracy even without raising the AI’s own accuracy. The phase diagram also says when to show the AI at all: withhold its signal when the AI is too weak, provide it in the complementarity band, and automate once the AI is strong enough.
Our work sits among several literatures but differs in what it treats as endogenous. Studies of optimal human-AI collaboration through sufficient statistics [2 ] leave out the conditionally dependent signals that are central here. Work on reducing overreliance [7 ] and on testing whether AI adds to human expertise [3 , 4 ] takes the intervention or the test as given, whereas we characterize the best deployment as a function of capability, overlap, and behavior. Closest in spirit, Guo et al. [ [12 ] define over- and under-reliance from the human’s adoption or switch rates against a Bayesian benchmark; our two parameters give a micro-foundation for their two sources of loss, trust for their reliance loss and correlation awareness for their discrimination loss, with a model of a worker who combines a continuous AI signal with her own rather than choosing per instance between the two recommendations. Other work on combining expert information [9 , 16 ] and learning-to-defer [14 , 15 , 20 , 21 ] takes the human’s decision rule as exogenous, whereas we derive it from behavioral primitives. Section 2 introduces the model, Section 3 develops the phase diagram and welfare decomposition, and Section 4 presents the empirical framework and results.
Model
A worker forms a tentative answer to a question, sees an AI assistant’s recommendation, and decides whether to revise. The evidence available about a question is a collection of cues , atomic units of evidence about the underlying answer that an agent might or might not access. For a question on classical mechanics, the cues are the relevant physical principles, the diagnostic features of the answer choices, and the patterns from prior similar problems. If a human and an AI both lean on the standard textbook derivation, there will be substantial overlap in cues whereas if they each approach the question through different reasoning paths there will be little overlap.
Formally, a binary state Y ∈ { 0 , 1 } Y \in\{0, 1\} Y ∈ { 0 , 1 } is drawn from the symmetric prior P ( Y = 1 ) = P ( Y = 0 ) = 1 / 2 P(Y = 1) = P(Y = 0) = 1/2 P ( Y = 1 ) = P ( Y = 0 ) = 1/2 .1 There is a finite collection of N N N cues { X 1 , … , X N } \{X_1, \ldots, X_N\} { X 1 , … , X N } , conditionally independent given Y Y Y :
X k ∣ Y = 1 ∼ N ( μ 1 , σ 2 ) , X k ∣ Y = 0 ∼ N ( μ 0 , σ 2 ) , μ 1 ≠ μ 0 , X_k \mid Y = 1 \sim\mathcal{N}(\mu_1, \sigma^2), \qquad X_k \mid Y = 0 \sim\mathcal{N}(\mu_0, \sigma^2), \qquad\mu_1 \ne\mu_0, X k ∣ Y = 1 ∼ N ( μ 1 , σ 2 ) , X k ∣ Y = 0 ∼ N ( μ 0 , σ 2 ) , μ 1 = μ 0 ,
where μ 1 , μ 0 ∈ R \mu_1, \mu_0 \in\mathbb{R} μ 1 , μ 0 ∈ R are the state-conditional means and σ 2 > 0 \sigma^2 > 0 σ 2 > 0 is the common conditional variance; the assumption μ 1 ≠ μ 0 \mu_1 \ne\mu_0 μ 1 = μ 0 ensures each cue is informative about Y Y Y . We denote the per-cue capability by θ : = ( μ 1 − μ 0 ) 2 / ( 2 σ 2 ) \theta:= (\mu_1 - \mu_0)^2/(2\sigma^2) θ := ( μ 1 − μ 0 ) 2 / ( 2 σ 2 ) , the signal-to-noise ratio of a single cue.2
The human observes the cues in X H ⊆ { 1 , … , N } \mathcal{X}_H \subseteq\{1, \ldots, N\} X H ⊆ { 1 , … , N } with cardinality n H n_H n H ; the AI observes the cues in X A \mathcal{X}_A X A with cardinality n A n_A n A ; their shared cues have cardinality n ∩ : = ∣ X H ∩ X A ∣ n_{\cap} := \lvert\mathcal{X}_H \cap\mathcal{X}_A\rvert n ∩ := ∣ X H ∩ X A ∣ . Each agent forms a log-likelihood ratio summarizing her evidence,
ℓ H : = ∑ k ∈ X H LLR ( X k ) , ℓ A : = ∑ k ∈ X A LLR ( X k ) , \ell_H := \sum_{k \in\mathcal{X}_H} \operatorname{LLR}(X_k), \qquad\ell_A := \sum_{k \in\mathcal{X}_A} \operatorname{LLR}(X_k), ℓ H := k ∈ X H ∑ LLR ( X k ) , ℓ A := k ∈ X A ∑ LLR ( X k ) ,
where LLR ( X k ) : = log [ ϕ ( X k ; μ 1 , σ 2 ) / ϕ ( X k ; μ 0 , σ 2 ) ] \operatorname{LLR}(X_k) := \log[\phi(X_k; \mu_1, \sigma^2)/\phi(X_k; \mu_0, \sigma^2)] LLR ( X k ) := log [ ϕ ( X k ; μ 1 , σ 2 ) / ϕ ( X k ; μ 0 , σ 2 )] is the standard Gaussian log-likelihood ratio, evaluating to ( μ 1 − μ 0 ) ( X k − ( μ 1 + μ 0 ) / 2 ) / σ 2 (\mu_1 - \mu_0)(X_k - (\mu_1 + \mu_0)/2)/\sigma^2 ( μ 1 − μ 0 ) ( X k − ( μ 1 + μ 0 ) /2 ) / σ 2 . Here, ϕ ( ⋅ ; μ , σ 2 ) \phi(\cdot; \mu, \sigma^2) ϕ ( ⋅ ; μ , σ 2 ) denotes the Gaussian density with mean μ \mu μ and variance σ 2 \sigma^2 σ 2 . Since posterior log-odds equal prior log-odds plus the log-likelihood ratio, and the prior is symmetric (log-prior-odds equal 0), for either agent B ∈ { H , A } B \in\{H, A\} B ∈ { H , A } , ℓ B \ell_B ℓ B is exactly B B B ’s posterior log-odds for Y = 1 Y = 1 Y = 1 given her cues.
The overlap coefficient
λ : = n ∩ n A ∈ [ 0 , 1 ] \lambda:= \frac{n_{\cap}}{n_A} \in[0,1] λ := n A n ∩ ∈ [ 0 , 1 ]
measures redundancy of the AI’s evidence from the human’s standpoint: if λ = 0.5 \lambda= 0.5 λ = 0.5 , half of the AI’s cues are also possessed by the human, so only half of the AI’s evidence is new to the human. Equivalently, λ \lambda λ is the population regression coefficient of ℓ H \ell_H ℓ H on ℓ A \ell_A ℓ A given Y Y Y .
Lemma 2.1 (Joint distribution of log-likelihood ratios). *Conditional on Y Y Y ,
( ℓ H ℓ A ) ∣ Y ∼ N ( ( 2 Y − 1 ) ( τ H τ A ) , Σ ) , Σ = ( 2 τ H 2 τ A λ 2 τ A λ 2 τ A ) , \begin{pmatrix} \ell_H \\ \ell_A \end{pmatrix} \mid Y \sim\mathcal{N}\left((2Y-1)\begin{pmatrix} \tau_H \\ \tau_A \end{pmatrix}, \Sigma\right), \qquad\Sigma= \begin{pmatrix} 2\tau_H & 2\tau_A\lambda\\ 2\tau_A\lambda& 2\tau_A \end{pmatrix}, ( ℓ H ℓ A ) ∣ Y ∼ N ( ( 2 Y − 1 ) ( τ H τ A ) , Σ ) , Σ = ( 2 τ H 2 τ A λ 2 τ A λ 2 τ A ) , where τ H : = θ n H \tau_H := \theta n_H τ H := θ n H and τ A : = θ n A \tau_A := \theta n_A τ A := θ n A are the agent capabilities.
The capability τ B \tau_B τ B measures how informative agent B B B ’s evidence is. By Lemma 2.1 , ℓ B \ell_B ℓ B has conditional mean of magnitude τ B \tau_B τ B and conditional standard deviation 2 τ B \sqrt{2\tau_B} 2 τ B , so its signal-to-noise ratio (SNR) given Y Y Y is τ B / 2 τ B = τ B / 2 \tau_B/\sqrt{2\tau_B}=\sqrt{\tau_B/2} τ B / 2 τ B = τ B /2 . An agent who classifies via 1 { ℓ B > 0 } 1\{\ell_B > 0\} 1 { ℓ B > 0 } is correct iff ℓ B \ell_B ℓ B shares the sign of ( 2 Y − 1 ) (2Y-1) ( 2 Y − 1 ) ; by symmetry, this happens with probability Φ ( τ B / 2 ) \Phi\left(\sqrt{\tau_B/2}\right) Φ ( τ B /2 ) , where Φ \Phi Φ is the standard normal CDF. Standalone classification accuracy is thus Φ \Phi Φ evaluated at the SNR, and is strictly increasing in τ B \tau_B τ B .
We refer to the cue pool { 1 , … , N } \{1,\ldots,N\} { 1 , … , N } as the domain (a property of the question type), and to X H \mathcal{X}_H X H as the human’s access within that domain. The following shows that, under uniform sampling, overlap depends on the domain and the human’s access but not on the AI’s capability.
Proposition 2.2 (Overlap as a domain property). Fix X H \mathcal{X}_H X H with ∣ X H ∣ = n H |\mathcal{X}_H|=n_H ∣ X H ∣ = n H . Suppose the AI’s cue access X A \mathcal{X}_A X A is a uniformly random subset of size n A n_A n A drawn without replacement from { 1 , … , N } \{1,\ldots,N\} { 1 , … , N } . Then E [ λ ] = n H / N \mathbb{E}[\lambda]=n_H/N E [ λ ] = n H / N , regardless of n A n_A n A .
If an AI scales by uniformly sampling more cues from a fixed pool, its overlap with a given human equals the human’s coverage fraction n H / N n_H/N n H / N in expectation (the share of all cues the human accesses), regardless of the AI’s capability. Overlap is therefore not tunable through AI scale alone. The invariance is a statement in expectation under uniform random sampling of X A \mathcal{X}_A X A , not a claim that all AI systems in a domain share the same realized λ \lambda λ : two AI systems whose cue access is drawn from different sampling distributions (different training corpora, training objectives, or architectures) can yield different realized λ \lambda λ even within the same domain.
In the analysis that follows we consider ( τ H , τ A , λ ) (\tau_H,\tau_A,\lambda) ( τ H , τ A , λ ) as separate deployment parameters, because different humans paired with different AIs in different domains generate different parameter combinations. Two feasibility constraints on ( τ H , τ A , λ ) (\tau_H,\tau_A,\lambda) ( τ H , τ A , λ ) deserve note. The model-agnostic constraint, from positive semidefiniteness of the covariance matrix in Lemma 2.1, is τ A ≤ τ H / λ 2 \tau_A \le\tau_H/\lambda^2 τ A ≤ τ H / λ 2 . The cue-pool generative model imposes the tighter bound τ A ≤ τ H / λ \tau_A \le\tau_H/\lambda τ A ≤ τ H / λ (since n ∩ ≤ n H n_{\cap}\le n_H n ∩ ≤ n H ); at this boundary, the AI’s evidence is Blackwell-equivalent to the human-AI team’s. All quantitative results in our analysis live within the cue-pool feasible region (which is contained in the broader PSD region).
The Worker’s Decision Rule
After processing its cues, the AI communicates a sufficient statistic for ℓ A \ell_A ℓ A to the human, such as a recommended answer, a probability report, or a textual explanation. We assume this communication is informationally equivalent to ℓ A \ell_A ℓ A (the below analysis is agnostic to the format of the information). We refer to the human decision-maker who has communicated with AI and has access to ( ℓ H , ℓ A ) (\ell_H,\ell_A) ( ℓ H , ℓ A ) as the worker . The worker forms a posterior probability for Y = 1 Y=1 Y = 1 and provides a binary decision. We model the worker via two parameters that capture distinct primitives: one a belief about the joint distribution of signals, the other a behavioral discount on the AI’s contribution.
Correlation awareness α ^ ∈ [ 0 , 1 ] \hat{\alpha}\in[0,1] α ^ ∈ [ 0 , 1 ] (a belief). The worker’s perceived overlap is α ^ ⋅ λ \hat{\alpha}\cdot\lambda α ^ ⋅ λ . At α ^ = 1 \hat{\alpha}=1 α ^ = 1 she perceives overlap correctly; at α ^ = 0 \hat{\alpha}=0 α ^ = 0 she perceives the signals as conditionally independent (correlation neglect); intermediate values represent partial awareness.
These parameters are jointly identified from cross-AI variation in λ \lambda λ (Appendix § 13 ).
Assumption 2.3 (Worker’s behavior). The worker has parameters ( α ^ , β ^ ) ∈ [ 0 , 1 ] × R > 0 (\hat{\alpha},\hat{\beta})\in[0,1]\times\mathbb{R}_{>0} ( α ^ , β ^ ) ∈ [ 0 , 1 ] × R > 0 and behaves as follows:
(i) Bayesian updating under perceived overlap. The worker performs Bayesian updating on ( ℓ H , ℓ A , Y ) (\ell_H,\ell_A,Y) ( ℓ H , ℓ A , Y ) under a belief that perceives overlap as α ^ λ \hat{\alpha}\lambda α ^ λ and capabilities ( τ H , τ A ) (\tau_H,\tau_A) ( τ H , τ A ) at their true values. This produces a Bayesian posterior log-odds.
(ii) Behavioral trust discount. After Bayesian updating, the worker scales the AI signal ℓ A \ell_A ℓ A by a multiplicative factor β ^ \hat{\beta} β ^ in the posterior log-odds. This discount is applied to the AI signal after Bayesian updating, with no underlying belief interpretation.
Under this assumption and the Gaussian signal structure, the worker’s decision rule satisfies the following linear aggregation of the agent log-likelihood ratios.3
Lemma 2.4 (Worker’s decision weights). Under Assumption 2.3 , a worker with parameters ( α ^ , β ^ ) ∈ [ 0 , 1 ] × R > 0 (\hat{\alpha},\hat{\beta}) \in[0,1] \times\mathbb{R}_{>0} ( α ^ , β ^ ) ∈ [ 0 , 1 ] × R > 0 uses the linear rule ℓ ^ = w H ℓ H + w A ℓ A \hat{\ell}=w_H\ell_H+w_A\ell_A ℓ ^ = w H ℓ H + w A ℓ A with weights
w H ( α ^ ) = τ H − τ A α ^ λ τ H − τ A α ^ 2 λ 2 , w A ( α ^ , β ^ ) = β ^ ⋅ τ H ( 1 − α ^ λ ) τ H − τ A α ^ 2 λ 2 . w_H(\hat{\alpha})=\frac{\tau_H-\tau_A\hat{\alpha}\lambda}{\tau_H-\tau_A\hat{\alpha}^2\lambda^2}, \qquad w_A(\hat{\alpha},\hat{\beta})=\hat{\beta}\cdot\frac{\tau_H(1-\hat{\alpha}\lambda)}{\tau_H-\tau_A\hat{\alpha}^2\lambda^2}. w H ( α ^ ) = τ H − τ A α ^ 2 λ 2 τ H − τ A α ^ λ , w A ( α ^ , β ^ ) = β ^ ⋅ τ H − τ A α ^ 2 λ 2 τ H ( 1 − α ^ λ ) . The human evidence weight depends only on α ^ \hat{\alpha} α ^ . The AI evidence weight equals β ^ \hat{\beta} β ^ times the Bayesian weight w A B a y e s ( α ^ ) : = τ H ( 1 − α ^ λ ) / ( τ H − τ A α ^ 2 λ 2 ) w_A^{\mathrm{Bayes}}(\hat{\alpha}):=\tau_H(1-\hat{\alpha}\lambda)/(\tau_H-\tau_A\hat{\alpha}^2\lambda^2) w A Bayes ( α ^ ) := τ H ( 1 − α ^ λ ) / ( τ H − τ A α ^ 2 λ 2 ) .
The worker treats ℓ ^ \hat{\ell} ℓ ^ as her log-odds for Y = 1 Y=1 Y = 1 . Under the symmetric prior, this gives the binary decision Y ^ : = 1 { ℓ ^ > 0 } \hat{Y}:=\mathbb{1}\{\hat{\ell}>0\} Y ^ := 1 { ℓ ^ > 0 } (for 0-1 loss) and the probability report p ^ : = expit ( ℓ ^ ) \hat{p}:=\operatorname{expit}(\hat{\ell}) p ^ := expit ( ℓ ^ ) (for log loss).
Remark 2.5 (Why β ^ \hat{\beta} β ^ is a behavioral discount, not a belief). A natural alternative interprets trust discount as a subjective belief about AI capability, replacing τ A \tau_A τ A by β ^ τ A \hat{\beta}\tau_A β ^ τ A in the worker’s perceived signal distribution. Because any such subjective-capability model leaves the Bayesian weight on ℓ A \ell_A ℓ A unchanged, rendering trust unidentifiable, we model β ^ \hat{\beta} β ^ as a behavioral discount applied after Bayesian updating; see Appendix § 6 .
Two Benchmarks
The two-parameter family ( α ^ , β ^ ) ∈ [ 0 , 1 ] × R > 0 (\hat{\alpha},\hat{\beta}) \in[0,1] \times\mathbb{R}_{>0} ( α ^ , β ^ ) ∈ [ 0 , 1 ] × R > 0 contains a continuum of worker behaviors. We consider two benchmarks at the extremes of correlation awareness.
Bayes-rational (BR). α ^ = 1 \hat{\alpha}=1 α ^ = 1 , β ^ = 1 \hat{\beta}=1 β ^ = 1 . The worker perceives overlap correctly and applies no trust discount. Substituting into Lemma 2.4 gives weights w H B R = τ H − τ A λ τ H − τ A λ 2 w_H^{\mathrm{BR}}=\frac{\tau_H-\tau_A\lambda}{\tau_H-\tau_A\lambda^2} w H BR = τ H − τ A λ 2 τ H − τ A λ and w A B R = τ H ( 1 − λ ) τ H − τ A λ 2 w_A^{\mathrm{BR}}=\frac{\tau_H(1-\lambda)}{\tau_H-\tau_A\lambda^2} w A BR = τ H − τ A λ 2 τ H ( 1 − λ ) . In the symmetric case τ H = τ A \tau_H=\tau_A τ H = τ A , both weights equal 1 / ( 1 + λ ) 1/(1+\lambda) 1/ ( 1 + λ ) , with the Bayesian shrinkage correcting for redundant evidence on both sides.
Correlation neglect (CN). α ^ = 0 \hat{\alpha}=0 α ^ = 0 , β ^ \hat{\beta} β ^ arbitrary. The worker treats ( ℓ H , ℓ A ) (\ell_H,\ell_A) ( ℓ H , ℓ A ) as conditionally independent given Y Y Y , yielding ℓ ^ = ℓ H + β ^ ℓ A \hat{\ell}=\ell_H+\hat{\beta}\ell_A ℓ ^ = ℓ H + β ^ ℓ A . The case β ^ = 1 \hat{\beta}=1 β ^ = 1 is the classical CN agent; β ^ < 1 \hat{\beta}<1 β ^ < 1 also discounts the AI signal.
The benchmarks differ in how the AI weight responds to true overlap: w A B R w_A^{\mathrm{BR}} w A BR depends on λ \lambda λ , whereas w A C N = β ^ w_A^{\mathrm{CN}}=\hat{\beta} w A CN = β ^ is invariant to λ \lambda λ — the CN worker treats AI evidence the same regardless of true redundancy. The interior of ( α ^ , β ^ ) (\hat{\alpha},\hat{\beta}) ( α ^ , β ^ ) space combines partial correlation awareness and trust miscalibration.
Analysis
This section develops three results that guide the empirical study presented later. The first is a closed-form expression for the worker’s classification accuracy that, under correlation neglect ( α ^ = 0 ) (\hat{\alpha}=0) ( α ^ = 0 ) , partitions the deployment space ( τ A , λ ) (\tau_A,\lambda) ( τ A , λ ) into three regimes: complementarity (human-AI team is most accurate), impairment (human alone is most accurate), and automation (AI alone is most accurate). The second is a welfare decomposition: the worker’s loss against the Bayes-rational benchmark splits additively into a correlation-neglect cost and an undertrust cost, which could then be calculated by the identified behavioral parameters α ^ \hat{\alpha} α ^ and β ^ \hat{\beta} β ^ . Our results depend only on the joint-Gaussian structure of Lemma 2.1; the cue-pool model is one interpretable construction yielding it, not a requirement.
Performance Objective
The worker’s decision rule under Assumption 2.3 can be evaluated using different performance objectives. We focus on binary-decision accuracy in the main text, which is the natural welfare statistic when the worker’s output is a binary label. Formally, the accuracy is the probability that the worker’s binary decision is correct: A : = P ( Y ^ = Y ) \mathcal{A}:=P(\hat{Y}=Y) A := P ( Y ^ = Y ) , where Y ^ = 1 { ℓ ^ > 0 } \hat{Y}=\mathbb{1}\{\hat{\ell}>0\} Y ^ = 1 { ℓ ^ > 0 } is her hardmax decision. Other objectives like log loss and switching-rate are derived and discussed in Appendix § 7 and are useful for probability reports and revision data.
Accuracy via the signal-to-noise (SNR) ratio. Under Assumption 2.3, the worker’s decision statistic ℓ ^ = w H ℓ H + w A ℓ A \hat{\ell} = w_H\ell_H + w_A\ell_A ℓ ^ = w H ℓ H + w A ℓ A is a linear combination of the jointly Gaussian pair ( ℓ H , ℓ A ) (\ell_H,\ell_A) ( ℓ H , ℓ A ) from Lemma 2.1 ; hence ℓ ^ ∣ Y \hat{\ell} \mid Y ℓ ^ ∣ Y is itself Gaussian. Accuracy under the rule 1 { ℓ ^ > 0 } \mathbf{1}\{\hat{\ell} > 0\} 1 { ℓ ^ > 0 } is a function of the SNR ratio of ℓ ^ \hat{\ell} ℓ ^ , namely the ratio of the conditional mean’s magnitude to the conditional standard deviation.
Lemma 3.1 (Accuracy of the worker’s decision rule). Under Assumption 2.3 and the symmetric prior, the worker classifying via Y ^ = 1 { ℓ ^ > 0 } \hat{Y} = 1\{\hat{\ell} > 0\} Y ^ = 1 { ℓ ^ > 0 } attains accuracy A H A = Φ ( r H A ) A_{HA} = \Phi(r_{HA}) A H A = Φ ( r H A ) , where r H A r_{HA} r H A is the signal-to-noise ratio of ℓ ^ \hat{\ell} ℓ ^ :
r H A ( α ^ , β ^ ; τ H , τ A , λ ) = w H τ H + w A τ A 2 ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) r_{HA}(\hat{\alpha},\hat{\beta};\tau_H,\tau_A,\lambda)=\frac{w_H\tau_H+w_A\tau_A}{\sqrt{2(w_H^2\tau_H+w_A^2\tau_A+2w_Hw_A\tau_A\lambda)}} r H A ( α ^ , β ^ ; τ H , τ A , λ ) = 2 ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) w H τ H + w A τ A and the weights ( w H , w A ) (w_H,w_A) ( w H , w A ) are given by Lemma 2.4 .
The signal-to-noise ratio r H A r_{HA} r H A has a clear reading. Conditional on Y Y Y , the worker’s decision statistic ℓ ^ \hat{\ell} ℓ ^ has mean ( 2 Y − 1 ) ( w H τ H + w A τ A ) (2Y-1)(w_H\tau_H+w_A\tau_A) ( 2 Y − 1 ) ( w H τ H + w A τ A ) and variance 2 ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) 2(w_H^2\tau_H+w_A^2\tau_A+2w_Hw_A\tau_A\lambda) 2 ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) . The variance decomposes into three terms: the human’s contribution w H 2 τ H w_H^2\tau_H w H 2 τ H , the AI’s contribution w A 2 τ A w_A^2\tau_A w A 2 τ A , and a cross-term 2 w H w A τ A λ 2w_Hw_A\tau_A\lambda 2 w H w A τ A λ capturing the double-counting penalty from overlapped evidence.
Bayes-rational rule is welfare-optimal. By Lemma 2.1 , ( ℓ H , ℓ A ) ∣ Y (\ell_H,\ell_A)\mid Y ( ℓ H , ℓ A ) ∣ Y is Gaussian with shared covariance across Y Y Y ; the linear direction maximizing r H A r_{HA} r H A from Lemma 3.1 therefore coincides with the Bayes-rational rule from § 2.2 , giving maximum SNR r ∗ : = r H A ( α ^ = 1 , β ^ = 1 ) r^* := r_{HA}(\hat{\alpha}=1,\hat{\beta}=1) r ∗ := r H A ( α ^ = 1 , β ^ = 1 ) and optimal accuracy A B R = Φ ( r ∗ ) A^{\mathrm{BR}}=\Phi(r^*) A BR = Φ ( r ∗ ) . The accuracy gap A B R − A H A ( α ^ , β ^ ) ≥ 0 A^{\mathrm{BR}}-A_{HA}(\hat{\alpha},\hat{\beta})\ge0 A BR − A H A ( α ^ , β ^ ) ≥ 0 is the worker’s loss relative to the BR benchmark.
Phase Diagram
Comparing the three accuracies A H = Φ ( τ H / 2 ) A_H=\Phi(\sqrt{\tau_H/2}) A H = Φ ( τ H /2 ) , A H A A_{HA} A H A from Lemma 3.1 , and A A I = Φ ( τ A / 2 ) A_{AI}=\Phi(\sqrt{\tau_A/2}) A A I = Φ ( τ A /2 ) partitions the deployment space ( τ A , λ ) (\tau_A,\lambda) ( τ A , λ ) , for each fixed β ^ > 0 \hat{\beta}>0 β ^ > 0 , into three regions: complementarity (HA optimal, i.e., the human-AI team is most accurate), impairment (H optimal), and automation (AI optimal). For clarity, we state the closed-form phase diagram at strict correlation neglect ( α ^ = 0 ) (\hat{\alpha}=0) ( α ^ = 0 ) ; the same qualitative forces extend continuously to partial correlation awareness, with larger α ^ \hat{\alpha} α ^ attenuating double-counting.
Theorem 3.2 (Phase diagram under correlation neglect). Fix α ^ = 0 \hat{\alpha}=0 α ^ = 0 , β ^ > 0 \hat{\beta}>0 β ^ > 0 , and the symmetric prior. 4
Define
τ a u g ( λ ; β ^ ) : = β ^ + 2 λ − 2 τ H , τ a u t o ( λ ; β ^ ) : = 2 β ^ − 1 + ( 2 β ^ − 1 ) 2 + 8 β ^ λ 4 β ^ λ τ H for λ > 0 , \tau_{\mathrm{aug}}(\lambda;\hat{\beta}):=\frac{\hat{\beta}+2\lambda-2}{\tau_H},\qquad\tau_{\mathrm{auto}}(\lambda;\hat{\beta}):=\frac{2\hat{\beta}-1+\sqrt{(2\hat{\beta}-1)^2+8\hat{\beta}\lambda}}{4\hat{\beta}\lambda}\tau_H \quad\text{for }\lambda>0, τ aug ( λ ; β ^ ) := τ H β ^ + 2 λ − 2 , τ auto ( λ ; β ^ ) := 4 β ^ λ 2 β ^ − 1 + ( 2 β ^ − 1 ) 2 + 8 β ^ λ τ H for λ > 0 , with τ a u t o ( 0 ; β ^ ) = + ∞ \tau_{\mathrm{auto}}(0;\hat{\beta})=+\infty τ auto ( 0 ; β ^ ) = + ∞ if β ^ ≥ 1 / 2 \hat{\beta}\ge1/2 β ^ ≥ 1/2 and τ H / ( 1 − 2 β ^ ) \tau_H/(1-2\hat{\beta}) τ H / ( 1 − 2 β ^ ) otherwise. Then for τ A > 0 \tau_A>0 τ A > 0 :
Low overlap ( λ ≤ 1 − β ^ / 2 (\lambda\le1-\hat{\beta}/2 ( λ ≤ 1 − β ^ /2 , when β ^ < 2 ) \hat{\beta}<2) β ^ < 2 ) : complementarity if τ A < τ a u t o ( λ ; β ^ ) \tau_A<\tau_{\mathrm{auto}}(\lambda;\hat{\beta}) τ A < τ auto ( λ ; β ^ ) ; automation if τ A ≥ τ a u t o ( λ ; β ^ ) \tau_A\ge\tau_{\mathrm{auto}}(\lambda;\hat{\beta}) τ A ≥ τ auto ( λ ; β ^ ) .
Moderate overlap ( 1 − β ^ / 2 < λ < 1 ) (1-\hat{\beta}/2<\lambda<1) ( 1 − β ^ /2 < λ < 1 ) : impairment if τ A < τ a u g ( λ ; β ^ ) \tau_A<\tau_{\mathrm{aug}}(\lambda;\hat{\beta}) τ A < τ aug ( λ ; β ^ ) ; complementarity if τ a u g ( λ ; β ^ ) ≤ τ A ≤ τ a u t o ( λ ; β ^ ) \tau_{\mathrm{aug}}(\lambda;\hat{\beta})\le\tau_A\le\tau_{\mathrm{auto}}(\lambda;\hat{\beta}) τ aug ( λ ; β ^ ) ≤ τ A ≤ τ auto ( λ ; β ^ ) ; automation if τ A > τ a u t o ( λ ; β ^ ) \tau_A>\tau_{\mathrm{auto}}(\lambda;\hat{\beta}) τ A > τ auto ( λ ; β ^ ) .
High overlap ( λ = 1 ) (\lambda=1) ( λ = 1 ) : the two thresholds coincide, τ a u g ( 1 ; β ^ ) = τ a u t o ( 1 ; β ^ ) = τ H \tau_{\mathrm{aug}}(1;\hat{\beta})=\tau_{\mathrm{auto}}(1;\hat{\beta})=\tau_H τ aug ( 1 ; β ^ ) = τ auto ( 1 ; β ^ ) = τ H , so the complementarity region collapses to the single point τ A = τ H \tau_A=\tau_H τ A = τ H .
Here, τ a u g \tau_{\mathrm{aug}} τ aug is strictly increasing in λ \lambda λ on ( 1 − β ^ / 2 , 1 ] (1-\hat{\beta}/2,1] ( 1 − β ^ /2 , 1 ] and τ a u t o \tau_{\mathrm{auto}} τ auto is strictly decreasing in λ \lambda λ on ( 0 , 1 ] (0,1] ( 0 , 1 ] .
The theorem yields several substantive insights that we elaborate below.
Insight 1: (Impairment Region) – A weak AI can hurt a worker who neglects correlation. The phase diagram has an impairment region where the worker assisted by the AI does worse than the worker alone (A H A < A H A_{HA}<A_H A H A < A H ). From Theorem 3.2 , this region exists whenever overlap satisfies λ > 1 − β ^ / 2 \lambda>1-\hat{\beta}/2 λ > 1 − β ^ /2 . The mechanism is direct. When overlap is high, much of the AI’s evidence is content the worker already has. The correlation-neglect worker, using the rule ℓ ^ = ℓ H + β ^ ℓ A \hat{\ell}=\ell_H+\hat{\beta}\ell_A ℓ ^ = ℓ H + β ^ ℓ A , treats ℓ A \ell_A ℓ A as if it were a fresh, independent signal. By giving ℓ A \ell_A ℓ A full weight β ^ \hat{\beta} β ^ , she effectively double-counts the overlapping content. When the AI is weak (small τ A \tau_A τ A ), the genuinely new information in ℓ A \ell_A ℓ A is too small to offset the double-counting penalty, and her accuracy falls below the human-alone benchmark.
Insight 2: (Automation Region) – Automation is inevitable as AI capability grows. From Theorem 3.2 , the automation threshold τ a u t o ( λ ; β ^ ) \tau_{\mathrm{auto}}(\lambda;\hat{\beta}) τ auto ( λ ; β ^ ) is finite for any λ > 0 \lambda>0 λ > 0 , so AI alone outperforms the team whenever τ A > τ a u t o \tau_A>\tau_{\mathrm{auto}} τ A > τ auto . The cue-pool interpretation imposes an upper bound τ A ≤ τ H / λ \tau_A\leq\tau_H/\lambda τ A ≤ τ H / λ ; at this boundary the AI’s evidence is Blackwell-equivalent to the team’s, so automation is welfare-optimal regardless of trust calibration. Since τ a u t o ( λ ; β ^ ) < τ H / λ \tau_{\mathrm{auto}}(\lambda;\hat{\beta})<\tau_H/\lambda τ auto ( λ ; β ^ ) < τ H / λ for all λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) , automation is reached at finite AI capability strictly inside the feasible region. This contrasts with the Bayes-rational case, where the team weakly outperforms either modality and strict automation is never optimal; therefore, under correlation neglect, inevitable automation is a signature of behavioral misspecification, not of the underlying information structure.
Insight 3: (Complementarity Region) – The complementarity region narrows as overlap grows. For any fixed β ^ > 0 \hat{\beta}>0 β ^ > 0 , the complementarity region in the ( τ A , λ ) (\tau_A,\lambda) ( τ A , λ ) plane has the form τ A ∈ [ τ a u g ( λ ; β ^ ) , τ a u t o ( λ ; β ^ ) ] \tau_A\in[\tau_{\mathrm{aug}}(\lambda;\hat{\beta}),\tau_{\mathrm{auto}}(\lambda;\hat{\beta})] τ A ∈ [ τ aug ( λ ; β ^ ) , τ auto ( λ ; β ^ )] when both thresholds are positive. As λ \lambda λ rises: the augmentation threshold τ a u g \tau_{\mathrm{aug}} τ aug is strictly increasing: more overlap requires a more capable AI before the team beats the human alone; meanwhile, the automation threshold τ a u t o \tau_{\mathrm{auto}} τ auto is strictly decreasing: more overlap means a less capable AI suffices for AI alone to beat the team. Therefore, the complementarity region shrinks from both ends as λ \lambda λ grows, and at λ = 1 \lambda=1 λ = 1 collapses to the single tie point τ A = τ H \tau_A=\tau_H τ A = τ H .
Insight 4: Trust is a double-edged lever under correlation neglect. The worker’s trust β ^ \hat{\beta} β ^ scales how strongly she weights the AI signal, with opposing effects across the two phase boundaries.
Corollary 3.3 (Comparative statics of phase boundaries in trust). The phase boundaries are strictly increasing in β ^ \hat{\beta} β ^ for λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) : ∂ τ a u g ∂ β ^ > 0 \frac{\partial\tau_{\mathrm{aug}}}{\partial\hat{\beta}}>0 ∂ β ^ ∂ τ aug > 0 and ∂ τ a u t o ∂ β ^ > 0 \frac{\partial\tau_{\mathrm{auto}}}{\partial\hat{\beta}}>0 ∂ β ^ ∂ τ auto > 0 .
A worker who trusts the AI signal more benefits more from its marginal information but also incurs a larger double-counting penalty. This has opposite effects at the two boundaries. At the lower boundary, higher trust amplifies double-counting when the AI is weak and redundant, so a more capable AI is needed before assistance beats the human-alone benchmark. Similarly, higher trust will also make the lowest level of overlap needed for impairment, 1 − β ^ / 2 1-\hat{\beta}/2 1 − β ^ /2 , lower. At the upper boundary, higher trust also lets the human-AI team make better use of a strong AI signal, so AI-alone dominance is delayed. Thus undertrust is protective against harmful redundant AI but makes automation arrive sooner, while overtrust expands impairment but can preserve complementarity up to higher AI capability.
Accuracy as a function of AI capability. Figure 2 traces A H \mathcal{A}_H A H , A H A \mathcal{A}_{HA} A H A , A A I \mathcal{A}_{AI} A A I vs. τ A \tau_A τ A at three overlap levels while fixing β ^ = 1 \hat{\beta}=1 β ^ = 1 , τ H = 1 \tau_H=1 τ H = 1 . At low overlap, A H A \mathcal{A}_{HA} A H A rises monotonically and dominates A H \mathcal{A}_H A H until A A I \mathcal{A}_{AI} A A I overtakes it at τ a u t o \tau_{\mathrm{auto}} τ auto . At moderate overlap, A H A \mathcal{A}_{HA} A H A is non-monotone: it dips below A H \mathcal{A}_H A H for small τ A \tau_A τ A (impairment), crosses at τ a u g \tau_{\mathrm{aug}} τ aug , and is overtaken by A A I \mathcal{A}_{AI} A A I at τ a u t o \tau_{\mathrm{auto}} τ auto . At λ = 1 \lambda=1 λ = 1 , the complementarity interval collapses to the triple tie at τ A = τ H \tau_A=\tau_H τ A = τ H . The non-monotonicity is the signature of correlation neglect: weak AI hurts via double-counting, moderate AI helps once its marginal information dominates, strong AI eventually dominates as the worker becomes a noise channel.
Figure 2. Accuracy vs. AI capability τ A \tau_A τ A at three overlap levels λ ∈ { 0.3 , 0.9 , 1.0 } \lambda\in\{0.3,0.9,1.0\} λ ∈ { 0.3 , 0.9 , 1.0 } under correlation neglect with calibrated trust (α ^ = 0 \hat{\alpha}=0 α ^ = 0 , β ^ = 1 \hat{\beta}=1 β ^ = 1 , τ H = 1 \tau_H=1 τ H = 1 ). Curves: human alone A H \mathcal{A}_H A H (gray), team A H A \mathcal{A}_{HA} A H A (red), AI alone A A I \mathcal{A}_{AI} A A I (blue). Shaded background marks the welfare-optimal regime (gray: impairment, red: complementarity, blue: automation). As λ \lambda λ rises, the team curve flattens, the impairment region appears (panel b), and widens until the AI is fully redundant at τ A = τ H \tau_A=\tau_H τ A = τ H (panel c).
The design implications are threefold. First, AI systems serving correlation-neglect workers benefit from reducing effective overlap with human evidence, not just from increasing τ A \tau_A τ A : at fixed τ A \tau_A τ A the team SNR is strictly decreasing in λ \lambda λ (∂ r H A 2 / ∂ λ ∣ α ^ = 0 < 0 \left.\partial r^2_{HA}/\partial\lambda\right|_{\hat{\alpha}=0}<0 ∂ r H A 2 / ∂ λ α ^ = 0 < 0 for any β ^ > 0 \hat{\beta}>0 β ^ > 0 ), giving the principle of training AI to predict what humans miss rather than what they already know. Second, whether to provide ℓ A \ell_A ℓ A at all is a complementary lever: from Theorem 3.2 , the team is welfare-optimal only on τ A ∈ [ τ a u g ( λ ; β ^ ) , τ a u t o ( λ ; β ^ ) ] \tau_A\in[\tau_{\mathrm{aug}}(\lambda;\hat{\beta}),\tau_{\mathrm{auto}}(\lambda;\hat{\beta})] τ A ∈ [ τ aug ( λ ; β ^ ) , τ auto ( λ ; β ^ )] , so the optimal disclosure rule is to withhold ℓ A \ell_A ℓ A when τ A < τ a u g \tau_A<\tau_{\mathrm{aug}} τ A < τ aug , provide it when τ A ∈ [ τ a u g , τ a u t o ] \tau_A\in[\tau_{\mathrm{aug}},\tau_{\mathrm{auto}}] τ A ∈ [ τ aug , τ auto ] , and automate when τ A > τ a u t o \tau_A>\tau_{\mathrm{auto}} τ A > τ auto . Third, in the impairment region, behavioral interventions encouraging undertrust (β ^ < 1 \hat{\beta}<1 β ^ < 1 ) can improve welfare for correlation-neglect workers: β ^ < 1 \hat{\beta}<1 β ^ < 1 does not correct the neglect but dampens its double-counting cost (Corollary 3.3), so the conventional calibration target β ^ = 1 \hat{\beta}=1 β ^ = 1 is not welfare-optimal there.
Welfare Decomposition
The phase diagram characterizes which deployment mode yields the highest accuracy. We now consider a complementary question: how much accuracy does the worker lose relative to the Bayesian-rational benchmark, and how much of this loss is attributable to correlation neglect versus undertrust?
Write A ( α ^ , β ^ ) \mathcal{A}(\hat{\alpha},\hat{\beta}) A ( α ^ , β ^ ) for the worker’s accuracy under behavioral primitives α ^ \hat{\alpha} α ^ and β ^ \hat{\beta} β ^ , and let A B R = A ( 1 , 1 ) \mathcal{A}^{\mathrm{BR}}=\mathcal{A}(1,1) A BR = A ( 1 , 1 ) be the Bayesian rational benchmark. We define
T W L ( α ^ , β ^ ) : = A B R − A ( α ^ , β ^ ) , C N C ( α ^ ) : = A B R − A ( α ^ , 1 ) , U T C ( α ^ , β ^ ) : = A ( α ^ , 1 ) − A ( α ^ , β ^ ) . \begin{aligned}
\mathrm{TWL}(\hat{\alpha},\hat{\beta}) &:= \mathcal{A}^{\mathrm{BR}}-\mathcal{A}(\hat{\alpha},\hat{\beta}),\\
\mathrm{CNC}(\hat{\alpha}) &:= \mathcal{A}^{\mathrm{BR}}-\mathcal{A}(\hat{\alpha},1),\\
\mathrm{UTC}(\hat{\alpha},\hat{\beta}) &:= \mathcal{A}(\hat{\alpha},1)-\mathcal{A}(\hat{\alpha},\hat{\beta}).
\end{aligned} TWL ( α ^ , β ^ ) CNC ( α ^ ) UTC ( α ^ , β ^ ) := A BR − A ( α ^ , β ^ ) , := A BR − A ( α ^ , 1 ) , := A ( α ^ , 1 ) − A ( α ^ , β ^ ) .
Here, TWL is the worker’s total welfare loss compared to the BR benchmark, which subsequently decomposes into two types of loss, CNC and UTC. The correlation-neglect cost CNC measures the loss from imperfect correlation awareness when trust is calibrated. The undertrust cost UTC measures the gain from calibrating trust while holding the worker’s actual awareness level fixed. We have the following exact decomposition.
Proposition 3.4 (Welfare decomposition). Within the cue-pool feasible region, for every worker ( α ^ , β ^ ) ∈ [ 0 , 1 ] × R > 0 (\hat{\alpha}, \hat{\beta}) \in[0,1] \times\mathbb{R}_{>0} ( α ^ , β ^ ) ∈ [ 0 , 1 ] × R > 0 , we have TWL ( α ^ , β ^ ) = CNC ( α ^ ) + UTC ( α ^ , β ^ ) \operatorname{TWL}(\hat{\alpha}, \hat{\beta}) = \operatorname{CNC}(\hat{\alpha}) + \operatorname{UTC}(\hat{\alpha}, \hat{\beta}) TWL ( α ^ , β ^ ) = CNC ( α ^ ) + UTC ( α ^ , β ^ ) . The correlation-neglect component is nonnegative: CNC ( α ^ ) ≥ 0 \operatorname{CNC}(\hat{\alpha}) \ge0 CNC ( α ^ ) ≥ 0 . If β ^ ∈ ( 0 , 1 ] \hat{\beta} \in(0,1] β ^ ∈ ( 0 , 1 ] and τ A ≥ τ H \tau_A \ge\tau_H τ A ≥ τ H , the undertrust component is also nonnegative; it vanishes when β ^ = 1 \hat{\beta}=1 β ^ = 1 .
As mentioned before, the two types of loss correspond to two design levers. Calibrating trust first recovers UTC; correcting the remaining correlation misperception then recovers CNC. We will later link this welfare decomposition to our empirical study. Finally, we define the welfare gain relative to the unaided human,
GAP ( α ^ , β ^ ) : = A ( α ^ , β ^ ) − A H . \operatorname{GAP}(\hat{\alpha}, \hat{\beta}) := \mathcal{A}(\hat{\alpha}, \hat{\beta}) - A_H. GAP ( α ^ , β ^ ) := A ( α ^ , β ^ ) − A H .
It is used to measure the benefit of AI assistance relative to human-alone performance. It is also used in our empirical study.
Empirical Evaluation
We instantiate the framework on ChatBench [8 ] , a controlled MTurk study in which 643 workers answer multiple-choice questions from MMLU [13 ] across math, physics, and moral reasoning. Each worker is randomly assigned to one of two AI assistants (GPT-4o or Llama-3.1-8b) under an answer-first protocol: she records an unassisted answer with stated confidence, then sees the AI’s response and records a final answer. After matching the answer-first user-AI rows to their corresponding user-alone rows and restricting to pairs where the AI committed to a definite letter recommendation, we obtain 2,199 paired observations covering 315 workers and 389 questions. We construct ℓ H = logit ( p H ( c ) ) \ell_H = \operatorname{logit}(p_H(c)) ℓ H = logit ( p H ( c )) from worker confidence and ℓ A = logit ( p A m , q ) \ell_A = \operatorname{logit}(p_A^{m,q}) ℓ A = logit ( p A m , q ) from AI model m m m ’s empirical accuracy on question q q q across up to 50 generations. The binary outcome Y Y Y is the ground-truth letter. Appendix § 12 details sample construction.
AI assistant produces significant welfare gains; different AI models lead to different optimal deployment modes Workers go from A H = 0.612 A_H = 0.612 A H = 0.612 alone to A H A = 0.824 A_{HA} = 0.824 A H A = 0.824 with AI assistance, a pooled welfare gain of GAP = + 0.213 \operatorname{GAP} = +0.213 GAP = + 0.213 (95% CI [ 0.191 , 0.235 ] [0.191, 0.235] [ 0.191 , 0.235 ] , never negative in 1,000 bootstrap samples). The two deployments fall in different empirical regimes (Table 1 ). For GPT-4o, the AI alone outperforms the team (A A I = 0.884 > A H A = 0.851 A_{AI} = 0.884 > A_{HA} = 0.851 A A I = 0.884 > A H A = 0.851 , p = 0.002 p = 0.002 p = 0.002 ): GPT is firmly in the framework’s automation regime. For Llama-3.1-8b, A H A = 0.795 A_{HA} = 0.795 A H A = 0.795 and A A I = 0.787 A_{AI} = 0.787 A A I = 0.787 are statistically indistinguishable (p = 0.55 p = 0.55 p = 0.55 ), placing Llama at the boundary between complementarity and automation. The cross-deployment GAP gap of + 0.081 +0.081 + 0.081 is approximately 93% explained by capability composition (95% CI [ 68 % , 148 % ] [68\%, 148\%] [ 68% , 148% ] ); a likelihood-ratio test of cell-by-AI interactions gives χ 2 ( 4 ) = 0.96 \chi^2(4) = 0.96 χ 2 ( 4 ) = 0.96 , p = 0.92 p = 0.92 p = 0.92 . This validates the framework’s central qualitative claim that AI capability τ A \tau_A τ A governs deployment-level welfare while behavior is approximately invariant across deployments.
Table 1. Per-deployment summary on the Has-Rec sample. Regimes are classified by paired comparison of A H A A_{HA} A H A and A A I A_{AI} A A I on the same observations; see Appendix § 13 for test details.
Workers substantially undertrust the AI; strict correlation neglect is rejected. Inverting Lemma 2.4 from per-deployment logistic regression weights identifies β ^ = 0.285 \hat{\beta} = 0.285 β ^ = 0.285 (95% CI [ 0.246 , 0.322 ] [0.246, 0.322] [ 0.246 , 0.322 ] ): workers apply roughly a quarter of the Bayes-rationale weight to the AI signal. A second approach using cross-question variation to estimate λ \lambda λ directly gives β ^ = 0.254 \hat{\beta} = 0.254 β ^ = 0.254 , in close agreement; see Appendix § 13 . Both decisively reject β ^ = 1 \hat{\beta} = 1 β ^ = 1 . Strict correlation neglect (α ^ = 0 \hat{\alpha} = 0 α ^ = 0 ) is also rejected: the inversion’s residual sum of squares jumps 141-fold when α ^ = 0 \hat{\alpha} = 0 α ^ = 0 is forced, and direct λ ^ \hat{\lambda} λ ^ is positive in 100% of bootstrap samples for both deployments. The data does not pin down α ^ \hat{\alpha} α ^ to a narrow range but decisively rejects this corner case.
Overlap λ \lambda λ varies meaningfully across AIs and across subject domains. Direct λ \lambda λ estimation from cross-question variation gives two empirical patterns that connect back to the framework’s cue-pool picture of overlap (Proposition 2.2 ). First, λ \lambda λ differs significantly across the two AIs: λ ^ L l a m a = 0.127 \hat{\lambda}^{\mathrm{Llama}} = 0.127 λ ^ Llama = 0.127 exceeds λ ^ G P T = 0.071 \hat{\lambda}^{\mathrm{GPT}} = 0.071 λ ^ GPT = 0.071 by + 0.056 +0.056 + 0.056 (95% CI [0.015, 0.097], two-sided p ≈ 0.009 p \approx0.009 p ≈ 0.009 ). The smaller model’s evidence overlaps more with what humans access; GPT brings more genuinely novel information. Second, λ \lambda λ varies dramatically across subjects, pooled across both AIs: math λ ^ ≈ 0.20 \hat{\lambda} \approx0.20 λ ^ ≈ 0.20 , physics λ ^ ^ ≈ 0.04 \hat{\hat{\lambda}} \approx0.04 λ ^ ^ ≈ 0.04 , moral reasoning λ ^ ≈ 0 \hat{\lambda} \approx0 λ ^ ≈ 0 , with the same pattern holding within each AI separately. Domains where humans and AI rely on shared informational cues exhibit substantial overlap; domains where reasoning paths are more idiosyncratic exhibit essentially none. Proposition 2.2 characterizes overlap as a property of the human-domain pair under uniform cue sampling; the cross-AI heterogeneity indicates that real AIs’ training data is more complicated.
Trust calibration and design implications. § 3.3 splits the worker’s accuracy loss against the Bayes-rational benchmark into a correlation-neglect cost CNC and an undertrust cost UTC, with UTC the accuracy recovered by calibrating trust (Proposition 3.4). Here, we measure the empirical counterpart of UTC. On items where the worker’s unaided answer and the AI’s recommendation disagree, we replace her final choice with a simulated trust-calibrated rule that selects whichever answer is more likely correct given the same two signals used to identify the behavioral parameters, her stated confidence ℓ H \ell_H ℓ H and the AI’s question-level accuracy ℓ A \ell_A ℓ A . Here, the correctness probabilities are estimated by cross-validated logistic regression (Appendix § 16 ). Like UTC, the resulting gain isolates the trust channel: it changes how she relies on the AI. For GPT-4o, the simulated rule follows the AI on about 92% of disagreements, versus the real workers’ 70%, and raises team accuracy by 2.6 percentage points, from 0.851 to 0.877, close to the AI-alone accuracy of 0.884. So for GPT-4o, which lies in the automation regime, calibrating trust moves the team nearly to the AI-alone benchmark. For Llama-3.1-8b, which sits at the complementarity-automation boundary, the simulated rule instead does worse than the actual workers (−1.4 percentage points). The reason is informational rather than behavioral: the simulated rule sees only two coarse signals, whereas the worker also reads the AI’s specific answer and can judge its reasoning, which for the weaker AI evidently outweighs calibrating trust. Two design implications follow from the result. First, trust calibration pays off more where the AI is strong, so interventions that raise trust should be targeted to AI-strong cases rather than applied uniformly. Meanwhile, besides behavior calibration, the granularity of information the worker receives is equally important.
Extensions to other applications. The contribution of our empirical framework extends beyond ChatBench’s setup with MMLU questions, and can be applied to other domains such as medicine, law, or programming, where human and AI expertise are combined. Estimating the model’s primitives requires only the observables present in any binary classification setting: the human-alone report ℓ H \ell_H ℓ H , the AI-alone report ℓ A \ell_A ℓ A , the AI-assisted human’s report ℓ \ell ℓ , and the ground-truth label Y Y Y . An example is professional radiologists diagnosing from X-ray imagery with the aid of AI assistance as in Agarwal et al. [2023]: ground truth comes from an expert diagnostic panel, the human and AI signals are the prior-corrected logits of the human-alone and AI-alone probability reports, and the same class-conditional moments identify τ H , τ A , λ , α ^ , \tau_H, \tau_A, \lambda, \hat{\alpha}, τ H , τ A , λ , α ^ , and β ^ \hat{\beta} β ^ .
Broader impacts. Our framework characterizes when sharing an AI recommendation helps or harms a human decision-maker, and identifies two levers for improving deployed human-AI teams: calibrating trust and reducing information overlap. Positively, the phase diagram offers deployment guidance (withhold, assist, or automate) that can prevent the accuracy losses documented when capable AI is naively added to human workflows, and the overlap lever suggests training assistants to supply complementary rather than redundant evidence. Risks include over-reliance on a stylized two-signal model: the behavioral parameters are estimated from a specific crowdsourced setting (ChatBench/MMLU) and need not transfer to high-stakes domains such as medicine or hiring, where miscalibrated ‘automation’ recommendations could cause harm. Our prescriptions should be treated as a modeling-informed starting point for extensive domain-specific validation.
Conclusion
We build a model of AI-assisted classification in which a worker combines her own evidence with an AI’s through two behavioral primitives, correlation awareness and trust. We derive a closed-form phase diagram over capability and overlap that separates complementarity, impairment, and automation. We examine the framework on ChatBench, finding substantial undertrust, regimes decided by AI capability and overlap, and heterogeneous gains from trust calibration. The framework potentially applies to any setup of AI-assisted decision making where human-alone, AI-alone, AI-assisted assessment, and ground truth are all observed.
Behavioral modeling of trust miscalibration
A natural alternative to model trust miscalibration is to interpret trust as a subjective belief about the AI’s capability, replacing τ A \tau_A τ A by β ^ τ A \hat{\beta}\tau_A β ^ τ A in the worker’s perceived signal distribution. This interpretation is not appropriate for the present classification model. The AI’s message is assumed to be informationally equivalent to its log-likelihood ratio ℓ A \ell_A ℓ A , or equivalently to a calibrated posterior probability. Once ℓ A \ell_A ℓ A is observed, its Bayesian contribution to posterior log-odds is already encoded in its realized value.
To verify this, suppose first that the worker perceives the AI signal as conditionally independent of her own signal and believes that
ℓ A ∣ Y = 1 ∼ N ( τ ^ A , 2 τ ^ A ) , ℓ A ∣ Y = 0 ∼ N ( − τ ^ A , 2 τ ^ A ) . \ell_A \mid Y=1 \sim N(\hat{\tau}_A,2\hat{\tau}_A), \qquad\ell_A \mid Y=0 \sim N(-\hat{\tau}_A,2\hat{\tau}_A). ℓ A ∣ Y = 1 ∼ N ( τ ^ A , 2 τ ^ A ) , ℓ A ∣ Y = 0 ∼ N ( − τ ^ A , 2 τ ^ A ) .
Then the log-likelihood ratio implied by this subjective model is
log ϕ ( ℓ A ; τ ^ A , 2 τ ^ A ) ϕ ( ℓ A ; − τ ^ A , 2 τ ^ A ) = ℓ A , \log\frac{\phi(\ell_A;\hat{\tau}_A,2\hat{\tau}_A)}{\phi(\ell_A;-\hat{\tau}_A,2\hat{\tau}_A)}=\ell_A, log ϕ ( ℓ A ; − τ ^ A , 2 τ ^ A ) ϕ ( ℓ A ; τ ^ A , 2 τ ^ A ) = ℓ A ,
which is independent of the perceived capability τ ^ A \hat{\tau}_A τ ^ A . Thus, changing the worker’s perceived AI precision changes the ex ante distribution she assigns to ℓ A \ell_A ℓ A , but it does not change the Bayesian weight placed on ℓ A \ell_A ℓ A as long as the worker still believes expit ( ℓ A ) \operatorname{expit}(\ell_A) expit ( ℓ A ) is a calibrated forecast. In particular, under perceived independence, any subjective-capability model yields the decision statistic ℓ ^ = ℓ H + ℓ A \hat{\ell}=\ell_H+\ell_A ℓ ^ = ℓ H + ℓ A , so trust would be behaviorally unidentifiable.
For this reason, we model β ^ \hat{\beta} β ^ as a behavioral weight on the AI’s communicated likelihood-ratio contribution, rather than as a subjective precision. β ^ ≠ 1 \hat{\beta}\ne1 β ^ = 1 should be interpreted as a departure from treating the AI’s probability report as a fully trusted calibrated posterior. If β ^ < 1 \hat{\beta}<1 β ^ < 1 , the worker behaves as though the AI’s log-odds are too extreme and shrinks them toward zero; if β ^ > 1 \hat{\beta}>1 β ^ > 1 , she behaves as though the AI’s log-odds are too conservative and amplifies them.
More performance objectives
Log Loss
The expected log loss is L log : = E [ − log P ( Y ∣ ℓ ^ ) ] \mathcal{L}_{\log}:=\mathbb{E}[-\log P(Y\mid\hat{\ell})] L l o g := E [ − log P ( Y ∣ ℓ ^ )] , where P ( Y ∣ ℓ ^ ) = expit ( ( 2 Y − 1 ) ℓ ^ ) P(Y\mid\hat{\ell})=\operatorname{expit}((2Y-1)\hat{\ell}) P ( Y ∣ ℓ ^ ) = expit (( 2 Y − 1 ) ℓ ^ ) is the worker’s reported probability for the realized state. The log loss is the natural statistic when the worker’s output is a probability rather than a binary decision. Unlike the error rate, which only depends on whether the decision is correct, log loss also penalizes the confidence with which the worker reports a wrong answer.
Lemma B.1 (Expected log loss). Define
Ψ ( m , v ) : = E Z ∼ N ( 0 , 1 ) [ log ( 1 + e − ( m + v Z ) ) ] . \Psi(m,v):=\mathbb{E}_{Z\sim N(0,1)}\left[\log(1+e^{-(m+\sqrt{v}Z)})\right]. Ψ ( m , v ) := E Z ∼ N ( 0 , 1 ) [ log ( 1 + e − ( m + v Z ) ) ] . Under Assumption 2.3 and the symmetric prior, the worker reporting posterior probability p ^ = expit ( ℓ ^ ) \hat{p}=\operatorname{expit}(\hat{\ell}) p ^ = expit ( ℓ ^ ) incurs expected log loss
L log ( α ^ , β ^ ; τ H , τ A , λ ) = Ψ ( m H A , σ H A 2 ) , \mathcal{L}_{\log}(\hat{\alpha},\hat{\beta};\tau_H,\tau_A,\lambda)=\Psi(m_{HA},\sigma_{HA}^{2}), L l o g ( α ^ , β ^ ; τ H , τ A , λ ) = Ψ ( m H A , σ H A 2 ) , *where m H A = w H τ H + w A τ A m_{HA}=w_H\tau_H+w_A\tau_A m H A = w H τ H + w A τ A and σ H A 2 = 2 ( w H 2 τ H + w A 2 τ A + 2 w H w A λ ) \sigma_{HA}^{2}=2(w_H^{2}\tau_H+w_A^{2}\tau_A+2w_Hw_A\lambda) σ H A 2 = 2 ( w H 2 τ H + w A 2 τ A + 2 w H w A λ ) are the conditional mean and variance of ℓ ^ \hat{\ell} ℓ ^ given Y = 1 Y=1 Y = 1 , and the weights ( w H , w A ) (w_H,w_A) ( w H , w A ) are given by Lemma 2.4. The function Ψ \Psi Ψ admits no elementary closed form but is computable by numerical integration. The standalone log-losses are recovered as L H = Ψ ( τ H , 2 τ H ) \mathcal{L}_H=\Psi(\tau_H,2\tau_H) L H = Ψ ( τ H , 2 τ H ) and L A = Ψ ( τ A , 2 τ A ) \mathcal{L}_A=\Psi(\tau_A,2\tau_A) L A = Ψ ( τ A , 2 τ A ) .
The log loss differs from the error rate in a substantive way. The error rate scores only whether the binary decision Y ^ = 1 { ℓ ^ > 0 } \hat{Y}=1\{\hat{\ell}>0\} Y ^ = 1 { ℓ ^ > 0 } is correct; it ignores the magnitude ∣ ℓ ^ ∣ |\hat{\ell}| ∣ ℓ ^ ∣ . The log loss scores both the direction of the worker’s belief and her confidence in it. Two workers with the same correctness rate can have different log losses if their probability reports differ in how confidently wrong they are: a worker who reports p ^ = 0.99 \hat{p}=0.99 p ^ = 0.99 on an answer that turns out wrong pays more under log loss than a worker who reports p ^ = 0.6 \hat{p}=0.6 p ^ = 0.6 on the same wrong answer. The log loss therefore captures calibration of probability reports, while the error rate captures only point-estimate correctness.
Log loss is the relevant welfare objective when the worker’s task involves communicating probabilities rather than binary decisions: medical diagnosis with uncertainty estimates or any setting where downstream users act on the worker’s confidence and not just her hard recommendation. In binary-decision settings, where the worker’s output is a hard label, error rate is the appropriate welfare objective.
Proof of Lemma B.1. The worker’s posterior report is p ^ = expit ( ℓ ^ ) \hat{p} = \operatorname{expit}(\hat{\ell}) p ^ = expit ( ℓ ^ ) . The log loss for outcome Y Y Y is
L log ( Y , p ^ ) = − Y log p ^ − ( 1 − Y ) log ( 1 − p ^ ) = log ( 1 + e − ( 2 Y − 1 ) ℓ ^ ) . \mathcal{L}_{\log}(Y,\hat{p}) = -Y \log\hat{p} - (1-Y)\log(1-\hat{p}) = \log\left(1+e^{-(2Y-1)\hat{\ell}}\right). L l o g ( Y , p ^ ) = − Y log p ^ − ( 1 − Y ) log ( 1 − p ^ ) = log ( 1 + e − ( 2 Y − 1 ) ℓ ^ ) .
using log expit ( x ) = − log ( 1 + e − x ) \log\operatorname{expit}(x)=-\log(1+e^{-x}) log expit ( x ) = − log ( 1 + e − x ) and log ( 1 − expit ( x ) ) = − log ( 1 + e x ) \log(1-\operatorname{expit}(x))=-\log(1+e^{x}) log ( 1 − expit ( x )) = − log ( 1 + e x ) . Taking expectations over the joint distribution of ( ℓ H , ℓ A , Y ) (\ell_H,\ell_A,Y) ( ℓ H , ℓ A , Y ) from Lemma 2.1 ,
L log ( α ^ , β ^ ; τ H , τ A , λ ) = E [ log ( 1 + e − ( 2 Y − 1 ) ℓ ^ ) ] . \mathcal{L}_{\log}(\hat{\alpha},\hat{\beta};\tau_H,\tau_A,\lambda)=\mathbb{E}\left[\log\left(1+e^{-(2Y-1)\hat{\ell}}\right)\right]. L l o g ( α ^ , β ^ ; τ H , τ A , λ ) = E [ log ( 1 + e − ( 2 Y − 1 ) ℓ ^ ) ] .
From the proof of Lemma 3.1 , ℓ ^ ∣ Y ∼ N ( ( 2 Y − 1 ) m H A , σ H A 2 ) \hat{\ell}\mid Y\sim\mathcal{N}((2Y-1)m_{HA},\sigma_{HA}^{2}) ℓ ^ ∣ Y ∼ N (( 2 Y − 1 ) m H A , σ H A 2 ) , where m H A = w H τ H + w A τ A m_{HA}=w_H\tau_H+w_A\tau_A m H A = w H τ H + w A τ A and σ H A 2 = 2 ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) \sigma_{HA}^{2}=2(w_H^{2}\tau_H+w_A^{2}\tau_A+2w_Hw_A\tau_A\lambda) σ H A 2 = 2 ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) . By symmetry of the conditional distributions under Y = 1 Y=1 Y = 1 versus Y = 0 Y=0 Y = 0 and the symmetric prior, the two conditional log-loss expectations are equal, so the unconditional expectation reduces to a one-dimensional Gaussian integral:
L log = E Z ∼ N ( 0 , 1 ) [ log ( 1 + e − ( m H A + σ H A Z ) ) ] = Ψ ( m H A , σ H A 2 ) . \mathcal{L}_{\log}=\mathbb{E}_{Z\sim\mathcal{N}(0,1)}\left[\log\left(1+e^{-(m_{HA}+\sigma_{HA}Z)}\right)\right]=\Psi\left(m_{HA},\sigma_{HA}^{2}\right). L l o g = E Z ∼ N ( 0 , 1 ) [ log ( 1 + e − ( m H A + σ H A Z ) ) ] = Ψ ( m H A , σ H A 2 ) .
Switching Rate
The switching rate is S : = P ( Y ^ H A ≠ Y ^ H ) S:=P(\hat{Y}_{HA}\ne\hat{Y}_H) S := P ( Y ^ H A = Y ^ H ) , the probability that AI assistance changes the worker’s decision relative to her unaided choice Y ^ H = 1 { ℓ H > 0 } \hat{Y}_H=1\{\ell_H>0\} Y ^ H = 1 { ℓ H > 0 } . The switching rate is not a welfare objective but a behavioral one: it measures how often AI assistance changes the worker’s decision. When the data record whether workers revised their tentative answers, the switching rate is what the data directly measure.
Lemma B.2 (Switching rate). Under Assumption 2.3 and the symmetric prior, the probability that AI assistance changes the worker’s decision is
S ( α ^ , β ^ ; τ H , τ A , λ ) = P ( sgn ( ℓ ^ ) ≠ sgn ( ℓ H ) ) = Φ ( τ H / 2 ) + Φ ( r H A ) − 2 Φ 2 ( τ H / 2 , r H A ; ρ ) , S(\hat{\alpha},\hat{\beta};\tau_H,\tau_A,\lambda)=P\left(\operatorname{sgn}(\hat{\ell})\ne\operatorname{sgn}(\ell_H)\right)=\Phi\left(\sqrt{\tau_H}/2\right)+\Phi(r_{HA})-2\Phi_2\left(\sqrt{\tau_H}/2,r_{HA};\rho\right), S ( α ^ , β ^ ; τ H , τ A , λ ) = P ( sgn ( ℓ ^ ) = sgn ( ℓ H ) ) = Φ ( τ H /2 ) + Φ ( r H A ) − 2 Φ 2 ( τ H /2 , r H A ; ρ ) , where Φ 2 ( a , b ; r ) : = Pr ( Z 1 ≤ a , Z 2 ≤ b ) \Phi_2(a,b;r):=\Pr(Z_1\le a,Z_2\le b) Φ 2 ( a , b ; r ) := Pr ( Z 1 ≤ a , Z 2 ≤ b ) for ( Z 1 , Z 2 ) ∼ N ( 0 , Σ r ) (Z_1,Z_2)\sim\mathcal{N}(0,\Sigma_r) ( Z 1 , Z 2 ) ∼ N ( 0 , Σ r ) with unit variances and correlation r r r , r H A r_{HA} r H A is the team SNR from Lemma 3.1 , and the conditional correlation between ℓ H \ell_H ℓ H and ℓ ^ \hat{\ell} ℓ ^ given Y Y Y is
ρ = w H τ H + w A τ A λ τ H ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) . \rho=\frac{w_H\tau_H+w_A\tau_A\lambda}{\sqrt{\tau_H\left(w_H^2\tau_H+w_A^2\tau_A+2w_Hw_A\tau_A\lambda\right)}}. ρ = τ H ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) w H τ H + w A τ A λ .
The switching rate is a behavioral objective: it measures how often AI assistance changes the worker’s binary decision, holding the worker’s beliefs ( α ^ , β ^ ) (\hat{\alpha},\hat{\beta}) ( α ^ , β ^ ) and deployment parameters fixed. The closed form has a structural reading. The first two terms, Φ ( τ H / 2 ) \Phi(\sqrt{\tau_H}/2) Φ ( τ H /2 ) and Φ ( r H A ) \Phi(r_{HA}) Φ ( r H A ) , are the marginal correctness probabilities of the human alone and the team. The third, Φ 2 ( τ H / 2 , r H A ; ρ ) \Phi_2(\sqrt{\tau_H}/2,r_{HA};\rho) Φ 2 ( τ H /2 , r H A ; ρ ) , is the joint probability that both the human alone and the team are correct, with the conditional correlation ρ \rho ρ measuring how aligned their decisions are. The switching rate equals the sum of marginals minus twice the joint, isolating exactly the disagreement events—the cases where AI assistance changed the worker’s binary decision. When the data record whether the worker’s tentative answer was revised after seeing the AI, the empirical switching rate is what the data measure directly, providing a model-fit check beyond the accuracy comparison.
Exploitable-to-dangerous ratio
Another useful deployment-level diagnostic is the ratio of probabilities of two events: exploitable cases where the human is wrong but the AI is right, and dangerous cases where the human is right but the AI is wrong. Define the standardized signals W H : = ( 2 Y − 1 ) ℓ H / 2 τ H W_H:=(2Y-1)\ell_H/\sqrt{2\tau_H} W H := ( 2 Y − 1 ) ℓ H / 2 τ H and W A : = ( 2 Y − 1 ) ℓ A / 2 τ A W_A:=(2Y-1)\ell_A/\sqrt{2\tau_A} W A := ( 2 Y − 1 ) ℓ A / 2 τ A , so that W B > 0 W_B>0 W B > 0 means agent B B B is correct. Under Lemma 2.1 ,
( W H , W A ) ∼ N ( ( τ H / 2 τ A / 2 ) , ( 1 ρ H A ρ H A 1 ) ) unconditionally , ρ H A : = λ τ A / τ H . (W_H,W_A)\sim\mathcal{N}\left(\begin{pmatrix}\sqrt{\tau_H/2}\\ \sqrt{\tau_A/2}\end{pmatrix},\begin{pmatrix}1&\rho_{HA}\\ \rho_{HA}&1\end{pmatrix}\right)\ \text{unconditionally},\qquad\rho_{HA}:=\lambda\sqrt{\tau_A/\tau_H}. ( W H , W A ) ∼ N ( ( τ H /2 τ A /2 ) , ( 1 ρ H A ρ H A 1 ) ) unconditionally , ρ H A := λ τ A / τ H .
The two probabilities are
Pr ( e x p l o i t a b l e ) = Pr ( W H ≤ 0 , W A > 0 ) = Φ 2 ( − τ H / 2 , τ A / 2 ; − ρ H A ) \Pr(\mathrm{exploitable})=\Pr(W_H\leq0,W_A>0)=\Phi_2\left(-\sqrt{\tau_H/2},\sqrt{\tau_A/2};-\rho_{HA}\right) Pr ( exploitable ) = Pr ( W H ≤ 0 , W A > 0 ) = Φ 2 ( − τ H /2 , τ A /2 ; − ρ H A )
Pr ( d a n g e r o u s ) = Pr ( W H > 0 , W A ≤ 0 ) = Φ 2 ( τ H / 2 , − τ A / 2 ; − ρ H A ) \Pr(\mathrm{dangerous})=\Pr(W_H>0,W_A\leq0)=\Phi_2\left(\sqrt{\tau_H/2},-\sqrt{\tau_A/2};-\rho_{HA}\right) Pr ( dangerous ) = Pr ( W H > 0 , W A ≤ 0 ) = Φ 2 ( τ H /2 , − τ A /2 ; − ρ H A )
where the negative correlation in Φ 2 \Phi_2 Φ 2 reflects the standard form (each event is rewritten as a ≤ \leq ≤ inequality, flipping one of the two variables and hence the sign of their correlation). The ratio Pr ( e x p l o i t a b l e ) / Pr ( d a n g e r o u s ) \Pr(\mathrm{exploitable})/\Pr(\mathrm{dangerous}) Pr ( exploitable ) / Pr ( dangerous ) exceeds 1 if and only if τ A > τ H \tau_A>\tau_H τ A > τ H : more capable AIs generate more exploitable than dangerous cases, providing a deployment-level handle on whether AI assistance has the potential to help.
Proofs for Section 2
Proof of Lemma 2.1 . Marginal moments. Conditional on Y = 1 Y=1 Y = 1 , X k ∼ N ( μ 1 , σ 2 ) X_k\sim\mathcal{N}(\mu_1,\sigma^2) X k ∼ N ( μ 1 , σ 2 ) , so
E [ L L R ( X k ) ∣ Y = 1 ] = μ 1 − μ 0 σ 2 ( μ 1 − μ 1 + μ 0 2 ) = θ \mathbb{E}[\mathrm{LLR}(X_k)\mid Y=1]=\frac{\mu_1-\mu_0}{\sigma^2}\left(\mu_1-\frac{\mu_1+\mu_0}{2}\right)=\theta E [ LLR ( X k ) ∣ Y = 1 ] = σ 2 μ 1 − μ 0 ( μ 1 − 2 μ 1 + μ 0 ) = θ
and similarly E [ L L R ( X k ) ∣ Y = 0 ] = − θ \mathbb{E}[\mathrm{LLR}(X_k)\mid Y=0]=-\theta E [ LLR ( X k ) ∣ Y = 0 ] = − θ . Since L L R ( X k ) \mathrm{LLR}(X_k) LLR ( X k ) is a linear transform of X k X_k X k with slope ( μ 1 − μ 0 ) / σ 2 (\mu_1-\mu_0)/\sigma^2 ( μ 1 − μ 0 ) / σ 2 ,
Var ( L L R ( X k ) ∣ Y ) = ( μ 1 − μ 0 σ 2 ) 2 σ 2 = 2 θ . \operatorname{Var}(\mathrm{LLR}(X_k)\mid Y)=\left(\frac{\mu_1-\mu_0}{\sigma^2}\right)^2\sigma^2=2\theta. Var ( LLR ( X k ) ∣ Y ) = ( σ 2 μ 1 − μ 0 ) 2 σ 2 = 2 θ .
Thus L L R ( X k ) ∣ Y ∼ N ( ( 2 Y − 1 ) θ , 2 θ ) \mathrm{LLR}(X_k)\mid Y\sim\mathcal{N}((2Y-1)\theta,2\theta) LLR ( X k ) ∣ Y ∼ N (( 2 Y − 1 ) θ , 2 θ ) . By conditional independence of cues given Y Y Y and the additivity ℓ B = ∑ k ∈ X B L L R ( X k ) \ell_B=\sum_{k\in\mathcal{X}_B}\mathrm{LLR}(X_k) ℓ B = ∑ k ∈ X B LLR ( X k ) ,
E [ ℓ B ∣ Y ] = ( 2 Y − 1 ) τ B , Var ( ℓ B ∣ Y ) = 2 τ B . \mathbb{E}[\ell_B\mid Y]=(2Y-1)\tau_B,\qquad\operatorname{Var}(\ell_B\mid Y)=2\tau_B. E [ ℓ B ∣ Y ] = ( 2 Y − 1 ) τ B , Var ( ℓ B ∣ Y ) = 2 τ B .
Conditional covariance. By conditional independence, Cov ( L L R ( X j ) , L L R ( X k ) ∣ Y ) = 2 θ ⋅ 1 { j = k } \operatorname{Cov}(\mathrm{LLR}(X_j),\mathrm{LLR}(X_k)\mid Y)=2\theta\cdot\mathbf{1}\{j=k\} Cov ( LLR ( X j ) , LLR ( X k ) ∣ Y ) = 2 θ ⋅ 1 { j = k } . Hence
Cov ( ℓ H , ℓ A ∣ Y ) = ∑ j ∈ X H ∑ k ∈ X A Cov ( L L R ( X j ) , L L R ( X k ) ∣ Y ) = 2 θ n ∩ = 2 τ A λ . \operatorname{Cov}(\ell_H,\ell_A\mid Y)=\sum_{j\in\mathcal{X}_H}\sum_{k\in\mathcal{X}_A}\operatorname{Cov}(\mathrm{LLR}(X_j),\mathrm{LLR}(X_k)\mid Y)=2\theta n_{\cap}=2\tau_A\lambda. Cov ( ℓ H , ℓ A ∣ Y ) = j ∈ X H ∑ k ∈ X A ∑ Cov ( LLR ( X j ) , LLR ( X k ) ∣ Y ) = 2 θ n ∩ = 2 τ A λ .
Joint Gaussianity. Every linear combination a ℓ H + b ℓ A = ∑ k c k L L R ( X k ) a\ell_H+b\ell_A=\sum_k c_k\mathrm{LLR}(X_k) a ℓ H + b ℓ A = ∑ k c k LLR ( X k ) with c k ∈ { a , b , a + b , 0 } c_k\in\{a,b,a+b,0\} c k ∈ { a , b , a + b , 0 } is a sum of independent Gaussians, so ( ℓ H , ℓ A ) (\ell_H,\ell_A) ( ℓ H , ℓ A ) is jointly Gaussian given Y Y Y .
Regression interpretation of λ \lambda λ . The population regression coefficient of ℓ H \ell_H ℓ H on ℓ A \ell_A ℓ A given Y Y Y is Cov ( ℓ H , ℓ A ∣ Y ) / Var ( ℓ A ∣ Y ) = 2 τ A λ / ( 2 τ A ) = λ \operatorname{Cov}(\ell_H,\ell_A\mid Y)/\operatorname{Var}(\ell_A\mid Y)=2\tau_A\lambda/(2\tau_A)=\lambda Cov ( ℓ H , ℓ A ∣ Y ) / Var ( ℓ A ∣ Y ) = 2 τ A λ / ( 2 τ A ) = λ .
Proof of Proposition 2.2 . Fix X H \mathcal{X}_H X H with ∣ X H ∣ = n H |\mathcal{X}_H|=n_H ∣ X H ∣ = n H , and let X A \mathcal{X}_A X A be a uniformly random subset of size n A n_A n A drawn without replacement from { 1 , … , N } \{1,\ldots,N\} { 1 , … , N } . Write the overlap count as n ∩ = ∑ i ∈ X H 1 { i ∈ X A } n_{\cap}=\sum_{i\in\mathcal{X}_H}\mathbf{1}\{i\in\mathcal{X}_A\} n ∩ = ∑ i ∈ X H 1 { i ∈ X A } . By symmetry of the uniform draw, P ( i ∈ X A ) = n A / N P(i\in\mathcal{X}_A)=n_A/N P ( i ∈ X A ) = n A / N for each i i i . Hence
E [ n ∩ ] = n H ⋅ n A N , E [ λ ] = E [ n ∩ ] n A = n H N , \mathbb{E}[n_{\cap}]=n_H\cdot\frac{n_A}{N},\qquad\mathbb{E}[\lambda]=\frac{\mathbb{E}[n_{\cap}]}{n_A}=\frac{n_H}{N}, E [ n ∩ ] = n H ⋅ N n A , E [ λ ] = n A E [ n ∩ ] = N n H ,
independent of n A n_A n A .
Proof of Lemma 2.4 . We construct the rule in two steps, corresponding to parts (i) and (ii) of Assumption 2.3 .
Step 1: Bayesian rule under perceived overlap. The worker’s belief about the joint distribution of ( ℓ H , ℓ A , Y ) (\ell_H,\ell_A,Y) ( ℓ H , ℓ A , Y ) replaces the true overlap λ \lambda λ with the perceived overlap α ^ λ \hat{\alpha}\lambda α ^ λ in Lemma 2.1 , retaining the true capabilities ( τ H , τ A ) (\tau_H,\tau_A) ( τ H , τ A ) . Her believed conditional joint distribution is therefore Gaussian with mean ( 2 Y − 1 ) ( τ H , τ A ) ⊤ (2Y-1)(\tau_H,\tau_A)^{\top} ( 2 Y − 1 ) ( τ H , τ A ) ⊤ and covariance
Σ ^ = ( 2 τ H 2 τ A α ^ λ 2 τ A α ^ λ 2 τ A ) (1) \widehat{\Sigma}=
\begin{pmatrix}
2\tau_H & 2\tau_A\hat{\alpha}\lambda\\
2\tau_A\hat{\alpha}\lambda& 2\tau_A
\end{pmatrix} \tag*{(1)} Σ = ( 2 τ H 2 τ A α ^ λ 2 τ A α ^ λ 2 τ A ) (1)
Under the symmetric prior, the worker’s believed posterior log-odds equals the believed log-likelihood ratio, which by the standard Gaussian formula is
ℓ ^ B a y e s = 2 μ ^ 1 ⊤ Σ ^ − 1 ( ℓ H , ℓ A ) ⊤ , μ ^ 1 : = ( τ H , τ A ) ⊤ . \hat{\ell}^{\mathrm{Bayes}}=2\hat{\mu}_1^{\top}\hat{\Sigma}^{-1}(\ell_H,\ell_A)^{\top},\qquad\hat{\mu}_1:=(\tau_H,\tau_A)^{\top}. ℓ ^ Bayes = 2 μ ^ 1 ⊤ Σ ^ − 1 ( ℓ H , ℓ A ) ⊤ , μ ^ 1 := ( τ H , τ A ) ⊤ .
The determinant is det Σ ^ = 4 τ A ( τ H − τ A α ^ 2 λ 2 ) = : 4 τ A D \det\hat{\Sigma}=4\tau_A(\tau_H-\tau_A\hat{\alpha}^2\lambda^2)=:4\tau_A D det Σ ^ = 4 τ A ( τ H − τ A α ^ 2 λ 2 ) =: 4 τ A D , where D : = τ H − τ A α ^ 2 λ 2 D:=\tau_H-\tau_A\hat{\alpha}^2\lambda^2 D := τ H − τ A α ^ 2 λ 2 . The inverse is
Σ ^ − 1 = 1 2 D ( 1 − α ^ λ − α ^ λ τ H / τ A ) . \hat{\Sigma}^{-1}=\frac{1}{2D}\begin{pmatrix}1&-\hat{\alpha}\lambda\\-\hat{\alpha}\lambda&\tau_H/\tau_A\end{pmatrix}. Σ ^ − 1 = 2 D 1 ( 1 − α ^ λ − α ^ λ τ H / τ A ) .
The row vector 2 μ ^ 1 ⊤ Σ ^ − 1 2\hat{\mu}_1^{\top}\hat{\Sigma}^{-1} 2 μ ^ 1 ⊤ Σ ^ − 1 has components
τ H − τ A α ^ λ D and τ H ( 1 − α ^ λ ) D , \frac{\tau_H-\tau_A\hat{\alpha}\lambda}{D}\quad\text{and}\quad\frac{\tau_H(1-\hat{\alpha}\lambda)}{D}, D τ H − τ A α ^ λ and D τ H ( 1 − α ^ λ ) ,
giving the Bayesian-rule weights
w H B a y e s ( α ^ ) = τ H − τ A α ^ λ τ H − τ A α ^ 2 λ 2 , w A B a y e s ( α ^ ) = τ H ( 1 − α ^ λ ) τ H − τ A α ^ 2 λ 2 . w_H^{\mathrm{Bayes}}(\hat{\alpha})=\frac{\tau_H-\tau_A\hat{\alpha}\lambda}{\tau_H-\tau_A\hat{\alpha}^2\lambda^2},\qquad
w_A^{\mathrm{Bayes}}(\hat{\alpha})=\frac{\tau_H(1-\hat{\alpha}\lambda)}{\tau_H-\tau_A\hat{\alpha}^2\lambda^2}. w H Bayes ( α ^ ) = τ H − τ A α ^ 2 λ 2 τ H − τ A α ^ λ , w A Bayes ( α ^ ) = τ H − τ A α ^ 2 λ 2 τ H ( 1 − α ^ λ ) .
Step 2: Trust discount on the AI signal. By part (ii) of Assumption 2.3, the worker applies a multiplicative trust factor β ^ \hat{\beta} β ^ to the AI signal contribution, leaving the human-side weight unchanged. Her decision rule is
ℓ ^ = w H B a y e s ( α ^ ) ⋅ ℓ H + β ^ ⋅ w A B a y e s ( α ^ ) ⋅ ℓ A , \hat{\ell}=w_H^{\mathrm{Bayes}}(\hat{\alpha})\cdot\ell_H+\hat{\beta}\cdot w_A^{\mathrm{Bayes}}(\hat{\alpha})\cdot\ell_A, ℓ ^ = w H Bayes ( α ^ ) ⋅ ℓ H + β ^ ⋅ w A Bayes ( α ^ ) ⋅ ℓ A ,
which is linear in ( ℓ H , ℓ A ) (\ell_H,\ell_A) ( ℓ H , ℓ A ) and yields the weights stated in the lemma. □
Proofs for Section 3
Proof of Lemma 3.1. The worker’s decision statistic is ℓ ^ = w H ℓ H + w A ℓ A \hat{\ell}=w_H\ell_H+w_A\ell_A ℓ ^ = w H ℓ H + w A ℓ A , a linear combination of the jointly Gaussian pair ( ℓ H , ℓ A ) (\ell_H,\ell_A) ( ℓ H , ℓ A ) from Lemma 2.1 . Hence ℓ ^ ∣ Y \hat{\ell}\mid Y ℓ ^ ∣ Y is Gaussian. We compute its conditional mean and standard deviation, then derive the accuracy.
Conditional mean. Using E [ ℓ H ∣ Y ] = ( 2 Y − 1 ) τ H \mathbb{E}[\ell_H\mid Y]=(2Y-1)\tau_H E [ ℓ H ∣ Y ] = ( 2 Y − 1 ) τ H and E [ ℓ A ∣ Y ] = ( 2 Y − 1 ) τ A \mathbb{E}[\ell_A\mid Y]=(2Y-1)\tau_A E [ ℓ A ∣ Y ] = ( 2 Y − 1 ) τ A from Lemma 2.1 ,
E [ ℓ ^ ∣ Y ] = ( 2 Y − 1 ) ( w H τ H + w A τ A ) . \mathbb{E}[\hat{\ell}\mid Y]=(2Y-1)(w_H\tau_H+w_A\tau_A). E [ ℓ ^ ∣ Y ] = ( 2 Y − 1 ) ( w H τ H + w A τ A ) .
Conditional variance. Using Var ( ℓ H ∣ Y ) = 2 τ H \operatorname{Var}(\ell_H\mid Y)=2\tau_H Var ( ℓ H ∣ Y ) = 2 τ H , Var ( ℓ A ∣ Y ) = 2 τ A \operatorname{Var}(\ell_A\mid Y)=2\tau_A Var ( ℓ A ∣ Y ) = 2 τ A , and Cov ( ℓ H , ℓ A ∣ Y ) = 2 τ A λ \operatorname{Cov}(\ell_H,\ell_A\mid Y)=2\tau_A\lambda Cov ( ℓ H , ℓ A ∣ Y ) = 2 τ A λ from Lemma 2.1 ,
Var ( ℓ ^ ∣ Y ) = w H 2 ⋅ 2 τ H + w A 2 ⋅ 2 τ A + 2 w H w A ⋅ 2 τ A λ = 2 ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) . \operatorname{Var}(\hat{\ell}\mid Y)=w_H^2\cdot2\tau_H+w_A^2\cdot2\tau_A+2w_Hw_A\cdot2\tau_A\lambda=2(w_H^2\tau_H+w_A^2\tau_A+2w_Hw_A\tau_A\lambda). Var ( ℓ ^ ∣ Y ) = w H 2 ⋅ 2 τ H + w A 2 ⋅ 2 τ A + 2 w H w A ⋅ 2 τ A λ = 2 ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) .
Accuracy. Classification is correct when ℓ ^ \hat{\ell} ℓ ^ has the same sign as ( 2 Y − 1 ) (2Y-1) ( 2 Y − 1 ) . Standardizing ℓ ^ \hat{\ell} ℓ ^ given Y = 1 Y=1 Y = 1 :
P ( ℓ ^ > 0 ∣ Y = 1 ) = P ( Z > − w H τ H + w A τ A Var ( ℓ ^ ∣ Y ) ) = Φ ( w H τ H + w A τ A Var ( ℓ ^ ∣ Y ) ) = Φ ( r H A ) , P(\hat{\ell}>0\mid Y=1)=P\left(Z>-\frac{w_H\tau_H+w_A\tau_A}{\sqrt{\operatorname{Var}(\hat{\ell}\mid Y)}}\right)=\Phi\left(\frac{w_H\tau_H+w_A\tau_A}{\sqrt{\operatorname{Var}(\hat{\ell}\mid Y)}}\right)=\Phi(r_{HA}), P ( ℓ ^ > 0 ∣ Y = 1 ) = P Z > − Var ( ℓ ^ ∣ Y ) w H τ H + w A τ A = Φ Var ( ℓ ^ ∣ Y ) w H τ H + w A τ A = Φ ( r H A ) ,
where Z ∼ N ( 0 , 1 ) Z\sim N(0,1) Z ∼ N ( 0 , 1 ) and r H A = ( w H τ H + w A τ A ) / Var ( ℓ ^ ∣ Y ) r_{HA}=(w_H\tau_H+w_A\tau_A)/\sqrt{\operatorname{Var}(\hat{\ell}\mid Y)} r H A = ( w H τ H + w A τ A ) / Var ( ℓ ^ ∣ Y ) . By symmetry of the conditional distributions, P ( ℓ ^ < 0 ∣ Y = 0 ) = Φ ( r H A ) P(\hat{\ell}<0\mid Y=0)=\Phi(r_{HA}) P ( ℓ ^ < 0 ∣ Y = 0 ) = Φ ( r H A ) as well. Under the symmetric prior, the unconditional accuracy is
A H A = 1 2 P ( ℓ ^ > 0 ∣ Y = 1 ) + 1 2 P ( ℓ ^ < 0 ∣ Y = 0 ) = Φ ( r H A ) . A_{HA}=\frac{1}{2}P(\hat{\ell}>0\mid Y=1)+\frac{1}{2}P(\hat{\ell}<0\mid Y=0)=\Phi(r_{HA}). A H A = 2 1 P ( ℓ ^ > 0 ∣ Y = 1 ) + 2 1 P ( ℓ ^ < 0 ∣ Y = 0 ) = Φ ( r H A ) .
Proof of Lemma B.2. AI assistance changes the worker’s decision when sgn ( ℓ ^ ) ≠ sgn ( ℓ H ) \operatorname{sgn}(\hat{\ell})\ne\operatorname{sgn}(\ell_H) sgn ( ℓ ^ ) = sgn ( ℓ H ) :
S = P ( ℓ H > 0 , ℓ ^ < 0 ) + P ( ℓ H < 0 , ℓ ^ > 0 ) . S=P(\ell_H>0,\hat{\ell}<0)+P(\ell_H<0,\hat{\ell}>0). S = P ( ℓ H > 0 , ℓ ^ < 0 ) + P ( ℓ H < 0 , ℓ ^ > 0 ) .
The pair ( ℓ H , ℓ ^ ) (\ell_H,\hat{\ell}) ( ℓ H , ℓ ^ ) is jointly Gaussian conditional on Y Y Y , with conditional means ( ( 2 Y − 1 ) τ H , ( 2 Y − 1 ) m H A ) ((2Y-1)\tau_H,(2Y-1)m_{HA}) (( 2 Y − 1 ) τ H , ( 2 Y − 1 ) m H A ) , conditional variances ( 2 τ H , σ H A 2 ) (2\tau_H,\sigma_{HA}^2) ( 2 τ H , σ H A 2 ) where m H A , σ H A 2 m_{HA},\sigma_{HA}^2 m H A , σ H A 2 are as in the proof of Lemma 3.1, and conditional covariance
Cov ( ℓ H , ℓ ^ ∣ Y ) = w H ⋅ 2 τ H + w A ⋅ 2 τ A λ = 2 ( w H τ H + w A τ A λ ) . \operatorname{Cov}(\ell_H,\hat{\ell}\mid Y)=w_H\cdot2\tau_H+w_A\cdot2\tau_A\lambda=2(w_H\tau_H+w_A\tau_A\lambda). Cov ( ℓ H , ℓ ^ ∣ Y ) = w H ⋅ 2 τ H + w A ⋅ 2 τ A λ = 2 ( w H τ H + w A τ A λ ) .
The conditional correlation between ℓ H \ell_H ℓ H and ℓ ^ \hat{\ell} ℓ ^ given Y Y Y is therefore
ρ = Cov ( ℓ H , ℓ ^ ∣ Y ) Var ( ℓ H ∣ Y ) Var ( ℓ ^ ∣ Y ) = w H τ H + w A τ A λ τ H ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) . \rho= \frac{\operatorname{Cov}(\ell_H,\hat{\ell}\mid Y)}{\sqrt{\operatorname{Var}(\ell_H\mid Y)\operatorname{Var}(\hat{\ell}\mid Y)}} = \frac{w_H\tau_H+w_A\tau_A\lambda}{\sqrt{\tau_H\left(w_H^2\tau_H+w_A^2\tau_A+2w_Hw_A\tau_A\lambda\right)}}. ρ = Var ( ℓ H ∣ Y ) Var ( ℓ ^ ∣ Y ) Cov ( ℓ H , ℓ ^ ∣ Y ) = τ H ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) w H τ H + w A τ A λ .
By symmetry of the conditional distributions under Y = 1 Y=1 Y = 1 versus Y = 0 Y=0 Y = 0 and the symmetric prior, S S S equals twice the conditional probability of the disagreement event under Y = 1 Y=1 Y = 1 . Standardizing ℓ H \ell_H ℓ H and ℓ ^ \hat{\ell} ℓ ^ to mean zero and unit variance under Y = 1 Y=1 Y = 1 :
ℓ H > 0 ⟺ Z H > − τ H / 2 , \ell_H > 0 \Longleftrightarrow Z_H > -\sqrt{\tau_H}/2, ℓ H > 0 ⟺ Z H > − τ H /2 ,
ℓ ^ > 0 ⟺ Z H A > − r H A , \hat{\ell} > 0 \Longleftrightarrow Z_{HA} > -r_{HA}, ℓ ^ > 0 ⟺ Z H A > − r H A ,
where ( Z H , Z H A ) (Z_H,Z_{HA}) ( Z H , Z H A ) is jointly standard normal with correlation ρ \rho ρ and r H A = m H A / σ H A r_{HA}=m_{HA}/\sigma_{HA} r H A = m H A / σ H A from Lemma 3.1. Using the identity P ( Z 1 > a , Z 2 < b ) = Φ ( b ) − Φ 2 ( a , b ; ρ ) P(Z_1>a,Z_2<b)=\Phi(b)-\Phi_2(a,b;\rho) P ( Z 1 > a , Z 2 < b ) = Φ ( b ) − Φ 2 ( a , b ; ρ ) for jointly standard normal ( Z 1 , Z 2 ) (Z_1,Z_2) ( Z 1 , Z 2 ) with correlation ρ \rho ρ , and similarly for the symmetric event,
P ( ℓ H > 0 , ℓ ^ < 0 ∣ Y = 1 ) = Φ ( − r H A ) − Φ 2 ( − τ H / 2 , − r H A ; ρ ) , P(\ell_H>0,\hat{\ell}<0\mid Y=1)=\Phi(-r_{HA})-\Phi_2(-\sqrt{\tau_H}/2,-r_{HA};\rho), P ( ℓ H > 0 , ℓ ^ < 0 ∣ Y = 1 ) = Φ ( − r H A ) − Φ 2 ( − τ H /2 , − r H A ; ρ ) ,
P ( ℓ H < 0 , ℓ ^ > 0 ∣ Y = 1 ) = Φ ( − τ H / 2 ) − Φ 2 ( − r H A , − τ H / 2 ; ρ ) . P(\ell_H<0,\hat{\ell}>0\mid Y=1)=\Phi(-\sqrt{\tau_H}/2)-\Phi_2(-r_{HA},-\sqrt{\tau_H}/2;\rho). P ( ℓ H < 0 , ℓ ^ > 0 ∣ Y = 1 ) = Φ ( − τ H /2 ) − Φ 2 ( − r H A , − τ H /2 ; ρ ) .
Adding and using Φ ( − x ) = 1 − Φ ( x ) \Phi(-x)=1-\Phi(x) Φ ( − x ) = 1 − Φ ( x ) and the symmetry Φ 2 ( − a , − b ; ρ ) = Φ 2 ( a , b ; ρ ) − Φ ( a ) − Φ ( b ) + 1 \Phi_2(-a,-b;\rho)=\Phi_2(a,b;\rho)-\Phi(a)-\Phi(b)+1 Φ 2 ( − a , − b ; ρ ) = Φ 2 ( a , b ; ρ ) − Φ ( a ) − Φ ( b ) + 1 (from rotation-invariance of the standard bivariate normal),
S = Φ ( τ H / 2 ) + Φ ( r H A ) − 2 Φ 2 ( τ H / 2 , r H A ; ρ ) . S=\Phi(\sqrt{\tau_H}/2)+\Phi(r_{HA})-2\Phi_2(\sqrt{\tau_H}/2,r_{HA};\rho). S = Φ ( τ H /2 ) + Φ ( r H A ) − 2 Φ 2 ( τ H /2 , r H A ; ρ ) .
Proof of Theorem 3.2. Under α ^ = 0 \hat{\alpha}=0 α ^ = 0 and the symmetric prior, the worker’s decision statistic is ℓ ^ = ℓ H + β ^ ℓ A \hat{\ell}=\ell_H+\hat{\beta}\ell_A ℓ ^ = ℓ H + β ^ ℓ A . The three modality accuracies are determined by their respective signal-to-noise ratios via Lemma 3.1. Since Φ \Phi Φ is strictly increasing, comparing accuracies reduces to comparing SNRs. Denote the three modality SNRs by r H , r A , r H A r_H,r_A,r_{HA} r H , r A , r H A . Direct substitution into Lemma 3.1 gives
r H = τ H / 2 , r A = τ A / 2 , r H A = τ H + β ^ τ A 2 ( τ H + β ^ ( β ^ + 2 λ ) τ A ) . r_H=\sqrt{\tau_H}/2,\qquad r_A=\sqrt{\tau_A}/2,\qquad r_{HA}=\frac{\tau_H+\hat{\beta}\tau_A}{\sqrt{2(\tau_H+\hat{\beta}(\hat{\beta}+2\lambda)\tau_A)}}. r H = τ H /2 , r A = τ A /2 , r H A = 2 ( τ H + β ^ ( β ^ + 2 λ ) τ A ) τ H + β ^ τ A .
where r H A r_{HA} r H A uses w H = 1 , w A = β ^ w_H=1,w_A=\hat{\beta} w H = 1 , w A = β ^ from Lemma 2.4 at α ^ = 0 \hat{\alpha}=0 α ^ = 0 .
The proof proceeds in five steps: derive the augmentation threshold (Step 1), the automation threshold (Step 2), the human-vs-AI comparison (Step 3), partition the deployment space by overlap regime (Step 4), and verify monotonicity of the thresholds (Step 5).
Step 1: Augmentation threshold. We derive the condition r H A > r H r_{HA}>r_H r H A > r H . Since both are positive, this is equivalent to r H A 2 > r H 2 r_{HA}^2>r_H^2 r H A 2 > r H 2 , i.e.,
( τ H + β ^ τ A ) 2 2 ( τ H + β ^ ( β ^ + 2 λ ) τ A ) > τ H 2 . \frac{(\tau_H+\hat{\beta}\tau_A)^2}{2(\tau_H+\hat{\beta}(\hat{\beta}+2\lambda)\tau_A)}>\frac{\tau_H}{2}. 2 ( τ H + β ^ ( β ^ + 2 λ ) τ A ) ( τ H + β ^ τ A ) 2 > 2 τ H .
The denominator τ H + β ^ ( β ^ + 2 λ ) τ A > 0 \tau_H+\hat{\beta}(\hat{\beta}+2\lambda)\tau_A>0 τ H + β ^ ( β ^ + 2 λ ) τ A > 0 for all ( τ H , τ A ) > 0 (\tau_H,\tau_A)>0 ( τ H , τ A ) > 0 and λ ≥ 0 \lambda\geq0 λ ≥ 0 . Multiplying through and expanding:
( τ H + β ^ τ A ) 2 > τ H ( τ H + β ^ ( β ^ + 2 λ ) τ A ) (\tau_H+\hat{\beta}\tau_A)^2>\tau_H(\tau_H+\hat{\beta}(\hat{\beta}+2\lambda)\tau_A) ( τ H + β ^ τ A ) 2 > τ H ( τ H + β ^ ( β ^ + 2 λ ) τ A )
⟺ τ H 2 + 2 β ^ τ H τ A + β ^ 2 τ A 2 > τ H 2 + β ^ 2 τ H τ A + 2 β ^ λ τ H τ A \Longleftrightarrow\tau_H^2+2\hat{\beta}\tau_H\tau_A+\hat{\beta}^2\tau_A^2>\tau_H^2+\hat{\beta}^2\tau_H\tau_A+2\hat{\beta}\lambda\tau_H\tau_A ⟺ τ H 2 + 2 β ^ τ H τ A + β ^ 2 τ A 2 > τ H 2 + β ^ 2 τ H τ A + 2 β ^ λ τ H τ A
⟺ β ^ τ A [ β ^ τ A + ( 2 − β ^ − 2 λ ) τ H ] > 0. \Longleftrightarrow\hat{\beta}\tau_A\left[\hat{\beta}\tau_A+(2-\hat{\beta}-2\lambda)\tau_H\right]>0. ⟺ β ^ τ A [ β ^ τ A + ( 2 − β ^ − 2 λ ) τ H ] > 0.
Since β ^ τ A > 0 \hat{\beta}\tau_A>0 β ^ τ A > 0 , this reduces to β ^ τ A > ( β ^ + 2 λ − 2 ) τ H \hat{\beta}\tau_A>(\hat{\beta}+2\lambda-2)\tau_H β ^ τ A > ( β ^ + 2 λ − 2 ) τ H , equivalently τ A > τ a u g ( λ ; β ^ ) : = ( ( β ^ + 2 λ − 2 ) / β ^ ) τ H \tau_A>\tau_{\mathrm{aug}}(\lambda;\hat{\beta}):=((\hat{\beta}+2\lambda-2)/\hat{\beta})\tau_H τ A > τ aug ( λ ; β ^ ) := (( β ^ + 2 λ − 2 ) / β ^ ) τ H . The threshold is positive iff β ^ + 2 λ − 2 > 0 \hat{\beta}+2\lambda-2>0 β ^ + 2 λ − 2 > 0 , i.e., λ > 1 − β ^ / 2 \lambda>1-\hat{\beta}/2 λ > 1 − β ^ /2 . When λ ≤ 1 − β ^ / 2 \lambda\leq1-\hat{\beta}/2 λ ≤ 1 − β ^ /2 , the threshold is non-positive and r H A > r H r_{HA}>r_H r H A > r H holds for every τ A > 0 \tau_A>0 τ A > 0 .
Step 2: Automation threshold. We derive the condition r H A > r A r_{HA}>r_A r H A > r A . By the same squaring,
( τ H + β ^ τ A ) 2 2 ( τ H + β ^ ( β ^ + 2 λ ) τ A ) > τ A 2 . \frac{(\tau_H+\hat{\beta}\tau_A)^2}{2(\tau_H+\hat{\beta}(\hat{\beta}+2\lambda)\tau_A)}>\frac{\tau_A}{2}. 2 ( τ H + β ^ ( β ^ + 2 λ ) τ A ) ( τ H + β ^ τ A ) 2 > 2 τ A .
and multiplying through gives
( τ H + β ^ τ A ) 2 > τ A ( τ H + β ^ ( β ^ + 2 λ ) τ A ) (\tau_H+\hat{\beta}\tau_A)^2>\tau_A(\tau_H+\hat{\beta}(\hat{\beta}+2\lambda)\tau_A) ( τ H + β ^ τ A ) 2 > τ A ( τ H + β ^ ( β ^ + 2 λ ) τ A )
τ H 2 + 2 β ^ τ H τ A + β ^ 2 τ A 2 > τ H τ A + β ^ 2 τ A 2 + 2 β ^ λ τ A 2 \tau_H^2+2\hat{\beta}\tau_H\tau_A+\hat{\beta}^2\tau_A^2>\tau_H\tau_A+\hat{\beta}^2\tau_A^2+2\hat{\beta}\lambda\tau_A^2 τ H 2 + 2 β ^ τ H τ A + β ^ 2 τ A 2 > τ H τ A + β ^ 2 τ A 2 + 2 β ^ λ τ A 2
τ H 2 + ( 2 β ^ − 1 ) τ H τ A − 2 β ^ λ τ A 2 > 0. ( ⋆ ) \tau_H^2+(2\hat{\beta}-1)\tau_H\tau_A-2\hat{\beta}\lambda\tau_A^2>0.\qquad(\star) τ H 2 + ( 2 β ^ − 1 ) τ H τ A − 2 β ^ λ τ A 2 > 0. ( ⋆ )
For λ > 0 \lambda>0 λ > 0 , ( ⋆ ) (\star) ( ⋆ ) is a quadratic in τ A \tau_A τ A with leading coefficient − 2 β ^ λ < 0 -2\hat{\beta}\lambda<0 − 2 β ^ λ < 0 (downward parabola). At τ A = 0 \tau_A=0 τ A = 0 , the LHS equals τ H 2 > 0 \tau_H^2>0 τ H 2 > 0 ; as τ A → ∞ \tau_A\to\infty τ A → ∞ , the LHS tends to − ∞ -\infty − ∞ . By continuity, the quadratic has a unique positive root τ a u t o ( λ ; β ^ ) \tau_{\mathrm{auto}}(\lambda;\hat{\beta}) τ auto ( λ ; β ^ ) , with ( ⋆ ) (\star) ( ⋆ ) holding iff τ A < τ a u t o \tau_A<\tau_{\mathrm{auto}} τ A < τ auto . The quadratic formula gives
τ a u t o ( λ ; β ^ ) = ( 2 β ^ − 1 ) + ( 2 β ^ − 1 ) 2 + 8 β ^ λ 4 β ^ λ τ H . \tau_{\mathrm{auto}}(\lambda;\hat{\beta})=
\frac{(2\hat{\beta}-1)+\sqrt{(2\hat{\beta}-1)^2+8\hat{\beta}\lambda}}{4\hat{\beta}\lambda}\,\tau_H. τ auto ( λ ; β ^ ) = 4 β ^ λ ( 2 β ^ − 1 ) + ( 2 β ^ − 1 ) 2 + 8 β ^ λ τ H .
where the positive sign is selected because the product of the two roots is − τ H 2 / ( 2 β ^ λ ) < 0 -\tau_H^2/(2\hat{\beta}\lambda)<0 − τ H 2 / ( 2 β ^ λ ) < 0 , forcing one positive and one negative root, and we take the positive one.
For λ = 0 \lambda=0 λ = 0 , ( ⋆ ) (\star) ( ⋆ ) reduces to τ H 2 + ( 2 β ^ − 1 ) τ H τ A > 0 \tau_H^2+(2\hat{\beta}-1)\tau_H\tau_A>0 τ H 2 + ( 2 β ^ − 1 ) τ H τ A > 0 , equivalently τ H + ( 2 β ^ − 1 ) τ A > 0 \tau_H+(2\hat{\beta}-1)\tau_A>0 τ H + ( 2 β ^ − 1 ) τ A > 0 . If β ^ ≥ 1 / 2 \hat{\beta}\ge1/2 β ^ ≥ 1/2 , both terms are non-negative and the condition holds for every τ A > 0 \tau_A>0 τ A > 0 , so τ a u t o ( 0 ; β ^ ) = + ∞ \tau_{\mathrm{auto}}(0;\hat{\beta})=+\infty τ auto ( 0 ; β ^ ) = + ∞ . If β ^ < 1 / 2 \hat{\beta}<1/2 β ^ < 1/2 , the condition becomes τ A < τ H / ( 1 − 2 β ^ ) \tau_A<\tau_H/(1-2\hat{\beta}) τ A < τ H / ( 1 − 2 β ^ ) , giving τ a u t o ( 0 ; β ^ ) = τ H / ( 1 − 2 β ^ ) \tau_{\mathrm{auto}}(0;\hat{\beta})=\tau_H/(1-2\hat{\beta}) τ auto ( 0 ; β ^ ) = τ H / ( 1 − 2 β ^ ) .
Step 3: Comparing r H r_H r H and $r_A. Since r H = τ H / 2 r_H=\sqrt{\tau_H/2} r H = τ H /2 and r A = τ A / 2 r_A=\sqrt{\tau_A/2} r A = τ A /2 , r H > r A ⟺ τ H > τ A r_H>r_A\Longleftrightarrow\tau_H>\tau_A r H > r A ⟺ τ H > τ A .
Step 4: Region partitioning. For each λ \lambda λ , the three score relationships partition τ A > 0 \tau_A>0 τ A > 0 into the three regimes.
Low overlap, 0 ≤ λ ≤ 1 − β ^ / 2 0\le\lambda\le1-\hat{\beta}/2 0 ≤ λ ≤ 1 − β ^ /2 (requires β ^ ≤ 2 \hat{\beta}\le2 β ^ ≤ 2 ): Step 1 gives r H A > r H r_{HA}>r_H r H A > r H for every τ A > 0 \tau_A>0 τ A > 0 , so H H H is never the top modality. We claim τ a u t o > τ H \tau_{\mathrm{auto}}>\tau_H τ auto > τ H in this regime. Evaluate ( ⋆ ) (\star) ( ⋆ ) at τ A = τ H \tau_A=\tau_H τ A = τ H : the LHS becomes 2 β ^ ( 1 − λ ) τ H 2 > 0 2\hat{\beta}(1-\lambda)\tau_H^2>0 2 β ^ ( 1 − λ ) τ H 2 > 0 for λ < 1 \lambda<1 λ < 1 , so r H A > r A r_{HA}>r_A r H A > r A at τ A = τ H \tau_A=\tau_H τ A = τ H , giving τ H < τ a u t o \tau_H<\tau_{\mathrm{auto}} τ H < τ auto . The choice between H A HA H A and A I AI A I then follows from Step 2:
Moderate overlap, 1 − β ^ / 2 < λ < 1 1-\hat{\beta}/2<\lambda<1 1 − β ^ /2 < λ < 1 : Step 1 gives a positive augmentation threshold τ a u g > 0 \tau_{\mathrm{aug}}>0 τ aug > 0 . We show τ a u g < τ H < τ a u t o \tau_{\mathrm{aug}}<\tau_H<\tau_{\mathrm{auto}} τ aug < τ H < τ auto . The first inequality follows from β ^ + 2 λ − 2 < β ^ \hat{\beta}+2\lambda-2<\hat{\beta} β ^ + 2 λ − 2 < β ^ (since λ < 1 \lambda<1 λ < 1 ), so τ a u g = ( β ^ + 2 λ − 2 ) / β ^ ⋅ τ H < τ H \tau_{\mathrm{aug}}=(\hat{\beta}+2\lambda-2)/\hat{\beta}\cdot\tau_H<\tau_H τ aug = ( β ^ + 2 λ − 2 ) / β ^ ⋅ τ H < τ H . For the second, evaluate ( ⋆ ) (\star) ( ⋆ ) at τ A = τ H \tau_A=\tau_H τ A = τ H : the LHS becomes
τ H 2 + ( 2 β ^ − 1 ) τ H 2 − 2 β ^ λ τ H 2 = 2 β ^ ( 1 − λ ) τ H 2 > 0 \tau_H^2+(2\hat{\beta}-1)\tau_H^2-2\hat{\beta}\lambda\tau_H^2=2\hat{\beta}(1-\lambda)\tau_H^2>0 τ H 2 + ( 2 β ^ − 1 ) τ H 2 − 2 β ^ λ τ H 2 = 2 β ^ ( 1 − λ ) τ H 2 > 0
for λ < 1 \lambda<1 λ < 1 . So r H A > r A r_{HA}>r_A r H A > r A at τ A = τ H \tau_A=\tau_H τ A = τ H , meaning τ H < τ a u t o \tau_H<\tau_{\mathrm{auto}} τ H < τ auto . The three regimes follow:
High overlap, λ = 1 \lambda=1 λ = 1 : substituting into τ a u g \tau_{\mathrm{aug}} τ aug , τ a u g ( 1 ; β ^ ) = ( ( β ^ + 2 − 2 ) / β ^ ) τ H = τ H \tau_{\mathrm{aug}}(1;\hat{\beta})=((\hat{\beta}+2-2)/\hat{\beta})\tau_H=\tau_H τ aug ( 1 ; β ^ ) = (( β ^ + 2 − 2 ) / β ^ ) τ H = τ H . Substituting into τ a u t o \tau_{\mathrm{auto}} τ auto , the radical equals ( 2 β ^ − 1 ) 2 + 8 β ^ = ( 2 β ^ + 1 ) 2 = 2 β ^ + 1 \sqrt{(2\hat{\beta}-1)^2+8\hat{\beta}}=\sqrt{(2\hat{\beta}+1)^2}=2\hat{\beta}+1 ( 2 β ^ − 1 ) 2 + 8 β ^ = ( 2 β ^ + 1 ) 2 = 2 β ^ + 1 (since β ^ > 0 \hat{\beta}>0 β ^ > 0 gives 2 β ^ + 1 > 0 2\hat{\beta}+1>0 2 β ^ + 1 > 0 ), so τ a u t o ( 1 ; β ^ ) = ( ( 2 β ^ − 1 ) + ( 2 β ^ + 1 ) ) / ( 4 β ^ ) ⋅ τ H = τ H \tau_{\mathrm{auto}}(1;\hat{\beta})=((2\hat{\beta}-1)+(2\hat{\beta}+1))/(4\hat{\beta})\cdot\tau_H=\tau_H τ auto ( 1 ; β ^ ) = (( 2 β ^ − 1 ) + ( 2 β ^ + 1 )) / ( 4 β ^ ) ⋅ τ H = τ H . Both thresholds equal τ H \tau_H τ H at λ = 1 \lambda=1 λ = 1 , and the complementarity region collapses to the single point τ A = τ H \tau_A=\tau_H τ A = τ H . At this point, the three score relationships give r H = r A = r H A = τ H / 2 r_H=r_A=r_{HA}=\sqrt{\tau_H/2} r H = r A = r H A = τ H /2 (a triple tie). For τ A < τ H \tau_A<\tau_H τ A < τ H : r H > r H A r_H>r_{HA} r H > r H A (Step 1, since τ a u g = τ H \tau_{\mathrm{aug}}=\tau_H τ aug = τ H and τ A < τ a u g \tau_A<\tau_{\mathrm{aug}} τ A < τ aug ) and r H > r A r_H>r_A r H > r A (Step 3), so H H H wins. For τ A > τ H \tau_A>\tau_H τ A > τ H : r A > r H A r_A>r_{HA} r A > r H A (Step 2, since τ a u t o = τ H \tau_{\mathrm{auto}}=\tau_H τ auto = τ H and τ A > τ a u t o \tau_A>\tau_{\mathrm{auto}} τ A > τ auto ) and r A > r H r_A>r_H r A > r H (Step 3), so A I AI A I wins.
Step 5: Monotonicity of thresholds. For the augmentation threshold,
∂ τ a u g ∂ λ = 2 τ H β ^ > 0 , \frac{\partial\tau_{\mathrm{aug}}}{\partial\lambda}=\frac{2\tau_H}{\hat{\beta}}>0, ∂ λ ∂ τ aug = β ^ 2 τ H > 0 ,
so τ a u g \tau_{\mathrm{aug}} τ aug is strictly increasing in λ \lambda λ on its positive domain ( 1 − β ^ / 2 , 1 ] (1-\hat{\beta}/2,1] ( 1 − β ^ /2 , 1 ] .
For the automation threshold, write τ a u t o ( λ ; β ^ ) / τ H = f ( λ ) / g ( λ ) \tau_{\mathrm{auto}}(\lambda;\hat{\beta})/\tau_H=f(\lambda)/g(\lambda) τ auto ( λ ; β ^ ) / τ H = f ( λ ) / g ( λ ) with f ( λ ) = ( 2 β ^ − 1 ) + S ( λ ) f(\lambda)=(2\hat{\beta}-1)+S(\lambda) f ( λ ) = ( 2 β ^ − 1 ) + S ( λ ) , g ( λ ) = 4 β ^ λ g(\lambda)=4\hat{\beta}\lambda g ( λ ) = 4 β ^ λ , and S ( λ ) : = ( 2 β ^ − 1 ) 2 + 8 β ^ λ S(\lambda):=\sqrt{(2\hat{\beta}-1)^2+8\hat{\beta}\lambda} S ( λ ) := ( 2 β ^ − 1 ) 2 + 8 β ^ λ . Direct differentiation:
f ′ ( λ ) = 4 β ^ S ( λ ) , g ′ ( λ ) = 4 β ^ , f'(\lambda)=\frac{4\hat{\beta}}{S(\lambda)},\qquad g'(\lambda)=4\hat{\beta}, f ′ ( λ ) = S ( λ ) 4 β ^ , g ′ ( λ ) = 4 β ^ ,
and by the quotient rule,
d d λ f g = f ′ g − f g ′ g 2 = ( 4 β ^ / S ) ( 4 β ^ λ ) − ( 4 β ^ ) f 16 β ^ 2 λ 2 = 4 β ^ λ − f ⋅ S 4 β ^ λ 2 ⋅ S . \frac{d}{d\lambda}\frac{f}{g}=\frac{f'g-fg'}{g^2}=\frac{(4\hat{\beta}/S)(4\hat{\beta}\lambda)-(4\hat{\beta})f}{16\hat{\beta}^2\lambda^2}=\frac{4\hat{\beta}\lambda-f\cdot S}{4\hat{\beta}\lambda^2\cdot S}. d λ d g f = g 2 f ′ g − f g ′ = 16 β ^ 2 λ 2 ( 4 β ^ / S ) ( 4 β ^ λ ) − ( 4 β ^ ) f = 4 β ^ λ 2 ⋅ S 4 β ^ λ − f ⋅ S .
The denominator is positive, so the sign of the derivative equals the sign of 4 β ^ λ − f S 4\hat{\beta}\lambda-fS 4 β ^ λ − f S . Computing f S fS f S using S 2 = ( 2 β ^ − 1 ) 2 + 8 β ^ λ S^2=(2\hat{\beta}-1)^2+8\hat{\beta}\lambda S 2 = ( 2 β ^ − 1 ) 2 + 8 β ^ λ :
f S = ( ( 2 β ^ − 1 ) + S ) S = ( 2 β ^ − 1 ) S + S 2 = ( 2 β ^ − 1 ) S + ( 2 β ^ − 1 ) 2 + 8 β ^ λ . fS=((2\hat{\beta}-1)+S)S=(2\hat{\beta}-1)S+S^2=(2\hat{\beta}-1)S+(2\hat{\beta}-1)^2+8\hat{\beta}\lambda. f S = (( 2 β ^ − 1 ) + S ) S = ( 2 β ^ − 1 ) S + S 2 = ( 2 β ^ − 1 ) S + ( 2 β ^ − 1 ) 2 + 8 β ^ λ .
Hence
4 β ^ λ − f S = − ( 2 β ^ − 1 ) S − ( 2 β ^ − 1 ) 2 − 4 β ^ λ . 4\hat{\beta}\lambda-fS=-(2\hat{\beta}-1)S-(2\hat{\beta}-1)^2-4\hat{\beta}\lambda. 4 β ^ λ − f S = − ( 2 β ^ − 1 ) S − ( 2 β ^ − 1 ) 2 − 4 β ^ λ .
We show this is strictly negative for λ > 0 \lambda>0 λ > 0 in two cases.
Case β ^ ≥ 1 / 2 \hat{\beta}\ge1/2 β ^ ≥ 1/2 : all three terms on the RHS are non-positive (since S > 0 S>0 S > 0 , ( 2 β ^ − 1 ) 2 ≥ 0 (2\hat{\beta}-1)^2\ge0 ( 2 β ^ − 1 ) 2 ≥ 0 , 4 β ^ λ > 0 4\hat{\beta}\lambda>0 4 β ^ λ > 0 ), and at least the last is strictly negative, so the sum is strictly negative.
Case β ^ < 1 / 2 \hat{\beta}<1/2 β ^ < 1/2 : − ( 2 β ^ − 1 ) > 0 -(2\hat{\beta}-1)>0 − ( 2 β ^ − 1 ) > 0 , so the first term is positive. We must verify that ∣ 2 β ^ − 1 ∣ ⋅ S < ( 2 β ^ − 1 ) 2 + 4 β ^ λ |2\hat{\beta}-1|\cdot S<(2\hat{\beta}-1)^2+4\hat{\beta}\lambda ∣2 β ^ − 1∣ ⋅ S < ( 2 β ^ − 1 ) 2 + 4 β ^ λ . Both sides are positive; squaring and using S 2 = ( 2 β ^ − 1 ) 2 + 8 β ^ λ S^2=(2\hat{\beta}-1)^2+8\hat{\beta}\lambda S 2 = ( 2 β ^ − 1 ) 2 + 8 β ^ λ , the inequality reduces (with u : = ( 2 β ^ − 1 ) 2 u:=(2\hat{\beta}-1)^2 u := ( 2 β ^ − 1 ) 2 and v : = 4 β ^ λ v:=4\hat{\beta}\lambda v := 4 β ^ λ ) to
u ( u + 2 v ) < ( u + v ) 2 = u 2 + 2 u v + v 2 ⟺ 0 < v 2 , u(u+2v)<(u+v)^2=u^2+2uv+v^2\quad\Longleftrightarrow\quad0<v^2, u ( u + 2 v ) < ( u + v ) 2 = u 2 + 2 uv + v 2 ⟺ 0 < v 2 ,
which holds for λ > 0 \lambda>0 λ > 0 .
In both cases, 4 β ^ λ − f S < 0 4\hat{\beta}\lambda-fS<0 4 β ^ λ − f S < 0 for λ > 0 \lambda>0 λ > 0 , so ∂ τ a u t o / ∂ λ < 0 \partial\tau_{\mathrm{auto}}/\partial\lambda<0 ∂ τ auto / ∂ λ < 0 on ( 0 , 1 ] (0,1] ( 0 , 1 ] . Hence τ a u t o \tau_{\mathrm{auto}} τ auto is strictly decreasing in λ \lambda λ on ( 0 , 1 ] (0,1] ( 0 , 1 ] .
The coincidence τ a u g ( 1 ; β ^ ) = τ a u t o ( 1 ; β ^ ) = τ H \tau_{\mathrm{aug}}(1;\hat{\beta})=\tau_{\mathrm{auto}}(1;\hat{\beta})=\tau_H τ aug ( 1 ; β ^ ) = τ auto ( 1 ; β ^ ) = τ H is verified in Step 4 (high-overlap case).
Proof of Proposition 3.4. Step 1: General split. Adding and subtracting A ( α ^ , 1 ) \mathcal{A}(\hat{\alpha},1) A ( α ^ , 1 ) inside T W L \mathrm{TWL} TWL gives
T W L ( α ^ , β ^ ) = A B R − A ( α ^ , β ^ ) = [ A B R − A ( α ^ , 1 ) ] + [ A ( α ^ , 1 ) − A ( α ^ , β ^ ) ] . \mathrm{TWL}(\hat{\alpha},\hat{\beta})=\mathcal{A}^{\mathrm{BR}}-\mathcal{A}(\hat{\alpha},\hat{\beta})=[\mathcal{A}^{\mathrm{BR}}-\mathcal{A}(\hat{\alpha},1)]+[\mathcal{A}(\hat{\alpha},1)-\mathcal{A}(\hat{\alpha},\hat{\beta})]. TWL ( α ^ , β ^ ) = A BR − A ( α ^ , β ^ ) = [ A BR − A ( α ^ , 1 )] + [ A ( α ^ , 1 ) − A ( α ^ , β ^ )] .
The first bracket is C N C ( α ^ ) \mathrm{CNC}(\hat{\alpha}) CNC ( α ^ ) by definition; the second is U T C ( α ^ , β ^ ) \mathrm{UTC}(\hat{\alpha},\hat{\beta}) UTC ( α ^ , β ^ ) .
Step 2: Sign of CNC. The Bayes-rational SNR r ∗ r^* r ∗ is defined as the maximum of r H A ( α ^ , β ^ ) r_{HA}(\hat{\alpha},\hat{\beta}) r H A ( α ^ , β ^ ) over all weight choices, so r ∗ ≥ r H A ( α ^ , 1 ) r^*\ge r_{HA}(\hat{\alpha},1) r ∗ ≥ r H A ( α ^ , 1 ) for any deployment, giving C N C ( α ^ ) ≥ 0 \mathrm{CNC}(\hat{\alpha})\ge0 CNC ( α ^ ) ≥ 0 .
Step 3: Sign of UTC for β ^ ∈ ( 0 , 1 ] \hat{\beta}\in(0,1] β ^ ∈ ( 0 , 1 ] and τ A ≥ τ H \tau_A\ge\tau_H τ A ≥ τ H . Fix α ^ ∈ [ 0 , 1 ] \hat{\alpha}\in[0,1] α ^ ∈ [ 0 , 1 ] and write u : = w H ( α ^ ) u:=w_H(\hat{\alpha}) u := w H ( α ^ ) and v : = w A ( α ^ , 1 ) v:=w_A(\hat{\alpha},1) v := w A ( α ^ , 1 ) . Within the cue-pool feasible region, u , v ≥ 0 u,v\ge0 u , v ≥ 0 and τ H − τ A λ ≥ 0 \tau_H-\tau_A\lambda\ge0 τ H − τ A λ ≥ 0 . The worker’s SNR is
r H A ( α ^ , β ^ ) = u τ H + β ^ v τ A 2 ( u 2 τ H + β ^ 2 v 2 τ A + 2 u β ^ v τ A λ ) . r_{HA}(\hat{\alpha},\hat{\beta})=\frac{u\tau_H+\hat{\beta}v\tau_A}{\sqrt{2(u^2\tau_H+\hat{\beta}^2v^2\tau_A+2u\hat{\beta}v\tau_A\lambda)}}. r H A ( α ^ , β ^ ) = 2 ( u 2 τ H + β ^ 2 v 2 τ A + 2 u β ^ v τ A λ ) u τ H + β ^ v τ A .
Differentiating r H A 2 r_{HA}^2 r H A 2 with respect to β ^ \hat{\beta} β ^ gives a nonnegative multiple of the linear function g ( β ^ ) = u τ H ( 1 − λ ) − β ^ v ( τ H − τ A λ ) g(\hat{\beta})=u\tau_H(1-\lambda)-\hat{\beta}v(\tau_H-\tau_A\lambda) g ( β ^ ) = u τ H ( 1 − λ ) − β ^ v ( τ H − τ A λ ) . Substituting the weights from Lemma 2.4 gives
g ( 1 ) = τ H λ ( 1 − α ^ ) ( τ A − τ H ) τ H − τ A α ^ 2 λ 2 ≥ 0. g(1)=\frac{\tau_H\lambda(1-\hat{\alpha})(\tau_A-\tau_H)}{\tau_H-\tau_A\hat{\alpha}^2\lambda^2}\ge0. g ( 1 ) = τ H − τ A α ^ 2 λ 2 τ H λ ( 1 − α ^ ) ( τ A − τ H ) ≥ 0.
For β ^ ∈ ( 0 , 1 ] \hat{\beta} \in(0,1] β ^ ∈ ( 0 , 1 ] , we therefore have g ( β ^ ) = g ( 1 ) + ( 1 − β ^ ) ν ( τ H − τ A λ ) ≥ 0 g(\hat{\beta}) = g(1) + (1-\hat{\beta})\nu(\tau_H-\tau_A\lambda) \ge0 g ( β ^ ) = g ( 1 ) + ( 1 − β ^ ) ν ( τ H − τ A λ ) ≥ 0 . Thus r H A ( α ^ , ⋅ ) r_{HA}(\hat{\alpha},\cdot) r H A ( α ^ , ⋅ ) is nondecreasing on ( 0 , 1 ] (0,1] ( 0 , 1 ] , so r H A ( α ^ , β ^ ) ≤ r H A ( α ^ , 1 ) r_{HA}(\hat{\alpha},\hat{\beta}) \le r_{HA}(\hat{\alpha},1) r H A ( α ^ , β ^ ) ≤ r H A ( α ^ , 1 ) , hence UTC ( α ^ , β ^ ) ≥ 0 \operatorname{UTC}(\hat{\alpha},\hat{\beta}) \ge0 UTC ( α ^ , β ^ ) ≥ 0 ; it vanishes when β ^ = 1 \hat{\beta}=1 β ^ = 1 . The degenerate case τ H = τ A \tau_H=\tau_A τ H = τ A and λ = 1 \lambda=1 λ = 1 follows by continuity, since the two signals coincide.
Generalization of the cue-pool construction
First, the assumption of equal per-cue capability can be relaxed while retaining conditional independence and Gaussian signals. Suppose
X k ∣ Y = y ∼ N ( μ y k , σ k 2 ) , θ k : = ( μ 1 k − μ 0 k ) 2 2 σ k 2 . X_k \mid Y=y \sim\mathcal{N}(\mu_{yk},\sigma_k^2), \qquad\theta_k := \frac{(\mu_{1k}-\mu_{0k})^2}{2\sigma_k^2}. X k ∣ Y = y ∼ N ( μ y k , σ k 2 ) , θ k := 2 σ k 2 ( μ 1 k − μ 0 k ) 2 .
where σ k 2 > 0 \sigma_k^2>0 σ k 2 > 0 is common across states but may differ across cues. This permits some cues to be highly informative and others nearly uninformative. Define
τ B : = ∑ k ∈ X B θ k , τ ∩ : = ∑ k ∈ X H ∩ X A θ k , λ : = τ ∩ τ A , \tau_B := \sum_{k\in\mathcal{X}_B}\theta_k, \qquad\tau_{\cap} := \sum_{k\in\mathcal{X}_H\cap\mathcal{X}_A}\theta_k, \qquad\lambda:= \frac{\tau_{\cap}}{\tau_A}, τ B := k ∈ X B ∑ θ k , τ ∩ := k ∈ X H ∩ X A ∑ θ k , λ := τ A τ ∩ ,
for B ∈ { H , A } B\in\{H,A\} B ∈ { H , A } , assuming positive agent capabilities. The overlap coefficient now measures the fraction of AI capability contributed by shared cues.
Proposition E.1 (Heterogeneous cues). For fixed access sets, the joint conditional distribution retains the form of Lemma 2.1:
( ℓ H ℓ A ) ∣ Y ∼ N ( ( 2 Y − 1 ) ( τ H τ A ) , ( 2 τ H 2 τ A λ 2 τ A λ 2 τ A ) ) . \begin{pmatrix}\ell_H\\\ell_A\end{pmatrix}\bigg|Y \sim\mathcal{N}\left((2Y-1)\begin{pmatrix}\tau_H\\\tau_A\end{pmatrix},\begin{pmatrix}2\tau_H&2\tau_A\lambda\\2\tau_A\lambda&2\tau_A\end{pmatrix}\right). ( ℓ H ℓ A ) Y ∼ N ( ( 2 Y − 1 ) ( τ H τ A ) , ( 2 τ H 2 τ A λ 2 τ A λ 2 τ A ) ) .
Proof. Cue k k k ’s log-likelihood ratio is Gaussian with conditional mean ( 2 Y − 1 ) θ k (2Y-1)\theta_k ( 2 Y − 1 ) θ k and variance 2 θ k 2\theta_k 2 θ k . Conditional independence implies that their sums are jointly Gaussian, with agent variances 2 τ B 2\tau_B 2 τ B and covariance 2 τ ∩ = 2 τ A λ 2\tau_{\cap}=2\tau_A\lambda 2 τ ∩ = 2 τ A λ .
Consequently, the decision rule, phase diagram, identification results, and welfare decomposition retain their forms under their existing assumptions, with λ \lambda λ interpreted as capability-weighted overlap. Heterogeneity concerns informativeness; individual cue realizations can still favor the wrong state.
The uniform-sampling argument in Proposition 2.2 also has a weighted analogue. Fix X H \mathcal{X}_H X H , let τ N : = ∑ k = 1 N θ k \tau_N := \sum_{k=1}^{N}\theta_k τ N := ∑ k = 1 N θ k , and sample X A \mathcal{X}_A X A uniformly among subsets of size n A n_A n A , independently of cue realizations. Each cue has inclusion probability n A / N n_A/N n A / N , so
E [ τ ∩ ] E [ τ A ] = ( n A / N ) τ H ( n A / N ) τ N = τ H τ N . \frac{\mathbb{E}[\tau_{\cap}]}{\mathbb{E}[\tau_A]}=\frac{(n_A/N)\tau_H}{(n_A/N)\tau_N}=\frac{\tau_H}{\tau_N}. E [ τ A ] E [ τ ∩ ] = ( n A / N ) τ N ( n A / N ) τ H = τ N τ H .
This ratio equals the human’s capability-weighted coverage of the domain and is independent of n A n_A n A . With heterogeneous capabilities, this is a ratio-of-expectations statement; it does not generally imply E [ λ ] = τ H / τ N \mathbb{E}[\lambda]=\tau_H/\tau_N E [ λ ] = τ H / τ N .
Second, we note that the accuracy and phase-diagram results depend only on the conditional joint-Gaussian law of ( ℓ H , ℓ A ) (\ell_H,\ell_A) ( ℓ H , ℓ A ) in Lemma 2.1, not on the cue-pool construction that produces it. That law arises as a Lyapunov-CLT type limit of aggregating many conditionally independent cues; for finite cue pools or heavy-tailed cues it holds only approximately.
Asymmetric Priors
This section extends our model to arbitrary prior probability π : = Pr ( Y = 1 ) ∈ [ 0 , 1 ] \pi:= \Pr(Y=1) \in[0,1] π := Pr ( Y = 1 ) ∈ [ 0 , 1 ] . We show that the qualitative structure that AI can impair the human or make automation optimal survives at every prior, with one sufficient condition on prior asymmetry (which we can define exactly) for establishing the complementarity region analogously to Theorem 3.2.
Under prior π \pi π , we retain the definition of primitive cues X k X_k X k , each agent’s log-likelihood ratios ℓ H \ell_H ℓ H and ℓ A \ell_A ℓ A , and the definition of overlap λ \lambda λ , correlation awareness α ^ \hat{\alpha} α ^ and trust β ^ \hat{\beta} β ^ . Note that each agent B ∈ { H , A } B\in\{H,A\} B ∈ { H , A } ’s posterior log-odds equals ℓ p r i o r + ℓ B \ell_{\mathrm{prior}}+\ell_B ℓ prior + ℓ B , where ℓ p r i o r : = log ( π / ( 1 − π ) ) \ell_{\mathrm{prior}} := \log(\pi/(1-\pi)) ℓ prior := log ( π / ( 1 − π )) is the prior. log-odds. We can show that ( ℓ H , ℓ A ) ∣ Y (\ell_H,\ell_A)\mid Y ( ℓ H , ℓ A ) ∣ Y follows the same joint distribution as Lemma 2.1, and that a worker with behavior given by Assumption 2.3 uses the linear rule ℓ ^ = ℓ p r i o r + w H ℓ H + w A ℓ A \hat{\ell}=\ell_{\mathrm{prior}}+w_H\ell_H+w_A\ell_A ℓ ^ = ℓ prior + w H ℓ H + w A ℓ A , where w H w_H w H and w A w_A w A follow the same definition as Lemma 2.4, but with the additive term ℓ p r i o r \ell_{\mathrm{prior}} ℓ prior in her decision statistic ℓ ^ \hat{\ell} ℓ ^ . The decision that maximizes accuracy (minimizes 0-1 loss) is Y ^ : = 1 { ℓ ^ > 0 } \hat{Y}:=1\{\hat{\ell}>0\} Y ^ := 1 { ℓ ^ > 0 } .
Unlike with symmetric priors (Lemma 3.1), the expected accuracy sums over two Gaussian CDFs:
Lemma F.1 (Accuracy under arbitrary priors). Under Assumption 2.3 and any prior π \pi π , the worker classifying via Y ^ : = 1 { ℓ ^ > 0 } \hat{Y}:=1\{\hat{\ell}>0\} Y ^ := 1 { ℓ ^ > 0 } attains accuracy
A H A ( α ^ , β ^ ; π , τ H , τ A , λ ) = π Φ ( m H A + ℓ p r i o r σ H A ) + ( 1 − π ) Φ ( m H A − ℓ p r i o r σ H A ) . \mathcal{A}_{HA}(\hat{\alpha},\hat{\beta};\pi,\tau_H,\tau_A,\lambda)
=
\pi\Phi\left(\frac{m_{HA}+\ell_{\mathrm{prior}}}{\sigma_{HA}}\right)
+
(1-\pi)\Phi\left(\frac{m_{HA}-\ell_{\mathrm{prior}}}{\sigma_{HA}}\right). A H A ( α ^ , β ^ ; π , τ H , τ A , λ ) = π Φ ( σ H A m H A + ℓ prior ) + ( 1 − π ) Φ ( σ H A m H A − ℓ prior ) . for any α ^ ∈ [ 0 , 1 ] \hat{\alpha}\in[0,1] α ^ ∈ [ 0 , 1 ] , β ^ > 0 \hat{\beta}>0 β ^ > 0 , where
m H A = w H τ H + w A τ A , σ H A = 2 ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) . m_{HA}=w_H\tau_H+w_A\tau_A,
\qquad
\sigma_{HA}=\sqrt{2\left(w_H^2\tau_H+w_A^2\tau_A+2w_Hw_A\tau_A\lambda\right)}. m H A = w H τ H + w A τ A , σ H A = 2 ( w H 2 τ H + w A 2 τ A + 2 w H w A τ A λ ) . Note that m H A / σ H A = r H A m_{HA}/\sigma_{HA}=r_{HA} m H A / σ H A = r H A is the same signal-to-noise ratio as Lemma 3.1.
Impairment and Automation Transfer Under All Priors
Proposition F.2 (Single-agent dominance under symmetric prior transfer to all priors). Fix α ^ = 0 \hat{\alpha}=0 α ^ = 0 and any β ^ > 0 \hat{\beta}>0 β ^ > 0 . For any admissible ( τ H , τ A , λ ) (\tau_H,\tau_A,\lambda) ( τ H , τ A , λ ) and any prior π \pi π , the expected accuracies as functions of prior π \pi π satisfy
A H A ( 1 / 2 ) ≤ A H ( 1 / 2 ) ⟹ A H A ( π ) ≤ A H ( π ) , ∀ π ∈ ( 0 , 1 ) , \mathcal{A}_{HA}(1/2)\leq\mathcal{A}_H(1/2)
\qquad\Longrightarrow\qquad
\mathcal{A}_{HA}(\pi)\leq\mathcal{A}_H(\pi),
\qquad\forall\pi\in(0,1), A H A ( 1/2 ) ≤ A H ( 1/2 ) ⟹ A H A ( π ) ≤ A H ( π ) , ∀ π ∈ ( 0 , 1 ) , and
A H A ( 1 / 2 ) ≤ A A ( 1 / 2 ) ⟹ A H A ( π ) ≤ A A ( π ) , ∀ π ∈ ( 0 , 1 ) . \mathcal{A}_{HA}(1/2)\leq\mathcal{A}_A(1/2)
\qquad\Longrightarrow\qquad
\mathcal{A}_{HA}(\pi)\leq\mathcal{A}_A(\pi),
\qquad\forall\pi\in(0,1). A H A ( 1/2 ) ≤ A A ( 1/2 ) ⟹ A H A ( π ) ≤ A A ( π ) , ∀ π ∈ ( 0 , 1 ) . This means if the team’s accuracy is weakly dominated by the human alone or by the AI alone at prior 1 / 2 1/2 1/2 , then the same dominance also holds at π \pi π .
Corollary F.3 (Impairment and automation regions under any prior are supersets of symmetric-prior regions). Fix α ^ = 0 \hat{\alpha}=0 α ^ = 0 and any β ^ , τ H > 0 \hat{\beta},\tau_H>0 β ^ , τ H > 0 . Let τ a u g ∗ ( λ ; β ^ ) \tau_{\mathrm{aug}}^*(\lambda;\hat{\beta}) τ aug ∗ ( λ ; β ^ ) and τ a u t o ∗ ( λ ; β ^ ) \tau_{\mathrm{auto}}^*(\lambda;\hat{\beta}) τ auto ∗ ( λ ; β ^ ) denote the symmetric-prior thresholds in Theorem 3.2, namely
τ a u g ∗ ( λ ; β ^ ) : = β ^ + 2 λ − 2 β ^ τ H , τ a u t o ∗ ( λ ; β ^ ) : = 2 β ^ − 1 + ( 2 β ^ − 1 ) 2 + 8 β ^ λ 4 β ^ λ τ H for λ > 0 , \tau_{\mathrm{aug}}^*(\lambda;\hat{\beta})
:=
\frac{\hat{\beta}+2\lambda-2}{\hat{\beta}}\tau_H,
\qquad
\tau_{\mathrm{auto}}^*(\lambda;\hat{\beta})
:=
\frac{2\hat{\beta}-1+\sqrt{(2\hat{\beta}-1)^2+8\hat{\beta}\lambda}}{4\hat{\beta}\lambda}\tau_H
\qquad\text{for }\lambda>0, τ aug ∗ ( λ ; β ^ ) := β ^ β ^ + 2 λ − 2 τ H , τ auto ∗ ( λ ; β ^ ) := 4 β ^ λ 2 β ^ − 1 + ( 2 β ^ − 1 ) 2 + 8 β ^ λ τ H for λ > 0 , with τ a u t o ∗ ( 0 ; β ^ ) = + ∞ \tau_{\mathrm{auto}}^*(0;\hat{\beta})=+\infty τ auto ∗ ( 0 ; β ^ ) = + ∞ if β ^ ≥ 1 / 2 \hat{\beta}\geq1/2 β ^ ≥ 1/2 and τ H / ( 1 − 2 β ^ ) \tau_H/(1-2\hat{\beta}) τ H / ( 1 − 2 β ^ ) otherwise. Then, for all priors π ∈ ( 0 , 1 ) \pi\in(0,1) π ∈ ( 0 , 1 ) and all admissible ( λ , τ A ) (\lambda,\tau_A) ( λ , τ A ) :
Low overlap ( λ ≤ 1 − β ^ / 2 (\lambda\leq1-\hat{\beta}/2 ( λ ≤ 1 − β ^ /2 , when β ^ < 2 \hat{\beta}<2 β ^ < 2 ): automation if τ A ≥ τ a u t o ∗ ( λ ; β ^ ) \tau_A\geq\tau_{\mathrm{auto}}^*(\lambda;\hat{\beta}) τ A ≥ τ auto ∗ ( λ ; β ^ ) .
Moderate or high overlap ( 1 − β ^ / 2 < λ ≤ 1 ) (1-\hat{\beta}/2<\lambda\leq1) ( 1 − β ^ /2 < λ ≤ 1 ) : impairment if τ A ≤ τ a u g ∗ ( λ ; β ^ ) \tau_A\leq\tau_{\mathrm{aug}}^*(\lambda;\hat{\beta}) τ A ≤ τ aug ∗ ( λ ; β ^ ) ; automation if τ A ≥ τ a u t o ∗ ( λ ; β ^ ) \tau_A\geq\tau_{\mathrm{auto}}^*(\lambda;\hat{\beta}) τ A ≥ τ auto ∗ ( λ ; β ^ ) .
These conditions characterize the full impairment and automation regions if and only if π = 1 / 2 \pi=1/2 π = 1/2 .
Full Phase Diagram Under Sufficiently Balanced Priors
Theorem F.4 (Phase diagram under correlation neglect with sufficiently balanced priors). Fix α ^ = 0 \hat{\alpha}=0 α ^ = 0 , π ∈ ( 0 , 1 ) \pi\in(0,1) π ∈ ( 0 , 1 ) , τ H > 0 \tau_H>0 τ H > 0 and β ^ > 0 \hat{\beta}>0 β ^ > 0 . Assume β ^ ≠ 1 / 2 \hat{\beta}\neq1/2 β ^ = 1/2 and ∣ ℓ p r i o r ∣ < ℓ ˉ ( τ H , β ^ ) |\ell_{\mathrm{prior}}|<\bar{\ell}(\tau_H,\hat{\beta}) ∣ ℓ prior ∣ < ℓ ˉ ( τ H , β ^ ) for a bound ℓ ˉ ( τ H , β ^ ) > 0 \bar{\ell}(\tau_H,\hat{\beta})>0 ℓ ˉ ( τ H , β ^ ) > 0 constructed explicitly in the proof. Then, there exists λ l o w : = 1 − β ^ / 2 \lambda_{\mathrm{low}}:=1-\hat{\beta}/2 λ low := 1 − β ^ /2 , λ h i g h ∈ ( λ l o w , 1 ] \lambda_{\mathrm{high}}\in(\lambda_{\mathrm{low}},1] λ high ∈ ( λ low , 1 ] , and functions τ a u g , τ a u t o \tau_{\mathrm{aug}},\tau_{\mathrm{auto}} τ aug , τ auto such that
Low overlap (λ ≤ λ l o w \lambda\leq\lambda_{\mathrm{low}} λ ≤ λ low , when β ^ < 2 \hat{\beta}<2 β ^ < 2 ): complementarity if τ A < τ a u t o ( λ ; β ^ ) \tau_A < \tau_{\mathrm{auto}}\left(\lambda;\hat{\beta}\right) τ A < τ auto ( λ ; β ^ ) ; automation if τ A ≥ τ a u t o ( λ ; β ^ ) \tau_A \geq\tau_{\mathrm{auto}}\left(\lambda;\hat{\beta}\right) τ A ≥ τ auto ( λ ; β ^ ) . 5
Moreover, τ a u g ( λ ; β ^ ) \tau_{\mathrm{aug}}\left(\lambda;\hat{\beta}\right) τ aug ( λ ; β ^ ) is strictly increasing in λ \lambda λ ; τ a u t o ( λ ; β ^ ) \tau_{\mathrm{auto}}\left(\lambda;\hat{\beta}\right) τ auto ( λ ; β ^ ) is strictly decreasing in λ \lambda λ (where finite); and τ a u g ( λ ; β ^ ) ≤ τ H ≤ τ a u t o ( λ ; β ^ ) \tau_{\mathrm{aug}}\left(\lambda;\hat{\beta}\right)\leq\tau_H\leq\tau_{\mathrm{auto}}\left(\lambda;\hat{\beta}\right) τ aug ( λ ; β ^ ) ≤ τ H ≤ τ auto ( λ ; β ^ ) , both with equality iff λ = λ h i g h \lambda=\lambda_{\mathrm{high}} λ = λ high . Also, λ h i g h = 1 \lambda_{\mathrm{high}}=1 λ high = 1 if and only if π = 1 / 2 \pi=1/2 π = 1/2 .
This is the same phase diagram as Theorem 3.2 , with the additional property that λ h i g h < 1 \lambda_{\mathrm{high}}<1 λ high < 1 when π ≠ 1 / 2 \pi\ne1/2 π = 1/2 : the high-overlap regime does not collapse to a single λ = 1 \lambda=1 λ = 1 . See Figure 3 for an illustration.
Figure 3. Phase diagram (regions as in Figure 1 ) under correlation neglect at trust β ^ = 0.285 \hat{\beta}=0.285 β ^ = 0.285 , for three priors. As the prior tilts away from π = 1 / 2 \pi=1/2 π = 1/2 , complementarity pinches off at λ h i g h < 1 \lambda_{\mathrm{high}}<1 λ high < 1 rather than at λ = 1 \lambda=1 λ = 1 (Theorem F.4 ).
Figure 1. Phase diagrams under correlation neglect (α ^ = 0 \hat{\alpha}=0 α ^ = 0 ) for three trust regimes. The deployment space ( τ A , λ ) (\tau_A,\lambda) ( τ A , λ ) is partitioned into complementarity (HA optimal, red), impairment (H optimal, gray), and automation (AI optimal, blue). The dashed black curve marks the cue-pool feasibility boundary τ A = τ H / λ \tau_A=\tau_H/\lambda τ A = τ H / λ . The augmentation threshold becomes positive at λ > 1 − β ^ / 2 \lambda>1-\hat{\beta}/2 λ > 1 − β ^ /2 (vertical-axis tick): 0.75, 0.50, 0.25 across panels (a)–(c), so the impairment region becomes active at lower overlap as trust rises. Dotted reference lines mark threshold values at fixed λ \lambda λ : at λ = 0.9 \lambda=0.9 λ = 0.9 (red), the augmentation threshold τ a u g \tau_{\mathrm{aug}} τ aug shifts from 0.60 to 0.80 to 0.87, and at λ = 0.5 \lambda=0.5 λ = 0.5 (blue), the automation threshold τ a u t o \tau_{\mathrm{auto}} τ auto shifts from 1.41 to 1.62 to 1.72.
Proof Sketch
The expected accuracy for any agent B ∈ { H , A , H A } B\in\{H,A,HA\} B ∈ { H , A , H A } can be written in terms of a shared function A B = A ( m B , σ B ; π ) \mathcal{A}_B=\mathcal{A}(m_B,\sigma_B;\pi) A B = A ( m B , σ B ; π ) , where ( m B , σ B ) (m_B,\sigma_B) ( m B , σ B ) and ℓ p r i o r \ell_{\mathrm{prior}} ℓ prior summarize the distribution of ℓ B \ell_B ℓ B or ℓ ^ \hat{\ell} ℓ ^ . When the prior is symmetric, A ( m B , σ B ; 1 / 2 ) = Φ ( m B / σ B ) \mathcal{A}(m_B,\sigma_B;1/2)=\Phi(m_B/\sigma_B) A ( m B , σ B ; 1/2 ) = Φ ( m B / σ B ) , where r B = m B / σ B r_B=m_B/\sigma_B r B = m B / σ B is the signal-to-noise ratio (SNR) in Lemma 3.1 .
Now fix an asymmetric prior π ≠ 1 / 2 \pi\ne1/2 π = 1/2 . In the ( m , σ ) (m,\sigma) ( m , σ ) plane, 6 we define SNR-rays along which r = m / σ r=m/\sigma r = m / σ is constant. We show that each SNR-ray has a unique calibrated point where σ 2 = 2 m \sigma^2=2m σ 2 = 2 m , where the statistic is a true log-likelihood ratio and its accuracy equals that of a calibrated single agent with signal-to-noise ratio r r r ; moreover, accuracy along the SNR-ray is maximized at the calibrated point, which is the unique maximum iff π ≠ 1 / 2 \pi\ne1/2 π = 1/2 . Additionally, accuracy among all calibrated points is strictly increasing in r r r . This yields a two-step SNR-ray projection lemma that upper bounds A ( m , σ ) \mathcal{A}(m,\sigma) A ( m , σ ) by a selected single agent’s accuracy: if the team is weakly dominated by a single agent at prior 1 / 2 1/2 1/2 , its signal-to-noise ratio is at most that agent’s; thus, under any prior π \pi π , the team performance A ( m , σ ) \mathcal{A}(m,\sigma) A ( m , σ ) is still at most the accuracy at the calibrated point A ( 2 m 2 / σ 2 , 2 m / σ ) \mathcal{A}(2m^2/\sigma^2,2m/\sigma) A ( 2 m 2 / σ 2 , 2 m / σ ) with the same signal-to-noise ratio, which is itself upper bounded by the single agent’s accuracy. This projection gives the single-agent dominance transfer property in Proposition F.2 , which immediately leads to Corollary F.3 .
Corollary F.3 implies that if any single-crossing augmentation and automation thresholds exist, they are lower- and upper-bounded respectively by the symmetric-prior thresholds τ a u g ∗ ( λ ; β ^ ) \tau_{\mathrm{aug}}^*\left(\lambda;\hat{\beta}\right) τ aug ∗ ( λ ; β ^ ) and τ a u t o ∗ ( λ ; β ^ ) \tau_{\mathrm{auto}}^{*}\left(\lambda;\hat{\beta}\right) τ auto ∗ ( λ ; β ^ ) . It remains to show the existence of such single crossings. For any fixed λ \lambda λ and under sufficient conditions, we show that for augmentation, A H A ( τ A ; λ ) A_{HA}\left(\tau_A;\lambda\right) A H A ( τ A ; λ ) is either increasing in τ A \tau_A τ A , or has a single minimum and then increases and approaches 1; and for automation, A H A ( τ A ; λ ) − A A ( τ A ) A_{HA}\left(\tau_A;\lambda\right)-A_A\left(\tau_A\right) A H A ( τ A ; λ ) − A A ( τ A ) has at most one single crossing to negative before τ A = τ a u t o ∗ ( λ ; β ^ ) \tau_A=\tau_{\mathrm{auto}}^{*}\left(\lambda;\hat{\beta}\right) τ A = τ auto ∗ ( λ ; β ^ ) , after which Corollary F.3 gives AI-alone dominance. The theorem’s upper bound on ∣ ℓ p r i o r ∣ \lvert\ell_{\mathrm{prior}}\rvert ∣ ℓ prior ∣ collects sufficient conditions for the correct signs of such derivatives. These results establish the existence of single-crossing thresholds τ a u g ( λ ; β ^ ) \tau_{\mathrm{aug}}\left(\lambda;\hat{\beta}\right) τ aug ( λ ; β ^ ) and τ a u t o ( λ ; β ^ ) \tau_{\mathrm{auto}}\left(\lambda;\hat{\beta}\right) τ auto ( λ ; β ^ ) under sufficiently balanced priors. Additionally, the implicit function theorem and derivative signs show that both thresholds are monotone in λ \lambda λ . Combining them with the H H H vs. A A A threshold (τ A = τ H \tau_A=\tau_H τ A = τ H ) proves Theorem F.4.
Sample construction
The ChatBench public release [8 ] contains 5,871 conversation logs (conversations.json), 10,450 user-answer rows (user_answers_filtered.csv) covering 643 workers, and 792 per-question AI-accuracy records (ai_alone_answers.csv) summarizing up to 50 free-text generations per AI-question pair (mean 49.9; 7 pairs have fewer due to sampling failures). We construct the matched answer-first sample as follows.
Filtering and merging. Restrict conversations.json to rows with condition = answer-first and answer_type = user-AI (the answer-first user-AI rows). Restrict user_answers_filtered.csv to answer_type = user-alone with non-missing confidence (the user-alone rows). Inner-join on (worker_id, question_id, model) to pair each post-AI answer with its unaided counterpart. Merge ai_alone_answers.csv on (model, question_id) to attach the per-question AI accuracy free_text_acc. This produces 2,894 matched pairs.
Ground truth. Each MMLU question has a unique correct letter. We recover ground truth from the user-alone sample by taking selected_answer of any worker who got the question right unaided. For two of the 396 questions in the matched sample, no user got the question right alone; we drop the resulting 10 rows (0.3% of 2,894) so that the AI-correctness indicator is well-defined for every observation. The remaining sample size is 2,884.
Has-Rec partition. The conversation logs include a flag conv_has_answer indicating whether the AI committed to a definite letter recommendation in its message. Pairs with conv_has_answer = True form the Has-Rec sample (2,199 rows; 315 unique workers; 389 unique questions); the rest form the No-Rec sample (685 rows). All analyses in §4 use the Has-Rec sample, where the AI’s signal is unambiguously a labeled recommendation.
Variable definitions. For each Has-Rec observation i = ( w , q , m ) i=(w,q,m) i = ( w , q , m ) (worker w w w , question q q q , AI model m m m ):
h _ a c c i ∈ { 0 , 1 } h\_acc_i\in\{0,1\} h _ a c c i ∈ { 0 , 1 } : whether the worker’s unaided answer is correct.
h _ c o n f i ∈ { not-, somewhat-, very-confident } h\_conf_i\in\{\text{not-, somewhat-, very-confident}\} h _ co n f i ∈ { not-, somewhat-, very-confident } : the worker’s stated confidence on the unaided answer.
F i ∈ { 0 , 1 } F_i\in\{0,1\} F i ∈ { 0 , 1 } : whether the worker’s post-AI answer matches ground truth.
a i _ c o r r e c t i ∈ { 0 , 1 } ai\_correct_i\in\{0,1\} ai _ cor r ec t i ∈ { 0 , 1 } : whether the AI’s letter recommendation matches ground truth.
p A m , q ∈ [ 0 , 1 ] p_A^{m,q}\in[0,1] p A m , q ∈ [ 0 , 1 ] : the AI’s empirical accuracy on question q q q across up to 50 independent free-text generations (the column free_text_acc).
Confidence calibration. We construct the empirical accuracy-by-confidence map p H ( c ) : = Pr ^ ( h _ a c c = 1 ∣ h _ c o n f = c ) p_H(c):=\widehat{\Pr}(h\_acc=1\mid h\_conf=c) p H ( c ) := Pr ( h _ a cc = 1 ∣ h _ co n f = c ) from the matched sample:
p H ( not-confident ) = 0.287 , p H ( somewhat-confident ) = 0.569 , p H ( very-confident ) = 0.787. p_H(\text{not-confident})=0.287,\quad p_H(\text{somewhat-confident})=0.569,\quad p_H(\text{very-confident})=0.787. p H ( not-confident ) = 0.287 , p H ( somewhat-confident ) = 0.569 , p H ( very-confident ) = 0.787. $
The signals used in the analysis are
ℓ H = logit ( clip ϵ ( p H ( h _ c o n f i ) ) ) , ℓ A i = logit ( clip ϵ ( p A m i , q i ) ) , \ell_H=\operatorname{logit}\left(\operatorname{clip}_{\epsilon}\left(p_H(h\_conf_i)\right)\right),\qquad\ell_A^i=\operatorname{logit}\left(\operatorname{clip}_{\epsilon}\left(p_A^{m_i,q_i}\right)\right), ℓ H = logit ( clip ϵ ( p H ( h _ co n f i ) ) ) , ℓ A i = logit ( clip ϵ ( p A m i , q i ) ) ,
where clip ϵ ( p ) : = max ( ϵ , min ( 1 − ϵ , p ) ) \operatorname{clip}_{\epsilon}(p) := \max(\epsilon,\min(1-\epsilon,p)) clip ϵ ( p ) := max ( ϵ , min ( 1 − ϵ , p )) with ϵ = 0.005 \epsilon= 0.005 ϵ = 0.005 to handle p A ∈ { 0 , 1 } p_A \in\{0,1\} p A ∈ { 0 , 1 } . About 62% of Has-Rec observations have p A ∈ { 0 , 1 } p_A \in\{0,1\} p A ∈ { 0 , 1 } (the AI gets every generation right or every generation wrong); ℓ A \ell_A ℓ A is therefore clipped to ± 5.293 \pm5.293 ± 5.293 for the majority of rows, and ∣ ℓ A ∣ \lvert\ell_A\rvert ∣ ℓ A ∣ should be read as a coarse three-level indicator of AI question-difficulty. Sensitivity to the choice of ϵ \epsilon ϵ is examined in §H.
Per-question H q H_q H q . For Approach 2 of §H, we use a finer human-side signal: H q : = 1 n q ∑ i : q i = q h a c c i H_q := \frac{1}{n_q}\sum_{i:q_i=q}h_{\mathrm{acc}i} H q := n q 1 ∑ i : q i = q h acc i , the empirical unaided accuracy on question q q q in the user-alone sample, and ℓ H H q : = logit ( clip ϵ ( H q , i ) ) \ell_H^{H_q} := \operatorname{logit}(\operatorname{clip}_{\epsilon}(H_{q,i})) ℓ H H q := logit ( clip ϵ ( H q , i )) . This gives 94 unique values of ℓ H H q \ell_H^{H_q} ℓ H H q versus only 3 unique values of ℓ H \ell_H ℓ H from confidence.
Sample-size summary.
Table 3.
Identification of the worker’s parameters and regime tests
Theoretically principled method
To use Theorem 3.2 for empirical analysis, we need to recover ( α ^ , β ^ ) (\hat{\alpha},\hat{\beta}) ( α ^ , β ^ ) from data on the worker’s behavior. The two parameters are separately identified if the same worker population is observed paired with two AI systems that differ in their overlap with the human’s evidence. We refer to a (worker population, AI system, task) configuration with parameters ( τ H , τ A , λ ) (\tau_H,\tau_A,\lambda) ( τ H , τ A , λ ) as a deployment , and assume the analyst observes data from two deployments d d d and d ′ d' d ′ (e.g., the same workers paired with GPT-4o on d d d and Llama on d ′ d' d ′ , on the same task) that share the worker population but differ in overlap: λ d ≠ λ d ′ \lambda^d \ne\lambda^{d'} λ d = λ d ′ . The distinguishing property of BR vs. CN (§ 2.2 ) provides the identification lever: at BR, the worker’s AI weight w A w_A w A depends on λ \lambda λ (she corrects for overlap); at CN, w A w_A w A does not (she treats the AI signal as independent). Across two deployments with λ d ≠ λ d ′ \lambda^d \ne\lambda^{d'} λ d = λ d ′ , the difference in w A w_A w A is informative about α ^ \hat{\alpha} α ^ .
Definition H.1 (Cross-deployment AI-weight ratio). R A ( α ^ , β ^ ; d , d ′ ) : = w A d ( α ^ , β ^ ) / w A d ′ ( α ^ , β ^ ) R_A(\hat{\alpha},\hat{\beta};d,d') := w_A^d(\hat{\alpha},\hat{\beta})/w_A^{d'}(\hat{\alpha},\hat{\beta}) R A ( α ^ , β ^ ; d , d ′ ) := w A d ( α ^ , β ^ ) / w A d ′ ( α ^ , β ^ ) , with w A d , w A d ′ w_A^d,w_A^{d'} w A d , w A d ′ from Lemma 2.4 . The empirical analogue is R ^ A : = w ^ A d / w ^ A d ′ \hat{R}_A := \hat{w}_A^d/\hat{w}_A^{d'} R ^ A := w ^ A d / w ^ A d ′ , where w ^ A d , w ^ A d ′ \hat{w}_A^d,\hat{w}_A^{d'} w ^ A d , w ^ A d ′ are estimated within each deployment d : = ( τ H , τ A d , λ d ) d := (\tau_H,\tau_A^d,\lambda^d) d := ( τ H , τ A d , λ d ) , d ′ : = ( τ H , τ A d ′ , λ d ′ ) d' := (\tau_H,\tau_A^{d'},\lambda^{d'}) d ′ := ( τ H , τ A d ′ , λ d ′ ) .
Proposition H.2 (Cross-deployment signature). Under Assumption 2.3,
Correlation neglect ( α ^ = 0 , any β ^ > 0 ) (\hat{\alpha}=0,\text{ any }\hat{\beta}>0) ( α ^ = 0 , any β ^ > 0 ) : R A C N = 1 R_A^{\mathrm{CN}}=1 R A CN = 1 .
Best response ( α ^ = 1 , β ^ = 1 ) (\hat{\alpha}=1,\hat{\beta}=1) ( α ^ = 1 , β ^ = 1 ) : When λ d ≠ λ d ′ \lambda^d\ne\lambda^{d'} λ d = λ d ′ , R A B R ≠ 1 R_A^{\mathrm{BR}}\ne1 R A BR = 1 generically.
Interior ( α ^ ∈ ( 0 , 1 ) ) (\hat{\alpha}\in(0,1)) ( α ^ ∈ ( 0 , 1 )) : R A R_A R A is continuous in α ^ \hat{\alpha} α ^ , and R A ≠ 1 R_A\ne1 R A = 1 generically. Moreover, when λ d ≠ λ d ′ \lambda^d\ne\lambda^{d'} λ d = λ d ′ , R A ( α ^ ) R_A(\hat{\alpha}) R A ( α ^ ) is locally invertible near α ^ = 0 \hat{\alpha}=0 α ^ = 0 with partial derivative λ d ′ − λ d ≠ 0 \lambda^{d'}-\lambda^d\ne0 λ d ′ − λ d = 0 .
The empirical R ^ A \hat{R}_A R ^ A thus separates correlation-neglect workers from best-response workers, and locally identifies an interior α ^ \hat{\alpha} α ^ near correlation neglect.
Two-step identification. Step 1 (recover α ^ \hat{\alpha} α ^ from R ^ A \hat{R}_A R ^ A ). Compute the BR-predicted ratio from deployment parameters ( τ H , τ A d , λ d ) (\tau_H,\tau_A^d,\lambda^d) ( τ H , τ A d , λ d ) , ( τ H , τ A d ′ , λ d ′ ) (\tau_H,\tau_A^{d'},\lambda^{d'}) ( τ H , τ A d ′ , λ d ′ ) ,
R A B R ( d , d ′ ) = ( 1 − λ d ) / ( τ H − τ A d ( λ d ) 2 ) ( 1 − λ d ′ ) / ( τ H − τ A d ′ ( λ d ′ ) 2 ) , R_A^{\mathrm{BR}}(d,d')=\frac{(1-\lambda^d)/(\tau_H-\tau_A^d(\lambda^d)^2)}{(1-\lambda^{d'})/(\tau_H-\tau_A^{d'}(\lambda^{d'})^2)}, R A BR ( d , d ′ ) = ( 1 − λ d ′ ) / ( τ H − τ A d ′ ( λ d ′ ) 2 ) ( 1 − λ d ) / ( τ H − τ A d ( λ d ) 2 ) ,
and compare to R ^ A \widehat{R}_A R A : R ^ A ≈ 1 \widehat{R}_A \approx1 R A ≈ 1 supports α ^ = 0 \widehat{\alpha}=0 α = 0 , R ^ A ≈ R A B R \widehat{R}_A \approx R_A^{\mathrm{BR}} R A ≈ R A BR supports α ^ = 1 \widehat{\alpha}=1 α = 1 , and intermediate values can pin down α ^ ∈ ( 0 , 1 ) \widehat{\alpha} \in(0,1) α ∈ ( 0 , 1 ) by solving
R ^ A = ( 1 − α ^ λ d ′ ) / ( τ H − τ A d ′ α ^ 2 ( λ d ′ ) 2 ) ( 1 − α ^ λ d ) / ( τ H − τ A d α ^ 2 ( λ d ) 2 ) . \widehat{R}_A=\frac{(1-\widehat{\alpha}\lambda^{d'})/(\tau_H-\tau_A^{d'}\widehat{\alpha}^{2}(\lambda^{d'})^{2})}{(1-\widehat{\alpha}\lambda^{d})/(\tau_H-\tau_A^{d}\widehat{\alpha}^{2}(\lambda^{d})^{2})}. R A = ( 1 − α λ d ) / ( τ H − τ A d α 2 ( λ d ) 2 ) ( 1 − α λ d ′ ) / ( τ H − τ A d ′ α 2 ( λ d ′ ) 2 ) .
Step 2 (recover β ^ \widehat{\beta} β from w ^ A \widehat{w}_A w A given α ^ \widehat{\alpha} α ). Inverting Lemma 2.4,
β ^ = w ^ A d τ H − τ A d α ^ 2 ( λ d ) 2 τ H ( 1 − α ^ λ d ) . \widehat{\beta}=\widehat{w}_A^{d}\frac{\tau_H-\tau_A^{d}\widehat{\alpha}^{2}(\lambda^{d})^{2}}{\tau_H(1-\widehat{\alpha}\lambda^{d})}. β = w A d τ H ( 1 − α λ d ) τ H − τ A d α 2 ( λ d ) 2 .
At α ^ = 0 \widehat{\alpha}=0 α = 0 , this reduces to β ^ = w ^ A d \widehat{\beta}=\widehat{w}_A^{d} β = w A d : the empirical AI weight estimates trust directly.
Proof of Proposition H.2. By Lemma 2.4, the worker’s AI weight at deployment d d d is
w A d ( α ^ , β ^ ) = β ^ ⋅ τ H ( 1 − α ^ λ d ) τ H − τ A d α ^ 2 ( λ d ) 2 . w_A^{d}(\widehat{\alpha},\widehat{\beta})=\widehat{\beta}\cdot\frac{\tau_H(1-\widehat{\alpha}\lambda^{d})}{\tau_H-\tau_A^{d}\widehat{\alpha}^{2}(\lambda^{d})^{2}}. w A d ( α , β ) = β ⋅ τ H − τ A d α 2 ( λ d ) 2 τ H ( 1 − α λ d ) .
The structural cross-deployment ratio is
R A ( α ^ , β ^ ; d , d ′ ) = w A d w A d ′ = ( 1 − α ^ λ d ) / ( τ H − τ A d α ^ 2 ( λ d ) 2 ) ( 1 − α ^ λ d ′ ) / ( τ H − τ A d ′ α ^ 2 ( λ d ′ ) 2 ) . R_A(\widehat{\alpha},\widehat{\beta};d,d')=\frac{w_A^{d}}{w_A^{d'}}=\frac{(1-\widehat{\alpha}\lambda^{d})/(\tau_H-\tau_A^{d}\widehat{\alpha}^{2}(\lambda^{d})^{2})}{(1-\widehat{\alpha}\lambda^{d'})/(\tau_H-\tau_A^{d'}\widehat{\alpha}^{2}(\lambda^{d'})^{2})}. R A ( α , β ; d , d ′ ) = w A d ′ w A d = ( 1 − α λ d ′ ) / ( τ H − τ A d ′ α 2 ( λ d ′ ) 2 ) ( 1 − α λ d ) / ( τ H − τ A d α 2 ( λ d ) 2 ) .
This is a rational function of α ^ \widehat{\alpha} α and deployment parameters. In particular, R A R_A R A does not depend on β ^ \widehat{\beta} β , which allows us to potentially identify α ^ \widehat{\alpha} α from ( d , d ′ ) (d,d') ( d , d ′ ) and the observed weights w A d w_A^{d} w A d and w A d ′ w_A^{d'} w A d ′ . We thus drop β ^ \widehat{\beta} β from the notation of R A R_A R A .
Note that the denominator after simplification is ( 1 − α ^ λ d ′ ) ( τ H − τ A d ′ α ^ 2 ( λ d ′ ) 2 ) (1-\widehat{\alpha}\lambda^{d'})(\tau_H-\tau_A^{d'}\widehat{\alpha}^{2}(\lambda^{d'})^{2}) ( 1 − α λ d ′ ) ( τ H − τ A d ′ α 2 ( λ d ′ ) 2 ) . On α ^ ∈ [ 0 , 1 ] \widehat{\alpha}\in[0,1] α ∈ [ 0 , 1 ] (no over-correlation), the second term is bounded away from zero under the model’s assumed parameter restriction τ A d ( λ d ) 2 < τ H \tau_A^{d}(\lambda^{d})^{2}<\tau_H τ A d ( λ d ) 2 < τ H , and the first term is bounded away from zero iff α ^ λ d ′ ≠ 1 \widehat{\alpha}\lambda^{d'}\ne1 α λ d ′ = 1 . Hence R A R_A R A is continuous in α ^ ∈ [ 0 , 1 ] \widehat{\alpha}\in[0,1] α ∈ [ 0 , 1 ] , except at α ^ = 1 \widehat{\alpha}=1 α = 1 if λ d ′ = 1 \lambda^{d'}=1 λ d ′ = 1 .
Special case: Correlation neglect. At α ^ = 0 \widehat{\alpha}=0 α = 0 , R A C N = 1 / τ H 1 / τ H = 1 R_A^{\mathrm{CN}}=\frac{1/\tau_H}{1/\tau_H}=1 R A CN = 1/ τ H 1/ τ H = 1 for every β ^ \widehat{\beta} β .
Ratio is generically not equal to 1 outside of correlation neglect. R A ( α ^ ; d , d ′ ) = 1 R_A(\widehat{\alpha};d,d')=1 R A ( α ; d , d ′ ) = 1 under the following necessary and sufficient condition:
τ H α ^ ( λ d ′ − λ d ) = ( 1 − α ^ λ d ′ ) τ A d ′ α ^ 2 ( λ d ′ ) 2 − ( 1 − α ^ λ d ′ ) τ A d α ^ 2 ( λ d ) 2 . (2) \tau_H\widehat{\alpha}(\lambda^{d'}-\lambda^{d})=(1-\widehat{\alpha}\lambda^{d'})\tau_A^{d'}\widehat{\alpha}^{2}(\lambda^{d'})^{2}-(1-\widehat{\alpha}\lambda^{d'})\tau_A^{d}\widehat{\alpha}^{2}(\lambda^{d})^{2}. \tag*{(2)} τ H α ( λ d ′ − λ d ) = ( 1 − α λ d ′ ) τ A d ′ α 2 ( λ d ′ ) 2 − ( 1 − α λ d ′ ) τ A d α 2 ( λ d ) 2 . (2)
Observe that (1) describes a specific knife-edge case that depends on all deployment parameters in d d d and d ′ d' d ′ , not just λ \lambda λ . To see this, note that it is equivalent to a unique requirement for τ A d ′ \tau_A^{d'} τ A d ′ , the second deployment’s AI capability:
τ A d ′ = τ H α ^ ( λ d ′ − λ d ) + ( 1 − α ^ λ d ′ ) τ A d α ^ 2 ( λ d ) 2 ( 1 − α ^ λ d ′ ) α ^ 2 ( λ d ′ ) 2 . \tau_A^{d'}=\frac{\tau_H\widehat{\alpha}(\lambda^{d'}-\lambda^{d})+(1-\widehat{\alpha}\lambda^{d'})\tau_A^{d}\widehat{\alpha}^{2}(\lambda^{d})^{2}}{(1-\widehat{\alpha}\lambda^{d'})\widehat{\alpha}^{2}(\lambda^{d'})^{2}}. τ A d ′ = ( 1 − α λ d ′ ) α 2 ( λ d ′ ) 2 τ H α ( λ d ′ − λ d ) + ( 1 − α λ d ′ ) τ A d α 2 ( λ d ) 2 .
whenever the denominator is nonzero. Also note that the proposition’s claim about best response (BR, α ^ = 1 , β ^ = 1 \widehat{\alpha}=1,\widehat{\beta}=1 α = 1 , β = 1 ) is a direct consequence of (1).
Derivatives (for solving α ^ \widehat{\alpha} α ): general case. The global log-derivative in α ^ \widehat{\alpha} α is
∂ ∂ α ^ log R A ( α ^ ; d , d ′ ) = λ d ′ − λ d ( 1 − α ^ λ d ) ( 1 − α ^ λ d ′ ) + 2 α ^ τ H [ τ A d ( λ d ) 2 − τ A d ′ ( λ d ′ ) 2 ] ( τ H − τ A d α ^ 2 ( λ d ) 2 ) ( τ H − τ A d ′ α ^ 2 ( λ d ′ ) 2 ) . (3) \frac{\partial}{\partial\widehat{\alpha}}\log R_A(\widehat{\alpha};d,d')=\frac{\lambda^{d'}-\lambda^{d}}{(1-\widehat{\alpha}\lambda^{d})(1-\widehat{\alpha}\lambda^{d'})}+\frac{2\widehat{\alpha}\tau_H\left[\tau_A^{d}(\lambda^{d})^{2}-\tau_A^{d'}(\lambda^{d'})^{2}\right]}{\left(\tau_H-\tau_A^{d}\widehat{\alpha}^{2}(\lambda^{d})^{2}\right)\left(\tau_H-\tau_A^{d'}\widehat{\alpha}^{2}(\lambda^{d'})^{2}\right)}. \tag*{(3)} ∂ α ∂ log R A ( α ; d , d ′ ) = ( 1 − α λ d ) ( 1 − α λ d ′ ) λ d ′ − λ d + ( τ H − τ A d α 2 ( λ d ) 2 ) ( τ H − τ A d ′ α 2 ( λ d ′ ) 2 ) 2 α τ H [ τ A d ( λ d ) 2 − τ A d ′ ( λ d ′ ) 2 ] . (3)
Specifically, the local derivative near α ^ = 0 \widehat{\alpha}=0 α = 0 is ∂ ∂ α ^ log R A ∣ α ^ = 0 = λ d ′ − λ d \left.\frac{\partial}{\partial\widehat{\alpha}}\log R_A\right|_{\widehat{\alpha}=0}=\lambda^{d'}-\lambda^{d} ∂ α ∂ log R A α = 0 = λ d ′ − λ d , so R A R_A R A is locally invertible near α ^ = 0 \widehat{\alpha}=0 α = 0 iff λ d ′ − λ d ≠ 0 \lambda^{d'}-\lambda^{d}\ne0 λ d ′ − λ d = 0 .
In practice, ( α ^ , β ^ ) (\widehat{\alpha},\widehat{\beta}) ( α , β ) may vary across topical domains, so identification applies within a domain (math, physics, etc.) rather than pooled.
Implementation details
§H.1 presents the two-step identification result theoretically. In the empirical work of §4 we implement two complementary procedures, both inversions of Lemma 2.4 , differing in how the human-side signal ℓ H \ell_H ℓ H and the overlap λ \lambda λ are constructed.
Approach 1: joint inversion using confidence-based ℓ H \ell_H ℓ H . For each deployment d ∈ { GPT-4o , Llama-3.1-8b } d \in\{\text{GPT-4o},\text{Llama-3.1-8b}\} d ∈ { GPT-4o , Llama-3.1-8b } , fit a logistic regression of F F F on ( ℓ H , ℓ A ) (\ell_H,\ell_A) ( ℓ H , ℓ A ) with intercept (no regularization, BFGS) on the Has-Rec rows for that deployment, recovering empirical weights ( w ^ H d , w ^ A d ) (\hat{w}_H^d,\hat{w}_A^d) ( w ^ H d , w ^ A d ) . Estimate the conditional-mean primitives by their plug-in moments,
τ ^ H = 1 2 ( ℓ H ∣ h _ a c c = 1 − ℓ H ∣ h _ a c c = 0 ) , τ ^ A d = 1 2 ( ℓ A ∣ a i _ c o r r e c t = 1 , d − ℓ A ∣ a i _ c o r r e c t = 0 , d ) , \hat{\tau}_H = \frac{1}{2}\left(\ell_H \mid h\_acc=1-\ell_H \mid h\_acc=0\right), \qquad\hat{\tau}_A^d = \frac{1}{2}\left(\ell_A \mid ai\_correct=1,d-\ell_A \mid ai\_correct=0,d\right), τ ^ H = 2 1 ( ℓ H ∣ h _ a cc = 1 − ℓ H ∣ h _ a cc = 0 ) , τ ^ A d = 2 1 ( ℓ A ∣ ai _ cor r ec t = 1 , d − ℓ A ∣ ai _ cor r ec t = 0 , d ) ,
treating τ ^ H \hat{\tau}_H τ ^ H as deployment-invariant (the empirical per-deployment values 0.374 and 0.411 agree to within 0.04). Then solve the four-equation system from Lemma 2.4 ,
w ^ H d = τ ^ H − τ ^ A d α ^ λ ^ d τ ^ H − τ ^ A d α ^ 2 ( λ ^ d ) 2 , w ^ A d = β ^ ⋅ τ ^ H ( 1 − α ^ λ ^ d ) τ ^ H − τ ^ A d α ^ 2 ( λ ^ d ) 2 , d ∈ { GPT , Llama } , \hat{w}_H^d=\frac{\hat{\tau}_H-\hat{\tau}_A^d\hat{\alpha}\hat{\lambda}^d}{\hat{\tau}_H-\hat{\tau}_A^d\hat{\alpha}^2(\hat{\lambda}^d)^2}, \qquad\hat{w}_A^d=\hat{\beta}\cdot\frac{\hat{\tau}_H(1-\hat{\alpha}\hat{\lambda}^d)}{\hat{\tau}_H-\hat{\tau}_A^d\hat{\alpha}^2(\hat{\lambda}^d)^2}, \qquad d\in\{\text{GPT},\text{Llama}\}, w ^ H d = τ ^ H − τ ^ A d α ^ 2 ( λ ^ d ) 2 τ ^ H − τ ^ A d α ^ λ ^ d , w ^ A d = β ^ ⋅ τ ^ H − τ ^ A d α ^ 2 ( λ ^ d ) 2 τ ^ H ( 1 − α ^ λ ^ d ) , d ∈ { GPT , Llama } ,
for the four unknowns ( α ^ , β ^ , λ ^ G P T , λ ^ L l a m a ) (\hat{\alpha},\hat{\beta},\hat{\lambda}^{\mathrm{GPT}},\hat{\lambda}^{\mathrm{Llama}}) ( α ^ , β ^ , λ ^ GPT , λ ^ Llama ) by trust-region-reflective nonlinear least squares with bounds α ^ ∈ [ 0 , 1 ] \hat{\alpha}\in[0,1] α ^ ∈ [ 0 , 1 ] , β ^ ∈ ( 0 , 5 ] \hat{\beta}\in(0,5] β ^ ∈ ( 0 , 5 ] , λ ^ d ∈ [ 0 , 0.99 ] \hat{\lambda}^d\in[0,0.99] λ ^ d ∈ [ 0 , 0.99 ] , and 100 random restarts. The global minimizer is
α ^ = 0.315 , β ^ = 0.285 , λ ^ G P T = 0.478 , λ ^ L l a m a = 0.365 , S S R = 5.5 × 10 − 3 . \hat{\alpha}=0.315,\qquad\hat{\beta}=0.285,\qquad\hat{\lambda}^{\mathrm{GPT}}=0.478,\qquad\hat{\lambda}^{\mathrm{Llama}}=0.365,\qquad\mathrm{SSR}=5.5\times10^{-3}. α ^ = 0.315 , β ^ = 0.285 , λ ^ GPT = 0.478 , λ ^ Llama = 0.365 , SSR = 5.5 × 1 0 − 3 .
A 200-replication non-parametric bootstrap gives β ^ ∈ [ 0.246 , 0.322 ] \hat{\beta}\in[0.246,0.322] β ^ ∈ [ 0.246 , 0.322 ] at the 95% level.
Approach 2: λ ^ \hat{\lambda} λ ^ from cross-question variation. This procedure replaces ℓ H \ell_H ℓ H with the per-question signal ℓ H H q \ell_H^{Hq} ℓ H H q from §G and estimates λ \lambda λ directly as the within-Y Y Y pooled covariance,
λ ^ d : = ∑ y ∈ { 0 , 1 } n d , y Cov ( ℓ H H q , ℓ A ∣ Y = y , d ) ∑ y ∈ { 0 , 1 } n d , y Var ( ℓ A ∣ Y = y , d ) , \hat{\lambda}^d:=\frac{\sum_{y\in\{0,1\}}n_{d,y}\operatorname{Cov}(\ell_H^{Hq},\ell_A\mid Y=y,d)}{\sum_{y\in\{0,1\}}n_{d,y}\operatorname{Var}(\ell_A\mid Y=y,d)}, λ ^ d := ∑ y ∈ { 0 , 1 } n d , y Var ( ℓ A ∣ Y = y , d ) ∑ y ∈ { 0 , 1 } n d , y Cov ( ℓ H H q , ℓ A ∣ Y = y , d ) ,
mirroring the population identity λ = Cov ( ℓ H , ℓ A ∣ Y ) / Var ( ℓ A ∣ Y ) \lambda=\operatorname{Cov}(\ell_H,\ell_A\mid Y)/\operatorname{Var}(\ell_A\mid Y) λ = Cov ( ℓ H , ℓ A ∣ Y ) / Var ( ℓ A ∣ Y ) implied by Lemma 2.1 . We obtain
λ ^ G P T = 0.071 ∈ [ 0.041 , 0.100 ] 95 % , λ ^ L l a m a = 0.127 ∈ [ 0.100 , 0.154 ] 95 % , \hat{\lambda}^{\mathrm{GPT}}=0.071\in[0.041,0.100]_{95\%},\qquad\hat{\lambda}^{\mathrm{Llama}}=0.127\in[0.100,0.154]_{95\%}, λ ^ GPT = 0.071 ∈ [ 0.041 , 0.100 ] 95% , λ ^ Llama = 0.127 ∈ [ 0.100 , 0.154 ] 95% ,
with both bootstrap CIs strictly above zero. We then re-run the per-deployment logistic regressions of F F F on ( ℓ H H q , ℓ A ) (\ell_H^{Hq},\ell_A) ( ℓ H H q , ℓ A ) to obtain weights under this finer signal, and solve the analogous system for ( α ^ , β ^ ) (\hat{\alpha},\hat{\beta}) ( α ^ , β ^ ) with λ ^ d \hat{\lambda}^d λ ^ d now treated as plug-in. The resulting fit gives β ^ = 0.254 \hat{\beta}=0.254 β ^ = 0.254 , in close agreement with Approach 1.
On the discrepancy in α ^ \hat{\alpha} α ^ . The two approaches agree on β ^ \hat{\beta} β ^ (the headline parameter) but not on α ^ \hat{\alpha} α ^ : Approach 1 gives an interior α ^ = 0.315 \hat{\alpha}=0.315 α ^ = 0.315 , Approach 2 returns α ^ = 1 \hat{\alpha}=1 α ^ = 1 at the upper bound. We interpret this as evidence that α ^ \hat{\alpha} α ^ is weakly identified by these moments — small differences in how ℓ H \ell_H ℓ H is constructed propagate through the nonlinear inversion to large differences in the α ^ \hat{\alpha} α ^ point estimate. The qualitative claim that α ^ > 0 \hat{\alpha}>0 α ^ > 0 (strict correlation neglect rejected) is robust across approaches and across three direct tests:
Approach 1’s residual sum of squares jumps 141-fold when α ^ = 0 \hat{\alpha}=0 α ^ = 0 is forced.
A likelihood-ratio test of whether ℓ A \ell_A ℓ A adds information beyond 4-cell ( h _ a c c , a i _ c o r r e c t ) (h\_acc, ai\_correct) ( h _ a cc , ai _ cor r ec t ) fixed effects gives χ 2 ( 1 ) = 3.93 \chi^2(1)=3.93 χ 2 ( 1 ) = 3.93 , p = 0.048 p=0.048 p = 0.048 .
Approach 2’s direct λ ^ d \hat{\lambda}^d λ ^ d are strictly positive in 100% of bootstrap samples for both deployments.
On the discrepancy in λ ^ \hat{\lambda} λ ^ . Both approaches yield positive estimates of overlap, but differ in their relative ordering across the two AIs. We adopt Approach 2 as our primary measure, as it estimates overlap directly from the conditional covariance structure of human and AI signals, rather than indirectly through a nonlinear inversion of worker behavior.
Robustness of β ^ \hat{\beta} β ^ . Varying the clipping constant ϵ ∈ { 0.001 , 0.005 , 0.01 , 0.02 } \epsilon\in\{0.001, 0.005, 0.01, 0.02\} ϵ ∈ { 0.001 , 0.005 , 0.01 , 0.02 } (§G) shifts β ^ \hat{\beta} β ^ across [ 0.22 , 0.37 ] [0.22, 0.37] [ 0.22 , 0.37 ] but leaves it well below 1 at every value, so the rejection of β ^ = 1 \hat{\beta}=1 β ^ = 1 is robust. A worker-cluster bootstrap that resamples the 315 workers (rather than 2,199 rows i.i.d.) gives β ^ ∈ [ 0.241 , 0.332 ] \hat{\beta} \in[0.241, 0.332] β ^ ∈ [ 0.241 , 0.332 ] at the 95% level, slightly wider than the row-bootstrap CI but with the same qualitative conclusion across all three headline quantities (GAP, β ^ \hat{\beta} β ^ , λ ^ L l a m a − λ ^ G P T \hat{\lambda}^{\mathrm{Llama}}-\hat{\lambda}^{\mathrm{GPT}} λ ^ Llama − λ ^ GPT ).
Test for A H A A_{HA} A H A versus A A I A_{AI} A A I . Regime classification in Table 1 compares team accuracy A H A A_{HA} A H A to AI accuracy A A I A_{AI} A A I on the same observations. Define the discordant pair counts
n 10 d : = ∣ { i ∈ d : F i = 1 , a i _ c o r r e c t i = 0 } ∣ , n 01 d : = ∣ { i ∈ d : F i = 0 , a i _ c o r r e c t i = 1 } ∣ . n_{10}^{d} := \left|\left\{i \in d : F_i = 1, ai\_correct_i = 0\right\}\right|,\qquad n_{01}^{d} := \left|\left\{i \in d : F_i = 0, ai\_correct_i = 1\right\}\right|. n 10 d := { i ∈ d : F i = 1 , ai _ cor r ec t i = 0 } , n 01 d := { i ∈ d : F i = 0 , ai _ cor r ec t i = 1 } .
Under H 0 : A H A = A A I H_0 : A_{HA}=A_{AI} H 0 : A H A = A A I , the conditional distribution of n 10 d n_{10}^{d} n 10 d given n 10 d + n 01 d n_{10}^{d}+n_{01}^{d} n 10 d + n 01 d is Binomial ( n 10 d + n 01 d , 1 2 ) \operatorname{Binomial}(n_{10}^{d}+n_{01}^{d}, \frac{1}{2}) Binomial ( n 10 d + n 01 d , 2 1 ) . The reported p p p -values are exact two-sided binomial p p p -values from this conditional distribution (the exact form of the McNemar test). The relevant counts are ( n 10 G P T , n 01 G P T ) = ( 51 , 89 ) (n_{10}^{\mathrm{GPT}}, n_{01}^{\mathrm{GPT}})=(51,89) ( n 10 GPT , n 01 GPT ) = ( 51 , 89 ) and ( n 10 L l a m a , n 01 L l a m a ) = ( 74 , 66 ) (n_{10}^{\mathrm{Llama}}, n_{01}^{\mathrm{Llama}})=(74,66) ( n 10 Llama , n 01 Llama ) = ( 74 , 66 ) . We classify a deployment as automation if the test rejects in favor of n 01 > n 10 n_{01}>n_{10} n 01 > n 10 , complementarity if it rejects in favor of n 10 > n 01 n_{10}>n_{01} n 10 > n 01 , and boundary otherwise.
Capability-versus-behavior decomposition
The cross-deployment GAP gap of + 0.081 +0.081 + 0.081 admits two non-exclusive interpretations: (i) AI-capability composition — different fractions of pairs in each ( h _ a c c , a i _ c o r r e c t ) (h\_acc, ai\_correct) ( h _ a cc , ai _ cor r ec t ) cell across deployments — or (ii) workers responding differently to the two AIs at the per-cell level. We separate them by holding worker behavior at the pooled per-cell rate Pr ( F = 1 ∣ h _ a c c , a i _ c o r r e c t ) \Pr(F=1 \mid h\_acc, ai\_correct) Pr ( F = 1 ∣ h _ a cc , ai _ cor r ec t ) across both deployments and letting only the cell composition vary by AI.
Counterfactual GAP gap. Let p ˉ ( h , a ) : = Pr ^ ( F = 1 ∣ h _ a c c = h , a i _ c o r r e c t = a ) \bar{p}(h,a):=\widehat{\Pr}(F=1 \mid h\_acc=h, ai\_correct=a) p ˉ ( h , a ) := Pr ( F = 1 ∣ h _ a cc = h , ai _ cor r ec t = a ) on the pooled Has-Rec sample. For each deployment d d d , define the counterfactual team accuracy
A ^ H A d , c f : = ∑ h , a ∈ { 0 , 1 } Pr ^ d ( h _ a c c = h , a i _ c o r r e c t = a ) ⋅ p ˉ ( h , a ) , \widehat{A}_{HA}^{d,\mathrm{cf}} := \sum_{h,a\in\{0,1\}} \widehat{\Pr}_{d}(h\_acc=h, ai\_correct=a)\cdot\bar{p}(h,a), A H A d , cf := h , a ∈ { 0 , 1 } ∑ Pr d ( h _ a cc = h , ai _ cor r ec t = a ) ⋅ p ˉ ( h , a ) ,
which assigns the pooled per-cell behavior to deployment d d d ’s observed cell composition. The capability share is the ratio of the counterfactual GAP gap (driven by composition alone) to the observed GAP gap:
CapShare : = ( A ^ H A G P T , c f − A ^ H A G P T ) − ( A ^ H A L l a m a , c f − A ^ H A L l a m a ) GAP G P T − GAP L l a m a . \operatorname{CapShare}:=\frac{\left(\widehat{A}_{HA}^{\mathrm{GPT,cf}}-\widehat{A}_{HA}^{\mathrm{GPT}}\right)-\left(\widehat{A}_{HA}^{\mathrm{Llama,cf}}-\widehat{A}_{HA}^{\mathrm{Llama}}\right)}{\operatorname{GAP}^{\mathrm{GPT}}-\operatorname{GAP}^{\mathrm{Llama}}}. CapShare := GAP GPT − GAP Llama ( A H A GPT , cf − A H A GPT ) − ( A H A Llama , cf − A H A Llama ) .
The point estimate is CapShare = 92.6 % =92.6\% = 92.6% . A non-parametric row bootstrap with 1,000 replications gives a median of 93.2% and a 95% confidence interval of [ 68 % , 148 % ] [68\%, 148\%] [ 68% , 148% ] . The wide upper end reflects the fact that the denominator GAP G P T − GAP L l a m a \operatorname{GAP}^{\mathrm{GPT}}-\operatorname{GAP}^{\mathrm{Llama}} GAP GPT − GAP Llama is itself a small estimated quantity ( + 0.081 ) (+0.081) ( + 0.081 ) ; the lower bound stays well above 50% across resamples.
Direct test of behavioral homogeneity. A complementary test compares two logistic regressions of F F F on ( h _ a c c , a i _ c o r r e c t ) (h\_acc, ai\_correct) ( h _ a cc , ai _ cor r ec t ) cell fixed effects: a reduced model with no AI interaction versus a full model interacting cell with deployment. The likelihood-ratio statistic is χ 2 ( 4 ) = 0.96 \chi^2(4)=0.96 χ 2 ( 4 ) = 0.96 (p = 0.92 p=0.92 p = 0.92 ); we cannot reject the null that workers respond identically to the two AIs at the per-cell level. The capability-share decomposition and the LR test thus point in the same direction: the cross-deployment welfare difference is driven primarily by the composition of ( h _ a c c , a i _ c o r r e c t ) (h\_acc, ai\_correct) ( h _ a cc , ai _ cor r ec t ) pairs each AI generates, not by deployment-specific worker behavior.
Overlap heterogeneity across AIs and subjects
The direct λ ^ d \hat{\lambda}^{d} λ ^ d estimator from Approach 2 of §H can be applied at finer-than-deployment granularity, by restricting the within-Y Y Y covariance computation to subsets of the Has-Rec sample. We report two cuts.
Across AIs. Pooled across all subjects, λ ^ G P T = 0.071 \hat{\lambda}^{\mathrm{GPT}}=0.071 λ ^ GPT = 0.071 and λ ^ L l a m a = 0.127 \hat{\lambda}^{\mathrm{Llama}}=0.127 λ ^ Llama = 0.127 , a difference of + 0.056 +0.056 + 0.056 . A row bootstrap on the difference gives a 95% CI of [ + 0.015 , + 0.097 ] [+0.015,+0.097] [ + 0.015 , + 0.097 ] (two-sided p ≈ 0.009 p\approx0.009 p ≈ 0.009 ), with 99.7% of 1,000 resamples having λ ^ L l a m a > λ ^ G P T \hat{\lambda}^{\mathrm{Llama}}>\hat{\lambda}^{\mathrm{GPT}} λ ^ Llama > λ ^ GPT . The smaller AI’s evidence overlaps more with what humans access on the same questions; this is consistent with the cue-pool reading of §2, in which a smaller model relies more heavily on cues that humans also access while a larger model contributes more genuinely novel evidence.
Across subjects. The Has-Rec sample is split as math (n = 1,388 n = 1{,}388 n = 1 , 388 ), physics (n = 497 n = 497 n = 497 ), and moral reasoning (n = 314 n = 314 n = 314 ). Direct λ ^ \hat{\lambda} λ ^ varies dramatically across subjects, and the same ordering holds within each AI separately:
Table 4.
Domains where humans and AI rely on shared procedural cues (math) exhibit substantial overlap; domains where reasoning paths are more idiosyncratic (moral reasoning) exhibit essentially none. This provides empirical content for the cue-pool framing: under the model, λ \lambda λ is a property of the human-domain pair (Proposition 2.2 ); the data shows the predicted variation across domains and additionally a cross-AI effect that the uniform-sampling model does not predict.
Limitations of these estimates. Both findings rely on the per-question ℓ H H q \ell_H^{Hq} ℓ H H q proxy, which is a population-level signal rather than an individual worker’s signal. Replacing ℓ H H q \ell_H^{Hq} ℓ H H q with the confidence-based ℓ H \ell_H ℓ H (Approach 1) does not give a directly comparable λ ^ \hat{\lambda} λ ^ because the latter has only three values per worker. We report the cross-AI and cross-subject patterns as evidence that λ \lambda λ varies meaningfully and in interpretable directions, not as precise structural estimates.
Empirical welfare evaluation
We evaluate the welfare consequences of trust calibration by simulating a trust-calibrated decision rule. For each observation i i i , recall h i : = h _ a c c i h_i := h\_acc_i h i := h _ a c c i indicate whether the human’s unaided answer is correct, a i : = a i _ c o r r e c t i a_i := ai\_correct_i a i := ai _ cor r ec t i whether the AI’s actual recommendation is correct, and F i F_i F i whether the worker’s final answer is correct. We simulate a reliance policy that chooses between the human’s and AI’s answers, and its accuracy is evaluated against the ground truth of the question.
Constructing the simulated policy. The policy uses X i = ( ℓ H i , ℓ A i ) X_i = (\ell_H^i, \ell_A^i) X i = ( ℓ H i , ℓ A i ) , where ℓ H i \ell_H^i ℓ H i is constructed from the worker’s stated confidence and ℓ A i \ell_A^i ℓ A i from the AI’s question-level accuracy, as described in Appendix G. These predictors summarize human confidence and AI performance on the question; they do not encode the content of the AI’s specific response.
Within the disagreement sample, define
p H d ( x ) : = Pr ( h i = 1 ∣ X i = x , i ∈ D d ) , p A d ( x ) : = Pr ( a i = 1 ∣ X i = x , i ∈ D d ) . p_H^d(x) := \Pr(h_i = 1 \mid X_i = x,\ i \in\mathcal{D}_d), \qquad p_A^d(x) := \Pr(a_i = 1 \mid X_i = x,\ i \in\mathcal{D}_d). p H d ( x ) := Pr ( h i = 1 ∣ X i = x , i ∈ D d ) , p A d ( x ) := Pr ( a i = 1 ∣ X i = x , i ∈ D d ) .
We estimate these conditional correctness probabilities using logistic regression on ( ℓ H , ℓ A ) (\ell_H, \ell_A) ( ℓ H , ℓ A ) with five-fold cross-validation. Each observation receives predictions from models that did not include that observation in their four folds.
For each held-out observation, define the reliance score and decision
s i : = p ^ A d ( X i ) − p ^ H d ( X i ) , π i : = 1 { s i > 0 } . s_i := \widehat{p}_A^d(X_i) - \widehat{p}_H^d(X_i), \qquad\pi_i := \mathbf{1}\{s_i > 0\}. s i := p A d ( X i ) − p H d ( X i ) , π i := 1 { s i > 0 } .
The simulated policy submits the AI’s answer when π i = 1 \pi_i = 1 π i = 1 and retains the human’s answer otherwise. We fix the threshold at zero rather than selecting it to maximize accuracy on the evaluation observations. With the true conditional probabilities, this rule is optimal among policies choosing between the two answers using X i X_i X i . The fitted logistic rule provides an empirical approximation to this benchmark.
Table 2. Accuracy gains from the logistic answer-selection policy. All entries are percentage points. The full-sample contribution weights the disagreement gain by the disagreement share.
Capability-dependent benefits of trust calibration. For GPT-4o, the simulated policy raises accuracy from approximately 0.851 to 0.877, bringing the team close to the AI-alone accuracy of 0.884. It follows the AI on approximately 92% of disagreements, compared with the workers’ observed reliance rate of 70%. The policy’s greater reliance and higher accuracy are consistent with a welfare cost from under-trust on the stronger AI.
For Llama-3.1-8b, the observed workers outperform the fitted policy. Within this comparison, replacing their decisions with the coarse-signal rule offers no improvement. The benefits of greater trust therefore depend on AI capability and cannot be assumed to transfer uniformly across deployments.
Information and the limits of the benchmark. In reality, the worker observes more than the two predictors available to the fitted rule. In particular, she reads the AI’s actual answer and can assess its reasoning and relevance to the question. The question-level accuracy measure averages across responses and omits these response-specific information. Workers’ superior performance relative to the fitted rule in the Llama deployment is consistent with their using this additional information.
The broader implication is that improving human–AI performance may require attention to information as well as behavior. Better-calibrated trust offers a measurable opportunity in the stronger-AI deployment, but a rule based only on coarse confidence and accuracy measures does not capture the worker’s full decision problem. Richer response-specific information would allow a more informative benchmark and a clearer assessment of the remaining scope for behavioral improvement.
Departures from theory and what they mean
The framework of §§2–3 is a binary-classification model with a Gaussian generative process, symmetric prior, and linear-rule decision-maker. The empirical setting departs from each of these in ways the analysis must accommodate.
Binary reduction of a four-way task. ChatBench is a four-way multiple-choice task; the model is binary. We define the outcome Y i Y_i Y i as the indicator that the worker’s chosen letter matches the unique correct letter on item i i i , collapsing the three wrong letters into Y i = 0 Y_i=0 Y i = 0 . Two consequences follow. (i) The model’s closed-form AI accuracy Φ ( τ ^ A d / 2 ) \Phi\left(\sqrt{\hat{\tau}_A^d/2}\right) Φ ( τ ^ A d /2 ) does not match empirical accuracy: at τ ^ A L l a m a = 1.64 \hat{\tau}_A^{\mathrm{Llama}}=1.64 τ ^ A Llama = 1.64 the formula gives 0.82 versus empirical p ^ A I L l a m a = 0.787 \hat{p}_{AI}^{\mathrm{Llama}}=0.787 p ^ A I Llama = 0.787 , and at τ ^ A G P T = 2.02 \hat{\tau}_A^{\mathrm{GPT}}=2.02 τ ^ A GPT = 2.02 it gives 0.84 versus empirical 0.884. We do not use the closed-form accuracy formula anywhere in the empirical section; every accuracy reported in §4 and Tables 1 and 2 is the empirical sample mean.
(ii) The empirical base rate Pr ^ ( h a c c = 1 ) = 0.612 \widehat{\Pr}(h_{\mathrm{acc}}=1)=0.612 Pr ( h acc = 1 ) = 0.612 is not 0.5, so the symmetric-prior assumption underlying Φ ( τ ^ H / 2 ) = Pr ( h a c c = 1 ) \Phi\left(\sqrt{\hat{\tau}_H/2}\right)=\Pr(h_{\mathrm{acc}}=1) Φ ( τ ^ H /2 ) = Pr ( h acc = 1 ) does not literally hold. We use prior-free conditional-mean estimators for τ ^ H \hat{\tau}_H τ ^ H and τ ^ A d \hat{\tau}_A^d τ ^ A d (Appendix H), which are invariant to the base rate.
ℓ A \ell_A ℓ A saturation and cross-deployment scale. About 62% of Has-Rec rows have p A m , q ∈ { 0 , 1 } p_A^{m,q}\in\{0,1\} p A m , q ∈ { 0 , 1 } (Appendix G), saturating ℓ A \ell_A ℓ A at ± 5.293 \pm5.293 ± 5.293 . Saturation is more severe for GPT-4o, where p A m , q = 1 p_A^{m,q}=1 p A m , q = 1 on the majority of items, than for Llama-3.1-8b. As a consequence, per-deployment regression coefficients w ^ A d \hat{w}_A^d w ^ A d are not on the same scale across deployments: w ^ A G P T \hat{w}_A^{\mathrm{GPT}} w ^ A GPT effectively measures the response to a near-binary "AI is right" indicator, while w ^ A L l a m a \hat{w}_A^{\mathrm{Llama}} w ^ A Llama uses more variation. Approach 1 (§H) absorbs this scale difference through the structural inversion, which divides through τ ^ A d \hat{\tau}_A^d τ ^ A d when recovering β ^ \hat{\beta} β ^ . Approach 2 uses cross-question covariance and is invariant to monotone rescaling of ℓ A \ell_A ℓ A . Direct comparison of w ^ A d \hat{w}_A^d w ^ A d across deployments would not be valid; the structural recovery of β ^ \hat{\beta} β ^ is.
Cue-pool generative story. The cue-pool model of §2 motivates λ \lambda λ as the fraction of the AI’s cues also accessed by the human. Language-model decisions are not literally generated by sampling discrete cues from a finite pool. We do not commit to the cue-pool generative process. The structural objects we estimate — the joint distribution of ( ℓ H , ℓ A , Y ) (\ell_H, \ell_A, Y) ( ℓ H , ℓ A , Y ) and its summary statistics ( τ H , τ A , λ ) (\tau_H, \tau_A, \lambda) ( τ H , τ A , λ ) — are well-defined regardless of the underlying generative process. The cue-pool story provides intuition for the cross-domain λ \lambda λ heterogeneity in Appendix § 15 , but it is not required for the identification or welfare results.
Between-subjects design. ChatBench randomizes each worker to exactly one AI, so the GPT-4o and Llama-3.1-8b samples are disjoint worker pools. Cross-deployment comparisons therefore rely on random assignment rather than within-worker variation. Random assignment makes large population-level imbalances unlikely in expectation but does not rule out them in any specific sample. The behavioral-homogeneity test of Appendix § 14 (χ 2 ( 4 ) = 0.96 \chi^2(4) = 0.96 χ 2 ( 4 ) = 0.96 , p = 0.92 p = 0.92 p = 0.92 ) provides a direct check: per-cell response rates are statistically indistinguishable across the two AI populations, supporting the cross-deployment comparability of β ^ \hat{\beta} β ^ and other behavioral parameters.
Reproducibility
All empirical claims in Section 4 and Appendices § 12 –§ 16 are computed by a single self-contained Python script that takes the three raw ChatBench files as input. It reproduces sample construction, signal calibration, the headline findings with bootstrap CIs (1,000 direct, 200 structural, 500 welfare), all regime, correlation-neglect, capability-share, and λ \lambda λ -heterogeneity tests, the β ^ = 1 \hat{\beta} = 1 β ^ = 1 counterfactual, and three robustness checks (worker-cluster bootstrap on all four headline quantities, ϵ \epsilon ϵ sensitivity, 5-fold cross-validated linear ceiling). All seeds are fixed; reference values appear in the script header; total runtime is approximately 4 minutes on a single CPU core. The script will be provided with the camera-ready version.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction state three contributions: a phase diagram in ( τ A , λ ) (\tau_A, \lambda) ( τ A , λ ) partitioning impairment, complementarity, and automation regions; identifiability of ( α ^ , β ^ ) (\hat{\alpha}, \hat{\beta}) ( α ^ , β ^ ) from cross-deployment variation in overlap; and a welfare decomposition into correlation-neglect and trust-miscalibration costs — which are delivered respectively in Theorem 3.2 , Proposition H.2 , and Proposition 3.4 , and instantiated empirically on ChatBench in §4.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix § 17 enumerates the main limitations of the empirical study as departures from the theoretical model.
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results are stated with full assumptions. Complete proofs are given in Appendices § 6 and the proof sections referenced from §2 and §3.
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix G fully describes sample construction; Appendix H specifies the estimation procedures; Appendix M describes reproducibility details.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.
In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All experiments use the publicly released ChatBench dataset [8 ] . Reproducibility details (sample construction, estimators, bootstrap settings, runtime) are in Appendix M; the code will be released publicly with the camera-ready version.
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: §4 and Appendices § 12 –§ 13 specifies necessary details.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Headline empirical claims in §4 (e.g., G A P = + 0.213 \mathrm{GAP} = +0.213 GAP = + 0.213 , β ^ = 0.285 \hat{\beta} = 0.285 β ^ = 0.285 , λ ^ L l a m a − λ ^ G P T = + 0.056 \hat{\lambda}^{\mathrm{Llama}} - \hat{\lambda}^{\mathrm{GPT}} = +0.056 λ ^ Llama − λ ^ GPT = + 0.056 ) are reported with 95% non-parametric bootstrap confidence intervals; and a worker-cluster bootstrap is reported in Appendix H as a robustness check.
Guidelines:
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Appendix M reports total runtime of approximately 4 minutes on a single CPU core for the entire pipeline.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?
Answer: [Yes]
Justification:
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes] Justification: The framework’s positive design implications are stated in §4 and §3. On the negative side, the analysis quantifies regimes where adding an AI assistant degrades worker accuracy, which cautions against blanket deployment. The paper introduces no new model or dataset with foreseeable misuse pathways.
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A]
Justification: The paper releases no pretrained model, image generator, or scraped dataset.
Guidelines:
The answer [N/A] means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external assets are cited at first use: ChatBench [8 ] for the human-AI interaction data and MMLU [13 ] for the underlying questions.
Guidelines:
The answer [N/A] means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [N/A]
Justification:
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: We did not run any new crowdsourcing or human-subjects experiment.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A] Justification: We did not run any new human-subjects research; all data come from the publicly released ChatBench study, which was conducted by its original authors.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: LLMs are studied as objects in this paper (GPT-4o and Llama-3.1-8b in the ChatBench data) but were not used as components of the methodology.
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.