betaFeedback

WorksMathematics

Measuring Public Institutions and their Agents: Disparities and Agreement, with Application to Asylum Decisions in the U.S. Immigration Courts

Abstract

Public institutions derive their legitimacy from the professionalism and impartiality of their agents. In order to maintain standards of conduct, the performance of an agent is often compared to the performance of her peers. But public institutions typically assign agents to different tasks, and as a result, naive comparisons may reflect the difficulty of the assignment and not an agent's performance. To address this concern, we develop measures of disparity and agreement that compare an agent's performance with the potential performance of randomly selected peers who could have been assigned the same set of tasks. We then give realistic conditions under which disparity is identified from the marginal distribution of each agent's actions. We also show how agreement, which is generally unidentified when agents are assigned to different tasks, can be bounded or estimated under sensitivity assumptions about the dependence between potential outcomes. Finally, we use a nonparametric Bayesian model to examine applications for asylum filed between 2007 and 2014 in the widely studied and often maligned U.S. immigration courts. Our results do not support the picture of widespread partiality asserted in previous empirical work. At both the judge and court level, we find limited disparity and substantial potential agreement.

Introduction

Public institutions are more likely to be perceived as trustworthy when they hold their agents accountable. To maintain accountability, it is necessary to evaluate agent performance.

If performance criteria are clear and easy to use, as when caseworkers from child protective services are evaluated on whether they made the mandatory number of visits to their clients last month, an audit study can determine if this condition was met. Similarly, teachers may be evaluated on whether a given percentage of students passed a standardized exam.

Yet even when criteria are clear, it is often difficult to determine whether they have been met. For example, officers are prohibited from using excessive force: “Officers may use force only when no reasonably effective, safe, and feasible alternative appears to exist and may use only the level of force that a reasonable officer on the scene would use under the same or similar circumstances” [8]. Similarly, U.S. immigration judges are required to grant asylum to applicants who complied with procedural criteria and demonstrate a well founded fear of persecution based on protected grounds (race, religion, nationality, political opinion or membership in a protected social category).

But how are we to assess whether an officer’s behavior is reasonable, or if an immigration judge has improperly denied admission to asylum seekers? That is the issue we address.

One approach is to use a ‘gold standard,’ measuring whether an agent’s performance on a set of model cases is consistent with the behavior expected under ideal conditions. But even though every agent is measured against the same standard, the model cases may not be representative of the situations that at least some agents encounter, as when difficult or easy cases are underrepresented. Moreover, agents may know how to respond to model cases, but behave differently in real life situations.

A second approach uses a ‘community standard.’ Here, there are no model cases. Instead, an agent’s behavior warrants further scrutiny (for outstanding or poor performance) if it deviates from the behavior of peers in actual situations. Although the community standard does not capture systemic departures from ideal behavior, as would occur if all agents discriminate in the same way, it is easy to implement—at least in theory—and widely used.

Yet determining whether an agent behaves like her peers is not always straightforward in practice. Encounters with the public are typically handled by just one agent. For example, one police officer makes a particular traffic stop. An evaluation using a community standard might compare officers according to the rate at which they use force in those stops, but then the challenge is to decide which other officers or stops make for a fair comparison. Simple disparity measures, such as the average difference in outcomes between agents and peers, can be misleading. In addition, such comparisons do not indicate whether agents and peers agree: this requires information on the unobserved joint distribution of potential outcomes.

This paper develops methodology for using community standards to assess the performance of public agents and the institutions they represent, focusing on the common situation in which each instance is handled by one agent. Our methodology is also applicable in other settings, e.g., large firms.

We proceed as follows. Measures of disparity, defined as a difference in performance outcomes between an agent and her peers in comparable situations, are given in § 2. While these can be identified from the marginal distributions of outcomes under each agent, agreement between the outcomes is typically of greater interest. Therefore, we also define measures of agreement, which depend on the unobserved joint distribution of agents’ potential behavior and are only partially identified from the marginal distributions. We tighten the traditional Fréchet-Hoeffding (hereafter FH) bounds under realistic assumptions about the evidence different agents see and how that evidence would be handled by another agent viewing the same case. We also outline additional conditions for estimation.

Our application to the U.S. immigration courts is presented in § 4. We begin by reviewing several influential studies that espoused the view, now widely held, that variation between courts and between judges within courts in the rate at which asylum seekers were granted reflect a fundamental lack of institutional consistency. We show that this work confuses disparities with agreement and fails to properly account for the fact that different judges adjudicate different types of cases. We then apply our methodology to data on asylum applications filed from May 2007 through September 30, 2014, using a non-parametric Bayesian model to estimate disparities between federal immigration judges and to bound the agreement between them. Our analyses suggest the conclusions in previous work are exaggerated and in need of tempering. Section 4 concludes.

Methodology: Estimands, Identification, and Estimation

We define estimands for disparity and agreement that compare the performance of agents with their peers. We consider the identification or partial identification of these estimands for the setting in which each case is handled by one agent only. We then discuss estimation.

We use the following notation. Let i∈{1,…,N}=Ii \in\{1,\ldots,N\} = \mathcal{I} index the set of cases, j∈{1,…,J}=Jj \in\{1,\ldots,J\} = \mathcal{J} the agents who ‘treat’ cases in I\mathcal{I}, Ij\mathcal{I}_j the set of cases assigned to agent jj, and Ji⊆J\mathcal{J}_i \subseteq\mathcal{J}, with two or more members, the set of agents ‘eligible’ to treat case i∈Ii \in\mathcal{I}. Each case is assigned to one agent: Ij∩Ij′=∅\mathcal{I}_j \cap\mathcal{I}_{j'} = \varnothing, j≠j′j \ne j'. Let ZiZ_i, taking values zi∈Jiz_i \in\mathcal{J}_i, denote the agent assigned to case ii. Also associated to each case are stochastic potential outcomes Yi(j)Y_i(j), j∈Jij \in\mathcal{J}_i, an observed outcome Yi=∑j∈Ji1{Zi=j}Yi(j)Y_i = \sum_{j \in\mathcal{J}_i} 1\{Z_i = j\}Y_i(j) that is continuous or binary, and a vector of recorded covariates Vi\mathbf{V}_i that does not depend on the agent assigned case ii.

We make the following assumptions, already reflected in the notation above:

A1. No Interference: For all i∈Ii \in\mathcal{I} and for any assignment vectors (z1,…,zN)(z_1,\ldots,z_N) and (z1′,…,zN′)(z'_1,\ldots,z'_N) with zi=zi′z_i = z'_i, Yi(z1,…,zN)=Yi(z1′,…,zN′)≡Yi(zi)Y_i(z_1,\ldots,z_N) = Y_i(z'_1,\ldots,z'_N) \equiv Y_i(z_i).

A2. Consistency: For all i∈Pi \in\mathcal{P} and j∈Jj \in\mathcal{J}, Zi=jZ_i = j implies Yi=Yi(j)Y_i = Y_i(j).

Under assumption A2, E[Yi∣Zi=j,Vi=v]=E[Yi(j)∣Zi=j,Vi=v]\mathbb{E}[Y_i \mid Z_i = j, \mathbf{V}_i = v] = \mathbb{E}[Y_i(j) \mid Z_i = j, \mathbf{V}_i = v].

Disparity

Measures Many measures have been used to quantify the disparity or inequality between different sub-groups of a population, e.g., men and women [9, 7]. Our goal is different. We aim to construct ‘local’ measures for comparing the performance of an agent jj with her peers and ‘global’ measures of institutional performance that take into account the behavior of all agents in J\mathcal{J}.

Previous researchers [e.g. 19] (GAO 2016) have used

D(j,j′)=Nj−1∑i∈IjYi−Nj′−1∑i∈Ij′Yi,(1)D(j,j') = N_j^{-1}\sum_{i\in\mathcal{I}_j}Y_i - N_{j'}^{-1}\sum_{i\in\mathcal{I}_{j'}}Y_i, \tag*{(1)}

the average difference for outcomes treated by jj versus outcomes treated by j′j', and

D(j,⋅)=Nj−1∑i∈IjYi−N−1∑i∈IYi,(2)D(j,\cdot) = N_j^{-1}\sum_{i\in\mathcal{I}_j}Y_i - N^{-1}\sum_{i\in\mathcal{I}}Y_i, \tag*{(2)}

the average deviation of jj from the institutional average, as disparity measures, where NjN_j is the cardinality of Ij\mathcal{I}_j.

In a completely randomized experiment with only two agents jj and j′j', (1) is an unbiased estimator of the average effect of assigning jj versus j′j' to the cases in I\mathcal{I}. But often agents are not randomly assigned to cases, and even when they are, they may not be eligible for assignment to the same cases. In such instances, (1) may not indicate how jj and j′j' differ were they assigned similar sets of cases. This suggests developing measures that compare agent jj only with peers who are eligible to be assigned to the cases jj is assigned.

Therefore, we define several local disparity measures that compare agent jj to her peers. The first is the average (taken over the distribution of cases assigned to agent jj) effect of assigning jj versus an eligible alternate jaj^a, randomly selected from Ji∖j\mathcal{J}_i \setminus j for each case ii:

D1(j)=Nj−1∑i∈IjE[Yi(j)−Yi(ja)∣Vi=vi].(3)D_1(j) = N_j^{-1} \sum_{i\in I_j} \mathbb{E}[Y_i(j) - Y_i(j^a) \mid V_i = v_i]. \tag*{(3)}

The second extends the measure (3) to the set of cases EjE_j that jj is eligible to treat,

D2(j)=(NjE)−1∑i∈EjE[Yi(j)−Yi(ja)∣Vi=vi],(4)D_2(j) = (N_j^E)^{-1} \sum_{i\in E_j} \mathbb{E}[Y_i(j) - Y_i(j^a) \mid V_i = v_i], \tag*{(4)}

where NjEN_j^E is the cardinality of EjE_j.

Analogously to the difference between the “effect of treatment on the treated” and the average treatment effect, D1(j)D_1(j) and D2(j)D_2(j) address somewhat different questions and may differ sharply, as is evident by re-expressing D2(j)D_2(j) as

D2(j)=(NjE)−1(NjD1(j)+∑i∈Ej−IjE[Yi(j)−Yi(ja)∣Vi=vi]);(5)D_2(j) = (N_j^E)^{-1} \left(N_jD_1(j) + \sum_{i\in E_j-I_j} \mathbb{E}[Y_i(j) - Y_i(j^a) \mid V_i = v_i]\right); \tag*{(5)}

for example, suppose Y∈{0,1}Y \in\{0,1\}, EjE_j consists of 100 cases of type ‘A,’ which jj treats with probability .75, for which Yi(j)−Yi(ja)=1Y_i(j) - Y_i(j^a) = 1, and 100 cases of type ‘B,’ which jj treats with probability .01, for which Yi(j)−Yi(ja)=−1Y_i(j) - Y_i(j^a) = -1. In line with the assessment practices typically employed by institutions, which evaluate agents only on the cases they were assigned, we restrict attention to measures defined over the distribution of case covariates in IjI_j.

The disparities above measure whether agent jj behaves similarly or not to her peers on average, but do not measure the magnitude of the differences between jj and her peers, which may be quite large even if (3) and (4) are small. Thus, we also consider a measure of the average size of the disparities,

D3(j)=Nj−1∑i∈Ij∣E[Yi(j)−Yi(ja)∣Vi=vi]∣.(6)D_3(j) = N_j^{-1} \sum_{i\in I_j} \left|\mathbb{E}[Y_i(j) - Y_i(j^a) \mid V_i = v_i]\right|. \tag*{(6)}

It is important to note that (6) is an average of the absolute value of the expected disparities, not the average of the expected absolute disparities, ∣Yi(j)−Yi(ja)∣|Y_i(j)-Y_i(j^a)|. The two are related by the inequality D3(j)≤Nj−1∑i∈IjE[∣Yi(j)−Yi(ja)∣∣Vi=vi]D_3(j) \le N_j^{-1}\sum_{i\in\mathcal{I}_j}\mathbb{E}[|Y_i(j)-Y_i(j^a)| \mid V_i=v_i]. We revisit this point in the subsequent section on agreement.

The local disparities above compare the performance of an agent with her peers, but do not measure institutional performance. To do so, we construct the global measure

D3(⋅)=∑j∈JNjND3(j),(7)D_3(\cdot)=\sum_{j\in\mathcal{J}}\frac{N_j}{N}D_3(j), \tag*{(7)}

the average of the disparities D3(j)D_3(j), weighted by caseload.

Identification Without additional assumptions, the disparities above are not identified ([27 page 62]). We consider identifying assumptions here, discussing additional assumptions for estimation subsequently.

Let ZiaZ_i^a, with values ja≠jj^a\ne j, denote an alternate agent selected for comparison with jj. The prima facie disparity between agent jj and her peers is then defined as

D1P(j)=Nj−1∑i∈Ij(E[Yi∣Zi=j,Vi=vi]−E[Yi∣Zia=ja,Vi=vi]).(8)D_{1P}(j)=N_j^{-1}\sum_{i\in\mathcal{I}_j}\left(\mathbb{E}[Y_i\mid Z_i=j,V_i=v_i]-\mathbb{E}[Y_i\mid Z_i^a=j^a,V_i=v_i]\right). \tag*{(8)}

We state assumptions under which D1P(j)=D1(j)D_{1P}(j)=D_1(j):

A3. Positivity (Eligibility): For all i∈Ii\in\mathcal{I} and j∈Jij\in\mathcal{J}_i, 0<Pr⁡(Zi=j∣Vi=vi)<10<\Pr(Z_i=j\mid V_i=v_i)<1.

A4. Unconfoundedness: For all i∈Ii\in\mathcal{I},

{Yi(j),Yi(ja)}j,ja∈Ji, j≠ja⊥ ⁣ ⁣ ⁣⊥(Zi,Zia)∣Vi,(9)\{Y_i(j),Y_i(j^a)\}_{j,j^a\in\mathcal{J}_i,\ j\ne j^a}\perp\!\!\!\perp(Z_i,Z_i^a)\mid V_i, \tag*{(9)}

where the symbol ⊥ ⁣ ⁣ ⁣⊥\perp\!\!\!\perp denotes statistical independence.

Under A1-A3, E[Yi∣Zi=j,Vi=vi]\mathbb{E}[Y_i\mid Z_i=j,V_i=v_i] is nonparametrically identified for each case i∈Pi\in\mathcal{P} and every agent j∈Jij\in\mathcal{J}_i. Under the additional assumption A4, E[Yi∣Zi=j,Vi=\mathbb{E}[Y_i\mid Z_i=j,V_i= Vi]=E[Yi(j)∣Vi=vi]V_i] = \mathbb{E}[Y_i(j) \mid V_i = v_i], j∈Jij \in\mathcal{J}_i. Thus, under A1-A4, D1(j)D_1(j), D3(j)D_3(j) and D3(⋅)D_3(\cdot) are nonparametrically identified.

Agreement

When a community standard is used to monitor the performance of agents, the agreement between agents’ outcomes is of interest. If agent jj tends to disagree with her peers, further scrutiny of her behavior may be warranted. Widespread disagreement among agents may suggest institutional failure. Thus we construct local and global measures of agreement analogous to the disparities in § 2.1.

Disparity measures are often used inappropriately to infer the agreement among agents. For example, Ramji-Nogales et al. [19] (hereafter RN), and GAO (2016) use the disparities D(j,j′)D(j,j^\prime) to infer widespread disagreement among federal immigration judges. Indeed, RN title their paper “Refugee Roulette: Disparities in Asylum Adjudication and Proposals for Reform.” Karen Musalo, another leading scholar, states “whether you win or lose should not depend on the roll of the dice of which judge gets your case” Rosenberg, Levinson, and McNeill, [20]. But such inferences are unwarranted, even in a completely randomized experiment. Disparity measures depend on the marginal distributions of agents’ potential outcomes, but agreement depends on the joint distribution. The marginal distributions may provide little information about the joint: for example, on a binary scale, two agents, each of whom grants asylum 50% of the time, will agree 50% of the time under independence, but may never agree or always agree if their classifications are dependent.

Measures for Categorical Outcomes For data measured on a nominal scale with CC categories, an obvious measure of agreement between two raters is pop^o, the percentage of cases on the diagonal of the C×CC \times C cross-classification table. In a landmark paper that shaped the subsequent literature, Cohen (1960) rejected the use of pop^o and proposed the measure κ=(po−pe)/(1−pe)∈[−1,1]\kappa= (p^o - p^e)/(1 - p^e) \in[-1,1], in which pep^e is the “chance agreement” estimated under the assumption that the two sets of ratings are independent. Many generalizations of κ\kappa, to three or more raters, also to ordinal data, have since been proposed [11].

Despite its widespread use with categorical data, we do not find κ\kappa or variants that adjust for chance agreement useful for measuring the performance of public institutions and their agents. If two agents independently classified each of KK cases, the joint distribution of outcomes would only be independent if the cases did not have common features that one or both agents used in their classifications. This is unlikely in real applications, and thus it is not clear why chance agreement should be the yardstick against which to measure actual agreement. Moreover, performance rankings based on κ\kappa do not necessarily reflect rankings based on pop^o, as κ1<κ2\kappa_1 < \kappa_2 does not imply p1o<p2op_1^o < p_2^o.

Similarly, while models for the association between agents may be useful for estimating agreement [2], measures of association are not indicative of agreement per se [3]. Agreement depends on the marginal distributions of outcomes of different agents, whereas association does not, when measured by the odds ratio or functions of the odds ratio. Consider the two by two table with row margins .9 and .1 and column margins .1 and .9, where the agents agree 20% of the time when the odds ratio is infinite, whereas in the two by two table with homogeneous margins .9 and .1, 82% of the cases lie on the diagonal under independence.

Let Pr⁡(Yi(j)=y,Yi(ja)=y′∣Vi=vi)≡πiy,y′(j,ja)\Pr(Y_i(j)=y, Y_i(j^a)=y' \mid V_i=v_i) \equiv\pi_i^{y,y'}(j,j^a) denote the probability agent j∈Jij \in\mathcal{J}_i responds yy to case ii and agent jaj^a responds y′y', and let ∑yπiy,y(j,ja)≡AiA(j,ja)\sum_y \pi_i^{y,y}(j,j^a) \equiv A_i^A(j,j^a) denote the probability jj and jaj^a agree, where jaj^a is randomly selected from Ji∖j\mathcal{J}_i \setminus j for each case ii as in § 2.1. Let Aiy∣y(ja∣j)≡Pr⁡(Yi(ja)=y∣Yi(j)=y,Vi=vi)A_i^{y \mid y}(j^a \mid j) \equiv\Pr(Y_i(j^a)=y \mid Y_i(j)=y, V_i=v_i).

To compare agent jj with her peers on the cases she was assigned, two measures are of primary interest, the probability jj agrees with peers,

A1A(j)=Nj−1∑i∈IjAiA(j,ja),(10)A_1^A(j) = N_j^{-1}\sum_{i\in\mathcal{I}_j} A_i^A(j,j^a), \tag*{(10)}

and the conditional probability jaj^{a} agrees with jj,

A1y∣y(j)=Nj−1∑i∈IjA1y∣y(ja∣j).(11)A^{y|y}_{1}(j)=N_j^{-1}\sum_{i\in\mathcal{I}_j}A^{y|y}_{1}(j^{a}\mid j). \tag*{(11)}

Global analogues of the local measures are also readily constructed,

A1(⋅)=∑j∈JNjNA1(j),(12)A_{1}(\cdot)=\sum_{j\in\mathcal{J}}\frac{N_j}{N}A_{1}(j), \tag*{(12)}
A1y∣y(⋅)=∑j∈JNjNA1y∣y(j).(13)A^{y|y}_{1}(\cdot)=\sum_{j\in\mathcal{J}}\frac{N_j}{N}A^{y|y}_{1}(j). \tag*{(13)}

While our measures of disparity and agreement do not depend on the fact that Yi(j)Y_i(j) and Yi(ja)Y_i(j^{a}) are binary, in such cases, agreement and disparity are related by the identities

Ai=(j,ja)=1−E[∣Yi(j)−Yi(ja)∣∣Vi=vi]=1−E[(Yi(j)−Yi(ja))2∣Vi=vi]=1−Di1(j,ja)−2πi0,1(j,ja),(14)\begin{aligned} A_i^{=}(j,j^{a}) &= 1-\mathbb{E}[|Y_i(j)-Y_i(j^{a})|\mid V_i=v_i]\\ &=1-\mathbb{E}[(Y_i(j)-Y_i(j^{a}))^2\mid V_i=v_i]\\ &=1-D_{i1}(j,j^{a})-2\pi_i^{0,1}(j,j^{a}), \tag*{(14)} \end{aligned}

where disparity Di1(j,ja)≡E[Yi(j)−Yi(ja)∣Vi=vi]D_{i1}(j,j^{a})\equiv\mathbb{E}[Y_i(j)-Y_i(j^{a})\mid V_i=v_i] is identified from the marginal distributions πiy(j)≡Pr⁡(Yi(j)=y∣Vi=vi)=E[Yi(j)∣Vi=vi]\pi_i^{y}(j)\equiv\Pr(Y_i(j)=y\mid V_i=v_i)=\mathbb{E}[Y_i(j)\mid V_i=v_i], and πiy(ja)≡Pr⁡(Yi(ja)=y∣Vi=vi)=E[Yi(ja)∣Vi=vi]\pi_i^{y}(j^{a})\equiv\Pr(Y_i(j^{a})=y\mid V_i=v_i)=\mathbb{E}[Y_i(j^{a})\mid V_i=v_i], but πi0,1(j,ja)\pi_i^{0,1}(j,j^{a}) is not. Without further assumptions, the agreement is only “partially identified” by the FH bounds

1−Di1(j,ja)−2min⁡(πi0(j),πi1(ja))≤Ai=(j,ja)1-D_{i1}(j,j^{a})-2\min(\pi_i^{0}(j),\pi_i^{1}(j^{a}))\leq A_i^{=}(j,j^{a})
≤1−Di1(j,ja)−2max⁡(0,πi0(j)+πi1(ja)−1).(15)\leq1-D_{i1}(j,j^{a})-2\max(0,\pi_i^{0}(j)+\pi_i^{1}(j^{a})-1). \tag*{(15)}

Bounds for measures (10)–(13) may be constructed from (15), a point we revisit subsequently.

Measures for Continuous Outcomes

Agreement between two raters jj and jaj^{a} on a continuous scale YY is commonly measured by the the intra-class correlation or concordance correlation coefficient,

2Cov⁡(Y(j),Y(ja))Var⁡(Y(j))+Var⁡(Y(ja))+E[Y(j)−Y(ja)]2(16)\frac{2\operatorname{Cov}(Y(j),Y(j^{a}))}{\operatorname{Var}(Y(j))+\operatorname{Var}(Y(j^{a}))+\mathbb{E}[Y(j)-Y(j^{a})]^2} \tag*{(16)}

(Lin 1989; Lin, Hedayat, Sinha and Yang 2002), motivating the local measure L(j)L(j),

Nj−1∑i∈Ij2Cov⁡(Yi(j),Yi(ja)∣Vi=vi)Var⁡(Yi(j)∣Vi=vi)+Var⁡(Yi(ja)∣Vi=vi)+E[Yi(j)−Yi(ja)∣Vi=vi]2(17)N_j^{-1}\sum_{i\in\mathcal{I}_j}\frac{2\operatorname{Cov}(Y_i(j),Y_i(j^{a})\mid V_i=v_i)}{\operatorname{Var}(Y_i(j)\mid V_i=v_i)+\operatorname{Var}(Y_i(j^{a})\mid V_i=v_i)+\mathbb{E}[Y_i(j)-Y_i(j^{a})\mid V_i=v_i]^2} \tag*{(17)}

Global agreement can then be measured as the weighted average of the local measures,

L(⋅)=∑j∈JNjNL(j).(18)L(\cdot)=\sum_{j\in\mathcal{J}}\frac{N_j}{N}L(j). \tag*{(18)}

Equation (16) is a natural extension of the second line of equation (14). When YY is continuous and bounded, m≤Y≤Mm\leq Y\leq M, we define

Ai(j,ja)≡1−E[(Yi(j)−Yi(ja))2](M−m)2,(19)A_i(j,j^{a})\equiv1-\frac{\mathbb{E}[(Y_i(j)-Y_i(j^{a}))^2]}{(M-m)^2}, \tag*{(19)}

normalized here to take value 0 under maximal disagreement, and which reduces to the categorical case discussed in the previous section when the outcome is binary. When YY is unbounded, observe that the numerator of (19) can be expanded as in

E[(Y1−Y2)2]=Var⁡(Y1)+Var⁡(Y2)−2Cov⁡(Y1,Y2)+(E[Y1]−E[Y2])2.\mathbb{E}[(Y_1-Y_2)^2]=\operatorname{Var}(Y_1)+\operatorname{Var}(Y_2)-2\operatorname{Cov}(Y_1,Y_2)+(\mathbb{E}[Y_1]-\mathbb{E}[Y_2])^2.

When the covariance is positive, (M−m)2(M-m)^2 can thus be replaced by the upper bound c2=Var⁡(Y1)+Var⁡(Y2)+(E[Y1]−E[Y2])2c_2=\operatorname{Var}(Y_1)+\operatorname{Var}(Y_2)+(\mathbb{E}[Y_1]-\mathbb{E}[Y_2])^2, in which case (19) reduces to (16).

Other extensions are possible. Using the first line of equation (14) would produce the numerator E[∣Yi(j)−Yi(ja)∣]\mathrm{E}[|Y_i(j)-Y_i(j^a)|]. The denominator would be M−mM-m in the bounded case, c1=E[∣Y1∣+∣Y2∣]c_1=\mathrm{E}[|Y_1|+|Y_2|] in the unbounded case, or if second moments exist, c2\sqrt{c_2} or min⁡(c1,c2)\min(c_1,\sqrt{c_2}).

Another approach is to replace the denominator with the FH bounds using the distribution functions FY1F_{Y_1} and FY2F_{Y_2}. Measures of conditional agreement can be constructed analogously. Further consideration of the continuous case, including the partial identification of agreement measures when one agent is observed per case, will be taken up in future work.

Identification. When each case is handled by one agent, measures A1=(j)A_1^{=}(j), A1=(⋅)A_1^{=}(\cdot), L(j)L(j), and L(⋅)L(\cdot) are identified up to the FH bounds. Point identification requires additional assumptions.

The simplest assumption is that outcomes are independent given observed covariates, Yi(j)⊥ ⁣ ⁣ ⁣⊥Yi(ja)∣ViY_i(j) \perp\!\!\!\perp Y_i(j^a) \mid V_i. Agreement is then identified from the marginal distributions. But this assumption will be violated in the common scenario in which ViV_i fails to include relevant but unrecorded information that both jj and jaj^a would see in handling case ii, where Ui(j)=Ui(ja)≡UiU_i(j)=U_i(j^a)\equiv U_i. For example, observed covariates rarely capture the weight of the evidence an asylum seeker might present at trial to support a fear of persecution. But even though UiU_i is unobserved, consideration of the way (Vi,Ui)(V_i,U_i) is likely to impact the joint distribution of outcomes is useful for tightening the FH bounds.

We begin by strengthening the unconfoundedness assumption A4:

A4.* Unconfoundedness: For all i∈Ii\in\mathcal{I},

{Yi(j),Yi(ja)}j,ja∈Ji, j≠ja,Ui⊥ ⁣ ⁣ ⁣⊥(Zi,Zia)∣Vi.(20)\{Y_i(j),Y_i(j^a)\}_{j,j^a\in\mathcal{J}_i,\ j\ne j^a},U_i \perp\!\!\!\perp(Z_i,Z_i^a)\mid V_i. \tag*{(20)}

Assumption A4* implies unconfoundedness given (Vi,Ui)(V_i,U_i) and also unconfoundedness given ViV_i (Assumption A4). In other words, for each case ii, UiU_i is a vector of “unconfounded unmeasured predictors”, not an unobserved confounder.

We also assume that if agents jj and jaj^a observed case ii, with information (Vi,Ui)(V_i,U_i), each agent would evaluate the case independently of the other agent, and the two agents would process the additional information in Ui\mathbf{U}_i in a similar manner:

A5. Independence of Agents Outcomes with Respect to Recorded and Unmeasured Information: For all i∈Ii \in\mathcal{I},

Yi(j)⊥ ⁣ ⁣⊥Yi(ja)∣Vi,Ui,(21)Y_i(j) \perp\!\!\perp Y_i(j^a) \mid\mathbf{V}_i,\mathbf{U}_i, \tag*{(21)}

A6. Common Interpretation of Unmeasured Information: If

E[Yi(j)∣Vi=v,Ui=u]≥E[Yi(j)∣Vi=v,Ui=u∗],\mathbb{E}[Y_i(j) \mid\mathbf{V}_i = \mathbf{v}, \mathbf{U}_i = \mathbf{u}] \ge\mathbb{E}[Y_i(j) \mid\mathbf{V}_i = \mathbf{v}, \mathbf{U}_i = \mathbf{u}^*],

then

E[Yi(ja)∣Vi=v,Ui=u]≥E[Yi(ja)∣Vi=v,Ui=u∗].\mathbb{E}[Y_i(j^a) \mid\mathbf{V}_i = \mathbf{v}, \mathbf{U}_i = \mathbf{u}] \ge\mathbb{E}[Y_i(j^a) \mid\mathbf{V}_i = \mathbf{v}, \mathbf{U}_i = \mathbf{u}^*].

A Covariance Bound Many measures, including A1=(j)A_1^{=}(j), A1=(⋅)A_1^{=}{(\cdot)}, L(j)L(j), L(⋅)L(\cdot), depend on the covariances Cov⁡(Yi(j),Yi(ja)∣Vi=vi)\operatorname{Cov}(Y_i(j), Y_i(j^a) \mid\mathbf{V}_i = \mathbf{v}_i). For example, in the binary case, (14) implies

A1=(j)=Nj−1∑i∈Pjπi=(j,ja)=Nj−1∑i∈Pj[1−πiy(j)−πiy(ja)+2πiy(j)πiy(ja)+2Cov⁡(Yi(j),Yi(ja)∣Vi=vi)]=Nj−1∑i∈Pj[1−πiy(j)−πiy(ja)+2πiy(j)πiy(ja)]+2E[Cov⁡(Yi(j),Yi(ja)∣Vi)].(22)\begin{aligned} A_1^{=}(j) = N_j^{-1} \sum_{i \in\mathcal{P}_j} \pi_i^{=}(j,j^a) \\ &= N_j^{-1} \sum_{i \in\mathcal{P}_j} \left[1-\pi_i^y(j)-\pi_i^y(j^a)+2\pi_i^y(j)\pi_i^y(j^a)+2\operatorname{Cov}(Y_i(j),Y_i(j^a)\mid\mathbf{V}_i=\mathbf{v}_i)\right] \\ &= N_j^{-1} \sum_{i \in\mathcal{P}_j} \left[1-\pi_i^y(j)-\pi_i^y(j^a)+2\pi_i^y(j)\pi_i^y(j^a)\right]+2\mathbb{E}\left[\operatorname{Cov}(Y_i(j),Y_i(j^a)\mid\mathbf{V}_i)\right]. \tag*{(22)} \end{aligned}

The expectation is taken over the empirical distribution of cases assigned to agent jj.

We now use assumptions A5 and A6 to bound the unidentified covariances, and thus the agreement measures. We begin with the covariance between agent outcomes, conditioned on the observed information, which can be written

Cov⁡(Yi(j),Yi(ja)∣Vi)=E[Cov⁡(Yi(j),Yi(ja)∣Ui,Vi)∣Vi]+Cov⁡(E[Yi(j)∣Ui,Vi],E[Yi(ja)∣Ui,Vi]∣Vi).(23)\operatorname{Cov}(Y_i(j),Y_i(j^a)\mid\mathbf{V}_i) = \mathbb{E}\left[\operatorname{Cov}(Y_i(j),Y_i(j^a)\mid\mathbf{U}_i,\mathbf{V}_i)\mid\mathbf{V}_i\right] +\operatorname{Cov}\left(\mathbb{E}[Y_i(j)\mid\mathbf{U}_i,\mathbf{V}_i],\mathbb{E}[Y_i(j^a)\mid\mathbf{U}_i,\mathbf{V}_i]\mid\mathbf{V}_i\right). \tag*{(23)}

By A5, the first term on the right hand side of (23) is 0. Furthermore, A6 implies the second term is positive, Cov⁡(E[Yi(j)∣Ui,Vi],E[Yi(ja)∣Ui,Vi]∣Vi)≥0\operatorname{Cov}\left(\mathbb{E}[Y_i(j) \mid\mathbf{U}_i,\mathbf{V}_i],\mathbb{E}[Y_i(j^a) \mid\mathbf{U}_i,\mathbf{V}_i] \mid\mathbf{V}_i\right) \ge0. It follows that Cov⁡(Yi(j),Yi(ja)∣Vi=v)≥0\operatorname{Cov}(Y_i(j),Y_i(j^a) \mid V_i=v) \ge0. Thus, in the categorical case, Ai−(j,ja)≥∑yπiy(j)πiy(ja)A_i^-(j,j^a) \ge\sum_y \pi_i^y(j)\pi_i^y(j^a).

To obtain upper bounds that are tighter than the FH bounds, we again use the law of total covariance, which, under A5, yields

Cov⁡(Yi(j),Yi(ja))=E[Cov⁡(E[Yi(j)∣Ui,Vi],E[Yi(ja)∣Ui,Vi]∣Vi)]+Cov⁡(E[Yi(j)∣Vi],E[Yi(ja)∣Vi]),(24)\operatorname{Cov}(Y_i(j),Y_i(j^a)) = \mathbb{E}\left[\operatorname{Cov}\left(\mathbb{E}[Y_i(j) \mid\mathbf{U}_i,\mathbf{V}_i],\mathbb{E}[Y_i(j^a) \mid\mathbf{U}_i,\mathbf{V}_i] \mid\mathbf{V}_i\right)\right] + \operatorname{Cov}\left(\mathbb{E}[Y_i(j) \mid\mathbf{V}_i],\mathbb{E}[Y_i(j^a) \mid\mathbf{V}_i]\right), \tag*{(24)}

a decomposition of the covariance as the sum of a “within” Vi\mathbf{V}_i component measuring the added covariation attributable to the unrecorded information Ui\mathbf{U}_i and a “between” component measuring the covariation between agents expectations, given the observed information Vi\mathbf{V}_i.

Of interest here is the unidentified “within component”. However, the “between” covariance is identified, and if positive, as when agents jj and jaj^a react similarly to observed conditions, it can serve as a yardstick for the unidentified within covariance, which can be written as a multiple α≥0\alpha\ge0 of the “between” component. We can then replace the within component by this multiple of the between component under various assumptions about the value of α\alpha. For example, if α=0\alpha= 0, the information in Ui\mathbf{U}_i adds no predictive value net of Vi\mathbf{V}_i; if α=1\alpha= 1, the predictive value of the unrecorded information is of equal magnitude to the predictive value of the recorded information; and so on. As the net contribution of the information in Ui\mathbf{U}_i increases, α\alpha increases, pushing agreement toward the upper FH bound.

Estimation

Disparity Our goal is to estimate the case-level expected outcomes E[Yi(j)∣Vi]\mathbb{E}[Y_i(j) \mid\mathbf{V}_i] and E[Yi(ja)∣Vi]\mathbb{E}[Y_i(j^a) \mid\mathbf{V}_i] for the set i∈Iji \in\mathcal{I}_j, from which the estimation of the disparity measures follow. While the estimation of E[Yi(j)∣Vi]\mathbb{E}[Y_i(j) \mid\mathbf{V}_i] is straightforward for i∈Iji \in\mathcal{I}_j under assumptions previously made, estimation of E[Yi(ja)∣Vi=vi]\mathbb{E}[Y_i(j^a) \mid\mathbf{V}_i = \mathbf{v}_i] for i∈Iji \in\mathcal{I}_j must be constructed from estimates of E[Yk(ja)∣Vk=vi]\mathbb{E}[Y_k(j^a) \mid\mathbf{V}_k = \mathbf{v}_i] for k∈Ijak \in\mathcal{I}_{j^a}, necessitating additional assumptions:

A7. Independent Sampling: The random data vectors (Q1,…,QN)(Q_1,\ldots,Q_N) are statistically independent, where Qi=({Yi(j)}j∈Ji,Zi,Zia,Vi)Q_i = (\{Y_i(j)\}_{j\in\mathcal{J}_i}, Z_i, Z_i^a, \mathbf{V}_i),

A8. Intra-Agent Consistency: For every ii, ja∈Ji∖jj^a \in\mathcal{J}_i \setminus j, and k∈Ijak \in\mathcal{I}_{j^a},

E[Yi(ja)∣Vi=v,Ui=u]=E[Yk(ja)∣Vk=v,Uk=u].\mathbb{E}[Y_i(j^a) \mid\mathbf{V}_i = \mathbf{v}, \mathbf{U}_i = \mathbf{u}] = \mathbb{E}[Y_k(j^a) \mid\mathbf{V}_k = \mathbf{v}, \mathbf{U}_k = \mathbf{u}].

Assumption A8 justifies using the outcomes of cases jaj^a was assigned to estimate the outcomes of cases jaj^a was eligible for but not in fact assigned. We assume agent jaj^a would treat both sets of cases the same, provided the same information u\mathbf{u} and v\mathbf{v} were observed.

A final assumption is necessary because U\mathbf{U} is not observed, and we can only estimate E[Yi(j)∣Vi=vi]\mathbb{E}[Y_i(j) \mid\mathbf{V}_i = \mathbf{v}_i] and E[Yk(ja)∣Vk=vi]\mathbb{E}[Y_k(j^a) \mid\mathbf{V}_k = \mathbf{v}_i].

A9. Exchangeability of Unmeasured Characteristics: For all i∈Ii \in\mathcal{I}, Ui∣Vi=v\mathbf{U}_i \mid\mathbf{V}_i = \mathbf{v} are independently and identically distributed.

Assumptions A8 and A9 jointly imply E[Yi(ja)∣Vi=v]=E[Yk(ja)∣Vk=v]\mathbb{E}[Y_i(j^a) \mid\mathbf{V}_i = \mathbf{v}] = \mathbb{E}[Y_k(j^a) \mid\mathbf{V}_k = \mathbf{v}].

Agreement In Section 2.2, we proposed a covariance bound for agreement measures such as Ai=(j)A_i^{=}(j) and Ai−(⋅)A_i^{-}(\cdot), which are partially identified from the marginal expectations E[Yi(j)∣Vi]\mathbb{E}[Y_i(j) \mid\mathbf{V}_i], E[Yi(ja)∣Vi]\mathbb{E}[Y_i(j^a) \mid\mathbf{V}_i]. Having estimated the marginal expectations under assumptions such as A1–A9, one can estimate agreement using various α\alpha values, replacing Cov⁡(Yi(j),Yi(ja)∣Vi)\operatorname{Cov}(Y_i(j), Y_i(j^a) \mid\mathbf{V}_i) with αCov⁡(E[Yi(j)∣Vi],E[Yi(ja)∣Vi])\alpha\operatorname{Cov}(\mathbb{E}[Y_i(j) \mid\mathbf{V}_i], \mathbb{E}[Y_i(j^a) \mid\mathbf{V}_i]) in equation (22). If the information in Ui\mathbf{U}_i is irrelevant, given Vi\mathbf{V}_i, α=0\alpha= 0 and the agents’ classifications are independent. As α\alpha increases, the value of the information in Ui\mathbf{U}_i increases, pushing agreement toward the upper FH bound. It is important to note that when we estimate agreement between judges using different values of α\alpha, we always threshold the agreement at the upper FH bound.

A related approach is to pretend Vi=(Vi1,Vi2)\mathbf{V}_i = (\mathbf{V}_{i1}, \mathbf{V}_{i2}) contains all the relevant information and to compute the relative contribution α(Vi2;Vi1)\alpha(\mathbf{V}_{i2}; \mathbf{V}_{i1}) when Vi2\mathbf{V}_{i2} is omitted. Both approaches are similar in spirit to sensitivity analyses in causal inference that ask how strong the association between confounders and outcomes and confounders and treatment must be to drive a causal effect to 0 (e.g., [6]). But here we are not attempting to minimize or maximize agreement, and as Ui\mathbf{U}_i is not a confounder for case ii only the additional association induced by Ui\mathbf{U}_i net of Vi\mathbf{V}_i is of interest.

Application: Disparity and Disagreement in the U.S.

Under the Geneva Convention of July, 28 1951 and its 1967 protocol, the United States, a signatory, is required to grant asylum to an applicant who arrives at a port of entry or is inside the USA, and qualifies as defined in the Refugee Act of 1980 as a person “unwilling or unable to return to” their home country due to a “well founded fear of persecution based on race, religion, nationality, membership in a particular social group, or political opinion.”

Two types of asylum cases end up in the Executive Office for Immigration Review (EOIR) courts. Affirmative applicants first petition the United States Citizenship and Immigration Service (USCIS), where an asylum officer conducts a “merits” review and decides whether to grant asylum to the applicant. Removal proceedings for unsuccessful applicants without lawful status are then initiated with a notice to appear (NTA) in court before an EIOR judge for a “master calendar” hearing. Persons who did not apply affirmatively and who have been apprehended by U.S. authorities also receive an NTA and may apply “defensively,” as may persons in expedited removal proceedings who demonstrate to the satisfaction of the asylum officer a credible fear of torture or persecution if returned to their home country.

Recipients of the NTA who fail to attend the master calendar hearing are typically issued a removal order “in absentia.” Those who attend are asked to admit to or deny the allegations in the NTA and to state what type of relief is sought, if any. For persons who request relief (e.g., asylum), the judge then schedules a merits hearing over which he/she normally presides. Some applicants withdraw their request prior to the hearing. The remaining applicants present evidence to support their request at the hearing, after which the judge rules on their application. Of the 56,898 affirmative applications filed in 2014, 14,758 applicants were granted asylum, and of the more than 30,000 defensive applicants, 8,775 were granted asylum [18].

A Review of Two Influential Studies

‘Large’ disparities in asylum grant rates within and across courts have been reported. [17, 19] (GAO 2016; TRAC 2007; TRAC 2017; TRAC 2020; TRAC 2021). We examine two prominent studies to illustrate the evidence immigration researchers and policy advocates have used to conclude the EOIR judiciary lacks impartiality.

Example 1: Refugee Roulette The dataset analyzed by RN contain 269,756 EOIR cases. After excluding “cases that were abandoned, withdrawn or disposed of in some other way” (p. 396), 140,428 EIOR “merits” cases remained. Subsequent exclusions include affirmative applications filed by Mexican nationals, who the authors claim were primarily seeking other forms of relief, and most applicants held in detention. Cases with multiple family members were bundled together under a “lead” case and treated unitarily, as asylum is typically granted (or not) to the lead and all secondary (rider) cases bundled with the lead.

RN then analyze almost 80,000 cases that were decided between 2000 and 2004, focusing on seventeen “high volume” courts in which more than 1,500 applications per court were handled during the study period, and to applicants from fifteen nationalities where more than 500 cases per nationality were filed during 2004. They view large cross-court differences, for example that between Orlando and Atlanta, with grant rates of 76% and 7%, respectively, for Chinese applicants, as evidence that judges do not behave impartially.

Substantial differences were also observed within courts, where judges are randomly assigned to cases. In Los Angeles, among the 23 judges who heard 50 or more cases with Chinese applicants, grant rates ranged from 9% to 81% for a disparity D(j,j′)=.72D(j,j') = .72 (.50 excluding the lowest and highest rates), and among the 23 judges in Miami who heard 50 or more cases from Columbian applicants, the grant rates ranged from 5% to 88% for a disparity D(j,j′)=.83D(j,j') = .83 (.71 excluding the lowest and highest rates). In Miami, Los Angeles and New York, approximately one third of the judges deviate from the court-wide average by more than 50%. RN conclude (p. 296): “In many cases, the most important moment in an asylum case is the instant in which a clerk randomly assigns an application to a particular asylum officer or immigration judge”.

RN also use logistic regression with a somewhat different sample of 66,443 affirmative cases to study the relationship of outcomes to judge and case characteristics. Applicants with legal representation, dependents, and who came from countries with a low national freedom index were more likely to obtain asylum. A judge’s prior work experience and gender, with female judges more likely to grant asylum, were also associated with outcomes. Given the large sample size, several other predictors were statistically significant, but with substantively insignificant coefficients.

Example 2: GAO 2016 The GAO analyze cases in 28 “high volume” courts that were decided between 1995 and 2014. Like RN, applications from “low volume” countries, from judges who heard fewer than fifty affirmative cases and fifty defensive cases in the court they primarily served were excluded, and applications with multiple family members were bundled. But whereas RN excluded applications that were abandoned, withdrawn, or otherwise closed, the GAO treated these as denials. Another important difference is that GAO excluded cases from applicants in detention. Cases where the applicant or the government appealed the judge’s decision were also excluded.

In an earlier study focusing on the period before May 2007, the GAO (2008) found substantial variation in grant rates among nationalities, courts, and judges. For the new period, May 2007 through fiscal year 2014, among affirmative applications, grant rates varied from less than 5% in Atlanta and Omaha to 66% in New York, from 0 to 31.3% among judges in Los Angeles, and from 22.9% to 83.3% among judges in New York. Similar results, with somewhat lower grant rates, were reported for defensive applications.

For each of four time periods and separately for defensive and affirmative applications, the GAO also estimated logistic-normal mixed effects models. Applicants with dependents and/or legal representation who applied affirmatively between May 2007 and the end of fiscal year 2014 were more likely to obtain asylum; RN reported similar results. Female judges and judges appointed by Bush senior were more likely to grant asylum than judges appointed by Bush junior or Obama, and years of service was negatively associated with success; 95% confidence intervals for variances of random effects for judges, courts, and nationality and language of the asylum seeker were all bounded away from 0.

The GAO also attempted to use these models to account for inconsistency between judges by constructing a 95% prediction interval for a ‘representative applicant’ who comes before two judges with the same characteristics, but different judge random effects. For the 2007-2014 period, the upper and lower endpoints differ by 47 and 57 percentage points, respectively, for affirmative and defensive applicants. Were judge characteristics not held constant, presumably the difference would be even larger.

Reanalysis of Disparity and Disagreement

The validity of our analyses depends on satisfying the assumptions in § 2. To do so, substantial departures from the analyses in RN and GAO are required.

Recipients of an NTA typically learn the identity of the judge assigned to their case prior to the master calendar hearing. Affirmative applicants apply for relief prior to assignment. The remaining recipients apply defensively (if at all) after the judge is assigned and are thus of two types. One type would apply regardless of the judge assigned, while the other may choose to apply depending on the judge assigned to the case. These two types cannot be differentiated from the data and therefore, unlike RN and the GAO, we only consider affirmative applications. RN also excluded cases that were abandoned, withdrawn, or otherwise closed. But while the judge assignment could affect whether or not a defensive application is filed, affirmative applications are filed before a judge is assigned. Therefore, like GAO, we classify these outcome as “not granted.”

To justify interpreting the measures (1) and (2) as evidence of disparity and inconsistency, RN cite the random assignment, within courts, of judges to cases. But this interpretation is unwarranted because some judges may be more likely to be assigned certain types of cases, e.g., cases with minors. In addition, judges do not necessarily serve for the entire study period, and the composition of cases may also vary over time. In fact, two judges with non-overlapping service periods are not even eligible to hear any of the same cases. We avoid this problem by comparing the judge randomly assigned to case ii only to other judges who could have been assigned that case. Consistent with this, and unlike RN and GAO and others who interpret “low” grant rates in some courts, as compared to others, as evidence of judicial bias, we do not make cross-court comparisons, as such interpretations are also consistent with other explanations, e.g., differences in the composition of cases.

Further, RN and GAO fail to guard against post-treatment confounding. Both analyze cases decided during their study periods. These include difficult cases filed long before the study period began and cases filed toward the end of the period that were quickly resolved. To avoid any potential problems due to non-ignorable censoring, we analyze cases filed (as versus decided) during our study period. They also associate outcomes to the judge who closed the case, who might be different than the judge initially assigned. This is problematic if reassignments are not randomized. But even if they were, it is not clear that an analysis using the closing judges would be preferred: outcomes for cases ii reassigned to judge jj may not be indicative of those that would have occurred had jj been initially assigned because actions taken by the initially assigned judge may shape the course of future proceedings [21]. They also note that applicants with legal representation were more likely to obtain asylum, and in their models of asylum outcomes, representation is included as a predictor. We do not include this predictor: in our data the decision to obtain representation or not was made for more than 10% of the cases after the master calendar hearing, and may have depended on the judge assigned.

Finally, even if all judges see similar cases on average, the largest within court disparities D(j,j′)D(j,j^{\prime}), used by RN and GAO as evidence of judicial inconsistency, may not be indicative of more typical disparities or levels of agreement between judges.

Data Our study period, May 1, 2007 to September 30, 2014, spans the same interval as period 4 of the GAO study. While GAO studied cases decided during this period we study affirmative applications submitted during this period to avoid potential biases due to censoring. Like RN and GAO, groups of individuals that applied together are treated as a single unit. During this period, 122,235 applications were filed. We consider the subset of 120,483 applications where the petitioner had not previously applied for asylum. See Section A of the Supplement for additional information on the construction of the data set used.

In 2014, there were 59 immigration courts. During our study period, more than half the cases were first heard in the top three courts (New York City (34.5% of cases), West Los Angeles (20.45%), San Fransisco (5.3%), and 90% of cases were heard at the top 16 locations: as we model outcomes court by court, attention is limited to these locations.

Each case ii is assigned to a judge. In 28.7% of cases, different judges presided over the first and last proceeding. We cannot account for the re-assignment mechanism. As in ‘intent to treat’ analyses, we associate the outcome of case ii to the judge first randomly assigned to the case. We must also select a judge jaj^{a} who could have been assigned case ii but was not.

Judges are typically assigned to cases after the NTA, which specifies the date of the master calendar hearing, has been issued: we assume jaj^{a} is chosen from the judges scheduled to be in court on that date. An alternate judge was available for 92,218 (of 102,388 cases in the top 16 courts); cases without alternates were omitted from analyses. For additional information on the 16 courts and the judges in these courts see Section B of the Supplement.

We use Bayesian models to compare assigned judges with their alternates. We do not attempt to estimate models for individual judges. Thus, in addition to case covariates Vi\mathbf{V}_{i}, to account for heterogeneity between judges, we include in our models covariates Wj\mathbf{W}_{j} that have been shown to be predictive of outcomes in previous work. Like GAO, we include cases that were granted (29.6%), denied (10.3%), abandoned (6.4%), withdrawn (17.5%) or other (36.2%). Unlike GAO, we do not exclude cases where the judge’s decision was appealed: the outcome Yi=1Y_{i}=1 if case ii resulted in a grant of asylum, 0 otherwise.

The case covariates Vi\mathbf{V}_{i} are: 1) RIDERS, 1 if there are riders attached to the case, 0 otherwise; 2) N_RIDERS, the number of riders; 3) ENGLISH, 1 if applicant speaks English, 0 otherwise, 4) SPANISH, 1 if applicant speaks Spanish, 0 otherwise, 5) CHINA, 1 if applicant is Chinese, 0 otherwise, 5) LANGUAGE, primary language spoken by the applicant, 6) NATIONALITY, the applicant’s nationality; 7) APP_YEAR, the year the asylum application was filed; 8) ON_TIME, a categorical variable coded 0 if the applicant’s arrival time is unknown, 1 if the arrival time is known and application was made within one year of arrival in the US, 2 otherwise; 9) NTA_YEAR, the year the notice to appear was sent; 10) HEARING_DATE, the date (year-month-date) the hearing was held.

The judge covariates Wj\mathbf{W}_{j} are: 1) SEX, 1 if male, 0 otherwise; 2) WORK1, 1 for previous work experience with Immigration and Naturalization Services or Immigration and Customs Enforcement, 0 otherwise; 3) WORK2, 1 for previous work experience with the Department of Justice, 0 otherwise, 4) MILITARY, 1 for prior military service, 0 otherwise; 5) YEAR_BA, year the judge received the bachelor’s degree; 6) YEAR_JD, year the judge received the law degree; 7) YEAR_AP, year the judge was appointed.

Model and Estimation For each court, we model the marginal probability that case ii would receive asylum under judge jj,

Pr⁡{Yi(j)=1∣Vi,Wj}=F{G(Vi,Wj,j)},\Pr\{Y_i(j) = 1 \mid\mathbf{V}_i, \mathbf{W}_j\} = F\{G(\mathbf{V}_i, \mathbf{W}_j, j)\},

where GG is an unknown prediction function and FF is a link function such as the probit, the standard normal cumulative distribution function.

We estimate the marginal probabilities using a variant of Bayesian Additive Regression Trees (Chipman, George, and McCulloch, 2010), hereafter BART, in which the prediction function is represented as a sum of mm regression trees:

G(Vi,Wj,j)=∑t=1mg(Vi,Wj,j;Tt,Mt),G(\mathbf{V}_i, \mathbf{W}_j, j) = \sum_{t=1}^{m} g(\mathbf{V}_i, \mathbf{W}_j, j; T_t, M_t),

where TtT_t is a binary regression tree that partitions the covariate space and MtM_t is the set of terminal-node parameters associated with that tree. We anticipate the grant probability varies smoothly over time and use soft trees as described in Linero and Yang [16], setting m=200m = 200 trees.

Regularization is achieved by specifying a prior distribution on (Tt,Mt)(T_t, M_t), as described in Linero [15] (see also [4], which limits the influence of any individual tree. Specifically, we shrink the node-parameters to 0 on the covariate scale, which shrinks the overall predictions to 0.5 on the probability scale, which errs on the side of uncertainty over whether judges agree or disagree.

We note that nonparametric approaches to causal inference are increasingly used because they can flexibly capture nonlinearities and interactions without requiring these features to be specified in advance [12]. Thus, we estimate and report the results from a fully nonparametric probit BART model with the R package SoftBart using covariates Vi\mathbf{V}_i and Wj\mathbf{W}_j, and judge indicators jj as predictors. The SoftBart implementation mitigates some of the biases that can result in large samples [1] (Li, Ding, and Mealli 2023) and posterior consistency is achieved under fairly general conditions [16]. Nevertheless, we also compare our results with estimates obtained from several other Bayesian models and find similar results. The first is a parametric logistic regression estimated with brms. It includes Vi\mathbf{V}_i, Wj\mathbf{W}_j, and random intercepts for nationality, language, and judge. The second is a semiparametric stan4bart model. In this specification, BART components are used for the covariates Vi\mathbf{V}_i and Wj\mathbf{W}_j, while judge-specific heterogeneity is represented by a judge random effect. We also compare these results with another fully nonparametric implementation of BART using the R package BART [22].

For each model, we draw 7,000 samples from the posterior distribution of the estimands using Markov chain Monte Carlo. We discard the first 2,000 warm-up iterations, retaining 5,000 draws for inference. The models produce similar posterior means, while the fully nonparametric model yields wider posterior distributions: we report the results from this model. See Section C of the Supplement for additional model and estimation details, and Section D for presentation of the results using the other models.

Estimation of Disparities and Agreement For each of the 5,000 draws above, we estimate the marginal grant probability for the assigned judge and an alternate, chosen with equal probability from the set of eligible alternates, defined to be the set of judges that held a master calendar hearing on the same day as case ii. We then plug these into the formulae from § 2, producing estimates of disparity and agreement: posterior summaries and uncertainty intervals are calculated directly from these. Note that an alternate judge is chosen once for each case used in all posterior samples.

As discussed in § 2, agreement, equation (12), is only partially identified, known only up to the covariance “within” Vi\mathbf{V}_i. We replace this term with the “between” covariance times a sensitivity parameter α\alpha as described in § 2.2. If a particular value of α\alpha under consideration produces a larger value of agreement for a case than that implied by the upper FH bound, we replace this larger value with the upper FH bound.

Results Figure 1 displays the posterior distribution of the measures D1(j)D_1(j) and D3(j)D_3(j) for the top 20 (of 119) judges, by caseload, in the New York City court NYC (upper panel) and the top 20 judges (of 98) in the Los Angeles court WLA. The difference between judge jj’s grant rate and the overall grant rate in the court she serves, Equation (2), is hereafter called “naïve disparity” and indicated by the black tick mark. Naïve disparity is used by RN to infer judicial bias, as previously noted.

Figure 1. Judge disparities D1(j)D_1(j) and D3(j)D_3(j) in New York City (NYC, top) and West Los Angeles (WLA, bottom).

In NYC, three judges (AAV, BAN, and SH) grant asylum at least 22% less often than the court-wide average, and four judges (GTC, NB, PMM, and TAB) grant asylum at least 22% more often; these judges also have the largest absolute disparities D3(j)D_3(j). Of these, only BAN appears less disparate after adjusting for case heterogeneity; for the other five, ∣D1(j)∣\lvert D_1(j)\rvert is almost as large or as large as D3(j)D_3(j), indicating AAV and SH are almost never more likely to grant asylum than their peers, while PLM, PMM, and TAB are almost never less likely. For the others judges, only for JBC, with naïve disparity -5.5%, does the 99% posterior credible interval of D1(j)D_1(j) cover 0. The remaining judges are clustered to the right of 0, mostly with mean disparities D1(j)D_1(j) about 3.8% and D3(j)D_3(j) about 18%.

In WLA, the story is quite different. Whereas the grant rate of 55.8% in NYC implies the naïve judge disparities are upper bounded at this value, in WLA, a judge could deviate from the court-wide rate of 8.8% by more than 90%. But the largest naïve disparity is less than 10%, and even the posterior means of D3(j)D_3(j) are less than 10% for all judges.

Figure 2 displays the posterior distribution of the absolute court-wide disparities D3(⋅)D_3(\cdot) for the 16 largest courts. In NYC, with highest grant rate, the posterior mean of 21% is nearly four times as large as the mean in WLA of 5.8%. But that does not imply a one to

Overall disparities $D_3(\cdot)$ by court

Figure 2. Overall disparities D3(⋅)D_3(\cdot) by court

one relationship between the court-wide rate and D3(⋅)D_3(\cdot); for example, in Baltimore (BAL), the posterior mean of 4.8% is smaller than that in WLA, but the grant rate of 30.8% is higher.

We now turn to agreement. Figure 3 displays the posterior distribution of A1(j)A_1^{(j)} for the top 20 judges in NYC (top panel) and WLA (bottom panel) under various scenarios. In NYC, the FH lower bound ranges from about 33% (JBC) to more than 50% (PLM). However, for countermonotonicity to hold, a judge must weigh the information in UiU_i differently from her peers. We believe it is more reasonable to think judges evaluate such information similarly, as per assumption (A6). If all relevant information is recorded, assumption (A5) reduces to Yi(j)⊥ ⁣ ⁣⊥Yi(ja)∣Vi=viY_i(j) \perp\!\!\perp Y_i(j^a) \mid V_i = \mathbf{v}_i, corresponding to independence. If the unrecorded information UiU_i is of equal predictive value (net of the recorded information Vi\mathbf{V}_i),

Figure 3. Judge agreement A1=(j)A^{=}_{1}(j) in New York City (NYC, top) and West Los Angeles (WLA, bottom).

E[Cov⁡(E[Yi(j)∣Ui,Vi],E[Yi(ja)∣Ui,Vi]∣Vi)]=Cov⁡(E[Yi(j)∣Vi],E[Yi(ja)∣Vi]),\mathbb{E}\left[\operatorname{Cov}\left(\mathbb{E}[Y_i(j) \mid U_i,V_i],\mathbb{E}[Y_i(j^a) \mid U_i,V_i] \mid V_i\right)\right]=\operatorname{Cov}\left(\mathbb{E}[Y_i(j) \mid V_i],\mathbb{E}[Y_i(j^a) \mid V_i]\right),

corresponding to α=1\alpha= 1. Increasing α\alpha further pushes agreement toward comonotonicity, and with α=3\alpha= 3, more than half of the judges achieve maximal agreement, with posterior means varying from 61% for SH to 85.8% for VGU.

In WLA, agreement is considerably higher. Under independence, average posterior agreement exceeds 80%, and agreement between RAR and her peers exceeds 90%. Even under countermonotonicity, agreement always exceeds 70%, and the difference between lower and upper bound agreement never exceeds 20%. With such high agreement under independence, there is not much additional relevant information in the unobserved covariates; even with α=3\alpha= 3, at best agreement over independence increases by about 9% for a few judges.

Figure 4 displays court-wide rates of agreement for the top 16 courts. The black tick mark represents agreement under “naïve independence” gc2+(1−gc)2g_c^2 + (1 - g_c)^2 where gcg_c is the court-wide grant rate. Courts with the lowest rates (CHA–0.00%, ATL–1.72%, OMA–2.91, PHO–5.28, ORL–7.72%, MIA–8.68%) tend to have the most agreement under all scenarios, with lower bounds in excess of 80%, and agreement under independence in excess of 87%, with little room for unobserved covariates to push agreement much higher. In courts with intermediate rates, e.g., WAS, with overall grant rate of 33.7%, and BAL with grant rate 30.8%, adjusting for case heterogeneity and judge heterogeneity matters more. In WAS, under “naïve independence” agreement would be 55.3%. But even if countermonotonicity were to hold, the posterior mean would exceed 69%, and the mean under independence (78%) is almost 42% higher than under “naïve independence”. And if α=1\alpha= 1, agreement virtually reaches its upper bound of 95%. But in Chicago, with grant rate 29.9%, only slightly less than WAS, the posterior mean lower bound of 43% is roughly 25 percentage points less than in WAS, and under independence, the posterior mean of 63% is only slightly greater than in NYC, and 15 percentage points less than in WAS. Further, unlike WAS, choosing α=1\alpha= 1 only increases agreement by a few (4.8) percentage points relative to independence.

Overall agreement by court

Figure 4. Overall agreement A1(⋅)A_1(\cdot) by court

Finally, we compute the average posterior agreement among judges in the top 16 courts (where 90% of cases were filed), weighted by the number of cases in each court. The FH lower bound is 62.7%, the FH upper bound, 88.4%. In between are “naïve independence”

(67.5%), independence (73%), α=1\alpha= 1 (78.2%), α=2\alpha= 2 (82.7%), and α=3\alpha= 3 (85.9%). Taking α=1\alpha= 1, which seems more than reasonable, as the recorded covariates include no information about the individual merits of the case, the agreement of 78.2% is about 16% higher than under “naïve independence.”

To provide another perspective on the agreement of 78.2%, consider the following thought experiment: 8 judges are eligible to hear the case, 7 of whom would grant (or not) asylum were they assigned. If two judges are selected uniformly at random (without replacement), the probability they would agree is 7/8×6/7=.757/8 \times6/7 = .75, roughly the agreement we find in our analysis with α=1\alpha= 1. In fact, even under independence (α=0\alpha= 0), we have 73.3% agreement, consistent with 6 out of 7 judges granting (or not granting) asylum. Even in NYC, agreement under independence (α=0\alpha= 0) is approximately 60%, consistent with 4 of 5.

Discussion

Public institutions are often evaluated using a community standard in which the performance of each agent is compared with that of peers. Naive comparisons may fail to account for the varied situations different agents face in the performance of their duties. We define estimands of disparity and agreement that account for this variability and show how to estimate these, at both the agent and institutional levels, when only one agent only is assigned to each case.

We apply our methodology to the U.S. asylum courts, revisiting earlier work that claimed the outcome of an asylum application depends more on the judge assigned to the case than on case merits. In the New York court, where a third of the cases are handled and grant rates are close to 50%, the disparities among the 20 judges who handled the most cases are larger than the other courts and agreement is lower. Yet disparity among these 20 judges is limited to six outlier judges with D3(j)D_3(j) greater than .3, and wide-spread disagreement does not characterize the court as a whole. In the other courts, where the grant rates are lower, there is less disparity between judges and high agreement, even though the potential for disparity is greater. Our findings suggest that the blanket condemnation of the EOIR judiciary as partial and unprofessional in previous work is largely unsubstantiated.

To ensure comparability with previous work on the asylum courts, particularly with GAO, we analyzed cases filed between 2007 and 2014. Yet our analysis remains relevant today. First, we challenge the interpretation of statistical evidence that continues to serve as the basis for a growing academic literature, and we provide a methodology for correcting the record using tools from causal inference. Second, the asylum system has undergone considerable stress in recent years due to both external pressures and domestic policies. For example, under Title 42, enacted during the COVID-19-era public health emergency, the United States expelled migrants who crossed the border illegally. Policymakers and advocates are now proposing reforms that consider in part how the institution functioned in the past. Mischaracterizing past performance may lead to misguided reforms.

It is also important to update our analysis and examine the more recent workings of the courts. One challenge will be that, due to the dramatic increase in filings following 2014, the average wait times for a hearing now exceeds four years, and many cases are unresolved after six or more years. Thus, any timely analysis must also account for right censoring.

Comparisons using community standards can also be used to address several issues of current concern that speak to the integrity of the courts, for example, whether judges appointed during the Trump administrations behave differently than judges appointed under the Obama or Biden administrations. Kim and Semet [13] conducted an early investigation of this question, albeit with limitations similar to those discussed in § 4. More generally, in any institutional setting, to assess whether agents in different subpopulations, e.g., male and female, behave differently, we can use the approach herein, by choosing a female (male) alternate jaj^{a} when jj is male (female) (Sobel et al., [21]).

Our methodology may also be useful in other settings where a community standard is used to compare agents. For example, large private sector firms often collect ratings from customers on their satisfaction with the agents with whom they interacted. These may then be used to compare agents with one another. In the public sector, teachers are evaluated using value added models and police officers by the number of stops or arrests made. In some instances, two or more agents may jointly handle an interaction with the public, while in others, multiple cases may be simultaneously handled by one agent. In future work, we shall address these cases to estimate the disparities and agreement defined herein.

We also intend to consider new estimands. Building on the representation of agreement in (14), we will develop new agreement measures applicable to both the discrete and continuous cases with 2 or more agents, and take up their estimation, both when all relevant agents are observed and when they are not. Similarly, we shall develop disparity measures for comparing more than two agents. We shall also extend our analyses to the case of multiple and multidimensional outcomes. Our analyses excluded mediators, most notably representation: in future work, we intend to extend the framework to handle mediating variables.

Evaluations using a community standard are becoming more common and more consequential as public institutions become increasingly accountable to the public or politicized (depending on one’s perspective). Statisticians have an important role to play in developing methodologies that faithfully portray the integrity of these institutions and the performance of their agents; good measurement protects the integrity of our public institutions, while mismeasurement undermines them, perhaps more than if there were no measurement at all.

Supplemental Information

Data Description

The Executive Office for Immigration Review (EOIR), a component of the U.S. Department of Justice, maintains administrative case data for immigration court proceedings. EOIR adjudicates immigration cases under authority delegated by the Attorney General, and immigration judges appointed by the Attorney General preside over immigration court hearings. EOIR provides raw case-level data from its case-file electronic database on its website. The results presented here use data accessed on June 1, 2026.

The database is relational with three main tables: the case table, the proceeding table, and the schedule table. The case table contains case-level information such as the nationality of the asylum seeker. The proceeding table records the proceeding or proceedings associated with each case. The schedule table records scheduled hearing sessions associated with proceedings—a proceeding can have multiple scheduled sessions.

A separate court application table records applications filed in immigration court, including asylum applications and their outcomes. Using this table, we identify affirmative asylum applications filed between May 1, 2007 and September 30, 2014. We restrict attention to cases classified as affirmative asylum cases (corresponding to cases in which EOIR starts the “clock”, i.e., C_ASY_TYPE = I), and we retain only the first asylum application associated with the lead case (the primary applicant in cases where multiple family members apply together). Later asylum applications or repeat attempts are excluded. Rider cases are not considered, although the number of riders attached to the lead is retained as a covariate.

For each retained lead case, we identify the first proceeding, the respondent’s first appearance before an immigration judge. We use the immigration judge recorded for that first proceeding as the initially assigned judge. The outcome is the eventual decision on the first asylum application, coded as grant or not grant. Specifically, we consider the outcomes GRANT, FULL GRANT and GRANT WCAT to be a “grant”, and outcomes ABANDONMENT, DENY, NOT ADJUDICATED, WITHDRAWN, ADMIN CLOSURE, COV/TRANSFER, and OTHER to be a “not grant.”

Model Sensitivity

We examine the sensitivity of the findings presented in Section 4.4 to the model and estimation strategy described in Section 4.3. We first discuss the challenges of model misspecification and regularization bias. We then consider three alternative models and show that the estimates from Section 4.4 are largely insensitive to the choice of model. As part of our model sensitivity analysis, we remake Figure 1–Figure 4 exactly as described in Section 4.4, except we substitute each alternative model for the estimation of the marginal grant rate. These figures are provided in section D of the Supplement.

Model Misspecification and Regularization Bias

In Section 4.4, we describe how the marginal probabilities were estimated nonparametrically using Bayesian Additive Regression Trees (BART, Chipman, George, and McCulloch (2010)). An advantage of BART is that it avoids model misspecification bias that can arise in the case of parametric models, where the true model may not be contained within the class of candidate models. Misspecification is particularly problematic when there is little overlap between cases, as discussed by Hill [12].

The disadvantage of nonparametric methods like BART is the potential for regularization bias ([1] (Li, Ding, and Mealli 2023)), arising from the deliberate restriction of the model class to avoid overfitting. Both model misspecification and regularization bias are difficult to assess directly. One solution is to fit a variety of models to see how sensitive the findings are to the choice of model. We take that approach here. We reanalyze the data using the same exact methodology as described in the main text, except we substitute a logistic regression model and a partially general linear model. We refer to these as simply the “semiparametric model” and “parametric model.” We also compare to another implementation of BART, which we refer to as the “nonparametric model.”

Model Choice

We consider three alternative models for the probability case ii would receive asylum under judge jj. The first is the alternative semiparametric model

Pr⁡{Yi(j)=1∣Vi,Wj}=F{G(Vi)+WjTγ+bj},\Pr\{Y_i(j) = 1 \mid\mathbf{V}_i, \mathbf{W}_j\} = F\{G(\mathbf{V}_i) + \mathbf{W}_j^\mathsf{T}\gamma+ b_j\},

where GG is an unknown prediction function and FF is the known link function. The vectors Vib\mathbf{V}_i^b and Wj\mathbf{W}_j are the case and judge covariates discussed in the main text, and bjb_j is a random effect for each judge.

We estimate the model using the R package stan4bart. The package samples from the posterior distribution of the function GG using Bayesian Additive Regression Trees as implemented in the R package bart and the remaining parameters using the R package rstanarm. We use the default link function (probit), number of trees (m=200m = 200), and priors (see Gelman, Hill, and Vehtari (2020) for details).

The second is the alternative parametric model (logistic regression),

Pr⁡{Yi(j)=1∣Vi,Wj}=F{β0+ViTβ+WjTγ+bj+an[i]+cℓ[i]}\Pr\{Y_i(j) = 1 \mid\mathbf{V}_i, \mathbf{W}_j\} = F\{\beta_0 + \mathbf{V}_i^\mathsf{T}\beta+ \mathbf{W}_j^\mathsf{T}\gamma+ b_j + a_{n[i]} + c_{\ell[i]}\}

where FF is the known link function, β0\beta_0 is an intercept, and bjb_j, n[i]n[i] and ℓ[i]\ell[i] are random effects for judge, nationality, and language, respectively. We sample from the posterior distribution of the model parameters using the R package brms and default priors, see Gelman, Hill, and Vehtari (2020) for details.

We also compare these models to the alternative nonparametric model described in the main text, fit using the R package BART and the default logit link function. For all models, we draw 7,000 samples from the posterior distribution. We discard the first 2,000 as warm-up, yielding 5,000 draws for inference as described in the main text. To ensure that the posterior draws are not sensitive to the choice of seed, we check for posterior convergence. Specifically, we run each model an additional four times (i.e. four chains with different seeds) and observe that the chains have mixed as indicated by a R-hat close to 1 and provide the same estimates as the original draws. Since the estimates are the same, we do not use these additional draws.

Additional Tables

Court LocationNumber of CasesNumber of JudgesGrant Rate (%)Number of Cases with AlternativesNumber of Judges with AlternativesNaïve Agreement (%)
NYC3907211955.83887311950.7
WLA24546988.8240299883.9
SFR58567726.055077661.5
MIA5347528.749345284.2
WAS45545333.734105355.3
ORL3667207.727522085.7
BOS29264014.423523875.3
NEW27983613.421883676.8
CHI25653729.920403758.1
BAL22793130.812272857.4
ATL2206261.718382596.6
PHO1912255.310932490.0
CHA132960.1163699.8
PHI12681713.26371777.0
OMA1167112.9802994.3
SEA8961523.83731563.8

Table 1. Case counts, judge counts, grant rates, and agreement under naïve independence (naïve agreement) by location

Additional Figures

Figure 5. Judge disparities D1(j)D_1(j) and D3(j)D_3(j) in New York City using the alternative nonparametric model (top), semiparametric model (middle), and parametric model (bottom).

Figure 6. Judge disparities D1(j)D_1(j) and D3(j)D_3(j) in West Los Angeles using the alternative nonparametric model (top), semiparametric model (middle), and parametric model (bottom).

Figure 7. Overall disparities D3(⋅)D_3(\cdot) by court using the nonparametric model (top), semiparametric model (middle), and parametric model (bottom).

Figure 8. Judge agreement Aˉ1(j)\bar{A}_{1}^{(j)} in New York City using the alternative nonparametric model (top), semiparametric model (middle), and parametric model (bottom).

Figure 9. Judge agreement A1(j)A_1^{(j)} in West Los Angeles using the alternative nonparametric model (top), semiparametric model (middle), and parametric model (bottom).

Figure 10. Overall agreement A1(⋅)A_1(\cdot) by court using the alternative nonparametric model (top), semiparametric model (middle), and parametric model (bottom).

References

References

  1. [1]Antonelli, J. (2023). “Bayesian propensity score methods and related approaches for confounding adjustment”. Handbook of Matching and Weighting Adjustments for Causal Inference. Ed. by J. R. Zubizarretta, E. A. Stuart, D. S. Small, and P. R. Rosenbaum. Chapman and Hall, pp. 501–528.DOI
  2. [2]Becker, M. P. (1989). “Using association models to analyse agreement data: two examples”. Statistics in Medicine 8.10, pp. 1199–1207.pubmed.ncbi.nlm.nih.gov/2814069
  3. [3]Bishop, Y. M., Feinberg, S. E., and Holland, P. W. (1975). Discrete multivariate statistics: Theory and practice. MIT Press, Cambridge, Massachusetts.
  4. [4]Chipman, H. A., George, E. I., and McCulloch, R. E. (1998). “Bayesian CART model search”. Journal of the American Statistical Association 93.443, pp. 935–948.DOI
  5. [6]Cinelli, C. and Hazlett, C. (2020). “Making sense of sensitivity: extending omitted variable bias”. Journal of the Royal Statistical Society, Series B 82, pp. 39–67.
  6. [7]Cook, B. L., McGuire, T. G., and Zaslavsky, A. M. (2012). “Measuring racial/ethnic disparities in health care: methods and practical issues”. Health services research 47.3pt2, pp. 1232–1254.DOI
  7. [8]Department of Justice (2022). Department of justice policy on use of force (JM 1-16.000).
  8. [9]Duan, N., Meng, X.-L., Lin, J. Y., Chen, C.-n., and Alegria, M. (2008). “Disparities in defining disparities: statistical conceptual frameworks”. Statistics in Medicine 27.20, pp. 3941–3956.DOI
  9. [11]Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC.
  10. [12]Hill, J. (2011). “Bayesian nonparametric modeling for causal inference”. Journal of Computational and Graphical Statistics 20.1, pp. 217–240.DOI
  11. [13]Kim, C. Y. and Semet, A. (2020). “An empirical study of political control over immigration adjudication”. The Georgetown Law Journal 108, p. 579.
  12. [15]Linero, A. R. (2022). “Softbart: Soft Bayesian additive regression trees”. arXiv preprint arXiv:210.16375.arxiv.org/abs/2210.16375
  13. [16]Linero, A. R. and Yang, Y. (2018). “Bayesian regression tree ensembles that adapt to smoothness and sparsity”. Journal of the Royal Statistical Society Series B: Statistical Methodology 80.5, pp. 1087–1110.arxiv.org/abs/1707.09461
  14. [17]Miller, B., Keith, L. C., and Holmes, J. S. (2014). Immigration Judges and U.S. Asylum Policy. University of Pennsylvania Press, Philadelphia PA.DOI
  15. [18]Mossaad, N. (2016). “Refugees and Asylees: 2014”. Department of Homeland Security, Office of Immigration Statistics.
  16. [19]Ramji-Nogales, J., Schoenholtz, A. I., and Schrag, P. G. (2007). “Refugee Roulette: Disparities in Asylum Adjudication”. Stanford Law Review 60: 295-411 60.2, pp. 295–411.
  17. [20]Rosenberg, M., Levinson, R., and McNeill, R. (Oct. 2017). They fled danger at home to make a high-stakes bet on U.S. immigration courts. Reuters.
  18. [21]Sobel, M. E., Wawro, G. J., and Farhang, S. (2023). “Association and causation: Attributes and effects of judges in equal employment opportunity commission litigation outcomes”. The Annals of Applied Statistics 17.4, pp. 3526–3549.DOI
  19. [22]Sparapani, R., Spanbauer, C., and McCulloch, R. E. (2021). “Nonparametric machine learning and efficient computation with Bayesian additive regression trees: The BART R package”. Journal of Statistical Software 97.1, pp. 1–66.DOI
  20. [27]Van der Vaart, A. W. (2000). Asymptotic statistics. Vol. 3. Cambridge university press.

Paper details

vlnc:W_t75e4gcr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents