betaFeedback

WorksMathematics

Studentized inference in randomized experiments when a few units dominate the outcome variation

Abstract

In completely randomized experiments with heavy-tailed outcomes, a few units can carry a share of the residual variation that does not vanish as the number of units grows. In this setting, we study the studentized difference in means and Lin's regression-adjusted estimator under a design-based framework. The studentized statistic has a nonnormal limit in general. Populations with the same marginal distributions of potential outcomes can have different limits, as the limit depends on how the two potential outcomes of each dominant unit are paired. When the outcome errors are drawn from a heavy-tailed law, the limit is random and, outside a degenerate case, varies with the population. Under balanced allocation, the conventional Wald interval has an explicit asymptotic coverage floor. At the nominal level of 0.950.95, the floor is about 0.420.42 for every population and about 0.840.84 when the two potential residuals of each unit are equal. Both floors are close to the best possible, and without balanced allocation, the coverage can fall to the smaller of the proportions of treated and control units.

Introduction

Randomized experiments are commonly analyzed with the difference in means or with a regression-adjusted estimator, together with a Wald interval based on a Neyman-type variance estimator. In the design-based framework, the covariates and potential outcomes are fixed, the treatment assignment is the only source of randomness, and the interval is justified by a finite-population central limit theorem (Lin [25]; Li and Ding (2017)). Such theorems require that no unit carry a share of the outcome variation that stays bounded away from zero as the number of units grows, a condition known as negligibility (Noether [27]; Erdős and Rényi [13]; Hájek [18]). In practice, however, outcomes such as revenue, spending, and charges are heavy-tailed, and a few units can carry much of the variation. As Young [33] noted, in published experiments, deleting a single observation can make about a third of the results that are significant at the level 0.01 insignificant. For example, in the RAND Health Insurance Experiment ([2]) studied in § 6, one person carries about a third of the residual variation of second-year medical spending.

Without negligibility, sums over a randomly selected subset of a finite population can have nonnormal limits with discrete components ([29, 22, 21, 8]), and permuted sums of heavy-tailed samples have limits that depend on the sample ([6]). For independent observations without finite variance, the limits of self-normalized sums are known ([26]). Student’s tt statistic is asymptotically normal if and only if the law of the observations is in the domain of attraction of the normal law ([17]), and under symmetry, it has explicit tail bounds ([12, 11, 28]). The Berry–Esseen bound of Bloznelis [9] for the finite-population Student statistic implies asymptotic normality under the condition of Erdős and Rényi [13], and the theory of studentized permutation and randomization tests ([20, 10] (Wu and Ding 2021)) establishes validity when the studentized statistic is asymptotically normal. Methods designed for heavy-tailed outcomes change either the estimator or the estimand ([4, 30, 16]). Concentration inequalities give intervals with finite-sample coverage ([15]), but they require known bounds on the potential outcomes. None of the above results gives the randomization law of the conventional studentized statistic when a few units dominate the residual variation. The limits for random subsets concern sums of one value per unit, whereas the studentized statistic is a ratio whose numerator and denominator depend on both potential outcomes of the dominant units.

This article studies the studentized difference in means and the interacted estimator of Lin [25], with covariates whose leverage vanishes, when the residual variation is not negligible. The studentized statistic has a limit law determined by the limits of the normalized residual pairs of the dominant units and of the second-moment matrix of the remaining units, provided the limit of the normalized variance estimator is positive with probability one (Theorem 3.2). The law is nonnormal in general. It depends on how the two potential residuals of each dominant unit are paired, not only on their marginal distributions, so populations with the same marginal distributions of potential outcomes can have different limiting coverage (Proposition 3.5). When the outcome errors are drawn from a law with tail index between one and two, the randomization law of the statistic for the population at hand converges to a random law determined by the largest residual pairs of the realized population, and this law varies with the population unless the spectral measure of the errors is concentrated on a line determined by the allocation (Proposition 3.4). Under balanced allocation, the Wald interval has an explicit asymptotic coverage floor for every population, about 0.42 at the nominal level of 0.95 (Proposition 4.1), and a higher floor, about 0.84, when the two potential residuals of each unit are equal (Theorem 4.2). Both floors are close to the best possible, and neither holds at the nominal level of 0.95 when 40% of the units are treated. Without balanced allocation, the coverage can fall to the smaller of the proportions of treated and control units (Proposition 4.3). The floors are randomization analogs of the tail bounds for Student’s tt statistic under symmetry, with the balance of the design in place of symmetric errors.

Simulations with heavy-tailed errors (§ 5) and a data application (§ 6) show how closely the limits describe populations of moderate size and how the coverage of the Wald interval behaves when a few units dominate.

Setup

Finite populations and the design

For each N≥2N \ge2, let ΠN={(Xi,Yi(1),Yi(0)):i≤N}\Pi_N = \{(X_i, Y_i(1), Y_i(0)) : i \le N\} be a finite population of covariates Xi∈RdX_i \in\mathbb{R}^d and potential outcomes Yi(1),Yi(0)∈RY_i(1), Y_i(0) \in\mathbb{R}. These quantities are fixed but may change with NN. We suppress their dependence on NN in the notation. The dimension d≥0d \ge0 is fixed, and d=0d = 0 means that there are no covariates. The estimand is the sample average treatment effect,

τN=1N∑i=1N{Yi(1)−Yi(0)}.\tau_N = \frac{1}{N} \sum_{i=1}^{N} \{Y_i(1) - Y_i(0)\}.

Assumption 1 (Complete randomization). For each NN, 1≤N1≤N−11 \le N_1 \le N - 1, and the vector of treatment indicators (Z1,…,ZN)(Z_1, \ldots, Z_N) is uniformly distributed over all binary vectors of length NN with N1N_1 entries equal to one. Also, N0=N−N1N_0 = N - N_1, and πN=N1/N→π∈(0,1)\pi_N = N_1/N \to\pi\in(0,1).

Projections and leverage

Let Y‾(z)=N−1∑i=1NYi(z)\overline{Y}(z)=N^{-1}\sum_{i=1}^{N}Y_i(z) for z∈{0,1}z\in\{0,1\}. If d≥1d\ge1, let X‾=N−1∑i=1NXi\overline{X}=N^{-1}\sum_{i=1}^{N}X_i and G=∑i(Xi−X‾)(Xi−X‾)⊤G=\sum_i(X_i-\overline{X})(X_i-\overline{X})^{\top}, which we assume to be nonsingular for every NN, and define the finite-population projection coefficients and residuals by

βz=G−1∑i=1N(Xi−X‾)Yi(z),ei(z)=Yi(z)−Y‾(z)−(Xi−X‾)⊤βz.(1)\beta_z=G^{-1}\sum_{i=1}^{N}(X_i-\overline{X})Y_i(z),\qquad e_i(z)=Y_i(z)-\overline{Y}(z)-(X_i-\overline{X})^{\top}\beta_z. \tag*{(1)}

If d=0d=0, let ei(z)=Yi(z)−Y‾(z)e_i(z)=Y_i(z)-\overline{Y}(z). No outcome model is assumed. By construction,

∑i=1Nei(z)=0,∑i=1N(Xi−X‾)ei(z)=0,(2)\sum_{i=1}^{N}e_i(z)=0,\qquad\sum_{i=1}^{N}(X_i-\overline{X})e_i(z)=0, \tag*{(2)}

where the second identity applies if d≥1d\ge1. The leverage of unit ii is hi=(Xi−X‾)⊤G−1(Xi−X‾)h_i=(X_i-\overline{X})^{\top}G^{-1}(X_i-\overline{X}) if d≥1d\ge1, and hi=0h_i=0 if d=0d=0. Then, 0≤hi≤10\le h_i\le1 and ∑ihi=d\sum_i h_i=d. Write ei=(ei(1),ei(0))⊤e_i=(e_i(1),e_i(0))^{\top} for the residual pair of unit ii, and let

rN2=∑i=1N∥ei∥2,ci=ei(1)πN+ei(0)1−πN.(3)r_N^2=\sum_{i=1}^{N}\lVert e_i\rVert^2,\qquad c_i=\frac{e_i(1)}{\pi_N}+\frac{e_i(0)}{1-\pi_N}. \tag*{(3)}

The quantity rN2r_N^2 is the total residual variation, and cic_i combines the two potential residuals of unit ii.

The estimator and the Wald interval

For z∈{0,1}z\in\{0,1\}, let X‾z\overline{X}_z and Y‾z\overline{Y}_z denote the averages of the covariates and of the observed outcomes Yi=ZiYi(1)+(1−Zi)Yi(0)Y_i=Z_iY_i(1)+(1-Z_i)Y_i(0) over {i:Zi=z}\{i:Z_i=z\}. If d≥1d\ge1, define the within-arm Gram matrix and slope by

Gz=∑i:Zi=z(Xi−X‾z)(Xi−X‾z)⊤,β^z=Gz−1∑i:Zi=z(Xi−X‾z)Yi,(4)G_z=\sum_{i:Z_i=z}(X_i-\overline{X}_z)(X_i-\overline{X}_z)^{\top},\qquad\widehat{\beta}_z=G_z^{-1}\sum_{i:Z_i=z}(X_i-\overline{X}_z)Y_i, \tag*{(4)}

with β^z=0\widehat{\beta}_z=0 if GzG_z is singular. The estimator is

τ^=Y‾1−Y‾0−(X‾1−X‾)⊤β^1+(X‾0−X‾)⊤β^0(5)\widehat{\tau}=\overline{Y}_1-\overline{Y}_0-(\overline{X}_1-\overline{X})^{\top}\widehat{\beta}_1+(\overline{X}_0-\overline{X})^{\top}\widehat{\beta}_0 \tag*{(5)}

if d≥1d\ge1, and the difference in means τ^=Y‾1−Y‾0\widehat{\tau}=\overline{Y}_1-\overline{Y}_0 if d=0d=0. If G1G_1 and G0G_0 are nonsingular, τ^\widehat{\tau} is the coefficient of ZiZ_i in the least squares regression of YiY_i on (1,Zi,Xi−X‾,Zi(Xi−X‾))(1,Z_i,X_i-\overline{X},Z_i(X_i-\overline{X})), the interacted estimator of Lin [25]. Suppose N1,N0≥2N_1, N_0 \ge2. For Zi=zZ_i=z, let e^i=Yi−Y‾z−(Xi−X‾z)⊤β^z\widehat{e}_i=Y_i-\overline{Y}_z-(X_i-\overline{X}_z)^\top\widehat{\beta}_z if d≥1d\ge1 and e^i=Yi−Y‾z\widehat{e}_i=Y_i-\overline{Y}_z if d=0d=0. Define

V^=s12N1+s02N0,sz2=1Nz−1∑i:Zi=ze^i2,TN=τ^−τNV^1/2.(6)\widehat{V}=\frac{s_1^2}{N_1}+\frac{s_0^2}{N_0}, \qquad s_z^2=\frac{1}{N_z-1}\sum_{i:Z_i=z}\widehat{e}_i^2, \qquad T_N=\frac{\widehat{\tau}-\tau_N}{\widehat{V}^{1/2}}. \tag*{(6)}

If d=0d=0, V^\widehat{V} is the Neyman variance estimator. If V^=0\widehat{V}=0, we set TN=0T_N=0 when τ^=τN\widehat{\tau}=\tau_N, and TN=+∞T_N=+\infty or TN=−∞T_N=-\infty according to the sign of τ^−τN\widehat{\tau}-\tau_N otherwise, so that {∣TN∣≤t}\{\lvert T_N\rvert\le t\} is the event that τ^±tV^1/2\widehat{\tau}\pm t\widehat{V}^{1/2} covers τN\tau_N, for t>0t>0. The Wald interval of nominal level 1−ς1-\varsigma is τ^±Φ−1(1−ς/2)V^1/2\widehat{\tau}\pm\Phi^{-1}(1-\varsigma/2)\widehat{V}^{1/2}, where Φ\Phi is the standard normal distribution function.

Exact decompositions

Define the residual contrast and, if d≥1d\ge1, the remainder from estimating the slopes by

LN=∑i=1N(Zi−πN){ei(1)N1+ei(0)N0},RN=−(X‾1−X‾)⊤(β^1−β1)+(X‾0−X‾)⊤(β^0−β0),(7)L_N=\sum_{i=1}^{N}(Z_i-\pi_N)\left\{\frac{e_i(1)}{N_1}+\frac{e_i(0)}{N_0}\right\}, \qquad R_N=-(\overline{X}_1-\overline{X})^\top(\widehat{\beta}_1-\beta_1)+(\overline{X}_0-\overline{X})^\top(\widehat{\beta}_0-\beta_0), \tag*{(7)}

and let RN=0R_N=0 if d=0d=0. Let e‾z(z)=Nz−1∑i:Zi=zei(z)\overline{e}_z(z)=N_z^{-1}\sum_{i:Z_i=z}e_i(z). If d=0d=0, let CN,z=Nze‾z(z)2C_{N,z}=N_z\overline{e}_z(z)^2. If d≥1d\ge1, let kz=∑i:Zi=z(Xi−X‾z)ei(z)k_z=\sum_{i:Z_i=z}(X_i-\overline{X}_z)e_i(z) and

CN,z=Nze‾z(z)2+kz⊤Gz−1kz(8)C_{N,z}=N_z\overline{e}_z(z)^2+k_z^\top G_z^{-1}k_z \tag*{(8)}

on the event that GzG_z is nonsingular, and CN,z=0C_{N,z}=0 otherwise.

Proposition 2.1. Suppose Assumption 1 holds. For every assignment,

τ^−τN=LN+RN,NLN=∑i=1N(Zi−πN)ci,(9)\widehat{\tau}-\tau_N=L_N+R_N, \qquad NL_N=\sum_{i=1}^{N}(Z_i-\pi_N)c_i, \tag*{(9)}

and E(NLN)=0\mathbb{E}(NL_N)=0 and var⁡(NLN)={N1N0/(N(N−1))}∑ici2\operatorname{var}(NL_N)=\{N_1N_0/(N(N-1))\}\sum_i c_i^2. If N1,N0≥2N_1,N_0\ge2 and either d=0d=0 or G1G_1 and G0G_0 are nonsingular, then CN,1,CN,0≥0C_{N,1},C_{N,0}\ge0 and

N2V^=∑z∈{0,1}N2Nz(Nz−1){∑i:Zi=zei(z)2−CN,z}.(10)N^2\widehat{V}=\sum_{z\in\{0,1\}}\frac{N^2}{N_z(N_z-1)}\left\{\sum_{i:Z_i=z}e_i(z)^2-C_{N,z}\right\}. \tag*{(10)}

Section S2 proves Proposition 2.1. The estimation error splits into the residual contrast LNL_N, which is the difference in means computed from the population residuals, and the remainder RNR_N from estimating the slopes. In each arm, the variance estimator sums the squared population residuals of the observed units, minus CN,zC_{N,z}, the part of that sum explained by regressing these residuals on an intercept and the covariates within arm zz. Both LNL_N and the leading term of (10) depend on the population only through the residual pairs ei\boldsymbol{e}_i.

Condition 1 (Vanishing leverage). The leverages satisfy max⁡ihi→0\max_i h_i \to0.

Condition 1 holds if d=0d=0. If d≥1d\ge1, it holds if N−1max⁡i∥Xi−X‾∥2→0N^{-1}\max_i \lVert X_i-\overline{X}\rVert^2\to0 and lim inf⁡Nλmin⁡(N−1G)>0\liminf_N\lambda_{\min}(N^{-1}G)>0, because hi≤∥Xi−X‾∥2/λmin⁡(G)h_i\le\lVert X_i-\overline{X}\rVert^2/\lambda_{\min}(G). It restricts the covariates only.

Lemma 2.2. Suppose Assumption 1 and Condition 1 hold. Then, if d≥1d\ge1, the probability that G1G_1 or G0G_0 is singular tends to zero. For every d≥0d\ge0, if rN>0r_N>0 for every NN, then NRN/rN→0NR_N/r_N\to0 and CN,z/rN2→0C_{N,z}/r_N^2\to0 in probability, for z∈{0,1}z\in\{0,1\}.

Section S2.3 proves Lemma 2.2 with a bound for each NN. Under Condition 1, estimating the slopes changes the numerator and the denominator of TNT_N only at a smaller order than the residual scale, regardless of the outcomes. The limits below are therefore determined by the residual pairs ei\boldsymbol{e}_i.

The limit of the studentized statistic

Residual configurations

When a few units carry a share of rN2r_N^2 that does not vanish, the numerator N(τ^−τN)N(\widehat{\tau}-\tau_N) and the denominator N2V^N^2\widehat{V} of TNT_N are dominated by the same units. For rN>0r_N>0, let e~i=ei/rN\widetilde{\boldsymbol{e}}_i=\boldsymbol{e}_i/r_N, so that ∑ie~i=0\sum_i\widetilde{\boldsymbol{e}}_i=0 by (2) and ∑i∥e~i∥2=1\sum_i\lVert\widetilde{\boldsymbol{e}}_i\rVert^2=1.

Definition 3.1. Let q≥1q\ge1, and for N≥2N\ge2 let vN,1,…,vN,N\boldsymbol{v}_{N,1},\ldots,\boldsymbol{v}_{N,N} be vectors in Rq\mathbb{R}^q such that sup⁡N∑i∥vN,i∥2\sup_N\sum_i\lVert\boldsymbol{v}_{N,i}\rVert^2 is finite. Let Ψ=(ψk)k∈K\Psi=(\psi_k)_{k\in\mathcal{K}} be a finite or countably infinite family of nonzero vectors in Rq\mathbb{R}^q such that K(ζ)={k:∥ψk∥>ζ}\mathcal{K}(\zeta)=\{k:\lVert\psi_k\rVert>\zeta\} is finite for every ζ>0\zeta>0, let ΣE\Sigma_E be a positive semidefinite q×qq\times q matrix, and let IN(ζ)={i:∥vN,i∥>ζ}\mathcal{I}_N(\zeta)=\{i:\lVert\boldsymbol{v}_{N,i}\rVert>\zeta\}. The array has limit configuration (Ψ,ΣE)(\Psi,\Sigma_E) if (i) and (ii) hold. (i) For every ζ>0\zeta>0 different from all ∥ψk∥\lVert\psi_k\rVert and all large NN, IN(ζ)\mathcal{I}_N(\zeta) has the same cardinality n(ζ)n(\zeta) as K(ζ)\mathcal{K}(\zeta), and there are enumerations of the two

sets, i1(N),…,in(ζ)(N)i_1(N),\ldots,i_{n(\zeta)}(N) and k1(N),…,kn(ζ)(N)k_1(N),\ldots,k_{n(\zeta)}(N), with max⁡j≤n(ζ)∥vN,ij(N)−ψkj(N)∥→0\max_{j\le n(\zeta)}\lVert v_{N,i_j}(N)-\psi_{k_j}(N)\rVert\to0.

(ii) The limit lim⁡ζ↓0lim sup⁡N→∞∥∑i∉IN(ζ)vN,ivN,iT−ΣE∥=0\lim_{\zeta\downarrow0}\limsup_{N\to\infty}\left\lVert\sum_{i\notin I_N(\zeta)}v_{N,i}v_{N,i}^{\mathsf T}-\Sigma_E\right\rVert=0, the limit in ζ\zeta being taken along values different from all ∥ψk∥\lVert\psi_k\rVert.

Condition 2 (Residual configuration). For every NN, rN>0r_N>0, and the array (e~i)i≤N(\widetilde e_i)_{i\le N} has a limit configuration (Ψ,ΣE)(\Psi,\Sigma_E) with q=2q=2, where ψk=(ψk,1,ψk,0)T\psi_k=(\psi_{k,1},\psi_{k,0})^{\mathsf T}.

The points ψk\psi_k are the limits of the normalized residual pairs of the units that keep a share of rN2r_N^2 that does not vanish, and ΣE\Sigma_E is the limit of the second-moment matrix of the remaining units. If Ψ\Psi is empty, max⁡i∥e~i∥→0\max_i\lVert\widetilde e_i\rVert\to0, which is the negligibility condition of Noether [27] for the residual pairs. Let σE,12\sigma_{E,1}^2 and σE,02\sigma_{E,0}^2 be the diagonal entries of ΣE\Sigma_E corresponding to ei(1)e_i(1) and ei(0)e_i(0), let wπ=(1/π,1/(1−π))T\mathbf w_\pi=(1/\pi,1/(1-\pi))^{\mathsf T}, let (Ak)k∈K(A_k)_{k\in\mathcal K} be independent Bernoulli(π)(\pi) variables, and let gE∼N(0,π(1−π)ΣE)\mathbf g_E\sim N(0,\pi(1-\pi)\Sigma_E) be independent of (Ak)(A_k). Define

UE=∑k(Ak−π)wπTψk+wπTgE,U_E=\sum_k(A_k-\pi)\mathbf w_\pi^{\mathsf T}\psi_k+\mathbf w_\pi^{\mathsf T}\mathbf g_E,
VE=∑k{Akψk,12π2+(1−Ak)ψk,02(1−π)2}+σE,12π+σE,021−π.(11)V_E=\sum_k\left\{\frac{A_k\psi_{k,1}^2}{\pi^2}+\frac{(1-A_k)\psi_{k,0}^2}{(1-\pi)^2}\right\}+\frac{\sigma_{E,1}^2}{\pi}+\frac{\sigma_{E,0}^2}{1-\pi}. \tag*{(11)}

Theorem 3.2. Suppose Assumption 1 and Conditions 1 and 2 hold. Then,

(NrN(τ^−τN),N2rN2V^)⇝(UE,VE).\left(\frac{N}{r_N}(\widehat{\tau}-\tau_N),\frac{N^2}{r_N^2}\widehat V\right)\rightsquigarrow(U_E,V_E).

Also, P(VE>0)=1\mathbb P(V_E>0)=1 if and only if ΣE≠0\Sigma_E\ne0, or Ψ\Psi is infinite, or some ψk\psi_k has both coordinates nonzero. In that case, TN⇝UE/VE1/2T_N\rightsquigarrow U_E/V_E^{1/2}.

Section S4.1 proves Theorem 3.2. If Ψ\Psi is empty, UEU_E is normal with mean zero and variance π(1−π)wπTΣEwπ\pi(1-\pi)\mathbf w_\pi^{\mathsf T}\Sigma_E\mathbf w_\pi, and VEV_E is the constant σE,12/π+σE,02/(1−π)\sigma_{E,1}^2/\pi+\sigma_{E,0}^2/(1-\pi). The constant exceeds the variance of UEU_E by (1,−1)ΣE(1,−1)T≥0(1,-1)\Sigma_E(1,-1)^{\mathsf T}\ge0. Thus, TNT_N has a normal limit with variance at most one, and the Wald interval is asymptotically conservative, as in Lin [25] and Li and Ding (2017). If Ψ\Psi is not empty and P(VE>0)=1\mathbb P(V_E>0)=1, the limit of TNT_N is nonnormal in general, and it depends on Ψ\Psi as a family of points in R2\mathbb R^2, not only on its two coordinate projections. A unit whose treatment residual is large but whose control residual is not enters the numerator through cic_i in both arms, because τN\tau_N contains both of its potential outcomes, but it enters the denominator only if it is treated. If ei(1)=ei(0)e_i(1)=e_i(0) for all ii and π=1/2\pi= 1/2, then ψk=αk(1,1)T/21/2\psi_k = \alpha_k(1,1)^{\mathsf T}/2^{1/2} for real αk\alpha_k, and

UEVE1/2=∑kαk(2Ak−1)+αEg(∑kαk2+αE2)1/2,(12)\frac{U_E}{V_E^{1/2}}=\frac{\sum_k\alpha_k(2A_k-1)+\alpha_Eg}{\left(\sum_k\alpha_k^2+\alpha_E^2\right)^{1/2}}, \tag*{(12)}

where 2Ak−12A_k-1 are independent Rademacher signs, g∼N(0,1)g \sim\mathcal{N}(0,1) is independent of them, and αE={(1,1)ΣE(1,1)T/2}1/2\alpha_E=\{(1,1)\Sigma_E(1,1)^{\mathsf T}/2\}^{1/2}. The right-hand side of (12) is a sum of Rademacher signs with fixed weights plus an independent normal variable, and its variance is one. It is not normal if Ψ\Psi is not empty, because its fourth cumulant is −2∑kαk4/(∑kαk2+αE2)2<0-2\sum_k\alpha_k^4/(\sum_k\alpha_k^2+\alpha_E^2)^2<0. Without equal residuals, the limit can be normal when Ψ\Psi is not empty. If every ψk\psi_k is proportional to (π,−(1−π))T(\pi,-(1-\pi))^{\mathsf T} and wπTΣEwπ>0w_\pi^{\mathsf T}\Sigma_Ew_\pi>0, then the points cancel from UEU_E and contribute only a constant to VEV_E.

Heavy-tailed outcome errors

Configurations with infinitely many points arise when outcome errors are drawn from a heavy-tailed law. Let S1\mathbb{S}^1 be the unit circle in R2\mathbb{R}^2.

Definition 3.3. Let γ>0\gamma>0. A random vector ε∈R2\varepsilon\in\mathbb{R}^2 is regularly varying with index γ\gamma and spectral measure Λ\Lambda, a Borel probability measure on S1\mathbb{S}^1, if P(∥ε∥>x)\mathbb{P}(\lVert\varepsilon\rVert>x) is regularly varying at infinity with index −γ-\gamma and P(ε/∥ε∥∈⋅∣∥ε∥>x)\mathbb{P}(\varepsilon/\lVert\varepsilon\rVert\in\cdot\mid\lVert\varepsilon\rVert>x) converges weakly to Λ\Lambda as x→∞x\to\infty.

Let T1<T2<⋯T_1<T_2<\cdots be the arrival times of a unit-rate Poisson process, let o1,o2,…\mathbf{o}_1,\mathbf{o}_2,\ldots be independent with law Λ\Lambda and independent of (Tk)(T_k), and define the random family Ψγ=(ψk)k≥1\Psi_\gamma=(\psi_k)_{k\geq1} by

ψk=Tk−1/γok(∑j≥1Tj−2/γ)1/2.(13)\psi_k=\frac{T_k^{-1/\gamma}\mathbf{o}_k}{\left(\sum_{j\geq1}T_j^{-2/\gamma}\right)^{1/2}}. \tag*{(13)}

Let UEU_E and VEV_E be defined by (11) with Ψ=Ψγ\Psi=\Psi_\gamma, ΣE=0\Sigma_E=0, and (Ak)(A_k) independent of Ψγ\Psi_\gamma.

Proposition 3.4. Let (Xi,εi)(X_i,\varepsilon_i) for i≥1i\geq1 be independent and identically distributed, where εi=(εi(1),εi(0))T\varepsilon_i=(\varepsilon_i(1),\varepsilon_i(0))^{\mathsf T}, and let ΠN\Pi_N consist of the first NN units, with Yi(z)=μz(Xi)+εi(z)Y_i(z)=\mu_z(X_i)+\varepsilon_i(z) for measurable functions μz\mu_z such that Eμz(X1)2<∞\mathbb{E}\mu_z(X_1)^2<\infty, where μz\mu_z is a constant if d=0d=0. Suppose that ε1\varepsilon_1 is independent of X1X_1, has mean zero, and is regularly varying with index γ∈(1,2)\gamma\in(1,2) and spectral measure Λ\Lambda. If d≥1d\geq1, suppose also that E∥X1∥2<∞\mathbb{E}\lVert X_1\rVert^2<\infty and that the covariance matrix of X1X_1 is nonsingular. Let the assignment satisfy Assumption 1 and be independent

of the population. On the event that G\mathbf{G} is singular, let hi=0h_i=0 and ei(z)=Yi(z)−Y‾(z)e_i(z)=Y_i(z)-\overline{Y}(z), and let ratios with denominator rN=0r_N=0 be zero.

(a) The share max⁡i∥ei∥2/rN2\max_i\lVert e_i\rVert^2/r_N^2 converges in distribution to the law of T1−2/γ/∑jTj−2/γ\mathrm{T}_1^{-2/\gamma}/\sum_j\mathrm{T}_j^{-2/\gamma}, which is concentrated on (0,1)(0,1). If d≥1d\ge1, also P(G singular)→0\mathbb{P}(\mathbf{G}\ \text{singular})\to0 and max⁡ihi→0\max_i h_i\to0 in probability.

(b) P(VE>0)=1\mathbb{P}(V_E>0)=1, and for every continuous function ff on [−∞,∞][-\infty,\infty],

E{f(TN)∣ΠN}⇝E{f(UE/VE1/2)∣Ψγ}.(14)\mathbb{E}\{f(T_N)\mid\Pi_N\}\rightsquigarrow\mathbb{E}\{f(U_E/V_E^{1/2})\mid\Psi_\gamma\}. \tag*{(14)}

(c) The random variable E{f(UE/VE1/2)∣Ψγ}\mathbb{E}\{f(U_E/V_E^{1/2})\mid\Psi_\gamma\} is almost surely constant for every continuous function ff on [−∞,∞][-\infty,\infty] if and only if Λ\Lambda is concentrated on the line {x∈R2:wπ⊤x=0}\{x\in\mathbb{R}^2:w_\pi^\top x=0\}.

Section S4.2 proves Proposition 3.4, using the limits of the ranked norms and directions in LePage et al. [23]. In (14), E(⋅∣ΠN)\mathbb{E}(\cdot\mid\Pi_N) is the randomization expectation with the population held fixed, the population being drawn once. By part (a), the largest residual pair keeps a share of rN2r_N^2 that does not vanish, so the negligibility condition fails. By part (b), the randomization law of TNT_N for the population at hand converges, in the sense of (14), to the law of Theorem 3.2 with ΣE=0\boldsymbol{\Sigma}_E=\mathbf{0}, whose points are the normalized points (13) of a Poisson process. This limit is a random law. By part (c), it varies with the population unless Λ\Lambda is concentrated on the line wπ⊤x=0w_\pi^\top x=0. In that case, every point lies on this line, the points cancel from UEU_E, and the limit is the point mass at zero. The limits of the bootstrap of the mean with infinite variance (Athreya, [5]) and of permuted sums of heavy-tailed samples (Aue et al., [6]; Berkes et al. [8]) also depend on the sample. The functions μz\mu_z allow conditional means that are nonlinear in XiX_i, and their projection residuals enter rN2r_N^2 only at the order NN, which is smaller than the order of the heavy-tailed errors.

Pairing of the potential outcomes

Each unit reveals one potential outcome, so the pairing of Yi(1)Y_i(1) with Yi(0)Y_i(0) is unobserved, and the variance of the unit-level effects is thus unidentified ([3]). The next proposition shows that this pairing can change the limit of TNT_N.

Proposition 3.5. Let d=0d = 0, suppose Assumption 1 holds with N1=N/2N_1 = N/2 along even NN, and let xN>0x_N > 0 satisfy xN/N1/2→∞x_N/N^{1/2} \to\infty, and let wi=(−1)iw_i = (-1)^i. In population I, let Y1(1)=Y1(0)=xNY_1(1) = Y_1(0) = x_N, Y2(1)=Y2(0)=0Y_2(1) = Y_2(0) = 0 and Yi(1)=Yi(0)=wiY_i(1) = Y_i(0) = w_i for i≥3i \ge3. In population II, let Y1(1)=xNY_1(1) = x_N, Y1(0)=0Y_1(0) = 0, Y2(1)=0Y_2(1) = 0, Y2(0)=xNY_2(0) = x_N and Yi(1)=Yi(0)=wiY_i(1) = Y_i(0) = w_i for i≥3i \ge3. Then, both populations have τN=0\tau_N = 0 and the same multisets {Yi(1):i≤N}\{Y_i(1) : i \le N\} and {Yi(0):i≤N}\{Y_i(0) : i \le N\}, but, along even NN,

PI(∣TN∣≤t)→0,PII(∣TN∣≤t)→Φ(t)/2(0<t<1),\mathbb{P}_{\mathrm{I}}(|T_N| \le t) \to0,\qquad\mathbb{P}_{\mathrm{II}}(|T_N| \le t) \to\Phi(t)/2 \qquad(0 < t < 1),
PI(∣TN∣≤t)→1,PII(∣TN∣≤t)→{1+Φ(t)}/2(t>1),\mathbb{P}_{\mathrm{I}}(|T_N| \le t) \to1,\qquad\mathbb{P}_{\mathrm{II}}(|T_N| \le t) \to\{1 + \Phi(t)\}/2 \qquad(t > 1),

where PI\mathbb{P}_{\mathrm{I}} and PII\mathbb{P}_{\mathrm{II}} are the randomization distributions for the two populations.

Section S4.4 proves Proposition 3.5. In the regular case, the pairing affects only (1,−1)ΣE(1,−1)⊤(1,-1)\Sigma_E(1,-1)^\top, the limit of ∑i{ei(1)−ei(0)}2/rN2\sum_i\{e_i(1)-e_i(0)\}^2/r_N^2. The limit of the normalized variance estimator exceeds the variance of the limit of the normalized estimator by this amount, so the Wald interval is conservative. With a residual configuration, it can instead change the shape of the limit law. The two populations of Proposition 3.5 have different limiting coverage at every critical value t≠1t \ne1, and the ordering of the two coverages reverses at t=1t = 1. Population I has the single point (1,1)⊤/21/2(1,1)^\top/2^{1/2} and ΣE=0\Sigma_E = 0. Population II has the points (1,0)⊤/21/2(1,0)^\top/2^{1/2} and (0,1)⊤/21/2(0,1)^\top/2^{1/2} and ΣE=0\Sigma_E = 0, so P(VE=0)=1/4\mathbb{P}(V_E = 0) = 1/4 in Theorem 3.2.

Coverage under balanced allocation

The limits of Theorem 3.2 depend on how ei(1)e_i(1) is paired with ei(0)e_i(0) at the dominant units, which cannot be identified from the observed data. Under balanced allocation, that is, under Assumption 1 with π=1/2\pi= 1/2, the coverage of the Wald interval nevertheless admits explicit lower bounds. Let Φˉ=1−Φ\bar{\Phi} = 1 - \Phi.

Proposition 4.1. Suppose Assumption 1 holds with π=1/2\pi= 1/2, and Condition 1 holds. Then, for every t>0t > 0,

lim inf⁡N→∞P(∣TN∣≤t)≥1−2c∗Φˉ(t)2,c∗=14Φˉ(21/2)≈3.1787.(15)\liminf_{N \to\infty} \mathbb{P}(|T_N| \le t) \ge\frac{1 - 2c_* \bar{\Phi}(t)}{2}, \qquad c_* = \frac{1}{4\bar{\Phi}(2^{1/2})} \approx3.1787. \tag*{(15)}

In particular, the Wald interval of nominal level 1−ς1-\varsigma has asymptotic coverage at least (1−c∗ς)/2(1-c_*\varsigma)/2, which is approximately 0.421 at ς=0.05\varsigma=0.05.

Section S5.2 proves Proposition 4.1. The bound requires no condition on the outcomes. When N1=N/2N_1=N/2, the assignments (Z1,…,ZN)(Z_1,\ldots,Z_N) and (1−Z1,…,1−ZN)(1-Z_1,\ldots,1-Z_N) are equally likely. Thus, when πN→1/2\pi_N\to1/2, changing the second assignment at a vanishing fraction of the units gives an assignment with the law of the first. At the two assignments, the numerators of TNT_N have the same absolute value to first order, and the sums of the squared population residuals over the observed arms add up to rN2r_N^2 to first order. For one of the two assignments, the variance estimate is therefore at least about 2rN2/N22r_N^2/N^2, and the interval covers τN\tau_N unless a weighted sum of the signs 2Zi−12Z_i-1, with squared weights summing to at most one, is large. Up to a term that vanishes in probability, this sum is a sum of independent Rademacher signs with the same weights, and c∗c_* is the sharp constant of the Gaussian comparison for such sums of Bentkus and Dzindzalieta [7].

A higher floor holds if the two potential residuals of the units are matched in aggregate. Let mi=ei(1)+ei(0)m_i=e_i(1)+e_i(0).

Condition 3 (Aggregate matching). There is η>0\eta>0 such that

lim inf⁡N∑imin⁡{ei(1)2,ei(0)2}∑imi2≥η2,\liminf_N \frac{\sum_i \min\{e_i(1)^2,e_i(0)^2\}}{\sum_i m_i^2} \ge\eta^2,

where the ratio is +∞+\infty if $\sum_i m_i^2=0.

If ei(1)=ei(0)e_i(1)=e_i(0) for all ii, Condition 3 holds with η=1/2\eta=1/2. If ei(1)=ρiei(0)e_i(1)=\rho_i e_i(0) with ρi∈[ρ,1/ρ]\rho_i\in[\rho,1/\rho] for some ρ∈(0,1]\rho\in(0,1], it holds with η=ρ/(1+ρ)\eta=\rho/(1+\rho) (Section S5.3). Geometrically, plot each unit as the point (ei(1),ei(0))(e_i(1),e_i(0)). Under balanced allocation, each unit enters the numerator of TNT_N through mim_i, which is proportional to the projection of its point onto the diagonal, with a sign given by its arm. It enters the variance estimator through the coordinate of its arm, whose square is at least min⁡{ei(1)2,ei(0)2}\min\{e_i(1)^2,e_i(0)^2\}, the squared distance from the point to the nearer coordinate axis. Condition 3 prevents the residual mass from concentrating, in aggregate, near the axes relative to its projection onto the diagonal. Mass near the line ei(1)=−ei(0)e_i(1)=-e_i(0) enters the variance estimator but contributes little to the numerator, and it increases the ratio in Condition 3.

Theorem 4.2. Suppose Assumption 1 holds with π=1/2\pi= 1/2, and Conditions 1 and 3 hold.

Then, for every t>0t > 0,

lim sup⁡N→∞P(∣TN∣>t)≤2c∗Φ‾(2ηt).(16)\limsup_{N \to\infty} \mathbb{P}(|T_N| > t) \le2c_* \overline{\Phi}(2\eta t). \tag*{(16)}

In particular, if ei(1)=ei(0)e_i(1) = e_i(0) for all ii, the Wald interval of nominal level 1−ς1-\varsigma has asymptotic coverage at least 1−c∗ς1-c_*\varsigma.

Section S5.3 proves Theorem 4.2. Theorem 4.2 requires no configuration condition on the residuals. For η=1/2\eta= 1/2, the floor is approximately 0.841 at ς=0.05\varsigma= 0.05 and 0.968 at ς=0.01\varsigma= 0.01. For η=1/3\eta= 1/3, which corresponds to pointwise matching with ρ=1/2\rho= 1/2, it is approximately 0.392 at ς=0.05\varsigma= 0.05. At ς=0.05\varsigma= 0.05, the floor of Theorem 4.2 exceeds that of Proposition 4.1 only if η>0.340\eta> 0.340. The proof replaces complete randomization by independent signs through a coupling of the type of [18]. The magnitudes ∣mi∣|m_i| are then fixed, as the absolute values of the observations are in the analysis of Student’s tt statistic under symmetry ([12]), and the bound has the form of the tail bound of [28] for that statistic under symmetry, with the constant of [7].

For K≥1K \ge1 and 0≤j≤K0 \le j \le K, let θK(j)={j/π−(K−j)/(1−π)}/{j/π2+(K−j)/(1−π)2}1/2\theta_K(j) = \{j/\pi- (K-j)/(1-\pi)\}/\{j/\pi^2 + (K-j)/(1-\pi)^2\}^{1/2}, where π\pi is the limit in Assumption 1. The next proposition shows that both floors are close to the best possible and that neither holds at the allocation π=0.4\pi= 0.4 and the nominal level 0.95.

Proposition 4.3. Let d=0d = 0, suppose Assumption 1 holds, let K≥1K \ge1 be fixed, let xN>0x_N > 0 satisfy xN/N1/2→∞x_N/N^{1/2} \to\infty, and let wN,iw_{N,i} satisfy sup⁡NN−1∑i≤NwN,i2<∞\sup_N N^{-1}\sum_{i \le N} w_{N,i}^2 < \infty.

(a) If Yi(1)=Yi(0)=xNY_i(1) = Y_i(0) = x_N for i≤Ki \le K and Yi(1)=Yi(0)=wN,iY_i(1) = Y_i(0) = w_{N,i} for i>Ki > K, then for every t>0t > 0 outside the finite set {∣θK(j)∣:0≤j≤K}\{|\theta_K(j)| : 0 \le j \le K\},

P(∣TN∣≤t)→∑j=0K(Kj)πj(1−π)K−j1{∣θK(j)∣≤t}.(17)\mathbb{P}(|T_N| \le t) \to\sum_{j=0}^{K} \binom{K}{j}\pi^j(1-\pi)^{K-j}1\{|\theta_K(j)| \le t\}. \tag*{(17)}

For K=4K = 4 and t=Φ−1(0.975)t = \Phi^{-1}(0.975), P(∣TN∣≤t)→1−π4−(1−π)4\mathbb{P}(|T_N| \le t) \to1-\pi^4-(1-\pi)^4 for every π∈(0,1)\pi\in(0,1). The limit is 7/87/8 at π=1/2\pi= 1/2 and approximately 0.589 at π=0.2\pi= 0.2.

(b) Let Yi(1)=Yi(0)=wN,iY_i(1) = Y_i(0) = w_{N,i} for i≥2i \ge2. If Y1(1)=xNY_1(1) = x_N and Y1(0)=0Y_1(0) = 0, then P(∣TN∣≤t)→π\mathbb{P}(|T_N| \le t) \to\pi for every t>1−πt > 1-\pi and P(∣TN∣≤t)→0\mathbb{P}(|T_N| \le t) \to0 for every t∈(0,1−π)t \in(0,1-\pi). If Y1(1)=0Y_1(1) = 0 and

Y1(0)=xNY_1(0) = x_N, the same holds with π\pi and 1−π1-\pi interchanged.

(c) Let π=0.4\pi= 0.4, Y1(1)=Y2(1)=5xNY_1(1) = Y_2(1) = 5x_N, Y1(0)=−2xNY_1(0) = -2x_N, Y2(0)=−xNY_2(0) = -x_N, and Yi(1)=Yi(0)=wN,iY_i(1) = Y_i(0) = w_{N,i} for i≥3i \ge3. Then, Condition 3 holds with η=5−1/2\eta= 5^{-1/2}, and for t=Φ−1(0.975)t = \Phi^{-1}(0.975), P(∣TN∣≤t)→1−(1−π)2=0.64\mathbb{P}(|T_N| \le t) \to1 - (1-\pi)^2 = 0.64.

Section S5.4 proves Proposition 4.3. In part (a), Condition 3 holds with η=1/2\eta= 1/2, and ∣θK(0)∣=∣θK(K)∣=K1/2|\theta_K(0)| = |\theta_K(K)| = K^{1/2}, whereas ∣θK(j)∣<(K−1)1/2|\theta_K(j)| < (K-1)^{1/2} for 0<j<K0 < j < K. For K=4K = 4 and t=Φ−1(0.975)t = \Phi^{-1}(0.975), the assignments that place all four dominant units in one arm are therefore not covered, and all other assignments are. At π=1/2\pi= 1/2, the limiting coverage is 7/87/8, so no floor valid for every population with η=1/2\eta= 1/2 exceeds 7/87/8 at ς=0.05\varsigma= 0.05, and the floor 0.841 of Theorem 4.2 is close to sharp. The probability π4+(1−π)4\pi^4 + (1-\pi)^4 of these assignments tends to one as π→0\pi\to0, and thus aggregate matching without balanced allocation gives no floor that holds for every π\pi. With K=2K = 2 and π=1/2\pi= 1/2, ∣θ2(j)∣∈{0,21/2}|\theta_2(j)| \in\{0, 2^{1/2}\}, so the limiting coverage is 1/21/2 for 0<t<21/20 < t < 2^{1/2}, whereas the floor 1−2c∗Φ‾(t)1 - 2c_* \overline{\Phi}(t) of Theorem 4.2 with η=1/2\eta= 1/2 tends to 1−2c∗Φ‾(21/2)=1/21 - 2c_* \overline{\Phi}(2^{1/2}) = 1/2 as t↑21/2t \uparrow2^{1/2}. The constant 2c∗2c_* in (4.2) therefore cannot be lowered.

In part (b), the single dominant unit lies on a coordinate axis, so the ratio in Condition 3 tends to zero. If that unit is treated, TN→1−πT_N \to1-\pi in probability and the interval covers. If it is assigned to control, its observed outcome Y1(0)=0Y_1(0) = 0 is of the size of the outcomes of the remaining units, so N2V^=O(N)N^2\widehat{V} = O(N), whereas N(τ^−τN)=−xN+Op(N1/2)N(\widehat{\tau} - \tau_N) = -x_N + O_p(N^{1/2}), and ∣TN∣→∞|T_N| \to\infty. At π=1/2\pi= 1/2, no floor valid for every population under balanced allocation exceeds 1/21/2, and the universal floor 0.421 of Proposition 4.1 is within a factor of 1.19 of this bound. For general π\pi, the two populations of part (b) have limiting coverage π\pi and 1−π1-\pi, so without balanced allocation the coverage can be as low as min⁡(π,1−π)\min(\pi,1-\pi). In part (c), only the assignments that place both dominant units in control are not covered. The floor of Theorem 4.2 at η=5−1/2\eta= 5^{-1/2} is approximately 0.747, above the limiting coverage 0.64, and for the first population of part (b) at π=0.4\pi= 0.4 the limiting coverage 0.4 is below the floor 0.421 of Proposition 4.1. Hence neither floor holds at the allocation π=0.4\pi= 0.4 and the nominal level 0.95.

Numerical studies

Finite populations and the limit laws

This study checks how closely the limits of Theorem 3.2 and Propositions 3.5 and 4.3 describe finite populations of moderate size. The populations of Propositions 3.5 and 4.3 have d=0d=0, N=20,000N=20{,}000 and xN=Nx_N=N, with wN,i=(−1)iw_{N,i}=(-1)^i, K=4K=4 in Proposition 4.3(a), and π∈{0.5,0.2}\pi\in\{0.5,0.2\} in Proposition 4.3. For Theorem 3.2, the population has N=20,000N=20{,}000, π=0.3\pi=0.3, one standard normal covariate, and potential outcomes Yi(1)=2Xi+εi(1)Y_i(1)=2X_i+\varepsilon_i(1) and Yi(0)=Xi+εi(0)Y_i(0)=X_i+\varepsilon_i(0). The error pairs are bivariate normal with unit variances and correlation 0.5, except for three units, whose error pairs are N1/2(2,0.5)N^{1/2}(2,0.5), N1/2(−1.5,1)N^{1/2}(-1.5,1) and N1/2(0.5,−2)N^{1/2}(0.5,-2). Its limit law is evaluated at its own three largest normalized residual pairs and the second-moment matrix of the remaining pairs, from 10610^6 draws. For each population, the study draws 100,000 assignments and records the coverage of τ^±tV^1/2\widehat{\tau}\pm t\widehat{V}^{1/2} for t∈{0.25,1.96}t\in\{0.25,1.96\}. Table 1 reports these coverages next to their limits.

Table 1: Coverage of τ^±tV^1/2\widehat{\tau}\pm t\widehat{V}^{1/2} over 100,000 assignments in populations of size N=20,000N=20{,}000, and its limit. The columns give the allocation π\pi, the largest share max⁡i∥ei∥2/rN2\max_i\lVert e_i\rVert^2/r_N^2 (share), and, for t=0.25t=0.25 and t=1.96t=1.96, the coverage over assignments (cov.) and the limit of the cited result (limit). The largest Monte Carlo standard error is 0.0016.

populationπ\pisharet=0.25t=0.25 cov.t=0.25t=0.25 limitt=1.96t=1.96 cov.t=1.96t=1.96 limit
Proposition 3.5, population I0.51.0000.0000.0001.0001.000
Proposition 3.5, population II0.50.5000.3010.2990.9870.988
Proposition 4.3(a), K=4K=40.50.2500.3720.3750.8750.875
Proposition 4.3(a), K=4K=40.20.2500.4070.4100.5880.589
Proposition 4.3(b)0.51.0000.0000.0000.5020.500
Proposition 4.3(b)0.21.0000.0000.0000.1980.200
Theorem 3.2, d=1d=10.30.3090.2030.2040.9980.998

Table 1.

The results match Theorem 3.2 and Propositions 3.5 and 4.3. Every coverage agrees with its limit within two Monte Carlo standard errors, including the limits zero and one. The two populations of Proposition 3.5 have the same marginal distributions of potential outcomes, yet their coverages at both critical values differ as the limits predict. In the population of Proposition 4.3(b), the coverage of the nominal 95% interval is close to the allocation π\pi.

Coverage with heavy-tailed errors

This study examines the coverage of the nominal 95% Wald interval when the errors are heavy-tailed. Each population has N=4,000N = 4{,}000 units with three independent standard normal covariates, Yi(1)=XiT(2,2,2)T+εi(1)Y_i(1) = \mathbf{X}_i^{\mathsf{T}}(2,2,2)^{\mathsf{T}} + \varepsilon_i(1) and Yi(0)=XiT(1,1,1)T+εi(0)Y_i(0) = \mathbf{X}_i^{\mathsf{T}}(1,1,1)^{\mathsf{T}} + \varepsilon_i(0). A symmetric Pareto variable with index γ\gamma is generated as PsPs, where P(P>x)=x−γ\mathbb{P}(P > x) = x^{-\gamma} for x≥1x \ge1 and ss is an independent random sign. The errors are symmetric Pareto with index γ∈{1.2,1.5,1.8}\gamma\in\{1.2,1.5,1.8\} and of three kinds: (i) additive errors that have εi(1)=εi(0)\varepsilon_i(1) = \varepsilon_i(0); (ii) matched errors that have εi(1)=ρiεi(0)\varepsilon_i(1) = \rho_i\varepsilon_i(0), with log⁡ρi\log\rho_i uniform on [log⁡(1/2),log⁡2][\log(1/2),\log2]; and (iii) independent errors that have εi(1)\varepsilon_i(1) and εi(0)\varepsilon_i(0) independent. The allocation is π∈{0.5,0.2}\pi\in\{0.5,0.2\}. Each setting has 200 populations, and each population has 1,000 assignments, analyzed with the interacted estimator on the three covariates and with the difference in means. For each population, the study computes max⁡i∥ei∥2/rN2\max_i \lVert e_i\rVert^2/r_N^2, ηN2=∑imin⁡{ei(1)2,ei(0)2}/∑imi2\eta_N^2 = \sum_i \min\{e_i(1)^2,e_i(0)^2\}/\sum_i m_i^2, which is the ratio of Condition 3 at NN, the floor 1−2c∗Φ{2ηNΦ−1(0.975)}1 - 2c_* \Phi\{2\eta_N\Phi^{-1}(0.975)\} of Theorem 4.2, and the coverage. Table 2 reports summaries of these quantities over the populations of each setting.

Table 2: Coverage of nominal 95% Wald intervals with symmetric Pareto errors of index γ\gamma, N=4,000N = 4{,}000, and 200 populations of 1,000 assignments each, for the interacted estimator on three normal covariates (τ^\hat{\tau}) and the difference in means (d=0d = 0). The columns give, for the interacted estimator, the median of ηN\eta_N over populations, the floor of Theorem 4.2 at this median and the median of max⁡i∥ei∥2/rN2\max_i \lVert e_i\rVert^2/r_N^2 (share), and for each estimator the mean and the minimum of the coverage and the number of populations whose coverage is more than two Monte Carlo standard errors below the larger of their own floor of Theorem 4.2 and the floor 0.421 of Proposition 4.1 (below). The Monte Carlo standard error at coverage xx is {x(1−x)/1000}1/2\{x(1-x)/1000\}^{1/2}, evaluated at that larger floor. No floor applies for π=0.2\pi= 0.2 (–). The largest leverage in any population is 0.0097.

γ\gammaerrorsπ\piηN\eta_Nfloorsharemeanminbelowmeanminbelow
1.2additive0.50.5000.8410.5160.9730.93300.9720.9320
1.2additive0.20.500–0.4660.9720.907–0.9720.914–
1.2matched0.50.4140.6680.5080.9670.93400.9670.9330
1.2matched0.20.415–0.5150.9600.685–0.9600.690–
1.2independent0.50.040<0< 00.5110.8860.48300.8860.4830
1.2independent0.20.045–0.4900.8790.180–0.8800.180–
1.5additive0.50.5000.8410.3780.9640.93800.9630.9410
1.5additive0.20.500–0.3560.9630.915–0.9620.929–
1.5matched0.50.4180.6780.3500.9640.93700.9640.9380
1.5matched0.20.413–0.3290.9560.839–0.9570.845–
1.5independent0.50.126<0< 00.3420.9540.48400.9550.4860
1.5independent0.20.134–0.2880.9330.202–0.9340.202–
1.8additive0.50.5000.8410.1980.9550.93400.9570.9320
1.8additive0.20.500–0.2090.9560.922–0.9560.930–
1.8matched0.50.4120.6630.1990.9580.94000.9590.9380
1.8matched0.20.413–0.2510.9550.867–0.9560.895–
1.8independent0.50.235<0< 00.1870.9700.48400.9690.4910
1.8independent0.20.232–0.2140.9740.531–0.9690.691–

Table 2.

The results match Propositions 4.1 and 4.3 and Theorem 4.2. Under balanced allocation, no population has coverage more than two Monte Carlo standard errors below the larger of its floor of Theorem 4.2 and the floor 0.421 of Proposition 4.1. The largest residual pair carries a large share of rN2r_N^2, and this share decreases as γ\gamma grows. With additive and matched errors, the mean coverage is at or above 0.95, and the lowest coverage stays close to it under balanced allocation. With independent errors, the median of ηN\eta_N is small and the floor of Theorem 4.2 at this median is not positive. Specifically, independent heavy-tailed errors place the dominant residual pairs near the coordinate axes, so ηN\eta_N falls below 0.257, where the floor of Theorem 4.2 is negative and only the floor 0.421 of Proposition 4.1 is informative. The mean coverage is below 0.95 for γ=1.2\gamma= 1.2, and for γ=1.5\gamma= 1.5 at π=0.2\pi= 0.2. The lowest coverage under balanced allocation is then close to 1/21/2, above the floor of Proposition 4.1, and the populations with the lowest coverage have one residual pair that carries almost all of rN2r_N^2, as in Proposition 4.3(b). At π=0.2\pi= 0.2, no floor applies, and with independent errors and γ≤1.5\gamma\le1.5 the lowest coverage falls close to π\pi. The interacted estimator and the difference in means have nearly the same mean coverage in every setting, because the covariates contribute to rN2r_N^2 only at the order NN, which is smaller than the order of the heavy-tailed errors (§ 3.2).

Data application

This section applies the results to data from three randomized experiments whose outcomes, medical spending, emergency-department charges, and customer spending, are heavy-tailed. The coverage of an interval cannot be evaluated from the experiment itself, because only one assignment is observed and τN\tau_N is unknown. We therefore impute the potential outcomes of each experiment under a sharp model of the effect and draw new assignments by complete randomization. The coverage is then the proportion of assignments whose interval contains τN\tau_N.

The first population comes from the RAND Health Insurance Experiment [2]. We use the person-year file distributed with Angrist and Pischke [1] and keep the 5,477 persons observed in both of their first two years in the experiment. The treated arm consists of the 1,843 persons assigned to free care, and the control arm consists of those assigned to a plan with cost sharing. The outcome YiY_i is the person’s total medical spending in the second year, and XiX_i is the total in the first year. The second population comes from the Oregon Health Insurance Experiment [14, 31]. We use the public-use emergency-department file for the Portland area and keep the 19,619 lottery participants who signed up alone and have both charge totals recorded. The treated arm consists of the 6,750 persons selected in the lottery. The outcome YiY_i is the sum of the charges for the person’s emergency-department visits from 10 March 2008 to 30 September 2009, and XiX_i is the same sum from 1 January 2007 to 9 March 2008, before the first lottery notification. The third population comes from the e-mail experiment of Hillstrom [19], in which 64,000 customers of a retailer were assigned at random to a men’s-merchandise e-mail, a women’s-merchandise e-mail or no e-mail. We keep the 42,613 customers assigned to the men’s e-mail, which forms the treated arm, or to no e-mail. The outcome YiY_i is the customer’s spending in the two weeks after the e-mail, and XiX_i is the spending in the preceding year.

The recorded outcome of each unit is its potential outcome under the arm to which it was assigned in the experiment, and the other potential outcome is imputed under one of two sharp models. The constant effect sets Yi(1)−Yi(0)=τNY_i(1)-Y_i(0)=\tau_N for every unit, with τN\tau_N equal to the difference between the average recorded outcomes of the treated and control arms. It gives ei(1)=ei(0)e_i(1)=e_i(0), so Condition 3 holds with η=1/2\eta=1/2. The proportional effect sets Yi(1)=ρYi(0)Y_i(1)=\rho Y_i(0) for every unit, with ρ\rho equal to the ratio of these averages. It gives ei(1)=ρei(0)e_i(1)=\rho e_i(0), and since ρ>1\rho>1 in the three experiments, Condition 3 holds with η=1/(1+ρ)\eta=1/(1+\rho). On the completed population, we draw 100,000 assignments by complete randomization with π=0.5\pi=0.5 and with π=0.2\pi=0.2, and compute the nominal 95% Wald interval for the difference in means (d=0d=0) and for the interacted estimator. The interacted estimator uses the covariate Xi=log⁡(1+Xi)X_i=\log(1+X_i), so d=1d=1. We do not use XiX_i itself, because its leverage is about 0.19 Table 3: Coverage of nominal 95% Wald intervals over 100,000 assignments in the RAND (N=5,477N = 5{,}477), Oregon (N=19,619N = 19{,}619) and retail (N=42,613N = 42{,}613) populations under a constant or proportional (prop.) effect, for the difference in means (d=0d = 0) and for the interacted estimator on Xi=log⁡(1+Xi)X_i = \log(1 + X_i) (log). The constant effect τN\tau_N is 155.55, 133.89 and 0.77, and the ratio ρ\rho of the proportional effect is 1.34, 1.03 and 2.18. The number of treated units N1N_1 is 2,738, 9,810 and 21,306 at π=0.5\pi= 0.5 and 1,095, 3,924 and 8,523 at π=0.2\pi= 0.2. All values are given for the three populations in the order listed. The columns give the maximum leverage max⁡ihi\max_i h_i of XiX_i, the largest share max⁡i∥ei∥2/rN2\max_i \lVert e_i\rVert^2/r_N^2 for each estimator, ηN\eta_N, the floor of Theorem 4.2 at ηN\eta_N, which applies at π=0.5\pi= 0.5, and the coverage. The largest Monte Carlo standard error is 0.0008.

for one person in the RAND population, outside the scope of Condition 1. Because the assignments are redrawn, the covariate need not precede the original assignment, as in the RAND population. Table 3 reports the coverage, together with the maximum leverage of XiX_i, the largest share max⁡i∥ei∥2/rN2\max_i \lVert e_i\rVert^2/r_N^2, the ratio ηN\eta_N of Condition 3 at NN defined in § 5.2, and the floor of Theorem 4.2 at ηN\eta_N.

populationeffectmax⁡ihi\max_i h_ishare d=0d = 0share logηN\eta_Nfloorπ=0.5\pi= 0.5 d=0d = 0π=0.5\pi= 0.5 logπ=0.2\pi= 0.2 d=0d = 0π=0.2\pi= 0.2 log
RANDconstant0.00140.3290.3380.5000.8410.9540.9540.9370.937
RANDprop.0.00140.3730.3810.4280.7030.9550.9550.9260.925
Oregonconstant0.00030.0160.0160.5000.8410.9510.9510.9470.947
Oregonprop.0.00030.0160.0160.4920.8290.9510.9510.9470.947
retailconstant0.00020.0260.0260.5000.8410.9510.9510.9450.945
retailprop.0.00020.0580.0580.3150.3080.9610.9610.9440.944

Table 3. Coverage of nominal 95% Wald intervals over 100,000 assignments in the RAND (N=5,477N = 5{,}477), Oregon (N=19,619N = 19{,}619) and retail (N=42,613N = 42{,}613) populations under a constant or proportional (prop.) effect, for the difference in means (d=0d = 0) and for the interacted estimator on Xi=log⁡(1+Xi)X_i = \log(1 + X_i) (log). The constant effect τN\tau_N is 155.55, 133.89 and 0.77, and the ratio ρ\rho of the proportional effect is 1.34, 1.03 and 2.18. The number of treated units N1N_1 is 2,738, 9,810 and 21,306 at π=0.5\pi= 0.5 and 1,095, 3,924 and 8,523 at π=0.2\pi= 0.2. All values are given for the three populations in the order listed. The columns give the maximum leverage max⁡ihi\max_i h_i of XiX_i, the largest share max⁡i∥ei∥2/rN2\max_i \lVert e_i\rVert^2/r_N^2 for each estimator, ηN\eta_N, the floor of Theorem 4.2 at ηN\eta_N, which applies at π=0.5\pi= 0.5, and the coverage. The largest Monte Carlo standard error is 0.0008.

In the RAND population, the largest residual pair carries a large share of rN2r_N^2 for both estimators, whereas in the Oregon and retail populations, its share is small. The maximum leverage of XiX_i is small in all three populations. Under balanced allocation, the coverage is above 0.95 in every population, well above the floors of Proposition 4.1 and Theorem 4.2. With π=0.2\pi= 0.2, the floors do not apply, and by Proposition 4.3 they need not hold. The coverage in the RAND population then falls below 0.95, more under the proportional effect than under the constant effect, whereas in the Oregon and retail populations, it stays close to 0.95. The covariate changes the largest share only slightly, and the two estimators have nearly the same coverage in every setting. Thus, in these populations, the largest loss of coverage occurs under unbalanced allocation in the population in which a few units dominate the residual variation.

Discussion

We studied the studentized difference in means and the interacted estimator, with covariates whose leverage vanishes, when a few units carry a share of the residual variation that does not vanish. The studentized statistic then has a limit that is nonnormal in general and depends on how the two potential residuals of the dominant units are paired. When the errors are heavy-tailed, this limit is random and, outside a degenerate case, varies with the population. Under balanced allocation, the Wald interval keeps explicit coverage floors, which are close to the best possible, and neither floor extends to every allocation.

Natural next steps are the best universal floor under balanced allocation, which lies between 0.421 and the coverage 1/21/2 of Proposition 4.3(b), a coverage floor for a fixed allocation π≠1/2\pi\ne1/2, and an interval whose coverage is guaranteed for every population, including populations whose covariates have leverage that does not vanish.

Data availability

The RAND data are distributed with Angrist and Pischke [1], the Oregon data are the public-use files of the Oregon Health Insurance Experiment [14], and the retail data are those of [19].

Supplementary Material

Section S1 of the Supplementary Material states conventions and external results. Sections S2–S5 prove the results of § 2–§ 4. The Supplementary Material is provided as an ancillary file with this article.

References

  1. [1]Angrist, J. D. and Pischke, J.-S. (2015), Mastering ’Metrics: The Path from Cause to Effect, Princeton: Princeton University Press.
  2. [2]Aron-Dine, A., Einav, L., and Finkelstein, A. (2013), “The RAND Health Insurance Experiment, three decades later,” Journal of Economic Perspectives, 27, 197–222.
  3. [3]Aronow, P. M., Green, D. P., and Lee, D. K. K. (2014), “Sharp bounds on the variance in randomized experiments,” The Annals of Statistics, 42, 850–871.
  4. [4]Athey, S., Bickel, P. J., Chen, A., Imbens, G. W., and Pollmann, M. (2023), “Semiparametric estimation of treatment effects in randomised experiments,” Journal of the Royal Statistical Society Series B: Statistical Methodology, 85, 1615–1638.
  5. [5]Athreya, K. B. (1987), “Bootstrap of the mean in the infinite variance case,” The Annals of Statistics, 15, 724–731.
  6. [6]Aue, A., Berkes, I., and Horváth, L. (2008), “Selection from a stable box,” Bernoulli, 14, 125–139.
  7. [7]Bentkus, V. K. and Dzindzaliета, D. (2015), “A tight Gaussian bound for weighted sums of Rademacher random variables,” Bernoulli, 21, 1231–1237.DOI
  8. [8]Berkes, I., Horváth, L., and Schauer, J. (2010), “Non-central limit theorems for random selections,” Probability Theory and Related Fields, 147, 449–479.DOI
  9. [9]Bloznelis, M. (1999), “A Berry–Esseen bound for finite population Student’s statistic,” The Annals of Probability, 27, 2089–2108.DOI
  10. [10]Chung, E. and Romano, J. P. (2013), “Exact and asymptotically robust permutation tests,” The Annals of Statistics, 41, 484–507.DOI
  11. [11]Edelman, D. (1990), “An inequality of optimal order for the tail probabilities of the T statistic under symmetry,” Journal of the American Statistical Association, 85, 120–122.DOI
  12. [12]Efron, B. (1969), “Student’s t-test under symmetry conditions,” Journal of the American Statistical Association, 64, 1278–1302.
  13. [13]Erdős, P. and Rényi, A. (1959), “On the central limit theorem for samples from a finite population,” Publications of the Mathematical Institute of the Hungarian Academy of Sciences, 4, 49–61.
  14. [14]Finkelstein, A., Taubman, S., Wright, B., Bernstein, M., Gruber, J., Newhouse, J. P., Allen, H., and Baicker, K. (2012), “The Oregon Health Insurance Experiment: evidence from the first year,” The Quarterly Journal of Economics, 127, 1057–1106.
  15. [15]Freidling, T. (2026), “Randomization inference with concentration inequalities,” ArXiv:2609.18586.arxiv.org/abs/2609.18586
  16. [16]Ghosh, A., Deb, N., Karmakar, B., and Sen, B. (2026), “Robustness and efficiency of Rosenbaum’s rank-based estimator in randomized trials: a design-based perspective,” Biometrika, 113. Article asag028.arxiv.org/abs/2111.15524
  17. [17]Giné, E., Götze, F., and Mason, D. M. (1997), “When is the Student t-statistic asymptotically standard normal?” The Annals of Probability, 25, 1514–1531.
  18. [18]Hájek, J. (1960), “Limiting distributions in simple random sampling from a finite population,” Publications of the Mathematical Institute of the Hungarian Academy of Sciences, 5, 361–374.
  19. [19]Hillstrom, K. (2008), “The MineThatData e-mail analytics and data mining challenge,” Data set, MineThatData, https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html.
  20. [20]Janssen, A. (1997), “Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens–Fisher problem,” Statistics & Probability Letters, 36, 9–21.DOI
  21. [21]Jiřina, M. (1982), “Limit theorems for samples from a finite population,” Journal of the Australian Mathematical Society (Series A), 32, 318–327.DOI
  22. [22]Kallenberg, O. (1973), “Canonical representations and convergence criteria for processes with interchangeable increments,” Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 27, 23–36.DOI
  23. [23]LePage, R., Woodroofe, M., and Zinn, J. (1981), “Convergence to a stable distribution via order statistics,” The Annals of Probability, 9, 624–632.DOI
  24. [25]Lin, W. (2013), “Agnostic notes on regression adjustments to experimental data: reexamining Freedman’s critique,” The Annals of Applied Statistics, 7, 295–318.DOI
  25. [26]Logan, B. F., Mallows, C. L., Rice, S. O., and Shepp, L. A. (1973), “Limit distributions of self-normalized sums,” The Annals of Probability, 1, 788–809.DOI
  26. [27]Noether, G. E. (1949), “On a theorem by Wald and Wolfowitz,” The Annals of Mathematical Statistics, 20, 455–458.DOI
  27. [28]Pinelis, I. (1994), “Extremal probabilistic problems and Hotelling’s T² test under a symmetry condition,” The Annals of Statistics, 22, 357–368.
  28. [29]Rosén, B. (1965), “Limit theorems for sampling from finite populations,” Arkiv för Matematik, 5, 383–424.DOI
  29. [30]Su, Y. and Li, X. (2024), “Treatment effect quantiles in stratified randomized experiments and matched observational studies,” Biometrika, 111, 235–254.arxiv.org/abs/2208.11798
  30. [31]Taubman, S. L., Allen, H. L., Wright, B. J., Baicker, K., and Finkelstein, A. N. (2014), “Medicaid increases emergency-department use: evidence from Oregon’s Health Insurance Experiment,” Science, 343, 263–268.
  31. [33]Young, A. (2019), “Channeling Fisher: randomization tests and the statistical insignificance of seemingly significant experimental results,” The Quarterly Journal of Economics, 134, 557–598.DOI

Paper details

vlnc:W_665nmdtc
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents