WorksMathematics
Studentized inference in randomized experiments when a few units dominate the outcome variation
Abstract
In completely randomized experiments with heavy-tailed outcomes, a few units can carry a share of the residual variation that does not vanish as the number of units grows. We study the studentized difference in means and Lin's regression-adjusted estimator, with covariates whose leverage vanishes, when the finite population is fixed, and the treatment assignment is the only source of randomness. The studentized statistic then has a nonnormal limit in general. Populations with the same marginal distributions of potential outcomes can have different limits, as the limit depends on the unobserved pairing mechanism of the two potential outcomes of each dominant unit. When the outcome errors are drawn from a heavy-tailed law, the limit is random and, outside a degenerate case, varies with the population. Under balanced allocation, the conventional Wald interval has an explicit asymptotic coverage floor. At the nominal level , the floor is about for every population and about when the two potential residuals of each unit are equal. Both floors are close to the best possible, and without balanced allocation, the coverage can fall to the smaller of the proportions of treated and control units.
Introduction
Randomized experiments are commonly analyzed with the difference in means or with a regression-adjusted estimator, together with a Wald interval based on a Neyman-type variance estimator. In the design-based framework, the covariates and potential outcomes are fixed, the treatment assignment is the only source of randomness, and the interval is justified by a finite-population central limit theorem [24] (Li and Ding 2017). Such theorems require that no unit carry a share of the outcome variation that stays bounded away from zero as the number of units grows, a condition known as negligibility [26, 13, 17]. In practice, however, outcomes such as revenue, spending, and charges are heavy-tailed, and a few units can carry much of the variation. As Young [32] noted, in published experiments, deleting a single observation can make about a third of the results that are significant at the level 0.01 insignificant. For example, in the RAND Health Insurance Experiment studied in § 6, one person carries about a third of the residual variation of second-year medical spending.
Without negligibility, sums over a randomly selected subset of a finite population can have nonnormal limits with discrete components [28, 21, 20, 8], and permuted sums of heavy-tailed samples have limits that depend on the sample [6]. For independent observations without finite variance, the limits of self-normalized sums are known [25]. Student’s statistic is asymptotically normal if and only if the law of the observations is in the domain of attraction of the normal law [16], and under symmetry, it has explicit tail bounds [12, 11, 27]. The Berry–Esseen bound of Bloznelis [9] for the finite-population Student statistic implies asymptotic normality under the condition of Erdős and Rényi [13], and the theory of studentized permutation and randomization tests [19, 10] (Wu and Ding 2021) establishes validity when the studentized statistic is asymptotically normal. Methods designed for heavy-tailed outcomes change either the estimator or the estimand [4, 29, 15]. None of the above results gives the randomization law of the conventional studentized statistic when a few units dominate the residual variation. The limits for random subsets concern sums of one value per unit, whereas the studentized statistic is a ratio whose numerator and denominator depend on both potential outcomes of the dominant units.
This article studies the studentized difference in means and the interacted estimator of Lin [24], with covariates whose leverage vanishes, when the residual variation is not negligible. The studentized statistic has a limit law determined by the limits of the normalized residual pairs of the dominant units and of the second-moment matrix of the remaining units, provided the limit of the normalized variance estimator is positive with probability one (Theorem 3.2). The law is nonnormal in general. It depends on how the two potential residuals of each dominant unit are paired, not only on their marginal distributions, so populations with the same marginal distributions of potential outcomes can have different limiting coverage (Proposition 3.5). When the outcome errors are drawn from a law with tail index between one and two, the randomization law of the statistic for the population at hand converges to a random law determined by the largest residual pairs of the realized population, and this law varies with the population unless the spectral measure of the errors is concentrated on a line determined by the allocation (Proposition 3.4). Under balanced allocation, the Wald interval has an explicit asymptotic coverage floor for every population, about 0.42 at the nominal level 0.95 (Proposition 4.1), and a higher floor, about 0.84, when the two potential residuals of each unit are equal (Theorem 4.2). Both floors are close to the best possible, and neither holds at the nominal level 0.95 when 40% of the units are treated. Without balanced allocation, the coverage can fall to the smaller of the proportions of treated and control units (Proposition 4.3). The floors are randomization analogs of the tail bounds for Student’s statistic under symmetry, with the balance of the design in place of symmetric errors.
Simulations with heavy-tailed errors (§ 5) and a data application (§ 6) show how closely the limits describe populations of moderate size and how the coverage of the Wald interval behaves when a few units dominate.
Setup
Finite populations and the design
For each , let be a finite population of covariates and potential outcomes . These quantities are fixed but may change with . We suppress their dependence on in the notation. The dimension is fixed, and means that there are no covariates. The estimand is the sample average treatment effect,
Assumption 1 (Complete randomization). For each , , and the vector of treatment indicators is uniformly distributed over all binary vectors of length with entries equal to one. Also, , and .
Projections and leverage
Let for . If , let and , which we assume to be nonsingular for every , and define the finite-population projection coefficients and residuals by
If , let . No outcome model is assumed. By construction,
where the second identity applies if . The leverage of unit is if , and if . Then, and . Write for the residual pair of unit , and let
The quantity is the total residual variation, and combines the two potential residuals of unit .
The estimator and the Wald interval
For , let and denote the averages of the covariates and of the observed outcomes over . If , define the within-arm Gram matrix and slope by
with if is singular. The estimator is
if , and the difference in means if . If and are nonsingular, is the coefficient of in the least squares regression of on , the interacted estimator of Lin [24]. Suppose . For , let if and if . Define
If , is the Neyman variance estimator. If , we set when , and or according to the sign of otherwise, so that is the event that covers , for . The Wald interval of nominal level is , where is the standard normal distribution function.
Exact decompositions
Define the residual contrast and, if , the remainder from estimating the slopes by
and let if . Let . If , let . If , let and
on the event that is nonsingular, and otherwise.
Proposition 2.1. Suppose Assumption 1 holds. For every assignment,
and and . If and either or and are nonsingular, then and
Section S2 proves Proposition 2.1. The estimation error splits into the residual contrast , which is the difference in means computed from the population residuals, and the remainder from estimating the slopes. In each arm, the variance estimator sums the squared population residuals of the observed units, minus , the part of that sum explained by regressing these residuals on an intercept and the covariates within arm . Both and the leading term of (10) depend on the population only through the residual pairs .
Condition 1 (Vanishing leverage). The leverages satisfy .
Condition 1 holds if . If , it holds if and , because . It restricts the covariates only.
Lemma 2.2. Suppose Assumption 1 and Condition 1 hold. Then, if , the probability that or is singular tends to zero. For every , if for every , then and in probability, for .
Section S2.3 proves Lemma 2.2 with a bound for each . Under Condition 1, estimating the slopes changes the numerator and the denominator of only at a smaller order than the residual scale, regardless of the outcomes. The limits below are therefore determined by the residual pairs .
The limit of the studentized statistic
Residual configurations
When a few units carry a share of that does not vanish, the numerator and the denominator of are dominated by the same units. For , let , so that by (2) and .
Definition 3.1. Let , and for let be vectors in such that is finite. Let be a finite or countably infinite family of nonzero vectors in such that is finite for every , let be a positive semidefinite matrix, and let . The array has limit configuration if (i) and (ii) hold. (i) For every different from all and all large , has the same cardinality as , and there are enumerations of the two
sets, and , with .
(ii) The limit , the limit in being taken along values different from all .
Condition 2 (Residual configuration). For every , , and the array has a limit configuration with , where .
The points are the limits of the normalized residual pairs of the units that keep a share of that does not vanish, and is the limit of the second-moment matrix of the remaining units. If is empty, , which is the negligibility condition of Noether [26] for the residual pairs. Let and be the diagonal entries of corresponding to and , let , let be independent Bernoulli variables, and let be independent of . Define
Theorem 3.2. Suppose Assumption 1 and Conditions 1 and 2 hold. Then,
Also, if and only if , or is infinite, or some has both coordinates nonzero. In that case, .
Section S4.1 proves Theorem 3.2. If is empty, is normal with mean zero and variance , and is the constant . The constant exceeds the variance of by . Thus, has a normal limit with variance at most one, and the Wald interval is asymptotically conservative, as in Lin [24] and Li and Ding (2017). If is not empty and , the limit of is nonnormal in general, and it depends on as a family of points in , not only on its two coordinate projections. A unit whose treatment residual is large but whose control residual is not enters the numerator through in both arms, because contains both of its potential outcomes, but it enters the denominator only if it is treated. If for all and , then for real , and
where are independent Rademacher signs, is independent of them, and . The right-hand side of (12) is a sum of Rademacher signs with fixed weights plus an independent normal variable, and its variance is one. It is not normal if is not empty, because its fourth cumulant is . Without equal residuals, the limit can be normal when is not empty. If every is proportional to and , then the points cancel from and contribute only a constant to .
Heavy-tailed outcome errors
Configurations with infinitely many points arise when outcome errors are drawn from a heavy-tailed law. Let be the unit circle in .
Definition 3.3. Let . A random vector is regularly varying with index and spectral measure , a Borel probability measure on , if is regularly varying at infinity with index and converges weakly to as .
Let be the arrival times of a unit-rate Poisson process, let be independent with law and independent of , and define the random family by
Let and be defined by (11) with , , and independent of .
Proposition 3.4. Let for be independent and identically distributed, where , and let consist of the first units, with for measurable functions such that , where is a constant if . Suppose that is independent of , has mean zero, and is regularly varying with index and spectral measure . If , suppose also that and that the covariance matrix of is nonsingular. Let the assignment satisfy Assumption 1 and be independent
of the population. On the event that is singular, let and , and let ratios with denominator be zero.
(a) The share converges in distribution to the law of , which is concentrated on . If , also and in probability.
(b) , and for every continuous function on ,
(c) The random variable is almost surely constant for every continuous function on if and only if is concentrated on the line .
Section S4.2 proves Proposition 3.4, using the limits of the ranked norms and directions in LePage et al. [22]. In (14), is the randomization expectation with the population held fixed, the population being drawn once. By part (a), the largest residual pair keeps a share of that does not vanish, so the negligibility condition fails. By part (b), the randomization law of for the population at hand converges, in the sense of (14), to the law of Theorem 3.2 with , whose points are the normalized points (13) of a Poisson process. This limit is a random law. By part (c), it varies with the population unless is concentrated on the line . In that case, every point lies on this line, the points cancel from , and the limit is the point mass at zero. The limits of the bootstrap of the mean with infinite variance (Athreya, [5]) and of permuted sums of heavy-tailed samples (Aue et al., [6]; Berkes et al. [8]) also depend on the sample. The functions allow conditional means that are nonlinear in , and their projection residuals enter only at the order , which is smaller than the order of the heavy-tailed errors.
Pairing of the potential outcomes
Each unit reveals one potential outcome, so the pairing of with is unobserved, and the variance of the unit-level effects is thus unidentified (Aronow et al., [3]). The next proposition shows that this pairing can change the limit of .
Proposition 3.5. Let , suppose Assumption 1 holds with along even , and let satisfy , and let . In population I, let , and for . In population II, let , , , and for . Then, both populations have and the same multisets and , but, along even ,
where and are the randomization distributions for the two populations.
Section S4.4 proves Proposition 3.5. In the regular case, the pairing affects only , the limit of . The limit of the normalized variance estimator exceeds the variance of the limit of the normalized estimator by this amount, so the Wald interval is conservative. With a residual configuration, it can instead change the shape of the limit law. The two populations of Proposition 3.5 have different limiting coverage at every critical value , and the ordering of the two coverages reverses at . Population I has the single point and . Population II has the points and and , so in Theorem 3.2.
Coverage under balanced allocation
The limits of Theorem 3.2 depend on how is paired with at the dominant units, which cannot be identified from the observed data. Under balanced allocation, that is, under Assumption 1 with , the coverage of the Wald interval nevertheless admits explicit lower bounds. Let .
Proposition 4.1. Suppose Assumption 1 holds with , and Condition 1 holds. Then, for every ,
In particular, the Wald interval of nominal level has asymptotic coverage at least , which is approximately 0.421 at .
Section S5.2 proves Proposition 4.1. The bound requires no condition on the outcomes. When , the assignments and are equally likely. Thus, when , changing the second assignment at a vanishing fraction of the units gives an assignment with the law of the first. At the two assignments, the numerators of have the same absolute value to first order, and the sums of the squared population residuals over the observed arms add up to to first order. For one of the two assignments, the variance estimate is therefore at least about , and the interval covers unless a weighted sum of the signs , with squared weights summing to at most one, is large. Up to a term that vanishes in probability, this sum is a sum of independent Rademacher signs with the same weights, and is the sharp constant of the Gaussian comparison for such sums of Bentkus and Dzindzalieta [7].
A higher floor holds if the two potential residuals of the units are matched in aggregate. Let .
Condition 3 (Aggregate matching). There is such that
where the ratio is if .
If for all , Condition 3 holds with . If with for some , it holds with (Section S5.3). Geometrically, plot each unit as the point . Under balanced allocation, each unit enters the numerator of through , which is proportional to the projection of its point onto the diagonal, with a sign given by its arm. It enters the variance estimator through the coordinate of its arm, whose square is at least , the squared distance from the point to the nearer coordinate axis. Condition 3 prevents the residual mass from concentrating, in aggregate, near the axes relative to its projection onto the diagonal. Mass near the line enters the variance estimator but contributes little to the numerator, and it increases the ratio in Condition 3.
Theorem 4.2. Suppose Assumption 1 holds with , and Conditions 1 and 3 hold.
Then, for every ,
In particular, if for all , the Wald interval of nominal level has asymptotic coverage at least .
Section S5.3 proves Theorem 4.2. Theorem 4.2 requires no configuration condition on the residuals. For , the floor is approximately 0.841 at and 0.968 at . For , which corresponds to pointwise matching with , it is approximately 0.392 at . At , the floor of Theorem 4.2 exceeds that of Proposition 4.1 only if . The proof replaces complete randomization by independent signs through a coupling of the type of Hájek [17]. The magnitudes are then fixed, as the absolute values of the observations are in the analysis of Student’s statistic under symmetry (Efron [12]), and the bound has the form of the tail bound of Pinelis [27] for that statistic under symmetry, with the constant of Bentkus and Dzindzalieta [7].
For and , let
where is the limit in Assumption 1. The next proposition shows that both floors are close to the best possible and that neither holds at the allocation and the nominal level 0.95.
Proposition 4.3. Let , suppose Assumption 1 holds, let be fixed, let satisfy , and let satisfy .
(a) If for and for , then for every outside the finite set ,
For and , for every . The limit is at and approximately 0.589 at .
(b) Let for . If and , then for every and for every . If and
, the same holds with and interchanged.
(c) Let , , , , and for . Then, Condition 3 holds with , and for , .
Section S5.4 proves Proposition 4.3. In part (a), Condition 3 holds with , and , whereas for . For and , the assignments that place all four dominant units in one arm are therefore not covered, and all other assignments are. At , the limiting coverage is , so no floor valid for every population with exceeds at , and the floor 0.841 of Theorem 4.2 is close to sharp. The probability of these assignments tends to one as , and thus aggregate matching without balanced allocation gives no floor that holds for every . With and , , so the limiting coverage is for , whereas the floor of Theorem 4.2 with tends to as . The constant in (16) therefore cannot be lowered.
In part (b), the single dominant unit lies on a coordinate axis, so the ratio in Condition 3 tends to zero. If that unit is treated, in probability and the interval covers. If it is assigned to control, its observed outcome is of the size of the outcomes of the remaining units, so , whereas , and . At , no floor valid for every population under balanced allocation exceeds , and the universal floor 0.421 of Proposition 4.1 is within a factor of 1.19 of this bound. For general , the two populations of part (b) have limiting coverage and , so without balanced allocation the coverage can be as low as . In part (c), only the assignments that place both dominant units in control are not covered. The floor of Theorem 4.2 at is approximately 0.747, above the limiting coverage 0.64, and for the first population of part (b) at the limiting coverage 0.4 is below the floor 0.421 of Proposition 4.1. Hence neither floor holds at the allocation and the nominal level 0.95.
Numerical studies
Finite populations and the limit laws
This study checks how closely the limits of Theorem 3.2 and Propositions 3.5 and 4.3 describe finite populations of moderate size. The populations of Propositions 3.5 and 4.3 have , and , with , in Proposition 4.3(a), and in Proposition 4.3. For Theorem 3.2, the population has , , one standard normal covariate, and potential outcomes and . The error pairs are bivariate normal with unit variances and correlation 0.5, except for three units, whose error pairs are , and . Its limit law is evaluated at its own three largest normalized residual pairs and the second-moment matrix of the remaining pairs, from draws. For each population, the study draws 100,000 assignments and records the coverage of for . Table 1 reports these coverages next to their limits.
| population | share | cov. | limit | cov. | limit | |
| Proposition 3.5, population I | 0.5 | 1.000 | 0.000 | 0.000 | 1.000 | 1.000 |
| Proposition 3.5, population II | 0.5 | 0.500 | 0.301 | 0.299 | 0.987 | 0.988 |
| Proposition 4.3(a), | 0.5 | 0.250 | 0.372 | 0.375 | 0.875 | 0.875 |
| Proposition 4.3(a), | 0.2 | 0.250 | 0.407 | 0.410 | 0.588 | 0.589 |
| Proposition 4.3(b) | 0.5 | 1.000 | 0.000 | 0.000 | 0.502 | 0.500 |
| Proposition 4.3(b) | 0.2 | 1.000 | 0.000 | 0.000 | 0.198 | 0.200 |
| Theorem 3.2, | 0.3 | 0.309 | 0.203 | 0.204 | 0.998 | 0.998 |
Table 1. Coverage of over 100,000 assignments in populations of size , and its limit. The columns give the allocation , the largest share (share), and, for and , the coverage over assignments (cov.) and the limit of the cited result (limit). The largest Monte Carlo standard error is 0.0016.
The results match Theorem 3.2 and Propositions 3.5 and 4.3. Every coverage agrees with its limit within two Monte Carlo standard errors, including the limits zero and one. The two populations of Proposition 3.5 have the same marginal distributions of potential outcomes, yet their coverages at both critical values differ as the limits predict. In the population of Proposition 4.3(b), the coverage of the nominal 95% interval is close to the allocation .
Coverage with heavy-tailed errors
This study examines the coverage of the nominal 95% Wald interval when the errors are heavy-tailed. Each population has units with three independent standard normal covariates, and . A symmetric Pareto variable with index is generated as , where for and is an independent random sign. The errors are symmetric Pareto with index and of three kinds: (i) additive errors that have ; (ii) matched errors that have , with uniform on ; and (iii) independent errors that have and independent. The allocation is . Each setting has 200 populations, and each population has 1,000 assignments, analyzed with the interacted estimator on the three covariates and with the difference in means. For each population, the study computes , , which is the ratio of Condition 3 at , the floor of Theorem 4.2, and the coverage. Table 2 reports summaries of these quantities over the populations of each setting.
| errors | floor | share | mean | min | below | mean | min | below | |||
| 1.2 | additive | 0.5 | 0.500 | 0.841 | 0.516 | 0.973 | 0.933 | 0 | 0.972 | 0.932 | 0 |
| 1.2 | additive | 0.2 | 0.500 | – | 0.466 | 0.972 | 0.907 | – | 0.972 | 0.914 | – |
| 1.2 | matched | 0.5 | 0.414 | 0.668 | 0.508 | 0.967 | 0.934 | 0 | 0.967 | 0.933 | 0 |
| 1.2 | matched | 0.2 | 0.415 | – | 0.515 | 0.960 | 0.685 | – | 0.960 | 0.690 | – |
| 1.2 | independent | 0.5 | 0.040 | 0.511 | 0.886 | 0.483 | 0 | 0.886 | 0.483 | 0 | |
| 1.2 | independent | 0.2 | 0.045 | – | 0.490 | 0.879 | 0.180 | – | 0.880 | 0.180 | – |
| 1.5 | additive | 0.5 | 0.500 | 0.841 | 0.378 | 0.964 | 0.938 | 0 | 0.963 | 0.941 | 0 |
| 1.5 | additive | 0.2 | 0.500 | – | 0.356 | 0.963 | 0.915 | – | 0.962 | 0.929 | – |
| 1.5 | matched | 0.5 | 0.418 | 0.678 | 0.350 | 0.964 | 0.937 | 0 | 0.964 | 0.938 | 0 |
| 1.5 | matched | 0.2 | 0.413 | – | 0.329 | 0.956 | 0.839 | – | 0.957 | 0.845 | – |
| 1.5 | independent | 0.5 | 0.126 | 0.342 | 0.954 | 0.484 | 0 | 0.955 | 0.486 | 0 | |
| 1.5 | independent | 0.2 | 0.134 | – | 0.288 | 0.933 | 0.202 | – | 0.934 | 0.202 | – |
| 1.8 | additive | 0.5 | 0.500 | 0.841 | 0.198 | 0.955 | 0.934 | 0 | 0.957 | 0.932 | 0 |
| 1.8 | additive | 0.2 | 0.500 | – | 0.209 | 0.956 | 0.922 | – | 0.956 | 0.930 | – |
| 1.8 | matched | 0.5 | 0.412 | 0.663 | 0.199 | 0.958 | 0.940 | 0 | 0.959 | 0.938 | 0 |
| 1.8 | matched | 0.2 | 0.413 | – | 0.251 | 0.955 | 0.867 | – | 0.956 | 0.895 | – |
| 1.8 | independent | 0.5 | 0.235 | 0.187 | 0.970 | 0.484 | 0 | 0.969 | 0.491 | 0 | |
| 1.8 | independent | 0.2 | 0.232 | – | 0.214 | 0.974 | 0.531 | – | 0.969 | 0.691 | – |
Table 2. Coverage of nominal 95% Wald intervals with symmetric Pareto errors of index , , and 200 populations of 1,000 assignments each, for the interacted estimator on three normal covariates () and the difference in means (). The columns give, for the interacted estimator, the median of over populations, the floor of Theorem 4.2 at this median and the median of (share), and for each estimator the mean and the minimum of the coverage and the number of populations whose coverage is more than two Monte Carlo standard errors below the larger of their own floor of Theorem 4.2 and the floor 0.421 of Proposition 4.1 (below). The Monte Carlo standard error at coverage is , evaluated at that larger floor. No floor applies for (–). The largest leverage in any population is 0.0097.
The results match Propositions 4.1 and 4.3 and Theorem 4.2. Under balanced allocation, no population has coverage more than two Monte Carlo standard errors below the larger of its floor of Theorem 4.2 and the floor 0.421 of Proposition 4.1. The largest residual pair carries a large share of , and this share decreases as grows. With additive and matched errors, the mean coverage is at or above 0.95, and the lowest coverage stays close to it under balanced allocation. With independent errors, the median of is small and the floor of Theorem 4.2 at this median is not positive. Specifically, independent heavy-tailed errors place the dominant residual pairs near the coordinate axes, so falls below 0.257, where the floor of Theorem 4.2 is negative and only the floor 0.421 of Proposition 4.1 is informative. The mean coverage is below 0.95 for , and for at . The lowest coverage under balanced allocation is then close to , above the floor of Proposition 4.1, and the populations with the lowest coverage have one residual pair that carries almost all of , as in Proposition 4.3(b). At , no floor applies, and with independent errors and the lowest coverage falls close to . The interacted estimator and the difference in means have nearly the same mean coverage in every setting, because the covariates contribute to only at the order , which is smaller than the order of the heavy-tailed errors (§ 3.2).
Data application
This section applies the results to data from three randomized experiments whose outcomes, medical spending, emergency-department charges, and customer spending, are heavy-tailed. The coverage of an interval cannot be evaluated from the experiment itself, because only one assignment is observed and is unknown. We therefore impute the potential outcomes of each experiment under a sharp model of the effect and draw new assignments by complete randomization. The coverage is then the proportion of assignments whose interval contains .
The first population comes from the RAND Health Insurance Experiment [2]. We use the person-year file distributed with Angrist and Pischke [1] and keep the 5,477 persons observed in both of their first two years in the experiment. The treated arm consists of the 1,843 persons assigned to free care, and the control arm consists of those assigned to a plan with cost sharing. The outcome is the person’s total medical spending in the second year, and is the total in the first year. The second population comes from the Oregon Health Insurance Experiment [14, 30]. We use the public-use emergency-department file for the Portland area and keep the 19,619 lottery participants who signed up alone and have both charge totals recorded. The treated arm consists of the 6,750 persons selected in the lottery. The outcome is the sum of the charges for the person’s emergency-department visits from 10 March 2008 to 30 September 2009, and is the same sum from 1 January 2007 to 9 March 2008, before the first lottery notification. The third population comes from the e-mail experiment of Hillstrom [18], in which 64,000 customers of a retailer were assigned at random to a men’s-merchandise e-mail, a women’s-merchandise e-mail or no e-mail. We keep the 42,613 customers assigned to the men’s e-mail, which forms the treated arm, or to no e-mail. The outcome is the customer’s spending in the two weeks after the e-mail, and is the spending in the preceding year.
The recorded outcome of each unit is its potential outcome under the arm to which it was assigned in the experiment, and the other potential outcome is imputed under one of two sharp models. The constant effect sets for every unit, with equal to the difference between the average recorded outcomes of the treated and control arms. It gives , so Condition 3 holds with . The proportional effect sets for every unit, with equal to the ratio of these averages. It gives , and since in the three experiments, Condition 3 holds with . On the completed population, we draw 100,000 assignments by complete randomization with and with , and compute the nominal 95% Wald interval for the difference in means () and for the interacted estimator. The interacted estimator uses the covariate , so . We do not use itself, because its leverage is about 0.19 for one person in the RAND population, outside the scope of Condition 1. Because the assignments are redrawn, the covariate need not precede the original assignment, as in the RAND population. Table 3 reports the coverage, together with the maximum leverage of , the largest share , the ratio of Condition 3 at defined in § 5.2, and the floor of Theorem 4.2 at .
| population | effect | share | share log | floor | log | log | ||||
| RAND | constant | 0.0014 | 0.329 | 0.338 | 0.500 | 0.841 | 0.954 | 0.954 | 0.937 | 0.937 |
| RAND | prop. | 0.0014 | 0.373 | 0.381 | 0.428 | 0.703 | 0.955 | 0.955 | 0.926 | 0.925 |
| Oregon | constant | 0.0003 | 0.016 | 0.016 | 0.500 | 0.841 | 0.951 | 0.951 | 0.947 | 0.947 |
| Oregon | prop. | 0.0003 | 0.016 | 0.016 | 0.492 | 0.829 | 0.951 | 0.951 | 0.947 | 0.947 |
| retail | constant | 0.0002 | 0.026 | 0.026 | 0.500 | 0.841 | 0.951 | 0.951 | 0.945 | 0.945 |
| retail | prop. | 0.0002 | 0.058 | 0.058 | 0.315 | 0.308 | 0.961 | 0.961 | 0.944 | 0.944 |
Table 3. Coverage of nominal 95% Wald intervals over 100,000 assignments in the RAND (), Oregon () and retail () populations under a constant or proportional (prop.) effect, for the difference in means () and for the interacted estimator on (log). The constant effect is 155.55, 133.89 and 0.77, and the ratio of the proportional effect is 1.34, 1.03 and 2.18. The number of treated units is 2,738, 9,810 and 21,306 at and 1,095, 3,924 and 8,523 at . All values are given for the three populations in the order listed. The columns give the maximum leverage of , the largest share for each estimator, , the floor of Theorem 4.2 at , which applies at , and the coverage. The largest Monte Carlo standard error is 0.0008.
In the RAND population, the largest residual pair carries a large share of for both estimators, whereas in the Oregon and retail populations, its share is small. The maximum leverage of is small in all three populations. Under balanced allocation, the coverage is above 0.95 in every population, well above the floors of Proposition 4.1 and Theorem 4.2. With , the floors do not apply, and by Proposition 4.3 they need not hold. The coverage in the RAND population then falls below 0.95, more under the proportional effect than under the constant effect, whereas in the Oregon and retail populations, it stays close to 0.95. The covariate changes the largest share only slightly, and the two estimators have nearly the same coverage in every setting. Thus, in these populations, the largest loss of coverage occurs under unbalanced allocation in the population in which a few units dominate the residual variation.
Discussion
We studied the studentized difference in means and the interacted estimator, with covariates whose leverage vanishes, when a few units carry a share of the residual variation that does not vanish. The studentized statistic then has a limit that is nonnormal in general and depends on how the two potential residuals of the dominant units are paired. When the errors are heavy-tailed, this limit is random and, outside a degenerate case, varies with the population. Under balanced allocation, the Wald interval keeps explicit coverage floors, which are close to the best possible, and neither floor extends to every allocation.
Natural next steps are the best universal floor under balanced allocation, which lies between 0.421 and the coverage of Proposition 4.3(b), a coverage floor for a fixed allocation , and an interval whose coverage is guaranteed for every population, including populations whose covariates have leverage that does not vanish.
Data and code availability
The RAND data are distributed with Angrist and Pischke [1], the Oregon data are the public-use files of the Oregon Health Insurance Experiment [14], and the retail data are those of Hillstrom [18].
Supplementary Material
Section S1 of the Supplementary Material states conventions and external results. Sections S2–S5 prove the results of § 2–§ 4. The Supplementary Material is provided as an ancillary file with this article.
References
- [1]Angrist, J. D. and Pischke, J.-S. (2015), Mastering ’Metrics: The Path from Cause to Effect, Princeton: Princeton University Press.
- [2]Aron-Dine, A., Einav, L., and Finkelstein, A. (2013), “The RAND Health Insurance Experiment, three decades later,” Journal of Economic Perspectives, 27, 197–222.
- [3]Aronow, P. M., Green, D. P., and Lee, D. K. K. (2014), “Sharp bounds on the variance in randomized experiments,” The Annals of Statistics, 42, 850–871.
- [4]Athey, S., Bickel, P. J., Chen, A., Imbens, G. W., and Pollmann, M. (2023), “Semi-parametric estimation of treatment effects in randomised experiments,” Journal of the Royal Statistical Society Series B: Statistical Methodology, 85, 1615–1638.DOI
- [5]Athreya, K. B. (1987), “Bootstrap of the mean in the infinite variance case,” The Annals of Statistics, 15, 724–731.
- [6]Aue, A., Berkes, I., and Horváth, L. (2008), “Selection from a stable box,” Bernoulli, 14, 125–139.
- [7]Bentkus, V. K. and Dzindzalieta, D. (2015), “A tight Gaussian bound for weighted sums of Rademacher random variables,” Bernoulli, 21, 1231–1237.DOI
- [8]Berkes, I., Horváth, L., and Schauer, J. (2010), “Non-central limit theorems for random selections,” Probability Theory and Related Fields, 147, 449–479.DOI
- [9]Bloznelis, M. (1999), “A Berry–Esseen bound for finite population Student’s statistic,” The Annals of Probability, 27, 2089–2108.DOI
- [10]Chung, E. and Romano, J. P. (2013), “Exact and asymptotically robust permutation tests,” The Annals of Statistics, 41, 484–507.DOI
- [11]Edelman, D. (1990), “An inequality of optimal order for the tail probabilities of the T statistic under symmetry,” Journal of the American Statistical Association, 85, 120–122.DOI
- [12]Efron, B. (1969), “Student’s t-test under symmetry conditions,” Journal of the American Statistical Association, 64, 1278–1302.DOI
- [13]Erdős, P. and Rényi, A. (1959), “On the central limit theorem for samples from a finite population,” Publications of the Mathematical Institute of the Hungarian Academy of Sciences, 4, 49–61.
- [14]Finkelstein, A., Taubman, S., Wright, B., Bernstein, M., Gruber, J., Newhouse, J. P., Allen, H., and Baicker, K. (2012), “The Oregon Health Insurance Experiment: evidence from the first year,” The Quarterly Journal of Economics, 127, 1057–1106.
- [15]Ghosh, A., Deb, N., Karmakar, B., and Sen, B. (2026), “Robustness and efficiency of Rosenbaum’s rank-based estimator in randomized trials: a design-based perspective,” Biometrika, 113. Article asag028.arxiv.org/abs/2111.15524
- [16]Giné, E., Götze, F., and Mason, D. M. (1997), “When is the Student t-statistic asymptotically standard normal?” The Annals of Probability, 25, 1514–1531.
- [17]Hájek, J. (1960), “Limiting distributions in simple random sampling from a finite population,” Publications of the Mathematical Institute of the Hungarian Academy of Sciences, 5, 361–374.
- [18]Hillstrom, K. (2008), “The MineThatData e-mail analytics and data mining challenge,” Data set, MineThatData, https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html.
- [19]Janssen, A. (1997), “Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens–Fisher problem,” Statistics & Probability Letters, 36, 9–21.DOI
- [20]Jiřina, M. (1982), “Limit theorems for samples from a finite population,” Journal of the Australian Mathematical Society (Series A), 32, 318–327.DOI
- [21]Kallenberg, O. (1973), “Canonical representations and convergence criteria for processes with interchangeable increments,” Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 27, 23–36.DOI
- [22]LePage, R., Woodroofe, M., and Zinn, J. (1981), “Convergence to a stable distribution via order statistics,” The Annals of Probability, 9, 624–632.DOI
- [24]Lin, W. (2013), “Agnostic notes on regression adjustments to experimental data: reexamining Freedman’s critique,” The Annals of Applied Statistics, 7, 295–318.DOI
- [25]Logan, B. F., Mallows, C. L., Rice, S. O., and Shepp, L. A. (1973), “Limit distributions of self-normalized sums,” The Annals of Probability, 1, 788–809.DOI
- [26]Noether, G. E. (1949), “On a theorem by Wald and Wolfowitz,” The Annals of Mathematical Statistics, 20, 455–458.DOI
- [27]Pinelis, I. (1994), “Extremal probabilistic problems and Hotelling’s T² test under a symmetry condition,” The Annals of Statistics, 22, 357–368.
- [28]Rosén, B. (1965), “Limit theorems for sampling from finite populations,” Arkiv för Matematik, 5, 383–424.DOI
- [29]Su, Y. and Li, X. (2024), “Treatment effect quantiles in stratified randomized experiments and matched observational studies,” Biometrika, 111, 235–254.arxiv.org/abs/2208.11798
- [30]Taubman, S. L., Allen, H. L., Wright, B. J., Baicker, K., and Finkelstein, A. N. (2014), “Medicaid increases emergency-department use: evidence from Oregon’s Health Insurance Experiment,” Science, 343, 263–268.
- [32]Young, A. (2019), “Channeling Fisher: randomization tests and the statistical insignificance of seemingly significant experimental results,” The Quarterly Journal of Economics, 134, 557–598.DOI