betaFeedback

WorksMathematics

On the Validity of Fisher Randomization Tests under the Weak Null

Abstract

Consider the problem of testing the Neyman weak null of no sample average treatment effect H0N:τ=0H_{0N}:τ= 0 in randomized experiments with binary outcomes. Although the Fisher Randomization Test (FRT) is typically justified under Fisher's sharp null of no individual treatment effect, we show that it is finite-sample-valid for the weak null H0NH_{0N} in three common combinations of study design and test statistic: 1) balanced completely randomized experiments with the difference-in-means test statistic, 2) matched pairs experiments with the same test statistic, and 3) balanced Bernoulli randomized experiments with the centered Horvitz-Thompson test statistic. As an immediate corollary, Fisher's exact test in a balanced completely randomized experiment is valid under the weak null, and so is McNemar's test in a matched pairs experiment. Similar results hold for the one-sided null H0:τ≤0H_0:τ\leq 0. Beyond balanced designs, we show that twice the one-sided FRT pp-value with the same test statistics is valid under H0NH_{0N}. In simulation experiments, the FRTs compare favorably to other exact procedures.

Introduction

The Fisher randomization test (FRT) has been used to analyze the results of experiments for almost a century [Fis30, Fis35, Pit37]. The development of the FRT played a crucial role in establishing random allocation of treatments as a fundamental principle in experimental design. Using solely the randomness of this allocation, without other assumptions on the experimental population, the FRT provides a basis for inference on the effect of treatment. For a historical overview and philosophical discussion of the test, see for instance [Dav08, Din17, ZZ23, Ros26].

The potential outcomes framework [Ney23] gives an elegant framework to reason about the test. Consider an experiment with nn units, where unit ii’s potential outcomes under treatment and control conditions are given by Yi(1)Y_i(1) and Yi(0)Y_i(0) respectively, fixed and unknown a priori. The sample average treatment effect τ=1n∑i=1n(Yi(1)−Yi(0))\tau= \frac{1}{n}\sum_{i=1}^{n}(Y_i(1) - Y_i(0)) is the parameter of interest. Upon administering a randomized treatment AiA_i to unit ii – where Ai=1A_i = 1 denotes treatment and Ai=0A_i = 0 denotes control – and under SUTVA [Rub80], we observe only the outcome corresponding to the treatment, i.e., Yi=Yi(Ai)Y_i = Y_i(A_i), or equivalently, Yi=AiYi(1)+(1−Ai)Yi(0)Y_i = A_iY_i(1) + (1 - A_i)Y_i(0). The FRT proceeds by imputing unknown potential outcomes with the observed data assuming the sharp null hypothesis, rerandomizing treatment to obtain a reference distribution, and comparing the observed data to the reference distribution. A precise definition is given in § 2. As delineated by Welch [Wel37], the usual justification of the FRT comes from Fisher’s sharp null where no unit responds to the treatment:

H0F:Yi(1)=Yi(0), i=1,…,n.(1)H_{0F}: Y_i(1) = Y_i(0), \ i = 1, \ldots, n. \tag*{(1)}

On the other hand, the scientific question underlying the experiment often involves the Neyman null or weak null representing efficacy on average over the nn units:

H0N:τ=0,H≤0N:τ≤0.(2)H_{0N}: \tau= 0, \qquad H_{\leq0N}: \tau\leq0. \tag*{(2)}

The gap between these hypotheses is substantive. In many settings, we expect at least some units to respond to the treatment even in a null case, due to inevitable heterogeneity among units. Finding examples or classes of FRTs valid under H0N,H≤0NH_{0N}, H_{\leq0N} is therefore important. To date, this property has been established in a few settings. One line of work establishes validity under H0NH_{0N} asymptotically [Din17, WD21, CF22], often by considering studentized or pre-pivoted statistics. Caughey et al. [CDLM23] show the validity of the FRT for bounded null hypotheses of the form H0:Yi(1)−Yi(0)≤δiH_0 : Y_i(1) - Y_i(0) \leq\delta_i, for any prespecified constants δi∈R\delta_i \in\mathbb{R}. More generally, as Ding argues [Din25], few examples of tests which are finite-sample valid under H0NH_{0N} are known. The most notable test is due to Chiba and Rigdon-Hudgens [Chi15, Chi17, RH15], which takes a maximum of FRT pp-values for all potential outcome tables in H0N,H≤0NH_{0N}, H_{\leq0N}. Concentration inequalities also enable valid tests [Fre26, SBF+26], although these are often conservative and can suffer from low power.

This note advances the literature on FRTs and randomization inference by showing that the FRT in three common settings is exactly finite-sample-valid for the weak null. For balanced completely randomized experiments, we show that the one- and two-sided FRTs with the standard difference-in-means statistic are valid for H≤0N,H0NH_{\leq0N}, H_{0N}, respectively, for all even nn. For balanced Bernoulli randomized experiments, the FRT with the centered Horvitz-Thompson statistic [3, 2 Hul26, SDT26] is also valid in finite samples under H0N,H≤0NH_{0N}, H_{\leq0N}. The same guarantees are true of the FRT in a matched pairs experiment with difference-in-means, where treatment is allocated with a fair coin in every pair of units. As a byproduct, these results establish the weak null validity of two very classical tests: Fisher’s exact test in a completely randomized experiment (CRE) and McNemar’s test in a matched pairs experiment. Relative to other exact randomization inference procedures [RH15, Chi17], running a single FRT may also be computationally more efficient. In the setting of an arbitrary unbalanced or balanced experiment with the estimators as above, we establish a simple rule of thumb that twice the one-sided FRT pp-value is valid under the weak null. We show in simulation that these three FRTs compare favorably in their respective experimental settings to tests based on concentration inequalities and the Chiba-Rigdon-Hudgens procedure.

Main Results

Before presenting our results, we collect some notation and recall the precise definitions of FRTs. The set of binary potential outcomes in the experiment {(Yi(1),Yi(0))}i=1n\{(Y_i(1),Y_i(0))\}_{i=1}^{n} will be called a table. Given a table, define nab=#{i:Yi(1)=a,Yi(0)=b}n_{ab}=\#\{i:Y_i(1)=a,Y_i(0)=b\} for a,b∈{0,1}a,b\in\{0,1\} to be the sizes of the principal strata [FR02]. Given a treatment vector AobsA^{\mathrm{obs}} drawn from some law A\mathcal{A} called the design, we observe data Yi(Aiobs)=AiobsYi(1)+(1−Aiobs)Yi(0)Y_i(A_i^{\mathrm{obs}})=A_i^{\mathrm{obs}}Y_i(1)+(1-A_i^{\mathrm{obs}})Y_i(0), i=1,…,ni=1,\ldots,n. We will refer to data points Yi=1Y_i=1 as events or ones. We consider designs where P(Ai=1)\mathbb{P}(A_i=1), the treatment probability, is constant for all ii. Call the design balanced if the treatment probability is 1/21/2 and unbalanced otherwise. Finally, call the design symmetric if A=(d)1n−AA\mathrel{\overset{(d)}{=}}\mathbf{1}_n-A, where 1n\mathbf{1}_n is the nn-dimensional vector of ones.

Let τ^:{0,1}n×{0,1}n→R\hat{\tau}:\{0,1\}^{n}\times\{0,1\}^{n}\to\mathbb{R} be any statistic of the treatment vector and data. The Fisher randomization test generates pp-values for the Fisher null H0FH_{0F} as follows. Given the observed data Yobs:=Y(Aobs)Y^{\mathrm{obs}}:=Y(A^{\mathrm{obs}}), impute the potential outcome table by assuming H0FH_{0F}, i.e., each unit’s unobserved outcome equals its observed outcome. Sample a new treatment vector Anew∼AA^{\mathrm{new}}\sim\mathcal{A} and set τ^new:=τ^(Anew,Y(Aobs))\hat{\tau}^{\mathrm{new}}:=\hat{\tau}(A^{\mathrm{new}},Y(A^{\mathrm{obs}})), τ^obs:=τ^(Aobs,Yobs)\hat{\tau}^{\mathrm{obs}}:=\hat{\tau}(A^{\mathrm{obs}},Y^{\mathrm{obs}}). Then the one-sided and two-sided FRT pp-values are defined as

pF+(Aobs):=PAnew∼A(τ^new≥τ^obs),(3)p_F^+(A^{\mathrm{obs}}):=\mathbb{P}_{A^{\mathrm{new}}\sim\mathcal{A}}\left(\hat{\tau}^{\mathrm{new}}\geq\hat{\tau}^{\mathrm{obs}}\right), \tag*{(3)}
pF±(Aobs):=PAnew∼A(∣τ^new∣≥∣τ^obs∣).(4)p_F^{\pm}(A^{\mathrm{obs}}):=\mathbb{P}_{A^{\mathrm{new}}\sim\mathcal{A}}\left(\left|\hat{\tau}^{\mathrm{new}}\right|\geq\left|\hat{\tau}^{\mathrm{obs}}\right|\right). \tag*{(4)}

We will often omit the dependence on AobsA^{\mathrm{obs}}. Any pp-value pp we discuss in the paper will be called valid for H0NH_{0N} at level-α\alpha if sup⁡H0NP(p≤α)≤α\sup_{H_{0N}}\mathbb{P}(p\leq\alpha)\leq\alpha. We adopt the same definition for H≤0NH_{\leq0N}. We will also call pp-values with this property Neyman valid, or say that they satisfy Neyman validity.

In this note, we take the test statistic τ^\hat{\tau} to be the centered Horvitz-Thompson statistic τ^cHT\hat{\tau}_{\mathrm{cHT}}. For experimental designs with treatment probability ρ\rho, the statistic is defined by

τ^cHT(A,Y)=∑i=1nAi(Yi−1/2)nρ−∑i=1n(1−Ai)(Yi−1/2)n(1−ρ).(5)\hat{\tau}_{\mathrm{cHT}}(A,Y)=\frac{\sum_{i=1}^{n}A_i(Y_i-1/2)}{n\rho}-\frac{\sum_{i=1}^{n}(1-A_i)(Y_i-1/2)}{n(1-\rho)}. \tag*{(5)}

The centered Horvitz-Thompson statistic is a special case of a class of estimators introduced in [3]. The estimator has recently been studied in several works [3, 2 Fre26, Hul26, SDT26] in both survey sampling and experimental settings. Although a strength of the FRT is that any test statistic may be accommodated, this choice is particularly natural and common: τ^cHT\hat{\tau}_{\mathrm{cHT}} is equivalent to difference-in-means in complete randomization and matched pairs, one of the most widely-used statistics for these experimental designs.

Completely Randomized Experiments. Our main results will show that the FRT based on τ^cHT\hat{\tau}_{\mathrm{cHT}} is Neyman valid for balanced complete randomization, Bernoulli randomization, and matched pairs. We begin with the balanced completely randomized experiment (CRE). In a balanced CRE, with nn even, n/2n/2 units are selected uniformly at random to be treated. Since τ^cHT\hat{\tau}_{\mathrm{cHT}} is equivalent to the difference in means estimator τ^DIM\hat{\tau}_{\mathrm{DIM}} in a CRE, we discuss them interchangeably. We have the following result.

Theorem 2.1. With a balanced CRE, pF+p_F^+ is valid under the weak null H≤0NH_{\leq0N} whenever 0<α≤1/20 < \alpha\leq1/2. The two-sided FRT pp-value pF±p_F^{\pm} is valid under H0NH_{0N}, for all α∈(0,1)\alpha\in(0,1).

The FRT with difference-in-means in a CRE is known to be equivalent to Fisher’s exact test for 2×22 \times2 tables [DD16], one of the oldest tests in statistics [Fis30]. It is defined as follows. Let OobsO^{\mathrm{obs}} be the total number of events observed in the data. Let OTobsO_T^{\mathrm{obs}} be the number of events observed in the treatment group. Under H0FH_{0F}, OTobsO_T^{\mathrm{obs}} has a hypergeometric distribution OTobs∼Hyper⁡(n,Oobs,n/2)O_T^{\mathrm{obs}} \sim\operatorname{Hyper}(n,O^{\mathrm{obs}},n/2). Then Fisher’s exact test generates a pp-value defined by P(Hyper⁡(n,Oobs,n/2)≥OTobs)\mathbb{P}(\operatorname{Hyper}(n,O^{\mathrm{obs}},n/2) \geq O_T^{\mathrm{obs}}). The two-sided case is analogous. Theorem 2.1 extends this validity to the weak null H0NH_{0N}.

Another approach to obtain exact tests for H0NH_{0N} based on randomization inference with CRE and τ^DIM\hat{\tau}_{\mathrm{DIM}} is provided by the procedures of Chiba and Rigdon-Hudgens [Chi15, Chi17, RH15]. Let Δ∈{−1,0,1}n\Delta\in\{-1,0,1\}^{n} be a vector of individual treatment effects compatible with the observed data, satisfying ∑iΔi=0\sum_i \Delta_i = 0. Let D\mathcal{D} be this set of vectors. Let pΔ±p_{\Delta}^{\pm} be the two-sided FRT pp-value assuming the null H0F,Δ:Yi(1)−Yi(0)=Δi,i=1,…,nH_{0F,\Delta}: Y_i(1)-Y_i(0)=\Delta_i, i=1,\ldots,n. Taking the worst-case over D\mathcal{D} yields a pp-value which is valid for H0NH_{0N}:

pCRH±:=sup⁡Δ∈DpΔ±.(6)p_{\mathrm{CRH}}^{\pm} := \sup_{\Delta\in\mathcal{D}} p_{\Delta}^{\pm}. \tag*{(6)}

Throughout this note, we will refer to this test as the Chiba-Rigdon-Hudgens procedure. A similar procedure can be obtained for H≤0NH_{\leq0N} and with the one-sided FRTs. These approaches require the computation of multiple FRTs, which can be prohibitive in some cases. For the balanced CRE, significant progress has been made. Algorithmic advances [LD16, ACL25] show O(nlog⁡n)O(n \log n) FRTs suffice to compute an exact confidence interval for τ\tau. Specializing to testing a single null value τ=0\tau=0, the results of [1] show that O(n)O(n) many FRT procedures suffice. Theorem 2.1 shows that, in fact, one FRT is enough to obtain a valid pp-value. Moreover, the two-sided pp-value pF±p_F^{\pm} is at least as powerful as (6) by construction and strictly more powerful for some alternatives, as simulation results show in Figure 1.

Figure 1. Results on CRE for two-sided tests when α=0.05\alpha= 0.05. Power is computed using 200,000 reassignments of the CRE; p0p_0 denotes the fraction of units with Yi(0)=1Y_i(0) = 1 and τ\tau is the sample average treatment effect. pCRH±p_{\mathrm{CRH}}^{\pm} is computed using the speedups of [1].

Bernoulli Randomized Experiments. Next, we consider balanced Bernoulli randomized experiments (BRE) in which treatments are administered via fair coin flips: AiA_i, i=1,…,ni=1,\ldots,n are independent Bernoulli random variables with success probability 1/21/2. Due to operational constraints, BRE appears frequently as the default design in A/B testing and online experimentation [PASJS+19, KTX20]. Our second main result shows the validity of the FRT with this test statistic for Bernoulli randomized experiments.

Theorem 2.2. Under a BRE, the FRT pp-value pF+p_F^+ constructed using the centered Horvitz-Thompson statistic is valid for the weak null H≤0NH_{\leq0N} whenever 0<α≤1/20 < \alpha\leq1/2. For every α∈(0,1)\alpha\in(0,1), pF±p_F^{\pm} is valid for H0NH_{0N}.

Randomization inference for the BRE appears to be less developed than for the CRE and for matched pairs designs. The most relevant contributions are work by Branson-Bind for the sharp null [4] and recent work by Zhang [Zha26] on exact confidence intervals for τ\tau. The work by Zhang shows for the Horvitz-Thompson estimator, different than (5), the Chiba-Rigdon-Hudgens procedure pCRH+p_{\mathrm{CRH}}^+ of (6) can be computed using at most two FRTs.

Matched Pairs Experiments. Lastly, consider the setting of a matched pairs experiment. We consider a pool of nn units, which are grouped into pairs p=1,…,n/2p=1,\ldots,n/2. We will index potential outcomes by (Ypi(1),Ypi(0))(Y_{pi}(1),Y_{pi}(0)), p=1,…,n/2p=1,\ldots,n/2, i=0,1i=0,1. In a matched pairs experiment, treatment is allocated independently at random in each pair, with exactly one unit receiving treatment. With ApA_p drawn i.i.d. Ber⁡(1/2)\operatorname{Ber}(1/2), we observe data

Yp1=Yp1(Ap),Yp0=Yp0(1−Ap),p=1,…,n/2.Y_{p1}=Y_{p1}(A_p), \qquad Y_{p0}=Y_{p0}(1-A_p), \qquad p=1,\ldots,n/2.

The SATE τ\tau is often estimated using the difference-in-means estimator τ^(A,Y)≔2n∑p=1n/2(2Ap−1)(Yp1−Yp0)\hat{\tau}(A,Y) \coloneqq\frac{2}{n}\sum_{p=1}^{n/2}(2A_p-1)(Y_{p1}-Y_{p0}), which is equal to the centered Horvitz-Thompson estimator (5). Our third result shows validity in this case.

Theorem 2.3. With a matched pairs experiment and the difference-in-means statistic, pF+p_F^+ is valid under the weak null H≤0NH_{\leq0N} whenever 0<α≤1/20<\alpha\leq1/2. The two-sided pp-value pF±p_F^{\pm} is valid under H0NH_{0N}, for all α∈(0,1)\alpha\in(0,1).

It turns out that the FRT pp-values pF+p_F^+, pF±p_F^{\pm} are equal to the one- and two-sided pp-values obtained from the exact binomial version of McNemar’s test [Ros17, p. 90], [Din17]. McNemar’s test [McN47] is a classical procedure used to analyze paired data. In a matched pairs experiment, call a pair pp discordant if (Yp1,Yp0)(Y_{p1},Y_{p0}) is (1,0)(1,0) or (0,1)(0,1). Let XX be the total number of discordant pairs, and among these, let X+X_+ be the number of pairs in which the treated unit is 1. Under H0FH_{0F}, X+∣X∼Bin⁡(X,1/2)X_+\mid X\sim\operatorname{Bin}(X,1/2). The test rejects if X+X_+ is more extreme than the appropriate Bin⁡(X,1/2)\operatorname{Bin}(X,1/2) quantiles, so it is valid conditional on XX. Theorem 2.3 shows Neyman validity, unconditionally, of McNemar’s test. This appears to be a novel conclusion.

Several recent works consider exact inference for the matched pairs experiment. Li, Spertus, and Stark [LSS25] develop the analogs of two confidence interval procedures from [RH15] in general stratified experiments. This includes the inversion of compatible randomization test pp-values as in (6) as well as Bonferroni combinations of hypergeometric confidence intervals. For matched pairs designs, Zhang [Zha26] shows that computing Rigdon-Hudgens confidence intervals for the matched pairs design has time complexity O(nlog⁡2n)O(n\log^2 n). Specializing Zhang’s results to testing, (6) can be computed with at most two FRTs. By design, pF±,pF+p_F^{\pm},p_F^+ are at least as powerful as pCRH±,pCRH+p_{\mathrm{CRH}}^{\pm},p_{\mathrm{CRH}}^+. [Wil26] develops the analog of the Rigdon-Hudgens attributable effects method by giving analytic solutions to the worst cases in the component randomization tests. Finally, Fogarty [Fog23] also develops sensitivity analysis procedures for the weak null in matched designs.

Unbalanced Experiments. For complete and Bernoulli randomization, it is fairly common to run unbalanced experiments where the treatment probability is ρ≠1/2\rho\ne1/2. The proof techniques of Theorems 2.1, 2.2 carefully leverage the symmetry of the balanced case and do not extend naturally to the unbalanced case. Instead, we show that a simple, practically appealing fix works: multiply the FRT pp-value by 2.

Theorem 2.4. Suppose Aobs∼AA^{\mathrm{obs}} \sim\mathcal{A} is exchangeable. Then sup⁡H0N≤0P(pF+≤α)≤2α\sup_{H_{0N} \le0} \mathbb{P}(p_F^+ \le\alpha) \le2\alpha.

By the same argument, with p‾F\overline{p}_F denoting the analogous lower-tail FRT pp-value, 4min⁡(pF+,p‾F)4\min(p_F^+,\overline{p}_F) is a valid pp-value for H0NH_{0N}. The proof relies on making a perhaps unexpected connection between the analysis of FRTs and tournaments, analogously to the analysis of the Jackknife+ [5]. Since only the exchangeability of the experimental design is used, we expect that the constant 2 in the guarantee can be sharpened to a constant CρC_\rho depending on the treatment probability ρ\rho. Our previous results show Cρ≤1,ρ=1/2C_\rho\le1, \rho= 1/2, for their respective designs. We leave a sharp characterization of CρC_\rho to future work.

Simulation Results

For each experimental design, we compare the two-sided FRTs to other exact tests for H0NH_{0N} when α=0.05\alpha= 0.05. One important class consists of tests based on concentration inequalities, developed in recent work by Freidling and Sandoval et al. [Fre26, SBF+26], which covers τ^cHT\hat{\tau}_{c\mathrm{HT}} and every design in this note. Broadly, the results demonstrate that the FRTs are more powerful. For the CRE, we compare pF±p_F^{\pm} to the Chiba-Rigdon-Hudgens pp-value pCRH±p_{\mathrm{CRH}}^{\pm} using the computational speedups from [1] along with the Barber-Hoeffding and empirical Bennett intervals from [Fre26, Prop. 4.1, 4.4]. For the BRE, we compare to pCRH±p_{\mathrm{CRH}}^{\pm} with the Horvitz-Thompson estimator, using Zhang’s speedups [Zha26]. We also use tests based on the sub-Bernoulli and empirical Bennett intervals for the centered Horvitz-Thompson estimator from [Fre26, Prop. 3.5, 3.6], which builds on [SBF+26].

For each design, we consider three different simulation suites. The first is modeled after clinical trials for rare outcomes. We assume the treatment effect τ\tau is small, 0.05, and the control baseline p0p_0—fraction of units with Yi(0)=1Y_i(0) = 1—is small, contained in {0%,2.5%,5%}\{0\%, 2.5\%, 5\%\}. The fraction of units harmed by the treatment is half of p0p_0. The second suite considers a situation where control events are more common; p0p_0 is 30%, τ\tau varies in {0.1,0.2,0.3}\{0.1, 0.2, 0.3\}, and the fraction of units harmed is 20%. Finally, the third suite mimics the second suite, except we impose a monotonicity constraint that there are no units harmed. In each of these suites, we take nn to be multiples of 20 up to 300. If hh is the fraction of units harmed by the treatment, then the sizes of the principal strata (n11,n10,n01,n00)(n_{11}, n_{10}, n_{01}, n_{00}) may be expressed in terms of p0,h,τp_0,h,\tau as n(p0−h,τ+h,h,1−p0−τ−h)n(p_0-h,\tau+h,h,1-p_0-\tau-h).

Figures 1 and 2 show the power of the comparator procedures in each of these simulation settings, with α=0.05\alpha= 0.05. As all methods in the 3 experimental designs are exactly finite-sample-valid, we do not plot the Type I error. Broadly, the FRT methods outperform the other methods for exact inference, with the concentration inequality based tests being fairly conservative in this binary setting. For the CRE, pCRH±p_{\mathrm{CRH}}^{\pm} is indeed different from the FRT pp-value as can be seen in the first column of Figure 1, which highlights a stark gap between the two methods in the rare outcome setting. Outside of this case the two pp-values are quite similar. Interestingly, the performance of the FRT in panel (A1) is quite promising, suggesting some applications to trials with rare binary outcomes. In the BRE, the FRT with τ^cHT\hat{\tau}_{c\mathrm{HT}} continues to outperform the concentration inequality based tests along with the exact method of [Zha26]. The performance for rare outcomes is much more modest than that of the FRT in a CRE with difference in means.

For the matched pairs experiment, we consider the same three simulation suites and use a uniformly chosen pairing between units for each nn, which is fixed across all reassignments of the design. We compare with the exact RH-type procedure of [Zha26] and tests using the matched pair concentration inequalities in [Fre26, Prop. 3.6, 3.7]. Finally, we also compare to the method of Wilson [Wil26]. The results are pictured in Figure 3. The paired FRT behaves similarly to pCRH±p_{\mathrm{CRH}}^{\pm}, except in the rare outcomes setting where the paired FRT is much more powerful. Wilson’s method is also quite competitive, losing only to the paired FRT in the rare outcomes setting and beating the concentration-based procedures in all settings.

Figure 3. Results for matched pairs experiments for two-sided tests when α=0.05\alpha= 0.05. Power is computed using 200,000 reassignments of the matched pairs design. Matching between units is selected uniformly at random and fixed over the 200,000 runs, for each nn. Again, p0p_0 denotes the fraction of units with Yi(0)=1Y_i(0) = 1 and τ\tau is the sample average treatment effect.

Matched pairs designs are often motivated on the basis of blocking on observed information to reduce variation. In Figure 4, we detail an additional simulation setting where the pairs are determined by matching on a scalar covariate. For each unit in the experiment, we sample Xi∼Beta⁡(1/7,9/7)X_i \sim\operatorname{Beta}(1/7, 9/7) and generate Yi(0)∣Xi=x∼Ber⁡(x)Y_i(0) \mid X_i = x \sim\operatorname{Ber}(x). Independently conditional on XiX_i, Yi(1)∣Xi=xY_i(1) \mid X_i = x is generated as Ber⁡((1+17x)/18)\operatorname{Ber}((1+17x)/18). These constants are chosen so that the resulting superpopulation mimics the rare outcome simulation suite described previously, with p0=0.1p_0 = 0.1, τ=0.05\tau= 0.05 and the fraction of units harmed being 0.05. For each n=100,120,…,300n = 100, 120, \ldots, 300, we sample XiX_i from the Beta distribution, sort units by XiX_i, and pair adjacent ones. The potential outcomes are generated as above and (Yi(1),Yi(0),Xi)i=1n(Y_i(1),Y_i(0),X_i)_{i=1}^{n} are held fixed when reassigning the treatment indicators in the matched pairs design. The right panel of Figure 4 shows the unconditional power of the previously compared methods, averaged over 100 samples from the superpopulation. The FRT remains the most powerful test; the CRH procedure and Wilson’s attributable effects procedure behave similarly.

Figure 4. Results for matched pairs experiments, using two-sided tests, α=0.05\alpha= 0.05. Pairs are obtained by matching on an informative covariate XiX_i generated iid from Beta⁡(1/7,9/7)\operatorname{Beta}(1/7, 9/7). Potential outcomes are generated from XiX_i as described in the text. The left panel shows the density of XX, and the right panel shows the power of each method, averaged over 100 Monte Carlo redraws of the superpopulation. 95% pointwise confidence bands are displayed for the variation due to superpopulation draws.

Discussion

In this note, we showcase three common circumstances in which the Fisher randomization test is valid under the weak null, allowing users of these tests to significantly upgrade their inferential conclusions. Generalizing and unifying the proof techniques in these theorems may give insights into general conditions for exact finite-sample validity of FRTs. Proving Neyman validity beyond the binary case remains a challenge. In Example A.1, we show that the FRT can be anticonservative when the potential outcomes are non-binary. We raise the problem of finding powerful and tractable FRTs, or randomization tests more generally, for bounded outcomes as an important direction for future work.

Acknowledgments. AI assistance (GPT6) was used in the preparation of this work, including generating figures, suggesting arguments, checking proofs, and revising the article. All proofs were verified by the authors and we take responsibility for any errors.

Figure 2. Results for BRE for two-sided tests when α=0.05\alpha= 0.05. Power is computed using 200,000 reassignments of the BRE; p0p_0 denotes the fraction of units with Yi(0)=1Y_i(0) = 1 and τ\tau is the sample average treatment effect. The pp-values pCRH±p^{\pm}_{\mathrm{CRH}} values are computed using the speedups of Zhang [Zha26].

Proofs

In all of the proofs, we will freely use ϵ\epsilon, ϵ′\epsilon' and BB, B′B' to denote independent copies of Rademacher and Bernoulli(1/2)(1/2) random variables. Let SkS_k be the simple random walk at time kk, ∑i=1kϵi\sum_{i=1}^{k}\epsilon_i with ϵi\epsilon_i iid Rademacher random variables. For brevity, we will write τ^cHT\hat{\tau}_{\mathrm{cHT}} simply as τ^\hat{\tau}.

At a technical level, the approach for each design will be to show that the sampling distribution of the statistic—the law of τ^obs\hat{\tau}^{\mathrm{obs}}—is more concentrated than the reference distribution—the law of τ^(Anew,Y(Aobs))\hat{\tau}(A^{\mathrm{new}},Y(A^{\mathrm{obs}}))—for all realizations of AobsA^{\mathrm{obs}}. At a heuristic level, we should expect this to be true for the centered Horvitz-Thompson estimator on the following grounds. For a balanced design, a brief algebraic manipulation shows that τ^cHT\hat{\tau}_{\mathrm{cHT}} may be rewritten as

τ^cHT(A,Y)=τ+1n∑i=1n(2Ai−1)(Yi(1)+Yi(0)−1).(7)\hat{\tau}_{\mathrm{cHT}}(A,Y)=\tau+\frac{1}{n}\sum_{i=1}^{n}(2A_i-1)(Y_i(1)+Y_i(0)-1). \tag*{(7)}

Under H0NH_{0N} and a symmetric design, τ^cHT\hat{\tau}_{\mathrm{cHT}} is symmetrically distributed around zero. In particular, τ=0\tau=0 and the summands (2Ai−1)(Yi(1)+Yi(0)−1)(2A_i-1)(Y_i(1)+Y_i(0)-1) are symmetric around zero, taking values in {−1,0,1}\{-1,0,1\}. Notice that Yi(1)+Yi(0)−1=0Y_i(1)+Y_i(0)-1=0 whenever (Yi(1),Yi(0))(Y_i(1),Y_i(0)) is (1,0)(1,0) or (0,1)(0,1). Under H0FH_{0F}, each of the summands is nonzero; under any potential outcome table in H0NH_{0N}, some of these summands may be zero. Therefore, we expect that the tails of the reference distribution should dominate those of the sampling distribution. Proposition A.1 shows that it suffices to check this tail dominance conditional on a well-chosen statistic CC.

Proposition A.1. Consider any symmetric design A∼AA\sim\mathcal{A}. For each table in H0NH_{0N}, suppose there exists a random variable CC such that for all t>0t>0,

PAnew(τ^(Anew,Y(Aobs))≥t)≥P(τ^(Aobs,Y(Aobs))≥t∣C).(8)\mathbb{P}_{A^{\mathrm{new}}}\left(\hat{\tau}(A^{\mathrm{new}},Y(A^{\mathrm{obs}}))\ge t\right)\ge\mathbb{P}\left(\hat{\tau}(A^{\mathrm{obs}},Y(A^{\mathrm{obs}}))\ge t\mid C\right). \tag*{(8)}

Then, for all 0<α≤1/20<\alpha\le1/2, sup⁡H0NP(pF+≤α)≤α\sup_{H_{0N}}\mathbb{P}(p_F^+\le\alpha)\le\alpha. If further τ^obs∣C\hat{\tau}^{\mathrm{obs}}\mid C is symmetric around zero, then sup⁡H0NP(pF+≤α)≤α\sup_{H_{0N}}\mathbb{P}(p_F^+\le\alpha)\le\alpha.

Proof. We first prove the guarantees for H0NH_{0N}. If AA is symmetric, then (7) shows that τ^(Anew,Y(Aobs))\hat{\tau}(A^{\mathrm{new}},Y(A^{\mathrm{obs}})) is symmetric around zero conditional on AobsA^{\mathrm{obs}}. For α≤1/2\alpha\le1/2, pF+≤αp_F^+\le\alpha implies τ^obs>0\hat{\tau}^{\mathrm{obs}}>0 by this symmetry. Let FˉC(t):=P(τ^obs≥t∣C)\bar{F}_C(t):=\mathbb{P}(\hat{\tau}^{\mathrm{obs}}\ge t\mid C) be the conditional sampling distribution. By (8) and the definition of the FRT pp-value, {pF+≤α}⊆{FˉC(τ^obs)≤α}\{p_F^+\le\alpha\}\subseteq\{\bar{F}_C(\hat{\tau}^{\mathrm{obs}})\le\alpha\}. Then P(pF+≤α∣C)≤P(FˉC(τ^obs)≤α∣C)≤α\mathbb{P}(p_F^+\le\alpha\mid C)\le\mathbb{P}(\bar{F}_C(\hat{\tau}^{\mathrm{obs}})\le\alpha\mid C)\le\alpha. Take expectations to conclude the validity of pF+p_F^+.

By assumption, τ^obs\hat{\tau}^{\mathrm{obs}} is symmetric around zero conditional on CC. Multiplying each side of (8) by two and using this symmetry and that of τ^new∣Aobs\hat{\tau}^{\mathrm{new}}\mid A^{\mathrm{obs}}, we obtain

PAnew(∣τ^(Anew,Y(Aobs))∣≥t)≥P(∣τ^(Aobs,Y(Aobs))∣≥t∣C).\mathbb{P}_{A^{\mathrm{new}}}\left(\left|\hat{\tau}(A^{\mathrm{new}},Y(A^{\mathrm{obs}}))\right|\ge t\right)\ge\mathbb{P}\left(\left|\hat{\tau}(A^{\mathrm{obs}},Y(A^{\mathrm{obs}}))\right|\ge t\mid C\right).

The same argument shows the conditional validity of pF+p_F^+. To extend to H≤0NH_{\le0N}, we exploit the following representation of τ^cHT\hat{\tau}_{\mathrm{cHT}}:

τ^(A,Y)=1n∑i=1n(2Ai−1)(2Yi−1).(9)\hat{\tau}(A,Y)=\frac{1}{n}\sum_{i=1}^{n}(2A_i-1)(2Y_i-1). \tag*{(9)}

Then, with D(Anew,Aobs):=τ^(Anew,Y(Aobs))−τ^(Aobs,Y(Aobs))D(A^{\mathrm{new}},A^{\mathrm{obs}}):=\hat{\tau}(A^{\mathrm{new}},Y(A^{\mathrm{obs}}))-\hat{\tau}(A^{\mathrm{obs}},Y(A^{\mathrm{obs}})),

D(Anew,Aobs)=2n∑i=1n(Ainew−Aiobs)(2Yiobs−1),D(A^{\mathrm{new}},A^{\mathrm{obs}})=\frac{2}{n}\sum_{i=1}^{n}(A_i^{\mathrm{new}}-A_i^{\mathrm{obs}})(2Y_i^{\mathrm{obs}}-1),

and pF+(Aobs)=PAnew(D(Anew,Aobs)≥0)p_F^+(A^{\mathrm{obs}})=\mathbb{P}_{A^{\mathrm{new}}}(D(A^{\mathrm{new}},A^{\mathrm{obs}})\ge0). For any table P={(Yi(1),Yi(0))i=1n}\mathcal{P}=\{(Y_i(1),Y_i(0))_{i=1}^{n}\} in H≤0NH_{\le0N}, there are at least ∣nτ∣|n\tau| units with potential outcomes (0,1)(0,1). Take one of these units and switch its potential outcomes to (1,1)(1,1) to obtain a table P′\mathcal{P}'. Then the difference of D(Anew,Aobs)D(A^{\mathrm{new}},A^{\mathrm{obs}}) under P′\mathcal{P}' and P\mathcal{P} is given by 4n(Ainew−Aiobs)Aiobs≤0\frac{4}{n}(A_i^{\mathrm{new}}-A_i^{\mathrm{obs}})A_i^{\mathrm{obs}}\le0. Thus, D(Anew,Aobs)D(A^{\mathrm{new}},A^{\mathrm{obs}}) decreases when moving from P\mathcal{P} to P′\mathcal{P}', and so pF+(Aobs)p_F^+(A^{\mathrm{obs}}) decreases. This yields PP(pF+≤α)≤PP′(pF+≤α)\mathbb{P}_{\mathcal{P}}(p_F^+\le\alpha)\le\mathbb{P}_{\mathcal{P}'}(p_F^+\le\alpha). Iterate this guarantee until a table P′∈H0N\mathcal{P}'\in H_{0N} is obtained. Then, use the first part of the proposition. □

Before treating each of the different designs, we record the following useful tail comparison lemma.

Lemma A.1 (Random walk tail comparison). For any nonnegative integers k′>kk'>k of the same parity and an integer threshold t>0t>0 of the same parity as kk, P(Sk′≥t)≥P(Sk≥t)\mathbb{P}(S_{k'}\ge t)\ge\mathbb{P}(S_k\ge t).

Proof. It suffices to show that P(Sk+2≥t)≥P(Sk≥t)\mathbb{P}(S_{k+2}\ge t)\ge\mathbb{P}(S_k\ge t) for every kk and t>0t>0. By conditioning on the last two increments of Sk+2S_{k+2}, we have

P(Sk+2≥t)=14P(Sk≥t+2)+12P(Sk≥t)+14P(Sk≥t−2).\mathbb{P}(S_{k+2}\ge t)=\frac{1}{4}\mathbb{P}(S_k\ge t+2)+\frac{1}{2}\mathbb{P}(S_k\ge t)+\frac{1}{4}\mathbb{P}(S_k\ge t-2).

Thus, P(Sk+2≥t)≥P(Sk≥t)\mathbb{P}(S_{k+2}\ge t)\ge\mathbb{P}(S_k\ge t) is equivalent to P(Sk≥t+2)−P(Sk≥t)≥P(Sk≥t)−P(Sk≥t−2)\mathbb{P}(S_k\ge t+2)-\mathbb{P}(S_k\ge t)\ge\mathbb{P}(S_k\ge t)-\mathbb{P}(S_k\ge t-2), or P(Sk=t)≤P(Sk=t−2)\mathbb{P}(S_k=t)\le\mathbb{P}(S_k=t-2). This is true by unimodality of SkS_k. □

Proofs for CRE

Throughout this section, we will refer to units ii for which Yi=1Y_i=1, Ai=1A_i=1 as treated ones, Yi=1Y_i=1, Ai=0A_i=0 as control ones, and so forth for the remaining combinations. Under a CRE, let OTobs=#{i:Aiobs=1,Yi(Aiobs)=1}O_T^{\mathrm{obs}}=\#\{i:A_i^{\mathrm{obs}}=1,Y_i(A_i^{\mathrm{obs}})=1\}, OCobs=#{i:Aiobs=0,Yi(Aiobs)=1}O_C^{\mathrm{obs}}=\#\{i:A_i^{\mathrm{obs}}=0,Y_i(A_i^{\mathrm{obs}})=1\} be the number of treated ones and control ones respectively. Let Oobs=OTobs+OCobs=∑iYiO^{\mathrm{obs}}=O_T^{\mathrm{obs}}+O_C^{\mathrm{obs}}=\sum_iY_i be the total number of ones in the data. Let Hyper⁡(N,K,n)\operatorname{Hyper}(N,K,n) denote the hypergeometric law with total urn size NN, number of successes KK, and number of draws nn.

Given observed data vector y∈{0,1}ny\in\{0,1\}^{n} and treatment vector a∈{0,1}na\in\{0,1\}^{n} with oTo_T many treated ones and oCo_C many control ones, call a table (Yi(1),Yi(0))i=1n(Y_i(1),Y_i(0))_{i=1}^{n} compatible with yy, aa if yi=Yi(ai)y_i=Y_i(a_i) for all ii. Let N(oT,oC)N(o_T,o_C) be the number of potential outcome tables compatible with yy that have strata (n11,n10,n01,n00)(n_{11},n_{10},n_{01},n_{00}). Lemma A.2 shows that NN only depends on yy through oT,oCo_T,o_C.

Proof of Theorem 2.1. We will apply Proposition A.1 by checking (8) with C=OobsC=O^{\mathrm{obs}}. Under a CRE, τ^obs\hat{\tau}^{\mathrm{obs}} is equal to 2n(OTobs−OCobs)=2n(2OTobs−Oobs)\frac{2}{n}(O_T^{\mathrm{obs}}-O_C^{\mathrm{obs}})=\frac{2}{n}(2O_T^{\mathrm{obs}}-O^{\mathrm{obs}}). It is easy to see that the reference distribution, the law of τ^(Anew,Y(Aobs))∣Aobs\hat{\tau}(A^{\mathrm{new}},Y(A^{\mathrm{obs}}))\mid A^{\mathrm{obs}}, is equal to the law of

2n(2H−Oobs),H∼Hyper⁡(n,Oobs,n/2),\frac{2}{n}\left(2H-O^{\mathrm{obs}}\right),\qquad H\sim\operatorname{Hyper}\left(n,O^{\mathrm{obs}},n/2\right),

which only depends on AobsA^{\mathrm{obs}} through OobsO^{\mathrm{obs}}. We check that

P(∣H−Oobs2∣≥t∣Oobs)≥P(∣OTobs−Oobs2∣≥t∣Oobs).(10)\mathbb{P}\left(\left|H-\frac{O^{\mathrm{obs}}}{2}\right|\ge t\mid O^{\mathrm{obs}}\right)\ge\mathbb{P}\left(\left|O_T^{\mathrm{obs}}-\frac{O^{\mathrm{obs}}}{2}\right|\ge t\mid O^{\mathrm{obs}}\right). \tag*{(10)}

Symmetry of both H−Oobs/2∣OobsH-O^{\mathrm{obs}}/2\mid O^{\mathrm{obs}} and OTobs−Oobs/2∣OobsO_T^{\mathrm{obs}}-O^{\mathrm{obs}}/2\mid O^{\mathrm{obs}} shows that the one-sided guarantee holds. To prove (10), we consider the likelihood ratio of the conditional distributions of OTobsO_T^{\mathrm{obs}} and HH, given OobsO^{\mathrm{obs}}. First, we calculate P(OTobs=oT,OCobs=oC)\mathbb{P}(O_T^{\mathrm{obs}}=o_T,O_C^{\mathrm{obs}}=o_C). Complete randomization is combinatorially equivalent to uniformly permuting the units and treating the first n/2n/2 units in the permuted order. We count the number of permutations for which the observed data consists of oTo_T many ones among the first n/2n/2 units and oCo_C many ones in the remaining n/2n/2 units. There are (n/2oT)\binom{n/2}{o_T} many ways to select the locations of the treated ones and (n/2oC)\binom{n/2}{o_C} many ways to select the locations of the control ones. There are N(oT,oC)N(o_T,o_C) many tables with the proper strata that are compatible with the chosen locations. Finally, the units may be permuted within each stratum, giving a factor of n11!n10!n01!n00!n_{11}!n_{10}!n_{01}!n_{00}!. Thus,

P(OTobs=oT,OCobs=oC)=n11!n10!n01!n00!(n/2oT)(n/2oC)N(oT,oC)n!.\mathbb{P}(O_T^{\mathrm{obs}}=o_T,O_C^{\mathrm{obs}}=o_C)=\frac{n_{11}!n_{10}!n_{01}!n_{00}!\binom{n/2}{o_T}\binom{n/2}{o_C}N(o_T,o_C)}{n!}.

Write Po(H=k)\mathbb{P}_o(H=k) as shorthand for P(H=k∣Oobs=o)\mathbb{P}(H=k\mid O^{\mathrm{obs}}=o). By rewriting the hypergeometric mass function, Po(H=k)=(n/2k)(n/2o−k)(no)\mathbb{P}_o(H=k)=\frac{\binom{n/2}{k}\binom{n/2}{o-k}}{\binom{n}{o}}. The likelihood ratio then satisfies

R(k):=P(OTobs=k∣Oobs=o)Po(H=k)∝N(k,o−k).(11)R(k):=\frac{\mathbb{P}(O_T^{\mathrm{obs}}=k\mid O^{\mathrm{obs}}=o)}{\mathbb{P}_o(H=k)}\propto N(k,o-k). \tag*{(11)}

Proposition A.2 shows that for every oT,oCo_T,o_C also N(oT,oC)=N(oC,oT)N(o_T,o_C)=N(o_C,o_T) and when oT≥oC+2o_T\ge o_C+2, N(oT,oC)≤N(oT−1,oC+1)N(o_T,o_C)\le N(o_T-1,o_C+1). Therefore, N(k,o−k)N(k,o-k) is nonincreasing as ∣k−o/2∣\lvert k-o/2\rvert increases. Because ∑kR(k)Po(H=k)=1\sum_k R(k)\mathbb{P}_o(H=k)=1, the likelihood ratio R(k)R(k) crosses 11 at most once, from above to below, as kk ranges over ⌈o/2+1⌉,…,o\lceil o/2+1\rceil,\ldots,o. Call this crossing point k∗k^*, so R(k)≤1R(k)\le1, k>k∗k>k^* and R(k)>1R(k)>1, k=⌈o/2+1⌉,…,k∗k=\lceil o/2+1\rceil,\ldots,k^*.

Fix any t>k∗−o/2t>k^*-o/2. Then

P(∣OTobs−o/2∣≥t∣Oobs=o)=∑∣k−o/2∣≥tR(k)Po(H=k)≤∑∣k−o/2∣≥tPo(H=k)=P(∣H−o/2∣≥t).(12)\mathbb{P}\left(\left|O_T^{\mathrm{obs}}-o/2\right|\ge t\mid O^{\mathrm{obs}}=o\right)=\sum_{\lvert k-o/2\rvert\ge t}R(k)\mathbb{P}_o(H=k)\le\sum_{\lvert k-o/2\rvert\ge t}\mathbb{P}_o(H=k)=\mathbb{P}\left(\left|H-o/2\right|\ge t\right). \tag*{(12)}

When t≤k∗−o/2t\le k^*-o/2, redo the above derivation with the lower tail event and take complements. This proves (10). Proposition A.1 gives the one and two-sided conclusions.

Lemma A.2. The completion count N(oT,oC)N(o_T,o_C) only depends on a,y∈{0,1}na,y \in\{0,1\}^{n} through oT,oCo_T,o_C.

Proof. Take any two data vectors y,y′∈{0,1}ny,y' \in\{0,1\}^{n} with the same number of treated ones and control ones oT,oCo_T,o_C. Take a permutation σ\sigma for which y′=σyy'=\sigma y and a′=σaa'=\sigma a. For any table P\mathcal{P} which is compatible with yy, applying the permutation σ\sigma to the rows of P\mathcal{P} yields another table P′\mathcal{P}' compatible with y′y'. This mapping is a bijection between the set of tables compatible with yy and those compatible with y′y'. Therefore the number only depends on oT,oCo_T,o_C.

Proposition A.2. Suppose n01=n10n_{01}=n_{10}, oC≥0o_C \ge0, and oC+2≤oT≤n/2o_C+2 \le o_T \le n/2. Then N(oT,oC)≤N(oT−1,oC+1)N(o_T,o_C) \le N(o_T-1,o_C+1). For any oC,oTo_C,o_T, N(oT,oC)=N(oC,oT)N(o_T,o_C)=N(o_C,o_T).

Proof. Let N(oT,oC)\mathcal{N}(o_T,o_C) be the set of tables with principal strata counts (n11,n10,n01,n00)(n_{11},n_{10},n_{01},n_{00}) and first n/2n/2 units treated, such that Yi(1)=1{i≤oT}Y_i(1)=1\{i\le o_T\} and Yn/2+i(0)=1{i≤oC}Y_{n/2+i}(0)=1\{i\le o_C\} for i=1,…,n/2i=1,\ldots,n/2. These are compatible with the observed data (oT,oC)(o_T,o_C) and by Lemma A.2, ∣N(oT,oC)∣=N(oT,oC)|\mathcal{N}(o_T,o_C)|=N(o_T,o_C). We will exhibit an injection Φ:N(oT,oC)↪N(oT−1,oC+1)\Phi:\mathcal{N}(o_T,o_C)\hookrightarrow\mathcal{N}(o_T-1,o_C+1).

To define the injection, fix a table P∈N(oT,oC)\mathcal{P}\in\mathcal{N}(o_T,o_C). See the left table of Figure 5 for a running visualization. We define five disjoint subsets of its rows. Let W={oT,n/2+oC+1}W=\{o_T,n/2+o_C+1\} be the row indices of the last treated 1 and the first control 0. Let VTV_T be the treated zeros {oT+1,…,n/2}\{o_T+1,\ldots,n/2\} and VCV_C the control ones {n/2+1,…,n/2+oC}\{n/2+1,\ldots,n/2+o_C\}. Finally, let UTU_T consist of the first oCo_C treated ones {1,…,oC}\{1,\ldots,o_C\} and UCU_C be the first n/2−oTn/2-o_T control zeros, excluding the row in WW: {n/2+oC+2,…,n+1+oC−oT}\{n/2+o_C+2,\ldots,n+1+o_C-o_T\}. Because oT≥oC+2o_T\ge o_C+2, ∣UT∣=∣VC∣|U_T|=|V_C| and ∣VT∣=∣UC∣|V_T|=|U_C|.

Example of the injection Phi showing two potential-outcome tables before and after the transformation

Figure 5. Example of the injection Φ\Phi with (n11,n10,n01,n00)=(5,2,2,5)(n_{11},n_{10},n_{01},n_{00})=(5,2,2,5) and oT=5,oC=3o_T=5,o_C=3. The ordered pairs are (5,11)(5,11), (1,8)(1,8), (2,9)(2,9), (3,10)(3,10), (6,12)(6,12), and (7,13)(7,13). The first three pairs are swapped and flipped, with changed entries in bold font. Grey entries are unobserved.

The idea is to swap the rows in WW and reverse their potential outcomes to define a table Φ(P)\Phi(\mathcal{P}) in N(oT−1,oC+1)\mathcal{N}(o_T-1,o_C+1). The issue is that the principal strata counts will not be maintained. We compensate for this by further swapping rows between UT,VCU_T,V_C and VT,UCV_T,U_C. To this end, we define a pairing of row indices by 1) pairing the rows in WW, 2) pairing the rows in UTU_T with the corresponding rows in VCV_C, in increasing row order, and 3) similarly with VTV_T to UCU_C. Define a list of row pairs {(pj,qj)}j=1K\{(p_j,q_j)\}_{j=1}^{K}, K=1+∣UT∪UC∣K=1+|U_T\cup U_C| by concatenating the matched pairs in WW, then those in (UT,VC)(U_T,V_C) followed by (VT,UC)(V_T,U_C). Explicitly,

(pk,qk)={(oT,n/2+oC+1)if k=1,(k−1,n/2+k−1)if k=2,…,oC+1,(k+oT−oC−1,k+n/2)if k>oC+1.(13)(p_k,q_k)= \begin{cases} (o_T,n/2+o_C+1) & \text{if } k=1,\\ (k-1,n/2+k-1) & \text{if } k=2,\ldots,o_C+1,\\ (k+o_T-o_C-1,k+n/2) & \text{if } k>o_C+1. \end{cases} \tag*{(13)}

Note that this pairing forces

Ypk(1)=Yqk(0),∀k≥2.(14)Y_{p_k}(1)=Y_{q_k}(0),\quad\forall k\ge2. \tag*{(14)}

Define

δk(P)≔∑i=1k(Ypi(1)−Ypi(0)+Yqi(1)−Yqi(0)),k=1,…,K,(15)\delta_k(\mathcal{P})\coloneqq\sum_{i=1}^{k}\left(Y_{p_i}(1)-Y_{p_i}(0)+Y_{q_i}(1)-Y_{q_i}(0)\right),\quad k=1,\ldots,K, \tag*{(15)}

to be the running imbalance between the pairs in this list. Let k∗(P)k_*(\mathcal{P}) be the first index for which this running sum hits zero. This index always exists. By construction of the set WW, δ1(P)≥0\delta_1(\mathcal{P}) \ge0. Because n10=n01n_{10} = n_{01} and VT∪VCV_T \cup V_C contains all rows with type (0,1)(0,1), δK(P)≤0\delta_K(\mathcal{P}) \le0. Finally, (14) forces ∣δk(P)−δk−1(P)∣≤1|\delta_k(\mathcal{P})-\delta_{k-1}(\mathcal{P})| \le1 for k≥2k \ge2.

Define the injection Φ\Phi as follows. For each 1≤j≤k∗(P)1 \le j \le k_*(\mathcal{P}), swap the rows pj,qjp_j,q_j, then swap the potential outcomes within each row. This results in a table Φ(P)\Phi(\mathcal{P}) with rows (Yi′(1),Yi′(0))i=1n(Y'_i(1),Y'_i(0))_{i=1}^n defined by

Ypi′(a)=Yqi(1−a),Yqi′(a)=Ypi(1−a),1≤i≤k∗(P),a∈{0,1},(16)Y'_{p_i}(a)=Y_{q_i}(1-a),\qquad Y'_{q_i}(a)=Y_{p_i}(1-a),\qquad1\le i\le k_*(\mathcal{P}),\quad a\in\{0,1\}, \tag*{(16)}

and all other rows unchanged. The swaps exchange types (1,0)(1,0) and (0,1)(0,1) while preserving (1,1)(1,1) and (0,0)(0,0). Since δk∗(P)(P)=0\delta_{k_*(\mathcal{P})}(\mathcal{P})=0, the numbers of all principal strata are preserved. The observed entries only change for the rows in WW. Thus Φ(P)∈N(oT−1,oC+1)\Phi(\mathcal{P})\in\mathcal{N}(o_T-1,o_C+1).

Finally, we show this map is an injection. Consider a table P′∈N(oT−1,oC+1)\mathcal{P}'\in\mathcal{N}(o_T-1,o_C+1) that is equal to Φ(P)\Phi(\mathcal{P}) for some P\mathcal{P}. Given P′\mathcal{P}', flipping the first treated zero to 1 and last control one to 0 yields the same observed data pattern as P\mathcal{P}, allowing us to recover the same sets W,UT,UC,VT,VCW,U_T,U_C,V_T,V_C. For all k≤k∗k\le k_*, we have δk(P′)=−δk(P)\delta_k(\mathcal{P}')=-\delta_k(\mathcal{P}). Thus, k∗(P′)=k∗(P)k_*(\mathcal{P}')=k_*(\mathcal{P}). Applying the same swap-reverse transformations as in Φ\Phi to P′\mathcal{P}' therefore recovers the original table P\mathcal{P}.

Symmetry. For any table P\mathcal{P} in N(oT,oC)\mathcal{N}(o_T,o_C), create a new table by switching each unit’s outcomes from (Yi(1),Yi(0))(Y_i(1),Y_i(0)) to (Yi(0),Yi(1))(Y_i(0),Y_i(1)) and flipping treatment and control. Because n10=n01n_{10}=n_{01}, this creates another table in N(oC,oT)\mathcal{N}(o_C,o_T). This transformation is a bijection. Therefore N(oT,oC)=N(oC,oT)\mathcal{N}(o_T,o_C)=\mathcal{N}(o_C,o_T).

Proofs for Bernoulli Randomized Experiments

Proof of Theorem 2.2. We will show that (8) holds with C=0C = 0. Recall that SkS_k denotes the simple random walk at time kk. By (7), under H0NH_{0N}, τ^obs\hat{\tau}^{\mathrm{obs}} is equal in distribution to 1nSn11+n00\frac{1}{n}S_{n_{11}+n_{00}}. Using (7) again, τ^CHT(Anew,Y(Aobs))=1n∑i=1n(2Ainew−1)(2Yi(Aiobs)−1)\hat{\tau}_{\mathrm{CHT}}(A^{\mathrm{new}},Y(A^{\mathrm{obs}})) = \frac{1}{n}\sum_{i=1}^{n}(2A_i^{\mathrm{new}}-1)(2Y_i(A_i^{\mathrm{obs}})-1). Since 2Yi(Aiobs)−1∈{−1,1}2Y_i(A_i^{\mathrm{obs}})-1 \in\{-1,1\}, the reference distribution of the FRT is equal to 1nSn\frac{1}{n}S_n, regardless of AiobsA_i^{\mathrm{obs}}. Under H0NH_{0N}, n10=n01n_{10}=n_{01}, so n≡n11+n00(mod2)n \equiv n_{11}+n_{00}\pmod2. Lemma A.1 shows

P(1nSn≥t)≥P(1nSn11+n00≥t).\mathbb{P}\left(\frac{1}{n}S_n \ge t\right) \ge\mathbb{P}\left(\frac{1}{n}S_{n_{11}+n_{00}} \ge t\right).

Apply Proposition A.1 to conclude.

Proofs for Matched Pairs Experiments

Proof of Theorem 2.3. In the matched pairs experiment, denote the summands in the difference in means estimator by δpobs:=(2Apobs−1)(Yp1(Apobs)−Yp0(1−Apobs))\delta_p^{\mathrm{obs}} := (2A_p^{\mathrm{obs}}-1)(Y_{p1}(A_p^{\mathrm{obs}})-Y_{p0}(1-A_p^{\mathrm{obs}})), or equivalently,

δpobs=Apobs(Yp1(1)−Yp0(0))+(1−Apobs)(Yp0(1)−Yp1(0)).(17)\delta_p^{\mathrm{obs}} = A_p^{\mathrm{obs}}(Y_{p1}(1)-Y_{p0}(0))+(1-A_p^{\mathrm{obs}})(Y_{p0}(1)-Y_{p1}(0)). \tag*{(17)}

Let δpnew\delta_p^{\mathrm{new}} be the summand in the resampled estimator, which by (17) is equal to Apnew(Yp1(Apobs)−Yp0(1−Apobs))+(1−Apnew)(Yp0(1−Apobs)−Yp1(Apobs))A_p^{\mathrm{new}}(Y_{p1}(A_p^{\mathrm{obs}})-Y_{p0}(1-A_p^{\mathrm{obs}}))+(1-A_p^{\mathrm{new}})(Y_{p0}(1-A_p^{\mathrm{obs}})-Y_{p1}(A_p^{\mathrm{obs}})). Thus,

δpnew=(d)ϵpnew∣δpobs∣,(18)\delta_p^{\mathrm{new}} \overset{(d)}{=} \epsilon_p^{\mathrm{new}} \lvert\delta_p^{\mathrm{obs}}\rvert, \tag*{(18)}

where the ϵpnew\epsilon_p^{\mathrm{new}} are iid Rademacher random variables, independent of everything else. Let Pobs=∑p=1n/2∣δpobs∣P^{\mathrm{obs}}=\sum_{p=1}^{n/2}\lvert\delta_p^{\mathrm{obs}}\rvert be the observed number of nonzero pair differences. Then, by (18),

{nτ^new/2∣Pobs=r}=(d)Sr,(19)\left\{n\hat{\tau}^{\mathrm{new}}/2 \mid P^{\mathrm{obs}}=r\right\} \overset{(d)}{=} S_r, \tag*{(19)}

the random walk at time rr.

We now turn to understanding the law of τ^obs\hat{\tau}^{\mathrm{obs}}. Using casework, we can characterize the distribution of δpobs\delta_p^{\mathrm{obs}} as a function of the 16 possible combinations of potential outcomes in pair pp, (Yp1(1),Yp1(0),Yp0(1),Yp0(0))(Y_{p1}(1),Y_{p1}(0),Y_{p0}(1),Y_{p0}(0)). For example, suppose the table for pair pp has the form

Yi(1)Y_i(1)Yi(0)Y_i(0)
p1p1aabb
p0p0bbaa

caption:

:::

for a,b∈{0,1}a,b \in\{0,1\}. With probability 1/21/2, the observed data is (a,a)(a,a), which forces δpobs=0\delta_p^{\mathrm{obs}}=0. The same is true in the other case when (b,b)(b,b) is observed. When the table is of the form

[a01a],[1aa0]\begin{bmatrix} a & 0 \\ 1 & a \end{bmatrix}, \begin{bmatrix} 1 & a \\ a & 0 \end{bmatrix}

δpobs (d)=B\delta_p^{\mathrm{obs}\,(d)} = B, a fair Bernoulli random variable. When it is of the form

[a10a],[0aa1],\begin{bmatrix} a & 1 \\ 0 & a \end{bmatrix}, \begin{bmatrix} 0 & a \\ a & 1 \end{bmatrix},

δpobs (d)=−B\delta_p^{\mathrm{obs}\,(d)} = -B. The last tables to handle are

[1100],[0011],[1010],[0101].\begin{bmatrix} 1 & 1 \\ 0 & 0 \end{bmatrix}, \begin{bmatrix} 0 & 0 \\ 1 & 1 \end{bmatrix}, \begin{bmatrix} 1 & 0 \\ 1 & 0 \end{bmatrix}, \begin{bmatrix} 0 & 1 \\ 0 & 1 \end{bmatrix}.

In the first two cases, δpobs (d)≡ϵ\delta_p^{\mathrm{obs}\,(d)} \equiv\epsilon. In the last two cases, δpobs=1\delta_p^{\mathrm{obs}} = 1 and δpobs=−1\delta_p^{\mathrm{obs}} = -1 respectively. Table 1 summarizes the distributions of δpobs\delta_p^{\mathrm{obs}} as a function of the potential outcomes of units p1,p0p1,p0.

Unit 1 / Unit 0(0,0)(0,0)(0,1)(0,1)(1,0)(1,0)(1,1)(1,1)
(0,0)(0,0)00−B-BBBϵ\epsilon
(0,1)(0,1)−B-B−1-100−B-B
(1,0)(1,0)BB0011BB
(1,1)(1,1)ϵ\epsilon−B-BBB00

Table 1. Distributions of δpobs\delta_p^{\mathrm{obs}} for all sixteen binary potential-outcome tables of a matched pair pp. Here, B∼Bernoulli⁡(1/2)B \sim\operatorname{Bernoulli}(1/2) and ϵ\epsilon is a Rademacher variable.

Define Δpi:=Ypi(1)−Ypi(0)\Delta_{pi} := Y_{pi}(1)-Y_{pi}(0). Inspecting Table 1, the sum of the individual treatment effects for the units in the pair satisfies

Δp0+Δp1={+2,if δpobs (d)≡+1,+1,if δpobs (d)≡B,0,if δpobs (d)≡0 or δpobs (d)≡ϵ,−1,if δpobs (d)≡−B,−2,if δpobs (d)≡−1.(20)\Delta_{p0}+\Delta_{p1} = \begin{cases} +2, & \text{if } \delta_p^{\mathrm{obs}\,(d)} \equiv+1,\\ +1, & \text{if } \delta_p^{\mathrm{obs}\,(d)} \equiv B,\\ 0, & \text{if } \delta_p^{\mathrm{obs}\,(d)} \equiv0 \text{ or } \delta_p^{\mathrm{obs}\,(d)} \equiv\epsilon,\\ -1, & \text{if } \delta_p^{\mathrm{obs}\,(d)} \equiv-B,\\ -2, & \text{if } \delta_p^{\mathrm{obs}\,(d)} \equiv-1. \end{cases} \tag*{(20)}

We will construct a partition G\mathcal{G} of the pairs p=1,…,n/2p=1,\ldots,n/2 into KK disjoint sets Gk\mathcal{G}_k for some KK, as follows. Let I+B\mathcal{I}_{+B} be the set of all pairs for which δpobs\delta_p^{\mathrm{obs}} has the same distribution as BB. Define the same for I−B,I−1,I+1,I0,Iϵ\mathcal{I}_{-B},\mathcal{I}_{-1},\mathcal{I}_{+1},\mathcal{I}_{0},\mathcal{I}_{\epsilon}. Let m+B,m−B,m−1,m+1,mϵm_{+B},m_{-B},m_{-1},m_{+1},m_{\epsilon} be the sizes of these sets. Under H0NH_{0N}, (20) shows

m+B−m−B+2(m+1−m−1)=0.(21)m_{+B}-m_{-B}+2(m_{+1}-m_{-1})=0. \tag*{(21)}

Suppose m+1−m−1=ν≥0m_{+1}-m_{-1}=\nu\ge0. For as many pairs p∈I+1p\in\mathcal{I}_{+1} as possible, match them with other pairs p′∈I−1p'\in\mathcal{I}_{-1}. For such a pair {p,p′}\{p,p'\}, let this be another set in G\mathcal{G}. The number of leftover pairs in I+1\mathcal{I}_{+1} is ν\nu. For every pair p∈I+Bp\in\mathcal{I}_{+B}, match it with a pair p′∈I−Bp'\in\mathcal{I}_{-B} and add {p,p′}\{p,p'\} to the partition G\mathcal{G}. By (21), there are 2ν2\nu unmatched pairs pp left over in I−B\mathcal{I}_{-B}. For every 2 leftover pairs p1,p2p_1,p_2 in I−B\mathcal{I}_{-B}, group them with an unmatched pair p3∈I+1p_3\in\mathcal{I}_{+1} and add {p1,p2,p3}\{p_1,p_2,p_3\} to the partition G\mathcal{G}. Finally, for all other pairs in Iϵ,I0\mathcal{I}_{\epsilon},\mathcal{I}_0, add them as singleton sets into G\mathcal{G}. For every set G∈GG\in\mathcal{G}, the sum ∑p∈Gδpobs\sum_{p\in G}\delta_p^{\mathrm{obs}} is distributed as one of the following:

0,ϵ,B−B′,1−B−B′,(22)0,\quad\epsilon,\quad B-B',\quad1-B-B', \tag*{(22)}

where B,B′B,B' are independent Bernoulli(1/2)(1/2) random variables. Moreover, the sums are independent across groups GG. Construct an analogous partition if m+1−m−1<0m_{+1}-m_{-1}<0. In this case, instead of +1−B−B′+1-B-B' being a possible law, the option is B+B′−1B+B'-1. Crucially, in all of these cases, the distribution is supported on {−1,0,1}\{-1,0,1\} and symmetric around zero: for each G∈GG\in\mathcal{G},

∑p∈Gδpobs(d) =(d) ∣∑p∈Gδpobs∣ϵ,(23)\sum_{p\in G}\delta_p^{\mathrm{obs}(d)}\ \overset{(d)}{=}\ \left|\sum_{p\in G}\delta_p^{\mathrm{obs}}\right|\epsilon, \tag*{(23)}

for an independent Rademacher random variable ϵ\epsilon. Therefore, conditional on PGobs:=∑G∈G∣∑p∈Gδpobs∣P_{\mathcal{G}}^{\mathrm{obs}}:=\sum_{G\in\mathcal{G}}\left|\sum_{p\in G}\delta_p^{\mathrm{obs}}\right|,

nτ^obs/2=∑G∈G∑p∈Gδpobs(d)=SPGobs.(24)n\widehat{\tau}^{\mathrm{obs}}/2=\sum_{G\in\mathcal{G}}\sum_{p\in G}\delta_p^{\mathrm{obs}(d)}=S_{P_{\mathcal{G}}^{\mathrm{obs}}}. \tag*{(24)}

Putting our analyses of the sampling distribution and reference distribution together, Lemma A.1 shows that for every t>0t>0,

PAnew(τ^(Anew,Y(Aobs))≥t)≥P(τ^obs≥t∣PGobs).(25)\mathbb{P}_{A^{\mathrm{new}}}\left(\widehat{\tau}(A^{\mathrm{new}},Y(A^{\mathrm{obs}}))\ge t\right)\ge\mathbb{P}\left(\widehat{\tau}^{\mathrm{obs}}\ge t\mid P_{\mathcal{G}}^{\mathrm{obs}}\right). \tag*{(25)}

Indeed, PGobs≡∑G∈G∑p∈Gδpobs≡∑p=1n/2δpobs≡Pobs(mod2)P_{\mathcal{G}}^{\mathrm{obs}}\equiv\sum_{G\in\mathcal{G}}\sum_{p\in G}\delta_p^{\mathrm{obs}}\equiv\sum_{p=1}^{n/2}\delta_p^{\mathrm{obs}}\equiv P^{\mathrm{obs}}\pmod2 using the fact that x≡∣x∣(mod2)x\equiv\lvert x\rvert\pmod2. By the triangle inequality, PGobs≤PobsP_{\mathcal{G}}^{\mathrm{obs}}\le P^{\mathrm{obs}}. This shows (8) with C=PGobsC=P_{\mathcal{G}}^{\mathrm{obs}}; applying Proposition A.1 completes the proof.

Proofs for Unbalanced Experiments

Proof of Theorem 2.4. Let ρ\rho be the treatment probability of the design. Under H≤0NH_{\le0}^{N}, there exists a pairing between all n10n_{10} units with potential outcomes (Yi(1),Yi(0))=(1,0)(Y_i(1),Y_i(0))=(1,0) and a subset of the n01n_{01} units with outcomes (0,1)(0,1). Let π\pi be the permutation which swaps the units in each pairing and leaves all other units untouched. Define for any vectors a,b∈{0,1}na,b\in\{0,1\}^{n} the difference

D(a,b)=nρ(1−ρ)[τ^(πa,Y(b))−τ^(b,Y(b))]=∑i=1n(aπ(i)−bi)(Yi(bi)−12).(26)D(a,b)=n\rho(1-\rho)\left[\widehat{\tau}(\pi a,Y(b))-\widehat{\tau}(b,Y(b))\right]=\sum_{i=1}^{n}(a_{\pi(i)}-b_i)\left(Y_i(b_i)-\frac{1}{2}\right). \tag*{(26)}

Then the exchangeability of AnewA^{\mathrm{new}} gives

pF+(Aobs)=PAnew(D(Anew,Aobs)≥0).(27)p_F^+(A^{\mathrm{obs}})=\mathbb{P}_{A^{\mathrm{new}}}\left(D(A^{\mathrm{new}},A^{\mathrm{obs}})\ge0\right). \tag*{(27)}

Following the coverage proof of the Jackknife+ [5], interpret D(a,b),D(b,a)D(a,b),D(b,a) as the scores in a two-way match between vectors a,b∈{0,1}na,b\in\{0,1\}^{n} in a tournament played between all vectors in {0,1}n\{0,1\}^{n}. Declare aa to be a winner in the match if D(a,b)≥0D(a,b)\ge0, and bb to be a winner if D(b,a)≥0D(b,a)\ge0. Observe that there can, in principle, be zero, one, or two winners. Let SS be the set of vectors whose opponent wins with probability at most α\alpha under A\mathcal{A}:

S:={a∈{0,1}n:Pb∈A(D(b,a)≥0)≤α}.S:=\left\{a\in\{0,1\}^{n}:\mathbb{P}_{b\in\mathcal{A}}\left(D(b,a)\ge0\right)\le\alpha\right\}.

Then P(pF+(Aobs)≤α)\mathbb{P}(p_F^+(A^{\mathrm{obs}})\le\alpha) is equal to P(Aobs∈S)\mathbb{P}(A^{\mathrm{obs}}\in S). We claim that there always exists at least one winner in a match, via the bound

D(a,b)+D(b,a)≥0.(28)D(a,b)+D(b,a)\ge0. \tag*{(28)}

Consider a pair i,ji,j where π(i)=j,π(j)=i\pi(i)=j,\pi(j)=i and (Yi(1),Yi(0))=(1,0),(Yj(1),Yj(0))=(0,1)(Y_i(1),Y_i(0))=(1,0),(Y_j(1),Y_j(0))=(0,1). Then the contribution of indices i,ji,j to D(a,b)+D(b,a)D(a,b)+D(b,a) is equal to

(aj−bi)(Yi(bi)−12)+(ai−bj)(Yj(bj)−12)+(bj−ai)(Yi(ai)−12)+(bi−aj)(Yj(aj)−12)=(aj−bi)(Yi(bi)−Yj(aj))+(ai−bj)(Yj(bj)−Yi(ai))=0.\begin{aligned} &(a_j-b_i)\left(Y_i(b_i)-\frac{1}{2}\right)+(a_i-b_j)\left(Y_j(b_j)-\frac{1}{2}\right)+\\ &(b_j-a_i)\left(Y_i(a_i)-\frac{1}{2}\right)+(b_i-a_j)\left(Y_j(a_j)-\frac{1}{2}\right)\\ &=(a_j-b_i)\left(Y_i(b_i)-Y_j(a_j)\right)+(a_i-b_j)\left(Y_j(b_j)-Y_i(a_i)\right)=0. \end{aligned}

The last equality follows because Yi(t)=tY_i(t)=t and Yj(t)=1−tY_j(t)=1-t for t∈{0,1}t\in\{0,1\}, so both terms vanish by the identity (x−y)(x+y−1)=0(x-y)(x+y-1)=0 for all x,y∈{0,1}x,y\in\{0,1\}.

For any single unit ii, the contribution of unit ii to D(a,b)+D(b,a)D(a,b)+D(b,a) is (ai−bi)(Yi(bi)−Yi(ai))(a_i-b_i)(Y_i(b_i)-Y_i(a_i)). When Yi(1)=Yi(0)Y_i(1)=Y_i(0), the contribution is zero; when (Yi(1),Yi(0))=(0,1)(Y_i(1),Y_i(0))=(0,1), the contribution is 1{ai≠bi}\mathbf{1}\{a_i\ne b_i\}. This proves (28). Finally,

P(Aobs∈S)2=P(Aobs∈S,Anew∈S)≤E[1{D(Aobs,Anew)≥0}+1{D(Anew,Aobs)≥0};Aobs∈S,Anew∈S]≤2E[1{D(Anew,Aobs)≥0};Aobs∈S,Anew∈S]≤2E[1{D(Anew,Aobs)≥0};Aobs∈S]≤2E[pF+;Aobs∈S]≤2αP(Aobs∈S).\begin{aligned} \mathbb{P}(A^{\mathrm{obs}}\in S)^2 &=\mathbb{P}(A^{\mathrm{obs}}\in S,A^{\mathrm{new}}\in S)\\ &\le\mathbb{E}\left[\mathbf{1}\{D(A^{\mathrm{obs}},A^{\mathrm{new}})\ge0\}+\mathbf{1}\{D(A^{\mathrm{new}},A^{\mathrm{obs}})\ge0\};A^{\mathrm{obs}}\in S,A^{\mathrm{new}}\in S\right]\\ &\le2\mathbb{E}\left[\mathbf{1}\{D(A^{\mathrm{new}},A^{\mathrm{obs}})\ge0\};A^{\mathrm{obs}}\in S,A^{\mathrm{new}}\in S\right]\\ &\le2\mathbb{E}\left[\mathbf{1}\{D(A^{\mathrm{new}},A^{\mathrm{obs}})\ge0\};A^{\mathrm{obs}}\in S\right]\le2\mathbb{E}\left[p_F^+;A^{\mathrm{obs}}\in S\right]\le2\alpha\mathbb{P}(A^{\mathrm{obs}}\in S). \end{aligned}

If P(Aobs∈S)=0\mathbb{P}(A^{\mathrm{obs}}\in S)=0, the claim is immediate. Otherwise, dividing by P(Aobs∈S)\mathbb{P}(A^{\mathrm{obs}}\in S) yields the desired bound.

Anticonservativeness in the Bounded Case

Example A.1. For each design considered in this paper, we give examples with nonbinary potential outcomes in which the level-α\alpha test based on pF+p_F^+ is anticonservative. Similar constructions show that pF±p_F^\pm can also be anticonservative.

For the CRE, take n=6n = 6 with (Yi(1))i=16=(0,0,1/2,1/2,1/2,1/2)(Y_i(1))_{i=1}^6 = (0,0,1/2,1/2,1/2,1/2) and (Yi(0))i=16=(0,0,0,0,1,1)(Y_i(0))_{i=1}^6 = (0,0,0,0,1,1), so τ=0\tau= 0. Consider pF+p_F^+ with α=0.05\alpha= 0.05. Let EE be the event where the treated set is either {3,5,6}\{3,5,6\} or {4,5,6}\{4,5,6\}. On EE, the observed data consists of (1/2,1/2,1/2)(1/2,1/2,1/2) in the treated arm and (0,0,0)(0,0,0) in the control arm. In the FRT reference distribution, the observed difference-in-means value of 1/21/2 is the unique largest value obtainable upon randomization. The one-sided FRT always rejects on this event and P(E)=0.10\mathbb{P}(E) = 0.10, violating the level α\alpha.

For the BRE, take n=6n = 6 and consider 4 units with type (0,0)(0,0) and two units with types (3/4,1/4)(3/4,1/4), (1/4,3/4)(1/4,3/4). By (7), the sampling distribution of τ^cHT\hat{\tau}_{\mathrm{cHT}} is S4/6S_4/6, where SrS_r is the simple random walk at time rr. For every observed assignment, (7) again shows that the reference distribution of the FRT is (S4′+ϵ/2+ϵ′/2)/6(S'_4+\epsilon/2+\epsilon'/2)/6, with ϵ,ϵ′\epsilon,\epsilon' independent Rademacher random variables. On the event S4=4S_4 = 4, which happens with probability 1/161/16, pF+=P(S4′+ϵ/2+ϵ′/2≥4)=3/64<0.05p_F^+ = \mathbb{P}(S'_4+\epsilon/2+\epsilon'/2 \ge4) = 3/64 < 0.05. Therefore the one-sided FRT rejects with probability at least 1/161/16.

For matched pairs, take n=12n = 12 units in 6 pairs. In each of the first four pairs, let the two units have types (1,1)(1,1) and (0,0)(0,0). In the fifth pair, both units have type (1/2,0)(1/2,0), and in the sixth, both have type (0,1/2)(0,1/2). Then τ=0\tau= 0, and the observed treated-minus-control differences in the last two pairs are always 1/21/2 and −1/2-1/2. Consequently, the sampling distribution of nτ^DIM/2n\hat{\tau}_{\mathrm{DIM}}/2 is S4S_4, whereas its FRT reference distribution is S4′+ϵ/2+ϵ′/2S'_4+\epsilon/2+\epsilon'/2. The same calculation as for the BRE gives pF+=3/64<0.05p_F^+ = 3/64 < 0.05 on an event of probability 1/161/16.

References

  1. [1][ACL25] P. M. Aronow, Haoge Chang, and Patrick Lopatto. Fast computation of exact confidence intervals for randomized experiments with binary outcomes. Journal of Econometrics, 251:106056, 2025. 4, 6, 8
  2. [2][AL26] PM Aronow and Patrick Lopatto. Minimax unbiased estimation for finite populations with bounded outcomes. arXiv preprint arXiv:2605.20572, 2026. 2, 3arxiv.org/abs/2605.20572
  3. [3][AM13] Peter M Aronow and Joel A Middleton. A class of unbiased estimators of the average treatment effect in randomized experiments. Journal of Causal Inference, 1(1):135–154, 2013. 2, 3DOI
  4. [4][BB19] Zach Branson and Marie-Abèle Bind. Randomization-based inference for bernoulli trial experiments and implications for observational studies. Statistical methods in medical research, 28(5):1378–1398, 2019. 5
  5. [5][BCRT21] Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife+. The Annals of Statistics, 49(1):486–507, 2021. 6, 21arxiv.org/abs/1905.02928

Paper details

vlnc:W_tywhppx9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

GPT6

View statement
Looking for related papers…
Contents