WorksMathematics
On the Validity of Fisher Randomization Tests under the Weak Null
Abstract
Consider the problem of testing the Neyman weak null of no sample average treatment effect in randomized experiments with binary outcomes. Although the Fisher Randomization Test (FRT) is typically justified under Fisher's sharp null of no individual treatment effect, we show that it is finite-sample-valid for the weak null in three common combinations of study design and test statistic: 1) balanced completely randomized experiments with the difference-in-means test statistic, 2) matched pairs experiments with the same test statistic, and 3) balanced Bernoulli randomized experiments with the centered Horvitz-Thompson test statistic. As an immediate corollary, Fisher's exact test in a balanced completely randomized experiment is valid under the weak null, and so is McNemar's test in a matched pairs experiment. Similar results hold for the one-sided null . Beyond balanced designs, we show that twice the one-sided FRT -value with the same test statistics is valid under . In simulation experiments, the FRTs compare favorably to other exact procedures.
Introduction
The Fisher randomization test (FRT) has been used to analyze the results of experiments for almost a century [Fis30, Fis35, Pit37]. The development of the FRT played a crucial role in establishing random allocation of treatments as a fundamental principle in experimental design. Using solely the randomness of this allocation, without other assumptions on the experimental population, the FRT provides a basis for inference on the effect of treatment. For a historical overview and philosophical discussion of the test, see for instance [Dav08, Din17, ZZ23, Ros26].
The potential outcomes framework [Ney23] gives an elegant framework to reason about the test. Consider an experiment with units, where unit ’s potential outcomes under treatment and control conditions are given by and respectively, fixed and unknown a priori. The sample average treatment effect is the parameter of interest. Upon administering a randomized treatment to unit – where denotes treatment and denotes control – and under SUTVA [Rub80], we observe only the outcome corresponding to the treatment, i.e., , or equivalently, . The FRT proceeds by imputing unknown potential outcomes with the observed data assuming the sharp null hypothesis, rerandomizing treatment to obtain a reference distribution, and comparing the observed data to the reference distribution. A precise definition is given in § 2. As delineated by Welch [Wel37], the usual justification of the FRT comes from Fisher’s sharp null where no unit responds to the treatment:
On the other hand, the scientific question underlying the experiment often involves the Neyman null or weak null representing efficacy on average over the units:
The gap between these hypotheses is substantive. In many settings, we expect at least some units to respond to the treatment even in a null case, due to inevitable heterogeneity among units. Finding examples or classes of FRTs valid under is therefore important. To date, this property has been established in a few settings. One line of work establishes validity under asymptotically [Din17, WD21, CF22], often by considering studentized or pre-pivoted statistics. Caughey et al. [CDLM23] show the validity of the FRT for bounded null hypotheses of the form , for any prespecified constants . More generally, as Ding argues [Din25], few examples of tests which are finite-sample valid under are known. The most notable test is due to Chiba and Rigdon-Hudgens [Chi15, Chi17, RH15], which takes a maximum of FRT -values for all potential outcome tables in . Concentration inequalities also enable valid tests [Fre26, SBF+26], although these are often conservative and can suffer from low power.
This note advances the literature on FRTs and randomization inference by showing that the FRT in three common settings is exactly finite-sample-valid for the weak null. For balanced completely randomized experiments, we show that the one- and two-sided FRTs with the standard difference-in-means statistic are valid for , respectively, for all even . For balanced Bernoulli randomized experiments, the FRT with the centered Horvitz-Thompson statistic [3, 2 Hul26, SDT26] is also valid in finite samples under . The same guarantees are true of the FRT in a matched pairs experiment with difference-in-means, where treatment is allocated with a fair coin in every pair of units. As a byproduct, these results establish the weak null validity of two very classical tests: Fisher’s exact test in a completely randomized experiment (CRE) and McNemar’s test in a matched pairs experiment. Relative to other exact randomization inference procedures [RH15, Chi17], running a single FRT may also be computationally more efficient. In the setting of an arbitrary unbalanced or balanced experiment with the estimators as above, we establish a simple rule of thumb that twice the one-sided FRT -value is valid under the weak null. We show in simulation that these three FRTs compare favorably in their respective experimental settings to tests based on concentration inequalities and the Chiba-Rigdon-Hudgens procedure.
Main Results
Before presenting our results, we collect some notation and recall the precise definitions of FRTs. The set of binary potential outcomes in the experiment will be called a table. Given a table, define for to be the sizes of the principal strata [FR02]. Given a treatment vector drawn from some law called the design, we observe data , . We will refer to data points as events or ones. We consider designs where , the treatment probability, is constant for all . Call the design balanced if the treatment probability is and unbalanced otherwise. Finally, call the design symmetric if , where is the -dimensional vector of ones.
Let be any statistic of the treatment vector and data. The Fisher randomization test generates -values for the Fisher null as follows. Given the observed data , impute the potential outcome table by assuming , i.e., each unit’s unobserved outcome equals its observed outcome. Sample a new treatment vector and set , . Then the one-sided and two-sided FRT -values are defined as
We will often omit the dependence on . Any -value we discuss in the paper will be called valid for at level- if . We adopt the same definition for . We will also call -values with this property Neyman valid, or say that they satisfy Neyman validity.
In this note, we take the test statistic to be the centered Horvitz-Thompson statistic . For experimental designs with treatment probability , the statistic is defined by
The centered Horvitz-Thompson statistic is a special case of a class of estimators introduced in [3]. The estimator has recently been studied in several works [3, 2 Fre26, Hul26, SDT26] in both survey sampling and experimental settings. Although a strength of the FRT is that any test statistic may be accommodated, this choice is particularly natural and common: is equivalent to difference-in-means in complete randomization and matched pairs, one of the most widely-used statistics for these experimental designs.
Completely Randomized Experiments. Our main results will show that the FRT based on is Neyman valid for balanced complete randomization, Bernoulli randomization, and matched pairs. We begin with the balanced completely randomized experiment (CRE). In a balanced CRE, with even, units are selected uniformly at random to be treated. Since is equivalent to the difference in means estimator in a CRE, we discuss them interchangeably. We have the following result.
Theorem 2.1. With a balanced CRE, is valid under the weak null whenever . The two-sided FRT -value is valid under , for all .
The FRT with difference-in-means in a CRE is known to be equivalent to Fisher’s exact test for tables [DD16], one of the oldest tests in statistics [Fis30]. It is defined as follows. Let be the total number of events observed in the data. Let be the number of events observed in the treatment group. Under , has a hypergeometric distribution . Then Fisher’s exact test generates a -value defined by . The two-sided case is analogous. Theorem 2.1 extends this validity to the weak null .
Another approach to obtain exact tests for based on randomization inference with CRE and is provided by the procedures of Chiba and Rigdon-Hudgens [Chi15, Chi17, RH15]. Let be a vector of individual treatment effects compatible with the observed data, satisfying . Let be this set of vectors. Let be the two-sided FRT -value assuming the null . Taking the worst-case over yields a -value which is valid for :
Throughout this note, we will refer to this test as the Chiba-Rigdon-Hudgens procedure. A similar procedure can be obtained for and with the one-sided FRTs. These approaches require the computation of multiple FRTs, which can be prohibitive in some cases. For the balanced CRE, significant progress has been made. Algorithmic advances [LD16, ACL25] show FRTs suffice to compute an exact confidence interval for . Specializing to testing a single null value , the results of [1] show that many FRT procedures suffice. Theorem 2.1 shows that, in fact, one FRT is enough to obtain a valid -value. Moreover, the two-sided -value is at least as powerful as (6) by construction and strictly more powerful for some alternatives, as simulation results show in Figure 1.
Figure 1. Results on CRE for two-sided tests when . Power is computed using 200,000 reassignments of the CRE; denotes the fraction of units with and is the sample average treatment effect. is computed using the speedups of [1].
Bernoulli Randomized Experiments. Next, we consider balanced Bernoulli randomized experiments (BRE) in which treatments are administered via fair coin flips: , are independent Bernoulli random variables with success probability . Due to operational constraints, BRE appears frequently as the default design in A/B testing and online experimentation [PASJS+19, KTX20]. Our second main result shows the validity of the FRT with this test statistic for Bernoulli randomized experiments.
Theorem 2.2. Under a BRE, the FRT -value constructed using the centered Horvitz-Thompson statistic is valid for the weak null whenever . For every , is valid for .
Randomization inference for the BRE appears to be less developed than for the CRE and for matched pairs designs. The most relevant contributions are work by Branson-Bind for the sharp null [4] and recent work by Zhang [Zha26] on exact confidence intervals for . The work by Zhang shows for the Horvitz-Thompson estimator, different than (5), the Chiba-Rigdon-Hudgens procedure of (6) can be computed using at most two FRTs.
Matched Pairs Experiments. Lastly, consider the setting of a matched pairs experiment. We consider a pool of units, which are grouped into pairs . We will index potential outcomes by , , . In a matched pairs experiment, treatment is allocated independently at random in each pair, with exactly one unit receiving treatment. With drawn i.i.d. , we observe data
The SATE is often estimated using the difference-in-means estimator , which is equal to the centered Horvitz-Thompson estimator (5). Our third result shows validity in this case.
Theorem 2.3. With a matched pairs experiment and the difference-in-means statistic, is valid under the weak null whenever . The two-sided -value is valid under , for all .
It turns out that the FRT -values , are equal to the one- and two-sided -values obtained from the exact binomial version of McNemar’s test [Ros17, p. 90], [Din17]. McNemar’s test [McN47] is a classical procedure used to analyze paired data. In a matched pairs experiment, call a pair discordant if is or . Let be the total number of discordant pairs, and among these, let be the number of pairs in which the treated unit is 1. Under , . The test rejects if is more extreme than the appropriate quantiles, so it is valid conditional on . Theorem 2.3 shows Neyman validity, unconditionally, of McNemar’s test. This appears to be a novel conclusion.
Several recent works consider exact inference for the matched pairs experiment. Li, Spertus, and Stark [LSS25] develop the analogs of two confidence interval procedures from [RH15] in general stratified experiments. This includes the inversion of compatible randomization test -values as in (6) as well as Bonferroni combinations of hypergeometric confidence intervals. For matched pairs designs, Zhang [Zha26] shows that computing Rigdon-Hudgens confidence intervals for the matched pairs design has time complexity . Specializing Zhang’s results to testing, (6) can be computed with at most two FRTs. By design, are at least as powerful as . [Wil26] develops the analog of the Rigdon-Hudgens attributable effects method by giving analytic solutions to the worst cases in the component randomization tests. Finally, Fogarty [Fog23] also develops sensitivity analysis procedures for the weak null in matched designs.
Unbalanced Experiments. For complete and Bernoulli randomization, it is fairly common to run unbalanced experiments where the treatment probability is . The proof techniques of Theorems 2.1, 2.2 carefully leverage the symmetry of the balanced case and do not extend naturally to the unbalanced case. Instead, we show that a simple, practically appealing fix works: multiply the FRT -value by 2.
Theorem 2.4. Suppose is exchangeable. Then .
By the same argument, with denoting the analogous lower-tail FRT -value, is a valid -value for . The proof relies on making a perhaps unexpected connection between the analysis of FRTs and tournaments, analogously to the analysis of the Jackknife+ [5]. Since only the exchangeability of the experimental design is used, we expect that the constant 2 in the guarantee can be sharpened to a constant depending on the treatment probability . Our previous results show , for their respective designs. We leave a sharp characterization of to future work.
Simulation Results
For each experimental design, we compare the two-sided FRTs to other exact tests for when . One important class consists of tests based on concentration inequalities, developed in recent work by Freidling and Sandoval et al. [Fre26, SBF+26], which covers and every design in this note. Broadly, the results demonstrate that the FRTs are more powerful. For the CRE, we compare to the Chiba-Rigdon-Hudgens -value using the computational speedups from [1] along with the Barber-Hoeffding and empirical Bennett intervals from [Fre26, Prop. 4.1, 4.4]. For the BRE, we compare to with the Horvitz-Thompson estimator, using Zhang’s speedups [Zha26]. We also use tests based on the sub-Bernoulli and empirical Bennett intervals for the centered Horvitz-Thompson estimator from [Fre26, Prop. 3.5, 3.6], which builds on [SBF+26].
For each design, we consider three different simulation suites. The first is modeled after clinical trials for rare outcomes. We assume the treatment effect is small, 0.05, and the control baseline —fraction of units with —is small, contained in . The fraction of units harmed by the treatment is half of . The second suite considers a situation where control events are more common; is 30%, varies in , and the fraction of units harmed is 20%. Finally, the third suite mimics the second suite, except we impose a monotonicity constraint that there are no units harmed. In each of these suites, we take to be multiples of 20 up to 300. If is the fraction of units harmed by the treatment, then the sizes of the principal strata may be expressed in terms of as .
Figures 1 and 2 show the power of the comparator procedures in each of these simulation settings, with . As all methods in the 3 experimental designs are exactly finite-sample-valid, we do not plot the Type I error. Broadly, the FRT methods outperform the other methods for exact inference, with the concentration inequality based tests being fairly conservative in this binary setting. For the CRE, is indeed different from the FRT -value as can be seen in the first column of Figure 1, which highlights a stark gap between the two methods in the rare outcome setting. Outside of this case the two -values are quite similar. Interestingly, the performance of the FRT in panel (A1) is quite promising, suggesting some applications to trials with rare binary outcomes. In the BRE, the FRT with continues to outperform the concentration inequality based tests along with the exact method of [Zha26]. The performance for rare outcomes is much more modest than that of the FRT in a CRE with difference in means.
For the matched pairs experiment, we consider the same three simulation suites and use a uniformly chosen pairing between units for each , which is fixed across all reassignments of the design. We compare with the exact RH-type procedure of [Zha26] and tests using the matched pair concentration inequalities in [Fre26, Prop. 3.6, 3.7]. Finally, we also compare to the method of Wilson [Wil26]. The results are pictured in Figure 3. The paired FRT behaves similarly to , except in the rare outcomes setting where the paired FRT is much more powerful. Wilson’s method is also quite competitive, losing only to the paired FRT in the rare outcomes setting and beating the concentration-based procedures in all settings.
Figure 3. Results for matched pairs experiments for two-sided tests when . Power is computed using 200,000 reassignments of the matched pairs design. Matching between units is selected uniformly at random and fixed over the 200,000 runs, for each . Again, denotes the fraction of units with and is the sample average treatment effect.
Matched pairs designs are often motivated on the basis of blocking on observed information to reduce variation. In Figure 4, we detail an additional simulation setting where the pairs are determined by matching on a scalar covariate. For each unit in the experiment, we sample and generate . Independently conditional on , is generated as . These constants are chosen so that the resulting superpopulation mimics the rare outcome simulation suite described previously, with , and the fraction of units harmed being 0.05. For each , we sample from the Beta distribution, sort units by , and pair adjacent ones. The potential outcomes are generated as above and are held fixed when reassigning the treatment indicators in the matched pairs design. The right panel of Figure 4 shows the unconditional power of the previously compared methods, averaged over 100 samples from the superpopulation. The FRT remains the most powerful test; the CRH procedure and Wilson’s attributable effects procedure behave similarly.
Figure 4. Results for matched pairs experiments, using two-sided tests, . Pairs are obtained by matching on an informative covariate generated iid from . Potential outcomes are generated from as described in the text. The left panel shows the density of , and the right panel shows the power of each method, averaged over 100 Monte Carlo redraws of the superpopulation. 95% pointwise confidence bands are displayed for the variation due to superpopulation draws.
Discussion
In this note, we showcase three common circumstances in which the Fisher randomization test is valid under the weak null, allowing users of these tests to significantly upgrade their inferential conclusions. Generalizing and unifying the proof techniques in these theorems may give insights into general conditions for exact finite-sample validity of FRTs. Proving Neyman validity beyond the binary case remains a challenge. In Example A.1, we show that the FRT can be anticonservative when the potential outcomes are non-binary. We raise the problem of finding powerful and tractable FRTs, or randomization tests more generally, for bounded outcomes as an important direction for future work.
Acknowledgments. AI assistance (GPT6) was used in the preparation of this work, including generating figures, suggesting arguments, checking proofs, and revising the article. All proofs were verified by the authors and we take responsibility for any errors.
Figure 2. Results for BRE for two-sided tests when . Power is computed using 200,000 reassignments of the BRE; denotes the fraction of units with and is the sample average treatment effect. The -values values are computed using the speedups of Zhang [Zha26].
Proofs
In all of the proofs, we will freely use , and , to denote independent copies of Rademacher and Bernoulli random variables. Let be the simple random walk at time , with iid Rademacher random variables. For brevity, we will write simply as .
At a technical level, the approach for each design will be to show that the sampling distribution of the statistic—the law of —is more concentrated than the reference distribution—the law of —for all realizations of . At a heuristic level, we should expect this to be true for the centered Horvitz-Thompson estimator on the following grounds. For a balanced design, a brief algebraic manipulation shows that may be rewritten as
Under and a symmetric design, is symmetrically distributed around zero. In particular, and the summands are symmetric around zero, taking values in . Notice that whenever is or . Under , each of the summands is nonzero; under any potential outcome table in , some of these summands may be zero. Therefore, we expect that the tails of the reference distribution should dominate those of the sampling distribution. Proposition A.1 shows that it suffices to check this tail dominance conditional on a well-chosen statistic .
Proposition A.1. Consider any symmetric design . For each table in , suppose there exists a random variable such that for all ,
Then, for all , . If further is symmetric around zero, then .
Proof. We first prove the guarantees for . If is symmetric, then (7) shows that is symmetric around zero conditional on . For , implies by this symmetry. Let be the conditional sampling distribution. By (8) and the definition of the FRT -value, . Then . Take expectations to conclude the validity of .
By assumption, is symmetric around zero conditional on . Multiplying each side of (8) by two and using this symmetry and that of , we obtain
The same argument shows the conditional validity of . To extend to , we exploit the following representation of :
Then, with ,
and . For any table in , there are at least units with potential outcomes . Take one of these units and switch its potential outcomes to to obtain a table . Then the difference of under and is given by . Thus, decreases when moving from to , and so decreases. This yields . Iterate this guarantee until a table is obtained. Then, use the first part of the proposition. □
Before treating each of the different designs, we record the following useful tail comparison lemma.
Lemma A.1 (Random walk tail comparison). For any nonnegative integers of the same parity and an integer threshold of the same parity as , .
Proof. It suffices to show that for every and . By conditioning on the last two increments of , we have
Thus, is equivalent to , or . This is true by unimodality of . □
Proofs for CRE
Throughout this section, we will refer to units for which , as treated ones, , as control ones, and so forth for the remaining combinations. Under a CRE, let , be the number of treated ones and control ones respectively. Let be the total number of ones in the data. Let denote the hypergeometric law with total urn size , number of successes , and number of draws .
Given observed data vector and treatment vector with many treated ones and many control ones, call a table compatible with , if for all . Let be the number of potential outcome tables compatible with that have strata . Lemma A.2 shows that only depends on through .
Proof of Theorem 2.1. We will apply Proposition A.1 by checking (8) with . Under a CRE, is equal to . It is easy to see that the reference distribution, the law of , is equal to the law of
which only depends on through . We check that
Symmetry of both and shows that the one-sided guarantee holds. To prove (10), we consider the likelihood ratio of the conditional distributions of and , given . First, we calculate . Complete randomization is combinatorially equivalent to uniformly permuting the units and treating the first units in the permuted order. We count the number of permutations for which the observed data consists of many ones among the first units and many ones in the remaining units. There are many ways to select the locations of the treated ones and many ways to select the locations of the control ones. There are many tables with the proper strata that are compatible with the chosen locations. Finally, the units may be permuted within each stratum, giving a factor of . Thus,
Write as shorthand for . By rewriting the hypergeometric mass function, . The likelihood ratio then satisfies
Proposition A.2 shows that for every also and when , . Therefore, is nonincreasing as increases. Because , the likelihood ratio crosses at most once, from above to below, as ranges over . Call this crossing point , so , and , .
Fix any . Then
When , redo the above derivation with the lower tail event and take complements. This proves (10). Proposition A.1 gives the one and two-sided conclusions.
Lemma A.2. The completion count only depends on through .
Proof. Take any two data vectors with the same number of treated ones and control ones . Take a permutation for which and . For any table which is compatible with , applying the permutation to the rows of yields another table compatible with . This mapping is a bijection between the set of tables compatible with and those compatible with . Therefore the number only depends on .
Proposition A.2. Suppose , , and . Then . For any , .
Proof. Let be the set of tables with principal strata counts and first units treated, such that and for . These are compatible with the observed data and by Lemma A.2, . We will exhibit an injection .
To define the injection, fix a table . See the left table of Figure 5 for a running visualization. We define five disjoint subsets of its rows. Let be the row indices of the last treated 1 and the first control 0. Let be the treated zeros and the control ones . Finally, let consist of the first treated ones and be the first control zeros, excluding the row in : . Because , and .

Figure 5. Example of the injection with and . The ordered pairs are , , , , , and . The first three pairs are swapped and flipped, with changed entries in bold font. Grey entries are unobserved.
The idea is to swap the rows in and reverse their potential outcomes to define a table in . The issue is that the principal strata counts will not be maintained. We compensate for this by further swapping rows between and . To this end, we define a pairing of row indices by 1) pairing the rows in , 2) pairing the rows in with the corresponding rows in , in increasing row order, and 3) similarly with to . Define a list of row pairs , by concatenating the matched pairs in , then those in followed by . Explicitly,
Note that this pairing forces
Define
to be the running imbalance between the pairs in this list. Let be the first index for which this running sum hits zero. This index always exists. By construction of the set , . Because and contains all rows with type , . Finally, (14) forces for .
Define the injection as follows. For each , swap the rows , then swap the potential outcomes within each row. This results in a table with rows defined by
and all other rows unchanged. The swaps exchange types and while preserving and . Since , the numbers of all principal strata are preserved. The observed entries only change for the rows in . Thus .
Finally, we show this map is an injection. Consider a table that is equal to for some . Given , flipping the first treated zero to 1 and last control one to 0 yields the same observed data pattern as , allowing us to recover the same sets . For all , we have . Thus, . Applying the same swap-reverse transformations as in to therefore recovers the original table .
Symmetry. For any table in , create a new table by switching each unit’s outcomes from to and flipping treatment and control. Because , this creates another table in . This transformation is a bijection. Therefore .
Proofs for Bernoulli Randomized Experiments
Proof of Theorem 2.2. We will show that (8) holds with . Recall that denotes the simple random walk at time . By (7), under , is equal in distribution to . Using (7) again, . Since , the reference distribution of the FRT is equal to , regardless of . Under , , so . Lemma A.1 shows
Apply Proposition A.1 to conclude.
Proofs for Matched Pairs Experiments
Proof of Theorem 2.3. In the matched pairs experiment, denote the summands in the difference in means estimator by , or equivalently,
Let be the summand in the resampled estimator, which by (17) is equal to . Thus,
where the are iid Rademacher random variables, independent of everything else. Let be the observed number of nonzero pair differences. Then, by (18),
the random walk at time .
We now turn to understanding the law of . Using casework, we can characterize the distribution of as a function of the 16 possible combinations of potential outcomes in pair , . For example, suppose the table for pair has the form
caption:
:::
for . With probability , the observed data is , which forces . The same is true in the other case when is observed. When the table is of the form
, a fair Bernoulli random variable. When it is of the form
. The last tables to handle are
In the first two cases, . In the last two cases, and respectively. Table 1 summarizes the distributions of as a function of the potential outcomes of units .
| Unit 1 / Unit 0 | ||||
Table 1. Distributions of for all sixteen binary potential-outcome tables of a matched pair . Here, and is a Rademacher variable.
Define . Inspecting Table 1, the sum of the individual treatment effects for the units in the pair satisfies
We will construct a partition of the pairs into disjoint sets for some , as follows. Let be the set of all pairs for which has the same distribution as . Define the same for . Let be the sizes of these sets. Under , (20) shows
Suppose . For as many pairs as possible, match them with other pairs . For such a pair , let this be another set in . The number of leftover pairs in is . For every pair , match it with a pair and add to the partition . By (21), there are unmatched pairs left over in . For every 2 leftover pairs in , group them with an unmatched pair and add to the partition . Finally, for all other pairs in , add them as singleton sets into . For every set , the sum is distributed as one of the following:
where are independent Bernoulli random variables. Moreover, the sums are independent across groups . Construct an analogous partition if . In this case, instead of being a possible law, the option is . Crucially, in all of these cases, the distribution is supported on and symmetric around zero: for each ,
for an independent Rademacher random variable . Therefore, conditional on ,
Putting our analyses of the sampling distribution and reference distribution together, Lemma A.1 shows that for every ,
Indeed, using the fact that . By the triangle inequality, . This shows (8) with ; applying Proposition A.1 completes the proof.
Proofs for Unbalanced Experiments
Proof of Theorem 2.4. Let be the treatment probability of the design. Under , there exists a pairing between all units with potential outcomes and a subset of the units with outcomes . Let be the permutation which swaps the units in each pairing and leaves all other units untouched. Define for any vectors the difference
Then the exchangeability of gives
Following the coverage proof of the Jackknife+ [5], interpret as the scores in a two-way match between vectors in a tournament played between all vectors in . Declare to be a winner in the match if , and to be a winner if . Observe that there can, in principle, be zero, one, or two winners. Let be the set of vectors whose opponent wins with probability at most under :
Then is equal to . We claim that there always exists at least one winner in a match, via the bound
Consider a pair where and . Then the contribution of indices to is equal to
The last equality follows because and for , so both terms vanish by the identity for all .
For any single unit , the contribution of unit to is . When , the contribution is zero; when , the contribution is . This proves (28). Finally,
If , the claim is immediate. Otherwise, dividing by yields the desired bound.
Anticonservativeness in the Bounded Case
Example A.1. For each design considered in this paper, we give examples with nonbinary potential outcomes in which the level- test based on is anticonservative. Similar constructions show that can also be anticonservative.
For the CRE, take with and , so . Consider with . Let be the event where the treated set is either or . On , the observed data consists of in the treated arm and in the control arm. In the FRT reference distribution, the observed difference-in-means value of is the unique largest value obtainable upon randomization. The one-sided FRT always rejects on this event and , violating the level .
For the BRE, take and consider 4 units with type and two units with types , . By (7), the sampling distribution of is , where is the simple random walk at time . For every observed assignment, (7) again shows that the reference distribution of the FRT is , with independent Rademacher random variables. On the event , which happens with probability , . Therefore the one-sided FRT rejects with probability at least .
For matched pairs, take units in 6 pairs. In each of the first four pairs, let the two units have types and . In the fifth pair, both units have type , and in the sixth, both have type . Then , and the observed treated-minus-control differences in the last two pairs are always and . Consequently, the sampling distribution of is , whereas its FRT reference distribution is . The same calculation as for the BRE gives on an event of probability .
References
- [1][ACL25] P. M. Aronow, Haoge Chang, and Patrick Lopatto. Fast computation of exact confidence intervals for randomized experiments with binary outcomes. Journal of Econometrics, 251:106056, 2025. 4, 6, 8
- [2][AL26] PM Aronow and Patrick Lopatto. Minimax unbiased estimation for finite populations with bounded outcomes. arXiv preprint arXiv:2605.20572, 2026. 2, 3arxiv.org/abs/2605.20572
- [3][AM13] Peter M Aronow and Joel A Middleton. A class of unbiased estimators of the average treatment effect in randomized experiments. Journal of Causal Inference, 1(1):135–154, 2013. 2, 3DOI
- [4][BB19] Zach Branson and Marie-Abèle Bind. Randomization-based inference for bernoulli trial experiments and implications for observational studies. Statistical methods in medical research, 28(5):1378–1398, 2019. 5
- [5][BCRT21] Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife+. The Annals of Statistics, 49(1):486–507, 2021. 6, 21arxiv.org/abs/1905.02928



