WorksMathematics
Principal-agent hypothesis testing under adaptive experimentation
Abstract
Principal-agent hypothesis testing studies statistical contracts in which a principal rewards an agent according to data the agent generates. Existing work focuses on the single-shot setting, where the agent produces a single dataset. We extend the framework to sequential settings, where evidence accumulates over time, rewards may be deferred to the end or disbursed incrementally, and the agent may stop the contract at a strategically chosen stopping time. We call a contract null-deterring if no ''null agent'' (those whose data-generating distribution lies in the null hypothesis) can profit in expectation under any stopping rule, and our main conceptual result shows that such contracts are characterized by e-processes. We then turn to the agent's best response and identify the optimal stopping rule through the Snell envelope of the reward process. We develop simpler, more interpretable approximations that often require the agent to have only partial knowledge of its distribution.
Introduction
Consider a drug regulator (e.g., the FDA) that controls access to a market. A pharmaceutical company can pay a cost to run a clinical trial of a new treatment. Based on the outcome of the trial, the regulator decides how widely the treatment may be marketed. There are competing interests at play: the regulator wants effective treatments to reach patients while keeping ineffective ones off the market, whereas the company wants its treatment authorized as widely, and as quickly, as possible.
This scenario is an example of a principal-agent problem: the strategic interaction between a principal, who controls access to a good, and an agent who is vying for access. Such examples are not hard to find in practice: recommendation algorithms, internal experimentation platforms at tech companies, and venture funding being a few common ones. Given their ubiquity, principal-agent problems have long been studied in economics (e.g., [19, 39, 47, 18]).
Recently, motivated by the FDA example above, Bates et al. [5] introduced a framework to analyze such dynamics from a statistical perspective, which they call principal-agent hypothesis testing. The framework is based on statistical contracts: protocols which guide the interaction between the principal and agent. The agent pays a cost to enter the contract and then generates data from a distribution—a distribution that is private and unknown to the principal, but possibly known to the agent. The principal allocates resources (rewards) based on inference made from this random sample. Importantly, the initial cost and reward mechanism was agreed to a priori by the agent, before generating the data.
In the example above, the trial outcome is a random sample drawn from the agent’s distribution. The principal does not know the treatment’s effectiveness beforehand. The reward, perhaps a license to profit or the scope of market authorization, is a function of the observed outcomes. In general, the reward may be profit, market share, a license, a binary accept/reject decision, etc.
Agents are assumed to be partitioned into null agents (e.g., companies with ineffective treatments) and non-null agents (those with effective treatments). Formally, there is a parameter space partitioned into (nulls) and (non-nulls) and each agent has a private type/parameter . Their data is drawn from . With this setup in mind, the principal-agent hypothesis testing framework of Bates et al. [5] is based on the following statistical contract:
Single-shot statistical contract [5]
The principal publicly announces a cost and a menu of reward functions.
An agent with hidden parameter may pay a cost to enter the contract. Upon entry, the agent selects a reward function .
The agent then generates data and receives reward .
The central design problem in statistical contract theory is to control the expected rewards allocated to null agents while allowing non-null agents to earn rewards commensurate with the evidence they submit [25]. Bates et al. [5] show that null-deterring contracts1—those ensuring that null agents cannot earn a positive expected profit—must consist of rescaled e-values, a property that follows almost directly from the definition (and motivation) of e-values. They further show that e-values characterize maximin optimal contracts, thus providing actionable advice for any principal looking to design statistical contracts.
There is a drawback, however. The contract above assumes that a single dataset is produced by the agent. But in many applications it is natural to consider sequential experiments. Data may take time to procure, or the agent may wish to obtain partial rewards based on the results thus far. Clinical trials are a paradigmatic case. They run over months or years, so rather than wait for the trial to conclude, a regulator may grant accelerated or conditional approval on interim evidence and broaden the authorization as more data arrive. In fact, sequential allocation and experimentation is now pervasive in the technology industry and elsewhere (e.g., Spotify [41], Netflix [30], Eppo [31], Growthbook [20], Abridge [34], Adobe [1], as well as in government and public policy [22, 23]).
Our goal in this work is to extend the single-shot statistical contract of Bates et al. [5] to the sequential setting, thus enabling adaptive experimentation and interim decision-making. Our contributions are as follows:
We propose two versions of a sequential statistical contract: one with deferred rewards and one with incremental rewards. Extending the notion of null deterrence from the single-shot setting, we define null-deterring sequential contracts as those where a null agent’s expected reward is at most the entry cost under every parameter and stopping strategy.
We classify the set of null-deterring contracts under both reward schedules using e-processes. For incremental reward schedules in particular, we show how to construct such contracts using functions called adjusters [14, 15, 42], which also arise in preserving accumulated evidence and combining e-processes [8].
We analyze the agent’s best response under both reward schedules, deriving utility bounds and simple policies with explicit performance guarantees. We also develop approximations that require less knowledge of the agent’s distribution and compare these policies experimentally.
We note that Bates et al. [5] also gave a sequential version of the single-shot setup above, but their time-horizon must be known in advance. That is, the contract can only proceed for some fixed number of rounds determined a priori. We give more detail on the differences between their sequential contract and ours in Appendix § 9, but here let us summarize three ways in which our contract differs: (i) we allow stopping rules for both the principal and the agent, (ii) we study two distinct reward schedules, and (iii) we handle unbounded horizons. That said, their sequential contract also has benefits over ours, such as the ability to consider per-round costs. We thus view our contribution as extending the single-shot contract in an alternative way to that originally proposed by Bates et al. [5].
Code is available at https://github.com/bchugg/statcontracts.
Related work
Since the original article of Bates et al. [5], there have been several follow-up works on principal-agent testing. Chen et al. [6] study a similar setup but focus on characterizing the principal’s optimal p-value threshold under strategic participation of the agent. Shi et al. [43] study the agent’s optimal response under general concave utility functions, and then later examine the situation of a principal interacting with multiple heterogeneous agents and focus on the issue of adverse selection [44]. Bates et al. [4] study a Bayesian version of the principal-agent problem, and Chugg et al. [9] study a multi-agent version.
Recently, Artola Velasco et al. [2] also studied principal-agent hypothesis testing in a sequential setting, though with different goals. In their model, the principal seeks to maximize social welfare by subsidizing a fraction of the agent’s experimentation cost, and the agent is uncertain about efficacy and updates a prior private belief. Despite these differences, the two works share key ingredients: both rely on e-values [37], and both characterize the agent’s optimal strategy via dynamic programming. We therefore view their work as complementary to ours, and together they highlight the many possible extensions of principal-agent problems to more dynamic settings.
Nguyen et al. [32] also study sequential statistical protocols in which an agent chooses whether to submit a project. The principal observes public testing outcomes and uses future access to testing to influence these decisions. In this setup, the agent repeatedly decides whether to submit a new project, whereas ours accumulates evidence about a fixed type. Velasievic et al. [48] use anytime-valid tests to design contracts that protect users of AI services from a provider covertly downgrading its model.
Our analysis of the agent’s parameter choice is closely related to recent work of Clerico et al. [11] on sequential testing with preferences over rejection times. Their EDO criterion coincides with our CGF-optimal criterion, and our bounds adapt their single-threshold bounds. The settings differ, however. They choose parameters to maximize a discounted reward for rejecting the null at a single significance level. Our agent faces a contract whose reward grows with the evidence and is not always paid out all at once. The bounds must therefore hold across many evidence levels and be combined with the agent’s stopping decision.
Contract theory has a long history outside of the particular statistical framing studied here. Its primary goal is to study the interactions of agents, at least one of whom has private information. We refer to Lanchim[a][29] and Laffont and Martimort [28] for overviews. The adjacent field of algorithmic (as opposed to statistical) contract theory [17] is also concerned with principal-agent problems, but emphasizes the computational aspect as opposed to the statistical aspect. There, the goals are to learn optimal payout vectors and to analyze the computational complexity of doing so [12, 7, 51].
Technical preliminaries
Let be a measurable space. An e-value for a set of probability distributions on is a nonnegative random variable such that for all . For readers unfamiliar with e-values, it is helpful to think of e-values as statistical cousins of p-values. While developed for similar purposes (in particular, to provide a measure of evidence against the null [10]) their definition in terms of expectations instead of tail probabilities gives them useful properties that p-values do not have. We refer to Ramdas and Wang [37] and Grünwald et al. [21] for more.
As our motivation in this work is to consider sequential settings, we will require the sequential version of an e-value, the e-process. To define it, we suppose that the space admits a filtration . Then an e-process for is an -adapted sequence of nonnegative random variables (that is, is -measurable for all ) such that is an e-value for at all stopping times . Recall that a stopping time is a random variable such that is -measurable for all . That is, at time , one can always determine whether the stopping time is now. All logarithms throughout the paper are natural logarithms.
Sequential Statistical Contracts
Our first order of business is to determine how to extend the reward menu in the single-shot contract to the sequential setting. Instead of a set of functions, we will have a set of processes. More specifically, will consist of a set of process families , each potentially parameterized by a sequence . To make things concrete, consider the process family defined by
where we use as shorthand for . When opting into the contract, the agent would choose a specific reward process but specify only initially. Then, before round , the agent chooses based on past observations. This example was not pulled from whole cloth: the underlying exponential process, initialized at one, is a nonnegative supermartingale for iid data with mean bounded in . (More generally, for , it is a nonnegative supermartingale whenever the data are iid, have nonpositive mean and are -sub-Gaussian, e.g., with .) As we’ll see below, nonnegative supermartingales result in null-deterring contracts.
We will consider two reward schedules: deferred and incremental. Under the deferred schedule, the agent receives payment only when the experiment ends. Under the incremental schedule, the agent receives rewards along the way, with each period’s reward becoming immediately available. Incremental rewards are natural when the rewards are themselves immediately actionable. A pharmaceutical company may want to take their drug to a corner of the market, or begin profiting, based on the evidence thus far. Deferred rewards arise in settings where evidence is accumulated sequentially, but the relevant transfer or decision is made only once, at the end. Examples include regulatory approval, prize competitions, grant funding, or venture financing. In such settings, the agent may gather evidence over time, while the eventual reward is determined only when the data gathering stops.
The reward schedule also restricts the reward process. Under the incremental schedule, previously disbursed rewards cannot be clawed back, so the cumulative reward should be nondecreasing over time. Under the deferred schedule, only the terminal reward is given. Thus, the total reward may fluctuate up and down according to the evidence.
With the example of the reward process above and the two reward schedules in mind, let us define a sequential statistical contract.
Sequential statistical contract
The principal publicly announces a cost , a menu of reward processes , a stopping time (which may be infinite), and the type of reward schedule.
If the agent pays to opt-in, they choose a reward process from . Set .
For
The agent chooses parameter(s) if applicable
A trial is run producing data , where is the agent’s type
The total reward is calculated. On the incremental reward schedule, the agent is awarded , ensured to be nonnegative.
If the contract is ended. On the deferred schedule, the agent is awarded .
The agent decides whether to stop or continue.
If they stop, on the deferred schedule, they are awarded .
Several remarks are in order. First, in addition to allowing the agent to stop the contract, we also allow the principal to commit ex ante to a stopping rule. This is useful in a variety of settings. For example, the principal may face a finite budget, a fixed review horizon, or some other operational constraint that limits how long the experiment can continue. Of course, the stopping rule need not be so explicit; it may depend on the growth rate of some utility function, for instance.
Second, we do not impose per-period costs on the agent. That is, once the agent has paid the entry cost , they do not make additional payments each round in order to generate further data. We interpret as the upfront cost of mounting the research program: setting up the experiment, building infrastructure, hiring personnel, obtaining approval, and incurring associated opportunity costs. This differs from the sequential model of Bates et al. [5]; see Appendix § 9.
More broadly, the costs of continuation need not be symmetric across the two parties. For instance, once the FDA has authorized a clinical trial, their marginal cost of allowing the study to continue may be quite low. But the agent may still face substantial costs from administering the treatment and monitoring patients.
Null Deterrence in Sequential Contracts
Following the intuition of Bates et al. [5] in the single-shot setting, the principal would like to design a menu such that, regardless of the agent’s strategy (i.e., their choice of , their optimization of if applicable, and when they choose to stop), the expected reward allocated to a null agent is at most . In the single-shot setting, null deterrence (ND) is the condition that for all . We extend this to the sequential setting as follows.
Definition 1. The sequential contract is null-deterring (ND) if
We may sometimes discuss null deterrence not of an entire contract itself, but of a single process
. In particular, if satisfies for all and all stopping times ,2 we will say that it is ND. Importantly, null deterrence does not require the principal to know the agent’s utility function. The condition holds under every agent strategy and it is thus an asymmetric guarantee, prioritizing the principal over the agent.
On the deferred reward schedule, (ND) should remind us of the definition of an e-process. Indeed, we will show below that all ND contracts on the deferred reward schedule are characterized by e-processes.
Things are not quite as simple on the incremental reward schedule, however. Many, if not most, e-processes are not almost surely increasing, and thus do not satisfy for all . Instead, we can construct ND contracts by using the concept of an adjuster [42, 8, 14, 15], which is a nonnegative, nondecreasing, right-continuous function with , satisfying
The adjuster is admissible if (3) holds with equality and [42]. Adjusters have their roots in the game-theoretic interpretation of e-processes, where they are viewed as the wealth process of a statistician playing a game against nature [38]. In such a setting, it is sometimes convenient to ensure that the wealth doesn’t decrease over time, which is what adjusters were developed to accomplish.
More formally, an adjuster satisfies the following guarantee: For any test supermartingale (meaning a nonnegative supermartingale with initial value 1) there exists a test supermartingale such that almost surely, where [42 Section 7]. This, together with the optional stopping theorem, implies that is an e-process, a fact which will undergird our results.
With the concept of adjusters in hand, we can state the following result regarding ND statistical contracts.
Proposition 1. Let be a sequential statistical contract. For the deferred reward schedule, is ND if and only if is an -process for for all . For the incremental reward schedule, suppose every is nondecreasing. Then is ND if and only if for all and all , where . In particular, is ND if is an -process for for every . Further, given an -process for , is ND for any adjuster , where .
Faced with Proposition 1, a natural question is whether all increasing -processes can be represented via adjusters. This turns out to be false. Appendix § 8.§ 3 gives a bounded nondecreasing -process that cannot be represented, or even dominated, by an adjusted running maximum.
Nevertheless, adjusted -processes will play an important role in our analysis in § 4, so let us linger on their details here. To foster some intuition, consider the following scheme. The principal fixes evidence levels , which we call milestones, and promises a milestone payment the first time the -process (i.e., the evidence) reaches . The cumulative rewards are then
where is the running maximum. When the agent is null, the probability that the evidence ever reaches is at most by Ville’s inequality [49, 40], so a payment of at level costs at most in expectation. The constraint in (4) thus caps the expected payout to a null agent at the entry cost , which we call the null-payout budget, and it is exactly (3) for . On this view, an adjuster is simply a choice of how to spend the null-payout budget across evidence levels, with milestone using the fraction .
The same reading applies to any adjuster, not only those with discrete milestone payments. Since is nondecreasing and right-continuous,
and Tonelli’s theorem turns (3) into the budget constraint
The contract therefore pays a baseline at the end of the first round, whatever the data, and pays the first time the evidence reaches . The baseline is reached with probability one, so it costs its full value, whereas a payment at level costs only the fraction of its value.
Shafer et al. [42] suggested the class of admissible adjusters for . Other examples include [8] and [27]. We can also consider mixing over :
The uniform mixture recovers the adjuster used by Choe and Ramdas [8].
Figure 1 compares the growth rates of these adjusters. Although all are admissible, they allocate their null-payout budget differently (right panel). The baseline reward matters. In particular, and , so these adjusters return half of the entry cost in round one, while the KV adjuster returns and the square-root adjuster has no baseline payment. The principal’s choice of adjuster thus encodes a preference over where on the evidence scale to reward the agent.
Figure 1. Depictions of the Shafer adjuster with , the square-root adjuster [8], the KV adjuster [27], and the Beta mixtures for . Left: the reward as a function of running maximum evidence. Right: the share of the null-payout budget spent on payments at evidence levels up to , namely , against log-evidence . Dots at zero show the unconditional first-round payment as a fraction of , and the remaining budget funds later milestones.
Remark 1. Definition 1 was formulated linearly in terms of the reward and cost. One could instead require for a nonnegative measurable function with . The e-process characterization in Proposition 1 then applies to for , extended by zero at time zero.
The Agent’s Strategy
The discussion on null-deterring contracts in the preceding section focuses on designing useful contracts from the principal’s perspective. But what about the agent? Faced with a null-deterring contract, what is the agent’s best response?
To answer this, we need to endow the agent with a utility function. Once such a function is introduced, the agent’s optimal strategy can be handled using machinery from stochastic control theory [36, 45]. In particular, it can be written in terms of a Bellman equation (or the Snell envelope, if the parameter policy is fixed in advance). We give the details in Appendix § 8.§ 1. But such solutions are not closed-form, not necessarily computationally tractable, and require that the agent possess full knowledge of their data-generating distribution . Therefore, in this section, we are interested in strategies that are more transparent, easier to interpret, and lend themselves to approximations that do not require full knowledge of the distribution.
The Bellman and Snell approaches given in the appendix handle general utility functions and e-processes, but our approximations require making a few simplifying assumptions. For one, we will study specific utility functions for the agent, one for the incremental reward schedule and one for the deferred schedule. For a fixed discount factor and , define
The superscripts refer to the reward schedule. Under the deferred schedule the agent is paid once, so only the final payout matters; under the incremental schedule every interim payment counts. Both objectives are common in financial models [see, e.g., 3, 26].
The two reward schedules (and their corresponding utility functions) affect the agent’s decision problem. Since the reward process must be nondecreasing on the incremental schedule, once the agent opts into the contract they should never exit. Therefore, the agent’s decision on the incremental schedule is how to choose the parameters (if any) to optimize the reward process. On the deferred schedule, meanwhile, the agent is concerned both with optimizing these parameters and also with potentially stopping and exiting the contract. Indeed, there’s no guarantee that is nondecreasing in .
We also assume throughout the rest of this section that the principal imposes no stopping time () and that the underlying e-process takes the form of a product:
where is positive and the superscript is the strategy determining the parameters and, if applicable, the stopping time. For , the rewards under the deferred schedule will be and those under the incremental schedule will be for some adjuster .
Let us summarize our path forward.
First we identify a parameter defined by
where is the set of values can take. We call the CGF-optimal parameter, so named because it can be defined by the CGF of the log-increment , as we’ll see below. Under both reward schedules, we bound the gap between the agent’s utility under and the optimal adaptive strategy as a function of . In particular, the gap is small when is small, where bounds the log-increments.
We then introduce the growth-optimal parameter , a first-order approximation to , defined by
which can be related to via . The GRO objective is easier to estimate from data by an empirical average and thus motivates several plug-in policies that can be instantiated even when the agent does not have full knowledge of their distribution. These plug-ins are presented in § 4.3 and the strategies are compared experimentally in § 5.
We begin by introducing the CGF-optimal parameter .
The CGF-optimal rule
For a fixed parameter , define the iid log-increments and their drift by
Recall that is the logarithm of the e-process in (7). For a threshold , write
with . In words, is the first time that the log-evidence reaches , and is the expected discount at that time. For a strategy that plays a constant parameter , we write and .
The reader may be wondering why we defined . Why are we studying threshold rules? First, they are natural under the incremental schedule when the reward process is an adjusted e-process, which pays exactly when the evidence first reaches (§ 3). Second, under the deferred schedule with discounted log utility, a reward-threshold rule is optimal at each fixed parameter with integrable log-increments (Appendix § 8.§ 2).
Fix and define the CGF
Consider what happens as the strategy which plays tries to reach a log-evidence of . Each round changes the log-evidence by and multiplies the eventual discount by . Since the increments are iid, conditioning on the first one gives the recurrence for , with for . If, for the moment, we posit that the solution to this recursive equation looks like for some fixed , then taking logs of both sides and rearranging gives . This motivates the definitions
For well-behaved CGFs, we have . Then is simply the smallest across all such roots and, when , it satisfies for every .3
The approximation suggests minimising . For , we call a parameter CGF-optimal if it attains the infimum and satisfies the recurrence relation with equality:
Equivalently, the CGF-optimal parameter satisfies . Such a parameter maximizes over . Only the lower bounds below require that the minimum in (14) is achieved. The upper bounds hold regardless.
Intriguingly, the exponent has recently appeared in another context: it is the optimal exponent of the exponential-decay-optimal (EDO) criterion of Clerico et al. [11, Section 7, equation (4)]. The same exponent appears because both problems discount the time needed to reach a given level of evidence. Proposition 2 below is thus similar to their Proposition 3.
The motivation for and rests on the approximation . We might wonder how tight this is. The following result shows that for every strategy , and that a fixed parameter nearly attains this bound when the overshoot (the amount by which the log-evidence exceeds at the crossing time) is small.
Proposition 2. Suppose the observations are iid under , and let satisfy for every . Then, for every predictable parameter strategy , the process is a nonnegative supermartingale under , and for all . Further, if a fixed parameter satisfies and almost surely for some , then, for every ,
Thus, taking and , whenever the stated assumptions hold, gives . That is, the fixed CGF rule achieves at least the fraction of the optimal discounted value under any adaptive strategy. The proof is in Appendix § 7.
Next we study the properties of the CGF-optimal rule under each reward schedule. Before we do, however, let us emphasize that the CGF-optimal rule still requires essentially full distributional knowledge from the agent, since a CGF that is finite near zero determines the distribution. Thus, it requires about as much information as the Bellman equation. However, it is significantly more transparent and, via results such as Proposition 2, allows us to analyze how close it is to the optimal strategy. Moreover, as we’ll see in § 4.3, it motivates various plug-in strategies which can be implemented under limited (or no) distributional knowledge.
λ_CGF under both reward schedules
We begin with the incremental reward schedule. Fix an adjuster and cumulative rewards for . The agent chooses predictable parameters to maximize
where the second equality follows from the adjuster representation (5) in § 3. We can integrate the crossing bounds in Proposition 2, taking and , to obtain bounds on the agent’s utility as follows.
Proposition 3. Suppose the observations are iid under and . Let be an adjuster and consider rewards for and . Define
and suppose . Then . If, in addition, there exists a parameter satisfying and almost surely for some , then
Remark 2. The upper bound also tells us when positive expected discounted profit is impossible. If , then for , so the adjuster budget gives
Thus is necessary for the agent’s expected discounted receipts to exceed the entry cost .
Next let us consider the deferred schedule. Here, the agent chooses both a parameter strategy and when to collect the reward. Stopping times may be infinite, in which case the agent never collects and we set . For any fixed parameter with integrable log-increments, a threshold rule attains the optimal stopping value (Appendix § 8.§ 2). We thus study the utility under the threshold rules . For , define
At the time , we have and . Ignoring overshoot, so that , gives the following approximation to the expected discounted utility:
where we’re using that . When , maximizing this approximation over gives
In this regime, the maximized value is a decreasing function of , hence , when it exists, also optimizes this approximation. Thus, the CGF-optimal policy under the deferred schedule uses and the stopping rule
For any other fixed parameter with , the corresponding reward threshold is . As in the case of incremental rewards, we can use properties of to give optimality guarantees for the CGF-optimal parameter and stopping rule.
Proposition 4. Suppose the observations are iid under and . Consider deferred rewards for , with . Let be the supremum of expected utility over predictable parameter strategies and all stopping times , and define
Then . If , suppose also that there exists a parameter satisfying and almost surely for some . Set and . Then
Thus the CGF-optimal policy attains at least the fraction of the optimal expected utility over all predictable strategies and stopping times. Appendix § 7 gives a corresponding guarantee for unbounded log-increments in terms of the overshoot’s second moment.
Approximate strategies
Let us now suppose that the agent does not know their full distribution. In this case they can try to approximate the CGF-optimal strategy via a first-order Taylor expansion. For fixed with and a CGF finite near zero, gives as . (Discount factors close to one, which describe patient agents, are the natural regime.) Therefore, to minimize we want to maximize , which suggests choosing
Under the incremental schedule the agent never stops, so they use throughout. Under deferred rewards, substituting and following (21) gives the threshold
The agent collects at the first round at which reaches this threshold.
Of course, these choices require knowing the drift. But there are many distributions for which it is reasonable to know the drift, even when full distributional knowledge is unavailable. For the exponential reward (1), for instance, depends only on the observation mean . The same holds for “sub-” processes [24] with a deterministic variance process, which cover a wide range of e-processes.
Further, when the drift is unknown, the agent can estimate it with the plug-in estimator
and bet an approximate maximizer of in the next round. Since the contract requires to be chosen before is observed, the agent starts from some and, after observing , sets . The resulting e-process is . Such a strategy has been called the “method of predictable plug-ins” [50] in other contexts. Examples are given in § 5. Under the deferred reward schedule, we use the following stopping time alongside the plug-in:
The floor at keeps the agent from collecting less than the entry cost . Next let us experimentally compare all the strategies we’ve discussed.
Numerical Illustrations
We compare the strategies under both reward schedules, setting . Appendix § 10 provides the implementation details.
Incremental schedule
We consider Bernoulli observations with mean , null mean at most , and increments for . The resulting “betting-style” e-process, is a common one in the e-value literature [50]. Since we are working under the incremental reward schedule, the reward process is for some adjuster .
We are interested in understanding the relative performance of our strategies compared to the optimal Bellman benchmark (Appendix § 8) as well as the effect of different adjusters. We consider two adjusters with three discrete milestone payments (see § 3), which differ in where on the evidence scale they spend their null-payout budget. Milestone uses the fraction of the budget, so the reward is , with . The low-milestone adjuster uses weights and the high-milestone adjuster uses .
Figure 2 shows that all three fixed policies plotted attain at least 98% of the Bellman value when . As agents become less patient and decreases, the gap widens. In other words, with stronger discounting, adaptive parameter choice matters more, regardless of the adjuster. At under the high-milestone adjuster, the growth-optimal rule earns 63.3% of the Bellman value and the CGF-optimal rule 86.9%. We also compare both rules with the best fixed parameter, found numerically. Its gap to the CGF-optimal rule reflects the cost of overshoot in Proposition 3.
Although the plug-in improves as the agent becomes more patient, it still falls short of the fixed policies at . To see why, relate it to the growth-optimal rule. In the Bernoulli example, the plug-in uses , with , where is the running mean. It starts at and sets . This is exactly with replaced by , so the plug-in is the growth-optimal rule with an estimated parameter. For close to one, is nearly optimal, converges to it, and the plug-in’s gap reflects the cost of learning. At , however, is far below the best fixed parameter of about 1.36. The plug-in’s early estimates are noisy, and after early successes they can be much larger than , moving the bet toward the better parameter. As a result, the plug-in matches the growth-optimal rule under the low-milestone adjuster and exceeds it under the high-milestone adjuster (69.2% versus 63.3% of the Bellman value).
Deferred schedule
We use Gaussian observations with the exponential reward (1), and Bernoulli observations with the betting reward above. The growth-optimal parameters are and , respectively. For Bernoulli data, Figure 3 also includes a Hoeffding baseline, which uses the sub-Gaussian reward , valid for any data in with mean at most . Its drift depends only on the mean, so, as with the Gaussian reward, an agent who knows only can implement the growth-optimal rule exactly, with and the drift threshold (25). The price is a smaller drift, and hence lower utility. Appendix § 10 quantifies the gap using Pinsker’s inequality, derives the CGF-optimal parameters, and specifies the plug-in rules.
Figure 3. Deferred utility relative to the Bellman benchmark (B.2). Sweeps over and use , and sweeps over use (Gaussian) and (Bernoulli), with . The CGF-optimal policy (21) uses . The CGF threshold and the drift threshold (25) use , and the plug-in uses the threshold rule (27). Ratios are estimated from 20,000 simulations with a cap of 2500 rounds.
In Figure 3, the CGF-optimal policy (21) is within 2% of the Bellman benchmark throughout the Gaussian settings and within 1.2% throughout the Bernoulli settings. The drift rule (25) performs similarly for patient agents, but loses more with stronger discounting. The plug-in rule (27) can lose substantially for weak signals or strong discounting.
For the bounded Bernoulli increments in Figure 3, Proposition 4 quantifies the loss from overshoot. Gaussian log-increments are unbounded above, but equation (A.5) in Appendix § 7 bounds the loss using the second moment of the overshoot. As hinted at by these results, the performance of our strategies can decline substantially when a single observation can carry the reward far past the threshold.
To give a concrete example, consider Bernoulli data with rare successes (, , ). Under , one success multiplies the evidence by about 55, a log-jump of 4.0 against the log-evidence threshold in (21), so . The CGF-optimal rule earns 0.184, whereas the smaller bet with its exact threshold from Proposition 6 earns 0.261. So the CGF-optimal rule loses at least 29.4% of the optimal value. More details are provided in Appendix § 10.
Conclusion
We have extended the framework of principal-agent hypothesis testing to settings with repeated and adaptive experimentation. We introduced sequential statistical contracts and showed how a principal can use e-processes to design contracts which deter the entry of null agents. We then focused on the agent, noting that techniques from stochastic control theory can be leveraged to optimize the reward process once they’ve opted into the contract. Such techniques are not closed-form, however, and require that the agent possess full knowledge of their distribution to implement them. We therefore provide several approximations which are more transparent and can be implemented with less information.
Acknowledgements. BC and AR acknowledge support from NSF grants IIS-2229881 and DMS-2310718.
Omitted Proofs
We give the proofs of the four propositions in the main text. When the parameter is fixed, we suppress its subscript in the increments, partial sums, and so on.
Proof of Proposition 1. If is ND, then by definition for any stopping time . The bound also holds when may be zero: since and rewards are nonnegative, . Dividing by shows that is an e-process for .
Conversely, on the deferred schedule, if is an e-process, then for all stopping times , which is precisely ND.
On the incremental schedule, suppose every is nondecreasing, so that exists in for each reward process. If is ND, then taking the deterministic stopping times gives , and monotone convergence yields . Conversely, if , then for every stopping time , so .
In particular, if is an e-process for every reward process, then for every , since is a stopping time. Monotone convergence gives the required limit bound, hence ND.
It remains to verify the adjuster construction. Let be an e-process for . By an equivalent definition of an e-process [38], for each there exists a test supermartingale under such that -a.s. for all . Moreover, since is an adjuster, there exists a test supermartingale under such that -a.s. Therefore, since is nondecreasing (and since lies in the domain of and ), for any ,
hence . We conclude that is an e-process for , immediately implying that is null-determining. □
The following identity supplies the overshoot correction used in the proofs below and extends the performance guarantees to unbounded increments.
Lemma 1. Fix with for some . Let be the tilted law defined by
For , the crossing time is finite under this law. Write for the overshoot. Then
More generally, for any nonnegative measurable ,
The formula extends to signed whenever one of its weighted positive or negative parts is integrable.
Proof. Suppress and write , , and . Tilting the one-observation law by and taking its iid product defines , since . Its density on is . Hence , and
Changing measure on each event gives
Sum over and use to obtain (A.2). Taking gives (A.1). Applying the formula separately to positive and negative parts gives the signed extension.
For and , both weighted parts are integrable on , because is bounded. Thus
In particular, if and , setting and gives
This identity also applies to unbounded increments.
Under the same assumptions, since for , it gives
Since , the same bound controls the relative loss against the optimal adaptive strategy. Near-optimality thus follows when is small, without a bounded-increment assumption. For Gaussian increments, this tilted second moment is finite. Indeed, tilting preserves normality, the lemma's geometric tail bound gives a finite tilted mean crossing time, and has finite tilted expectation by independence.
Proof of Proposition 2. For a predictable strategy , write . Independence of from and the conditional moment bound give
Thus is a nonnegative supermartingale with initial value one, and obeys for . If is finite, by definition, hence .
Under the additional assumptions for , Lemma 1 applies with . Its overshoot satisfies , so
The remaining inequality follows because a fixed parameter is a predictable strategy.
Proof of Proposition 3. The baseline is paid in round one, and each increase is paid at its first evidence crossing. Tonelli’s theorem gives (16), since these payments are nonnegative. Integrating the upper crossing bound gives for every predictable policy.
Under the lower-bound assumptions, Proposition 2 gives
The middle inequality in (4.11) follows because the fixed CGF policy is admissible.
Proof of Proposition 4. Write and , so that for . For an adaptive strategy, abbreviate as . Let . Fix any strategy. Since is -measurable, is independent of , and for every (§ 4.1),
so is a nonnegative supermartingale with . Thus for any stopping time . Since for all ,
The positive part of the utility is therefore integrable. If its negative part is not integrable, the upper bound is immediate. Otherwise, set . If , the utility is zero. For , Jensen’s inequality under the probability measure with density gives
where the second inequality uses . Enlarging the range to and maximizing gives and the bound , which is positive and also covers .
Under the additional assumptions for the lower bound, set and . The overshoot identity (A.4) gives
because under the tilted law and is nonincreasing for . The threshold rule is admissible, so .
Additional Results
Bellman and Snell characterizations
Once the agent has chosen a reward process, they face a joint optimization over parameter selection and stopping, which can be framed as a stochastic control problem. Let denote the agent’s utility from collecting reward at time . For deferred rewards with for , let and define, after round while the contract is active,
Here ranges over predictable parameter policies and over almost surely finite stopping times. The reward and time arguments are frozen when the principal stops at .
The Bellman equation characterizes under a Markov assumption: the conditional law of given and depends only on , and the principal stops at the first entry of into some termination set , that is, . Examples include a deterministic horizon , a rule that stops once the reward reaches a target or leaves an interval, a time-varying boundary , and a growth-rate rule . A path-dependent rule, or one driven by the principal’s private information (such as a budget), is not Markov in . Augmenting the state can restore the Markov property, but at the cost of a larger state, or of state variables the agent cannot observe.
Proposition 5 (Bellman characterization). Suppose the Markov assumption holds and the stopped payoffs are uniformly integrable over admissible policies and stopping times. For on ,
with at termination. The value at entry is , since the agent has already opted in. If the supremum admits a measurable maximizing selector and the resulting first stopping time is almost surely finite, selecting that maximizer while the continuation value exceeds the stopping payoff is optimal.
This follows from the principle of optimality and optional stopping under uniform integrability [35 Chapter 3]. The selector and finiteness conditions are needed only for an optimal strategy to exist, not for the Bellman equation itself. The numerical benchmarks use bounded action sets with integrable envelopes for the log-increments, which ensure uniform integrability of discounted log payoffs. Incremental rewards require a state that also records past payments, since discounted receipts depend on when each payment was made; for milestone contracts, current evidence and the next unpaid milestone suffice (Appendix § 10).
The Bellman equation is rarely solvable in closed form. Even for Gaussian data it is a non-linear integral equation, coupled with a state-dependent optimization over , and must be solved numerically. In practice, however, e-processes are seldom optimized round by round. One either commits in advance to a predictable plug-in rule [50] or mixes over with a data-free distribution. Either way, the reward process is determined by the data, and only the stopping decision remains. For , write and define its Snell envelope, the smallest supermartingale that dominates , by
Theorem 1 (Snell characterization). Suppose is an -stopping time and . If is almost surely finite, it attains .
Proof. For , the recursion makes a uniformly integrable martingale. Thus , while domination and optional stopping give for every admissible .
The integrability condition can be weakened to uniform integrability of , the usual class-D assumption [35 Chapter 1]. For a fixed parameter with , it holds for discounted log utility, since , whose expectation is at most . For conditions under which the expected log utility of an e-value is well-defined, see De Roos et al. [16 Lemma 2.5].
Unlike Proposition 5, Theorem 1 does not require to be Markov, only an -stopping time. For instance, a principal who stops once the reward falls more than a fraction below its running maximum, , uses a path-dependent rule that cannot be written in terms of . The Snell envelope thus trades per-round optimization of for more general principal stopping rules. Both characterizations generally require computing a value function, and both also apply to other adapted payoffs, including cumulative discounted incremental receipts.
Section 4.2 also allows , with payoff zero. For discounted log utility under the integrability conditions above, this changes neither value. Indeed, is then integrable and, by the strong law of large numbers, almost surely on , so dominated convergence gives for every stopping time .
Exact fixed-parameter stopping value
For a fixed parameter, the optimal stopping boundary has an exact series representation [13, 33].
Proposition 6. Consider deferred rewards and for , with and utility , with . Fix with . Set and
The rule attains the supremum of expected utility over all stopping times at this parameter, without a positive-drift assumption.
Under the assumptions of Proposition 6, let be independent of the observations, with for \Pr(N_\delta \ge t) = \delta^t$, discounting is equivalent to crossing before this deadline:
For a fixed parameter set . The boundary is the expected record high, . Since the agent must run the first trial, the exact value is
where is an independent copy of , independent of . When , this reduces to ; this includes the experiments with .
Proof of Proposition 6. Write , so the payoff at time is . Use the independent geometric deadline from (B.4) and let . Since , we have .
Spitzer’s identity [46] gives . Averaging over and exchanging the order of summation,
Define the candidate value function
Step 1: U dominates the payoff, with equality above . By Jensen’s inequality, . If then (as ), so .
Step 2: a one-step recursion. By the memoryless property of the geometric law, with probability we have and , while conditional on we have , where is independent of . Hence
If , then for every (both vanish when ), so . If , then , so . Thus is a nonnegative supermartingale, and a martingale from time one up to the hitting time .
Step 3: upper bound. For any stopping time , Step 1 and optional stopping from time one (via Fatou’s lemma) give
where the stopped term is defined as zero on .
Step 4: attainment. The stopped process from Step 2 is an integrable martingale from time one at every finite horizon. On , Step 1 gives . Consequently, for ,
On , , so monotonicity of bounds the second term by . The first term converges to by monotone convergence, since at every finite crossing. Thus attains the upper bound , proving (B.5). If , Step 2 also gives .
A limitation of the adjuster construction
Here we observe that not all nondecreasing -processes come from adjusters.
Proposition 7. There exists a bounded nondecreasing -process with for a singleton null that cannot be represented as
for any deterministic adjuster and any -process for on the same filtration. In fact, no such pair satisfies almost surely for every .
Proof. Let be uniform on under , with trivial and for . Set and for . This process is bounded and nondecreasing. For every stopping time , , so . Thus is an -process and defines a null-deterring incremental contract with entry cost .
Suppose almost surely. Since , nonnegativity and monotonicity of force . Moreover, every adjuster satisfies for , because
On , we must have : otherwise . On , we have and . On , nonnegativity gives . Hence almost surely, with strict inequality on an event of probability . It follows that , contradicting the e-process property.
Finally, if almost surely, the adjuster guarantee in Proposition 1 gives . Equality of expectations and the assumed domination force almost surely, which was just ruled out.
Comparison to the Sequential Contract of Bates et al.
As we discussed in the introduction, Bates et al. [5] also provided a sequential version of their one-shot contract. We state it here:
Bates et al. [5] multi-round contract
The initial license value is .
For : (a) The agent chooses to withdraw from their license, or continue without a withdrawal. (b) The agent may pay to run stage of the trial, in which case: i. The agent selects an update function from the menu ii. The trial produces evidence iii. The license value is updated as . (c) Otherwise, license value is updated as .
While their motivation is similar to ours, the two contracts differ in their treatment of costs, payments, horizons, and the agent’s optimization problem. Let us draw attention to each of these features in turn.
Per-round costs and payments. First, the contract above uses per-round costs, and allows the agent to withdraw money from the license won so far. This can be beneficial for some applications, as it makes the cost structure explicit and part of the agent’s strategy. In particular, the contract is well-suited to settings in which continued experimentation requires a sequence of distinct investments and the agent actively manages both investment and consumption.
We instead model as the up-front cost of mounting the research program. Any costs of delay or continuation are represented through the agent’s utility rather than through additional payments in the contract. In other words, per-round costs are possible in our setting, though they are modeled only on the agent’s side—they are not part of the structure of the contract itself.
Finite-horizon and stopping times. Per-round costs make an unbounded horizon more delicate. The agent’s net-profit supermartingale need not be nonnegative, so optional stopping does not apply to arbitrary stopping times without additional admissibility conditions. (For instance, if cumulative investment is bounded by , the net profit is bounded below by , and optional stopping can be applied to the shifted process.) Consequently, Bates et al. work with a finite number of possible rounds. This fixed horizon allows the problem to be solved by backwards induction. Our framework, meanwhile, allows an unbounded horizon and introduces stopping rules for both the principal and the agent.
The agent’s best response. Both we and Bates et al. study the agent’s best response, and both do so via dynamic programming. Bates et al. formulate the finite-horizon contract as a Markov decision process whose state includes the current time and license value. At each stage, the agent chooses whether to stop and, if continuing, which e-value update to use. Our Bellman formulation is analogous, but permits an unbounded horizon, an explicit principal stopping rule, and general time-dependent utility. For a fixed reward process, we additionally characterize the optimal stopping time through its Snell envelope.
Overall, the two frameworks should be viewed as complementary extensions of the single-shot contract. Bates et al. emphasize a finite sequence of costly investments with flexible withdrawals and reinvestment. We emphasize a potentially indefinite stream of evidence, a fixed entry cost, principal and agent stopping, and approximations to the optimal stopping rule under different levels of agent information.
Experimental Details
Code to replicate the experiments is available at https://github.com/bchugg/statcontracts.
For Figure 2, we use Bernoulli observations and the milestone allocations in § 5.1. For the betting multiplier , the log-increment CGF is
Figure 2. The utility of various strategies under the incremental reward schedule relative to the Bellman benchmark (B.2). The low- and high-milestone adjusters allocate 80% of their null-payout budget (see § 3) to the first and last milestone, respectively. The plug-in is that of § 4.3 and uses the clipped sample mean. Standard errors of the ratios are below 0.0016, so error bars are omitted.
We compute the CGF-optimal parameter by numerically solving ; a prescribed fixed bet uses its own root . All reported Bernoulli settings satisfy , so and an interior CGF-optimal bet exists. The mean plug-in starts at , then uses and , continuing to sample when the bet is zero.
A policy’s value is . For fixed bets we compute these crossing values recursively and search numerically for the best parameter. The adaptive benchmark solves a discretized Bellman equation whose state records current evidence and the next unpaid milestone. We evaluate the adaptive and plug-in policies on 500,000 simulated paths each, and the plotted ratios have standard errors below 0.0016. Simulation horizons make the omitted expected receipts smaller than .
Figure 3 uses the same Bernoulli family and a Gaussian family. For with , use , . Then
Maximizing the drift gives . Maximizing the CGF over and solving the moment-root equation gives
The experiments use .
The Gaussian plug-in starts at , then uses and estimated drift , with . For the Bernoulli plug-in’s deferred threshold, the estimated drift is when , and zero otherwise. This uses the clipped fitted law, a variant of the empirical drift estimate in § 4.3.
The Hoeffding baseline uses with and the drift threshold corresponding to . By Pinsker’s inequality, its drift is at most the betting drift . For patient agents this gap carries over to utility: with and , the maximized value (20) is approximately , which is linear in the drift. The Hoeffding baseline should therefore earn about the fraction of the value of the drift rule with the betting reward. At , this fraction falls from 0.998 at to 0.934 at , and the simulated ratios agree with it to within 0.003.
We obtain the deferred Bellman benchmark by policy iteration on a log-evidence grid, averaging over Gaussian increments by quadrature and summing over the two Bernoulli outcomes exactly. Each policy and the benchmark are evaluated on 20,000 simulated paths under the original observation law. All paths run for at least one trial and are capped at , with payoff even if the threshold was not reached. The ratios therefore have sampling uncertainty and Bellman discretization error.
For the large-overshoot example in § 5.2, we compare with log-evidence threshold against with exact threshold . The error relative to the uncapped values is below .
References
- [1]Adobe. Experimentation panel. https://experienceleague.adobe.com/en/docs/analytics-platform/using/cja-workspace/panels/experimentation, September 2026. Adobe Customer Journey Analytics documentation, last updated 28 Sep 2026, accessed 2026-10-14.
- [2]Ander Artola Velasco, Stratis Tsirtsis, and Manuel Gomez-Rodriguez. Optimizing social utility in sequential experiments. In Advances in Neural Information Processing Systems, volume 39, 2026. To appear.
- [4]Stephen Bates, Michael I. Jordan, Michael Sklar, and Jake A. Soloff. Incentive-theoretic Bayesian inference for collaborative science. arXiv preprint arXiv:2307.03748, 2023.arxiv.org/abs/2307.03748
- [5]Stephen Bates, Michael I. Jordan, Michael Sklar, and Jake A. Soloff. Principal-agent hypothesis testing. Journal of the American Statistical Association, 2026.arxiv.org/abs/2205.06812
- [6]Yatong Chen, Safwan Hossain, and Yiling Chen. Strategic hypothesis testing. In Advances in Neural Information Processing Systems, volume 38, pages 162375–162402. Curran Associates, Inc., 2025.
- [7]Yurong Chen, Zhaohua Chen, Xiaotie Deng, and Zhiyi Huang. Are bounded contracts learnable and approximately optimal? In Proceedings of the 25th ACM Conference on Economics and Computation, pages 315–344. Association for Computing Machinery, 2024.arxiv.org/abs/2402.14486
- [8]Yo Joong Choe and Aaditya Ramdas. Combining evidence across filtrations. Journal of the Royal Statistical Society Series B: Statistical Methodology, 88(4):1492–1514, 2026.arxiv.org/abs/2402.09698
- [9]Ben Chugg, Michael I. Jordan, and Aaditya Ramdas. Strategic multi-agent resource allocation via statistical contracts. arXiv preprint arXiv:2610.18336, 2026.arxiv.org/abs/2610.18336
- [10]Ben Chugg, Aaditya Ramdas, and Peter Grünwald. E-values as statistical evidence: A comparison to Bayes factors, likelihoods, and p-values. Synthese, 208(3):130, 2026.arxiv.org/abs/2603.24421
- [11]Eugenio Clerico, Tobias Wegel, Iskander Azangulov, and Patrick Rebeshini. Time-sensitive anytime-valid testing. In Advances in Neural Information Processing Systems, volume 39, 2026. To appear.arxiv.org/abs/2605.06521
- [12]Alon Cohen, Argyrios Deligkas, and Moran Koren. Learning approximately optimal contracts. In International Symposium on Algorithmic Game Theory, pages 331–346. Springer, 2022.
- [13]D. A. Darling, T. Liggett, and H. M. Taylor. Optimal stopping for partial sums. The Annals of Mathematical Statistics, 43(4):1363–1368, 1972.DOI
- [14]A. Philip Dawid, Steven de Rooij, Peter Grünwald, Wouter M. Koolen, Glenn Shafer, Alexander Shen, Nikolai Vereshchagin, and Vladimir Vovk. Probability-free pricing of adjusted American lookbacks. arXiv preprint arXiv:1108.4113, 2011.arxiv.org/abs/1108.4113
- [15]A. Philip Dawid, Steven de Rooij, Glenn Shafer, Alexander Shen, Nikolai Vereshchagin, and Vladimir Vovk. Insuring against loss of evidence in game-theoretic probability. Statistics & Probability Letters, 81(1):157–162, 2011.
- [16]Dante de Roos, Ben Chugg, Peter Grünwald, and Aaditya Ramdas. Rao–Blackwellized e-variables. Biometrika, 2026. Advance article.arxiv.org/abs/2512.16759
- [17]Paul Dütting, Michal Feldman, and Inbal Talgam-Cohen. Algorithmic contract theory: A survey. Foundations and Trends® in Theoretical Computer Science, 16(3–4):211–411, 2024.arxiv.org/abs/2412.16384
- [18]Dino Gerardi and Lucas Maestri. A principal-agent model of sequential testing. Theoretical Economics, 7(3):425–463, 2012.
- [19]Sanford J. Grossman and Oliver D. Hart. An analysis of the principal-agent problem. In G. Dionne and S. E. Harrington, editors, Foundations of Insurance Economics: Readings in Economics and Finance, Huebner International Series on Risk, Insurance and Economic Security, pages 302–340. Springer, 1992.
- [20]GrowthBook. Sequential testing. https://docs.growthbook.io/statistics/sequential, n.d. GrowthBook documentation, no publication date stated, accessed 2026-10-14.
- [21]Peter Grünwald, Rianne de Heide, and Wouter Koolen. Safe testing. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(5):1091–1128, 2024.
- [22]Peter Henderson, Ben Chugg, Brandon Anderson, and Daniel E Ho. Beyond ads: Sequential decision-making algorithms in law and public policy. In Proceedings of the 2022 Symposium on Computer Science and Law, pages 87–100, 2022.DOI
- [23]Peter Henderson, Ben Chugg, Brandon Anderson, Kristen Altenburger, Alex Turk, John Guyton, Jacob Goldin, and Daniel E Ho. Integrating reward maximization and population estimation: Sequential decision-making for internal revenue service audit selection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5087–5095, 2023.
- [24]Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform Chernoff bounds via nonnegative supermartingales. Probability Surveys, 17:257–317, 2020.DOI
- [25]Michael I. Jordan. A collectivist, economic vision for AI. Communications of the ACM, 69(9):67–75, 2026.DOI
- [27]Wouter M. Koolen and Vladimir Vovk. Buy low, sell high. Theoretical Computer Science, 558:144–158, 2014.
- [28]Jean-Jacques Laffont and David Martimort. The Theory of Incentives: The Principal-Agent Model. Princeton University Press, 2002.
- [29]Cintya Lanchimba. Contract theory. In Industrial Organization: The Economics of Markets, Firms, and Competition, pages 91–124. Palgrave Macmillan, 2026.
- [30]Michael Lindon, Chris Sanden, Vache Shirikian, Yanjun Liu, Minal Mishra, and Martin Tingley. Sequential A/B testing keeps the world streaming Netflix, part 1: Continuous data. https://netflixtechblog.com/sequential-a-b-testing-keeps-the-world-streaming-netflix-part-1-continuous-data-cba6c7ed49df, February 2024. Netflix Technology Blog, accessed 2026-10-14.
- [31]Ryan Lucht. What is sequential testing? (with examples). https://www.geteppo.com/blog/sequential-testing, July 2024. Eppo blog, accessed 2026-10-14.
- [32]Drew T. Nguyen, Alireza Fallah, and Michael I. Jordan. Incentive design in sequential statistical protocols. ICML 2026 Workshop on New Frontiers in Game-Theoretic Learning, 2026. URL https://openreview.net/forum?id=VKt5UUOYCK.
- [33]A. A. Novikov and A. N. Shiryaev. On an effective solution of the optimal stopping problem for random walks. Theory of Probability & Its Applications, 49(2):344–354, 2005.DOI
- [34]Michael Oberst, Davis Liang, and Zachary C. Lipton. Pioneering the science of AI evaluation. https://www.abridge.com/ai/science-ai-evaluation, September 2024. Abridge whitepaper, published 2024-09-19, last updated 2025-08-07, accessed 2026-10-14.
- [35]Goran Peskir and Albert Shiryaev. Optimal Stopping and Free-Boundary Problems. Birkhäuser, 2006.
- [36]Martin L. Puterman. Markov decision processes. In D. P. Heyman and M. J. Sobel, editors, Stochastic Models, volume 2 of Handbooks in Operations Research and Management Science, chapter 8, pages 331–434. Elsevier, 1990.
- [37]Aaditya Ramdas and Ruodu Wang. Hypothesis testing with e-values. Foundations and Trends® in Statistics, 1(1–2):1–390, 2025.DOI
- [38]Aaditya Ramdas, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4):576–601, 2023.arxiv.org/abs/2210.01948
- [39]William P. Rogerson. The first-order approach to principal-agent problems. Econometrica, 53(6):1357–1367, 1985.DOI
- [40]Johannes Ruf, Martin Larsson, Wouter M. Koolen, and Aaditya Ramdas. A composite generalization of Ville’s martingale theorem using e-processes. Electronic Journal of Probability, 28, 2023.arxiv.org/abs/2203.04485
- [41]Mårten Schultzberg and Sebastian Ankargren. Choosing a sequential testing framework — comparisons and discussions. https://engineering.atspotify.com/2023/03/choosing-sequential-testing-framework-comparisons-and-discussions, March 2023. Spotify Engineering Blog, accessed 2026-10-14.
- [42]Glenn Shafer, Alexander Shen, Nikolai Vereshchagin, and Vladimir Vovk. Test martingales, Bayes factors and p-values. Statistical Science, 26(1):84–101, 2011.
- [43]Flora C. Shi, Stephen Bates, and Martin J. Wainwright. Sharp results for hypothesis testing with risk-sensitive agents. arXiv preprint arXiv:2412.16452, 2024.arxiv.org/abs/2412.16452
- [44]Flora C. Shi, Martin J. Wainwright, and Stephen Bates. Instance-adaptive hypothesis tests with heterogeneous agents. arXiv preprint arXiv:2510.21178, 2025.arxiv.org/abs/2510.21178
- [45]Albert N. Shiryaev. Optimal Stopping Rules. Springer, 2008.
- [46]Frank Spitzer. A combinatorial lemma and its application to probability theory. Transactions of the American Mathematical Society, 82(2):323–339, 1956.
- [47]Roland Vaubel. Principal-agent problems in international organizations. The Review of International Organizations, 1(2):125–138, 2006.DOI
- [48]Boris Velasevic, Eden Saig, and Eric Mazumdar. Anytime contracts for AI services. arXiv preprint arXiv:2610.22781, 2026.arxiv.org/abs/2610.22781
- [49]Jean Ville. Étude critique de la notion de collectif. Gauthier-Villars, Paris, 1939.
- [50]Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(1):1–27, 2024.
- [51]Banghua Zhu, Stephen Bates, Zhuoran Yang, Yixin Wang, Jiantao Jiao, and Michael I. Jordan. The sample complexity of online contract design. In Proceedings of the 24th ACM Conference on Economics and Computation, page 1188. Association for Computing Machinery, 2023. Full version: arXiv:2211.05732.arxiv.org/abs/2211.05732


