StreamLTT: Risk Certification for Policy Selection under Stream Dependence

Introduction

Modern prediction systems are ultimately deployed through decisions rather than raw predictions ([13, 40, 11]). A practitioner may choose a confidence threshold, prediction set width, abstention rule, alarm threshold, intervention rule, or another operating policy subject to an explicit risk requirement ([16, 29]). A broad literature has developed statistical tools for uncertainty-aware and risk-controlled decision-making, including selective classification, conformal prediction, and conformal risk control ([15, 2, 3]). Learn then Test (LTT) provides a natural framework for policy selection: given a finite candidate family, it tests each policy against a prescribed risk target and corrects for multiplicity so that the resulting policy region is certified simultaneously ([4]). For independent calibration losses, a common implementation constructs $p$ values for individual policies using a central limit theorem (CLT) approximation to empirical risk; we refer to this instantiation as IID-LTT. This separation between prediction, certification, and subsequent selection makes LTT particularly attractive when deployment must satisfy an explicit risk budget.

A central difficulty arises when certification data arrive as a stream. Forecasts are temporally ordered, operating regimes evolve, and observations may be coupled through past outcomes, recurring entities, or shared structure. Even for a fixed policy, its conditional loss distribution can therefore depend on the accumulated history and evolve throughout deployment, so a single stationary marginal risk no longer describes the risks encountered over time. This challenge has motivated a broad literature on online calibration, which uses sequential feedback to maintain predictive calibration or coverage under distribution shift and nonexchangeability [17, 42, 18, 41].

Policy certification, however, poses a different statistical problem. In LTT, the candidate family is fixed before certification, and each policy must be tested against a risk constraint before multiplicity correction controls the probability of certifying any unsafe candidate [4]. Under stream dependence, the challenge is therefore not only how to construct a valid test, but also what risk should be certified when the risk of a fixed policy can evolve along the stream. Existing tools address the inference side of this problem: long-run variance estimators adjust uncertainty for dependent averages [32], block resampling approximates sampling distributions under local dependence [33], and sequential concentration controls deviations for adapted observations [24]. Yet these approaches still require a pre-specified risk target, leaving open how risk itself should be defined when it evolves with the observed history.

We address this gap by defining risk along the information flow of deployment. Immediately before each outcome is revealed, we consider the conditional risk of the corresponding decision given the available history and average these risks along the stream, yielding the average predictable risk. This target does not require a stationary marginal distribution and reduces to the standard population risk in the IID setting. Conditional centering further turns empirical deviations from this target into a martingale sum, providing the basis for StreamLTT, our extension of LTT to policy certification over dependent deployment streams. We develop a test that adapts to predictable quadratic variation, together with a martingale Hoeffding alternative [22] that requires only bounded losses and provides finite-sample validity. The resulting $p$-values plug directly into the multiple-testing layer of LTT [4], yielding a simultaneously certified policy region under family-wise error rate control [23]. A deployment policy can then be selected from this region without repeating certification, retaining the modular selection principle of LTT while aligning certification with the information structure of an evolving stream.

Our experiments evaluate StreamLTT along two dimensions: the tightness of the certified risk frontier and the control of false certification. Across seven real-world forecasting datasets and seven deployment risks, StreamLTT produces substantially tighter risk certificates than IID-LTT and alternatives based on robust variance estimation, block resampling, and concentration bounds, while empirical family-wise error closely follows the prescribed level. A finite-sample ablation further separates the two sources of improvement: the predictable-risk formulation already yields a tight certified frontier, while adaptation to predictable variance makes more effective use of the available family-wise error budget. These results distinguish the benefit of changing the risk being certified from that of adapting the statistical test to the variability of the stream.

Our contributions are threefold:

• We formulate risk certification for policy selection under dependent and nonstationary deployment through average predictable risk, which follows the information available at each decision time and recovers the standard population risk in the IID setting.

• We develop StreamLTT by using conditional centering to obtain martingale fluctuations, leading to a test that adapts to predictable variance and a Hoeffding alternative with finite sample validity. Combined with control of the family wise error rate, StreamLTT certifies an entire policy region while preserving validity under subsequent policy selection.

• Across seven forecasting datasets and seven deployment risks, StreamLTT yields substantially tighter certification than IID, robust variance, bootstrap, and concentration based alternatives while empirical family wise error follows the prescribed level. A finite sample ablation further separates the benefit of the predictable risk formulation from that of predictable variance adaptation.

Problem Formulation and Preliminaries

Consider a prediction system deployed over an ordered stream $(X_1, Y_1), \ldots, (X_N, Y_N)$. At round $n$, the covariates $X_n$ and the history from previous rounds are available before a deployment decision is made, whereas the outcome $Y_n$ is revealed only afterward. Let $\{\mathcal{F}_n\}_{n\ge0}$ denote the associated filtration, with the history, current covariates, model output, and deployment decision at round $n$ measurable with respect to $\mathcal{F}_{n-1}$. Here, we make no assumptions about independence nor stationarity. The conditional law of the stream may vary over time and depend on previously observed outcomes, recurring entities, or other shared context.

A finite family of deployment policies $\Theta:= \{\theta_1, \ldots, \theta_M\}$ is fixed before certification. Each policy may specify a confidence threshold, prediction-set width, alarm threshold, or another operating rule applied to a frozen predictive model. At round $n$, policy $\theta$ incurs an adapted bounded loss $L_n(\theta) \in[0, B(\theta)]$, where the deterministic bound $B(\theta) < \infty$ is fixed before certification. Given a prescribed risk target $\alpha$ and family-wise error level $\delta$, the objective is to construct a certified policy region such that the probability of including any policy whose deployment risk exceeds $\alpha$ is controlled at level $\delta$.

Learn-then-Test (LTT) casts risk-controlled policy selection as a multiple-testing problem (Angelopoulos et al. [4]). In the standard setting, each candidate $\theta$ has population risk $R(\theta) := \mathbb{E}[L(\theta)]$. A policy is considered safe when $R(\theta) \le\alpha$, so LTT tests the complementary null

A valid $p$-value is constructed for each candidate and passed to a multiple-testing procedure. If the rejection rule controls the family-wise error rate (FWER) at level $\delta$, the rejected hypotheses define a certified region $\widehat{\Theta}_{\mathrm{safe}}$ satisfying

Thus, with probability at least $1-\delta$, every policy retained in the certified region satisfies the prescribed risk requirement. Since the guarantee holds simultaneously over the region, a final operating point may subsequently be selected from $\widehat{\Theta}_{\mathrm{safe}}$ without retesting the candidate family.

For independent calibration losses, LTT admits several policy level tests, including finite sample bounded loss tests and asymptotic approximations. In this work, we refer to the CLT based instantiation as IID-LTT. These standard constructions target a common population risk and rely on independent calibration observations. They therefore do not directly address the deployment setting considered here, where losses may be dependent and their conditional risks may evolve with the observed history.

Problem formulation. Let $R_N(\theta)$ denote the stream-level deployment risk associated with policy $\theta$, whose precise form will be specified in Section 3. For a prescribed risk level $\alpha$, define the corresponding safe region as

Given the observed stream, the certification problem is to construct a data-dependent region $\widehat{\Theta}_{\mathrm{safe}} \subseteq\Theta$ satisfying

Thus, the statistical task is to specify a stream-level risk $R_N(\theta)$ compatible with the deployment information structure and to construct valid policy-level tests for the unsafe hypotheses $H_j : R_N(\theta_j) > \alpha$. Section 3 instantiates $R_N$ as average predictable risk and develops the corresponding dependence-aware tests.

StreamLTT: Learn-then-Test under Stream Dependence

We now develop StreamLTT for the stream-level certification problem introduced in Section 2. Rather than imposing a common population risk, StreamLTT averages the conditional risks encountered along the deployment stream. Centering losses around this predictable target induces martingale fluctuations, enabling finite-sample and variance-adaptive policy tests under dependence. The resulting $p$-values are combined with family-wise error control to obtain a simultaneously certified policy region.

From Marginal to Predictable Risk

For a fixed candidate policy $\theta\in\Theta$, let $L_n(\theta)$ denote its bounded deployment loss at round $n$. Before $Y_n$ is revealed, define the one-step predictable risk $r_n(\theta) := \mathbb{E}[L_n(\theta) \mid\mathcal{F}_{n-1}]$, the conditional risk of deploying $\theta$ given the information available when the decision is made. Unlike a fixed population risk, $p_n(\theta)$ may change with the observed history and operating environment.

The quantity $R_{\mathrm{pred},N}(\theta)$ therefore measures the average conditional risk encountered along the realized deployment history, rather than risk under a single stationary population. It allows the conditional risk to vary across rounds and requires neither independence nor stationarity. In the IID setting with common mean $R(\theta)$, $p_n(\theta) = R(\theta)$ for every $n$, so $R_{\mathrm{pred},N}(\theta) = R(\theta)$ and the standard LTT target is recovered.

This target also yields the statistical structure needed for certification. Let

Since $\mathbb{E}[Z_n(\theta) \mid\mathcal{F}_{n-1}] = 0$, $\{Z_n(\theta)\}_{n\geq1}$ is a martingale-difference sequence, and hence

Thus, dependence in the losses is retained in the risk target, while fluctuations of empirical risk around that target admit martingale-based inference.

Stream-Aware Policy Testing

For each candidate $\theta_j \in\Theta$, StreamLTT tests $H_j : R_{\mathrm{pred},N}(\theta_j) > \alpha$, so rejecting $H_j$ certifies $R_{\mathrm{pred},N}(\theta_j) \leq\alpha$. By Eq. (6), empirical risk differs from this target by a martingale average. This structure permits a finite-sample test under bounded losses without specifying the dependence among the losses.

Theorem 3.2 provides a finite-sample guarantee using only the deterministic loss bound $B_j$. The same robustness can make the test conservative: its normalization reflects the worst-case loss range rather than the conditional variability actually encountered along the stream. When the latter is substantially smaller, a variance-sensitive normalization can provide tighter certification. The proof is given in Appendix B.1.

Variance-adaptive testing. To capture this stream-specific variability, define the predictable quadratic variation

Whereas $B_j$ gives a uniform worst-case bound, $V_N^2(\theta)$ accumulates the conditional variance of the martingale increments along the deployment stream. We require this variation to admit a stable asymptotic scale.

Since bounded losses yield uniformly bounded martingale increments, $v_{N,j} \to\infty$ makes the conditional Lindeberg condition automatic. Together with Assumption 3.3, the martingale central limit theorem then yields the variance-adaptive test below.

Theorem 3.4 trades the finite-sample robustness of Theorem 3.2 for adaptation to the predictable variability of the stream. When this variability is well below the worst-case range, the adaptive normalization can reduce conservatism and yield tighter certification. The proof is given in Appendix § 10.§ 2. Under IID losses, the construction recovers the standard population-risk test.

Estimating the predictable variance. The adaptive test depends on $V_N(\theta)$, which is generally unknown. For Bernoulli losses, $\mathbb{E}[Z_n^2(\theta) \mid\mathcal{F}_{n-1}] = p_n(\theta)(1-p_n(\theta))$. Given an online estimate $\widehat{p}_n(\theta)$ of $p_n(\theta)$, we use the plug-in estimator

The next result shows that consistency of the one-step risk estimates is sufficient for the variance estimation required by Theorem 3.4.

Lemma 3.6 therefore links estimation of the one-step predictable risks directly to the normalization used by the variance-adaptive test. The proof is given in Appendix § 10.§ 3.

From Policy Tests to a Certified Region

The preceding results establish valid policy-level tests for predictable risk. We now lift these guarantees from individual candidates to the entire policy family. Let $\mathcal{R}_N \subseteq\{1,\ldots,M\}$ denote the rejection set returned by the multiple-testing procedure and define $\widehat{\Theta}_{\mathrm{safe}} := \{\theta_j : j \in\mathcal{R}_N\}$. The corresponding safe region is $\Theta_{\mathrm{safe},N} := \{\theta\in\Theta: R_{\mathrm{pred},N}(\theta) \le\alpha\}$. Controlling false rejections across the candidates then yields simultaneous certification of $\widehat{\Theta}_{\mathrm{safe}}$.

Algorithm 1 StreamLTT for risk-certified policy selection

Require: Certification stream $\{(X_n,Y_n)\}_{n=1}^{N}$; candidate family $\Theta=\{\theta_1,\ldots,\theta_M\}$; loss $L_n(\theta)$; risk target $\alpha$; bounds $\{B(\theta_j)\}_{j=1}^{M}$; FWER level $\delta$; MODE $\in\{\mathrm{ADAPTIVE},\mathrm{HOEFFDING}\}$.

Ensure: Certified region $\widehat{\Theta}_{\mathrm{safe}}$; optional policy $\widehat{\theta}$.

1. Freeze the candidate family and all test-specific quantities.

2. Initialize empirical-risk summaries and, for ADAPTIVE mode, predictable-variance summaries.

3. for $n=1,\ldots,N$ do

4. Observe $\mathcal{F}_{n-1}$ and form decisions for all $\theta_j\in\Theta$.

5. Reveal $Y_n$ and record $L_n(\theta_j)$ for all $\theta_j\in\Theta$.

6. if MODE = ADAPTIVE then

7. Estimate $\widehat{p}_n(\theta_j)$ and update the predictable-variance summaries via Eq. (10).

8. end if

9. end for

10. for $j=1,\ldots,M$ do

11. Compute $\widehat{R}_N(\theta_j)\leftarrow N^{-1}\sum_{n=1}^{N}L_n(\theta_j)$.

12. if MODE = ADAPTIVE then

13. Compute $p_j\leftarrow p_j^{\mathrm{SL}}$ via Eq. (9).

14. else

15. Compute $p_j\leftarrow p_j^{\mathrm{H}}$ via Eq. (7).

16. end if

17. end for

18. Apply FWER control at level $\delta$ to $\{p_j\}_{j=1}^{M}$ and form $\widehat{\Theta}_{\mathrm{safe}}$ from the rejected hypotheses.

19. if $\widehat{\Theta}_{\mathrm{safe}}=\varnothing$ then

20. return “no certified policy”.

21. end if

22. Optionally select $\widehat{\theta}\in\widehat{\Theta}_{\mathrm{safe}}$ according to the deployment criterion.

23. return $\widehat{\Theta}_{\mathrm{safe}}$ and optional $\widehat{\theta}$.

Theorem 3.7 lifts the policy-level validity established above to simultaneous certification of the candidate family. The policy-level tests account for dependence along the deployment stream, whereas FWER control accounts for multiplicity across candidate policies. The first guarantee ensures that the entire certified region is safe with probability at least $1-\delta$; the second shows that any subsequent data-dependent selection from this region inherits the same risk control without additional testing.

Complete procedure. Algorithm 1 summarizes the resulting StreamLTT pipeline. Each candidate is evaluated using either the finite-sample Hoeffding test or the variance-adaptive test, and the resulting $p$-values are combined with FWER control to form $\widehat{\Theta}_{\mathrm{safe}}$. A deployment policy may then be selected from the certified region according to the downstream objective. For $M$ candidates, StreamLTT maintains $O(M)$ running summaries and requires $O(M)$ computation per stream round.

Experiments

We evaluate StreamLTT along two complementary dimensions of risk certification: the tightness of the certified risk frontier and the control of false certification. We refer to the variance adaptive test in Theorem 3.4 as StreamLTT-A and the finite-sample Hoeffding test in Theorem 3.2 as StreamLTT-H. Our primary comparison evaluates StreamLTT-A against IID, dependence aware, and sequential certification baselines. StreamLTT-H provides a finite sample counterpart that isolates the effect of predictable variance adaptation.

Experimental Setup

Datasets and forecasting protocol. We consider seven real world forecasting datasets spanning three domains: Electricity [37], Appliances Energy (Candianedo et al., 2017), and Household Power [20] for energy forecasting; PEMS04 and PEMS08 [19] for traffic forecasting; and Beijing Weather [8] and Jena Weather (Max Planck Institute for Biogeochemistry, 2016) for temperature forecasting. The datasets vary in sequence length and sampling frequency, and their temporal order is preserved throughout the experiment. Each dataset is split chronologically into a training set (20%), a validation set (2%), a calibration set (8%), and a certification stream (70%). A temporal convolutional network (TCN) [5] is trained for one step forecasting on the training set, selected on the validation set, and then frozen. The calibration set determines auxiliary quantities, such as severe miss tolerances and regime thresholds, which are fixed before certification. The remaining observations are processed sequentially as the certification stream, with the same frozen forecasts shared across all certification methods. At each round, only information available before the corresponding outcome is revealed is used; history dependent decisions may additionally use previously revealed outcomes. Model specifications, preprocessing, and further protocol details are provided in Appendix § 11.§ 1.

Deployment risks. We consider seven binary deployment risks spanning pointwise, regime dependent, directional, and history dependent errors, as summarized in Table 1. These risks range from standard forecasting failures to decisions whose losses depend on the current operating regime or previously observed outcomes, providing diverse dependence structures for stream level certification. All auxiliary quantities are fixed before certification, and each risk defines a separate certification problem. Complete risk constructions are provided in Appendix § 11.§ 2.

Risk	Structure	Error event	$\alpha$
Interval miss	Pointwise	$Y_i \notin C_m(X_i)$	.10
Lower-tail miss	Pointwise	$Y_i < \ell_i(m)$	.10
Severe miss	Pointwise	$\lvert Y_i-\widehat{\mu}_i\rvert> ms_i+e_0$	.10
Task-regime miss	Regime	$G_i=1,\ Y_i\notin C_m(X_i)$	.05
Opposite direction	Temporal	$a_i(\theta)z_i^d=-1$	.10
Missed reversal	History	$r_i=1,\ \widehat{r}_i(\theta)=0$	.10
False reversal	History	$r_i=0,\ \widehat{r}_i(\theta)=1$	.10

Methods. Our primary method is StreamLTT-A, the variance adaptive test in Theorem 3.4. We compare it with IID-LTT [4], LTT-HB-Holm [22, 6], HAC-CLT-Holm [32], CircularBlockBootstrap-Holm [33], and EB-Howard-Bonferroni [24], covering IID inference, bounded loss concentration, long run variance correction, block resampling, and sequential concentration, respectively. We also evaluate StreamLTT-H, the finite sample test in Theorem 3.2, which retains the same predictable risk target as StreamLTT-A while removing predictable variance adaptation. All methods use the same risk targets and family wise levels. Further details are provided in Appendix § 11.§ 3.

Evaluation protocol and metrics. We evaluate all $7 \times7 = 49$ dataset–risk combinations at $\delta\in\{0.10, 0.20\}$ using 2,000 independent repetitions per configuration. We assess certification through risk tightness and false certification control. Empirical FWER is the fraction of repetitions in which at least one certified policy has predictable risk above $\alpha$. For each nonempty certified region, $R_{\max}$ denotes its largest predictable risk, and $\overline{R}_{\max}$ its mean across repetitions. We report the risk gap $\alpha-\overline{R}_{\max}$ and FWER gap $\delta-\overline{\mathrm{FWER}}$, where values near zero indicate fuller use of the respective risk and family wise budgets. We additionally report the certified fraction, the proportion of candidate policies retained after certification. Further evaluation details, risk wise and dataset level results, and Monte Carlo uncertainty are provided in Appendix § 11.§ 4.

Results

StreamLTT-A approaches both certification boundaries. Figure 1 shows that StreamLTT-A uses substantially more of both the risk and family wise error budgets. At $\delta=0.10$, its mean risk gap is 0.0090, placing the certified frontier close to the prescribed risk boundary, compared with 0.0302 for IID-LTT and approximately 0.030–0.032 for the dependence and concentration based alternatives. This tighter frontier is not accompanied by excessive false certification: StreamLTT-

A attains empirical FWER 0.0902, close to the prescribed $\delta= 0.10$. In contrast, the competing procedures leave considerably more slack in the risk budget, the family wise budget, or both. StreamLTT-A therefore expands certification toward the admissible risk boundary while maintaining the intended family wise control.

Table 2 resolves this aggregate improvement at the dataset level for three representative risks; complete results for all seven risks are provided in the appendix. For interval miss, the baseline frontiers remain between 0.043 and 0.052, whereas StreamLTT-A reaches 0.095–0.097 across all seven datasets. The separation persists for lower tail miss, where the baseline values range from 0.052 to 0.070 while StreamLTT-A remains at 0.095–0.097. For the history dependent missed reversal risk, the gain is particularly pronounced on PEMS04 and PEMS08, where StreamLTT-A reaches 0.095 compared with approximately 0.048–0.050 for the competing methods. Across all 49 dataset–risk combinations, StreamLTT-A retains 76.7% of the candidate family while maintaining mean empirical FWER 0.0902 at the prescribed $\delta= 0.10$.

Risk	Method	Electricity	Appliances	Household	PEMS04	PEMS08	Beijing	Jena
Interval ($\alpha= .10$)	IID-LTT	.051/.028	.046/.013	.048/.020	.051/.017	.044/.015	.047/.019	.045/.012
Interval ($\alpha= .10$)	LTT-HB-Holm	.050/.006	.045/.004	.047/.009	.051/.006	.043/.006	.046/.005	.045/.004
Interval ($\alpha= .10$)	HAC-CLT-Holm	.051/.035	.046/.014	.048/.025	.051/.021	.044/.019	.047/.024	.045/.017
Interval ($\alpha= .10$)	BlockBootstrap-Holm	.051/.034	.046/.015	.048/.025	.052/.021	.044/.019	.047/.023	.045/.018
Interval ($\alpha= .10$)	EB-Howard-Bonf.	.050/.000	.045/.000	.047/.000	.050/.000	.043/.000	.046/.000	.044/.000
Interval ($\alpha= .10$)	StreamLTT-A	.096/.084	.096/.082	.097/.102	.096/.094	.095/.087	.096/.089	.096/.083
Lower tail ($\alpha= .10$)	IID-LTT	.055/.045	.069/.031	.053/.032	.065/.025	.066/.026	.069/.020	.064/.025
Lower tail ($\alpha= .10$)	LTT-HB-Holm	.053/.014	.069/.010	.052/.009	.065/.009	.066/.009	.069/.004	.063/.007
Lower tail ($\alpha= .10$)	HAC-CLT-Holm	.055/.052	.069/.037	.053/.037	.066/.030	.066/.031	.070/.023	.064/.032
Lower tail ($\alpha= .10$)	BlockBootstrap-Holm	.055/.052	.069/.036	.053/.036	.066/.031	.066/.031	.070/.023	.064/.031
Lower tail ($\alpha= .10$)	EB-Howard-Bonf.	.052/.000	.068/.000	.052/.000	.065/.000	.065/.000	.069/.000	.063/.000
Lower tail ($\alpha= .10$)	StreamLTT-A	.097/.095	.096/.097	.095/.093	.097/.080	.097/.088	.096/.081	.097/.085
Missed reversal ($\alpha= .10$)	IID-LTT	.080/.087	.064/.026	.075/.040	.048/.016	.050/.018	.089/.091	.081/.047
Missed reversal ($\alpha= .10$)	LTT-HB-Holm	.077/.030	.063/.008	.074/.015	.048/.005	.049/.006	.086/.031	.080/.018
Missed reversal ($\alpha= .10$)	HAC-CLT-Holm	.081/.099	.064/.033	.075/.045	.049/.022	.050/.023	.090/.099	.081/.052
Missed reversal ($\alpha= .10$)	BlockBootstrap-Holm	.081/.097	.064/.033	.075/.047	.049/.022	.050/.023	.090/.099	.081/.053
Missed reversal ($\alpha= .10$)	EB-Howard-Bonf.	.073/.000	.063/.000	.074/.000	.048/.001	.049/.000	.079/.000	.080/.000
Missed reversal ($\alpha= .10$)	StreamLTT-A	.097/.088	.095/.092	.095/.085	.095/.090	.095/.089	.096/.093	.082/.092

Dependence aware inference alone does not remove certification slack. The risk gaps help isolate the source of the improvement. At $\delta= 0.10$, IID-LTT has a mean gap of 0.0302, while HAC-CLT-Holm and CircularBlockBootstrap-Holm attain 0.0300 despite explicitly accounting for temporal dependence. StreamLTT-A reduces this gap to 0.0090. Thus, adjusting the sampling distribution for dependence has little effect on the certified frontier when the underlying risk target remains unchanged. StreamLTT instead certifies average predictable risk, conditioning risk on the information available at each decision time. The empirical separation therefore supports the distinction in Section 3.1: dependence aware inference changes how risk is tested, whereas StreamLTT also changes what risk is certified along the deployment stream.

StreamLTT-A adapts to the available family-wise budget. Increasing $\delta$ from 0.10 to 0.20 provides a direct test of whether the certification procedure responds to a larger tolerance for false certification. StreamLTT-A’s empirical FWER increases from 0.0902 to 0.1888, closely tracking the corresponding prescribed levels, while its mean risk gap remains small and decreases from 0.0090 to 0.0078. The baselines respond much less strongly: at $\delta= 0.20$, IID-LTT, HAC-CLT-Holm, and CircularBlockBootstrap-Holm reach FWER only 0.1125, 0.1217, and 0.1208, respectively, while LTT-HB-Holm remains at 0.0376 and EB-Howard-Bonferroni near zero. Their risk gaps also remain between 0.0283 and 0.0321. Thus, increasing the family wise budget primarily expands certification under StreamLTT-A, rather than leaving the additional error budget unused.

Ablation: predictable variance adaptation primarily improves family wise budget utilization. StreamLTT-H and StreamLTT-A certify the same average predictable risk and differ only in the policy level test, allowing us to isolate the effect of predictable variance adaptation. Their certified frontiers are already similar: at $\delta= 0.10$ and $0.20$, StreamLTT-H attains mean risk gaps 0.0118 and 0.0112, compared with 0.0090 and 0.0078 for StreamLTT-A. The certified fractions are likewise close, increasing from 75.3% to 76.7% at $\delta= 0.10$ and from 75.7% to 77.1% at $\delta= 0.20$.

The main difference appears in false certification. StreamLTT-H has empirical FWER only 0.0001 and 0.0006 at the two family wise levels, leaving nearly the entire error budget unused, whereas StreamLTT-A reaches 0.0902 and 0.1888. Thus, the tight frontier is already largely present when predictable risk is paired with the finite sample test; predictable variance adaptation provides a modest additional improvement in frontier tightness and certified fraction, but its dominant empirical effect is to reduce the conservatism of policy level testing and make substantially fuller use of the available family wise budget.

Conclusion

We introduced StreamLTT for risk certified policy selection over dependent and nonstationary deployment streams. StreamLTT replaces a fixed population risk with average predictable risk, aligning certification with the information available at each decision time and yielding martingale fluctuations for policy level testing. This structure supports both a finite sample bounded loss test and a variance adaptive test, which combine with family wise error control to certify the candidate policy region. Across seven forecasting datasets and seven deployment risks, StreamLTT-A produced substantially tighter certified frontiers than IID, dependence aware, bootstrap, and sequential concentration baselines, while empirical false certification closely tracked the prescribed family wise level. The finite sample ablation further separated the two sources of improvement: the predictable risk formulation accounted for most of the frontier tightening, while predictable variance adaptation primarily reduced testing conservatism and made fuller use of the available family wise budget. Together, these results extend LTT to evolving deployment streams by addressing both what risk should be certified and how it should be tested under dependence.

Reproducibility Statement

Complete proofs and experimental details are provided in the appendix to support reproduction of the theoretical and empirical results.

Ethics Statement

This study used public datasets and needs no ethical approval.

AI Use Statement

Generative AI was used only to assist with writing and language polishing.

Supplementary File

Related Works

StreamLTT is related to the following works on risk controlled prediction, sequential calibration, inference under dependence, and multiple testing.

Risk Control and Sequential Calibration

Conformal prediction provides distribution free predictive uncertainty under exchangeability [38, 36, 28, 2], with extensions to regression, classification, and more general predictive settings [35]. Related work on selective prediction studies decisions with abstention and the tradeoff between prediction frequency and reliability [9, 12, 15, 16]. More recently, conformal risk control extends calibration from coverage to general monotone losses [3], while Learn then Test (LTT) casts risk controlled policy selection as a multiple testing problem [4]. StreamLTT retains the LTT architecture of testing a fixed candidate family and certifying the rejected policies simultaneously, but replaces the stationary population risk with average predictable risk.

A complementary literature studies conformal calibration when data arrive sequentially or exchangeability fails. Adaptive Conformal Inference and its extensions use sequential feedback to maintain coverage under distribution shift [17, 42, 18], while SPCI exploits temporal structure in sequential conformity scores [41]. These methods primarily adapt predictive calibration as new observations arrive. StreamLTT instead keeps the candidate policy family fixed and asks which policies satisfy a deployment risk constraint. Online calibration and StreamLTT therefore operate at complementary stages: prediction can adapt over time, while certification evaluates a fixed family of deployment policies.

Inference under Dependence

Classical inference under temporal dependence often retains the target estimand while modifying its sampling approximation. HAC methods estimate long run variance under serial dependence [32, 1], while block bootstrap methods preserve dependence by resampling consecutive observations [26, 33, 34, 27]. These approaches motivate our HAC and block bootstrap baselines. They address how an empirical statistic should be normalized or resampled under dependence, but do not by themselves specify which deployment risk should replace a stationary marginal expectation when conditional risk changes along the stream.

Martingale concentration provides a complementary route to sequential inference. Classical bounded increment and variance sensitive inequalities include Hoeffding, Bentkus, and Freedman type bounds [22, 6, 14], with self normalized extensions developed for martingale processes [10]. More recent work uses nonnegative supermartingales to obtain time uniform concentration and confidence sequences [24, 25], including betting based inference for bounded means [39]. StreamLTT uses this martingale perspective after defining the risk target: centering each loss by its one step predictable risk yields a martingale difference sequence. Boundedness then gives the finite sample test, while predictable quadratic variation supports the variance adaptive construction.

Simultaneous Policy Certification

Certification across a finite policy family additionally requires multiplicity control. Classical procedures such as Bonferroni and Holm provide control of the family wise error rate, with related stepwise and closed testing procedures providing further refinement [23, 30, 21]. LTT connects these tools to policy selection by interpreting rejection of an unsafe risk null as certification of the corresponding policy [4]. StreamLTT preserves this multiplicity testing layer: once valid $p$ values for predictable risk are available, standard family wise correction yields a simultaneously certified region and permits subsequent selection within that region.

Proofs for StreamLTT

This appendix provides the proofs and supporting arguments for Section 3. We follow the order of the main text: the finite-sample StreamLTT test, the variance-adaptive StreamLTT test, consistency of the predictable-variance estimator, and simultaneous certification with post-certification selection.

Proof of the Finite-Sample StreamLTT Test

Proof of Theorem 3.2. Fix $\theta_j \in\Theta$ and suppress the argument $\theta_j$. Recall $p_n = \mathbb{E}[L_n \mid\mathcal{F}_{n-1}]$ and $Z_n = L_n-p_n$. By construction, $\mathbb{E}[Z_n \mid\mathcal{F}_{n-1}] = 0$, so $\{Z_n\}_{n\ge1}$ is a martingale-difference sequence with respect to $\{\mathcal{F}_n\}_{n\ge0}$.

Since $0 \le L_n \le B_j$ and conditional expectation preserves these bounds, $0 \le p_n \le B_j$. Conditional on $\mathcal{F}_{n-1}$,

so $Z_n$ lies in an interval of length $B_j$. Hoeffding’s lemma therefore gives, for every $s > 0$,

Define

Then $\{M_k,\mathcal{F}_k\}_{k=0}^{N}$ is a nonnegative supermartingale, since

Hence $\mathbb{E}[M_N] \le1$.

For any $t > 0$, on the event $\{\sum_{n=1}^{N} Z_n \le-Nt\}$,

Markov’s inequality therefore yields

Optimizing over $s > 0$ at $s=4t/B_j^2$ gives

where (6) was used.

Fix $z \in(0,1)$ and define $t_z := B_j\sqrt{\log(1/z)/(2N)}$. By (7), $p_j^{\mathrm{H}} \le z$ implies $\widehat{R}_N \le\alpha-t_z$. Under $H_j:R_{\mathrm{pred},N}>\alpha$,

Thus

Applying (13),

The cases $z=0$ and $z=1$ are immediate. Hence $p_j^{\mathrm{H}}$ is finite-sample super-uniform under $H_j:R_{\mathrm{pred},N}(\theta_j)>\alpha$. ∎

Proof of the Variance-Adaptive StreamLTT Test

Proof of Theorem 3.4. Fix $\theta_j \in\Theta$ and suppress the argument $\theta_j$. Recall $p_n = \mathbb{E}[L_n \mid\mathcal{F}_{n-1}]$ and $Z_n = L_n - p_n$. By construction, $\{Z_n\}_{n\ge1}$ is a martingale-difference sequence with respect to $\{\mathcal{F}_n\}_{n\ge0}$, and Eq. (6) gives

Since $0 \le L_n \le B_j$, we have $0 \le p_n \le B_j$ and therefore $|Z_n| \le B_j$. By Assumption 3.3, there exists a deterministic sequence $v_{N,j}^2 > 0$ such that $v_{N,j}^2 \to\infty$ and

where $V_N^2$ is defined in Eq. (8).

For any $\varepsilon> 0$, $v_{N,j} \to\infty$ implies $\varepsilon v_{N,j} > B_j$ for all sufficiently large $N$. Hence

eventually. Thus the conditional Lindeberg condition holds.

Combining the conditional Lindeberg condition with predictable-variance stabilization, the martingale central limit theorem gives

Moreover, $V_N/v_{N,j}\xrightarrow{p}1$, so Slutsky’s theorem yields

By the variance-consistency condition in Theorem 3.4, $\widehat{V}_N/V_N\xrightarrow{p}1$. Applying Slutsky’s theorem again,

The statistic underlying Eq. (9) can be written as

Under $H_j:R_{\mathrm{pred},N}>\alpha$, the second term is positive. Since $\Phi$ is increasing,

Consequently, for every $z\in(0,1)$,

Using Eq. (14),

The endpoints $z=0$ and $z=1$ are immediate. Hence $p_j^{\mathrm{SL}}$ is asymptotically super-uniform under $H_j:R_{\mathrm{pred},N}(\theta_j)>\alpha$.

Proof of Predictable-Variance Consistency

Proof of Lemma 3.6. Fix $\theta_j \in\Theta$ and suppress the argument $\theta_j$. Since $L_n \in\{0,1\}$,

Therefore,

while Eq. (10) gives

Let $g(x) := x(1-x)$ for $x \in[0,1]$. Since $|g(x)-g(y)| \leq|x-y|$ for all $x,y \in[0,1]$,

where the second inequality follows from Cauchy–Schwarz.

By the assumed nondegenerate variance limit $V_N^2/N \overset{p}{\to} \sigma_j^2>0$,

Hence $\widehat{V}_N^2/V_N^2 \overset{p}{\to} 1$, and the continuous mapping theorem gives

Finally, define $v_{N,j}^2:=N\sigma_j^2$. Then $v_{N,j}^2\to\infty$ and

Thus Assumption 3.3 also holds. $\square$

Proof of Simultaneous StreamLTT Certification

Proof of Theorem 3.7. At horizon $N$, define the unsafe-policy index set

and let $\mathcal{R}_N$ denote the rejection set returned by the multiple-testing procedure. By definition,

Under this convention, false certification occurs exactly when at least one unsafe policy is rejected:

Suppose first that the policy-level $p$-values are finite-sample super-uniform under their corresponding null hypotheses and that the multiple-testing procedure controls the FWER at level $\delta$. Then

Together with Eq. (15), this gives

or equivalently,

We next establish the post-certification statement. Let $\widehat{\theta}$ be any possibly data-dependent selection satisfying $\widehat{\theta}\in\widehat{\Theta}_{\mathrm{safe}}$ whenever $\widehat{\Theta}_{\mathrm{safe}}\ne\varnothing$. On the event $\widehat{\Theta}_{\mathrm{safe}}\subseteq\Theta_{\mathrm{safe},N}$, every policy in the certified region satisfies the predictable-risk constraint. Therefore,

It follows that

This proves both finite-sample statements.

For the variance-adaptive StreamLTT test, consider Bonferroni correction, which rejects candidate $j$ whenever $p_j^{\mathrm{SL}}\le\delta/M$. By the union bound,

For each fixed candidate $\theta_j$, Theorem 3.4 gives

Since $M$ is fixed and finite, taking the limit superior in Eq. (17) yields

Using Eq. (15),

which is equivalent to

Finally, Eq. (16) holds for every $N$. Hence

This proves both asymptotic statements.

Experimental Details

This section provides additional details for the experiments in § 4, including the forecasting protocol, construction of the deployment risks, certification methods, and evaluation metrics. All model configurations, auxiliary quantities, and test specific choices are fixed before certification.

Datasets and Forecasting Protocol

We consider seven real world forecasting datasets spanning energy, traffic, and temperature forecasting. The energy datasets are Electricity [37], Appliances Energy [7], and Household Power [20]; the traffic datasets are PEMS04 and PEMS08 [19]; and the temperature datasets are Beijing Weather [8] and Jena Weather [31]. The datasets differ in sequence length, sampling frequency, and

caption: Summary of the forecasting datasets used in the experiments.

:::

temporal structure, while their chronological order is preserved throughout preprocessing, forecasting, and certification.

Each dataset is split chronologically into a training set (20%), a validation set (2%), a calibration set (8%), and a certification stream (70%). A temporal convolutional network (TCN) is trained for one step forecasting on the training set, selected using the validation set, and then frozen. The calibration set determines auxiliary quantities required by the deployment risks, such as severe miss tolerances and regime thresholds, which remain fixed throughout certification. The final 70% of observations form the certification stream and are processed sequentially in their original temporal order.

Forecasting model. We use the same TCN forecasting architecture across all seven datasets. The model takes the previous 24 observations of the target variable as input and predicts the next outcome; Electricity additionally includes temporal and customer information. Inputs and targets are standardized using statistics from the training set. Models are trained by minimizing mean squared error with validation based early stopping, and the selected model is frozen before calibration and certification. The resulting forecasts are shared across all certification methods.

At each certification round, predictions and deployment decisions use only information available before the corresponding outcome is revealed. History dependent decisions may additionally use previously revealed outcomes, whereas unrevealed outcomes remain unavailable.

tbl:3 summarizes the application domains and forecasting tasks.

Deployment Risk Construction

We consider the seven binary deployment risks summarized in Table 1. They span pointwise forecasting failures, regime dependent events, directional decisions, and history dependent reversal errors. Each loss takes values in $\{0,1\}$, matching the Bernoulli setting used for predictable variance estimation in § 3.2.

For the interval based risks, let $\hat{\mu}_i$ denote the frozen point forecast and define

where $s_i$ is a calibration based residual scale fixed before certification, and $m$ indexes the candidate interval widths.

Interval miss. The interval miss loss is $L_i^{\mathrm{int}}(m) = \mathbb{I}\{Y_i \notin C_m(X_i)\}$, with target $\alpha= 0.10$.

Lower tail miss. The lower tail loss isolates outcomes below the lower prediction limit, $L_i^{\mathrm{low}}(m) = \mathbb{I}\{Y_i < \ell_i(m)\}$, with $\alpha= 0.10$.

Severe interval miss. Let $e_0$ denote an additional tolerance determined from the calibration set. We define $L_i^{\mathrm{sev}}(m) = \mathbb{I}\{|Y_i - \hat{\mu}_i| > ms_i + e_0\}$, with $\alpha= 0.10$. This risk captures errors exceeding the ordinary uncertainty scale by the prescribed tolerance.

Task regime miss. Let $G_i \in\{0,1\}$ indicate an extreme operating regime determined from the calibration set. The corresponding loss is $L_i^{\mathrm{reg}}(m) = \mathbb{I}\{G_i = 1,\, Y_i \notin C_m(X_i)\}$, with $\alpha= 0.05$. This is a joint event risk that emphasizes coverage conditional on $G_i = 1$.

Opposite direction. Let $a_i(\theta) \in\{-1,0,1\}$ denote the action issued by policy $\theta$ and $z_i^d \in\{-1,0,1\}$ the realized direction. We define $L_i^{\mathrm{dir}}(\theta) = \mathbb{I}\{a_i(\theta)z_i^d = -1\}$, with $\alpha= 0.10$.

Missed reversal. Let $r_i \in\{0,1\}$ denote the realized trend reversal indicator and $\widehat{r}_i(\theta) \in\{0,1\}$ the corresponding policy prediction. The loss is $L_i^{\mathrm{miss}}(\theta) = \mathbb{I}\{r_i = 1, \widehat{r}_i(\theta) = 0\}$, with $\alpha= 0.10$.

False reversal. The complementary false alert loss is $L_i^{\mathrm{false}}(\theta) = \mathbb{I}\{r_i = 0, \widehat{r}_i(\theta) = 1\}$, with $\alpha= 0.10$. Together, the missed and false reversal risks distinguish the two asymmetric failure modes of reversal detection.

All quantities defining these risks are either fixed using precertification data or constructed from information available before the corresponding outcome is revealed.

Certification Methods

Our primary implementation, StreamLTT-A, uses the variance adaptive test in Theorem 3.4. We compare it with five baselines: IID-LTT [4], LTT-HB-Holm [22, 6], HAC-CLT-Holm [32], CircularBlockBootstrap-Holm [33], and EB-Howard-Bonferroni [24]. We additionally evaluate StreamLTT-H, which uses the same average predictable risk target as StreamLTT-A but replaces the adaptive test with the finite sample test in Theorem 3.2.

IID-LTT and LTT-HB-Holm both retain the standard LTT population risk under independent calibration losses. IID-LTT uses a CLT approximation to empirical risk, whereas LTT-HB-Holm uses the finite sample Hoeffding–Bentkus bounded loss test. HAC-CLT-Holm instead adjusts the long run variance for temporal dependence, while CircularBlockBootstrap-Holm approximates the dependent sampling distribution through circular block resampling. EB-Howard-Bonferroni uses an empirical Bernstein concentration construction.

For StreamLTT-A, the one step predictable risk $p_n(\theta) = \mathbb{E}[L_n(\theta) \mid\mathcal{F}_{n-1}]$ is estimated using only information available before $Y_n$ is revealed. Given $\widehat{p}_n(\theta)$, the predictable quadratic variation is estimated as

as in Eq. equation 10. Pointwise risks use the current forecasting information and policy quantities, while regime and history dependent risks add the corresponding information available in $\mathcal{F}_{n-1}$. All estimator configurations are fixed before certification.

StreamLTT-H retains the same predictable risk formulation but uses the finite sample martingale Hoeffding $p$ value in Eq. equation 7. Comparing StreamLTT-H with StreamLTT-A therefore isolates predictable variance adaptation while holding the certified risk target fixed.

Evaluation Protocol and Metrics

We evaluate all $7 \times7 = 49$ dataset–risk combinations at $\delta\in\{0.10, 0.20\}$ using 2,000 independent repetitions per configuration.

A family wise error occurs when at least one certified policy exceeds the prescribed predictable risk target,

Empirical FWER is the frequency of this event across repetitions.

For each nonempty certified region, define

and let $\overline{R}_{\max}$ denote its mean across nonempty repetitions. We measure certification tightness by the risk gap $\alpha- \overline{R}_{\max}$ and family wise budget utilization by the FWER gap $\delta- \widehat{\mathrm{FWER}}$. Values near zero indicate that the corresponding certification boundary is approached closely.

As a complementary measure, the certified fraction $|\widehat{\Theta}_{\mathrm{safe}}|/|\Theta|$ records the proportion of candidate policies retained after certification. Monte Carlo uncertainty is computed from the 2,000 repetitions. For empirical FWER $\widehat{q}$, its Monte Carlo standard error is $\sqrt{\widehat{q}(1-\widehat{q})/2000}$; we additionally compute binomial confidence intervals for FWER and Monte Carlo uncertainty for $\overline{R}_{\max}$ and the certified fraction.

Additional Experimental Results

This section complements § 4.2 with complete dataset level results for all seven deployment risks at both family wise levels, the finite sample StreamLTT ablation, and Monte Carlo diagnostics.

Dataset Level Results at $\delta= 0.10$

Table 4 reports the complete dataset level comparison at $\delta= 0.10$. Each entry gives $\overline{R}_{\max}/\widehat{\mathrm{FWER}}$, allowing frontier tightness and false certification to be compared jointly across datasets and risks.

Risk	Method	Electricity	Appliances	Household	PEMS04	PEMS08	Beijing	Jena
Interval miss ($\alpha= .10$)	IID-LTT	.051/.028	.046/.013	.048/.020	.051/.017	.044/.015	.047/.019	.045/.012
Interval miss ($\alpha= .10$)	LTT-HB-Holm	.050/.006	.045/.004	.047/.009	.051/.006	.043/.006	.046/.005	.045/.004
Interval miss ($\alpha= .10$)	HAC-CLT-Holm	.051/.035	.046/.014	.048/.025	.051/.021	.044/.019	.047/.024	.045/.017
Interval miss ($\alpha= .10$)	BlockBootstrap-Holm	.051/.034	.046/.015	.048/.025	.052/.021	.044/.019	.047/.023	.045/.018
Interval miss ($\alpha= .10$)	EB-Howard-Bonf.	.050/.000	.045/.000	.047/.000	.050/.000	.043/.000	.046/.000	.044/.000
Interval miss ($\alpha= .10$)	StreamLTT-A	.096/.084	.096/.082	.097/.102	.096/.094	.095/.087	.096/.089	.096/.083
Lower tail miss ($\alpha= .10$)	IID-LTT	.055/.045	.069/.031	.053/.032	.065/.025	.066/.026	.069/.020	.064/.025
Lower tail miss ($\alpha= .10$)	LTT-HB-Holm	.053/.014	.069/.010	.052/.009	.065/.009	.066/.009	.069/.004	.063/.007
Lower tail miss ($\alpha= .10$)	HAC-CLT-Holm	.055/.052	.069/.037	.053/.037	.066/.030	.066/.031	.070/.023	.064/.032
Lower tail miss ($\alpha= .10$)	BlockBootstrap-Holm	.055/.052	.069/.036	.053/.036	.066/.031	.066/.031	.070/.023	.064/.031
Lower tail miss ($\alpha= .10$)	EB-Howard-Bonf.	.052/.000	.068/.000	.052/.000	.065/.000	.065/.000	.069/.000	.063/.000
Lower tail miss ($\alpha= .10$)	StreamLTT-A	.097/.095	.096/.097	.095/.093	.097/.080	.097/.088	.096/.081	.097/.085
Severe miss ($\alpha= .10$)	IID-LTT	.054/.083	.080/.082	.063/.079	.075/.090	.069/.091	.072/.097	.073/.096
Severe miss ($\alpha= .10$)	LTT-HB-Holm	.051/.029	.078/.026	.060/.024	.073/.029	.066/.028	.070/.030	.071/.031
Severe miss ($\alpha= .10$)	HAC-CLT-Holm	.054/.092	.080/.094	.063/.087	.075/.097	.069/.099	.073/.105	.073/.107
Severe miss ($\alpha= .10$)	BlockBootstrap-Holm	.054/.091	.080/.093	.063/.086	.075/.096	.069/.098	.073/.105	.073/.105
Severe miss ($\alpha= .10$)	EB-Howard-Bonf.	.050/.000	.078/.000	.059/.000	.072/.000	.065/.000	.069/.000	.070/.000
Severe miss ($\alpha= .10$)	StreamLTT-A	.072/.085	.086/.082	.063/.080	.075/.090	.069/.091	.072/.097	.073/.096
Task regime miss ($\alpha= .05$)	IID-LTT	.025/.097	.016/.016	.032/.017	.040/.048	.040/.059	.025/.015	.038/.050
Task regime miss ($\alpha= .05$)	LTT-HB-Holm	.024/.033	.016/.005	.032/.005	.040/.017	.040/.018	.025/.004	.038/.017
Task regime miss ($\alpha= .05$)	HAC-CLT-Holm	.026/.099	.016/.018	.032/.021	.040/.049	.040/.060	.025/.018	.038/.051
Task regime miss ($\alpha= .05$)	BlockBootstrap-Holm	.026/.100	.016/.017	.032/.021	.040/.048	.040/.060	.025/.017	.038/.050
Task regime miss ($\alpha= .05$)	EB-Howard-Bonf.	.023/.000	.016/.000	.032/.000	.039/.000	.040/.000	.025/.000	.038/.000
Task regime miss ($\alpha= .05$)	StreamLTT-A	.034/.097	.044/.117	.047/.086	.046/.093	.045/.105	.048/.084	.045/.096
Opposite direction ($\alpha= .10$)	IID-LTT	.095/.085	.094/.086	.083/.085	.088/.087	.091/.088	.074/.088	.054/.089
Opposite direction ($\alpha= .10$)	LTT-HB-Holm	.095/.032	.094/.029	.082/.027	.088/.026	.090/.029	.072/.026	.052/.034
Opposite direction ($\alpha= .10$)	HAC-CLT-Holm	.095/.095	.094/.097	.083/.091	.089/.099	.091/.098	.074/.101	.055/.099
Opposite direction ($\alpha= .10$)	BlockBootstrap-Holm	.095/.097	.094/.096	.084/.092	.089/.098	.091/.098	.074/.098	.055/.099
Opposite direction ($\alpha= .10$)	EB-Howard-Bonf.	.087/.000	.091/.000	.082/.000	.087/.000	.090/.000	.071/.000	.050/.000
Opposite direction ($\alpha= .10$)	StreamLTT-A	.097/.087	.094/.087	.083/.086	.097/.087	.097/.088	.092/.090	.055/.091
Missed reversal ($\alpha= .10$)	IID-LTT	.080/.087	.064/.026	.075/.040	.048/.016	.050/.018	.089/.091	.081/.047
Missed reversal ($\alpha= .10$)	LTT-HB-Holm	.077/.030	.063/.008	.074/.015	.048/.005	.049/.006	.086/.031	.080/.018
Missed reversal ($\alpha= .10$)	HAC-CLT-Holm	.081/.099	.064/.033	.075/.045	.049/.022	.050/.023	.090/.099	.081/.052
Missed reversal ($\alpha= .10$)	BlockBootstrap-Holm	.081/.097	.064/.033	.075/.047	.049/.022	.050/.023	.090/.099	.081/.053
Missed reversal ($\alpha= .10$)	EB-Howard-Bonf.	.073/.000	.063/.000	.074/.000	.048/.001	.049/.000	.079/.000	.080/.000
Missed reversal ($\alpha= .10$)	StreamLTT-A	.097/.088	.095/.092	.095/.085	.095/.099	.095/.089	.096/.093	.082/.092
False reversal ($\alpha= .10$)	IID-LTT	.089/.088	.094/.086	.084/.081	.049/.018	.045/.020	.091/.105	.076/.098
False reversal ($\alpha= .10$)	LTT-HB-Holm	.089/.029	.094/.033	.083/.028	.048/.004	.044/.006	.090/.028	.074/.030
False reversal ($\alpha= .10$)	HAC-CLT-Holm	.090/.098	.094/.096	.084/.091	.049/.022	.045/.023	.091/.110	.076/.111
False reversal ($\alpha= .10$)	BlockBootstrap-Holm	.090/.100	.094/.096	.084/.089	.049/.022	.045/.024	.091/.110	.076/.107
False reversal ($\alpha= .10$)	EB-Howard-Bonf.	.088/.000	.090/.000	.082/.000	.048/.000	.044/.000	.090/.000	.073/.000
False reversal ($\alpha= .10$)	StreamLTT-A	.097/.088	.097/.086	.096/.081	.096/.086	.096/.088	.098/.105	.094/.100

The complete results show that the aggregate behavior in § 4.2 is not driven by a single dataset or risk. The largest separation appears for interval and lower tail miss, where StreamLTT-A consistently moves the frontier close to $\alpha= 0.10$ while the competing methods remain substantially below it. Similar gains appear for missed and false reversal, particularly on datasets where the baseline frontier is far from the target.

Risk	Method	Electricity	Appliances	Household	PEMS04	PEMS08	Beijing	Jena
Interval miss ($\alpha= .10$)	IID-LTT	.052/.054	.046/.022	.049/.038	.052/.035	.044/.032	.048/.033	.046/.029
	LTT-HB-Holm	.050/.016	.046/.009	.048/.016	.051/.011	.043/.011	.046/.011	.045/.008
	HAC-CLT-Holm	.053/.060	.047/.029	.049/.046	.053/.044	.045/.041	.048/.038	.046/.038
	BlockBootstrap-Holm	.053/.061	.047/.028	.049/.046	.053/.044	.045/.042	.048/.038	.046/.038
	EB-Howard-Bonf.	.050/.000	.045/.000	.047/.000	.050/.000	.043/.000	.046/.000	.044/.000
	StreamLTT-A	.097/.185	.096/.178	.097/.211	.097/.186	.095/.181	.097/.177	.097/.182
Lower tail miss ($\alpha= .10$)	IID-LTT	.057/.095	.070/.063	.055/.062	.066/.050	.067/.051	.070/.035	.065/.059
	LTT-HB-Holm	.054/.030	.069/.021	.052/.017	.065/.018	.066/.018	.069/.014	.063/.018
	HAC-CLT-Holm	.057/.102	.070/.070	.055/.068	.067/.058	.068/.065	.070/.044	.065/.064
	BlockBootstrap-Holm	.057/.101	.070/.070	.055/.068	.067/.057	.068/.065	.070/.043	.065/.065
	EB-Howard-Bonf.	.052/.000	.068/.000	.052/.000	.065/.000	.065/.000	.069/.000	.063/.000
	StreamLTT-A	.098/.184	.096/.183	.096/.199	.098/.175	.098/.185	.097/.175	.097/.184
Severe miss ($\alpha= .10$)	IID-LTT	.059/.180	.082/.173	.067/.179	.078/.188	.072/.189	.075/.195	.076/.185
	LTT-HB-Holm	.053/.056	.079/.057	.062/.058	.074/.067	.068/.066	.071/.067	.072/.063
	HAC-CLT-Holm	.060/.194	.082/.186	.067/.191	.078/.202	.072/.198	.076/.202	.076/.199
	BlockBootstrap-Holm	.059/.192	.082/.185	.067/.188	.078/.199	.072/.197	.076/.203	.076/.198
	EB-Howard-Bonf.	.050/.000	.078/.000	.059/.000	.072/.000	.065/.000	.069/.000	.070/.000
	StreamLTT-A	.075/.185	.088/.175	.067/.180	.078/.188	.072/.190	.075/.196	.076/.189
Task regime ($\alpha= .05$)	IID-LTT	.028/.189	.017/.033	.032/.034	.040/.093	.041/.105	.026/.032	.039/.096
	LTT-HB-Holm	.025/.064	.016/.007	.032/.010	.040/.029	.040/.038	.025/.009	.038/.034
	HAC-CLT-Holm	.028/.190	.017/.035	.033/.038	.040/.098	.041/.111	.026/.034	.039/.099
	BlockBootstrap-Holm	.028/.188	.017/.035	.033/.040	.040/.096	.041/.108	.026/.034	.039/.100
	EB-Howard-Bonf.	.023/.000	.016/.000	.032/.000	.039/.000	.040/.000	.025/.000	.038/.000
	StreamLTT-A	.036/.189	.045/.217	.048/.193	.046/.191	.045/.212	.048/.192	.046/.197
Opposite direction ($\alpha= .10$)	IID-LTT	.096/.187	.095/.178	.085/.172	.090/.180	.092/.190	.077/.189	.060/.196
	LTT-HB-Holm	.095/.061	.094/.064	.083/.060	.088/.057	.090/.063	.073/.056	.053/.063
	HAC-CLT-Holm	.096/.198	.095/.191	.085/.186	.090/.190	.092/.203	.077/.199	.060/.207
	BlockBootstrap-Holm	.096/.196	.095/.189	.085/.183	.090/.190	.092/.203	.077/.199	.060/.206
	EB-Howard-Bonf.	.087/.000	.092/.000	.082/.000	.087/.000	.090/.000	.071/.000	.055/.000
	StreamLTT-A	.097/.188	.095/.180	.085/.179	.098/.182	.098/.192	.093/.192	.060/.200
Missed reversal ($\alpha= .10$)	IID-LTT	.084/.173	.065/.058	.076/.085	.050/.042	.051/.041	.092/.191	.082/.090
	LTT-HB-Holm	.079/.061	.064/.018	.074/.030	.048/.011	.049/.011	.088/.064	.081/.035
	HAC-CLT-Holm	.085/.182	.065/.066	.076/.096	.050/.051	.051/.047	.093/.206	.082/.100
	BlockBootstrap-Holm	.084/.180	.065/.066	.076/.095	.050/.051	.051/.046	.093/.204	.082/.100
	EB-Howard-Bonf.	.073/.000	.063/.000	.074/.000	.048/.001	.049/.000	.079/.000	.080/.000
	StreamLTT-A	.098/.173	.095/.191	.095/.192	.096/.186	.096/.185	.097/.195	.084/.201
False reversal ($\alpha= .10$)	IID-LTT	.091/.183	.095/.178	.085/.163	.050/.043	.046/.039	.092/.209	.079/.199
	LTT-HB-Holm	.089/.066	.094/.058	.083/.057	.048/.010	.045/.012	.091/.075	.075/.070
	HAC-CLT-Holm	.091/.197	.095/.193	.085/.178	.050/.050	.047/.043	.092/.226	.079/.210
	BlockBootstrap-Holm	.091/.195	.095/.192	.085/.177	.050/.049	.047/.043	.092/.224	.079/.208
	EB-Howard-Bonf.	.088/.000	.091/.000	.082/.000	.048/.000	.044/.000	.090/.000	.073/.000
	StreamLTT-A	.098/.186	.097/.179	.097/.163	.097/.192	.097/.198	.098/.214	.095/.202

Other risks illustrate why frontier tightness and FWER must be interpreted jointly. For severe miss and opposite direction, HAC-CLT-Holm and the block bootstrap often attain empirical FWER near 0.10, but their frontiers remain close to IID-LTT on several datasets. Conversely, StreamLTT-A maintains FWER near the prescribed level while improving the frontier where additional admissible policies are available. The attainable frontier can itself vary by dataset, as seen for opposite direction on Jena and for severe miss on several streams.

Dataset Level Results at =0.20

Table 5 reports the corresponding dataset level results after increasing the family wise level to $\delta= 0.20$. This comparison shows how each procedure responds when a larger false certification budget is available.

The larger family wise budget produces two distinct behaviors. StreamLTT-A responds across nearly all dataset–risk pairs: empirical FWER generally moves toward 0.20, while the certified frontier remains close to the corresponding risk target whenever the candidate family contains policies near that boundary. For example, the interval and lower tail frontiers remain around 0.095–0.098 across all seven datasets, while their empirical FWER values move from the neighborhood of 0.10 at $\delta= 0.10$ toward 0.20.

The baseline response is less uniform. HAC-CLT-Holm and BlockBootstrap-Holm approach the larger FWER budget for some risks, such as severe miss and opposite direction, but remain substantially below it for interval, lower tail, task regime, and several reversal settings. LTT-HB-Holm remains more conservative throughout, while EB-Howard-Bonferroni leaves essentially the entire family wise budget unused. Moreover, increasing $\delta$ changes many baseline frontiers only modestly. For interval miss, for example, their $\bar{R}_{\max}$ values remain around 0.04–0.05, whereas StreamLTT-A remains near 0.10.

Finite Sample StreamLTT Ablation

To complement the aggregate ablation in § 4.2, Table 6 reports the risk wise results for StreamLTT-H after averaging across the seven datasets. StreamLTT-H retains the same average predictable risk target as StreamLTT-A but uses the finite sample martingale Hoeffding test, isolating the effect of removing predictable variance adaptation.

Risk	$\delta= 0.10$			$\delta= 0.20$
	$\bar{R}_{\max}$	FWER	Frac.	$\bar{R}_{\max}$	FWER	Frac.
Interval miss	.092	.000	.322	.093	.001	.325
Lower tail miss	.094	.000	.699	.094	.001	.704
Severe miss	.070	.000	.981	.070	.001	.982
Task regime miss	.042	.000	.647	.043	.000	.652
Opposite direction	.086	.000	.947	.086	.001	.953
Missed reversal	.091	.000	.864	.091	.001	.866
False reversal	.093	.000	.809	.094	.001	.814

Across risks, StreamLTT-H already produces relatively tight certified frontiers, while its empirical FWER remains essentially zero at both family wise levels. Increasing $\delta$ from 0.10 to 0.20 changes the frontier and certified fraction only modestly, consistent with the conservatism of the finite sample Hoeffding test. Together with the StreamLTT-A results reported above, this risk wise breakdown supports the aggregate ablation: the predictable risk formulation accounts for most of the frontier tightening, whereas predictable variance adaptation primarily enables the procedure to use more of the available family wise budget.