WorksMathematics
Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation
Abstract
Reinforcement learning (RL) is increasingly used to train robust, adaptable systems across healthcare, e-commerce, and robotics. In these settings, legal, ethical, or logistical concerns may preclude certain key variables that drive decisions from being collected during deployment time. This gives rise to a setting known as runtime confounding: the logged data hold full observability, but deployment time lacks knowledge of certain variables, known as runtime confounders. Unlike single timepoint and non-adaptive settings, runtime confounding under reinforcement learning has the potential to propagate over time, as earlier decisions affect future states, observations, and action assignments, leading to biased learning. We address these problems with three contributions: we (1) develop policy learning algorithms under runtime-confounding, (2) develop semi-parametric efficient off-policy estimation procedures, and (3) provide diagnostics for understanding the price of runtime confounding.
Introduction
Sequential decision-making rules are often deployed in settings with less information than was available during training. For example, a clinical decision-support tool may be trained on records that include laboratory results, clinician notes, or socioeconomic variables, but some of these variables that relate to sensitive subgroup classification, such as socioeconomics, may not be allowed as inputs when the tool is deployed to protect against discriminative profiling. This problem can occur in other settings with high-stakes decision-making, such as adjusting credit limits or child welfare screening. Despite having knowledge of all variables that influence outcomes in historical logs, during deployment time, one may not have access to an individual’s protected attributes for ethical, legal, or logistical reasons. More formally, these variables observed in historical data but are unavailable at deployment are known as runtime confounders, and the setting of misaligned logging data and deployment data is referred to as runtime confounding [6].
While a large part of reinforcement learning (RL) literature studies a related problem of unobserved confounding, to the best of our knowledge, no work studies runtime confounding precisely. More precisely, past work assume both logging and deployment lack confounders, and handle the non-identifiability with a range of approaches including causal bounds and online identification [2] (Zhang and Bareinboim 2019), sensitivity analysis [9, 14], and proximal or partially observable formulations [16, 4, 18]. In runtime confounding, the difficulty lies in the fact that the deployed policy can only condition on a variable set that does not include the runtime confounders, leading to a different policy class than that used in the historical data and violations to sequential randomization during deployment. Coston et al. [6] address runtime confounding in single-period counterfactual prediction, and recent work studies a conformal prediction approach in the same setting [3].
We address this gap by proposing a semiparametric-efficient framework for policy learning and off-policy evaluation under runtime confounding. The key observation is that the runtime restriction enters only through the numerator of an importance ratio. Hence, one may take a reweighting approach to recover the value of any policy in the runtime-confounding policy class, with all nuisances estimated from the full data. As a result, reinforcement learning under runtime confounding can fit within the ideas of doubly robust off-policy evaluation [7, 17, 8, 19] with cross-fitted nuisances [5], as well as the literature on dynamic treatment regimes for restricted policies [12, 15] and the broader literature on policy learning with observational data [1].
Our contributions are as follows.
Policy learning. We propose a doubly robust policy learning algorithm for reinforcement learning under runtime confounding over a class of runtime deployable policies (Algorithm 1). Its regret relative to the full-information optimum is an irreducible runtime regret gap and efficient rates (Theorem 3).
Semiparametric-Efficient Off-Policy Evaluation. For a fixed runtime policy, we propose a doubly-robust estimator that is asymptotically normal and attains the semiparametric efficiency bound, yielding valid confidence intervals and testing procedures (Theorem 2).
Diagnosing and Characterizing the Price of Runtime Confounding. Because the runtime confounders are logged, we can test the price of runtime confounding. We propose a test for the irreducible runtime regret gap, giving a decision-relevant quantity for having to accept or choosing these runtime confounders.
Problem Setting
Notation
We consider a setting where we have complete observation logging data available over sample trajectories and timepoints. Trajectories are denoted by
where is the full-state consisting of (available at deployment) and (recorded in the logs, but unavailable at runtime), is the action, and is the reward, bounded by . Define the histories
so that is the runtime version of the history, or with each full state replaced by its runtime component . The observed data are generated by a logging policy which we allow to be known or unknown by the researcher, where rewards and states are generated according to and , respectively. Importantly, during deployment is unavailable so that newer target policies which depend on are infeasible. Figure 1 outlines our problem setting.

Figure 1. Problem setting. is observed in the historical data, but unmeasured during deployment and in turn unavailable to the target policy.
We occasionally write to make dependence on the policy explicit, though we often omit this notation for brevity. Letting denote a fixed sequence of actions, we let denote the counterfactual reward under action sequence , and similarly let denote the counterfactual state at time that would be observed under action sequence .
Target Parameters and Identification
Our target is the -discounted objective function under a fixed policy :
In RL, one typically seeks to learn an optimal policy, and evaluate fixed policies, which act on the full history , or current contexts and any relevant past information. What makes runtime confounding unique compared to standard RL settings is that we do not observe at runtime, so policies that require are infeasible. In turn, we instead aim to perform inference and learning over the policy class which acts only over the variables available at runtime:
Importantly, is not immediately identified for policies other than the logging policy that generated the observed data. We assume the following identification conditions to identify for any .
(Id1) Sequential ignorability. For all , and for any generic action sequence ,
(Id2) Consistency: and for all .
(Id3) Overlap. for all and .
(Id1) – (Id3) are standard identification assumptions in policy learning [13, 20] and reinforcement learning [8].
With these identification assumptions, one can identify from the observed data. Let the superscript RC denote the runtime version of any fixed object. Then define the runtime importance ratio as:
Here, the importance weight takes a particular form: In contrast to the non RC setting which uses , the RC importance weight has the numerator correspond to runtime-confounded quantities while the denominator is the full information policy. For , we then recursively define the Q- and Value function.
where . Again, the primary difference is the action density that we marginalize over—a runtime policy.
Under Assumptions (Id1)-(Id3), is identified by the observed data functionals:
Importantly, the identifying functional in (3) does not discard the historical . Discarding variables that unavailable at runtime is a common practice in data science and machine learning ([6]), but will tend to yield invalid identifying functionals when the discarded variables are truly confounders. Specifically, focusing on the importance-weighted identifying functional in (3), if we were to to instead use the importance weights
the resulting identifying functional would generally not equal . In turn, standard methods for both policy learning and inference ([8]) that naively estimate this functional would target the wrong parameter inducing unintended bias.
Efficient Estimation and Policy Learning
Both policy-learning and off-policy estimation will build on the efficient influence function (EIF) for the identifying functional (3) of the target parameter . EIFs are crucial ingredients for constructing estimators of statistical parameters, and enable valid asymptotic inference even when estimators of underlying nuisance functions themselves converge at slower rates ([10]). Throughout, we will estimate the EIF of using nuisance models for the propensity and the conditional outcome model, denoted by . Writing for the Bellman residual, the efficient influence function of at fixed is given by the following Theorem:
Theorem 1 (Efficient Influence Function under Runtime-Confounding) Fix . Under our assumptions ((Id1)-(Id3)), the efficient influence function of is
where . The proof is found in Section A.1.
The EIF in (4) is analogous to the EIF considered by Kallus and Uehara [8] for non-Markov decision processes, where here the importance weights are restricted to accommodate policies .
Theorem 1 motivates an estimator based on the EIF , where we use cross-fitting ([5]) to ensure the resulting asymptotic distribution remains tractable. Specifically, letting denote nuisances cross-fitted on folds not containing unit , estimate the objective by
The full estimation procedure is outlined in Algorithm 1.
Algorithm 1: Doubly robust policy learning and inference under runtime confounding
Input: Logged trajectories $\{O_i\}_{i=1}^n$ with $S_{it}=(X_{it},U_{it})$; policy class $\Pi\subseteq\Pi_X$; folds $K$; overlap constant $\eta_g$; $C_Q=R_{\max}\sum_t\gamma^{t-1}$
Output: $\hat{\pi}$, $\hat{J}(\cdot)$, confidence intervals
Split $[n]$ into folds $\mathcal{I}_1,\ldots,\mathcal{I}_K$
// Propensities (full information), fitted on $\mathcal{I}_{-k}$
for $k=1,\ldots,K$ do
for $t=1,\ldots,T$ do
Fit $\hat{g}^{(-k)}_t(a\mid H_t)$ by regressing $A_t$ on $H_t$
end
end
// Cross-fitted value estimate for $\pi\in\Pi_X$
Function Value$(\pi)$:
for $k=1,\ldots,K$ do
Set $\hat{V}^{(-k)}_{T+1}\equiv 0$
// Fitted-$Q$ evaluation on $\mathcal{I}_{-k}$
for $t=T,\ldots,1$ do
Fit $\hat{q}^{\pi,(-k)}_t(H_t,a)$ by regressing $R_t+\gamma\hat{V}^{(-k)}_{t+1}(H_{t+1})$ on $(H_t,A_t)$
$\hat{V}^{(-k)}_t(H_t)\leftarrow\sum_a\pi_t(a\mid H_t^X)\hat{q}^{\pi,(-k)}_t(H_t,a)$
end
// Evaluation on the held-out fold $\mathcal{I}_k$
for $i\in\mathcal{I}_k$ do
$\hat{\rho}^{\mathrm{RC}}_{i,1:t}\leftarrow\prod_{s\leq t}\frac{\pi_s(A_{is}\mid H^X_{is})}{\hat{g}^{(-k)}_s(A_{is}\mid H_{is})}$ for $t\in[T]$
$\hat{\delta}^{\pi}_{it}\leftarrow R_{it}+\gamma\hat{V}^{(-k)}_{t+1}(H_{i,t+1})-\hat{q}^{\pi,(-k)}_t(H_{it},A_{it})$
$\psi_i(\pi)\leftarrow\hat{V}^{(-k)}_1(H_{i1})+\sum_{t=1}^T\gamma^{t-1}\hat{\rho}^{\mathrm{RC}}_{i,1:t}\hat{\delta}^{\pi}_{it}$
end
end
return $\hat{J}(\pi)=\frac{1}{n}\sum_{i=1}^n\psi_i(\pi)$ and $\hat{\sigma}^2(\pi)=\frac{1}{n}\sum_{i=1}^n(\psi_i(\pi)-\hat{J}(\pi))^2$
// Learning and inference
Policy learning: $\hat{\pi}\leftarrow\operatorname*{arg\,max}_{\pi\in\Pi}\operatorname{Value}(\pi)$
Inference for a fixed $\pi$: report $\hat{J}(\pi)\pm z_{1-\alpha/2}\hat{\sigma}(\pi)/\sqrt{n}$
Error Decomposition with the EIF
We now describe the high-level decomposition used to produce our policy learning and off-policy evaluation guarantees. Let denote the empirical measure, denote with cross-fitted nuisance estimates, and and denote estimation error of each respective nuisance function. To analyze , we decompose our error into the following terms:
where the remainder is given by:
This is a standard decomposition in the semiparametric literature [10], where term (E) is sampling error, term (R) denotes nuisance bias, and term (C) is controlled by cross-fitting. This decomposition to establish regret bounds and conditions under which attains asymptotic normality.
Before introducing our results, we introduce the following estimation assumptions, which are sufficient to control the remainder term in (7).
(Est1) Product rate at . .
(Est2) Consistency at . for each .
(Est3) Complexity. has finite uniform entropy integral bounded by .
(Est1)-(Est2) are standard rate assumptions in semiparametric inference supporting the use of non-Donsker function classes in estimation. (Est3) is an entropy condition on the runtime policy class, rather than the nuisance function estimator, used for regret analysis. In the coming sections, we establish off-policy evaluation (3.1) and policy learning (3.2) results based on these assumptions.
Efficient Off-Policy Evaluation
We now fix and give conditions under which in (5) is asymptotically normal and efficient based on Theorem 1.
Theorem 2 (Asymptotic normality and efficiency). Under our identification assumptions (Id1)-(Id3), and estimation assumptions (Est1)-(Est2),
where
is the semiparametric efficiency bound. Moreover is consistent, so is an asymptotically valid confidence interval. The proof is found in Section A.3.
This result gives us a way to characterize uncertainty for runtime-confounded policies, and do valid testing procedures. The finding is analogous to the results from Kallus and Uehara [8], but RC adjusted.
Policy Learning under Runtime-Confounding
Let over the unrestricted (full-information) policies, and . The learned policy maximizes the cross-fitted estimator (5) over runtime policies,
so that takes only as input and is deployable.
Theorem 3 (Regret under runtime confounding). Under the identification assumptions (Id1)–(Id3), (Est1) at and , and (Est3),
where and where we recall that is our positivity constraint for the logger. The proof is found in Section A.2.
The result states that our learned policy, under sufficient regularity conditions, follows the same finding of rates with an irreducible runtime gap .
Diagnosing and Testing Runtime Confounding
One may ask the gap we are losing in the decision-making process by using variables available only at runtime. This can inform future experimentation or provide an interpretable gap when it is unavoidable for domain experts. We lower-bound and test the irreducible runtime gap , which is the value lost by withholding at deployment. Because is a difference of maxima, inference on this parameter can be non-regular [11]. To address this difficult, we employ a sample-splitting approach. Specifically, we split the sample into parts and . On , we learn a full-information policy on and a runtime policy by Algorithm 1. On , estimate the runtime gap by
By Theorem 2, is asymptotically normal. Assuming and ,
is an asymptotically valid confidence interval for , and we can test , by . Under this can fail if the two policies coincide so their scores are equal, so we impose .
Discussion
Under high-stakes or high-latency decision-making, deployment of standard methods can be biased due to variables present in training logs not being present during implementation. We introduce policy learning and efficient off-policy evaluation methods for reinforcement learning under runtime confounding to overcome this gap. We additionally provide diagnostics to understand bias from using a runtime confounded policy, informing future design of experiments and the price of runtime confounding. Extensions to online settings and decaying overlap are left for future work.
Acknowledgments
This work was supported by National Institutes of Health (NIH) grant T32HL139426. We thank Rachel Nethery for helping birth this research.
References
Proofs
Proof of Theorem 1
Proof. In general, note the factorization given by
The EIF will be derived using the point mass contamination method, by perturbations on each of these densities, except , which is assumed to be logged. Let , , we consider perturbations of the form
T=1 We start with the case when . Define , .
Our target parameter is
Perturb along each population object: , . Then, by the chain rule, we have
Noting the usual calculation,
This reduces to
Hence, the EIF at reduces to , for importance weight at one timepoint given by .
General T Now, we wish to extend this logic to general . The difference is that is a recursive object:
In the same manner as above, we have
where we have used that
For convenience, define the following time IS-weighted EIF of as
Setting the base case for our weights to 1, , gives as our object as interest.
Since the value function is a function of the future timepoints (e.g. for being a function of ), we require unrolling this sum and understanding .
Borrowing the key calculation from above on , we perturb the distributions, and calculate the EIF on :
Plugging this into (the directional derivative for along ), using in the first term and in the second, gives
where in the last line is evaluated at the point mass.
Unroll this from , with ,
Hence, the EIF form follows.
Note, if is unknown and we need to account for local perturbations in the derivation, just as in the iid case, our score is orthogonal yielding the same EIF as above.
Proof of Theorem 3
Step 1: Regret decomposition. Adding and subtracting , the value at the optimal runtime-confounded policy, we can decompose the regret as
is irreducible, representing the gap between the optimal policy and optimal runtime policy. It remains to bound the estimation term, which is the only term depending on the data.
Step 1: Value-error decomposition. We can bound the estimation term as follows:
The first inequality follows by adding and subtracting and , and applying the basic inequality. The second equality follows plugging in (6) and expanding. The final inequality follows from the triangle inequality and realizability.
Step 2: Empirical process control, on supremum terms.
Term 2 : By (Id3), , and , and the same bounds hold for the estimated versions. Hence and admit an envelope . Then, by symmetrization and Dudley’s entropy bound under (Est3),
Term 2 : Cross-fitting splits the term over folds of size . Let the function class implied by this process be given by . Then,
Conditional on the training folds , the nuisance estimates are fixed, so is a fixed class of differences between elements of and , with envelope . Its entropy is at most the sum of the entropies of these two classes, so by (Est3). Dudley’s integral on the evaluation fold gives . Since is fixed, Markov’s inequality gives .
Step 3: Second-order remainder (R) By (7), applied fold by fold with , for any ,
where the second inequality uses from (Id3), and the third follows from Cauchy–Schwarz. (Est1) then shows .
Proof of Theorem 2
Proof
Decomposition. Our decomposition yielded
We show the result by demonstrating and that (R) and (C) are .
Term (E) and the variance. By (Id3), is bounded by , and . Since trajectories are i.i.d., the central limit theorem gives . We are explicit about the form of . Write , where
for Bellman residual . Expanding the variance of the sum,
Mean Zero. By the tower property and the Bellman equation ,
Covariances with . Conditioning on and pulling out the measurable factors and
Covariances across periods. For , is -measurable, so
Variances. , so
and .
Combining. Substituting into (11), we find
Hence
which is the efficiency bound since is the EIF (Theorem 1).
Term (R). See Theorem 2 for the proof. By the bound on the remainder in the proof of Theorem 3, at the fixed ,
by (Est1).
Term (C). Fix a fold with units. Conditional on the training folds , is a fixed function so
By Chebyshev’s inequality, this term is . The difference can be decomposed as
By (Id3), , . Also, , and is at most . Combining, we can bound the difference by
Hence by (Est2), .
Consistency of our Variance Estimator. Since ,
With cross-fitting, . Conditional on , is a fixed function evaluated on the independent fold , so by Cauchy--Schwarz
The right side is by our analysis on Term (C), so by Markov’s inequality applied conditionally on , for each .
It remains to show . The terms are i.i.d. and bounded by , so by the law of large numbers in probability. Therefore in probability.
References
- [1]Susan Athey and Stefan Wager. Policy learning with observational data. Econometrica, 89 (1):133–161, 2021.arxiv.org/abs/1702.02896
- [2]Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: A causal approach. In Advances in Neural Information Processing Systems, volume 28, 2015.
- [3]Keith Barnatchez, Kevin P. Josey, Rachel C. Nethery, and Giovanni Parmigiani. Debiased machine learning for conformal prediction of counterfactual outcomes under runtime confounding. In Proceedings of the Fifth Conference on Causal Learning and Reasoning, volume 323 of Proceedings of Machine Learning Research, pages 1345–1379, 2026.arxiv.org/abs/2604.03772
- [4]Andrew Bennett and Nathan Kallus. Proximal reinforcement learning: Efficient off-policy evaluation in partially observed Markov decision processes. Operations Research, 72 (3):1071–1086, 2024.arxiv.org/abs/2110.15332
- [5]Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1–C68, 2018.
- [6]Amanda Coston, Edward Kennedy, and Alexandra Chouldechova. Counterfactual predictions under runtime confounding. Advances in neural information processing systems, 33:4150–4162, 2020.arxiv.org/abs/2006.16916
- [7]Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 652–661, 2016.arxiv.org/abs/1511.03722
- [8]Nathan Kallus and Masatoshi Uehara. Double reinforcement learning for efficient off-policy evaluation in markov decision processes. Journal of Machine Learning Research, 21(167):1–63, 2020. URL https://jmlr.org/papers/v21/19-827.html.arxiv.org/abs/1908.08526
- [9]Nathan Kallus and Angela Zhou. Confounding-robust policy improvement. In Advances in Neural Information Processing Systems, volume 31, 2018.arxiv.org/abs/1805.08593
- [10]Edward H Kennedy. Semiparametric doubly robust targeted double machine learning: a review. Handbook of statistical methods for precision medicine, pages 207–236, 2024.arxiv.org/abs/2203.06469
- [11]Alexander R. Luedtke and Mark J. van der Laan. Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy. The Annals of Statistics, 44(2):713–742, 2016a.DOI
- [12]Alexander R. Luedtke and Mark J. van der Laan. Super-learning of an optimal dynamic treatment rule. The International Journal of Biostatistics, 12(1):305–332, 2016b. doi: 10.1515/ijb-2015-0052.DOI
- [13]Susan A. Murphy, Mark J. van der Laan, James M. Robins, and Conduct Problems Prevention Research Group. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 96(456):1410–1423, 2001. doi: 10.1198/016214501753383227.
- [14]Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy evaluation for sequential decisions under unobserved confounding. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2003.05623
- [15]Shosei Sakaguchi. Estimation of optimal dynamic treatment assignment rules under policy constraints. Quantitative Economics, 16(3):981–1022, 2025. doi: 10.3982/QE22288.arxiv.org/abs/2106.05031
- [16]Guy Tennenholtz, Uri Shalit, and Shie Mannor. Off-policy evaluation in partially observable environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10276–10283, 2020.arxiv.org/abs/1909.03739
- [17]Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2139–2148, 2016.arxiv.org/abs/1604.00923
- [18]Masatoshi Uehara, Haruka Kiyohara, Andrew Bennett, Victor Chernozhukov, Nan Jiang, Nathan Kallus, Chengchun Shi, and Wen Sun. Future-dependent value-based off-policy evaluation in POMDPs. In Advances in Neural Information Processing Systems, volume 36, 2023.arxiv.org/abs/2207.13081
- [19]Mark J. van der Laan and Alexander R. Luedtke. Targeted learning of the mean outcome under an optimal dynamic treatment rule. Journal of causal inference, 3(1):61–95, 2015.
- [20]Baqun Zhang, Anastasios A. Tsiatis, Eric B. Laber, and Marie Davidian. Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions. Biometrika, 100(3):681–694, 2013.DOI