betaFeedback

WorksComputer Science

DECODE: Decomposable Environments for COntrolled Sequential-Decision Evaluation

Abstract

When a sequential decision system falls short of an optimum, its score does not reveal which modifications would improve its decisions. DECODE is a benchmark for testing such modifications through controlled experiments on planning and available information. Its consumer-credit collections generator provides five partially observed environments that share known dynamics matrices and vary a memory parameter and process noise. The experiments change selected inputs or planning rules while keeping the rest of the solver fixed. Each comparison specifies the solver's state inputs, response inputs and planning horizon. On the deterministic base environment, which was selected for a large gap between six-month and one-month planning with the hidden state, extending planning from one to six months increases the scores of two gradient-boosted solvers by 15 to 17 percentage points of the exact optimum, computed with the hidden state and borrower traits. Even at this horizon, the best graded solver, whose repayment response is fitted on logs, remains 15.6 points below that optimum. In a specified intervention sequence, three changes to its state inputs and their evaluation during search lower its score by 6.3 points, and replacing its fitted repayment model with the true response, evaluated with hidden borrower traits, then raises it by 15.6, a net recovery of nine of those points. Because the intermediate effects are conditional on the repayment model and the order of the changes, they do not uniquely assign the gap to separate causes. Within these limits, DECODE makes hypotheses about solver performance testable through specified interventions under known dynamics and explicit information conditions.

Introduction

Benchmarks for sequential decision systems often summarize performance with a final score. A diagnostic evaluation can then test how specific changes in planning or available information affect policy value. Interpreting the resulting change in value requires stating what information each reference receives and what decision rule it applies. DECODE provides partially observed environments with known equations and specified reference policies for these comparisons. Each environment is a finite-horizon partially observable Markov decision process (POMDP) [19] with time-invariant dynamics. In the consumer-credit collections setting used here, a hidden account state evolves under these dynamics. Monthly actions affect its evolution and repayment through known response maps, while policy value is expected net cash up to the first recovery within the evaluation horizon. Under partial observation, a solver’s belief is the probability distribution of the hidden state given everything it has observed. To evaluate changes within this setting, DECODE combines the generator, a common policy-value evaluator and the reference policies into a measurement instrument. Knowing the world’s equations allows changes to a solver’s information and planning inputs to be evaluated against references computed through closed-form or path-dependent calculations on deterministic members and Monte Carlo estimates under process noise. Each reference’s guarantees depend on its information inputs and planning rule. In particular, the optimum with access to the hidden state differs from the filter-based reference.

This paper proposes a reproducible instrument with specified input interventions and reference policies, together with an empirical characterization of the released environments. Its comparisons test controlled changes to information and planning inputs for solvers fitted on partially observed collection logs. Graded solvers use logged observations, while reference solvers receive specified input substitutions, including ground-truth quantities. Both are evaluated under the same grading rule. The environments vary along two dynamics parameters, a memory parameter and process noise, which change the dependence on history and the uncertainty of future states. These parameters define the five settings released as “members”, with construction checks (Figure 1). The environments are decomposable in the sense that the transition, the reading model and the repayment response are specified separately, so each can be varied in the world or substituted into a solver one at a time. This property does not imply an additive decomposition of a performance gap. This design relates to diagnostic suites [27, 33], generators with certified optima on the full state [17] or on a sufficient information state (Querdane et al. 2026), and oracle-based analyses of estimator errors [8]. The policy-value experiments are complemented by predictive probes that test whether history improves prediction of the repayment response. A separate comparison asks the corresponding predictive question for observed outcomes in public loan-performance data. These predictive gains do not by themselves establish improvements in policy value. The release supplies the generator, the five members and the artifacts supporting these measurements.

Overview of DECODE

Figure 1. Overview of DECODE.

On the member with the memory parameter and process noise both set to zero (Base), extending the planning horizon from one month to six increases the score of the same fitted solver by 15 to 17 points of the exact optimum. The construction checks of Base require a gap of 15 to 40 points between the exact and onemonth optima on a training-world prefix and a best plan that varies across loans, so this setting is selected to reward planning over the horizon and plans that differ across loans. Contrasts between the tested solvers at a fixed horizon have point estimates below 4 points. Even at the longer horizon, the best graded solver, whose repayment response is fitted on logs, remains 15.6 points below the exact optimum. In the declared intervention sequence, three changes to the solver’s state inputs and their evaluation during search lower its score by 6.3 points, and supplying the world’s response, evaluated with the true hidden borrower traits, then raises it by 15.6, a net recovery of nine of those points. The step effects are conditional on the fitted response and the order of the changes. The step that replaces the history features with filter features is small in that sequence, but the filter ignores repayment outcomes. Beyond these changes in horizon and inputs, the benchmark checks how a planner evaluates and updates its own plans. Chaining marginal repayment probabilities overstates the filter-based planner’s value relative to joint integration of the same plan under the same belief. Allowing the planner to re-plan on fresh observations improves value on the noisy members relative to executing its initial plan.

This paper is organized as follows. Section 2 describes the world generator, evaluation protocol, reference policies and solvers. Section 3 presents the experimental results. Section 4 discusses their interpretation and limitations. Section 5 reviews related work. Section 6 concludes the paper.

DECODE Benchmark Environment

This section explains how DECODE’s environments, evaluation protocol and reference calculations fit together in a diagnostic experiment. The world equations define how the state evolves, what a solver observes and how actions affect repayment. These equations govern the training logs and the trajectories used to evaluate policies. The protocol specifies how the logs are collected and how policy value is measured. Reference comparisons use this protocol to assess specified changes in information and planning, while predictive probes measure how history contributes to prediction. Construction checks state the conditions for releasing the environments, and the solver descriptions identify the models, information inputs and decision rules compared under this protocol.

A world is a population of clients whose monthly outcomes depend on a hidden state that evolves from one month to the next. In the consumer debt collection setting used here, clients are delinquent loans, outcomes are repayments, and actions are contact choices and settlement offers, each at a cost. A graded solver never directly accesses the hidden state. It observes the public account state and a noisy reading of the hidden state, and chooses an action per client per month.

Two parameters vary these dynamics, the memory parameter δ\delta and the process-noise scale σ\sigma. Five settings are released as members. Base sets both parameters to zero. Mem-0.5 and Mem-4 use δ=0.5\delta= 0.5 and 4 with no process noise, while Noise-0.1 and Noise-0.2 use σ=0.1\sigma= 0.1 and 0.2 with δ=0\delta= 0. Mem-4 and Noise-0.2 also use lower monthly base rates. The configurations are listed in Table 5 in Appendix § 10. Terms are defined at first use and collected in the glossary of Appendix § 9.

MemberTransition modelMemory δ\deltaNoise σ\sigmaBase rateChecks
Baselinear, deterministic000.038pass, re-anchored
Mem-0.5action-rotation0.500.038pass, one clause amended
Mem-4action-rotation400.026pass, re-anchored
Noise-0.1linear, stochastic00.10.038pass
Noise-0.2linear, stochastic00.20.032pass, re-anchored

Table 5. Members and release checks.

Dynamics and outcomes

Each client carries a hidden state zt∈R6z_t \in\mathbb{R}^{6} drawn from a standard Gaussian population at origination and evolving once per month. An action at=(ct,dt)a_t = (c_t, d_t) pairs a contact channel ctc_t, one of none, short message service (SMS), WhatsApp, or phone, with a settlement offer dtd_t. The channel enters the state transition through contact intensities w(c)=0,0.4,0.4,w(c) = 0, 0.4, 0.4, and 1 respectively. SMS and WhatsApp therefore move the state equally, at equal cost, and differ only in their effect on repayment. With these intensities, the state follows a linear-Gaussian state-space model [20] with an action-dependent rotation, given by:

z~t+1=AQw(ct)z~t+bw(ct)+ηt,ηt∼N(0,σ2I),(1)\tilde{z}_{t+1} = A Q_{w(c_t)} \tilde{z}_t + b w(c_t) + \eta_t, \qquad\eta_t \sim\mathcal{N}(0,\sigma^2 I), \tag*{(1)}

where tt indexes months, zt∈R6z_t \in\mathbb{R}^{6} is a client’s hidden state and zt+1z_{t+1} its next-month state. The contact channel ctc_t has intensity w(ct)w(c_t) as specified above. The matrix A∈R6×6A \in\mathbb{R}^{6 \times6} is a contraction and b∈R6b \in\mathbb{R}^{6} a fixed input direction. The rotation Qw=exp⁡(wδS)Q_w = \exp(w \delta S) uses channel intensity ww, memory parameter δ\delta and a fixed skew-symmetric matrix S∈R6×6S \in\mathbb{R}^{6 \times6}, with exp⁡\exp denoting the matrix exponential. The process innovation ηt\eta_t follows a Gaussian distribution, denoted N\mathcal{N}, with zero mean and covariance σ2I\sigma^{2}I. Here σ\sigma is the process-noise standard deviation per component and II the 6×66 \times6 identity matrix. The memory parameter δ\delta scales the rotation in this transition. At δ=0\delta= 0, QwQ_w is the identity and the action adds the input bw(c)b w(c), so the contributions of past actions add up independently and their order matters only through how recently each was taken. At δ>0\delta> 0 each contact also rotates the current state, which carries the inputs of all earlier contacts, so each contact’s effect depends on the state and on every earlier contact. This interaction is what δ\delta controls.

The process-noise scale σ\sigma, the second parameter, controls the monthly innovations added to this evolution. When σ=0\sigma= 0 the state is a deterministic function of the client’s initial draw and the action history. At σ>0\sigma> 0 the state becomes random, but each innovation is shrunk by AA in every later month, so the accumulated variance converges to a finite level, at most σ2/(1−∥A∥22)\sigma^{2}/(1-\lVert A\rVert_{2}^{2}) per component of the hidden state, where ∥A∥2\lVert A\rVert_{2} is the spectral norm of the transition matrix AA in Eq. (1). The solver observes this evolving state through a two-dimensional linear reading each month. The corresponding measurement equation is written as:

ot=Czt+εt,εt∼N(0,0.62I2),(2)o_t = C z_t + \varepsilon_t,\qquad\varepsilon_t \sim\mathcal{N}(0,0.6^{2}I_2), \tag*{(2)}

where ot∈R2o_t \in\mathbb{R}^{2} is the reading at month tt, ztz_t is the hidden state of Eq. (1), and C∈R2×6C \in\mathbb{R}^{2 \times6} is the fixed loading matrix. The measurement noise εt\varepsilon_t is zero-mean Gaussian with standard deviation 0.6 in each component, and I2I_2 is the 2×22 \times2 identity matrix. Besides oto_t, the solver observes the public account state xtx_t (delinquency ageing, balance, contact fatigue, demographics), and the logged records also carry realized repayments. Repayment depends on the public account state as well as the hidden state and borrower traits. Resilience, payment friction and contactability correspond to the three borrower factors of JUBILEE [31], and regional economic stress to its regional shock. DECODE combines these mechanisms with the latent-state dynamics and observation model defined above.

Conditional on these model inputs, repayment is a monthly Bernoulli draw, rt∼Bernoulli⁡(expit⁡(yt))r_t \sim\operatorname{Bernoulli}(\operatorname{expit}(y_t)), where rtr_t equals one for a repayment and zero otherwise, yty_t is the logit of the repayment probability, and expit is the logistic sigmoid. The corresponding logit is additive in the grouped terms, as in logistic regression [5], and is given by:

yt=β0+γct+g(xt)+hct(dt)−ϕct(xt)ft+κuctTzt+θcTτi+ξt,(3)y_t = \beta_0 + \gamma_{c_t} + g(x_t) + h_{c_t}(d_t) - \phi_{c_t}(x_t) f_t + \kappa u_{c_t}^{\mathsf{T}} z_t + \theta_c^{\mathsf{T}}\tau_i + \xi_t, \tag*{(3)}

where yty_t is the repayment logit for borrower ii at month tt, ctc_t is the contact channel and dtd_t the settlement discount as a fraction of balance. The subscript cc indexes channels. The public account state xtx_t is described after Eq. (2), and ztz_t is the hidden state in Eq. (1). The term β0\beta_0 is the logit of the member’s base rate, γc\gamma_c a channel offset, gg collects the public-state effects, hch_c is a saturating response to the discount dtd_t, and ftf_t is the contact-fatigue component of xtx_t, with a sensitivity ϕc(xt)\phi_c(x_t) that depends on the borrower’s observed income. Both hch_c and ϕc\phi_c are the same for every contact channel and vanish on no-contact months.

The remaining terms describe latent effects and outcome noise. The term κuctTzt\kappa u_{c_t}^{\mathsf{T}}z_t is the hidden score of the acting channel ctc_t, with coupling vectors ucu_c and scale κ\kappa. The term τi\tau_i collects borrower ii’s resilience, payment friction and contactability, together with regional economic stress, with fixed weights θc\theta_c. The superscript T\mathsf{T} denotes transpose, so the products with ucu_c and θc\theta_c are inner products. Finally, ξt∼N(0,0.22)\xi_t \sim\mathcal{N}(0,0.2^2) is Gaussian logit noise with zero mean and standard deviation 0.2, drawn per loan-month. All additive terms are on the log-odds scale. These repayment events also update the public account state. In the generator, a recovery is a payment event and can leave the loan open. It reduces the balance and resets delinquency to zero, returning the loan to performing status. The loan closes as cured only if the remaining balance falls below the larger of 50 currency units and 5% of the original principal. Charge-off is a competing terminal event that removes the loan without a recovery.

Besides their effect on repayment, two hidden traits affect how the historical records are generated. Payment friction steers the logging policy toward phone contact and away from no contact, and contactability steers it the other way. Because they influence both action assignment and repayment, they confound treatment and outcome. Graded solvers do not observe the traits or ξt\xi_t directly. The oracle used for grading has access to the hidden state and the equations.

The fixed matrices and vectors in these equations are shared by every client of a world. One random draw of them is called a “geometry”. It comprises AA and bb in the transition (Eq. (1)), the loading CC of the reading (Eq. 2), the coupling vectors ucu_c of the hidden score (Eq. 3), and, on members with δ>0\delta> 0, the rotation generator SS. The released members share one geometry. Robustness sweeps examine four further draws of these same components.

Protocol

Training and evaluation worlds share each member’s dynamics but use independent seeds (Figure 2). The training logs supply the data used to construct graded solvers, while reference policies use the known world equations. With probability ε=0.03\varepsilon= 0.03, logging substitutes a uniformly drawn contact channel and discount. The no-contact action is never randomized. Exploration excludes liquidation ages above 240 days, where liquidation age counts days in the state of at least 90 days past due. Decisions use a grid pairing four channels with 21 discount levels from zero to half the balance, giving (4×21)6(4 \times21)^6 six-month plans.

World generation and policy-value evaluation

Figure 2. World generation and policy-value evaluation.

After the logged history, solvers take over at month six and re-decide monthly. The harness supplies the earlier readings, channels, offers and contact fatigue, together with the current account state. CE-Seq, the certainty-equivalent (CE) sequential solver of Section 2.5, additionally loads earlier account states directly from the evaluation database, revealing past repayments. Cohorts are selected in loan-identifier order from loans open at the decision point, with the same fixed cohort used for every graded solver. Counterfactual trajectories use common random numbers across the compared policies. Under process noise, innovation paths are independent across loans and shared across references.

Policy value is expected net cash up to the first recovery, with discounts counted as lost and contact costs included. Monthly cash and costs are weighted by the probability of no earlier recovery or charge-off. The grader evaluates (3) at ξt=0\xi_t = 0 and integrates repayment probabilities without drawing payments. During grading, solvers therefore receive readings and deterministic account updates, without new realized payment outcomes. This objective differs from the generator’s complete cash flow, which can retain a loan after payment.

Reference ladder

A disclosure replaces one input of a graded solver with a specified reference quantity. The resulting copy is labeled a reference solver and does not compete with the graded solvers. Its change in value depends on how the fitted system uses that input and can be negative. It differs from the decision-theoretic value of free information for an optimal decision maker who can ignore it ([16]). The policy-value references in tbl:1 use the world’s response equations and public account state under the declared grading objective and action grid. The exact optimum is the fully observed Markov decision process (MDP) upper bound [14].

Without process noise, the state path is fixed by the initial state and channel sequence. Discounts do not alter that path, allowing the search to enumerate 464^6 channel sequences and optimize their discount grids separately. The reference filter instead forms a Gaussian reading-model belief by a Kalman recursion (Kalman

ReferenceInformationCalculationInterpretation
Exact optimum (σ=0\sigma= 0)True hidden state and borrower traitsEnumerate channel sequences and optimize discounts by backward dynamic programmingMDP upper bound for the grading objective
One-month optimum (σ=0\sigma= 0)True hidden state and borrower traitsOptimize the current month and re-decide monthlyReference for myopic planning
Filter plannerGaussian belief from readings, with true borrower traitsIntegrate monthly repayment probabilities and chain their marginalsValue of a specified policy, without POMDP optimality
True-state re-planning (σ>0\sigma> 0)Current true hidden state and borrower traitsMonthly re-planning, evaluated over shared innovation pathsNormalizer, not an upper bound

caption: Policy-value references.

:::

1960), using action-dependent transitions on the memory members. It treats logged actions as fixed inputs and ignores repayment outcomes. Since the logging policy reads delinquency, actions can carry outcome information, and this belief differs from the posterior given the full public history. The filter planner masks future readings and neglects the effect of actions on future information [2]. Under process noise, future innovation variance enters both the belief-based and true-state references. The search integrates each month’s probability over the predictive distribution and chains its marginals, omitting cross-month dependence. Scores are percentages of each member’s upper reference, the exact optimum without process noise and true-state monthly re-planning under noise. Under noise this reference normalizes scores and is not an upper bound. At the path count in Figure 2, each reference’s estimated Monte Carlo standard error is below 0.5% of its normalizer. The standard errors are themselves estimated and quantify Monte Carlo precision. They do not certify policy optimality.

Predictive probes measure how past readings and actions inform the repayment response (Table 2). Each score is the out-of-fold R2R^2, the coefficient of determination over three folds, of a gradient-boosted regressor [7] fitted to the true repayment logit yty_t. Rows are phone-contact months among the exploration rows with a complete six-month window, fixing the acting channel within randomized assignment. Account fields enter at their start-of-month values, before the repayment outcome.

ProbeInformation set
SnapshotCurrent month’s public state, reading, and offered discount
HistorySnapshot set plus the last six months of readings and actions (recorded by channel)
True stateSnapshot set plus the hidden state ztz_t
ContrastWhat it measures
Separation, history − snapshotIncrease in fitted predictive R2R^2 from the six-month window
Headroom, true state − historyIncrease in fitted predictive R2R^2 from adding the true state

caption: Information probes and contrasts.

:::

A separate information reference uses the residual hidden-score variance for a predictor that knows all other logit terms. It is written as:

Rref2=1−Var⁡(κ ucT(zt−μt))Var⁡(yt).(4)R^2_{\mathrm{ref}} = 1 - \frac{\operatorname{Var}\left(\kappa\,u_c^\mathsf{T}(z_t-\mu_t)\right)}{\operatorname{Var}(y_t)}. \tag*{(4)}

where Rref2R^2_{\mathrm{ref}} is the information reference and Var denotes variance across the probes’ measurement rows. The repayment logit yty_t, hidden-score scale κ\kappa, coupling vector ucu_c and transpose notation are defined after Eq. (3). Here cc is the phone channel, ztz_t is the hidden state at month tt, and μt\mu_t is its reading-model posterior mean with logged actions treated as fixed inputs. The numerator is the variance of the residual hidden score, and the denominator is the total logit variance.

The probes use entering account states and logged actions that can reveal past repayments omitted by this reading-model calculation. Conversely, the information reference treats hidden traits and logit noise as known, although the probes cannot observe them. It therefore does not bound or validate the probes, and crossing it is not by itself evidence of an implementation fault.

Checks

The released environments are subject to construction checks on recovery rates and information measures. Recovery-rate bands require the share of loans recovered within one month and within nine months to fall inside pre-set ranges. These checks are applied to the training and evaluation worlds of the four members other than Base and to a 100,000-borrower prefix of Base’s training world. The three members with a deterministic reference additionally have information clauses on the probes and values (Appendix § 10). These are construction criteria, rather than external validation of collection behavior.

Check criteria were fixed before generation. One information clause was amended once before release, and three members were regenerated with an adjusted base rate (Appendix § 10). The release carries each member’s check report and regeneration manifest. The reports record passing checks for the released configurations.

Solvers

The graded solvers in Table 3 combine logged information with a decision rule. GBM-lags and GBM-flat use calibrated gradient-boosted machines (GBMs) ([7]) to predict repayment. Their charge-off hazard, cash formula and fatigue dynamics are supplied by the generator, so only the repayment response is fitted. The disclosures use GBM-lags. All other solvers except LLM-agent, a rule set written by a large language model (LLM), also fit a predictor from the training logs.

SolverInformationModel or policyDecision ruleHorizon
GBM-lagsPublic state, current reading, six months of readings and channelsTwo calibrated GBM classifiersMyopic or sequential, reading and history features frozen during search1, 6
GBM-flatPublic state and current readingAs GBM-lagsAs GBM-lags, without history1, 6
CE-SeqCurrent state and reading, full pre-decision history including account statesRecurrent belief with two calibrated headsSequential, future readings masked6
GBM-cashPublic state and current readingGBM cash predictor, Tweedie lossMyopic1
LLM-agentDelinquency ageing and balanceFixed rules written from censored logsRules applied monthly1

Table 3. Graded solvers.

A myopic solver maximizes current-month expected value, whereas a sequential solver uses the channel enumeration and discount optimization of § 2.3 with its own predicted probabilities, executes the first action and re-plans next month. Only GBM-lags and GBM-flat are graded at both horizons. They form one solver class, differing only in their history inputs, so their horizon contrasts compare each solver with itself.

Horizons in Table 3 are in months. CE denotes certainty-equivalent chaining of predicted monthly probabilities. Training rows, calibration and decision rules are detailed in Appendix § 11.

The recurrent configurations in § 3.2 are predictive memory encoders, evaluated separately from these policy-value comparisons.

Results

Closed-loop contrasts are reported with 95% paired bootstrap intervals over loans, with 10,000 resamples that draw the same loans for every solver. These are percentile intervals, with no correction for multiple comparisons. Each fitted solver uses one training world and one fit, so the closed-loop intervals do not include variation from retraining. Contrasts between graded solvers use the fixed cohort optimum as the normalizer. Contrasts of the declared intervention sequence and of the two fitted refinements of GBM-lags (§ 3.1) recompute it within each resample (Appendices § 15 and § 12). Monte Carlo references carry per-reference standard errors (§ 2.3). A point is one percentage point of the member’s upper reference.

For the predictive measurements, probe scores (Table 2) are out-of-fold means over three cross-validation folds. The recurrent configurations of § 3.2 report the mean and population standard deviation over five training seeds on Mem-4 and two on the other members. A point of R2R^2 is 0.01.

Base

On Base, within the GBM class, the only class graded at both horizons, extending the planning horizon from one month to six raises the score of GBM-lags by 16.62 points ([15.31, 18.02]) and that of GBM-flat by 15.06 ([13.54, 16.66]) of the exact optimum. At a fixed horizon, the largest point estimate is 3.75 ([2.75, 4.78]), between GBM-lags and GBM-flat. Across fitted solver classes, the largest contrast is 2.12 ([0.82, 3.34]), by which GBM-lags exceeds CE-Seq despite the latter’s access to earlier account states (Figure 3, Appendix § 15).

Figure 3. Closed-loop performance on Base.

The connected pairs in Figure 3(a) are the within-class horizon contrasts, unconnected markers are solvers graded at one horizon, and the dotted rule marks the one-month optimum. LLM-agent, graded once at the one-month horizon, reaches 64.74% of the exact optimum, 0.85 points ([0.05, 1.73]) below the closest fitted solver, GBM-flat at one month (Table 10).

ContrastPoints95% interval
GBM-lags, six months minus one month+16.62[15.31, 18.02]
GBM-flat, six months minus one month+15.06[13.54, 16.66]
GBM-lags minus GBM-flat, six months+3.75[2.75, 4.78]
GBM-lags minus CE-Seq, six months+2.12[0.82, 3.34]
CE-Seq minus GBM-flat, six months+1.63[0.17, 3.21]
GBM-lags minus GBM-flat, one month+2.19[1.32, 3.15]
GBM-lags minus GBM-cash, one month+1.46[0.61, 2.44]
GBM-flat minus LLM-agent, one month+0.85[0.05, 1.73]
exact optimum minus GBM-lags, six months+15.60[14.36, 17.01]
exact optimum minus one-month optimum+23.87[22.43, 25.40]

Table 10. Paired contrasts on Base.

Full disclosure, whose endpoint is the filter planner of Table 1, improves the score of GBM-lags by 9.35 points ([8.68, 10.02]) of the 15.60 ([14.36, 17.01]) separating it from the exact optimum. The declared intervention sequence (Figure 3b, Appendix § 12) replaces the fatigue interpolation, the state features, their propagation during search and the fitted response. The response replacement uses each loan’s true hidden traits. Replacing the history features by the reference filter’s features changes the score by −0.68 points, with an interval that includes zero. The largest step, +15.61 points ([14.14, 17.16]), replaces the fitted response after the preceding changes. It exceeds the full gain because the exact-fatigue, filter-feature and rolled-belief steps together lower the score by 6.26 points. The full gain and two steps are shown with 95% intervals, with step values conditional on the sequence and fitted response.

A linear model fitted separately for each contact channel predicts the hidden score from the current reading and the last six months of readings and actions, with a row-weighted mean out-of-sample R2R^2 of 0.56 (0.06 when a single model is fitted across the three channels). This oracle-side diagnostic is trained on the hidden score, whereas the solvers learn from repayment labels. The structured estimator it motivated scores −1.73 points ([−2.60, −0.86]) against GBM-lags, while an 11-lag window scores +0.63 ([−0.12, +1.47]). These two fitted refinements use the same logs and cohort as GBM-lags, and neither establishes an improvement (Appendix § 12).

These comparisons leave the source of the remaining gap unresolved. Contact exploration randomizes actions with ϵ=0.03\epsilon= 0.03 only in eligible states and is held fixed across the comparisons, so its contribution is not identified.

For a fixed plan, certainty-equivalent chaining inflates the filter-based planner’s self-reported value, the value it computes for that plan, by approximately 11% relative to the value of the same plan integrated jointly under the filter’s belief (95% loan-bootstrap interval [10.8, 11.6]%). On Base, this estimate uses 4,000 loans and 512 draws per loan and is conditional on one geometry and the specified filter belief, with posterior draws held fixed when resampling loans. Graded and self-reported values are reported separately throughout.

Memory

The memory sweep compares the snapshot, history and true-state probe scores across values of δ\delta (Figure 4a). It uses pilot worlds, smaller unreleased worlds of 100,000 borrowers over 12 months generated on the members’ geometry. The released members correspond to δ∈{0,0.5,4}\delta\in\{0, 0.5, 4\}. On this geometry, the history-minus-snapshot separation is +0.127 at δ=0\delta= 0, +0.175 at δ=0.5\delta= 0.5 and +0.114 at δ=4\delta= 4. At δ=4\delta= 4, headroom is +0.430 and the snapshot score is 0.338. These fitted contrasts differ from the margin to the information reference of Eq. (4) reported below.

Figure 4. Memory measurements.

The geometry sweep repeats these measurements across five independent draws of the world’s matrices (Figure 4b). The first ordering clause requires separation at δ=0.5\delta= 0.5 to exceed that at δ=0\delta= 0. The second requires greater headroom and no higher snapshot score at δ=4\delta= 4 than at δ=0\delta= 0. Each clause holds in four of five draws. The snapshot score falls from δ=0\delta= 0 to 4 in five of five, and the within-world separation is positive in all fifteen draw–δ\delta cells. Worlds across δ\delta are not matched populations. Each history-minus-snapshot contrast uses the same rows within its own world. Snapshot scores also vary across seeds and geometry draws. These runs do not isolate an effect of δ\delta on that score.

To test how learned memories use the history measured by the probes, each recurrent configuration uses a gated recurrent unit (GRU) or long short-term memory (LSTM) network to compress a loan’s full history of readings and actions into a vector. A small network combines this vector with the current month’s snapshot to predict the probes’ target. Its out-of-fold R2R^2 minus the history probe’s score defines its memory value above that probe.

On Mem-4, the information reference lies 0.298 in R2R^2 above the history probe. The recurrent configurations of Table 7 have gains of approximately zero to 38% of that margin at the stated training budgets, while the fitted reference filter has a gain of 69% (Figure 5a, Appendix § 13). These percentages are a normalization against the information reference, not fractions of attainable predictive information.

Figure 5. Memory and re-observation.

ConfigurationΔR2\Delta R^2 vs historyshareat epoch cap
recurrent (GRU), small−0.002±0.003-0.002 \pm0.003≈0\approx0no stopping log
recurrent (GRU), small, long budget+0.083±0.002+0.083 \pm0.00227.9%yes
recurrent (LSTM), large+0.071±0.006+0.071 \pm0.00623.7%yes
recurrent (GRU), large+0.112±0.002+0.112 \pm0.00237.7%yes
reference filter (fitted)+0.205+0.20568.8%
information reference (no fit)+0.298+0.298100%

Table 7. Memory value on Mem-4.

On Mem-4, the long-budget configuration and the two large configurations reached the epoch cap in every fold. Training was still improving in most folds of the two large configurations and in about half the folds of the long-budget configuration. The reported gains therefore describe the stated training budgets. Because capacity also varies across configurations, the comparison does not isolate an architecture effect. At δ≤0.5\delta\le0.5 the recurrent scores were close. The limited replication does not establish equivalence between architectures.

Process noise

The re-observation gain is the value of re-planning every month on the updated filter belief minus the value of executing the plan chosen at the decision point. On Base it is approximately zero for the reference filter. It is positive on both noisy members, with the larger gain at σ=0.2\sigma= 0.2 (Figure 5b). The Base point uses a closed-form comparison, while the noisy points use Monte Carlo evaluation. Measured against each member’s upper reference, re-planning with the filter belief gains 0.57 points at σ=0.1\sigma= 0.1 and 3.66 points at σ=0.2\sigma= 0.2. The corresponding gains in expected recovered value per loan, in the world’s currency units, are +9.4 ([4.3, 14.8]) and +87.1 ([63.0, 116.1]). These 95% intervals resample loans together with their 24 paths and are conditional on each member’s evaluation world and geometry. The two noisy members also differ in base rate, so this comparison does not isolate the effect of σ\sigma alone.

Predictive comparison with real data

The predictive measurements can also be extended to observed loan outcomes by testing whether history improves prediction. The real-data comparison uses 20,677 delinquency spells, or episodes of delinquency, constructed from Fannie Mae’s Single-Family Loan Performance Data [6], freely available upon registration. Here, cure means a return to current status within six months. Adding the pre-entry block (twelve months of delinquency states, prior-spell count, months since the previous cure) raises the area under the receiver operating characteristic curve (AUC) for cure prediction by +0.041 ([0.031, 0.050]). The interval resamples loans within folds and is conditional on the fitted models and partition. It uses the same mean-over-folds statistic as the point estimate, with 10,000 resamples. Sensitivity to the partition is reported in Appendix § 16.

The corresponding AUC gains in the synthetic worlds span +0.010 to +0.013, with all three intervals excluding zero (Appendix § 16). The real portfolio’s estimate lies above every synthetic interval. The two sides differ in observations and targets. The real side is a mortgage portfolio observed without contact actions. Its label is cure within six months, whereas the synthetic label is recovery within nine months. Its history block also differs from the synthetic readings and actions. The comparison concerns prediction. It does not validate the effects of collection policies.

Discussion

DECODE tests how specified changes to information and planning affect solver performance under known dynamics. The results support interpretations tied to the inputs, decision rules and grading objective used in each comparison.

On Base, extending lookahead within the same six-month evaluation window improves value more than the tested solver differences at a fixed horizon, for the two GBMs tested at both horizons. Costs, fatigue and state transitions can explain why current choices affect later returns, although their individual contributions were not isolated. Solver contrasts also change information inputs, and Base was selected for a gap between the exact and one-month optima on a training-world prefix. The declared intervention sequence recovers most of the remaining gap, with effects conditional on the fitted response and preceding steps. Some steps lower value, and the filter-feature interval includes zero. Since the filter omits pre-decision repayment outcomes, its small step cannot establish that inference is generally unimportant. The final replacement supplies the true response and hidden traits, combining information with model knowledge. Separating an information penalty from a learning gap would require an optimum under the solver’s information constraints, which is not computed here [28]. Neither fitted refinement establishes an improvement. The oracle-side readout predicts a hidden score, a different task from learning repayment probabilities. Exploration, specification and extrapolation remain possible explanations of the residual gap. The logging design is unchanged, so these experiments do not isolate exploration as its cause.

History can help prediction even at zero memory parameter because readings remain partial and noisy and action recency still affects the state. The parameter δ\delta changes how effects depend on state and earlier actions. The history length supplied to the solver, its representation and planning lookahead are separate design choices. A peak in fitted history-minus-snapshot separation does not establish an optimum of memory or data complexity. It may reflect a balance between historical information and the predictor’s ability to use it, a mechanism these experiments do not isolate. Separation is positive across all measured geometry cells, but the ordering clauses are not universal. Unmatched cross-parameter worlds also prevent attributing snapshot changes to δ\delta alone. The filter uses known dynamic and observation equations, while recurrent encoders learn a representation and several were still improving at the training cap. Varying capacities and budgets prevent a general architecture ranking, including for the single attention configuration (Appendix F). The information reference does not bound the probes, so shares of its margin are not fractions of attainable information. History also improves prediction in the real mortgage portfolio, with an estimate above the synthetic intervals. Populations, covariates and labels differ, including six-month cure against nine-month recovery. Agreement in direction neither calibrates synthetic effect sizes nor validates the effects of collection policies. Predictive gains alone do not establish better decisions.

The observed self-report inflation comes from chaining monthly mean repayment probabilities, which omits their dependence under state uncertainty. It is measured against joint integration of the same plan under the same filter belief and concerns that valuation rule, plan and belief. Re-planning on updated filter beliefs improves value on both noisy members. Gaussian innovations can make an initial plan less appropriate, but the contractive dynamics keep this adaptation within a stable process. Base’s near-zero gain is empirical, not an identity at zero process noise, since readings remain noisy. Larger benefits under stronger or abrupt shocks remain a hypothesis. Adaptation depends on changes being observed, changing the preferred action and being handled by the model. Regime changes and altered repayment mechanisms were not tested, and different base rates between noisy members prevent attributing their difference in gains to δ\delta alone.

These findings support diagnoses conditional on the evaluated configurations. Graded solvers are evaluated on Base only. The memory members support predictive measurements and the noisy members a comparison between reference policies. On Base, fitted policy comparisons use one geometry, one training world and one fit per solver. Intervals reflect the 4,000-loan cohort, without retraining variability. Monte Carlo errors quantify uncertainty in reference estimates, not policy optimality. Under noise, true-state re-planning normalizes scores without bounding attainable value. The policy values concern the first-recovery grading objective. GBM-lags and GBM-flat plan with the generator’s charge-off hazard, cash formula and fatigue dynamics, so their horizon contrast measures planning with a partly known model. No offline reinforcement learning method that learns a value function over the planning horizon was graded. Broader reliability claims require replication across training samples and worlds.

Related work

Several benchmark suites summarize solver performance through policy returns. Offline reinforcement learning Levine et al. [21] suites such as D4RL Fu et al. [9] standardize datasets and report policy returns from logs that often include more exploratory data than deployed policies produce Qin et al. [32]. Off-policy evaluation (OPE) benchmarks, COBS Voloshin et al. [38] and DOPE Fu et al. [10], compare value estimators against returns from running the policy.

Studies in model-based reinforcement learning also compare methods by the return of the policy trained against a learned model Janner et al. [18]; Yu et al. [39]. The value-equivalence line Grimm et al. [11] judges models by the value-based planning they support. Learned recurrent state-space models are graded by downstream return Hafner et al. [13], with no reference belief to compare against. DECODE separately evaluates changes to model inputs and planning. Its memory experiments examine the memory component of the world-model decomposition Ha & Schmidhuber [12]. They compare sequence encoders under a common readout design.

The relative performance of memory architectures depends on the task’s memory length Ni et al. [26], and recurrent model-free baselines perform on par with specialized methods when their architecture and hyperparameters are chosen carefully Ni et al. [25]. For DECODE, the comparisons are conditional on the memory parameter and the information used by each reference.

Planning under the released dynamics is a partially observable Markov decision problem Kaelbling et al. [19]. Work on value overestimation identifies biases from the maximization step Thrun & Schwartz [35]; van Hasselt et al. [36], selection among noisy estimates Smith & Winkler [34], and selecting a policy on the sample that estimated its model Mannor et al. [22]. The inflation measured here comes from chaining marginal repayment probabilities instead of integrating their joint effect on survival and cash flows. It is measured against the same plan integrated jointly under the filter’s belief.

For systems fitted on historical data, interpretation also depends on how the logs were collected. Identification rests on the randomization the logging design provides Bottou et al. [3]. Sensitivity analyses under unobserved confounding give bounds that can widen with the horizon Namkoong et al. [24]. Reinforcement learning has been deployed for collections Abe et al. [1], evaluated by importance weighting on logged outcomes with no reference optimum.

JUBILEE Prado et al. [31] evaluates OPE under hidden assignment dependence, using an initial policy intervention followed by a common behavior policy. DECODE examines controlled information and planning changes in sequential decision systems, including the response to specified information disclosures. Its disclosure contrasts differ from the decision-theoretic value of information Howard [16].

Diagnostic suites use tractable environments to study specific capabilities. bsuite Osband et al. [27] includes experiments scored by regret normalized between random and optimal policies. MDP Playground Rajan et al. [33] varies hardness dimensions over toy environments with known optima, uses the optimum for reward normalization, and summarizes learning with area under the training curve. POPGym Morad et al. [23] isolates memory models on partially observable tasks and rescales each environment so that the worst and best observed trials map to 0 and 1. Memory Gym Pleines et al. [30] reports success rates and executed-command counts. Memory Maze Pasukonis et al. [29] reports episode scores against a privileged-information oracle and probes agents’ representations against the true maze layout. DECODE compares its predictive scores with the information reference, conditional on the reading model, and policy values with a specified filter-based planner.

Within this diagnostic literature, a study of deep Q-learning uses exact tabular solutions to disentangle estimator-internal error sources, characterizing one algorithm class on a fixed domain set Fu et al. [8]. Its oracle-stopping headroom concerns stopping during learning, while the self-report inflation measured here concerns valuation of a fixed plan.

A complementary approach generates environments with certified optima along a difficulty ladder Ibrahim et al. [17], reporting normalized optimality gaps and discounted regret in a fully observed setting. Querando et al. 2026 extend this approach to partially observed tasks with certified optima on a sufficient information state. Their linear-quadratic-Gaussian (LQG) reference separates an analytic information penalty from the learning gap under the stated model assumptions.

DECODE’s contribution concerns controlled interventions in solvers whose repayment responses are fitted on partially observed collection logs. These experiments vary selected information and planning inputs under the same grading objective. No optimum under the solvers’ information constraints is computed, so the contrasts measure the effects of specified interventions rather than separating information loss from learning error.

Conclusion

DECODE supports diagnostic experiments on partially observed sequential decision systems through controlled changes to information and planning inputs. On Base, a member selected for a large gap between the exact and one-month optima, extending the horizon within a fitted solver raises its score by 15 to 17 points of the exact optimum. At a fixed horizon, contrasts between the tested solvers have point estimates below 4 points. The declared intervention sequence recovers nine of the 15.6 points separating GBM-lags from the exact optimum. The effects of these steps are conditional on their order, the response model and the information supplied.

A separate fixed-plan comparison checks the planner’s internal valuation. On Base, chaining marginal repayment probabilities inflates the filter-based planner’s self-report relative to joint integration of the same plan under the same belief.

The diagnoses drawn from these experiments concern the specified input substitutions, information sets and training budgets in one domain. They do not assign a unique cause to the total performance gap. The generator and reference policies allow these conditions to be varied and their consequences measured.

Acknowledgments

The authors review all AI-assisted contributions and take responsibility for the manuscript, code and reported results.

LLM statement

Claude Fable 5 (Anthropic) assisted with software implementation, literature retrieval, analysis checks, manuscript drafting and revision.

Loan records, rewards and policy-value estimates are produced by the simulator and evaluation code using the specified configurations and random seeds. Reported statistics are computed by code from saved inputs and outputs.

Glossary

The terms defined in the body of the paper are collected here for reference, in alphabetical order (Table 4).

TermMeaning
Base, Mem-0.5, Mem-4, Noise-0.1, Noise-0.2the five released environments, with parameter settings in Table 5
beliefthe probability distribution of the hidden state given everything a solver has observed
certainty-equivalent (CE)chaining monthly repayment probabilities without their cross-month dependence
closed-loop gradinggrading in which each solver re-decides every month on the readings its own actions generate
decision pointmonth six of an evaluation loan, when a solver takes over after six months of public history, with six months left to decide
disclosurethe replacement of one input of a graded solver by a specified reference quantity
exact optimumthe value of the best plan computed with the true hidden state and borrower traits at the decision point, available on members without process noise
exploration rowslogged loan-months on which the logging policy drew a contact channel and a discount at random
filter beliefthe reference filter’s Gaussian approximation, ignoring repayment outcomes
gapthe difference between the exact optimum and a solver’s value, in points
geometryone random draw of the fixed matrices and vectors in a world’s equations. The five released members share one geometry
headroomthe true-state probe score minus the history probe score
hidden scorethe channel-specific hidden-state term κu⊤zt\kappa u^\top z_t, with terms defined after Eq. (3)
hidden traitsthree unobserved borrower factors and a regional economic-stress term entering repayment
information clausesfloors and ceilings, fixed before generation, on probe scores, and on Base also on values, that a member with an exact reference must pass
information referencethe analytic R2R^2 reference conditional on the specified reading model and its information set
logging policythe fixed rule that generated the training logs
memberone released setting of the memory parameter, the process-noise scale and the monthly base rate
memory parameter δ\deltathe scale of the rotation each contact applies to the hidden state, which makes contacts interact with the state
myopic solvera solver that chooses each month the action with the highest expected value for that month alone
one-month optimumthe best policy that plans one month at a time with the true hidden state
oraclethe grader’s side of the world, which holds the hidden state and the equations
pilot worlda smaller unreleased world, of 100,000 borrowers over 12 months, used for sweeps and construction checks
pointone percentage point of a member’s upper reference
probethe out-of-fold R2R^2 of a gradient-boosted model predicting the true repayment logit from one information set, snapshot, history, or true state (Table 2)
process-noise scale σ\sigmathe standard deviation of the monthly innovation in the hidden state
readingthe noisy two-dimensional linear observation of the hidden state
re-anchoringregenerating a member with an adjusted base rate after its aggregate recovery fell outside the recovery-rate band
recovery-rate bandsranges into which the share of loans recovered within one month and within nine months must fall
referencea value computed from the world’s equations, against which a solver’s result is read
reference filterthe Gaussian reading-model recursion with fixed actions, ignoring repayment outcomes
reference ladderthe analytic and Monte Carlo references available under each member’s dynamics
reference solvera labeled copy of a graded solver that receives a disclosure and never competes
re-observation gainthe value of re-planning every month on the updated belief minus the value of executing the plan chosen at the decision point
rotationthe orthogonal matrix Qw=exp⁡(ωδS)Q_w=\exp(\omega\delta S) that turns the hidden state, with terms defined after Eq. (1)
self-reported valuethe value a planner computes for its own plan
separationthe history probe score minus the snapshot probe score
sequential solvera solver that plans the whole remaining horizon every month, executes the first action, and re-plans
solvera decision system graded by the instrument, built from the training logs
solver classa solver’s architecture and estimator, which can be graded at more than one horizon
upper referencethe normalizer of each member, the exact optimum without process noise and the value of monthly re-planning with the true state under noise

Table 4. Glossary.

Members

The five members of Section 2 are listed in Table 5 with their transition model, their parameters, their monthly base rate, and the outcome of their construction checks. All reports record passing recovery-rate bands, and the three members with σ=0\sigma= 0 also pass their information clauses. These clauses set floors on the separation or the headroom of the probes of Table 2, so that the history or the hidden state carries information the current month lacks, and, on the memory members, ceilings on what the current month alone predicts, so that the reading does not leak the hidden state. On Base the clauses are also stated on values, requiring that planning over the horizon matter, that the best plan vary across loans, and that a solver without history stay well below the exact optimum. The first requires the exact optimum to exceed the one-month optimum by 15 to 40 points, and Base measured 27.1 on a 100,000-borrower prefix of its training world. The history-free clause caps a solver without history, planning over six months, at 85 points. This solver was fit and graded on the same training-world prefix, where Base measured 59.2 before any solver of Section 3 was graded. The clauses on plan variation require that no single channel sequence be optimal for more than 60% of loans, where Base measured 25%, and that the best of the twelve most frequent optimal channel sequences, applied to every loan with a 20% discount on contact months, stay at least 2 points below the exact optimum, where Base measured 19.9 points. A coverage clause requires at least 50 logged rows in each of 27 cells formed by income terciles, three contact-fatigue levels and three channel classes, and the smallest Base cell held 7,629 rows. The two memory members also pass a per-loan consistency check, under which the plan optimum is never below the one-month optimum, and on Base the same holds on every graded loan. The thresholds of every clause are recorded with the release.

The release carries each member’s machine-readable check report and a manifest recording the configuration hash, the seeds, and the pinned generation batch size that regeneration requires. Check criteria were fixed before generation, with one exception. An information clause on Mem-0.5 failed and was replaced once before release by a clause targeting the same mechanism, and the amendment is in the release records. Three members were re-anchored, that is, regenerated with an adjusted base rate after their aggregate recovery fell outside the recovery-rate band, two on pilot or check worlds before definitive generation and one after its definitive world failed the check. The release contains each member’s final configuration and check report and, for Mem-4, the configuration and check report of the definitive world that failed.

Two further verifications were run at construction time on pilot worlds. With δ=0\delta= 0 the generator reproduces bit for bit the output of its version without the memory parameter, so the parameter changes nothing when it is off, and the grader’s replay of each client’s hidden-state path under the logged actions agrees with the generator to within 10−1210^{-12}.

Solver training details

The information inputs and decision rules of the graded solvers are summarized in Table 3. GBM-lags and GBM-flat each fit two calibrated classifiers using training-world months six onward. The contact classifier uses exploration rows, while the no-contact classifier uses logged no-contact rows. GBM-lags adds a six-month window of readings and channel codes. Both solvers keep their reading features, and GBM-lags its history features, fixed at the decision month during sequential search.

CE-Seq fits a recurrent belief model on the full training panel, summarizing the account’s observed history from its first month. Its contact head uses exploration months and its no-contact head uses logged no-contact months, each with its own isotonic calibrator. Here certainty-equivalent means chaining predicted monthly repayment probabilities as fixed quantities along a plan. This differs from planning with the state fixed at its conditional mean [2]. Future readings are masked during search. Its pre-decision account-state sequence is read directly from the evaluation database, in addition to the history supplied by the harness.

GBM-cash fits expected monthly cash with a Tweedie loss on all training-log rows, using a temporal validation split. It chooses one-month actions. LLM-agent uses state-conditional rules written once after a language model examined a censored copy of the training logs, without access to the evaluation world. The rules were frozen before evaluation and applied monthly.

Declared intervention sequence

The full disclosure of § 3.1 is reached from GBM-lags in four specified steps, each of which changes one input or its numerical evaluation and is graded on the same 4,000 loans with one shared-index bootstrap (Table 6). Values are percentages of the exact optimum, and steps measure changes relative to the preceding row. Their sum equals the full-disclosure gain by identity. The exact-fatigue step replaces interpolation with exact evaluation of the observed fatigue along each candidate path. It is a numerical control, not a disclosure of hidden information. The filter-feature step has an interval that includes zero, and propagating those features lowers value. The exact-response step raises value after these preceding changes, exceeding the total gain. These effects are conditional on the preceding steps and on the fitted response. The remaining 6.25 points between full disclosure and the exact optimum combine differences in information with the planner’s approximations. The filter ignores pre-decision repayment outcomes and its planner chains marginal probabilities. The remaining distance therefore does not measure state-inference error alone. The intervals of the sequence contrasts divide each bootstrap resample’s mean difference by that resample’s mean optimum.

SolverChange from the row aboveValueStep95% interval
GBM-lagsnone84.40
exact fatigueexact per-path fatigue instead of interpolation83.65−0.75-0.75[−1.18,−0.34][-1.18,-0.34]
filter featuresreference-filter features instead of history82.97−0.68-0.68[−1.60,+0.22][-1.60,+0.22]
rolled belieffilter belief propagated instead of frozen78.14−4.83-4.83[−6.29,−3.43][-6.29,-3.43]
exact responsetrue response with hidden traits instead of fitted response93.75+15.61+15.61[14.14,17.16][14.14,17.16]
full disclosureall four changes together+9.35+9.35[8.68,10.02][8.68,10.02]

Table 6. Declared intervention sequence on Base.

Measured directly against GBM-lags, the three intermediate reference solvers of the declared intervention sequence score −0.75-0.75 ([−1.18,−0.34][-1.18,-0.34]), −1.43-1.43 ([−2.37,−0.55][-2.37,-0.55]) and −6.26-6.26 ([−7.77,−4.80][-7.77,-4.80]). They serve as controls in that sequence, rather than additional fitted competitors. Exact fatigue changes the numerical evaluation of a public state, whereas the filter replacements change the state information.

The two fitted refinements of Section 3.1 are a structured estimator and an 11-lag window. The structured estimator replaces the history features with a linear correction in logit space, fitted on top of a history-free GBM. Its specification drew on the oracle-side diagnostic, and its search uses exact per-path fatigue. It scores −1.73-1.73 ([−2.60,−0.86][-2.60,-0.86]) against GBM-lags. Against the exact-fatigue control, its difference is −0.98-0.98 points. The 11-lag window adds public history and has an interval that includes zero at +0.63+0.63 ([−0.12,+1.47][-0.12,+1.47]). These comparisons do not establish an improvement over GBM-lags.

A separate experiment compares response-component substitutions under its own protocol, with the fixed cohort optimum as normalizer and spans not additive to the declared intervention sequence. In that experiment, propagating the exact contact component improves value, whereas rolling filter features into the fitted response lowers it in the declared intervention sequence. The fitted inputs also differ, so this contrast does not isolate order alone. Relative to the exact-fatigue control, substituting the exact contact response with frozen features changes value by +6.7+6.7 points ([+5.7,+7.8][+5.7,+7.8]), propagating its effect across months adds $+3.3 ([+2.3, +4.0]), and substituting the exact response on no-contact months, the organic component, changes value by −0.32 ([−0.63, −0.05]). The no-contact model is fit on observational rows. These contrasts do not isolate the effect of randomizing no contact.

Memory values

The memory scores of Figure 5a on Mem-4 are tabulated in Table 7 as gains above the history probe at the stated training budgets. The recurrent LSTM (Hochreiter & Schmidhuber [15]) and GRU (Cho et al. [4]) variants report means with population standard deviations over five training seeds. The share column divides each gain by the 0.298 margin between the history probe and the information reference, rather than by attainable information. The final column records whether the epoch cap was reached.

On Base’s training world, the history probe reaches R2R^2 of 0.624, a fitted pipeline on the reference filter’s outputs 0.633, and a fitted recurrent memory 0.639. These scores lie below the 0.674 information reference, which does not bound them (§ 2.3). These levels are from the training world and are not comparable to the pilot-world sweep of Figure 4.

Attention

Alongside the recurrent configurations of Table 7, one attention-based sequence model was evaluated with the same readout head, folds, and data. The encoder is a two-layer transformer (Vaswani et al. [37]) of model width 64 with four heads, causal masking, and learned positions. Its epoch cap and patience match the standard recurrent configurations’ at a lower learning rate. In Table 8, the upper block compares training budgets and the lower block reports memory value above the history probe for each member. One recurrent configuration of Table 7 ran a longer budget, so shares credited to it and the values reported here are not comparable at equal optimization effort. Each member cell trains three folds per seed. Dispersion is the population standard deviation over seed means. The two-seed cells include individual values, showing opposite signs on Mem-0.5. Every transformer fold reached the epoch cap, twelve of the fifteen Mem-4 folds were still improving there, and three had peaked before the cap. These measurements are conditional on the stated training budget. A single small configuration at one budget does not establish a general ranking of attention-based and recurrent memories. The per-fold stopping logs are recorded with the measurement.

configurationepoch cappatiencelearning rate
attention (this appendix)3053×10−43 \times10^{-4}
recurrent (Table 7)30510−310^{-3}
recurrent, long budget (Table 7)1001510−310^{-3}
memberseedsmemory valueper-seed
Base2−0.021-0.021−0.022-0.022, −0.020-0.020
Mem-0.520.000−0.008-0.008, +0.008+0.008
Mem-45−0.090±0.005-0.090 \pm0.005

Table 8. Attention model budgets and memory values.

Closed-loop values

Every closed-loop value on Base, including the exact and one-month optima, a random policy, and the single-observation LLM-agent reported in § 3.1, is listed in Table 9. Values are levels expressed as percentages of the exact optimum. Every solver row is also plotted in Figure 3(a), and the random policy provides the comparison floor. Comparisons between rows use the paired contrasts of Table 10 under the bootstrap protocol of § 3.

Solver1-month horizon6-month horizon
GBM-lags67.7884.40
GBM-flat65.5980.65
CE-Seq82.28
GBM-cash66.32
LLM-agent64.74
random policy29.11
1-month optimum76.13
exact optimum100

Table 9. Closed-loop values on Base.

The paired contrasts in Table 10 are in points of the exact optimum, with 95% paired bootstrap intervals over the 4,000 evaluation loans. The 10,000 resamples draw the same loans for every solver and use the fixed cohort optimum as the normalizer.

Predictive gains from pre-entry history

The AUC gain of § 3.4 is reported in Table 11 as the out-of-fold ΔAUC\Delta\mathrm{AUC} from adding pre-entry history. The real portfolio and the released members use the same estimator and a five-fold, loan-grouped evaluation policy. The real-side outcome is cure within six months, while the synthetic label is recovery within nine. The intervals use 10,000 resamples of loans within folds and condition on the fitted models and partition. The final column gives the minimum and maximum over 20 random loan-grouped partitions with refitted models. This descriptive range is not a confidence interval.

DataΔ\DeltaAUC95% intervalPartition range
Fannie Mae spells (real, n=20,677n = 20{,}677)+0.041[0.031, 0.050][0.030, 0.045]
Base+0.013[0.010, 0.016][0.011, 0.014]
Mem-0.5+0.013[0.010, 0.016][0.012, 0.017]
Mem-4+0.010[0.007, 0.013][0.008, 0.013]

Table 11. Predictive gains from pre-entry history.

The real-data source comprises quarterly files from 2000Q1 through 2025Q4. The data license restricts redistribution, so the release records a cohort fingerprint rather than loan identifiers. The identifier list is withheld under the data license. The real-data calculation was reproduced with scikit-learn 1.7.2. Reproduction under the release’s pinned 1.9.0 environment has not been verified.

References

  1. [1]Naoki Abe, Prem Melville, Cezar Pendus, Chandan K. Reddy, David L. Jensen, Vince P. Thomas, James J. Bennett, Gary F. Anderson, Brent R. Cooley, Melissa Kowalczyk, Mark Domick, and Timothy Gardinier. Optimizing debt collections using constrained reinforcement learning. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 75–84, 2010.DOI
  2. [2]Yaakov Bar-Shalom and Edison Tse. Dual effect, certainty equivalence, and separation in stochastic control. IEEE Transactions on Automatic Control, 19(5):494–500, 1974.DOI
  3. [3]Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14:3207–3260, 2013.
  4. [4]Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724–1734, 2014.arxiv.org/abs/1406.1078
  5. [5]David R. Cox. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (Methodological), 20(2):215–232, 1958.
  6. [6]Fannie Mae. Single-family loan performance data. https://capitalmarkets.fanniemae.com/credit-risk-transfer/single-family-credit-risk-transfer/fannie-mae-single-family-loan-performance-data, 2026. Quarterly files 2000Q1–2025Q4, accessed May 2026.
  7. [7]Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5):1189–1232, 2001.DOI
  8. [8]Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep Q-learning algorithms. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2021–2030, 2019.arxiv.org/abs/1902.10250
  9. [9]Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.arxiv.org/abs/2004.07219
  10. [10]Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R. Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Tom Le Paine. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations, 2021.arxiv.org/abs/2103.16596
  11. [11]Christopher Grimm, André Barreto, Satinder Singh, and David Silver. The value equivalence principle for model-based reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pp. 5541–5552, 2020.arxiv.org/abs/2011.03506
  12. [12]David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018.DOI
  13. [13]Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering Atari with discrete world models. In International Conference on Learning Representations, 2021.arxiv.org/abs/2010.02193
  14. [14]Milos Hauskrecht. Value-function approximations for partially observable Markov decision processes. Journal of Artificial Intelligence Research, 13:33–94, 2000. doi: 10.1613/jair.678.DOI
  15. [15]Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
  16. [16]Ronald A. Howard. Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2 (1):22–26, 1966.DOI
  17. [17]Sinan Ibrahim, Grégoire Ouerdane, Hadi Salloum, Henni Ouerdane, Stefan Streif, and Pavel Osinenko. Benchmarking reinforcement learning via stochastic converse optimality: Generating systems with known optimal policies. arXiv preprint arXiv:2603.17631, 2026.arxiv.org/abs/2603.17631
  18. [18]Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, volume 32, 2019.
  19. [19]Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2):99–134, 1998.
  20. [20]Rudolf E. Kalman. A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82(1):35–45, 1960.DOI
  21. [21]Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.arxiv.org/abs/2005.01643
  22. [22]Shie Mannor, Duncan Simester, Peng Sun, and John N. Tsitsiklis. Bias and variance approximation in value function estimates. Management Science, 53(2):308–322, 2007.DOI
  23. [23]Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. POPGym: Benchmarking partially observable reinforcement learning. In International Conference on Learning Representations, 2023.arxiv.org/abs/2303.01859
  24. [24]Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy evaluation for sequential decisions under unobserved confounding. In Advances in Neural Information Processing Systems, volume 33, pp. 18819–18831, 2020.arxiv.org/abs/2003.05623
  25. [25]Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be a strong baseline for many POMDPs. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16691–16723, 2022.arxiv.org/abs/2110.05038
  26. [26]Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Luc Bacon. When do transformers shine in RL? Decoupling memory from credit assignment. In Advances in Neural Information Processing Systems, volume 36, pp. 50429–50452, 2023.
  27. [27]Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020.
  28. [28]Grégoire Ouerdane, Sinan Ibrahim, Stefan Streif, and Pavel Onisenko. Ground-truth-first benchmarking of reinforcement learning via converse optimality: From smooth, fully observed systems to non-smooth or partially observed tasks. arXiv preprint arXiv:2610.30954, 2026.
  29. [29]Jurgis Pasukonis, Timothy Lillicrap, and Danijar Hafner. Evaluating long-term memory in 3D mazes. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.13383
  30. [30]Marco Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss. Memory Gym: Partially observable challenges to memory-based agents. In International Conference on Learning Representations, 2023.
  31. [31]Diego Silva Prado, Tiago Amaral Penteado de Jesus, Bruno Caldas de Souza, and Cristiano Bevitori Maffia De Oliveira. JUBILEE: A benchmark environment for off-policy evaluation in debt collections. arXiv preprint arXiv:2611.10972, 2026. URL https://arxiv.org/abs/2611.10972.
  32. [32]Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. NeoRL: A near real-world benchmark for offline reinforcement learning. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, volume 35, pp. 24753–24765, 2022.arxiv.org/abs/2102.00714
  33. [33]Raghu Rajan, Jessica Lizeth Borja Diaz, Suresh Guttikonda, Fabio Ferreira, André Biedenkapp, Jan Ole von Hartz, and Frank Hutter. MDP Playground: An analysis and debug testbed for reinforcement learning. Journal of Artificial Intelligence Research, 77:821–890, 2023.DOI
  34. [34]James E. Smith and Robert L. Winkler. The optimizer’s curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52(3):311–322, 2006.DOI
  35. [35]Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Michael Mozer, Paul Smolensky, David Touretzky, Jeffrey Elman, and Andreas Weigend (eds.), Proceedings of the 1993 Connectionist Models Summer School, pp. 255–263. Lawrence Erlbaum, Hillsdale, NJ, 1993.
  36. [36]Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp. 2094–2100, 2016.arxiv.org/abs/1509.06461
  37. [37]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1706.03762
  38. [38]Cameron Voloshin, Hoang M. Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
  39. [39]Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, volume 33, pp. 14129–14142, 2020.arxiv.org/abs/2005.13239

Paper details

vlnc:W_wvteef4y
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Fable 5 (bare)

View statement
Looking for related papers…
Contents