WorksComputer Science
Agent Verifiers in Counterfactual Analysis: A Pilot Study Benchmarking Agentic Heterogeneous Treatment Effects
Abstract
Agents can expand the covariate space of randomized experiments by retrieving fast-moving local context missed by slow-changing survey baseline covariates, helping characterize who benefits most from an intervention. Verifying such covariates is difficult: individual treatment effects are unobserved, retrieval leakage can pull post-treatment information into nominally pretreatment summaries, and encoder leakage can inject post-treatment pretraining knowledge into representations even after temporal filtering of inputs. Guided by a two-channel information bound separating summary provenance from encoder weights, we propose a three-layer benchmark protocol: (i) retrieval admissibility via pre-treatment date filtering; (ii) heterogeneity validation scored rank-weighted average treatment effects (RATEs); and (iii) encoder admissibility checked with chronologically vintaged encoders, future-fact probes, and weight perturbations. Across 11 datasets anchored by Uganda's Youth Opportunities Program, we compare session-isolated retrieval (sandboxed, bounded-budget agent runs per study) against a persistent workspace teammate (xAI's GrokBot) that builds one date-verified corpus per study while blinded to outcomes. The workspace teammate yields higher mean RATE ratios in 6/11 datasets for text alone and 7 for combined features, though paired tests show neither workflow universally dominates. Even on fixed retrieved text, encoder choice significantly shifts heterogeneity signal. Chronologically vintaged probes show no clear evidence of weight leakage, but single-record adapter perturbation analyses show that memorized future facts, once present, persist until injected noise degrades model output. Temporal filtering of retrieval is necessary but insufficient: verifying agent-generated covariates requires jointly examining source provenance, out-of-sample treatment-effect signal, and encoder weights.
Workshop on Who Verifies the Agents? Toward Reliable Agent Development, NeurIPS 2026.
Introduction
Randomized controlled trials (RCTs) estimate average treatment effects, but practical decisions—such as targeted interventions and personalized treatment allocation—depend on identifying who benefits most. The individual effect is not jointly observable; for baseline covariates , the prediction target in conditional average treatment effect (CATE) estimation is instead [1, 2, 3, 4]. The quality of any CATE estimate is bounded by the quality of the pretreatment covariates it conditions on. Yet rarely do researchers know in advance which signals matter most.
Recent work has expanded the covariate space of experiments with new data modalities: satellite imagery [5, 6], text [7], and vision-language representations [8]. In particular, archived web text records fast-moving local events—political unrest, price shocks, infrastructure projects, the local context around an RCT—that slow-changing survey-obtained baseline covariates may struggle to capture. Language-model agents are a potentially promising tool for retrieving such information at large scale and low cost.
Deploying such agents in causal studies, however, poses a verification problem absent from existing agent benchmarks, which evaluate task completion, instruction following, and tool use in interactive environments [9] or aggregate capability metrics across tasks [10]. Rather than scoring task execution, we ask whether an agent’s data-producing behavior yields representations that are causally admissible for downstream treatment-effect estimation. Checking whether agent-retrieved context improves CATE estimation is difficult because unit-level effects are unobserved and retrieved documents, summaries, or embeddings can leak post-treatment outcomes into pretreatment covariates.
Moreover, post-treatment leakage can corrupt causal inference through two distinct channels. First, under retrieval leakage, an agent pulls post-treatment or retrospectively edited pages into nominally pretreatment summaries. Second, under neural-weight-based (encoder) leakage, a pretrained encoder injects knowledge from post-treatment pretraining corpora into embeddings even when the input text predates the intervention.
To address leakage and evaluation jointly (Figure 1)—guided by a two-channel information bound separating summary provenance from pretraining weights (Theorem 1)—we here propose a three-layer benchmark framework for agentic heterogeneous treatment effects: Layer 1 (retrieval admissibility) counters retrieval leakage by restricting acquisition to verifiably pre-intervention sources; Layer 2 (heterogeneity validation) addresses unobserved unit-level effects and search overfitting by scoring out-of-sample prioritization with cross-fitted rank-weighted average treatment effect (RATE) ratios [11]; and Layer 3 (encoder admissibility) targets weight-based leakage by auditing pretrained encoders with chronologically vintaged models [12], future-fact probes, explicit-label controls, and weight perturbations.
Figure 1. Verifying agentic workflows in causal systems. a, In standard task benchmarks, every unit’s ground-truth answer is observed directly. b, In counterfactual analysis, individual treatment effects are never jointly observed and post- information can leak through retrieved documents or pretrained encoder weights, motivating temporal filtering (Layer 1), aggregate RATE validation (Layer 2), and vintage and -stability probes (Layer 3).
To test how distinct agent designs perform under this protocol, we anchor our evaluation on the 2008 Uganda Youth Opportunities Program (YOP) RCT [13, 14] within an 11-dataset benchmark (encompassing 47 causal contrasts) comparing two retrieval architectures. The first is session-isolated retrieval: a sandboxed CLI agent harness that runs ephemeral, bounded-budget web and archive sessions per study context and filters out post-cutoff dates ex post. The second is a persistent workspace teammate (xAI’s GrokBot): working in a cloud VM with shell and PDF-parsing tools, it receives study contexts without outcome data, compiles one shared corpus per study, and passes an automated date-and-keyword check before estimation begins.
In Uganda, both agentic text workflows exceed matched tabular RATE ratios on all four skill and human-capital outcomes, whereas all representations yield negative ratios on income. Across all 188 paired RATE tests (spanning 47 contrasts over context-level, area-pooled, text-only, and combined specifications), only 7 workflow differences survive multiplicity adjustment: 5 favor the workspace teammate and 2 favor session-isolated retrieval. Meanwhile, swapping pretrained text encoders significantly shifts CATE rankings in both directions, and while chronologically vintaged probes detect no post-event shift on clean historical text, controlled weight perturbations show that memorized future facts remain recoverable until injected noise degrades model fluency.
Background and problem setup
Admissible agentic representations. For unit , let denote the intervention date and the raw covariates; the CATE over is . A representation is admissible only if it can be constructed exclusively from information available before ; we write for this verifiably pre- information set. Let denote the study-defined locality of unit . For each document , let denote its locality and its publication or archival timestamp. The eligible historical corpus is
Let collect every parameter or model weight used by the retrieval-and-summarization policy. The policy is non-anticipating only when is -measurable and
Admissibility also requires that the policy parameters do not depend on post- information. In practice, units in the same locality and period share a context . Session-isolated retrieval applies (2) independently to each context in an ephemeral session with separate search noise . A workspace teammate instead pools pre- documents across all contexts in a study inside one persistent workspace before mapping verified excerpts to units. Sharing one corpus across contexts reduces per-unit search noise, but any post-treatment leak in a shared source then contaminates every unit mapped to it—whereas post- leakage in shared policy weights affects all units under either topology.
Encoded representations and two-channel admissibility. Once a summary is assembled, mapping it through a numerical text encoder yields a learned representation and defines a CATE over ,
Because post-treatment information can enter at either stage—during text acquisition () or during embedding ()—admissibility has two parts (see Figure 2): retrieval admissibility excludes documents or policy inputs with unverifiable timing from , whereas encoder admissibility requires that inject no knowledge learned after (weight-based leakage).

Figure 2. Three paths from experimental units to CATE representations. a, Baseline representations (census covariates, Earth-observation imagery) are admissible but miss fast-changing political and economic context. b, Naive web retrieval captures that context but leaks future outcomes into nominally pretreatment representations. c, Ordered along data flow, the agentic framework filters temporal provenance at retrieval (Layer 1), audits the encoder for weight-based leakage (Layer 3), and scores out-of-sample heterogeneity via the RATE protocol (Layer 2).
Autonomy tiers in agentic heterogeneity analysis. Retrieving and encoding external context is one instance of a broader design space in which an agent selects covariates, engineers features, acquires new data, or optimizes the estimation pipeline itself. Let . For a fixed agent specification, let be a selected covariate subset and measurable feature maps. Orthogonal to our three verification layers (which audit representations), we distinguish five tiers of agent autonomy (which classify what the agent constructs), yielding representations :
where , , and the historical corpus retain their definitions above, and is the representation induced by a complete pipeline selected using performance feedback on development data. For fixed construction rules (, or conditioning also on the development sample for ), each tier induces a CATE,
Moving up these autonomy tiers progressively expands the agent’s scope. Tiers 1 and 2 select or nonlinearly transform existing survey variables ( and ), whereas Tier 3 adds historical context from outside the recorded covariates: session-isolated retrieval and GrokBot both operate at Tier 3, differing in session persistence, tooling, and date verification, and are evaluated below via standalone text (recovering ) and combined features (inducing below). Tier 4 iteratively revises these choices together with the CATE estimator (formalized as RATIO in Appendix E); for this learned rule, the CATE is conditional on the development data used to select and fit the pipeline. Across all five tiers, every input and model weight must remain admissible before , and any outcome-guided search must be confined to training splits (Section 3).
Additional heterogeneity resolved by acquired covariates. Stepping from internal transformations () to external acquisition () enlarges the baseline conditioning set along a nested ordering of -algebras (whereas Verification Layers 1–3 below are complementary checks across distinct pipeline components): for fixed selection and construction rules, the generated -algebras satisfy
Thus, constructing features from can aid finite-sample estimation without expanding the information in the full baseline, whereas acquiring external features can expand it. Assuming , define . For a fixed retrieval-and-summarization specification and encoder (with weights ), define the additional resolved heterogeneity as
The equality follows from and the law of total variance. Consequently, exactly when the acquired covariates resolve conditional mean treatment-effect differences beyond . Note that characterizes incremental signal of over (not standalone text performance ). To isolate pure retrieved-context signal alongside combined performance, our empirical benchmark evaluates standalone text , baseline covariates , and combined features ; because , , and are unobserved, we score out-of-sample prioritization using the rank-weighted functional below.
The RATE ratio as a verification metric. Because CATEs have no per-unit ground truth, we evaluate whether a representation separates high- from low-benefit units out of sample using a rank-based functional. Given out-of-fold priority scores from a CATE estimator, the rank-weighted average treatment effect (RATE) weights units by how strongly they are prioritized, with AUTOC weighting as primary and Qini weighting as a robustness check [11]. With standard error clustered at the randomization-unit level, the RATE ratio is
Under the regularity conditions of Yadlowsky et al. [11], the RATE ratio (7) is asymptotically standard normal under ; we write AUTOC/SE and Qini/SE for its AUTOC- and Qini-weighted instantiations.
The benchmark protocol: three verification layers
Structural decomposition: a two-channel bound. Even when retrieved documents predate , post-treatment information can enter through post- pretraining records in the encoder weights or through the agentic retrieval policy. Theorem 1 bounds the pretraining term and keeps the summary term explicit. Let denote a post- quantity correlated with the potential outcomes—the realized outcome or, in an encoder-leakage audit, a post-cutoff event label (such as a subject’s later death year or a subsequent earthquake). The encoder weights are produced by a pretraining algorithm
where is the pre- portion of the pretraining corpus, consists of post- training records referencing the event (leaving the -dependence of and implicit), and is the algorithm’s internal randomness. This is a closed structural model: every data source or selection rule through which can affect training must appear among . Three assumptions—representation construction, a closed training model, and per-record KL stability with constant —are stated in Appendix A.
Theorem 1 (Two-channel information bound). Under Assumptions 1–3,
Moreover,
For , the product is interpreted as zero.
Theorem 1 motivates the three-layer structure below (proofs and a differential-privacy corollary in Appendix A). In the retrieval channel , excluding post- documents prevents direct outcome contamination—motivating Layer 1’s temporal filter (Section 3.1), though timestamps alone cannot bound policy weights (Remark 2). Because admissible pre- summaries still lack unit-level counterfactuals and risk search overfitting—while leaked summaries would inflate out-of-sample scores—Layer 2 pairs Layer 1 with cross-fitted RATE validation (Section 3.2). In the weight channel, post-treatment influence is bounded by , the number of event-referencing pretraining records times per-record KL sensitivity. Because is unknown for off-the-shelf encoders, Theorem 1 bounds this channel only structurally; in Layer 3, we probe it empirically with vintaged encoders (Section 3.3) and calibrate it at exact values under Gaussian noise (Section 4).
Layer 1: retrieval admissibility
Temporal provenance filter. For each located, dated experimental unit, both workflows withhold treatment assignments and outcomes from the agent while enforcing pre- date bounds at retrieval (Appendix § 7). Session-isolated retrieval applies ex-post no-post filtering to per-context web and archive searches, dropping explicitly post-cutoff items while retaining undated excerpts. The workspace teammate (GrokBot) compiles one shared corpus per study and enforces stricter known-pre filtering, retaining only sources with verified pre-intervention dates and no outcome mentions. Even under strict date filtering, however, document timestamps cannot rule out post- knowledge in the retrieval agent’s own weights .
Layer 2: heterogeneity validation
Cross-fitted RATE scoring. Once candidate representations pass Layer 1’s temporal filters, Layer 2 evaluates whether they resolve genuine out-of-sample treatment-effect heterogeneity. To isolate the contribution of each representation—comparing tabular covariates , agentic text , and concatenations such as —we train causal forests [2, 15] on identical samples, cross-fitting folds, and hyperparameters. From the resulting out-of-fold priority scores and shared doubly robust evaluation scores, we compute the AUTOC-weighted RATE ratio (7) with standard errors clustered at the randomization-unit level, and report Qini-weighted ratios in robustness checks.
RATIO: a proposed Tier-4 search specification. Whereas our benchmark evaluates fixed Tier-3 workflows, Tier-4 pipeline search (Equation (4)) must guard against adaptive overfitting; we therefore discuss—though leave as a secondary extension for future investigation—a sample-split search loop, RATIO (Appendix § 10). Subject to the admissibility constraints, an agent proposes covariate subsets, feature maps, retrieval policies, encoders, and estimator settings across fixed cluster-preserving development splits, using validation-subset RATE ratios solely as an optimization signal. Within a fixed search budget, RATIO selects the candidate with the highest mean validation ratio, refits it once on the full development sample, and evaluates the frozen pipeline once on the held-out test split.
Yet, even strict sample splitting only helps prevent overfitting; it cannot detect post-treatment knowledge memorized in pretrained encoder weights, which requires Layer 3.
Layer 3: encoder admissibility
Chronological vintage probes. Passing Layer 1 guarantees only that the input text predates ; the encoder can still inject post- facts memorized during pretraining. We probe this channel with chronologically vintaged encoders trained on chronologically bounded corpora [12]: hold the input text fixed and always pre-event, vary only the encoder vintage, and ask whether later vintages recover future facts from the same historical text better than earlier vintages. Because successive vintages change the entire intervening corpus and its optimization trajectory alongside , vintage comparisons serve as observational diagnostics. The death-year probe predicts subjects’ later death years from biography snapshots written while the subjects were alive. The earthquake probe embeds pre-disaster Wikipedia pages for struck and matched control cities using encoders trained before and after each quake. Weight leakage would pull struck cities’ embeddings—more than controls’—toward disaster terms in post-quake encoders.
Calibration instruments. Observational probes alone cannot distinguish a leakage-free encoder from an insensitive probe, nor attribute a detected shift to specific post- records; two calibration instruments help close these gaps. First, to verify that our embedding and linear probe can recover a future label when it enters the text—a check sensitive to token pooling and input length—each death-year probe pairs clean historical biographies with positive controls that prepend or append the subject’s death year explicitly. Second, to isolate single-record parameter memorization from broader corpus drift between vintages—and to make the per-record stability constant in Theorem 1 finite and analytically exact—we pair observational probes with a Gaussian-perturbed adapter experiment that also tests whether post-hoc weight noise can erase memorized facts without destroying fluency.
Empirical verification: workflows, encoders, and leakage audits
Anchor experiment: Uganda YOP
We first apply Layers 1 and 2 to our anchor field experiment, Uganda’s 2008 Youth Opportunities Program (YOP), which randomized unconditional cash grants (approximately US$400 per member) to young-adult groups across 439 design clusters (1710–1715 matched individuals [13, 14]). Across 5 intention-to-treat outcomes, we compare baseline survey covariates against text from session-isolated retrieval, which ran an independent session per parish, and from the workspace teammate (GrokBot), which re-verified candidate URLs from the session-isolated run alongside pre-2008 district development and humanitarian reports. Using the small OpenAI encoder, we extract 4 principal components across the study’s unlabeled text embeddings—a low-dimensional compression chosen to curb split dilution on place-level embeddings in axis-aligned trees (Appendix § 7)—and fit honest causal forests (1000 trees) across shared 5-fold cross-fitting splits. Temporally filtered local text outperforms baseline survey covariates on all four skill and human-capital outcomes—training hours, vocational training, skilled trade, and human capital (Figure 3). On income, however, every representation yields a negative AUTOC/SE ratio (−1.15 for session-isolated text, −0.53 for GrokBot, and −0.55 for tabular covariates). To test whether a persistent workspace teammate systematically improves upon session-isolated retrieval beyond this single trial, we next compare both pipelines across the full multi-study battery.
Figure 3. Uganda YOP matched RATE ratios. Pooled out-of-fold AUTOC/SE for tabular covariates, session-isolated retrieval, and workspace-teammate (GrokBot) text. All representations share identical samples and saved folds (Qini/SE in Table 4).
| Outcome | AUTOC/SE | Qini/SE | ||||
| Tabular | Session-isolated retrieval | Workspace teammate | Tabular | Session-isolated retrieval | Workspace teammate | |
| Training hours | −0.65 | 1.77 | 2.22 | 0.64 | 2.81 | 2.67 |
| Vocational training | −0.22 | 2.21 | 2.98 | 1.22 | 3.23 | 3.83 |
| Skilled trade | 0.05 | 2.31 | 2.32 | −0.73 | 2.21 | 2.10 |
| Human capital | −1.35 | 1.55 | 0.35 | −2.07 | 1.44 | 1.07 |
| Income | −0.55 | −1.15 | −0.53 | 0.25 | −1.14 | −0.08 |
Table 4. Uganda YOP RATE ratios by outcome and representation. AUTOC/SE and Qini/SE ratios use 2000 shared resamples. Session-isolated retrieval and workspace-teammate (GrokBot) workflows retain their respective filters, with a common embedding and forest specification across outcomes.
Multi-study workflow and encoder sensitivity
Across the full battery of 11 datasets and 47 eligible treatment–outcome contrasts, the GrokBot workspace teammate achieves a higher within-study mean AUTOC/SE in 6 datasets for text alone and 7 for combined tabular-plus-text features (Table 1). Across context-level and area-pooled variants of text and combined features (188 paired AUTOC tests; context-level means in Table 1), 7 differences pass Holm multiplicity adjustment: 5 favor the workspace teammate and 2 favor session-isolated retrieval (Appendix § 9). GrokBot gains most when local context lives in structured administrative PDFs: in the Kenya digital-ID trial, parsing county census and development tables lifts mean text AUTOC/SE from 0.78 to 2.50 (and combined features from 0.99 to 3.43, passing Holm adjustment on tax registration). It also wins Holm-significant contrasts in direct air capture (area-pooled text) and the footbridge arm of trolley dilemmas. Conversely, session-isolated retrieval tends to win when strict known-pre temporal filtering leaves GrokBot with sparse governance reports—in civic honesty on area-pooled text (context-level text AUTOC/SE likewise drops from 0.25 to −2.44). Because the two workflows vary base model, tooling, search scope, and date filtering together, this comparison evaluates complete acquisition pipelines.
| Study | Contrasts | Tabular | Text Session-isolated retrieval | Text Workspace teammate | Tabular + text Session-isolated retrieval | Tabular + text Workspace teammate |
| Uganda YOP | 5 | −0.54 | 1.34 | 1.47 | 0.97 | 1.16 |
| Kenya digital ID | 5 | 3.18 | 0.78 | 2.50 | 0.99 | 3.43 |
| Direct air capture | 7 | −0.52 | 0.87 | 0.50 | 0.55 | 0.19 |
| Just transition | 7 | −0.82 | −0.51 | −0.15 | −0.78 | −0.71 |
| COVID-19 boosters | 5 | −1.08 | −0.26 | 0.44 | −0.35 | 0.04 |
| Bangladesh masks | 2 | 0.48 | −0.63 | −0.16 | 0.53 | 0.33 |
| Holiday advertisements | 2 | −0.08 | 0.65 | 0.06 | 1.18 | 1.39 |
| IST-1 stroke | 4 | −0.17 | −0.26 | −0.27 | −0.63 | −0.38 |
| Moral Machine | 7 | 1.60 | 1.94 | 1.37 | 4.61 | 4.85 |
| Trolley dilemmas | 2 | 8.87 | 3.95 | 5.13 | 10.32 | 9.88 |
| Civic honesty | 1 | 3.28 | 0.25 | −2.44 | 3.91 | 2.49 |
Table 1. Session-isolated retrieval and workspace-teammate (GrokBot) workflow comparison. Within-study means of contrast-specific AUTOC/SE ratios. The shared tabular benchmark uses identical samples and evaluation scores.
Our primary causal-forest benchmark fixes a single encoder compressed to 4 principal components fitted across unlabeled study embeddings—a tree-specific parsimony choice supported by pilot causal-forest width sweeps where PCA budgets diluted place-level signal (Appendix § 7). To test whether 4 components remain optimal once direct regularization replaces axis-aligned tree splitting—and to test sensitivity to encoder choice—we next run an R-learner analysis [16] that confines all scaling, PCA, nuisance estimation, and penalty tuning strictly to each outer training fold (Appendix D.2). Evaluating uncompressed (pre-PCA) embeddings (256–768 dimensions) alongside an in-fold 4-PC small-encoder control, we compare the large OpenAI encoder and MPNet against the small OpenAI baseline. Across 376 paired AUTOC comparisons on uncompressed embeddings, 16 remain significant after Holm–Bonferroni adjustment, split evenly between favoring the alternative encoder (8) and favoring the small OpenAI baseline (8); comparing uncompressed against in-fold 4-component small-encoder embeddings likewise yields 6 significant differences across 188 tests (4 favoring uncompressed embeddings, 2 favoring PCA). Compression to 4 PCs is thus an estimator-matched choice for causal forests rather than universally dominant under regularization, and both encoder choice and compression materially shift CATE rankings in either direction. Taken together, the 11-study RCT analyses show that date-filtered agentic context can capture treatment-effect heterogeneity missed by baseline surveys, yet downstream CATE rankings remain sensitive to both retrieval workflow and pretrained encoder choice—motivating Layer 3’s direct audits of encoder memorization below.
Encoder leakage audits: vintage probes and β-calibration
Observational vintage probes and pooling controls. To test whether pretrained encoders inject memorized post-event facts into temporally clean text, we apply Layer 3 using ChronoGPT encoders with 2007, 2014, and 2024 training cutoffs [12] while holding historical inputs fixed (Figure 4; Appendix § 8). In the death-year probe on 2007 () and 2014 () biography snapshots of then-living subjects, clean text yields negative rank correlations with death year across all vintages (Spearman ); binary late-death AUC rises across vintages in the smaller 2007 cohort (0.556 to 0.702) but falls from 0.534 (2014 vintage) to below chance at 0.423 (2024 vintage) in the larger 2014 cohort (Table 6). Explicit-label controls clarify both the rationale and scope of our primary pooling window: to equalize sequence length across articles and prevent short prepended control labels from washing out over long texts, our primary pooling rule averages the first 16 token states (Appendix § 8). Because ChronoGPT uses left-to-right causal attention, hidden states at tokens 1–16 cannot attend to any token after position 16. Prepending the death year at token 1 therefore enters the 16-token average directly and yields near-perfect recovery (), whereas appending it at the end of the article (past token 16) cannot alter the first 16 hidden states at all, leaving the pooled embedding identical to that of clean text—illustrating how pooling rules govern what a leakage probe can detect. To test place-level shocks that mirror geographic RCTs, we embed pre-disaster pages for 13 affected/control city pairs (9 earthquakes). For the 2024 encoder, the post-earthquake shift is 0.0049 (95% CI [−0.0465, 0.0577]); as detailed in Appendix § 8, confidence intervals still include zero after subtracting flood or city-term control shifts and when pooling the final 16 tokens or encoding city names alone.
Figure 4. Vintage comparisons and explicit-label controls. a, Spearman correlations () with actual death year for clean biographies (identical to the suffix control) and biographies with the death year prepended, across cohorts ( and ) and encoder vintages. b, Earthquake gap shifts with 95% pair-bootstrap intervals for encoders trained before or after each event; counts city pairs. 13 pairs are nested in 9 events.
| Snapshot / vintage / condition | MAE | Spearman | Bal. acc. | AUC | |
| 2007 snapshot / 2007 vintage / Clean | 76 | 1.716 | 0.305 | 0.563 | |
| 2007 snapshot / 2007 vintage / Prefix control | 76 | 0.094 | 0.979 | 1.000 | 1.000 |
| 2007 snapshot / 2014 vintage / Clean | 76 | 1.586 | 0.409 | 0.683 | |
| 2007 snapshot / 2014 vintage / Prefix control | 76 | 0.085 | 0.979 | 1.000 | 1.000 |
| 2007 snapshot / 2024 vintage / Clean | 76 | 1.510 | 0.381 | 0.702 | |
| 2007 snapshot / 2024 vintage / Prefix control | 76 | 0.086 | 0.979 | 1.000 | 1.000 |
| 2014 snapshot / 2014 vintage / Clean | 184 | 1.536 | 0.303 | 0.534 | |
| 2014 snapshot / 2014 vintage / Prefix control | 184 | 0.052 | 0.980 | 1.000 | 1.000 |
| 2014 snapshot / 2024 vintage / Clean | 184 | 1.607 | 0.272 | 0.423 | |
| 2014 snapshot / 2024 vintage / Prefix control | 184 | 0.056 | 0.980 | 1.000 | 1.000 |
Table 6. Death-year vintage probe results by snapshot cohort and encoder vintage. Suffix-control scores equal the corresponding Clean rows identically under first-16-token causal pooling and are omitted.
Controlled Gaussian -calibration. Because observational vintage probes cannot isolate single-record memorization from broader drift in the training corpus or yield a known per-record KL stability bound for the pretraining process, we complement them with a restricted training procedure in which is analytically exact (Figure 5). Starting from a frozen open-weight Pythia-70M encoder, we deterministically fit a rank-4 residual adapter with 2,048 trainable coefficients to each of two corpora that differ only in one record assigning a fictitious person a death year of 2031 or 2041, yielding distinct noiseless adapter vectors and . Deterministic fitting alone produces point-mass parameter laws with infinite KL divergence (), leaving Theorem 1 vacuous. We therefore inject isotropic Gaussian output noise around and for two purposes: first, it converts training into a randomized Gaussian mechanism with closed-form stability so Theorem 1 can be calibrated across exact regimes; second, it tests whether post-hoc weight perturbation—a post-training weight-perturbation mitigation (Corollary 1)—can mask memorized post-treatment facts without retraining. Because the evaluation biography (mean-pooled across all input tokens) is identical across all randomized models and contains neither death year, the summary term vanishes; with and a balanced draw between the two candidate years (, capping binary entropy at nats), Theorem 1 gives . At each of 7 noise scales, we fit a standardized linear probe to embeddings from 1,024 randomized models and test it on 4,096 independently drawn models.

Figure 5. Testing single-record weight memorization under Gaussian weight noise. Two corpora share pre-2026 background texts () and differ in one post-cutoff record () stating that a fictional subject died in 2031 (World , weights ) or 2041 (World , weights ). Adding Gaussian noise sets an exact one-record KL bound ; encoding a clean 2026 biography (containing neither year) then tests whether still predicts the memorized year and how much held-out fluency degrades as noise rises.
Empirical recovery tracks the theoretical bound closely, but suppressing weight leakage via post-hoc noise severely degrades model fluency (Figure 6; Appendix § 8.§ 2). Primary-probe accuracy drops from 92.8% at (32 nats) and 71.6% at (8) to 52.1% at (0.5), reaching chance once ; at , primary accuracy is 50.3% (matching the 49.5–51.4% null controls) and endpoint projection also falls to chance (Table 7). Across these same noise scales, however, held-out token negative log-likelihood jumps from 4.82 (frozen base; 5.52 for the noiseless fitted adapters) to 12.20 already at (32)—where probe recovery is still 92.8%—and 33.29 at , showing that isotropic weight noise destroys fluency well before memorized future facts disappear. Moreover, at (32) the probe recovers the future year with 90.7% accuracy even from an unrelated biography, showing that parameter leakage shifts embeddings globally beyond text naming the training subject.

Figure 6. Controlled Gaussian calibration. a, Accuracy of the primary and endpoint-projection probes (pointwise 95% Wilson intervals), plotted alongside the Bayes-optimal weight classifier (analytic weight oracle) and the primary probe under the identical-mean null control. Because the identical-mean null control has true (which cannot appear on the log axis), each null run is plotted at the coordinate of the signal run that shares its Gaussian noise variance . b, The entropy-capped upper bound, exact Gaussian weight-channel mutual information, and simultaneous probe information lower bounds (). c, Mean held-out token negative log-likelihood across 16 noisy model draws per setting; horizontal lines give the frozen-base value and the mean of the two deterministic fits.
| Primary linear probe | Endpoint projection | ||||||
| Acc. [CI] | AUC | Acc. [CI] | AUC | ||||
| (0.0078125) | 50.3 [48.8, 51.8] | 0.501 | 0 | 49.1 [47.6, 50.7] | 0.482 | 0 | |
| (0.03125) | 49.8 [48.3, 51.3] | 0.500 | 0 | 52.5 [51.0, 54.0] | 0.529 | ||
| (0.125) | 48.8 [47.3, 50.3] | 0.492 | 0 | 54.0 [52.4, 55.5] | 0.556 | 0.000453 | |
| (0.5) | 52.1 [50.6, 53.6] | 0.516 | 0 | 59.1 [57.6, 60.6] | 0.625 | 0.00895 | |
| (2) | 58.4 [56.9, 59.9] | 0.621 | 0.00726 | 68.1 [66.6, 69.5] | 0.742 | 0.0505 | |
| (8) | 71.6 [70.2, 73.0] | 0.790 | 0.0771 | 81.3 [80.1, 82.5] | 0.892 | 0.184 | |
| (32) | 92.8 [92.0, 93.6] | 0.982 | 0.401 | 96.4 [95.8, 96.9] | 0.995 | 0.507 |
Table 7. Target-biography probe calibration. Accuracy is in percent with pointwise 95% Wilson intervals; AUC is the area under the ROC curve. is the simultaneous 95% information lower bound in nats. The endpoint projection was both noiseless training-world embeddings. All 7 noise settings use 1,024 training and 4,096 independent test model releases.
Discussion and limitations
Takeaway. Across 11 studies, neither retrieval architecture dominates CATE ranking, and temporal filters at retrieval cannot confirm admissibility alone: pretrained encoders shift CATE rankings and can retain memorized future facts, while post-hoc weight noise degrades language-model fluency well before erasing that memorization.
Beyond causal inference. The protocol’s three verification layers apply beyond causal inference whenever agents construct features under an information constraint. Temporal provenance filtering applies to any forecasting, nowcasting, or prediction task where labels must not leak into inputs. Paired, cluster-robust RATE testing likewise applies whenever researchers rank units with agent-generated signals. Vintage probes transfer wherever a pretrained model’s corpus may overlap with evaluation targets, echoing contamination checks for language-model benchmarks.
Limitations. A hosted workspace teammate relies on unpinned model weights and a shared corpus whose sources still require direct audit. Our observational leakage probes cover specific encoders, token-pooling rules, and historical events, while the controlled experiment studies one invented fact under noisy adapter perturbation.
Data and code availability. The encoder-probe suite is available at https://huggingface.co/datasets/cjerzak/AgenticHeterogeneity. Tabular baselines can be recomputed from publicly archived trial data [14]; related code and data for the Uganda RCT are at https://huggingface.co/datasets/cjerzak/ImageHeterogeneity.
Two-channel information bound: assumptions, proofs, and scope
All random elements are assumed to take values in standard Borel spaces, so regular conditional laws exist. Conditional statements are understood for -almost every corpus , and mutual information and KL divergence use natural logarithms. Let denote a post- quantity correlated with the potential outcomes, such as the realized outcome or a post-cutoff event label (e.g., a subject’s death year or a subsequent earthquake). Temporal document filtering may exclude future documents, but can remain predictable from past text or enter through either the summary policy’s parameters or the encoder’s weights.
Assumption 1 (Representation construction). is a measurable deterministic function of . No independence between , , and is imposed.
Assumption 2 (Closed training model). and . In particular, no omitted training input or selection rule may depend on once is fixed.
Assumption 3 (Per-record KL stability). The records in are mutually independent conditional on , and there exists such that, for -almost every , every admissible realization of , every index , and every admissible replacement record ,
where denotes with its -th coordinate replaced by .
The constant is a uniform upper bound on the relative-entropy influence of one training record. Conditional independence of given is a substantive assumption on how post- records are generated. Likewise, a finite is not automatic: deterministic training can produce disjoint point masses and infinite KL.
Lemma 1 (Conditional data processing). Under Assumption 1,
The final term is the open summary/retrieval channel. It may contain ordinary predictive pretreatment signal as well as information introduced by a post-period retrieval or summarization policy.
Lemma 2 (Fact-to-group information bound). Under Assumption 2, .
Lemma 3 (Group information under KL stability). Under Assumption 3, .
Proof of Lemma 1. Conditional on , is a deterministic measurable function of . Conditional data processing and the chain rule give
No independence is used.
Proof of Lemma 2. If , the claim is immediate. Otherwise, for -almost every the first term below is finite, and we may expand by the chain rule in two ways:
Assumption 2 gives , so the second term on the left vanishes. The second term on the right is nonnegative. Rearranging, . Average over .
Proof of Lemma 3. If , both sides are zero. Otherwise, fix a value outside the exceptional null set and suppress it. By the chain rule, with . Fix and condition on . Write and take to be an independent copy of ; mutual independence of the coordinates then gives , so by the definition of conditional mutual information, convexity of relative entropy in its second argument, and Jensen’s inequality,
For any realizations of , both and are mixtures of over with the same mixing law, again by independence. Joint convexity of relative entropy therefore gives
by Assumption 3. Averaging over shows each of the summands is at most ; averaging over completes the proof.
Proof of Theorem 1. Apply Lemma 1 and bound its first summand using Lemmas 2 and 3. The displayed subsidiary chain is exactly Lemmas 2 and 3.
The encoder-only bound follows only under the additional condition , equivalently when regular conditional laws are used. Temporal document filtering alone does not imply this condition.
Corollary 1 (Differential privacy as a special case). Under Assumptions 1 and 2, suppose the conditional-independence clause of Assumption 3 holds and is pure -differentially private under bounded (replacement) adjacency of one training record while is held fixed. Then the KL-stability clause holds with , and
Proof. For the output laws on replacement-neighboring datasets, pure differential privacy in both directions gives . Hence
The last inequality uses and . Hence Assumption 3 holds with , and the bound follows from Theorem 1. Approximate privacy is not covered by this likelihood-ratio argument, and add/remove adjacency would require the corresponding replacement or group-privacy constants.
Remark 1 (Conditioning on ). The quantity measures residual association after conditioning on the chosen corpus. When the relevant information is finite,
Because conditioning can increase mutual information, can exceed ; interpreting it as post-treatment leakage therefore requires a sufficiency or causal model for .
Remark 2 (Residual retrieval channel). Theorem 1 retains the retrieval and summarization channel as . The term can mix legitimate predictive pretreatment content with leakage through , and timestamps alone do not separate them. Such a separation would require a baseline representation capturing the admissible pretreatment information in together with an explicit causal or conditional-independence model relative to . If the policy is itself trained, an analogous stability analysis may bound the contribution of its weights; it cannot simply be assumed to vanish.
Study battery, retrieval architectures, and primary estimation
Study battery and matched samples. Uganda estimation samples contain 1710–1715 observations in 439 design clusters, depending on the outcome. Each contrast uses identical units, weights, folds, propensities, and evaluation scores across representations. Beyond Uganda YOP, our 11-dataset battery (Table 2) comprises four located preference-conjoint experiments—Kenya digital ID [17], US direct air capture siting [18], US just energy transition policy [19], and US COVID-19 booster uptake [20], with each randomized attribute defining a separate contrast—alongside six public field, clinical, and behavioral experiments: Bangladesh masks [21], holiday advertisements [22], IST-1 stroke [23], Moral Machine [24], trolley dilemmas [25], and civic honesty [26].
| Study | Contrasts | Analysis rows | Places |
| Uganda YOP | 5 | 1,710–1,715 | 296 |
| Kenya digital ID | 5 | 14,076 | 1,224 |
| Direct air capture | 7 | 29,068 | 494 |
| Just transition | 7 | 9,220 | 407 |
| COVID-19 boosters | 5 | 9,655 | 579 |
| Bangladesh masks | 2 | 595 | 34 |
| Holiday advertisements | 2 | 6,717–6,773 | 754–788 |
| IST-1 stroke | 4 | 19,285–19,435 | 36 |
| Moral Machine | 7 | 10,618–116,950 | 71 |
| Trolley dilemmas | 2 | 135,996 | 41 |
| Civic honesty | 1 | 16,104 | 40 |
Table 2. Completed battery and eligible samples. Ranges span the eligible treatment–outcome contrasts within each dataset.
Retrieval workflows and provenance
Under session-isolated retrieval (Table 3), an automated worker pool spawns an ephemeral, read-only CLI session per study context (with repeated retrieval draws on the six multi-site public trials), calling a pinned gpt-5.4-mini deployment with bounded web-search, Wayback, and lightweight page-reading tools and checking the returned factor, verbatim-evidence, URL, and recorded-date fields ex post. Under the workspace teammate workflow, xAI’s GrokBot on grok.com runs in a persistent cloud virtual machine with a browser, PDF-parsing, and Python tooling. For each study, our benchmark harness provides GrokBot with a self-contained handoff package containing canonical place registries, contrast definitions, and candidate URLs from the session-isolated run while withholding treatment assignments and evaluation outcomes until the corpus is frozen. Within its VM, GrokBot builds a shared study corpus—filtering out-of-window candidates, parsing dated administrative PDFs, querying archived web snapshots and bibliographic records, and mapping verbatim excerpts across places—before an offline validator verifies known-pre publication intervals and excludes study-outcome keywords.
| Dimension | Session-isolated retrieval | Workspace teammate (GrokBot) |
| Compute runtime | Ephemeral, read-only CLI sessions calling a pinned gpt-5.4-mini deployment | Persistent cloud Linux VM on grok.com with browser, PDF-parsing, and Python tools |
| Retrieval topology () | Isolated per-context sessions (16-turn, 24-call budget; repeated draws on six public trials); no cross-context memory | Stateful study-wide pass over the canonical place registry; shared source cache and scripted place mapping |
| Source & archive tooling | Web search, Wayback lookup, and lightweight HTML/short-PDF text reader (no headless JS rendering or bulk PDF table parsing) | Browser rendering, bulk PDF table parsing, Wayback snapshots, PubMed/GitHub APIs, and candidate-URL re-verification |
| Temporal & leakage filter | Prompt window plus ex-post no-post date filter (retains undated rows) and published-trial self-citation regex filter | Ex-ante offline validator: known-pre date-interval bounds and study-outcome keyword exclusion |
| Outcome blinding | Worker inputs omit assignments and outcomes by script construction | Handoff package withholds evaluation outcomes until pre-evaluation corpus freeze |
| Governance trade-off | Pinned model deployment and fixed per-context prompt, but cross-session search noise and undated excerpts | Zero undated items and richer document parsing, but unpinned model weights and shared candidate URLs |
Table 3. Architectural and governance comparison of the two Tier-3 agent workflows.
Both workflows have provenance limits. In GrokBot’s run logs, we verify that outcomes were withheld during retrieval, though xAI does not expose an exact checkpoint identifier. Because GrokBot’s handoff included candidate URLs from the session-isolated run, the two corpora are not independent retrieval samples; replication also differs (repeated per-context draws in six studies under session-isolated retrieval versus one study-wide pass under GrokBot), and in IST-1—where patient recruitment began in 1991 and ran through 1996 prior to the trial’s May 1997 publication—we tightened the date cutoff from May 1997 (session-isolated) to December 31, 1990 (GrokBot) so all sources predate patient enrollment.
Pilot prompt-optimization check. During pilot development, we also tested an outcome-guided prompt loop in which an optimizer model iteratively rewrote retrieval prompts to maximize training-fold RATE ratios while holding out evaluation folds. Selecting the best of candidate prompts raised in-sample scores but failed to generalize out of sample—motivating strict outcome blinding during Tier-3 retrieval and a dedicated untouched test split in RATIO (Appendix § 10).
Primary representations and causal-forest estimation. Across the end-to-end verification pipeline (Figure 7), retained evidence paragraphs are concatenated per context and embedded using text-embedding-3-small. The original no-post filter retains undated sources; GrokBot’s known-pre requires a parseable date on or before the configured cutoff. Study-specific PCA uses all unlabeled context vectors and retains 4 components to suit axis-aligned tree splitting. Because text embeddings vary only across study contexts (as few as 34–40 places; Table 2) and reproducible signal concentrates in the first component, wider PCA budgets (16, 64, or 128 components) depleted splits on noisy trailing axes and never improved context-level AUTOC/SE in pilot sweeps. Area-pooled variants average context embeddings within administrative areas before PCA. Honest causal forests [2, 15] use 1000 trees and the saved 5-fold place-component partitions. Design clusters follow the assignment or repeated-observation unit appropriate to each dataset.

Figure 7. The agentic pipeline. Located, dated experimental units pass through agentic retrieval and temporal filtering (Layer 1)—instantiated either via session-isolated per-context runs or via a workspace teammate (GrokBot)—followed by text embedding, cross-fitted CATE estimation and RATE scoring (Layer 2), and encoder-weight leakage audits (Layer 3). Orange stages mark retrieval-provenance and encoder-leakage checks.
We pool one out-of-fold priority score per observation to estimate the RATE, using AUTOC weighting for the primary analysis and Qini weighting as a sensitivity check [11]. Priority scores are held fixed, all representations are evaluated against the same doubly robust scores, and standard errors and paired differences are computed from 2000 cluster-level half-sample bootstrap draws (following the grf RATE procedure) shared across the compared representations. All 188 primary workflow comparisons (text, combined, and their area-pooled variants) form one Holm family. Paired RATE differences between text and tabular representations are adjusted for multiplicity as a separate Holm family. As a sensitivity check, we re-estimate standard errors and paired tests with clustering by place component rather than by the study’s design cluster. For study-level summaries, we report unweighted means of contrast-specific ratios [27].
Layer 3 leakage probes and Gaussian calibration
Observational vintage probes
Chronologically vintaged encoders. The probes use ChronoGPT [12], a family of language models trained with chronologically bounded corpora. The vintages used are the chronologically consistent instruct checkpoints with cutoffs 2007-12-31, 2014-12-31, and 2024-12-31 (manelalab/chronogpt-instruct-v1-{20071231, 20141231, 20241231}). All probes hold the input text fixed and historical, varying the encoder vintage.
Death-year probe. Historical Wikipedia biographies concern people alive at snapshot time who later died (2007 cohort: 76; 2014 cohort: 184). Our primary probe configuration averages the first 16 hidden states with a 128-token input limit. This opening-16-token window was selected during pilot calibration because full biographies almost always exceed 128 tokens—causing end-of-article suffix controls to be truncated before encoding even under last-16-token pooling—while averaging the final 16 states of the 128-token window diluted a prepended death-year label across more than 100 intervening tokens, leaving the linear probe no more accurate than on clean text (zero MAE or rank-correlation gain). Averaging the first 16 causal states equalizes sequence length across snapshots and aligns the pooled representation with the prefix control, at the cost of ignoring text past token 16 (and, on unstripped Wikipedia revisions, occasionally pooling leading infobox or hatnote markup alongside or ahead of the subject’s name); suffix-control scores therefore match clean text identically in this configuration (Table 5 and Table 6).
| Snapshot / condition | MAE | Spearman | Window bal. acc. | AUC | |
| 2007 / Clean | 76 | [1.510, 1.716] | [−0.390, −0.053] | [0.305, 0.409] | [0.563, 0.702] |
| 2007 / Prefix control | 76 | [0.085, 0.094] | 0.979 | 1.000 | 1.000 |
| 2014 / Clean | 184 | [1.536, 1.607] | [−0.067, −0.018] | [0.272, 0.303] | [0.423, 0.534] |
| 2014 / Prefix control | 184 | [0.052, 0.056] | 0.980 | 1.000 | 1.000 |
Table 5. Death-year vintage probe summary. Bracketed intervals span the available vintages within each cohort. MAE is in years; window accuracy is balanced accuracy; AUC concerns late-death classification.
Earthquake probe. Starting from 11 candidate earthquakes (16 candidate city pairs, 29 historical city pages), restricting to complete pre-event affected/control pairs yields 13 comparisons across 9 events. For encoder and concept , the pair gap is , where denotes cosine distance between unit-normalized embeddings. We define a pair’s post-event shift by subtracting its mean pre-event gap from its post-event gap; because smaller cosine distance indicates closer embedding proximity, a negative post-event shift means that post-quake encoders pull struck places closer to earthquake terms relative to matched controls. To test specificity to earthquakes, we subtract the corresponding shift for a control concept (floods or generic city terms) from the earthquake-concept shift. Matching the death-year audit, our primary probe configuration averages the first 16 hidden states with a 128-token input limit. Because unstripped Wikipedia city pages often open with infobox markup inside the first 16 tokens, we also test last-16-token pooling across the 128-token window and city-name-only inputs (32-token cap). Both city-level checks yield 2024 post-event shifts whose 95% confidence intervals include zero, whereas bare region-name prompts (e.g., province or prefecture names by which major quakes are commonly named) pull struck regions closer to earthquake terms (negative gap shift).
Controlled Gaussian -calibration
In the controlled Pythia-70M calibration, deterministic full-batch Adam yields two fixed rank-4 final-block MLP up-projection vectors ( and , with 2,048 trainable coefficients and all other weights frozen) whose point-mass parameter laws would have infinite KL divergence without randomization; adding common isotropic Gaussian output perturbations induces an exact two-world stability constant while testing whether adding Gaussian weight noise after training can mask memorized post-treatment facts without retraining the encoder. Unlike the ChronoGPT prefix pooling rule, each Pythia embedding averages final hidden states across the entire fixed input biography. As an upper-reference linear benchmark, the noiseless-direction projection projects noisy embeddings onto the exact noiseless vector difference between both known training worlds, isolating whether signal missed by the primary linear probe reflects weight noise rather than high-dimensional probe estimation variance. Its classification threshold is fitted only on training draws. To form the information lower bounds, we compute a one-sided exact binomial lower confidence bound on probe accuracy—Bonferroni-adjusted across all 56 combinations of probe type, biography input, signal versus null-control condition, and noise scale—and substitute that bound into the binary Fano inequality (Table 7; Figure 6).
Contrast-level workflow results and encoder sensitivity
Contrast-level RATE ratios and paired workflow tests
Across all 47 contrasts, we compute primary RATE ratios per representation (Table 9) and paired workspace-teammate (GrokBot) minus session-isolated retrieval AUTOC differences adjusted within a single 188-test Holm family. Table 8 reports all 14 comparisons in this family with Holm-adjusted ; the remaining 174 comparisons yield Holm (in studies where administrative areas match retrieval contexts, such as ZIP-level holiday advertisements, context-level and area-pooled specifications coincide).
| Contrast and representation | AUTOC | CI lower | CI upper | Holm |
| Civic honesty / money vs. no money / wallet reported / area-pooled text | −0.0329 | −0.05 | −0.0157 | 0.0311 |
| Civic honesty / money vs. no money / wallet reported / tabular + area text | −0.02 | −0.0341 | −0.00601 | 0.896 |
| Direct air capture / no community benefits / profile selected / tabular + area text | −0.0213 | −0.0344 | −0.00828 | 0.241 |
| Direct air capture / enhanced oil recovery storage / profile selected / area-pooled text | 0.0243 | 0.0114 | 0.0372 | 0.0392 |
| Kenya digital ID / social protection / profile selected / text | 0.0391 | 0.0169 | 0.0613 | 0.1 |
| Kenya digital ID / social protection / profile selected / tabular + text | 0.0314 | 0.0143 | 0.0485 | 0.0588 |
| Kenya digital ID / tax registration / profile selected / tabular + text | 0.0386 | 0.0186 | 0.0586 | 0.0288 |
| Moral Machine / more vs. fewer characters / characters spared / text | 0.0189 | 0.00865 | 0.0292 | 0.0563 |
| Moral Machine / more vs. fewer characters / characters spared / area-pooled text | 0.0136 | 0.00534 | 0.0219 | 0.225 |
| Trolley dilemmas / footbridge vs. switch / sacrifice acceptable / text | 0.0153 | 0.0115 | 0.0192 | |
| Trolley dilemmas / footbridge vs. switch / sacrifice acceptable / area-pooled text | 0.0236 | 0.0192 | 0.028 | |
| Trolley dilemmas / footbridge vs. switch / sacrifice acceptable / tabular + area text | 0.0109 | 0.00625 | 0.0155 | 0.000756 |
| Trolley dilemmas / loop vs. switch / sacrifice acceptable / text | −0.00872 | −0.0121 | −0.0053 | 0.000104 |
| Trolley dilemmas / loop vs. switch / sacrifice acceptable / area-pooled text | −0.00619 | −0.0105 | −0.00189 | 0.834 |
Table 8. Paired workspace-teammate minus session-isolated retrieval AUTOC differences (Holm ). Pointwise 95% normal intervals use 2000 shared cluster-bootstrap draws across the 188-test workflow multiplicity family; all remaining 174 comparisons yield Holm .
Uncompressed embeddings and penalized R-learner estimation
As an exploratory post-hoc sensitivity check added after inspecting the primary causal-forest results, we evaluate uncompressed (pre-PCA) embeddings with -penalized R-learners. We re-embed the complete retained evidence with the small and large OpenAI encoders (text-embedding-3-small and text-embedding-3-large, both evaluated at 256 Matryoshka dimensions) and sentence-transformers/all-mpnet-base-v2 at 768 dimensions [28]. OpenAI inputs use nonoverlapping byte-bounded chunks; MPNet uses complete token windows. Splitting documents into chunks avoids truncation, so all input text contributes to the representation; we then combine chunk embeddings via length-weighted averaging. Because architecture, dimension, and chunking vary together across models, we compare complete encoding pipelines.
The CATE estimator is an R-learner [16] parameterizing on standardized features and minimizing weighted R-loss . Here is an slope penalty applied only to the heterogeneity slope vector while leaving the CATE intercept unpenalized: because uncompressed embeddings span 256–768 dimensions across as few as 34 unique places, penalizing prevents high-dimensional overfitting and shrinks weak heterogeneity toward a constant treatment effect () without shrinking the average effect . Within each saved outer fold, nested grouped validation chooses the slope penalty over a logarithmic grid from to , in half-log steps. Nuisance fits, imputation, scaling, and the small-encoder 4-component control use only their designated training partitions. Outcome nuisances use fixed-penalty -regularized linear regression; propensities use known design probabilities or training-only logistic regression. Missing text is represented by zeros and an availability indicator. All specifications share the frozen evaluation samples and scores; CATE predictions are not clipped. Unlike unsupervised PCA compression, fitting an R-learner on uncompressed embeddings regularizes the representation directly under causal R-loss.
Of 376 paired encoder contrasts, 16 remain significant after Holm–Bonferroni adjustment: 8 favor the alternative encoder and 8 favor the small OpenAI encoder. Across 188 paired tests comparing uncompressed (256-dimensional) embeddings from the small OpenAI encoder against its in-fold 4component PCA control, 6 remain significant after adjustment: 4 favor the uncompressed embeddings and 2 favor the 4-component control. In the separate Holm family testing paired differences over the tabular baseline, 43 of 752 comparisons are significant (24 positive, 19 negative). Slope penalties reach a grid boundary in 1377 of 3995 outer fits, primarily reflecting shrinkage toward a constant CATE () when fold-level heterogeneity is weak (study-level means in Table 10).
| Study / contrast / outcome | Tabular | Text: Session-isolated retrieval | Text: Workspace teammate | Tabular + text: Session-isolated retrieval | Tabular + text: Workspace teammate |
| Bangladesh masks / mask promotion / proper mask wearing | 1.81 | −0.54 | −0.88 | 1.35 | 1.05 |
| Bangladesh masks / mask promotion / physical distancing | −0.86 | −0.73 | 0.57 | −0.29 | −0.40 |
| COVID-19 boosters / 90% efficacy / profile accepted | −0.01 | 0.28 | 0.34 | 0.57 | −0.05 |
| COVID-19 boosters / two-year duration / profile accepted | −2.85 | −0.49 | 0.24 | −2.44 | −1.50 |
| COVID-19 boosters / variant protection / profile accepted | −1.53 | −0.67 | 0.66 | 0.70 | 0.29 |
| COVID-19 boosters / mRNA manufacturer / profile accepted | −1.85 | −0.57 | −0.61 | −0.75 | −0.43 |
| COVID-19 boosters / $1,000 incentive / profile accepted | 0.82 | 0.16 | 1.57 | 0.15 | 1.88 | |||||
| Civic honesty / money vs. no money / wallet reported | 3.28 | 0.25 | −2.44 | 3.91 | 2.49 |
| Direct air capture / fossil-fuel owner / profile selected | −1.49 | 0.29 | 0.60 | 0.48 | −0.22 |
| Direct air capture / fossil-fuel energy / profile selected | −0.51 | 2.60 | 2.78 | 0.87 | 2.30 |
| Direct air capture / no consultation / profile selected | 0.24 | 0.49 | −0.99 | 0.18 | −0.57 |
| Direct air capture / no local jobs / profile selected | 0.42 | 2.44 | 2.36 | 2.47 | 2.15 |
| Direct air capture / no community benefits / profile selected | −2.15 | −0.06 | −2.04 | −0.67 | −2.92 |
| Direct air capture / enhanced oil recovery storage / profile selected | 0.63 | −0.17 | 1.68 | 0.17 | 1.80 |
| Direct air capture / private funding / profile selected | −0.76 | 0.49 | −0.87 | 0.31 | −1.23 |
| Holiday advertisements / Thanksgiving ads / post-Thanksgiving cases | 0.36 | 0.86 | −0.24 | 1.63 | 1.78 |
| Holiday advertisements / Christmas ads / post-Christmas cases | −0.52 | 0.44 | 0.36 | 0.73 | 1.00 |
| IST-1 stroke / aspirin / dead or dependent (6 mo.) | 0.52 | 1.12 | 0.00 | 0.63 | −0.35 |
| IST-1 stroke / aspirin / death (14 d) | 0.76 | −1.47 | 0.36 | 0.66 | 1.32 |
| IST-1 stroke / heparin / dead or dependent (6 mo.) | −1.64 | −0.90 | −1.14 | −2.78 | −1.82 |
| IST-1 stroke / heparin / death (14 d) | −0.30 | 0.20 | −0.29 | −1.02 | −0.67 |
| Just transition / any income support / profile selected | −1.68 | 0.82 | −0.72 | −1.11 | −1.35 |
| Just transition / any benefit support / profile selected | −1.13 | −0.26 | 0.05 | 0.08 | −0.90 |
| Just transition / relocation voucher / profile selected | −2.29 | −0.73 | −1.09 | −1.49 | −0.93 |
| Just transition / clean-energy retraining / profile selected | −1.13 | −0.82 | 2.57 | 0.88 | 1.83 |
| Just transition / $100k salary / profile selected | 0.72 | −0.08 | 0.10 | 0.20 | −0.03 | |||||
| Just transition / short retraining time / profile selected | 0.62 | −1.24 | 0.68 | −2.24 | −1.53 |
| Just transition / school & community investment / profile selected | −0.87 | −1.29 | −2.61 | −1.75 | −2.08 |
| Kenya digital ID / public services / profile selected | 1.40 | 0.38 | 1.74 | 1.15 | 2.05 |
| Kenya digital ID / security / profile selected | 4.68 | 2.92 | 3.64 | 2.33 | 4.84 |
| Kenya digital ID / social protection / profile selected | 5.00 | −0.05 | 4.64 | 2.32 | 6.13 |
| Kenya digital ID / tax registration / profile selected | 2.43 | −0.42 | 0.46 | −1.38 | 2.84 |
| Kenya digital ID / voting / profile selected | 2.38 | 1.05 | 2.01 | 0.54 | 1.30 |
| Moral Machine / young vs. elderly / characters spared | 2.81 | 4.35 | 3.76 | 8.51 | 6.94 |
| Moral Machine / female vs. male / characters spared | 1.95 | 3.27 | 0.49 | 4.13 | 3.99 |
| Moral Machine / fit vs. large / characters spared | −1.85 | −0.30 | −2.07 | 1.15 | 2.09 |
| Moral Machine / higher vs. lower status / characters spared | −0.40 | 1.14 | −1.22 | 1.24 | 1.61 |
| Moral Machine / humans vs. pets / characters spared | 10.07 | 6.79 | 8.48 | 14.52 | 14.61 |
| Moral Machine / more vs. fewer characters / characters spared | 0.76 | −1.06 | 3.89 | 4.07 | 5.10 |
| Moral Machine / lawful vs. unlawful / characters spared | −2.18 | −0.60 | −3.73 | −1.36 | −0.41 |
| Trolley dilemmas / footbridge vs. switch / sacrifice acceptable | 12.50 | 5.96 | 13.58 | 12.94 | 14.50 |
| Trolley dilemmas / loop vs. switch / sacrifice acceptable | 5.24 | 1.93 | −3.32 | 7.70 | 5.26 |
| Uganda YOP / cash grant (ITT) / training hours (log) | −0.65 | 1.77 | 2.22 | 1.23 | 1.58 |
| Uganda YOP / cash grant (ITT) / vocational training | −0.22 | 2.21 | 2.98 | 2.70 | 3.15 |
| Uganda YOP / cash grant (ITT) / skilled trade | 0.05 | 2.31 | 2.32 | 1.50 | 2.55 |
Table 9. Primary AUTOC/SE by treatment–outcome contrast.
| Study / contrast / outcome | Tabular | Text: Session-isolated retrieval | Text: Workspace teammate | Tabular + text: Session-isolated retrieval | Tabular + text: Workspace teammate |
| Uganda YOP / cash grant (ITT) / human capital | −1.35 | 1.55 | 0.35 | 0.09 | −0.91 |
| Uganda YOP / cash grant (ITT) / income | −0.55 | −1.15 | −0.53 | −0.67 | −0.55 |
Table 9 (continued).
| Study / workflow / features | Tabular | Small, 4 PCs | Small (256-d) | Large (256-d) | MPNet (768-d) |
| Uganda YOP / session-isolated retrieval / text | −0.77 | −0.90 | −0.36 | 0.77 | 1.27 |
| Uganda YOP / session-isolated retrieval / tabular + text | −0.77 | −1.05 | −0.27 | 0.71 | 1.02 |
| Uganda YOP / workspace teammate / text | −0.77 | 0.39 | 1.21 | 1.59 | 1.13 |
| Uganda YOP / workspace teammate / tabular + text | −0.77 | −0.01 | 1.19 | 1.36 | 1.11 |
| Kenya digital ID / session-isolated retrieval / text | 0.19 | −0.49 | 0.64 | −0.03 | −0.19 |
| Kenya digital ID / session-isolated retrieval / tabular + text | 0.19 | 0.17 | 0.62 | −0.03 | −0.14 |
| Kenya digital ID / workspace teammate / text | 0.19 | −0.16 | 2.28 | 2.67 | 2.96 |
| Kenya digital ID / workspace teammate / tabular + text | 0.19 | 0.75 | 1.99 | 2.05 | 2.90 |
| Direct air capture / session-isolated retrieval / text | −1.31 | −0.66 | −0.22 | −1.05 | −1.01 |
| Direct air capture / session-isolated retrieval / tabular + text | −1.31 | −1.09 | −0.12 | −0.95 | −1.12 |
| Direct air capture / workspace teammate / text | −1.31 | −0.86 | −1.07 | −1.18 | −1.01 |
| Direct air capture / workspace teammate / tabular + text | −1.31 | −1.28 | −0.99 | −1.20 | −1.03 |
| Just transition / session-isolated retrieval / text | −1.65 | −1.85 | −1.67 | −1.45 | −2.10 |
| Just transition / session-isolated retrieval / tabular + text | −1.65 | −1.72 | −1.66 | −1.23 | −2.12 |
| Just transition / workspace teammate / text | −1.65 | −1.81 | −1.70 | −1.61 | −1.68 |
| Just transition / workspace teammate / tabular + text | −1.65 | −1.83 | −1.79 | −1.65 | −1.69 |
| COVID-19 boosters / session-isolated retrieval / text | −1.21 | −0.96 | −0.43 | −1.15 | −0.30 |
| COVID-19 boosters / session-isolated retrieval / tabular + text | −1.21 | −1.05 | −0.34 | −0.90 | −0.23 |
| COVID-19 boosters / workspace teammate / text | −1.21 | −0.06 | −0.36 | −0.08 | −0.25 |
| COVID-19 boosters / workspace teammate / tabular + text | −1.21 | −0.20 | −0.26 | −0.12 | −0.38 |
| Bangladesh masks / session-isolated retrieval / text | 0.49 | −1.57 | −1.47 | −0.84 | −1.14 |
| Bangladesh masks / session-isolated retrieval / tabular + text | 0.49 | 0.38 | −0.40 | −0.73 | −1.05 |
| Bangladesh masks / workspace teammate / text | 0.49 | −1.41 | −1.09 | −1.21 | −0.96 |
| Bangladesh masks / workspace teammate / tabular + text | 0.49 | 0.56 | −0.54 | 0.03 | −0.59 |
| Holiday advertisements / session-isolated retrieval / text | −1.61 | −1.58 | −0.68 | −1.61 | −1.23 |
| Holiday advertisements / session-isolated retrieval / tabular + text | −1.61 | −1.67 | −0.69 | −1.56 | −1.22 |
| Holiday advertisements / workspace teammate / text | −1.61 | −1.48 | 0.66 | −0.73 | −1.18 |
| Holiday advertisements / workspace teammate / tabular + text | −1.61 | −1.56 | 0.78 | −0.89 | −1.18 |
| IST-1 stroke / session-isolated retrieval / text | 0.05 | −1.06 | −0.88 | −0.97 | −0.80 |
| IST-1 stroke / session-isolated retrieval / tabular + text | 0.05 | −0.03 | −0.46 | −0.68 | −1.05 |
| IST-1 stroke / workspace teammate / text | 0.05 | −1.30 | −1.40 | −0.94 | −1.15 |
| IST-1 stroke / workspace teammate / tabular + text | 0.05 | −0.12 | −1.12 | −0.23 | −1.18 |
| Moral Machine / session-isolated retrieval / text | 1.69 | 1.28 | 2.23 | 1.36 | 2.48 |
| Moral Machine / session-isolated retrieval / tabular + text | 1.69 | 2.42 | 3.16 | 2.96 | 3.34 |
| Moral Machine / workspace teammate / text | 1.69 | 2.00 | 2.62 | 2.04 | 2.30 |
| Moral Machine / workspace teammate / tabular + text | 1.69 | 2.36 | 3.24 | 2.68 | 3.20 |
Table 10. Uncompressed-embedding R-learner sensitivity across encoders. Within-study mean AUTOC/SE against a shared tabular R-learner benchmark, with encoder dimensions and chunk pooling specified in Appendix D.2.
| Study / workflow / features | Tabular | Small, 4 PCs | Small (256-d) | Large (256-d) | MPNet (768-d) |
| Trolley dilemmas / session-isolated retrieval / text | 11.22 | −8.08 | −5.09 | −4.67 | 7.56 |
| Trolley dilemmas / session-isolated retrieval / tabular + text | 11.22 | 11.69 | 12.23 | 10.69 | 12.04 |
| Trolley dilemmas / workspace teammate / text | 11.22 | 2.22 | −4.51 | −2.36 | −3.91 |
| Trolley dilemmas / workspace teammate / tabular + text | 11.22 | 11.89 | 10.31 | 8.34 | 9.38 |
| Civic honesty / session-isolated retrieval / text | 2.76 | −2.31 | −4.21 | −2.64 | −0.74 |
| Civic honesty / session-isolated retrieval / tabular + text | 2.76 | 2.37 | −0.43 | −0.03 | 2.99 |
| Civic honesty / workspace teammate / text | 2.76 | −2.40 | −0.26 | 0.72 | −1.28 |
| Civic honesty / workspace teammate / tabular + text | 2.76 | 2.55 | 2.00 | 2.94 | 0.19 |
Table 10 (continued).
RATIO: adaptive pipeline optimization
RATIO implements the Tier-4 capability in Equation (4) by using RATE-ratio probes to revise the entire analysis pipeline. Let
collect covariate selection, feature construction, the retrieval-and-summarization specification (including ), encoder weights, feature integration, and CATE-estimator settings, respectively. Write for the representation induced by this configuration. The treatment, outcome, RATE weighting, and admissibility requirements are fixed before search; the agent proposes configurations within these constraints.
Development splits and validation scores. Split the sample by randomization cluster into development indices and an untouched test set . Partition the development sample into validation subsets , retaining the same splits across candidates. For each configuration and subset , fit the priority rule, preprocessing, and nuisance models on , and let denote the resulting RATE estimate on . The subset probe is
where the standard error clusters at the randomization-unit level.
Agent feedback and selection. Let be the agent’s proposal policy and let its history after evaluated candidates be
The feedback can trigger changes to any component of , including new retrieval or feature construction. After evaluating candidates within a prespecified search budget, select
using a prespecified tie rule. Because search inspects validation subsets repeatedly, this mean serves only as an optimization objective, not a standard-normal test statistic.
Final evaluation and scope. Freeze the selected specification and refit it on . Evaluate its resulting priority scores once on , reporting one RATE estimate and a cluster-robust standard error under the benchmark’s regularity conditions. The agent receives no test-set feedback during search, and the reported evaluation does not trigger another optimization round. A comparison with a prespecified baseline uses a paired, covariance-aware RATE contrast, as in § 3.
For an admissible selected pipeline, the information-gain identity in Equation (6) applies conditional on its development data; maximizing subset RATE ratios does not imply maximizing . An optional extension probes subpopulations defined by pretreatment covariates. Any selected subgroup is frozen before test-set evaluation, and the resulting RATE pertains to that subgroup.
References
- [1]Susan Athey and Guido Imbens. Recursive Partitioning for Heterogeneous Causal Effects. Proceedings of the National Academy of Sciences, 113(27):7353–7360, 2016.
- [2]Stefan Wager and Susan Athey. Estimation and Inference of Heterogeneous Treatment Effects using Random Forests. Journal of the American Statistical Association, 113(523):1228–1242, 2018.arxiv.org/abs/1510.04342
- [3]Sören R. Künzel, Jasjeet S. Sekhon, Peter J. Bickel, and Bin Yu. Metalearners for Estimating Heterogeneous Treatment Effects using Machine Learning. Proceedings of the National Academy of Sciences, 116(10):4156–4165, 2019.DOI
- [4]Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/Debiased Machine Learning for Treatment and Structural Parameters. The Econometrics Journal, 21(1):C1–C68, 2018.
- [5]Connor T. Jerzak, Fredrik Johansson, and Adel Daoud. Image-based Treatment Effect Heterogeneity. Proceedings of the Second Conference on Causal Learning and Reasoning (CLeaR), Proceedings of Machine Learning Research (PMLR), 213:531–552, 2023.arxiv.org/abs/2206.06417
- [6]Connor T. Jerzak, Ritwik Vashistha, and Adel Daoud. Effect Heterogeneity with Earth Observation in Randomized Controlled Trials: Exploring the Role of Data, Model, and Evaluation Metric Choice. arXiv preprint arXiv:2407.11674, 2024.arxiv.org/abs/2407.11674
- [7]Naoki Egami, Christian J. Fong, Justin Grimmer, Margaret E. Roberts, and Brandon M. Stewart. How to Make Causal Inferences Using Texts. Science Advances, 8(42):eabg2652, 2022.DOI
- [8]Satiyabooshan Murugaboopathy, Connor T. Jerzak, and Adel Daoud. Platonic representations for poverty mapping: Unified vision-language codes or agent-induced novelty? In Proceedings of the International Conference on Social Computing (ICSC), 2027.
- [9]Xiao Liu, Hao Yu, Hanchen Zhang, et al. AgentBench: Evaluating LLMs as Agents. In International Conference on Learning Representations, 2024. arXiv:2308.03688.arxiv.org/abs/2308.03688
- [10]Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, et al. Holistic Evaluation of Language Models. Transactions on Machine Learning Research, 2023. arXiv:2211.09110.arxiv.org/abs/2211.09110
- [11]Steve Yadlowsky, Scott Fleming, Nigam Shah, Emma Brunskill, and Stefan Wager. Evaluating treatment prioritization rules via rank-weighted average treatment effects. Journal of the American Statistical Association, 120(549):38–51, 2025.DOI
- [12]Songrun He, Linying Lv, Asaf Manela, and Jimmy Wu. Chronologically consistent large language models. arXiv preprint arXiv:2502.21206, 2025.arxiv.org/abs/2502.21206
- [13]Christopher Blattman, Nathan Fiala, and Sebastian Martinez. Generating Skilled Self-Employment in Developing Countries: Experimental Evidence from Uganda. The Quarterly Journal of Economics, 129(2):697–752, 2014.
- [14]Christopher Blattman, Nathan Fiala, and Sebastian Martinez. The Long-Term Impacts of Grants on Poverty: Nine-Year Evidence from Uganda’s Youth Opportunities Program. American Economic Review: Insights, 2(3):287–304, 2020.DOI
- [15]Susan Athey, Julie Tibshirani, and Stefan Wager. Generalized Random Forests. The Annals of Statistics, 47(2):1148–1178, 2019.arxiv.org/abs/1610.01271
- [16]Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299–319, 2021. doi: 10.1093/biomet/asaa076.DOI
- [17]Lisa Garbe, Nina McMurry, Alexandra Scacco, and Kelly Zhang. Who wants to be legible? Digitalization and intergroup inequality in Kenya. Comparative Political Studies, 58(9):1803–1853, 2025. doi: 10.1177/00104140241276971.DOI
- [18]Celina Scott-Buechler et al. Communities conditionally support deployment of direct air capture for carbon dioxide removal in the United States. Communications Earth & Environment, 5:175, 2024. doi: 10.1038/s43247-024-01334-6.
- [19]Alexander F. Gazmararian. Fossil fuel communities support climate policy coupled with just transition assistance. Energy Policy, 184:113880, 2024. doi: 10.1016/j.enpol.2023.113880.DOI
- [20]Shyam Raman, Douglas Kriner, Nicolas Ziebarth, Kosali Simon, and Sarah Kreps. COVID-19 booster uptake among US adults: Assessing the impact of vaccine attributes, incentives, and context in a choice-based experiment. Social Science & Medicine, 310:115277, 2022. doi: 10.1016/j.socscimed.2022.115277.DOI
- [21]Jason Abaluck et al. Impact of community masking on COVID-19: A cluster-randomized trial in Bangladesh. Science, 375:eabi9069, 2022. doi: 10.1126/science.abi9069.DOI
- [22]Emily Breza et al. Effects of a large-scale social media advertising campaign on holiday travel and COVID-19 infections: a cluster randomized controlled trial. Nature Medicine, 27: 1622–1628, 2021. doi: 10.1038/s41591-021-01487-3.DOI
- [23]Peter A. G. Sandercock, Maciej Niewada, Anna Członkowska, and International Stroke Trial Collaborative Group. The International Stroke Trial database. Trials, 12:101, 2011. doi: 10.1186/1745-6215-12-101.DOI
- [24]Edmond Awad et al. The Moral Machine experiment. Nature, 563:59–64, 2018. doi: 10.1038/s41586-018-0637-6.DOI
- [25]Edmond Awad et al. Universals and variations in moral decisions made in 42 countries by 70,000 participants. Proceedings of the National Academy of Sciences, 117(5):2332–2337, 2020. doi: 10.1073/pnas.1911517117.DOI
- [26]Alain Cohn, Michel André Maréchal, David Tannenbaum, and Christian Lukas Zünd. Civic honesty around the globe. Science, 365(6448):70–73, 2019. doi: 10.1126/science.aau8712.DOI
- [27]Entejar Alam, Po orbita Kundu, and Antonio R. Linero. Decision Theoretic Subgroup Detection With Bayesian Machine Learning. arXiv preprint arXiv:2509.05832, 2025.arxiv.org/abs/2509.05832
- [28]Sentence Transformers. all-mpnet-base-v2 model card. Hugging Face. URL https://huggingface.co/sentence-transformers/all-mpnet-base-v2. Revision e8c3b32e.


