betaFeedback

WorksComputer Science

Where Causal Discovery Breaks on Real Data: Measuring the Assumptions One Property at a Time

Abstract

Causal discovery is usually evaluated on simulators or on systems whose ground truth follows from physics or design. When it fails on other real data, the failure is attributed to assumptions that may not hold, but which ones fail, and by how much, is rarely measured. We measure the observable properties of the data they bear on, one property at a time, on a real panel. The case study follows 2,901 US public firms over 31 quarters, with 71 types of corporate action from SEC 8-K filings, 11 financial-statement items, and an expert graph over 651 action-state pairs. The panel breaks measurable requirements of discovery: tails that survive winsorisation, balance-sheet items and revenue close to random walks, and interventions too rare to detect effects of a tenth of a standard deviation in 40 of 71 actions. Even so, a marginal test ranks expert-labelled direct above not-direct pairs within each action (AUROC 0.658) better than every null we built, so the panel does carry information about which pairs are direct. Tuned discovery methods, in configurations feasible at this panel size, recover less of it than a simple pairwise cross-correlation when pairs are ranked within each action, and no more when they are ranked within each target, which removes what the target alone predicts. Most direct pairs (132 of 222) involve actions below the support needed for 80% power against a 0.1-SD effect. For flow targets, conditioning on the target's other expert-labelled parents, often filed alongside the action, erodes the marginal discrimination, and most not-direct associations survive it: even with the graph's parent sets supplied, the expected separation does not emerge. For each property, we give a diagnostic computable before any discovery method is run.

*Equal contribution. †Corresponding authors. ‡Work done while at Intuit AI Research.

Introduction

Most of what we know about causal discovery methods comes from data where the answer is known by construction: simulators, which can reward artefacts of the generator [42], and a few real systems whose ground truth follows from physics or design, such as river networks [52], climate model output [46] and industrial processes [37]. They share a regime: continuous variables, observed densely, with mechanisms that are stable over time. Much of the data people want to run discovery on differs on each count. Business, clinical and economic panels are often short and wide, with interventions that are discrete, rare and dated, and outcomes with extreme tails. When discovery does poorly on such data, the usual explanation is a list of assumptions that might be violated. They are rarely measured, so a practitioner cannot tell which violation matters, or whether a better method would have helped.

This paper measures the observable properties of the data that bear on these assumptions and on the statistical machinery of discovery, before a method is run. Causal sufficiency or faithfulness cannot in general be verified from observational data, and we do not claim to verify them; tails, persistence, collinearity and support can be measured. We take one real panel with an expert graph that is not estimated from it and, for each property, report a diagnostic, its value, the assumption it bears on, and what it does to discovery and to its evaluation.

The panel covers 2,901 US public firms over 31 quarters. The interventions are 71 types of corporate action filed on SEC Form 8-K, and the outcomes are 11 financial-statement items. Annotators labelled each action–state pair as direct, not direct or uncertain, and judged each direct effect one-time or persistent. The shapes are expert judgements, not estimates, and no method is asked to recover a lag. Finance is the setting, not the subject.

The paper follows that order. The panel breaks the assumptions behind discovery in measurable ways: extreme tails, near-unit-root persistence, strongly interrelated states, and interventions too rare for small effects (§ 4). The same properties force a calibrated evaluation: shift nulls sit above 0.5 and alias the effect at short distances, and scoring each pair by how often its target is labelled direct for other actions gives 0.860 (§ 6). We compute these references with one marginal test, an event-study comparison of firms that fire an action with firms that do not. It ranks direct above not-direct pairs within an action at AUROC 0.658, above all 8 nulls, so the data are not empty, and its detection rates later show which direct pairs a marginal comparison can see. Discovery methods from each family, tuned on the labels and scored the same way, reach 0.528–0.700 within action, below pairwise cross-correlation (0.824), and none has a larger excess over its own null than the baselines. Within target, which removes what the target alone predicts, none is above cross-correlation either (0.643 against at most 0.579; Section 7). Two questions account for the scores: why so few direct pairs are seen even marginally, and why conditioning does not separate the rest. Most direct pairs sit on actions too rare for a marginal comparison to find an effect of 0.1 SD (132 of the 222), the share detected rises from 31% in the lowest support quartile to 80% in the highest, and extreme rows hide part of the rest (Section 7.1). For flow targets, conditioning on the expert parents lowers the AUROC from 0.647 to 0.533, on levels mostly through same-quarter parents, which shrink direct coefficients but leave not-direct ones intact. Most not-direct detections survive: the panel carries dependence the graph does not account for. It survives firm and quarter effects and dropping the filing quarter, annotator disagreement explains few detections, and a confounder must be 11 times as strong as log firm size. Stock targets fail before conditioning: they do not separate even marginally (Section 7.1).

We contribute CauFin, a benchmark of corporate actions and financial states with an expert graph (§ 3); a catalogue of data properties tied to discovery assumptions, with diagnostics that transfer to other panels (§ 4 and § 8); an evaluation protocol calibrated against matched nulls (§ 6); and discovery methods from each family scored under that protocol, in the configurations this panel size allows, with an account of why they fail (Sections 7 and 7.1). Because the expert graph is partial and may contain errors, a low score alone does not show that a method fails; we bound how much label error could explain (Section M). The diagnostics apply to any panel of rare, dated interventions; the values we report describe this one. We will release the panel, the labels, the code and the extraction prompts upon acceptance.

Related work

Causal discovery benchmarks. Benchmarks are synthetic, semi-synthetic or real [21, 3]. Synthetic ones generate data from known graphs [57, 31] and can reward artefacts of the generator [42]; CausalTime [11] learns generators from real series, and TimeGraph [17] and CausalDynamics [23] scale generation to thousands of coupled continuous systems. Real benchmarks with known structure (protein signalling [47], cause–effect pairs [38], rivers [52], climate models [46], industrial processes [37]) share one regime: continuous variables, observed densely, under stable mechanisms, and most ask for the whole graph. CausalBench [12] pairs discrete interventions with continuous outcomes, but the interventions are designed; FinCARE [35] takes firm ground truth from 10-K filings but simulates the data. We know of no benchmark whose interventions occur naturally, rarely and on known dates in an observational panel, with edges labelled from domain mechanism rather than from the data. Gentzel et al. [19] argue for interventional evaluation; our labels are expert judgements, not interventions.

Methods and their assumptions. Methods test conditional independence (PC [51], PCMCI+ [45] (Runge 2020), LPCMCI [20]), optimise a score over graphs (GES [13], DYNOTEARS [39]), orient edges with non-Gaussian noise (VAR-LiNGAM [25]), ask whether one variable’s past predicts another’s [22], or use language models as priors or graph generators [5, 26, 43] (Kıcıman et al. 2024). Strong faithfulness fails for a large share of parameters [55], and single wrong independence decisions propagate through PC [16]. Extensions relax one assumption at a time: latent confounding [20], nonstationarity [24, 18, 4, 56], hidden confounding across environments [28]. Mixed binary and continuous data strain several families at once: partial-correlation tests assume a Gaussian model, and additive noise is ill-defined for a binary cause; MiTCD [10] first recovers latent continuous variables from the discrete ones. We know of no work that measures the full set of requirements on one real dataset against an expert graph.

Evaluation and negative controls. Our evaluation follows negative-control calibration in medicine [48] (Schümie et al. 2016; Schümie et al. 2018) and Bertrand et al. [7], who showed that placebo interventions in serially correlated panels are declared significant far too often unless the dependence is modelled. In perturbation-response prediction, simple baselines matched deep models [1] until the metrics were recalibrated against positive and negative controls [36]: a precedent for checking the metric before the method.

Causal inference in finance. Empirical finance identifies single effects, not graphs, with instruments, discontinuities and difference-in-differences [2]. Our marginal test is close to an event study [32]; unlike staggered-adoption estimators [8, 53], it does not correct for effects that differ across adoption cohorts. Its firm-clustered inference [41, 9] is unreliable when few clusters are treated [33], the usual case for a rare action.

The CauFin benchmark

CauFin is a partial discovery benchmark. A method may return any graph over the panel’s variables, but one block is scored: edges from an action filed in quarter tt to a state in the next four quarters.

Edge labels. We observe firms i=1,…,Ni=1,\ldots,N over quarters t=1,…,Tt=1,\ldots,T, with binary actions ai,tka^{k}_{i,t} and continuous states si,tjs^{j}_{i,t}. For horizon h∈{1,…,4}h\in\{1,\ldots,4\} let Wt,hW_{t,h} be every other observed variable dated before t+ht+h, including the target’s own past. Wt,hW_{t,h} includes variables dated after tt that the action may itself move, so following Pearl [40] we use the controlled direct effect, with all of them held fixed:

Δhkj(w)=E[st+hj∣do⁡(atk=1,Wt,h=w)]−E[st+hj∣do⁡(atk=0,Wt,h=w)].(1)\Delta^{kj}_{h}(w)=\mathbb{E}\left[s^{j}_{t+h}\mid\operatorname{do}(a^{k}_{t}=1,W_{t,h}=w)\right] -\mathbb{E}\left[s^{j}_{t+h}\mid\operatorname{do}(a^{k}_{t}=0,W_{t,h}=w)\right]. \tag*{(1)}

The pair is direct if Δhkj(w)≠0\Delta^{kj}_{h}(w)\ne0 for some hh and ww: the action moves the state by a route that passes through no other observed variable. It is not direct if Δhkj(w)=0\Delta^{kj}_{h}(w)=0 for every hh and ww: any effect runs through other observed variables, or there is none. A pair is uncertain if the answer depends on an attribute uu of the event that the panel does not record, so that Δhkj(w;u)≠0\Delta^{kj}_{h}(w;u)\ne0 for some uu and =0=0 for others. The attribute is either a detail of the action (an exit plan moves other comprehensive income only if it involves a foreign subsidiary or a pension curtailment) or a later management decision (an at-the-market equity programme moves equity only if it is used). Uncertain pairs are left out of every metric. The definitions are relative to the variable set: a route through an unrecorded quantity (working capital, headcount, a subsidiary’s balance sheet) counts as direct, and the labels are judged in this sense.

Panel. The panel covers 2,901 firms, identified by SEC Central Index Key, over the 31 quarters from 2018Q2 to 2025Q4 (89,931 firm-quarters, Table 4). It keeps only firms present in every quarter, so firms that listed, delisted or were acquired are absent; 1.8% of state values are unreported.

Firms2,901
Quarters (2018Q2–2025Q4)31
Firm-quarters89,931
Actions (8-K items)71 (10)
States11
State values reported98.2%
Firings per action, median (min–max)620 (24–20,826)
Firm-quarters with the action, median0.7%
Actions with fewer than 100 firings10
Action–state pairs781
direct / not direct / uncertain222/ 429/ 130
direct: persistent / one-time / no shape140/ 62/ 20
Actions with both classes43 of 70

Table 4. The panel and the expert graph.

Actions. The 71 action types are subtypes of 10 8-K items (Table 5), kept because they record voluntary corporate decisions and fire often enough to test; subtypes separate events under one item whose accounting differs. A language model assigns the events each 8-K discloses to subtypes (3% by embedding similarity instead), and events that fit no subtype are dropped. An action is binary per firm-quarter and dated by the calendar quarter of its filing. The median action fires 620 times (0.7% of firm-quarters), the rarest 24 times, and one item-5.02 subtype in 23% of all firm-quarters. An audit of 30 events finds none invented and two with the wrong subtype, and about 1.6% of firm-quarters miss an event; a missed or invented event hits an action’s direct and not-direct targets alike and pulls the within-action AUROC towards 0.5, while a wrong subtype has no fixed sign (Section B).

ItemEventSubtypes
1.01Entry into a material definitive agreement23
1.02Termination of a material definitive agreement12
2.01Completion of an acquisition or disposition4
2.03Creation of a direct financial obligation5
2.05Exit or disposal activities4
3.02Unregistered sale of equity securities1
3.03Material modification of holders’ rights4
5.02Departure or appointment of directors and officers5
5.03Amendment to articles or by-laws; change in fiscal year3
8.01Other events10
Total71

Table 5. Action subtypes per 8-K item.

States. The 11 states are line items from the firms’ 10-Q and 10-K filings: current and noncurrent assets, current liabilities, equity, revenues, operating income, net income, the three cash-flow totals and other comprehensive income. There are no totals for assets, liabilities or the change in cash. Each is dated by the calendar quarter in which the reporting period ends. Flows filed year to date are converted to single quarters, and an audit of 40 values against the XBRL filings finds none of the 31 it could verify wrong (Section B). 98.2% of values are reported; the rest are left missing, not imputed, and drop out of any regression that uses them; exact zeros, some probably unreported, are kept as reported (Section B). Each state is moved to the start of the quarter (Figure 1a), divided by the firm’s median total assets (current plus noncurrent, so current assets alone where noncurrent assets is zero, and the median across firms for the two firms whose median is zero), and winsorised at the 1st and 99th percentiles within state and quarter.

Figure 1. The benchmark. (a) States are recorded at the start of each quarter, so the first change an action filed in quarter qq can produce is the change over qq, indexed h=1h=1; h=0h=0, the last change before the filing, is not tested. Annotators judged each direct effect one-time or persistent (counts are pairs; cells are schematic; 20 direct pairs have no shape); the shapes are judgements, not estimates. (b) Labels of all 781 action–state pairs, actions grouped by 8-K item. Near-uniform rows and blocks feed the label-only baseline of § 6.

Annotation. Three annotators labelled each action–state pair as direct, not direct or uncertain, judging whether an accounting mechanism exists rather than how large it is (Section C). 222 pairs are direct, 429 not direct and 130 uncertain, leaving 651 scored pairs. The 121 state–state pairs were also labelled, at lags 1–4, but are not scored; the 4 direct ones are all self-loops of the four stocks, so no flow has a self-edge and no state is a lagged direct parent of another; identities that hold within a quarter, such as revenue entering net income, are not lagged effects. The annotators followed a written rubric without seeing the panel data, and agree unanimously on 442 scored pairs but on only 95 of the 222 direct ones (Fleiss’ κ\kappa 0.60; Section C). One item-1.02 subtype has all its pairs uncertain, so 70 of the 71 actions enter the metrics. The labels are structured by row and block (Figure 1b): other comprehensive income is direct in 2 of 66 pairs, personnel changes (item 5.02) in 2 of 50. The labels encode the plausibility of an accounting mechanism, not an experimentally verified effect.

Timing. Labels belong to the pair, not to a horizon. The annotators also judged each direct effect one-time (a change in a single quarter) or persistent (carrying into later quarters): 140 are persistent, 62 one-time, and 20 have no shape. The shapes are part of the benchmark but are not evaluated here; scoring them is left to future work. They fix only the test window, h≤4h \le4; no lag is scored, because the same action takes effect at different horizons across firms with deal timing, fiscal calendars and management discretion. States are stamped at the start of the quarter, so for an action filed in quarter tt the state sts_t precedes the filing and the change at horizon hh is over quarter t+h−1t+h-1 (Figure 1a); every change the score uses is dated after sts_t.

Properties of the data

For each property we give a diagnostic and its value on the panel; Section 7.1 takes up what the properties hide from a marginal comparison and predict for discovery methods.

Heavy tails

The scaled levels have excess kurtosis of 4,043–83,688 as reported and 18–451 after winsorising (Figure 2a). The top 1% of rows still carry 27% to 76% of each state’s sum of squares, so for the most tail-heavy states a least-squares fit on 89,931 rows is dominated by fewer than a thousand.

Figure 2. (a) Excess kurtosis of each scaled level, as reported and after 1% winsorisation (log scale). (b) Within-firm AR(1) coefficients of the levels; boxes span firms, the blue tick is the median after Kendall’s correction.

Persistence

Persistence splits the states in two, though not into stocks and flows. The median within-firm AR(1)AR(1) coefficient is 0.83–0.93 for 5 states, the four balance-sheet items and revenue, and −0.05–0.43 for the rest (Figure 2b); revenue is a flow that behaves like a stock (0.83). Over 31 quarters the estimate is biased towards zero by roughly (1+3ρ)/T(1 + 3\rho)/T (Kendall, [29]); after that correction, 42%–64% of firms have a coefficient above one for the persistent states: over 31 quarters they are hard to tell from random walks. The residuals of a level regression on a rare binary action then inherit that persistence, and the firm-quarters are worth only 3,645–58,452 independent observations (ICC 0.018–0.795). The one-quarter change is close to stationary. Persistence also decides which shift nulls are valid (§ 6).

Heteroscedasticity against firm size is real but small: a Breusch–Pagan test rejects in all 2,604 cells even with cluster-robust inference, but the auxiliary R2R^2, which we report instead, has median 0.0180 (Section G).

Collinearity

Accounting ties line items together, and deterministic relations violate faithfulness outright. Which ones can be checked depends on what is reported: here 3 standard identities cannot be computed at all (Section G). No state is determined by the others, but some are largely explained by them: regressed on the other ten at the same date, the best-explained state has R2R^2 0.83, which leaves a test that conditions on the others less unique variation.

Sparse and bundled interventions

10 of the 71 actions fire fewer than 100 times. To see what support buys we plant an additive effect of δ\delta standard deviations into the real panel at random firm-quarters and rerun the marginal test of § 6 cell by cell (Section F). With no effect planted, the harness rejects in 0.064 of replicates at nominal 0.05 (the test’s own size is in Section K). An effect of 0.1 SD is found in 0.47 of replicates at 200 firings and needs about 752 for 80% power, which 40 of the 71 actions lack; at 0.05 SD, power is only 0.65 at 2,000 firings. Real actions cluster in fewer firms than planted ones, and under clustered inference the treated firms set power, so the grid bounds what the test can see from above; it does not estimate how much of the graph is visible. Support is also confounded with the labels (§ 6). Actions also arrive together: the median action shares its firm-quarter with another in 89% of firings, the least bundled in 62%, and for 18 of the 71 actions the most frequent partner is a subtype of the same 8-K item. Effects of actions that always co-occur cannot be separated. Same-item partners may also be one event that extraction split in two.

Experimental setup

Methods. We run discovery methods from four families. Constraint-based: PCMCI+ (Runge, [44]), which assumes causal sufficiency, and SVAR-RFCI (Malinsky and Spirtes, [34]), which allows latent confounders; conditional-independence methods are too expensive at this panel size for more representatives. Continuous optimisation: DYNOTEARS (Pamfil et al., [39]) and its nonparametric variant NTS-NOTEARS (Sun et al., [54]). Functional: longitudinal LiNGAM (Kadowaki et al., [27]), one non-Gaussian model per quarter across firms. Granger: a linear VAR Granger test (Granger, [22]), fitted on lagged rows that never cross two firms, and MiTCD (Chen and Zhao, [10]), a neural Granger model that embeds binary variables before learning a sparse graph. Two pairwise baselines set the floor a method should clear: lagged cross-correlation averaged over firms, and the event-triggered average, the mean change after an action with no control group.

Input and tuning. Methods receive the quarterly states and binary action indicators with a lag window of one to four quarters, and search four panel strategies (pooled levels, firm effects, firm and quarter effects, first differences) crossed with two state scales (raw dollars, or division by the firm’s median total assets), all winsorised at 1% and standardised. The transformations apply only to the states, so actions stay binary and the demeaned arms are not full within estimators. VAR Granger, longitudinal LiNGAM and the two baselines have no other hyperparameters and are evaluated under all eight combinations; DYNOTEARS, NTS-NOTEARS and MiTCD also search regularisation and optimisation hyperparameters within a common budget of 40 configurations. PCMCI+ and SVAR-RFCI are too expensive to tune: each runs in one fixed configuration, with conditioning sets capped at two variables,1 under each preprocessing combination, and its significance level is swept over four values to rank edges (Section 1.1); their results show what a feasible instantiation recovers, not that the unrestricted algorithms fail. Configurations are selected by pair-level within-action AUROC on the full label set, so every entry in Table 1 is the best over the configurations searched: an upper bound, not a held-out estimate.

MethodFamilyAUROCExcessWithin tgt.
PCMCI+CB0.650 [0.604, 0.695]0.091 [0.045, 0.140]0.548 [0.489, 0.606]
DYNOTEARSCO0.528 [0.509, 0.547]0.032 [0.011, 0.054]0.501 [0.456, 0.544]
NTS-NOTEARSCO0.575 [0.540, 0.610]0.052 [0.025, 0.081]0.380 [0.314, 0.447]
Long. LiNGAMFN0.700 [0.653, 0.747]0.057 [0.021, 0.095]0.506 [0.458, 0.555]
VAR GrangerGR0.620 [0.573, 0.667]0.069 [0.019, 0.117]0.579 [0.528, 0.630]
MiTCDGR0.541 [0.514, 0.595]0.000 [0.000, 0.001]0.482 [0.452, 0.524]
Cross-correlationbase0.824 [0.780, 0.865]0.237 [0.162, 0.309]0.643 [0.585, 0.698]
Event-trig. avg.base0.646 [0.588, 0.702]0.111 [0.053, 0.168]0.571 [0.505, 0.634]

Table 1. Discovery methods scored as in § 6, with 90% intervals from 2,000 bootstrap resamples of actions. AUROC: pair-level within-action; excess: over each method’s restricted per-firm circular-shift null (§ 6; 40 replicates, except PCMCI+, 20); within target: ranked within target over the labelled pairs (“within tgt.”), where an action-blind score gets 0.5. Null replicates refit each method’s best configuration. Methods are tuned on the full label set, so each entry is an upper bound. Bold: best score.

Contemporaneous edges. Because sts_t is fixed before any action in quarter tt is filed (§ 3), a contemporaneous edge at→sta_t \to s_t cannot be causal, and methods are scored on lags one to four only. On quarter-end data the same first change would sit at lag zero, where a lagged method never looks.

Scoring. A method’s output is collapsed to one score per action–state pair over lags one to four and scored as in § 6: the within-action AUROC, its excess over the method’s own restricted shift null, and the within-target AUROC.

How to evaluate in this setting

On this panel chance is not 0.5. Support differs across actions, the labels have structure of their own, and a null that keeps the data’s properties scores well above 0.5, so a method at 0.6 can be reading nothing. We fix the metric and its references before any method is run; most are negative controls [48], built to share the data’s properties but not the effect.

A marginal test. Every reference below (the nulls, the placebo, the shift curve) is recomputed thousands of times through one fixed score; a discovery method is too slow for that, and harder to read under a null. Rare, dated, binary interventions in a panel are the setting of the event study [32], so we use the simplest test of that kind: firms that fire an action against firms that do not, the pooled, one-quarter-change counterpart of the event-triggered average of § 5 with a control group. The test is also the paper’s instrument: its known size gives the references in this section, the power of § 4.4 and the detection rates of Section 7.1. It uses changes because the persistent states make a level regression on a rare action unreliable (§ 4.2). For action kk, state jj and h∈{1,…,4}h \in\{1,\ldots,4\} we regress si,t+hj−si,t+h−1js^{j}_{i,t+h}-s^{j}_{i,t+h-1} on ai,tka^{k}_{i,t}, pooled over the firm-quarters where the change is observed, with firm-clustered errors and degrees of freedom set by the number of firms that do and do not fire the action (Section K). A pair’s score is −log⁡10p-\log_{10}p of a joint test that any horizon differs from zero; other ways of combining the horizons give 0.638 to 0.658 (§ 21). The test is oversized when few firms fire the action; we flag the 87 such pairs.

Pooled AUROC ranks by firing frequency. Direct pairs sit on rarer actions (median 528 firings against 702), and 83% of pairs in the top support quartile are not direct, against at most 63% in the others; the most frequent action, a personnel change (item 5.02), is almost never linked directly to a line item. Pooled across actions, any score that grows with sample size partly ranks by firing frequency: the marginal test scores 0.609 pooled and 0.658 within action.

Within-action AUROC. We therefore compute the AUROC within action, comparing each direct target of an action with each not-direct target of the same action. All targets of an action share its firings, so support differences cancel. 43 of 70 actions have both classes (869 comparisons). The other 27 are invisible to the AUROC: 25 have no direct target and 2 no not-direct one, so a ranking metric says nothing about them, and for these we report detection rates instead (Section E).

What the target alone predicts. The within-action AUROC removes differences between actions but not between targets, which differ widely in how often they are labelled direct. Scoring each pair by the share of other actions labelled direct on its target, with no panel data, already gives 0.860. This is not a method to beat but a measure of what the target alone supplies, and it can be discounted two ways. The nulls of § 6 keep labels and targets fixed, so the excess over a null already leaves it out. Ranking within target (each score replaced by its percentile among the scored pairs on its target) removes it from the AUROC itself, so a target-only score gets 0.5, but also discards real signal shared across a target’s actions. Here the two disagree: the marginal test keeps an excess of 0.120 over its null but scores 0.564 within target, which points to real effects shared across a target’s actions. The within-action AUROC is therefore descriptive; the excess over the null and the within-target AUROC, which discount the target, are the evaluation, and we report both throughout.

Chance depends on the null. We build 8 nulls of 400 replicates each: label permutation within action, firm-block permutation with and without activity strata, row permutation, and circular shifts within firm, per action or per firm, at any distance or at least five quarters (Section D). Label and firm-block permutation centre on 0.501 and the shifts reach 0.556 (Figure 3a, Table 6), so the excess shrinks from 0.157 to 0.102, a factor of 1.5, as the null keeps more of the data. A shift keeps which firms file each action and moves only when. The gap is composition: filers differ from non-filers more in some states than others, and direct labels concentrate on particular targets, so even a shifted action ranks direct pairs above not-direct ones more often than not. The shift null is the right reference; label and firm-block permutation are the most lenient. A placebo that scores each action against the changes at h=−4,…,−1h=-4,\ldots,-1, four quarters that end before the quarter preceding the filing, gives 0.568 against 0.540 under the restricted shift null (Section D), an excess about a quarter of the forward one and no pre-action difference beyond what the null carries.

Figure 3. (a) Within-action AUROC under each null (400 replicates); the line is the observed value. (b) AUROC with every action circularly shifted by the same distance; at one and two the bar spans both directions. Restricted nulls draw only from the shaded offsets.

NullMeanSDExcess
Firm-block permutation0.5010.0330.157
Label permutation0.5010.0310.157
Row permutation0.5140.0320.144
Stratified firm permutation0.5270.0340.131
Shift per action, distance ≥5\ge50.5310.0270.127
Shift per firm, distance ≥5\ge50.5380.0270.120
Shift per action, any distance0.5490.0280.109
Shift per firm, any distance0.5560.0280.102

Table 6. Within-action AUROC of the marginal test under each null, from 400 replicates. Excess is the observed 0.658 minus the null mean. No replicate of any null reaches the observed value, so every empirical pp is at its floor of 0.0025.

Short shifts are not nulls. The test looks four quarters past each firing, so a shift of one to four quarters can leave the real effect inside the window: at one quarter the AUROC is still 0.627–0.675 against 0.658 observed, and it falls to 0.480–0.507 only at five to eight (Figure 3b). The primary null therefore draws only shifts of five quarters or more. Each method’s excess in Table 1 is against the same null refitted through the method: its best configuration is rerun on panels with every action shifted within firm by five quarters or more. Methods search levels and first differences (§ 5); in levels a persistent step stays put wherever the shifted indicator lands, so the null keeps part of any persistent effect and the excess of a levels configuration is conservative.

The panel carries information about the labels. Against these references the marginal test scores 0.658. No replicate of any null reaches it (p≤0.0025p \le0.0025), and the excess over the primary null, the restricted per-firm shift, is 0.120. On pairs whose action fires in at least 80 firms, where the test holds its size in simulation (Section K), the AUROC is 0.662, an excess of 0.120, so the flagged pairs do not drive the headline. Nor do contested labels: unanimous pairs alone give 0.647, and random flips bring the AUROC down to the null’s 95th percentile only once 30% of not-direct labels are made direct (Section M). The panel carries information about which pairs the annotators call direct. Section 7 scores the methods against the same kind of reference, each against a null refitted through its own pipeline, and Section 7.1 asks which direct pairs the marginal test misses and why.

How do discovery methods perform on the panel?

No discovery method reaches cross-correlation (Table 1). The best within action, longitudinal LiNGAM, scores 0.700 with an excess of 0.057 over its own restricted-shift null, against 0.824 and 0.237 for cross-correlation and 0.646 and 0.111 for the event-triggered average (for configurations in levels the excess is conservative; § 6). LiNGAM’s own null sits at 0.643 and it scores 0.506 within target, so its AUROC reflects which firms act and which targets are labelled, the composition the references of § 6 are built to expose. Within target the gap narrows to within the intervals: VAR Granger scores 0.579, against 0.643 for cross-correlation and 0.571 for the event-triggered average; NTS-NOTEARS falls below chance (0.380), and MiTCD has no excess over its null. SVAR-RFCI [34] outputs a PAG (§ 18.§ 2): no direct pair receives a → or ◦ → mark in any of the 32 cap-2 runs, and almost every adjacency it keeps is bidirected, which with conditioning sets this small we read as computational stress, not latent confounding. The marginal test of § 6 is not in the table; it scores 0.658 within action, an excess of 0.120, and 0.564 within target. It is fixed in advance rather than selected on the labels. A probe trained on the labels stays within cross-correlation’s intervals (Section H).

Why do discovery methods struggle?

Discovery has to tell direct effects from associations induced by indirect paths, shared causes and temporal dependence, whether through conditional-independence tests, prediction or structural assumptions. We ask whether the distinctions the expert graph implies are recoverable from the panel.

The expert-parent test. Under the causal Markov and faithfulness conditions, with causal sufficiency and a correct graph, an action that is not a direct parent of a target is independent of it given the target’s parents, while a direct parent stays dependent given the other parents ([51]). Conditioning on a superset of the parents preserves the independence, provided the extra variables precede the target ((Runge et al., 2019)). These are implications of the graph, not of any algorithm, and we test them by supplying the conditioning sets from the expert graph instead of estimating them. The parent set is conservative: every annotated direct or uncertain action–state and state–state parent, so no candidate is dropped because its label was disputed. Extra non-parents cost power, not validity, and uncertain edges are never scored. For each flow pair and lag ℓ∈{1,…,4}\ell\in\{1,\ldots,4\}, we regress the target’s level sts_t on at−ℓa_{t-\ell}, alone and conditional on its other candidate parents at lags 1–4, with firm-clustered errors, and score the pair by max⁡ℓ∣tℓ∣\max_{\ell}|t_\ell| (Section J). The graph can still omit parents, place them beyond four quarters or leave common causes unobserved, so this is a favourable, graph-supplied conditioning test. Raw levels test the graph’s implication most directly; firm scaling, whose denominator uses outcomes across the panel, and first differences, which change the graph’s variables, are complementary checks.

Supplying the expert parents lowers the within-action AUROC in all four representations (Table 2; from 0.647 to 0.533 on raw levels), and substantial dependence remains among not-direct pairs. After conditioning, 14–44% of not-direct pairs remain detected, and 64–83% of those detected marginally survive, about as many as direct ones (Table 13).

RepresentationMarginal WAMarginal WTGiven parents WAGiven parents WT
Raw level0.6470.6220.5330.523
Scaled level0.6770.5760.5950.549
Raw change0.5840.6030.5410.556
Scaled change0.6200.6410.5730.644

Table 2. Expert-parent test on flow targets: AUROC within action (“WA”) and within target (“WT”), alone and conditioned on the target’s other expert parents. Raw states are as reported; scaled states are divided by the firm’s median total assets. Firm-clustered standard errors; uncertain pairs are in every conditioning set but not scored (115 direct, 319 not-direct pairs).

SpecificationDirect — AloneDirect — Given parentsDirect — KeptNot direct — AloneNot direct — Given parentsNot direct — Kept
Scaled level70.4%53.9%70%50.5%43.9%83%
Scaled change40.9%33.0%74%20.7%16.0%64%
Raw level73.9%56.5%76%50.5%41.7%78%
Raw change26.1%25.2%83%16.9%13.8%70%

Table 13. Expert-parent test on flow targets: share of pairs detected alone and given the target’s other expert parents, and the share of pairs detected alone that stay detected. A pair is detected if max⁡ℓ∣tℓ∣>2.49\max_{\ell}|t_{\ell}| > 2.49, a family-wise 5% over four lags, with firm-clustered standard errors. Direct: 115 pairs; not direct: 319.

A complementary nonlinear test produces the same pattern. We use the generalised covariance measure with gradient-boosted trees to residualise both the focal action and target against the expert parents. This relaxes the linearity assumption for the nuisance relationships, although it tests residual covariance rather than general conditional independence. The within-action AUROC still falls from 0.682 to 0.587 on scaled levels and from 0.609 to 0.568 on scaled changes (Table 12).

Conditioning setRaw levelRaw changeScaled levelScaled change
None (marginal)0.641 / 0.6020.611 / 0.5990.682 / 0.6050.609 / 0.647
Expert parents0.505 / 0.4690.562 / 0.5880.587 / 0.5330.568 / 0.622

Table 12. Generalised covariance measure on flow targets with nonlinear nuisance regressions: AUROC within action / within target, alone and conditioned on the expert parents. Gradient-boosted residualisation with three firm-grouped cross-fitting folds; uncertain pairs are in every conditioning set but not scored.

As an annotation-sensitivity analysis, we also test whether annotation disagreement has left relevant parents out of the conditioning sets. Since extra preceding variables preserve the graph-implied independence, though they may reduce power, disputed edges can remain candidate parents: a disputed not-direct edge may be a missing true parent, while a disputed direct edge that is truly not-direct is only an extra control. We therefore score only unanimous pairs and progressively add the disputed not-direct edges as parents, but discrimination is not restored (Table 15). Demeaning outcomes by firm and quarter does not restore it either (Table 16). Stock targets remain within 0.05 of chance within action, with or without the parents (Table 17).

Table 15: Adding the 35 disputed not-direct flow edges to the parent sets, none →\to all, on the 343 unanimously labelled flow pairs. Loss: fall in within-action AUROC from conditioning. Kept: share of not-direct detections that survive conditioning. Ratio: median direct ∣β^∣|\hat{\beta}| given the parents over its marginal value.

SpecificationLossKeptRatio
Scaled level0.094 →\to 0.06883% →\to 79%0.59 →\to 0.55
Scaled change0.047 →\to 0.05864% →\to 64%1.05 →\to 1.07
Raw level0.126 →\to 0.09481% →\to 78%0.56 →\to 0.53
Raw change0.052 →\to 0.05875% →\to 75%1.05 →\to 1.05

Table 15.

LevelChange
Conditioning setWAWTWAWT
None (marginal)0.6470.6490.5490.601
Expert parents0.5380.5790.5570.556

Table 16. Expert-parent test on flow targets with raw state outcomes demeaned by firm and quarter: AUROC within action (“WA”) and within target (“WT”), alone and conditioned on the target’s other expert parents. The states are exactly two-way demeaned after choosing the level/change representation; actions stay as filed. Firm-clustered standard errors; uncertain pairs are in every conditioning set but not scored.

LevelChange
Conditioning setWAWTWAWT
None (marginal)0.4520.3950.5160.597
Expert parents0.5320.5320.5160.516

Table 17. Expert-parent test on stock targets with raw states: AUROC within action (“WA”) and within target (“WT”), alone and conditioned on the target’s other expert parents. Firm-clustered standard errors; uncertain pairs are in every conditioning set but not scored (217 pairs).

How this relates to the data properties. The test exposes two obstacles tied to § 4. First, many direct pairs are invisible even marginally: on firm-scaled states, 30% are missed on levels and 59% on one-quarter changes (Table 13). Sparse support explains much of this. In the marginal test, direct-pair detection rises from 31% to 80% across support quartiles, and pairs below the support needed to detect a planted effect of 0.1 SD are detected at 37%, against 69% for the rest. Heavy tails hide more: removing the most extreme 1% of rows raises direct detection from 50% to 55%.

Second, not-direct dependence largely survives conditioning on the graph. On levels, the full parent set shrinks direct coefficients to 0.51 of their marginal size on raw states and 0.56 on scaled ones, mostly through same-quarter parents: with bundled actions (§ 4.4), co-filed parents absorb part of the focal action’s association. On scaled changes the parents leave the within-target AUROC unchanged (0.641 to 0.644), so there the obstacle is power, not bundling. Yet 64–83% of marginally detected not-direct associations survive across the four specifications, and 64% on scaled levels after demeaning by firm and quarter (Section J). Nor do they rest on the filing quarter, whose change can precede the filing: without it, 71% of the 73 not-direct pairs detected on scaled changes across all targets survive (63% of 148 with it; Section M). Label errors visible in the annotators’ disagreement explain few: Dawid–Skene expects 5.7 of the 148 to be truly direct (12.7 under a second model). A confounder explaining 1% of the residual variance of action and outcome would erase the median detection, as would one 11 times as strong as log firm size; the target’s own pre-filing change would not at 50 times (Section M). What remains could be omitted parents (a flow’s own lags are not conditioned on), latent common causes, or limits of the representation or tests.

What this means for the methods. The expert-parent test bypasses one of the central tasks of discovery: choosing which variables should be conditioned on. Its candidate parent sets come directly from the expert graph. The discovery methods receive no such sets and must recover directness from a panel with sparse interventions, persistent states, strongly related variables and bundled actions. That the linear and gradient-boosted tests separate direct from not-direct pairs only weakly even with graph-supplied parents identifies a difficulty that can contribute to the limited discrimination of the evaluated method families, especially within target (Table 1). This is not an impossibility result: the graph may be incomplete, and other procedures or assumptions may recover more.

A checklist for a new dataset

The diagnostics of § 4 and § 6 form a checklist for any panel of rare, dated interventions (Table 3, Section A). Before running discovery, measure heavy tails, persistence, heteroscedasticity, collinearity, the power each intervention’s support allows, and how often interventions are bundled; none of these needs labels or a method. Before reading a score, compute it under several nulls and report the excess over the strictest valid one, restrict shifts beyond the test window, and report a label-only baseline and the units the metric drops. A label-only baseline takes one line of code, and on our panel it beats the marginal test and every score in Table 1.

PropertyDiagnosticWarning signCase studyResponse
Heavy tailsKurtosis; sum-of-squares share of top 1%A few rows carry most of the variation27%–76% after winsorisingReport with and without the top rows; rank or robust tests
PersistenceWithin-firm AR(1), bias-corrected; ICCCorrected coefficient near one5 states at 0.83–0.93; 42%–64% of firms above one after correctionTest changes, not levels; give methods differenced states as well as levels; restrict shift nulls; beware Nickell bias
HeteroscedasticityAuxiliary R2R^2 on scaleA large auxiliary R2R^2, not a rejectionmedian 0.0180Cluster-robust inference; ignore the rejection rate
CollinearityComputable identities; R2R^2 on the other variablesR2R^2 close to oneup to 0.83Drop or combine collinear variables
Sparse interventionsPlanted-effect power against supportMany interventions below the required support40 of 71 below 80% at 0.1 SDReport power; pool or drop rare interventions
BundlingShare of firings shared with another interventionMost firings sharedmedian 89% of firings (§ 4.4)Merge or test bundled interventions jointly
Null choiceMetric under several nullsNull mean well above 0.50.501–0.556Report excess over the strictest valid null
Shift aliasingMetric against shift distanceShort shifts score near the observed value0.627–0.675 at one quarterRestrict shifts beyond the test window
Label structureLabel-only baseline; units dropped by the metricBaseline at or above the method0.860 from target base rates; 27 actions droppedReport both beside the score

Table 3. Diagnostics to run before discovery on a panel. The last three concern the evaluation.

Limitations and conclusion

The panel is balanced by selection, so firms that entered or left are absent. Actions are extracted by a language model, and an extraction error looks like an action without effect. The expert graph is partial and may contain errors. Labels are coarse in time, quarters hide the order of events within them, and binary indicators ignore the size of an action. The properties and the evaluation use one linear, additive estimator, so nonlinear or firm-varying effects are outside what it can see.

On this panel several of the conditions discovery relies on are measurably violated: extreme tails, 5 of 11 states close to random walks, strongly interrelated states and rare interventions. Against references calibrated for this panel (§ 6), the data carry information about which pairs are direct, yet standard discovery methods, in the configurations this panel size allows, do not beat pairwise cross-correlation within action or within target. Most direct pairs sit on actions too rare to detect a 0.1-SD effect even marginally, and most not-direct associations survive the expert parents, so the panel carries dependence the graph does not account for. The data properties that stress a method’s assumptions can be measured before it is run, with the checklist of § 8.

Checklist

Table Table 3 lists the diagnostics of § 8 in the order we would run them. The first six need neither labels nor a discovery method. Case-study values are reference points, not thresholds, and the warning signs are qualitative on purpose: a cut-off calibrated on one dataset would be false precision. Collinearity should be expected in any accounting panel.

Panel details

The 71 action types are subtypes of 10 8-K items: material agreements and their termination (1.01, 1.02), completed acquisitions and disposals (2.01), new direct financial obligations (2.03), exit and disposal plans (2.05), unregistered equity sales and changes to holders’ rights (3.02, 3.03), changes of directors and officers (5.02), amendments to articles or by-laws (5.03) and other events (8.01). The 11 states are current and noncurrent assets, current liabilities, equity, revenues, operating income, net income, the three cash-flow totals and other comprehensive income. Because only firms present in every quarter are kept, firms that entered, exited or were acquired during the window, and the filings that accompany those events, are not in the panel. The start-of-quarter stamp means sts_t is fixed before any action in quarter tt is filed, but a bad quarter can still prompt an action later in the same quarter.

Extraction audit. We audited the extraction against the filings. Of 30 extracted events, sampled to over-represent rare subtypes, 24 are supported by the 8-K, 4 are ambiguous and 2 carry the wrong subtype of the right item; none is invented. The sample is small: zero of 30 still allows an invention rate of up to about 10% at 95% confidence. Against SEC item metadata for 300 firms, about 1.6% of firm-quarters miss an event the taxonomy covers, most of the apparent gap being events that fit no subtype (item 8.01 above all). The embedding-assigned events were not audited separately, and the audit does not measure how often one event is split into two subtypes, which would inflate bundling (§ 4.4). A missed or invented event hits an action’s direct and not-direct targets alike, so it pulls the within-action AUROC towards 0.5 rather than inflating it. A wrong subtype is different: it moves an event to an action with other labels, and its effect on the AUROC has no fixed sign.

State construction. Filers report flows cumulatively over the fiscal year, so we take the discrete-quarter value where the filing reports one and otherwise difference the year-to-date figures; Q4 is the annual figure minus the nine-month one, and a revenue difference that comes out negative is left missing. Stocks are the balance at the quarter end. In an audit of 40 values against the XBRL filings, all sit in the calendar quarter of their period end, including fiscal years ending in September, October and January; 31 match the filed value and 9 could not be verified. None of the 31 is wrong, which bounds the error rate below about 9% at 95% confidence. Some reported values are exact zeros that are probably unreported rather than true zeros: noncurrent assets is zero in every quarter for many firms, and other comprehensive income is often exactly zero. We keep these as reported.

Annotation

Three annotators labelled each of the 781 action–state pairs independently as a direct cause, not a direct cause, or uncertain; the two grades of uncertainty they could choose are merged; one annotator left one pair blank. The annotators disagreed on 281 pairs. A financial expert resolved about 250 of them and set out the principles by which the rest were resolved. Uncertain pairs are held out of every metric. For the pairs judged direct, the annotators separately judged the shape of the effect, one-time (a change in a single quarter) or persistent (a change that carries into later quarters), and the shapes were adjudicated in the same way; 20 direct pairs were not given one. Labels and shapes are judgements about mechanism; none is estimated from the panel. The annotators followed a written rubric and did not see the panel data. All three agree on 442 of the scored pairs: 347 of the 429 not-direct and 95 of the 222 direct ones, so most direct labels are not unanimous and the positive class is the noisier one. Fleiss’ κ\kappa over the three classes is 0.60 on the 780 pairs all three labelled.

Nulls

Label permutation shuffles labels within action. Firm-block permutation reassigns whole action histories between firms; the stratified version reassigns them only within deciles of total action firings, so firm activity is preserved. Row permutation shuffles firm-quarters. Circular shifts rotate each action’s series within firm, with one offset per action or one per firm and action, either at any distance or at least five quarters from the identity. The restricted per-firm shift is primary (§ 6).

Shift distance. Shifting every action by the same distance (Figure 3b) gives 0.627–0.675 at one quarter, against 0.658 observed, 0.629 at two, averaged over the two directions, 0.517 at four, 0.480–0.507 at five to eight, and rises only to 0.554 at fifteen. An unrestricted shift null draws its distance at random and so includes the short shifts, which is why it sits above the restricted one.

Placebo. The placebo scores each action against the changes at h=−4,…,−1h=-4,\ldots,-1, the changes over quarters t−5t-5 to t−2t-2 for an action filed in quarter tt. Its null is the per-firm shift restricted to at least ten quarters, so that no shift moves the real effect into the four quarters before the action; it gives 0.568 against 0.540 (p=0.15p=0.15, 100 replicates).

Actions with one class

The 25 actions with no direct target and the 2 with no not-direct target drop out of the within-action AUROC. Calling a pair detected when any horizon is significant at a family-wise 5%, the median zero-direct action has 0.273 of its pairs flagged (0.083 under the primary null), association without directness, and the median all-direct action has 0.611 recovered (null 0.053).

Planted effects and support

We draw firm-quarters uniformly at random, add δ\delta standard deviations of the target’s one-quarter change at h=1h=1, and run the cell test at 5%, with 200 replicates per point (Figure 4a). Planted firm-quarters spread over almost as many firms as firings, while real actions cluster, and under clustered inference treated firms set power; the grid is therefore optimistic. Figure 4b shows the support behind each scored pair by label. In the top support quartile 83% of pairs are not direct, against at most 63% elsewhere. For 18 of the 71 actions the most frequent co-filed partner is a subtype of the same item, and the least-bundled action still co-occurs in 62% of its firings.

Figure 4. (a) Detection of a planted additive effect at 5% against firm-quarters treated; grey bars count real actions per support bin (right axis). (b) Support of the action behind each scored pair, by label.

Heteroscedasticity and accounting relations

A Breusch–Pagan test of the action-to-state residuals against firm size rejects in all 2,604 cells; with size entered as terciles, the median statistic is 66,896 for the classic test, 633 for Koenker’s and 70 with cluster-robust inference. Size explains a median 0.0082 of the squared residuals’ variance as terciles and 0.0180 as log size.

Without totals for assets, liabilities or the change in cash, three standard identities (total assets as the sum of its parts, the balance sheet, the cash-flow sum) cannot be computed; the first would be true by construction. Three relations can be. Current plus noncurrent assets less current liabilities and equity is implied noncurrent liabilities and should never be negative; it is in 0.7% of firm-quarters, which bounds one kind of extraction error. The change in equity less net income and other comprehensive income is within 1% of total assets in 56% of firm-quarters, and operating and net income are within 1% of total assets of each other in 69%.

Supervised probe

Cross-correlation is the strongest score in Table 1. To ask whether the panel’s other label-free statistics add to it, we train a probe that learns the map from 78 statistics of each pair to the labels, with each action held out in turn. It is neither a discovery method nor a ceiling on what the panel holds.

Results. The probe reaches 0.841 within action (90% interval 0.799–0.880), an excess of 0.143 over its null, against 0.824 for cross-correlation, which is one of its features and is selected over eight preprocessing choices on the labels (§ 5). Ranked within target, it reaches 0.681 (0.632–0.733), above every point estimate in Table 1 (0.643 for cross-correlation and 0.579 for the best discovery method), though cross-correlation lies within its interval. The other statistics may carry target-specific information that a single lagged correlation does not.

Features. For each action–state pair (A,S)(A,S), we construct 78 label-free features at lags ℓ∈{1,2,3,4}\ell\in\{1,2,3,4\}: the 39 statistics of Table 7, each on the raw panel and again on the firm-scaled panel, in which every state is divided by the firm’s median total assets before winsorisation (§ 5). Most use t^(Y∼X)\hat{t}(Y \sim X), the signed homoskedastic Wald statistic for the coefficient of XX in a marginal regression of YY on XX. We interpret this statistic as a studentized measure of association rather than as a calibrated hypothesis test. Applying it to state levels, one-period changes, cumulative displacements, and firm-demeaned outcomes captures persistent, transitory, and accumulated responses while separating pooled from within-firm variation.

FeatureCountDefinitionInterpretation
Level association4t^(Sit∼Ai,t−ℓ)\hat{t}(S_{it} \sim A_{i,t-\ell})Persistent association visible in the state level.
Change association4t^(Sit−Si,t−1∼Ai,t−ℓ)\hat{t}(S_{it}-S_{i,t-1} \sim A_{i,t-\ell})Transitory or flow-like association visible in one-period changes.
Cumulative displacement3t^(Sit−Si,t−ℓ∼Ai,t−ℓ), ℓ∈{2,3,4}\hat{t}(S_{it}-S_{i,t-\ell} \sim A_{i,t-\ell}),\ \ell\in\{2,3,4\}Association with displacement accumulated over multiple quarters. Lag one is omitted because it duplicates the one-period change.
Firm-demeaned level4t^(S~it∼A~i,t−ℓ)\hat{t}(\widetilde{S}_{it} \sim\widetilde{A}_{i,t-\ell})Within-firm level association after removing time-invariant firm differences.
Firm-demeaned change4t^(ΔS~it∼A~i,t−ℓ)\hat{t}(\Delta\widetilde{S}_{it} \sim\widetilde{A}_{i,t-\ell})Within-firm association with one-period changes.
Quarterly-slope mean tt-statistic4β^‾ℓ/{sd⁡(β^qℓ)/Qℓ}\overline{\widehat{\beta}}_{\ell}/\{\operatorname{sd}(\widehat{\beta}_{q\ell})/\sqrt{Q_{\ell}}\}Strength and consistency of the cross-sectional association across eligible quarters.
Slope dispersion4sd⁡(β^qℓ)\operatorname{sd}(\widehat{\beta}_{q\ell})Temporal heterogeneity in the quarter-specific association.
Slope coherence4∣β^‾ℓ∣/∣β^qℓ∣|\overline{\widehat{\beta}}_{\ell}|/|\widehat{\beta}_{q\ell}|Magnitude-weighted sign consistency across quarters.
Lagged cross-correlation4 (8)1∣Iℓ∣∑i∈Iℓ∣corr⁡t(Ai,t−ℓ,Sit)∣\frac{1}{|\mathcal{I}_{\ell}|}\sum_{i\in\mathcal{I}_{\ell}}|\operatorname{corr}_{t}(A_{i,t-\ell},S_{it})|Within-firm temporal alignment, averaged over firms in which both series vary.

Table 7. Label-free observational summaries used by the supervised probe. Here Z~it=Zit−Z‾i\widetilde{Z}_{it}=Z_{it}-\overline{Z}_{i}, QℓQ_{\ell} is the number of eligible quarters, and Iℓ\mathcal{I}_{\ell} is the set of firms contributing to the lag-ℓ\ell cross-correlation. Counts are per panel; the full set applies every row to the winsorised and the firm-scaled panel, and cross-correlation to levels and changes on each.

We supplement these regression summaries with measures of temporal stability and lagged alignment. Let βqℓ\beta_{q\ell} denote the cross-sectional slope from regressing SiqS_{iq} on Ai,q−ℓA_{i,q-\ell} across firms in quarter qq. Its distribution across quarters captures average strength, dispersion, and sign coherence. Lagged cross-correlations are computed within each eligible firm and then averaged across firms, on levels and on one-period changes, unscaled and firm-scaled; the firm-scaled change variant is the configuration of Table 1. Table 7 gives the full feature set.

Training and evaluation. We fit a gradient-boosted classifier with 300 trees, learning rate 0.05, maximum depth 3, and minimum leaf size 20. The primary evaluation uses leave-one-action-out cross-fitting. In each fold, all pairs involving one source action are withheld, the classifier is trained on pairs involving the remaining actions, and the held-out pairs are scored. Repeating this procedure over all actions produces one out-of-fold score for each of the 651 strictly labelled pairs.

We evaluate the probe at the action–state pair level using within-action and within-target AU-ROC. Its excess is taken against the same restricted per-firm shift null as the methods, with 40 replicates, and its interval treats the null mean as fixed. Each replicate shifts every action’s firings, recomputes the 78 features, including the four cross-correlation fits, and reruns the full leave-one-action-out cross-fitting, so the null probe is trained and scored exactly as the observed one.

Discovery-method protocol

Methods that cannot take missing values run on the panel with the incomplete state series interpolated linearly within firm; the two firms with an entirely unobserved series are dropped. A complete-case panel of the firms with no missing value is the robustness arm. Under firm scaling, states are additionally divided by the firm’s median total assets as in § 3; under either scale, they are winsorised at 1% and standardised. Firms are never concatenated into one series without a boundary: lagged rows that would pair one firm’s last quarters with the next firm’s first are dropped or masked. PCMCI+ and SVAR-RFCI run with conditioning sets of at most two variables, a feasibility limit, and binary actions are tested with a regression-based Wald test rather than partial correlation. Both are run at four significance levels. PCMCI+ edges are ranked by the strictest level at which they survive; SVAR-RFCI’s PAG marks are reported separately in § 18.§ 2. The event-triggered average scores only the action-to-state block, and no adapter reads the labels. Tuned scores are upper bounds.

Method configurations

Shared protocol. All methods run on the panel of 2,899 firms over 31 quarters. We search four panel strategies: pooled levels, firm effects (each state demeaned within firm), firm and quarter effects (two-way demeaning), and first-differenced states. We cross them with two state scales: raw dollars, and division by the firm’s median total assets as in § 3. Under both scales, states are winsorised at 1% and standardised to zero mean and unit variance. All transformations apply to the states only; binary action indicators are neither differenced, demeaned nor rescaled. This gives eight preprocessing combinations. The lag window is fixed at one to four quarters for every method. Methods that also estimate contemporaneous edges keep them in the fit, but only lags 1–4 are scored.

A configuration is scored by pair-level within-action AUROC: each method’s lagged edge scores are collapsed to (action, state) pairs by taking the strongest lag, uncertain pairs are excluded, and each action’s direct pairs are ranked against its not-direct pairs. Edge-pruning thresholds are never tuned, since the score ranks every emitted edge. The selected configuration is the one with the highest score on the full label set, so the reported numbers are upper bounds over the configurations searched.

Methods with no hyperparameters to tune. VAR Granger, longitudinal LiNGAM, cross-correlation and the event-triggered average have no hyperparameters beyond the lag window, which is fixed. Each is therefore run once under each of the eight preprocessing combinations, and the best is reported.

  • VAR Granger: a pooled VAR of every variable on lags 1–4 of all variables, fitted within firm so no lag crosses a firm boundary; an edge scores 1−p1-p for its coefficient.

  • Longitudinal LiNGAM: four lags, default settings.

  • Cross-correlation: the per-firm absolute lagged correlation at lags 1–4, averaged over the firms in which both series vary.

  • Event-triggered average: the mean change in the state after each firing, with no controls.

Methods with tuned hyperparameters. DYNOTEARS, NTS-NOTEARS and MiTCD search the panel strategy and state scale jointly with their own hyperparameters using TPE, with the same budget of 40 configurations each, steered by the same score.

  • DYNOTEARS: p=4p=4; λW\lambda_W and λA\lambda_A searched on a log scale over [10−5,10−1][10^{-5}, 10^{-1}]. The least-squares term is evaluated through the Gram matrix of the lagged design, which minimises the same objective as the reference implementation (causalnex) at lower computational cost.

  • NTS-NOTEARS: lag 4, 100 outer iterations, htol=10−5h_{\mathrm{tol}}=10^{-5}; λ1\lambda_1 and λ2\lambda_2 searched on a log scale over [10−5,10−1][10^{-5}, 10^{-1}]. Three seeds per configuration, scored by the median.

  • MiTCD: lag 4, 2,000 training iterations; learning rate [10−4,10−1][10^{-4}, 10^{-1}], λsparse∈[10−3,1]\lambda_{\mathrm{sparse}}\in[10^{-3},1], λridge∈[10−4,10−1]\lambda_{\mathrm{ridge}}\in[10^{-4},10^{-1}] and λrec∈[10−2,1]\lambda_{\mathrm{rec}}\in[10^{-2},1], all on a log scale. Three seeds per configuration, scored by the median. Every configuration runs at the full iteration budget: a shorter screening stage left low-learning-rate configurations unconverged, so its scores could not rank them.

MethodSearchedConfigurations
PCMCI+preprocessing only; α\alpha swept (4 points) to rank8 (32 runs)
SVAR-RFCIpreprocessing; α\alpha swept; PAG marks reported32 runs
DYNOTEARSpreprocessing, λW\lambda_W, λA\lambda_A40 (TPE)
NTS-NOTEARSpreprocessing, λ1\lambda_1, λ2\lambda_240 (TPE)
MiTCDpreprocessing, learning rate, three penalties40 (TPE)
Longitudinal LiNGAMpreprocessing only8
VAR Grangerpreprocessing only8
Cross-correlationpreprocessing only8
Event-triggered averagepreprocessing only8

Table 8. What each method searches. Preprocessing: four panel strategies (pooled levels, firm effects, firm and quarter effects, first differences) crossed with raw or firm-scaled states. Configurations: the number scored.

Constraint-based methods: computational restrictions. PCMCI+ and SVAR-RFCI run a conditional-independence test for every candidate link and conditioning set, which with 2,899 firms and 82 variables is computationally expensive. We therefore make three restrictions, all for computational reasons.

  1. Capped conditioning sets. PCMCI+ conditions on at most two variables in its skeleton phase, and on at most five parents of each of the two variables in its MCI phase. SVAR-RFCI conditions on at most two variables, tests at most one conditioning set of each size per link, and adds no parents of the earlier variable. The upstream defaults are unbounded. Small caps limit which confounders the tests can block, so these runs test less conditioning than the methods were designed for.

  1. One fixed configuration. Beyond the caps, each method is run with fixed settings and no hyperparameter search: the regression-based CI test (with a Wald variant for binary variables), a lag window of 1–4 (PCMCI+) or 0–4 (SVAR-RFCI), and, for SVAR-RFCI, its orientation phase enabled. Only the preprocessing combination varies.

  1. Four significance levels. We use α∈{0.01,0.1,0.5,0.9}\alpha\in\{0.01, 0.1, 0.5, 0.9\} under each preprocessing combination, for 32 runs per method. For PCMCI+, the four edge sets within a combination are pooled into a ranking: an edge scores 1−α∗1-\alpha^*, where α∗\alpha^* is the strictest level at which it is retained, and edges never retained rank last. This ranking has at most five distinct levels, so its AUROC is coarse. SVAR-RFCI instead returns PAG marks, which are retained and reported without converting them to a direct-edge score.

We test whether the caps drive the results by fixing the preprocessing variant selected at cap 2 and changing only the relevant conditioning cap. For PCMCI+, this is the first-differenced, winsorised variant; increasing max_conds_dim from 2 through 8 leaves both evaluation scores nearly flat, with overlapping 90% bootstrap intervals (Figure 5). Raising this cap alone therefore does not improve PCMCI+’s discrimination in this sensitivity analysis. The corresponding SVAR-RFCI sensitivity is reported in Table 11. These one-factor checks do not evaluate otherwise unrestricted runs: the remaining method settings stay fixed, and preprocessing is not reselected at each cap.

Sensitivity of PCMCI+ to the skeleton conditioning-set cap

Figure 5. PCMCI+ sensitivity to the skeleton conditioning-set cap. The first-differenced, winsorised variant selected at cap 2 is held fixed while max_conds_dim varies from 2 through 8. Points show within-action and within-target AUROC; error bars are 90% bootstrap intervals over actions. Neither score improves systematically as the cap increases.

Capα=0.01\alpha= 0.01α=0.1\alpha= 0.1α=0.5\alpha= 0.5α=0.9\alpha= 0.9
262 / 160 / 094 / 128 / 0172 / 50 / 0221 / 1 / 0
357 / 165 / 089 / 133 / 0171 / 51 / 0221 / 1 / 0
453 / 168 / 185 / 137 / 0164 / 58 / 0221 / 1 / 0
549 / 172 / 179 / 143 / 0160 / 62 / 0221 / 1 / 0

Table 11. SVAR-RFCI sensitivity to the global conditioning cap for the 222 expert-direct pairs. The pooled, winsorised variant selected at cap 2 is held fixed while max_p_global varies. Each cell reports the number marked bidirected / not adjacent / oriented at any lag 1–4; “oriented” combines →\to and ∘→\circ\to.

MethodAUROCPanel transformationScaleTuned hyperparameters
PCMCI+0.650First differencesRaw–
DYNOTEARS0.528Firm and quarter effectsRawλW=0.043\lambda_W = 0.043, λA=0.0026\lambda_A = 0.0026
NTS-NOTEARS0.575First differencesRawλ1=0.0011\lambda_1 = 0.0011, λ2=0.0013\lambda_2 = 0.0013
Long. LiNGAM0.700First differencesRaw–
VAR Granger0.620Firm and quarter effectsRaw–
MiTCD0.541Firm effectsRawLearning rate 6.2×10−46.2 \times10^{-4}; λsparse=0.21\lambda_{\mathrm{sparse}} = 0.21; λridge=0.0023\lambda_{\mathrm{ridge}} = 0.0023; λrec=0.14\lambda_{\mathrm{rec}} = 0.14
Cross-correlation0.824First differencesFirm-scaled–
Event-trig. avg.0.646Firm and quarter effectsFirm-scaled–

Table 9. Selected configurations behind Table 1. “Raw” means no firm scaling. Hyperparameters are shown only for methods for which they were searched.

SVAR-RFCI PAG output

SVAR-RFCI is evaluated on the marks it outputs rather than mapped into the direct-edge score of Table 1. Its output is almost entirely bidirected. In every preprocessing combination and significance level, every lagged adjacency it keeps is marked ↔\leftrightarrow, apart from at most eight →\rightarrow or ∘ ⁣→\circ\!\rightarrow marks in the whole PAG (Table 10). Loosening α\alpha only adds more: at α=0.9\alpha= 0.9, about 17,000–18,600 of the 26,896 possible lagged cells are bidirected, as are almost all labelled pairs.

Transformationα\alphaWhole PAG ↔\leftrightarrowWhole PAG →\toWhole PAG ∘ ⁣→\circ\!\toDirect (222) ↔\leftrightarrowDirect (222) noneNot-direct (429) ↔\leftrightarrowNot-direct (429) none
Winsorised
Pooled0.013,6532062160107322
0.16,0112194128186243
0.511,122001725034782
0.918,5760022114263
Firm effects0.012,651224817476352
0.14,7430081141153276
0.59,765001665633396
0.917,6730021934290
Firm and quarter effects0.012,600225316972356
0.14,6810084138151278
0.59,703001566633297
0.917,6480021934272
First-differenced0.012,416356415874355
0.14,4310098124157272
0.59,3980016656324105
0.917,3060022024254
Firm-scaled
Pooled0.013,5663262160129300
0.15,83300100122225204
0.511,072001695338346
0.918,4930021934272
Firm effects0.012,8264158164107322
0.15,0630093129208221
0.510,228001606236663
0.917,9360022024272
Firm and quarter effects0.012,7874161161107322
0.15,0080095127208221
0.510,179001705236465
0.917,9570021754281
First-differenced0.012,4994470152118311
0.14,62400112110208221
0.59,767001843837455
0.917,4710021934272

Table 10. SVAR-RFCI output by preprocessing, panel transformation and significance level. Left: lagged edge marks in the whole PAG (82 variables, lags 1–4; 26,896 possible cells). Right: labelled action–state pairs with a bidirected (↔\leftrightarrow) mark at any lag 1–4, or not adjacent at any lag. No labelled pair receives a →\to or ∘ ⁣→\circ\!\to mark except one not-direct pair at α=0.01\alpha= 0.01 under firm effects and under firm and quarter effects (winsorised). “Winsorised”: raw-dollar states trimmed 1% over all firm-quarters; “firm-scaled”: states divided by the firm’s median total assets and trimmed 1% within each quarter. Both are standardised.

A bidirected mark A↔YA \leftrightarrow Y asserts that neither variable is an ancestor of the other, so their dependence is attributed to a latent common cause. An expert-direct pair, by definition, is one in which the action is an ancestor of the state. In the reported cap-2 runs, SVAR-RFCI never recovers this ancestry: no direct pair receives a →\rightarrow or ∘ ⁣→\circ\!\rightarrow mark. Whenever it keeps a direct pair adjacent, it instead marks the pair bidirected, contradicting the label.

The bidirected marks also fail to distinguish direct from not-direct pairs. At α=0.5\alpha= 0.5 under winsorised pooled levels, they cover 77% of direct pairs and 81% of not-direct pairs, and the two shares remain close across settings. SVAR-RFCI therefore fails to recover direct edges not by abstaining, but by marking the action–state dependencies it keeps as bidirected.

Conditioning-cap sensitivity. We hold fixed the pooled, winsorised variant selected at cap 2 and vary only max_p_global from 2 to 5 (Table 11). Higher caps prune more adjacencies, but do not recover orientations for expert-direct pairs. Across all cap and significance-level combinations, every direct pair is either marked bidirected or absent, except for one pair that gains a ∘ ⁣→\circ\!\to mark at caps 4 and 5 under α=0.01\alpha= 0.01. At α=0.9\alpha= 0.9, 221 of the 222 direct pairs are bidirected and one is absent at every cap. A directed-edge adapter therefore remains at 0.5 AUROC at every cap.

The cap-2 choice alone therefore does not explain SVAR-RFCI’s result: increasing it does not recover directed separation. The sensitivity still holds the other computational restrictions fixed, including one conditioning set per size.

Expert-parent test

The expert-parent test of Section 7.1 uses the four representations in Table 2. Scaled states are divided by the firm’s median total assets and winsorised at 1% within each quarter, as in § 3; raw states are reported in dollars and winsorised at 1% over all firm-quarters. Change scores difference the winsorised states. Each specification uses one complete-case sample for all of its fits: 63,288 firm-quarters for levels and 60,233 for changes. Actions are binary throughout. The conditioning set holds the target’s other expert parents at every lag 1–4, uncertain parents included. Only the focal action at the lag being tested is excluded; its other lags stay in the set when it is a parent. For a stock, the set also contains its own lagged level. Uncertain pairs are not scored. Adding earlier variables to a set that already contains the parents cannot break the separation of a not-direct pair, so including uncertain parents is conservative. The detection cut-off, 2.49, is the Šidák level for a family-wise 5% over four lags. Standard errors are clustered by firm, with tt on G−1G - 1 degrees of freedom. Table 13 gives detection rates by class for flow targets, and Table 14 conditions on one timing class of parents at a time.

Table 14: Flow targets: conditioning on one timing class of the expert parent set at a time. Relative to the focal action at lag ℓ\ell, pre-source, same-time and post-source parents occur at lags >ℓ> \ell, =ℓ= \ell and <ℓ< \ell. Each cell gives the AUROC within action / within target and, in brackets, the median ratio of the direct pairs’ ∣β^∣|\hat{\beta}| to the marginal fit. Uncertain pairs are in every conditioning set but not scored.

ParentsScaled levelScaled changeRaw levelRaw change
Pre-source0.679 / 0.595 (0.87)0.611 / 0.626 (1.00)0.633 / 0.610 (0.84)0.571 / 0.595 (1.00)
Same-time0.636 / 0.548 (0.76)0.592 / 0.644 (1.03)0.554 / 0.552 (0.70)0.530 / 0.558 (1.01)
Post-source0.660 / 0.579 (0.87)0.622 / 0.639 (1.00)0.633 / 0.611 (0.82)0.560 / 0.579 (1.00)

Table 14.

Nonlinear conditional association. We repeat the test with the generalised covariance measure. The focal action lag and target are each regressed on the conditioning set using gradient-boosted trees (depth 4, 100 trees) with three firm-grouped cross-fitting folds. The score is the standardised mean product of their out-of-fold residuals, maximised over lags 1–4. The rows, pairs and conditioning sets are the same as in the linear test; only the residualisation changes. This tests residual covariance after flexible adjustment, not general conditional independence. It yields the same ordering (Table 12).

Under the full parent set, the median direct coefficient retains 0.51 of its marginal magnitude on raw levels and 0.56 on scaled levels, against 1.04 and 1.03 on raw and scaled changes. The corresponding median standard-error ratios are 0.91 on raw levels and 1.09 and 1.07 on raw and scaled changes. For not-direct pairs, coefficient ratios are 0.91 on raw levels and 0.97–0.99 on changes; standard-error ratios on changes are 1.00–1.01.

Parents the graph may be missing. Disagreement among annotators marks edges the graph may have wrong. We fix the evaluation set to the 343 unanimously labelled flow pairs (59 direct, 191 within-action comparisons) and add to the parent sets, at lags 1–4, the 35 not-direct flow edges with at least one direct or uncertain vote, in 100 random nested orders from none to all of them. Adding them never restores discrimination (Table 15): the level loss shrinks, the change loss stays at 0.05–0.06, direct coefficients barely move, and 64–79% of the not-direct detections still survive with all 35 added. The ranges over the 100 orders are within 0.05 of AUROC and 6 points of retention at every step.

Firm and quarter demeaning. We rerun the test with the states two-way demeaned before the fit and the actions left as filed (Table 16). Conditioning on the expert parents lowers the within-action AUROC on raw levels, while slightly raising it on raw changes; on both scaled representations, which the table does not show, it also lowers it. It again fails to separate the classes. On scaled levels, demeaning the state outcomes lowers detection in both classes, but 64% of the not-direct detections (32% to 24%) and 75% of the direct detections (45% to 37%) survive the parents.

Stock targets. Stock targets do not separate the classes even marginally under either score, and conditioning on the expert parents leaves them close to 0.5 (Table 17). Their conditioning sets also hold the stock’s own lagged level.

Size of the cell and pair tests

Table 18 reports the rejection rate of each inference convention under a true null, from 500 simulated cells per row, at a nominal 5%, so each rate carries a Monte Carlo standard error of about 0.01. Each simulated panel has 400 firms over 20 quarters, and each treated firm fires the action four times. The outcome is Gaussian, a firm effect plus independent noise, so the table measures size on well-behaved data; the panel’s heavy tails and exact zeros can change it. The treated count is the number of firms that fire the simulated action. The cell tt test with min⁡(G1,G0)−1\min(G_1, G_0) - 1 degrees of freedom, which we use, rejects at between 0.034 and 0.056 across the grid. Referencing the same statistic to G−1G - 1 degrees of freedom rejects at 0.074 with ten treated firms. The Bell–McCaffrey correction [6] and the restricted subcluster wild bootstrap, with weights drawn per firm-quarter rather than per firm [33], are about as well sized as our rule. The joint Wald test over four horizons is not: it rejects at 0.156 with ten treated firms and falls to 0.062 only at 80. We therefore flag the 87 pairs whose action fires in fewer than 80 firms; they remain in the ranking.

Treatedtt, G1−1G_1-1 dftt, G−1G-1 dfCR2-BMWild boot.Joint FF
100.0500.0740.0480.0440.156
200.0520.0740.0460.0480.084
300.0480.0560.0460.0420.070
400.0340.0500.0320.0320.098
800.0560.0600.0560.0500.062

Table 18. Rejection rates under the null at a nominal 5%, on simulated Gaussian outcomes.

Ordinary standard errors. On the panel itself, refitting the marginal test with ordinary instead of firm-clustered standard errors barely moves the median ∣t∣|t| but changes significance in 16% of cells. The overall rejection rate is unchanged, but detections move from direct to not-direct pairs (50% to 45% and 37% to 40%), and the AUROC of each pair’s smallest pp value falls from 0.640 to 0.587. Clustering corrects within-firm dependence and unequal variance together, so the two are not separated here.

Choice of pair score

We considered four pair scores: the smallest per-horizon pp value, the largest and the mean per-horizon ∣t∣|t|, and the joint Wald test. The rule fixed in advance was to take the score with the largest excess over its own restricted per-firm shift null. The rule was applied to the pairs on which the annotators agreed; Table 19 gives the same comparison on all scored pairs. The joint Wald test also has the largest excess there, and the four within-action AUROCs span 0.638 to 0.658, so no conclusion in the paper depends on the choice.

Pair scoreWithin-actionPooledShift nullExcess
Minimum pp0.6400.6010.5330.106
Maximum ∣t∣|t|0.6380.6040.5330.104
Mean ∣t∣|t|0.6440.6130.5370.107
Joint Wald*0.6580.6090.5380.120

Table 19. Pair scores on all scored pairs. The shift null is the restricted per-firm shift. *Score used throughout.

Sensitivity to the labels, the inference and confounding

The labels are expert judgements, so the headline measures how far the panel agrees with them. Here we ask how much that agreement depends on which judgements are trusted, on the inference, and on unobserved confounding. The score is the marginal test of § 6, ranked within action unless stated. The reference for a relabelled set is the 95th percentile of the restricted per-firm shift null under the final labels, 0.581; the panel nulls are not recomputed for each relabelling, except for the unanimous subset below, but the label-permutation null in Table 20 is.

LabelsPairsDirectComparisonsWithin-actionLabel perm. q95Within tgt.
Final labels (headline)6512228690.6580.5470.564
Unanimous pairs only442953400.6470.5890.516
No vote for the other class5941856520.6870.5550.592
Direct if ≥2\ge2 direct votes6512008610.6530.5450.559
Direct if ≥1\ge1 direct vote6512408830.6750.5460.581
Direct only if all 3 vote direct651956010.5660.5640.541
Annotator A1 alone5571385990.6680.5560.572
Annotator A2 alone5871817380.6490.5640.563
Annotator A3 alone6132168050.6550.5590.559
Uncertain counted direct7813521,3580.6430.5550.563
Uncertain counted not direct7812221,1840.6440.5470.589
Uncertain by vote majority7812671,2800.6580.5390.588

Table 20. The marginal test under alternative label sets. Within-action AUROC, the 95th percentile of a label-permutation null recomputed for each set, and the within-target AUROC. Uncertain pairs are left out except in the last block.

Which labels are trusted. On the 442 unanimously labelled pairs the AUROC is 0.647, an excess of 0.098 over the restricted shift null recomputed on that subset (p=0.007p = 0.007). The pair score was chosen on this subset (§ 21), but on all scored pairs the four candidates span only 0.638–0.658. Dropping the 57 pairs with a vote for the other class raises the AUROC to 0.687; those pairs alone form only 14 comparisons, too few to score. Each annotator’s labels alone give 0.649–0.668, and the three ways of scoring the uncertain pairs give 0.643–0.658 (Table 20). Only the strictest definition, which turns the 127 non-unanimous direct pairs into not-direct ones, falls to its label-permutation reference (0.566). Ranked within target, the unanimous subset scores 0.516; with only 95 direct pairs, it is too small to resolve the signal shared across a target’s actions (§ 6).

How many labels would have to be wrong. Under random flips, the mean AUROC falls to the original benchmark’s 95th-percentile reference, 0.581, only once 30% of not-direct labels are made direct, or 20% of each class are swapped; flipping direct labels to not direct never gets there up to 50%. The null can itself move when the class configuration changes, so these are breakdown points against the benchmark’s reference, not against a null recomputed for each hypothetical graph. An adversary that flips the most damaging label at each step needs 12 flips (1.8% of scored pairs, all not direct to direct), or 14 if it may flip only non-unanimous pairs. That is a worst case over arbitrary errors. A Dawid and Skene [15] model of the votes reproduces the final label for 98.2% of pairs (96.6% with uncertain votes as a third response). Restricted to the 17 pairs it considers plausibly mislabelled (10 under the second model), the adversary stops at 0.641 (0.639), and labels redrawn from its posteriors give a mean of 0.665, with 2.5% quantile 0.654 (0.671 and 0.660).

Inference and scaling. Clustering by firm and by quarter, which also absorbs shocks shared across firms within a quarter, leaves the AUROC of the smallest per-horizon pp (Section K) at 0.643 against 0.640, and moves the within-target AUROC from 0.569 to 0.599. The median standard error grows by a factor of 1.07, and detection falls more for not-direct pairs (37% to 24%) than for direct ones (50% to 43%). Firm scaling divides by a median taken over the whole window, after many of the actions. Taking it over the first eight quarters instead gives 0.625 (0.571 within target).

Timing within the filing quarter. States are stamped at the start of a quarter, so the change at h=1h = 1 spans the filing quarter and part of it can precede the filing: a bad January can prompt an 8-K in March. Dropping h=1h = 1 removes that ambiguity. On h=2,3,4h = 2, 3, 4 the marginal test gives 0.612 against a restricted shift null of 0.537, an excess of 0.076 against 0.120 on all four horizons (100 replicates), and 0.585 within target against 0.564. The h=2h = 2 change alone, with flows measured as st+2−sts_{t+2} - s_t so that nothing of the filing quarter enters, gives 0.570 against 0.520. Lagged cross-correlation computed on the marginal test’s panel, not the configuration selected for Table 1, moves from 0.705 to 0.692 within action, still 0.041 above its binding null (40 replicates), and from 0.669 to 0.585 within target.

For survival, the marginal and expert-parent fits share one complete-case sample, smaller than the headline one because each row needs the parents’ lags, and cover all targets, stocks included. A pair is detected if any horizon clears the Šidák level for four horizons, and survives if it stays detected given the expert parents at its peak horizon. Detection falls, as it must when the horizon holding the first effect is removed, from 46.8% to 35.1% of direct pairs and from 34.5% to 17.0% of not-direct ones, but survival does not: of the 73 not-direct pairs detected on h=2,3,4h = 2, 3, 4, 71% survive, against 63% of the 148 detected on all four (62% and 69% of direct pairs). On flow targets alone, the population of Table 13, not-direct survival rises from 67% (61 of 91) to 74% (34 of 46); the table’s 64% comes from the separate fits described at the start of Section J, on their own sample. The results are not driven by the horizon at which the order within a quarter is ambiguous.

Label error and confounding behind the not-direct detections. Of the 148 not-direct pairs the marginal test detects on the complete-case sample above, 11 carry any direct vote, and Dawid–Skene expects 5.7 of the 148 (12.7) to be truly direct. Dawid–Skene is a sensitivity model, not a measure of the graph’s accuracy: it infers errors from the pattern of disagreement, so a mistake all three annotators share is invisible to it. Label errors consistent with that disagreement explain few of the detections. For confounding we compute, at each detected pair’s strongest horizon, the robustness value of Cinelli and Hazlett [14]: the share of the residual variance of both the action and the outcome that a confounder must explain to erase the estimate. Its median is 0.013 for both classes: a confounder explaining 1.3% of the residual variance of both the action and the outcome would erase the median detection. With about 80,000 firm-quarters per cell the robustness value is close to ∣t∣/df|t|/\sqrt{\mathrm{df}}, so any ∣t∣|t| near 3 gives a value near 1%; Cinelli and Hazlett [14] read it against observed covariates, and so do we. Log firm size explains a median 0.27% of the action’s residual variance and 0.05% of the outcome’s. Erasing the median detected not-direct pair takes a confounder 11 times as strong as log firm size, 38 times as strong as the firm’s other actions and 33 times as strong as quarter shocks (16 and 42 for direct pairs), and the target’s own change before the filing erases neither within 50 times its strength. Relative to these observed benchmarks, erasing the median association takes a confounder far stronger, in its partial associations with both action and outcome, than any covariate we observe. By the same measure, the not-direct detections are as robust as the direct ones the graph accepts.

Per-state tables

Tables 21 and 22 give, per state, the persistence of § 4.2 and the tails of § 4.1.

Stateρ^\hat{\rho}IQRρ^K\hat{\rho}_{K}ρ^K>1\hat{\rho}_{K} > 1ICCFirms
Equity0.93[0.84, 0.99]1.0564%0.782,901
Noncurrent assets0.91[0.82, 0.97]1.0359%0.262,264
Current assets0.87[0.74, 0.95]0.9947%0.372,901
Current liabilities0.84[0.68, 0.94]0.9642%0.802,901
Revenues0.83[0.58, 0.96]0.9542%0.702,741
Operating income0.43[0.14, 0.69]0.518%0.702,888
Net income0.28[0.05, 0.57]0.344%0.622,893
Operating cash flow0.14[-0.09, 0.43]0.193%0.642,901
Investing cash flow0.05[-0.07, 0.25]0.091%0.072,815
Financing cash flow-0.01[-0.12, 0.16]0.031%0.382,890
Other comprehensive income-0.05[-0.21, 0.12]-0.030%0.022,339

Table 21. Within-firm persistence of the scaled levels. ρ^\hat{\rho} is the median per-firm OLS coefficient and IQR its interquartile range across firms; ρ^K\hat{\rho}_{K} is the median after Kendall’s correction, with the share of firms above one. A firm is dropped from a state’s row if that state is observed in fewer than 12 of its quarters.

StateExcess kurtosisSkewTop 1%
rawwinsorisedSS share
Noncurrent assets9,76945116.259%
Financing cash flow77,082867.665%
Current assets10,009765.627%
Current liabilities9,495687.976%
Net income81,35866-7.674%
Equity4,04360-7.373%
Operating income83,68855-6.969%
Investing cash flow37,86347-4.248%
Operating cash flow47,96838-5.457%
Revenues15,270193.328%
Other comprehensive income50,84718-0.737%

Table 22. Distribution of the scaled levels, as reported and after 1% winsorisation within quarter. Skew and the share of the sum of squares in the top 1% of rows are after winsorisation.

References

  1. [1]Constantin Ahlmann-Eltze, Wolfgang Huber, and Simon Anders. Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines. Nature Methods, 22:1657–1661, 2025.
  2. [2]Joshua D Angrist and Jörn-Steffen Pischke. Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton University Press, 2009.
  3. [3]Charles K Assaad, Emilie Devijver, and Eric Gaussier. Survey and evaluation of causal discovery methods for time series. Journal of Artificial Intelligence Research, 73:767–819, 2022.DOI
  4. [4]Carles Balsells-Rodas, Xavier Sumba, Tanmayee Narendra, Ruibo Tu, Gabriele Schweikert, Hedvig Kjellström, and Yingzhen Li. Causal discovery from conditionally stationary time series. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of PMLR, pages 2715–2741, 2025.
  5. [5]Taiyu Ban, Lyuzhou Chen, Derui Lyu, Xiangyu Wang, Qinrui Zhu, Qiang Tu, and Huanhuan Chen. Integrating large language model for improved causal discovery. IEEE Transactions on Artificial Intelligence, 6(11):3030–3042, 2025. doi: 10.1109/TAI.2025.3560927.DOI
  6. [6]Robert M. Bell and Daniel F. McCaffrey. Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 28(2):169–181, 2002.
  7. [7]Marianne Bertrand, Esther Duflo, and Sendhil Mullainathan. How much should we trust differences-in-differences estimates? Quarterly Journal of Economics, 119(1):249–275, 2004.
  8. [8]Brantly Callaway and Pedro H. C. Sant’Anna. Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2):200–230, 2021.
  9. [9]A. Colin Cameron and Douglas L. Miller. A practitioner’s guide to cluster-robust inference. Journal of Human Resources, 50(2):317–372, 2015.
  10. [10]Jiawei Chen and Chunhui Zhao. Addressing information asymmetry: Deep temporal causality discovery for mixed time series. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(7):5723–5741, 2025. doi: 10.1109/TPAMI.2025.3553957.DOI
  11. [11]Yuxiao Cheng, Ziqian Wang, Tingxiong Xiao, Qin Zhong, Jinli Suo, and Kunlun He. CausalTime: Realistically generated time-series for benchmarking of causal discovery. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.01753
  12. [12]Mathieu Chevalley, Yusuf H. Roohani, Arash Mehrjou, Jure Leskovec, and Patrick Schwab. A large-scale benchmark for network inference from single-cell perturbation data. Communications Biology, 8:412, 2025. doi: 10.1038/s42003-025-07764-y.DOI
  13. [13]David Maxwell Chickering. Optimal structure identification with greedy search. Journal of Machine Learning Research, 3:507–554, 2002.
  14. [14]Carlos Cinelli and Chad Hazlett. Making sense of sensitivity: Extending omitted variable bias. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 82(1):39–67, 2020.
  15. [15]A. Philip Dawid and Allan M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20–28, 1979.
  16. [16]Sofia Faltenbacher, Jonas Wahl, Rebecca Herman, and Jakob Runge. How PC-based methods err: Towards better reporting of assumption violations and small sample errors. In Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, volume 337 of PMLR, pages 1498–1519, 2026.arxiv.org/abs/2502.14719
  17. [17]Muhammad Hasan Ferdous, Emam Hossain, and Md Osman Gani. TimeGraph: Synthetic benchmark datasets for robust time-series causal discovery. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2025. doi: 10.1145/3711896.3737439.DOI
  18. [18]Shanyun Gao, Raghavendra Addanki, Tong Yu, Ryan Rossi, and Murat Kocaoglu. Causal discovery in semi-stationary time series. In Advances in Neural Information Processing Systems, volume 36, pages 46624–46657, 2023.arxiv.org/abs/2407.07291
  19. [19]Amanda Gentzel, Dan Garant, and David Jensen. The case for evaluating causal models using interventional measures and empirical data. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1910.05387
  20. [20]Andreas Gerhardus and Jakob Runge. High-recall causal discovery for autocorrelated time series with latent confounders. In Advances in Neural Information Processing Systems, volume 33, pages 12615–12625, 2020.arxiv.org/abs/2007.01884
  21. [21]Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in Genetics, 10:524, 2019.DOI
  22. [22]Clive W. J. Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica, 37(3):424–438, 1969.
  23. [23]Benjamin Herdeanu, Juan Nathaniel, Carla Roesch, Jatan Buch, Gregor Ramien, Johannes Haux, and Pierre Gentine. CausalDynamics: A large-scale benchmark for structural discovery of dynamical causal models. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 38, 2025.arxiv.org/abs/2505.16620
  24. [24]Biwei Huang, Kun Zhang, Jiji Zhang, Joseph Ramsey, Ruben Sanchez-Romero, Clark Glymour, and Bernhard Schölkopf. Causal discovery from heterogeneous/nonstationary data. Journal of Machine Learning Research, 21(89):1–53, 2020.arxiv.org/abs/1903.01672
  25. [25]Aapo Hyvärinen, Kun Zhang, Shohei Shimizu, and Patrik O. Hoyer. Estimation of a structural vector autoregression model using non-Gaussianity. Journal of Machine Learning Research, 11:1709–1731, 2010.DOI
  26. [26]Thomas Jiralerspong, Xiaoyin Chen, Yash More, Vedant Shah, and Yoshua Bengio. Efficient causal graph discovery using large language models. In ICLR 2024 Workshop: How Far Are We From AGI, 2024.arxiv.org/abs/2402.01207
  27. [27]Kento Kadowaki, Shohei Shimizu, and Takashi Washio. Estimation of causal structures in longitudinal data using non-Gaussianity. In IEEE International Workshop on Machine Learning for Signal Processing, 2013.DOI
  28. [28]Rickard K. A. Karlsson and Jesse H. Krijthe. Detecting hidden confounding in observational data using multiple environments. In Advances in Neural Information Processing Systems, volume 36, pages 44280–44309, 2023.arxiv.org/abs/2205.13935
  29. [29]Maurice G. Kendall. Note on bias in the estimation of autocorrelation. Biometrika, 41(3/4):403–404, 1954.DOI
  30. [31]Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-based neural DAG learning. In International Conference on Learning Representations, 2020.arxiv.org/abs/1906.02226
  31. [32]A. Craig MacKinlay. Event studies in economics and finance. Journal of Economic Literature, 35(1):13–39, 1997.
  32. [33]James G. MacKinnon and Matthew D. Webb. The wild bootstrap for few (treated) clusters. The Econometrics Journal, 21(2):114–135, 2018.DOI
  33. [34]Daniel Malinsky and Peter Spirtes. Causal structure learning from multivariate time series in settings with unmeasured confounding. In Proceedings of the 2018 ACM SIGKDD Workshop on Causal Discovery, volume 92 of Proceedings of Machine Learning Research, pages 23–47, 2018.
  34. [35]Alejandro Michel, Abhinav Arun, Bhaskarjit Sarmah, and Stefano Pasquali. FinCARE: Financial causal analysis with reasoning and evidence. arXiv preprint arXiv:2510.20221, 2025.arxiv.org/abs/2510.20221
  35. [36]Henry E. Miller, Gabriel M. Mejia, Francis J. A. Leblanc, Brendan Swain, Bo Wang, and Lucas Paulo de Lima Camillo. Deep learning perturbation models can outperform baselines on calibrated metrics. Nature Biotechnology, 2026. doi: 10.1038/s41587-026-03307-w.DOI
  36. [37]Søren Wengel Mogensen, Karin Rathsman, and Per Nilsson. Causal discovery in a complex industrial system: A time series benchmark. In Proceedings of the Third Conference on Causal Learning and Reasoning, volume 236 of PMLR, pages 1218–1236, 2024.arxiv.org/abs/2310.18654
  37. [38]Joris M Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Distinguishing cause from effect using observational data: Methods and benchmarks. Journal of Machine Learning Research, 17(32):1–102, 2016.
  38. [39]Roxana Pamfil, Nisara Sriwattanaworachai, Shaan Desai, Philip Pilgerstorfer, Konstantinos Georgatzis, Paul Beaumont, and Bryon Aragam. DYNOTEARS: Structure learning from time-series data. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, volume 108 of PMLR, pages 1595–1605, 2020.arxiv.org/abs/2002.00498
  39. [40]Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2nd edition, 2009.
  40. [41]Mitchell A. Petersen. Estimating standard errors in finance panel data sets: Comparing approaches. Review of Financial Studies, 22(1):435–480, 2009.
  41. [42]Alexander G. Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the simulated DAG! causal discovery benchmarks may be easy to game. In Advances in Neural Information Processing Systems, volume 34, 2021.
  42. [43]Amartya Roy, N. Devharish, Shreya Ganguly, and Kripabandhu Ghosh. Causal-LLM: A unified one-shot framework for prompt- and data-driven causal graph discovery. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 8259–8279, 2025.DOI
  43. [44]Jakob Runge. Discovering contemporaneous and lagged causal relations in autocorrelated nonlinear time series datasets. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence, volume 124 of PMLR, pages 1388–1397, 2020.arxiv.org/abs/2003.03685
  44. [45]Jakob Runge, Peer Nowack, Marlene Kretschmer, Seth Flaxman, and Dino Sejdinovic. Detecting and quantifying causal associations in large nonlinear time series datasets. Science Advances, 5(11):eaau4996, 2019.DOI
  45. [46]Jakob Runge, Xavier-Andoni Tibau, Matthias Bruhns, Jordi Muñoz-Marí, and Gustau Camps-Valls. The causality for climate competition. In Proceedings of the NeurIPS 2019 Competition and Demonstration Track, volume 123 of PMLR, pages 110–120, 2020.
  46. [47]Karen Sachs, Omar Perez, Dana Pe’er, Douglas A. Lauffenburger, and Garry P Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308(5721):523–529, 2005.
  47. [48]Martijn J. Schuemie, Patrick B. Ryan, William DuMouchel, Marc A. Suchard, and David Madigan. Interpreting observational studies: Why empirical calibration is needed to correct p-values. Statistics in Medicine, 33(2):209–218, 2014.
  48. [51]Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, Prediction, and Search. MIT Press, 2nd edition, 2000.
  49. [52]Gideon Stein, Maha Shadaydeh, Jan Blunk, Niklas Penzel, and Joachim Denzler. CausalRivers: Scaling up benchmarking of causal discovery for real-world time-series. In International Conference on Learning Representations, 2025.arxiv.org/abs/2503.17452
  50. [53]Liyang Sun and Sarah Abraham. Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics, 225(2):175–199, 2021.
  51. [54]Xiangyu Sun, Oliver Schulte, Guiliang Liu, and Pascal Poupart. NTS-NOTEARS: Learning non-parametric DBNs with prior knowledge. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 1942–1964, 2023.
  52. [55]Caroline Uhler, Garvesh Raskutti, Peter Bühlmann, and Bin Yu. Geometry of the faithfulness assumption in causal inference. Annals of Statistics, 41(2):436–463, 2013.DOI
  53. [56]Michael Wieck-Sosa, Michel F. C. Haddad, and Aaditya Ramdas. The dynamic generalized covariance measure for conditional independence testing with nonstationary time series. arXiv preprint arXiv:2504.21647, 2025.
  54. [57]Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. DAGs with NO TEARS: Continuous optimization for structure learning. Advances in Neural Information Processing Systems, 31, 2018.arxiv.org/abs/1803.01422

Paper details

vlnc:W_kk5phwkk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents