Patching the Blind Spots: Reversible LLM Rule Overlay for Frozen Fraud Detectors

Introduction

Gradient-boosted trees dominate tabular fraud classification ([12, 1]), trained once on historical labelled transactions, then degrade under the temporal distribution shift ubiquitous in financial data ([14, 6, 2]): new fraud patterns emerge, the population shifts, and discrimination quietly erodes with no visible retraining signal.

In production, such a detector also has loopholes: fraud patterns it scores too low, which persist after deployment. Closing them is constrained in two ways in our setting. The model’s scores and cut-offs drive downstream decisions, so changing the model means a redeployment. Its training data are selection-biased: a rule engine and a screening model filter transactions before they are recorded, so training sees approved traffic only, while the model serves the full population (a small unbiased cohort is kept for evaluation; § 3). Patterns that rarely pass the filters are under-represented, and the usual fix—continued training on a fresh window—fits the same bias: in our experiments it regresses on the unbiased cohort (§ 4). The practical question is how to find a frozen model’s loopholes and improve how it generalizes, without modifying it.

Key idea: LLMs reason about the frozen GBM’s blind spots and patch them proactively. Instead of updating the model, we let large language models reason over its diagnostics (SHAP attributions and a digest of its trees) and propose human-readable rules for the fraud it misses; the adaptation window is used only to screen and calibrate them, so no model parameter changes. Generation is steered through the model’s weak dimensions: each rule’s primary anchor must be a feature the model underweights by SHAP. The rules are calibrated into a band-conditioned log-lift overlay on the frozen base—additive in logit space, hence reversible—and mapped back to the production score scale, so the overlay runs at the existing cut-off. Every amendment is a human-auditable predicate, in line with calls for interpretable high-stakes models ([16]).

Prior work uses LLMs on tabular data either as the predictor itself, on serialized rows ([9, 5]), or to engineer features and rules for a model that is then trained or retrained: CAAFE generates features from dataset descriptions ([10]), FeatLLM turns LLM-written rules into features for few-shot learning [8], and OCTree refines feature-generation rules with decision-tree feedback [13]. Closest to us, DeLTa has an LLM rewrite a tree ensemble’s rules into one that corrects the ensemble’s outputs (Ye et al., 2025). These methods target accuracy on public benchmarks, where the predictor can be rebuilt freely. Our setting differs in three ways: the model is deployed and frozen, so the rules act as a reversible overlay at its existing cut-off; generation is anchored on the features the model underweights rather than on refining its existing tree rules; and the training data are selection-biased, so the gain must hold on a population the model was not trained on.

As the rule proposer, a single LLM produces high-variance outcomes not predictable from in-distribution signal: the backbone that performs best on the real-world generalization test is among the worst on the BAF variants, and vice versa § 4. We therefore prompt three architecturally diverse backbones [3] (Wei et al. 2022; Qwen Team 2026; Gemma Team 2026) across independent shots and average all retained shots in logit space [4], with no dataset-specific selection.

Contributions

1. Patching a frozen, deployed model. LLM rules steered through the detector’s SHAP-underweighted dimensions, calibrated into a reversible overlay that runs at the production cut-off with every parameter fixed before deployment.

2. Evidence under selection bias and designed shift. On an unbiased cohort, continued training regresses while the overlay gains recall at equal precision; on seven bias-shift variants it raises precision at equal recall on every variant. Pooling three LLMs bounds the high, dataset-dependent variance of single-LLM benefit without oracle selection § 4.

Method

our framework adds a calibrated, LLM-generated rule overlay to a frozen gradient-boosted classifier. Figure 1 summarizes the pipeline, from a model-derived prompt context through LLM proposal, empirical screening, band-conditioned calibration, and a reversible overlay (top), and how the overlay is deployed beside the production score (bottom).

Problem Setup and Frozen-Model Constraint

Let $x \in\mathcal{X}$ be a transaction feature vector and $y \in\{0,1\}$ the fraud label. A tree-ensemble base scorer $f$ (constructed in § 3) is trained once on historical data and frozen thereafter. Its production score $s(x) \in[0,1000]$ is alerted at a cut-off $t$, i.e. when $s(x) \ge t$. Our framework never modifies $f$, $s$, or $t$. Because $f$ is never retrained, the generalization-gap failure mode described above is avoided by construction § 4. We call the labeled slice used to generate, screen, and calibrate the overlay the adaptation window $A$; it is not a held-out evaluation set. Every fitted quantity—rules, lifts, and the lookup tables below—comes from $A$ and is frozen: no out-of-time (OOT) labels or score distributions are used at any stage. The overlay’s base is $p(x) = F_A(s(x))$, the score’s percentile under $A$’s empirical distribution (a fixed lookup table).

Model-Aware LLM Rule Proposal

For each dataset the framework runs 3 LLMs × 10 independent generation shots: each shot is a single generation pass with the same prompt (Appendix § 8), yielding a noisy candidate dictionary of diverse predicates.

The prompt has three parts: (i) a domain primer describing the fraud-detection context and feature encodings; (ii) a digest of the frozen model’s trees—per-feature split counts and gain, frequent feature sets on high-gain paths, and low-value leaves; and (iii) a SHAP attribution gap on the adaptation window: features are split at the median of their mean SHAP over fraud transactions into those the model relies on and those it underweights, with decile (or category) fraud-lift tables for the underweighted ones. The LLM sees these summaries, never the model itself. It reasons about how fraud would present in the underweighted dimensions—behaviours a detector anchored on the first group would not flag—and every candidate’s primary anchor must come from that set; strongly-modeled features may appear only as secondary filters narrowing a pattern to a behavioural archetype. Each candidate must cite supporting lift from the tables. Candidates are conjunctions of threshold conditions (feature $\circ\theta$), $\circ\in\{<,\le,>,\ge,=\}$, or missing / non-missing tests, joined by $\land$ (no disjunctions), with thresholds given as training-set percentiles.

Why an LLM rather than a rule miner? The adaptation window’s per-band strata are small, so a tree learner or itemset miner fits only patterns already present, while the LLM proposes archetypes from domain priors; its output is also a named hypothesis with a rationale (§ 5.2). We do not compare empirically against rule-mining baselines and flag this as the primary gap. The three backbones (exact identifiers in § 3) span chain-of-thought reasoning, sliding-window attention, and sparse mixture-of-experts; no single one wins uniformly across datasets (§ 4), motivating the all-source ensemble.

Empirical Support Screening

Predicates are compiled with their percentile thresholds taken from the training data; any that fail to parse or compile are dropped. On the adaptation window, a predicate is kept if it fires, its fraud rate exceeds the window’s base rate (lift $> 1$), and it is not a duplicate of an already kept predicate (canonical form); on BAF, predicates built only from strongly-correlated features are also dropped. It then enters the pool $\mathcal{K}_b$ of every band $b$ in whose score zone $Z_b$ (§ 2.4) it fires on at least three transactions with lift $> 1$. Shots left with no rule in any band are discarded (the compilable-shot filter); retained shot counts are reported in Section 3. Screening checks empirical support only; it does not prove semantic correctness or guarantee out-of-time transfer.

Band-Conditioned Log-Lift Residual Overlay

We operate at a set of production cut-offs $t_b$ (alert-rate bands $b$). For each band the framework fits an independent rule pool $\mathcal{K}_b$ and lift set on the adaptation window restricted to band $b$’s score zone $Z_b = \{x \in A : s(x) \ge t_b\}$. Let $R_k(x) \in\{0,1\}$ indicate whether rule $k$ fires on $x$. For a rule $k \in\mathcal{K}_b$, its calibrated fraud lift is the band-conditioned precision ratio

estimated by a decile-standardized average: $Z_b$ is split into base-score deciles, and the within-decile precision ratios are averaged weighted by rule-firing support. This controls for score-dependent base rates and targets—rather than exactly equals—(1). The overlay for cut-off $t_b$ applies $\mathcal{K}_b$ to every transaction:

where $\sigma(\cdot)$ is the sigmoid and $p(\mathbf{x}) = F_A(s(\mathbf{x}))$ (§ 2.1). A firing rule with $\operatorname{lift}_k > 1$ raises the score, one below 1 lowers it (log is natural, matching the logit scale).

Per-band calibration is load-bearing: estimated globally, a rule’s coefficient is diluted by transactions outside its active zone, and D-Industrial avg$\Delta$R@P falls from +2.26 to $-2.41\%$ (Appendix § 9).

Multi-Shot Averaging, Scaled Score, and Evaluation

Each retained shot $j$ (across all three LLMs) produces $p^*_{j,b}(\mathbf{x})$ via Eq. (2). Stochastic generation produces heterogeneous candidate pools across shots; the ensemble combines all retained shots $S$ by uniform averaging in logit space, $z_b(\mathbf{x}) = |S|^{-1}\sum_{j\in S}\operatorname{logit}(p^*_{j,b}(\mathbf{x}))$. Every shot receives equal weight and the same combination is applied to all datasets; logit-space averaging matches the additive parameterization of Eq. (2) and is a standard variance-reduction strategy (Dietterich, [4]).

Scaled overlay score and deployment. To run at the existing cut-off, $z_b$ is mapped back to the production scale by a cut-off table fitted on $A$: $\tau_b(t)$ is the value at which $z_b$ alerts as many transactions of $A$ as $s \ge t$, for every $t$ on a 0.1-point grid, and $\tilde{s}_b(\mathbf{x}) = \max\{t : z_b(\mathbf{x}) \ge\tau_b(t)\}$.

The overlay alerts when $\tilde{s}_b(\mathbf{x}) \ge t_b$, the production cut-off (Figure 1, bottom). Serving needs only $F_A$, the rules with their lifts, and $t_b$; nothing is refitted on new traffic, and with the rules switched off $\tilde{s}_b = s$.

Evaluation metric. On OOT data we compare production ($s \ge t$) and the overlay ($\tilde{s}_b \ge t$) through their precision–recall points over the same threshold grid. Let $P_b$, $R_b$ be production’s precision and recall at its cut-off $t_b$. $\Delta$R@P is the overlay’s highest recall at any threshold with precision $\ge P_b$, minus production’s own highest recall at precision $\ge P_b$; $\Delta$P@R is defined symmetrically at recall $\ge R_b$. Subtracting production’s own best removes what it would gain by merely moving its threshold; no interpolation is used, and a level the overlay never reaches counts as recall (precision) zero. We report averages over a split’s cut-offs, avg$\Delta$R@P and avg$\Delta$P@R, as absolute differences in %; the precision and recall levels they average over are given in § 3, and every operating point in Appendix § 11. This pair is the sole comparison metric; we do not use extra-TP counts or PR-AUC/AUC as comparison axes.

Experimental Setup

Datasets: generalization and robustness to variants. We evaluate on two datasets that test the framework’s two target capabilities (Table 1): generalizing to an unseen population, and staying robust across designed distribution-shift variants. D-Industrial detects unauthorized-transaction fraud at a real-world e-commerce platform, trained on sampled data: training covers approved traffic only, richer in fraud signal but selection-biased, while evaluation uses a small unbiased cohort approved regardless of score (§ 1). This is where continued training generalizes worst. BAF (Jesus et al., [11]) is a public bank-account benchmark with seven OOT splits of designed bias-shift variants and no train/evaluation population mismatch: gradient adaptation is well-posed there, so it tests whether the overlay stays robust when retraining is a viable alternative.

Dataset	Shift type	#feat	#trees (LGB / RF)	Blend $\alpha$	Split design
D-Industrial	selection bias; unbiased eval $\ne$ biased train	500	2,219 / 500	0.75	Temporal, six months: train Oct–Nov 2025 (biased, approved-only); validate Dec 2025 and evaluate Jan 1–Mar 7, 2026, both from the unbiased cohort
BAF	public; designed bias-shift variants	30	892 / 200	1.00	Temporal: train months 0–4; evaluate held-out months 5–7 (OOT-1, OOT-2) plus five designed bias-shift variants (Var-I–V)

Baselines. Frozen base is the scorer under test, never modified; all $\Delta$ metrics are relative to it. It is itself ensemble-tuned: a rank blend $\alpha\operatorname{rank}(\mathrm{LGB})+(1-\alpha)\operatorname{rank}(\mathrm{RF})$ (Ke et al., [12]), ranks taken against the adaptation window, with $\alpha$ swept over a grid and kept at the value maximizing validation PR-AUC (Table 1), so overlay gains are measured over an already ensemble-tuned baseline. CT-100 adds 100 boosting rounds on the adaptation window (a straightforward continued-training baseline). CT-Tuned applies one boosting round at a lower learning rate (lr=0.03), a single round-count/learning-rate sensitivity point evaluated on D-Industrial only. Retrain fully retrains on the combined historical-plus-adaptation data. Each is scored on the same scale and evaluated like the overlay, with its cut-off table fitted on the adaptation window.

Operating cut-offs. All methods are evaluated at the production cut-offs $t_b$ on the 0–1000 score scale: D-Industrial $\{998,\ldots,993\}$ (6) and BAF $\{999,995,990,985,980,975,970\}$ (7). At these cut-offs production’s OOT precision and recall span 30.3–56.5% and 23.5–62.9% on D-Industrial, and 10.1–56.2% and 3.1–50.1% across the BAF splits; avg$\Delta$R@P and avg$\Delta$P@R average over these precision and recall levels respectively.

LLM backbones and active shot counts. Three architecturally diverse open-weight backbones each generate 10 independent shots per dataset: LLM-CoT = DeepSeek-R1-Distill-Qwen-32B-4bit (DeepSeek-AI, [3]) (chain-of-thought); LLM-SlW = Gemma4-31B-it-8bit (Gemma Team, [7]) (sliding-window attention); LLM-MoE = Qwen3.5-35B-A3B-4bit (Qwen Team, [15]) (mixture-of-experts). Shots left with no rule after screening are discarded; active counts after filtering (LLM-CoT / LLM-SlW / LLM-MoE) are D-Industrial 9 / 8 / 8 and BAF 10 / 6 / 9 (19/14/17 total). The all-3 ensemble averages uniformly over all retained shots. Generation ran locally on one Apple M4 Max workstation (128 GB unified memory); no hosted API was used, so transaction data never leaves the environment.

Results

Continued Training vs. the Overlay

We compare the all-3 ensemble overlay with three continued-training variants—CT-Tuned, CT-100, and full Retrain—under the same avg$\Delta$R@P / avg$\Delta$P@R metric (full figures in Appendix § 7, Table 3).

Dataset (split)	CT-100 $\Delta R@P$	CT-100 $\Delta P@R$	CT-Tuned $\Delta R@P$	CT-Tuned $\Delta P@R$	Retrain $\Delta R@P$	Retrain $\Delta P@R$	Ours $\Delta R@P$	Ours $\Delta P@R$
D-Industrial OOT	−46.39%	−34.80%	−7.69%	−2.85%	−46.39%	−41.73%	+2.26%	+1.60%
BAF (7 splits, range)	[−0.39%, −0.04%]	[−0.29%, +0.32%]	—	[−0.75%, +0.09%]	[+0.08%, +0.71%]	[−0.13%, +0.27%]	[+0.02%, +0.30%]

D-Industrial: the generalization test. The fair comparator here is a carefully tuned continued-training update, not a naive one. Even CT-Tuned (one boosting round, lr=0.03) regresses, at avg$\Delta$R@P −7.69% and avg$\Delta$P@R −2.85%; the naive CT-100 and full Retrain never reach production’s precision at any cut-off (avg$\Delta$R@P −46.39%: no recall at those precision levels) and lose 34.80% and 41.73% of precision at equal recall. The overlay instead gains avg$\Delta$R@P +2.26% (95% CI [+1.84%, +2.90%], paired block bootstrap over 66 calendar-day blocks, 1000 replicates) and avg$\Delta$P@R +1.60% ([+1.28%, +2.04%]), positive at every cut-off over the OOT period and on average in each month (Appendix § 12), with no redeployment. The cause is structural, not under-tuning: every training example carries the same selection bias, so no learning rate or round count turns the biased gradient direction into an unbiased one ([14]); a frozen base sidesteps the mechanism entirely.

BAF: a robustness floor. BAF tests robustness to designed variants rather than generalization across a population gap, so gradient updates are well-posed and the overlay should not be expected to lead—and it does not. What it shows is that the overlay does no harm: avg$\Delta$P@R is positive on all seven splits ([+0.02%, +0.30%]) and avg$\Delta$R@P stays near neutral ([−0.13%, +0.27%]; negative on OOT-1 and Var-V). CT-100 is no better ([−0.39%, −0.04%] / [−0.29%, +0.32%]); Retrain raises precision at equal recall more ([+0.08%, +0.71%]) but loses recall at equal precision on six splits (down to −0.75%), and both need a redeploy. The claim is a robustness floor, not an improvement.

Proposal-Source Variance and Ensemble Stabilization

Table 2 reports both metrics per backbone and for the all-3 ensemble. The central finding is high single-LLM variance: the regimes invert the ranking—LLM-MoE is best industrially (+3.88% avg$\Delta$R@P) yet negative on six of seven BAF splits, while LLM-CoT is slightly negative on D-Industrial (−0.28%) yet the best single source on all seven. No in-distribution signal separates them.

Dataset	Split	Precision	Recall	CoT	SIW	MoE	Ensemble (All 3)
D-Industrial	OOT	30.3–56.5%	23.5–62.9%	−0.28% / −0.15%	+2.27% / +1.56%	+3.88% / +3.09%	+2.26% / +1.60%
BAF	OOT-1	17.3–53.2%	4.1–41.4%	−0.12% / +0.12%	−0.21% / +0.06%	−0.18% / −0.09%	−0.10% / +0.07%
	OOT-2	19.1–52.8%	3.9–41.2%	−0.08% / +0.09%	−0.41% / −0.17%	−0.26% / −0.14%	0.00% / +0.06%
	Var-I	18.0–56.2%	3.1–35.3%	+0.21% / +0.12%	−0.07% / +0.17%	−0.22% / +0.04%	+0.04% / +0.08%
	Var-II	14.6–51.8%	6.4–50.1%	+0.37% / +0.25%	+0.20% / +0.13%	+0.19% / +0.13%	+0.27% / +0.30%
	Var-III	10.1–36.6%	4.5–43.1%	+0.23% / +0.26%	−0.22% / +0.18%	−0.07% / +0.41%	+0.04% / +0.28%
	Var-IV	10.1–36.6%	4.5–43.1%	+0.23% / +0.26%	−0.22% / +0.18%	−0.07% / +0.41%	+0.04% / +0.28%
	Var-V	10.8–40.4%	5.1–45.6%	+0.10% / +0.01%	−0.51% / −0.13%	−0.36% / −0.28%	−0.13% / +0.02%

Ensembling all three resolves this without an oracle: avg$\Delta$P@R is positive on all eight splits and avg$\Delta$R@P on five (the exceptions within 0.13% of neutral), trading part of LLM-MoE’s industrial peak (+2.26% vs +3.88%) to avoid its BAF degradation. The gain also needs the rest of the pipeline (Appendix § 9): without the log-lift overlay or the per-shot ensemble, the same LLM rules fall below the frozen base on every split (avg$\Delta R@P$ −2.5 to −46% on D-Industrial, −17 to −33% on BAF), and global instead of per-band calibration lowers avg$\Delta R@P$ on every split.

Analysis of LLM-Proposed Rules

LLM Source and Proposal Behaviour

Appendix § 10 (Table 5) covers all seven subsets. The inversion carries over to pairs: LLM-SIW+LLM-MoE, the best pair on D-Industrial (+2.80% avg$\Delta R@P$), is worst or near-worst on every BAF split; LLM-CoT pairs invert this.

Table 5: Full LLM-source ablation: avg$\Delta$R@P (top) and avg$\Delta$P@R (bottom), %, per split, all seven backbone combinations (full pipeline). All 3 = our default ensemble. CoT/SlW/MoE abbreviate LLM-CoT/LLM-SlW/LLM-MoE (§ 3).

Dataset	Split	CoT	SlW	MoE	CoT+SlW	CoT+MoE	SlW+MoE	All 3
D-Industrial	OOT	−0.28%	+2.27%	+3.88%	+1.14%	+2.36%	+2.80%	+2.26%
BAF	OOT-1	−0.12%	−0.21%	−0.18%	+0.06%	−0.03%	−0.33%	−0.10%
BAF	OOT-2	−0.08%	−0.41%	−0.26%	+0.14%	+0.08%	−0.29%	0.00%
BAF	Var-I	+0.21%	−0.07%	−0.22%	+0.27%	+0.16%	−0.20%	+0.04%
BAF	Var-II	+0.37%	+0.20%	+0.19%	+0.27%	+0.33%	+0.18%	+0.27%
BAF	Var-III	+0.23%	−0.22%	−0.07%	+0.27%	+0.26%	−0.18%	+0.04%
BAF	Var-IV	+0.23%	−0.22%	−0.07%	+0.27%	+0.26%	−0.18%	+0.04%
BAF	Var-V	+0.10%	−0.51%	−0.36%	+0.02%	−0.12%	−0.42%	−0.13%
avg$\Delta$P@R
D-Industrial	OOT	−0.15%	+1.56%	+3.09%	+0.78%	+1.80%	+2.34%	+1.60%
BAF	OOT-1	+0.12%	+0.06%	−0.09%	+0.27%	+0.06%	−0.05%	+0.07%
BAF	OOT-2	+0.09%	−0.17%	−0.14%	+0.24%	+0.06%	−0.04%	+0.06%
BAF	Var-I	+0.12%	+0.17%	+0.04%	+0.19%	+0.13%	+0.44%	+0.08%
BAF	Var-II	+0.25%	+0.13%	+0.13%	+0.23%	+0.16%	+0.11%	+0.30%
BAF	Var-III	+0.26%	+0.18%	+0.41%	+0.25%	+0.33%	+0.37%	+0.28%
BAF	Var-IV	+0.26%	+0.18%	+0.41%	+0.25%	+0.33%	+0.37%	+0.28%
BAF	Var-V	+0.01%	−0.13%	−0.28%	+0.09%	+0.09%	+0.01%	+0.02%

The backbones also differ in how they follow the output contract (Appendix § 8). On D-Industrial, 44% of LLM-CoT’s returned predicates name features that do not exist in the model—half of them near-misses of real names—against 2% or less for the other two, and 62% of LLM-SIW’s are repeats within a shot; on both datasets, 69–73% of LLM-MoE’s pass the support screen. The retained rule sets barely overlap: only 14% (D-Industrial) and 16% (BAF) of distinct retained predicates come from more than one shot, and LLM-CoT shares just 2 of its 22 D-Industrial rules with each other backbone. Each backbone thus contributes largely different patterns—the variance ensemble absorbs.

Representative Risk Patterns

Each retained predicate carries the LLM’s name and rationale beside its measured lift. The LLMs largely follow the steering instruction: the first condition of 97% (D-Industrial) and 99% (BAF) of retained predicates is an underweighted feature, with strongly-modeled features as secondary filters in 9% and 33%. Rules typically combine individually weak evidence into a selective conjunction. On D-Industrial, LLM-CoT cites a 1.09× lift for failed payments at one shipping address and proposes failed_txn(addr,14d)>p90 ∧ buyer_seg>p80, reasoning that repeated failed payments to one address test card authorization; on the adaptation window the rule reaches 71.6× lift on 6,249 transactions. On BAF, LLM-MoE combines session_len>p90 (1.44×) and free_email (1.22×) with housing=0 into a 6.7× rule (862 transactions). Calibration lift does not predict which generalize: the highest-lift D-Industrial predicates come from LLM-CoT, the weakest single source there.

Conclusion

We patched a frozen fraud detector’s blind spots by steering LLM proposals through its underweighted dimensions into a reversible log-lift overlay that runs at the production cut-off. It generalizes where continued training regresses, stays robust where retraining offers no edge, and reports each amendment as an auditable predicate; pooling backbones bounds the worst case.

Limitations

Scope of the recall advantage

Scope of the recall advantage. The recall advantage is demonstrated under a generalization gap between training and evaluation populations. On BAF, where no such gap exists, neither method dominates on recall and we claim only that the overlay does no harm.

Continued-training baselines. We evaluate two continued-training configurations on D-Industrial (CT-100 and CT-Tuned); both trail the frozen baseline, but two configurations do not span the hyperparameter space. Importance-weighted or propensity-adjusted continued training is the principled remedy; we do not evaluate it, as the selection mechanism is a separate model whose scores are unavailable to the adaptation procedure.

Inference cost. The three-model ensemble requires roughly $3\times$ the LLM inference budget of a single-source overlay; a single-source fallback is feasible but sacrifices variance reduction.

Offline evaluation. All results are offline evaluations against historical labels, with every parameter fixed on the adaptation window; no live A/B test has been conducted.

Ethics Statement

Data and privacy. Experiments on D-Industrial use historical transaction records accessed under the platform’s internal data-governance program. No human-subject data were collected, no directly identifying attributes are used as model features, and feature names are paraphrased throughout. BAF is a public benchmark released for fraud-detection research.

Risk of harm. Fraud decisions carry asymmetric costs: a false alert burdens a legitimate customer, a missed fraud harms the platform and its users. The overlay alerts at the production cut-off and never blocks autonomously; deployments should retain human review and an appeals path.

Continued-Training Comparison

Table 3 gives the full continued-training comparison summarized in § 4.1: the all-3 ensemble overlay against three continued-training variants (CT-100, CT-Tuned, full Retrain) under the same avg$\Delta$R@P / avg$\Delta$P@R metric, across both datasets. Each method’s score is mapped to the production scale by its own cut-off table fitted on the adaptation window (§ 2.5).

Prompt Design

Each generation shot sends a single prompt to the LLM; there is no second LLM pass. Listing 1 reproduces the template behind all reported results (D-Industrial and BAF, all three backbones), as recovered from the logged LLM calls. Four placeholders are filled per dataset:

• {domain primer}: a description of the fraud-detection task and of the dataset’s features and their encodings. It is dataset-specific and not reproduced; for D-Industrial it describes proprietary data.

• {model digest}: a summary of the frozen model’s trees—split counts, total gain and split depths for the features with the most gain, the names of the remaining features, the feature sets most frequent on high-gain leaf paths, and a few low-value leaves.

• {SHAP gap}: features split at the median of their mean SHAP over fraud transactions on the adaptation window, into those above it (“forbidden”, listed with their mean SHAP) and those at or below it (“target”), each target feature with a decile or category table of counts, fraud rate, and lift over the window’s base rate. The template’s “SHAP <= 0” refers to this target set.

• {number of rules}: 50.

All other text is verbatim, with non-ASCII glyphs rendered in ASCII (an em dash as --, $\le$ as <=, and a left-pointing marker as <-). The returned predicates are then compiled, screened and assigned to band pools in code (§ 2.3), not by the LLM.

{domain primer}
---
{model digest}
---
{SHAP gap}
---
## Task: Adversarial Rule Generation

You are a red-team analyst. The LightGBM model
already captures fraud patterns
anchored on its **forbidden features** (high SHAP
for fraud, listed above).
Your job is to find fraud patterns
systematically missed by reasoning in the **target feature** dimensions it underweights
.

Work through the following three steps in your head
before writing the output.

### Step 1 -- Adversarial evasion reasoning

Think like a fraudster who knows LGB monitors the
forbidden features closely.
Design fraud behaviours that primarily exploit the
target features
(SHAP <= 0, listed above) as the main discriminating
anchor.
You may also include forbidden features as secondary
filters when the combination
reveals a behavioural archetype that the target
features alone would not isolate -- for
example, a target-feature profile that only
becomes high-precision when
narrowed by a forbidden-feature threshold.
For each candidate pattern, ask: *why would a
fraudster naturally exhibit this
combination? Does it correspond to a real
behavioural archetype?*

### Step 2 -- Data grounding

For each evasion pattern, check the decile/category
tables in the SHAP Gap
section. Look for target-feature groups with **lift
> 1.5 (<=)***. Prefer
patterns that are both theoretically coherent AND
supported by elevated lift
in the data. If no <- appears for a candidate
feature, weigh it lower.

### Step 3 -- Predicate construction

Translate the evasion pattern into a conjunction
predicate using:
- Feature names exactly as listed above (no invented
  names).
- Operators: `>`, `<`, `>=`, `<=`, `==`, `isna`, `notna`
- Percentile tokens: `p5`, `p10`, `p15`, `p20`, `p25`,
  `p30`, `p40`,
  `p50`, `p60`, `p70`, `p75`, `p80`, `p85`, `p90`,
  `p95`, `p99`.
- Join conditions with `AND` only (no top-level OR).
---

### Output format

Produce exactly **{number of rules}** rules as a
JSON array. Each item:

```json
[
  {
    "id": "R_001",
    "name": "short label (5 words max)",
    "evasion_reasoning": "why a fraudster shows this
      pattern in target features; if forbidden
      features appear as secondary filters, explain
      why that narrows to a distinct archetype",

"data_evidence": "specific lift values or decile
ranges from the table that support this",
"target_features_used": ["feat_a", "feat_b"],
"forbidden_features_used": ["feat_c"],
"predicate": "feat_a > p80 AND feat_b == 3 AND
feat_c > p90"
}

Constraints:

• Use ONLY feature names that appear verbatim in the digest or SHAP gap above.

• The primary anchor condition(s) must be target features (SHAP <= 0). Forbidden features may appear as secondary filters in a conjunction.

• Each rule must address a distinct evasion archetype — no near-duplicates.

• Rank from strongest evidence + clearest evasion logic to weakest.

Respond with the JSON array only — no explanation, no markdown fences outside the JSON.

Listing 1: Rule-proposal prompt template used for every shot (anonymized placeholders, ASCII).

Full Pipeline Ablation

Table 4 reports both metrics across all eight evaluation splits. Conditions denote removing components from Full (complete pipeline: all three backbones, per-band calibration, per-shot logit ensemble): −PerBand replaces per-band with global calibration; −Ens uses a merged deduplicated rule pool with no per-shot ensemble; −Ens −PerBand combines both; −LogLift scores each shot by its rule fire rate instead of the log-lift overlay; Rules-only uses the merged pool’s fire rate. Every condition is evaluated like Full, with its own cut-off table fitted on the adaptation window.

Table 4: Full pipeline ablation: avg$\Delta$R@P (top) and avg$\Delta$P@R (bottom), %, per split. Full = complete pipeline; each other column removes components (definitions in Appendix § 9).

Dataset	Split	Full	−PerBand	−Ens	−Ens −PerBand	−LogLift	Rules-only
D-Industrial	OOT	+2.26%	−2.41%	−2.48%	−46.39%	−35.73%	−24.76%
BAF	OOT-1	−0.10%	−0.42%	−21.42%	−22.87%	−26.02%	−25.76%
BAF	OOT-2	0.00%	−0.38%	−20.44%	−20.97%	−25.46%	−25.16%
BAF	Var-I	+0.04%	−0.09%	−17.45%	−19.12%	−21.46%	−21.04%
BAF	Var-II	+0.27%	−0.06%	−19.96%	−24.10%	−32.92%	−32.10%
BAF	Var-III	+0.04%	−0.20%	−17.85%	−21.41%	−26.79%	−26.40%
BAF	Var-IV	+0.04%	−0.20%	−17.85%	−21.41%	−26.79%	−26.40%
BAF	Var-V	−0.13%	−0.70%	−21.97%	−24.63%	−29.18%	−28.44%
avg$\Delta$R@P
D-Industrial	OOT	+1.60%	−2.99%	−5.11%	−41.54%	−31.45%	−24.11%
BAF	OOT-1	+0.07%	−0.25%	−15.46%	−19.38%	−26.84%	−26.00%
BAF	OOT-2	+0.06%	−0.21%	−16.18%	−19.68%	−28.15%	−27.15%
BAF	Var-I	+0.08%	+0.25%	−15.73%	−17.79%	−26.75%	−26.37%
BAF	Var-II	+0.30%	+0.10%	−13.45%	−16.36%	−23.78%	−22.71%
BAF	Var-III	+0.28%	+0.26%	−9.25%	−11.49%	−16.42%	−15.69%
BAF	Var-IV	+0.28%	+0.26%	−9.25%	−11.49%	−16.42%	−15.69%
BAF	Var-V	+0.02%	+0.27%	−11.20%	−12.10%	−17.61%	−16.74%

Full LLM-Source Ablation

Table 5 reports both metrics across all eight evaluation splits for all seven combinations of the three LLM backbones (full pipeline). No combination dominates uniformly: LLM-MoE alone gives the highest avg$\Delta R@P$ on D-Industrial (+3.88%) and LLM-SlW+LLM-MoE the highest of the pairs (+2.80%), but both are negative on most BAF splits, while LLM-CoT—slightly negative on D-Industrial—is the best single source on every BAF split. All 3 remains strongly positive on D-Industrial and avoids the worst corners on BAF.

All Operating Points

Table 6 lists, for every cut-off $t_b$ and split, production ($s \ge t_b$) and the all-3 ensemble overlay ($\tilde{s}_b \ge t_b$) side by side on OOT data—precision and recall—with the $\Delta R@P$ and $\Delta P@R$ checks at that cut-off. The two need not alert the same volume on OOT; the checks compare them at equal precision and equal recall instead. Var-IV shares Var-III’s evaluation population.

Table 6: All operating points, all-3 ensemble: production $s \ge t_b$ vs. overlay $\tilde{s}_b \ge t_b$ on OOT data (precision P%, recall R%), and $\Delta R@P$ / $\Delta P@R$ (%) at each cut-off. Every parameter is fixed on the adaptation window.

Split	$t_b$	Production P%	Production R%	Overlay P%	Overlay R%	$\Delta R@P$	$\Delta P@R$
D-Ind. OOT	998	56.48	23.47	57.51	24.37	+2.57%	+1.03%
	997	49.07	35.67	50.30	38.00	+3.31%	+2.04%
	996	43.29	45.80	44.06	47.67	+1.88%	+1.66%
	995	37.99	52.41	38.73	54.00	+2.41%	+1.91%
	994	33.99	58.12	34.58	59.27	+1.55%	+1.96%
	993	30.32	62.86	30.91	64.00	+1.84%	+1.02%
BAF OOT-1	999	53.15	4.12	53.95	4.05	+0.21%	+0.30%
	995	34.89	13.88	34.68	13.53	−0.59%	−0.11%
	990	28.15	22.40	28.55	21.74	−0.31%	−0.34%
	985	23.76	28.35	24.39	28.00	+0.94%	+0.52%
	980	20.77	33.07	21.08	33.31	+0.94%	+0.60%
	975	18.94	37.68	19.00	37.05	−1.68%	−0.45%
	970	17.35	41.42	17.56	40.96	−0.21%	−0.01%
BAF OOT-2	999	52.83	3.89	52.68	3.75	+0.31%	+0.71%
	995	37.20	13.83	36.60	13.24	−1.53%	−0.47%
	990	29.74	21.86	30.08	21.09	−0.35%	−0.22%
	985	25.23	27.52	25.54	26.93	+1.08%	+0.36%
	980	22.57	32.97	22.64	32.80	+0.69%	+0.10%
	975	20.51	37.32	20.70	36.80	−0.14%	−0.01%
	970	19.07	41.17	19.30	40.93	−0.10%	−0.08%
BAF Var-I	999	56.25	3.13	57.79	3.09	−0.03%	−0.25%
	995	33.98	10.88	34.50	10.63	+0.07%	+0.07%
	990	28.05	17.72	28.91	17.48	0.00%	+0.56%
	985	24.64	23.45	25.22	23.11	+0.59%	+0.39%
	980	21.85	28.11	22.23	28.04	+0.56%	+0.30%
	975	19.65	32.00	19.97	31.62	−0.28%	−0.17%
	970	17.99	35.27	18.03	34.64	−0.63%	−0.33%
BAF Var-II	999	51.84	6.36	53.04	6.36	−0.21%	+1.18%
	995	31.96	20.29	32.32	20.01	−0.28%	−0.41%
	990	24.67	30.09	25.27	29.78	+0.21%	+0.36%
	985	20.40	36.69	20.91	36.28	+1.04%	+0.35%
	980	18.03	42.49	18.24	42.43	+0.03%	+0.13%
	975	16.03	46.39	16.39	46.53	+1.04%	+0.45%
	970	14.58	50.14	14.76	49.69	+0.07%	+0.05%
BAF Var-III	999	36.59	4.52	38.27	4.68	+0.04%	+1.59%
	995	21.17	15.12	21.60	15.05	0.00%	+0.01%
	990	16.54	23.87	16.96	23.57	−0.72%	+0.02%
	985	13.95	30.13	14.32	29.83	+1.09%	+0.26%
	980	12.45	35.82	12.53	35.52	−0.15%	−0.08%
	975	11.10	39.63	11.30	39.59	+0.08%	+0.19%
	970	10.11	43.06	10.22	42.61	−0.08%	−0.01%
BAF Var-IV	999	36.59	4.52	38.27	4.68	+0.04%	+1.59%
	995	21.17	15.12	21.60	15.05	0.00%	+0.01%
	990	16.54	23.87	16.96	23.57	−0.72%	+0.02%
	985	13.95	30.13	14.32	29.83	+1.09%	+0.26%
	980	12.45	35.82	12.53	35.52	−0.15%	−0.08%
	975	11.10	39.63	11.30	39.59	+0.08%	+0.19%
	970	10.11	43.06	10.22	42.61	−0.08%	−0.01%
BAF Var-V	999	40.36	5.05	40.54	5.09	+0.04%	+0.18%
	995	22.43	16.48	22.76	16.29	−0.04%	−0.07%
	990	17.52	25.45	17.76	25.00	+0.41%	+0.11%
	985	14.74	31.86	15.03	31.18	…

D-Industrial per OOT Month

Table 7 breaks the D-Industrial results of the all-3 ensemble down by OOT month, with the cut-offs and every table fixed on the adaptation window and precision, recall, and the checks computed within each month. March covers only its first seven days, so its per-cut-off values rest on few frauds.

Table 7: D-Industrial per OOT month, all-3 ensemble: $\Delta R@P$ (top) and $\Delta P@R$ (bottom) at each production cut-off $t_b$ and their average (%).

Month	998	997	996	995	994	993	Avg
$\Delta R@P$
Jan 2026	+6.10%	+1.72%	+0.47%	+2.27%	+1.56%	+2.27%	+2.40%
Feb 2026	+2.29%	+5.01%	+3.48%	+3.16%	+0.98%	+1.74%	+2.77%
Mar 2026	−0.79%	+4.74%	+3.56%	+1.98%	+0.79%	−0.40%	+1.65%
$\Delta P@R$
Jan 2026	+2.72%	+2.37%	+0.88%	+1.64%	+2.35%	+1.10%	+1.84%
Feb 2026	+1.13%	+2.96%	+2.78%	+2.08%	+0.84%	+0.84%	+1.77%
Mar 2026	−0.28%	+2.00%	+1.91%	+1.64%	+1.78%	−0.61%	+1.07%