betaFeedback

WorksComputer Science

Which AI Use Cases Deserve a Pilot? A Risk and Value Framework for Funding, Scoping, and Scaling AI

Abstract

A composite score can rank AI use cases while leaving the permitted operating scope and the evidence required for expansion unspecified. We treat the unit of decision as a use case at a proposed scope (authority, human mediation, limits, volume, environment, population) and define three decisions: screen the scope, fund the experiment, and authorize expansion only when residual risk is acceptable and evidence of incremental net value applies at that scope. A risk track (R1-R4) is implemented as an explicit decision table with vetoes and documented precedence, and a value track follows four evidence stages with each transition needing its own evidence. Funding uses expected value of perfect information as an upper bound and the value of sample information for the specific experiment. A quantified expected-loss check, required where policy says so, computes the containment an expansion needs and credits a control only on evidence measured at the proposed scope; in a synthetic scenario the highest-value control design fails that test while a smaller scope passes. A reproducible Python package with 279 passing tests demonstrates the framework on synthetic refund, reconciliation and knowledge-assistant use cases, including a synthetic matched-scope fixture and a transferability stress test (value evidence is simulated, not realized), and compares it with a public AWS sample. Thresholds and appetite are prototype policy requiring field validation.

Introduction

Organizations are asked to choose among many candidate AI projects. A composite score can rank AI use cases while leaving the permitted operating scope and the evidence required for expansion unspecified. This paper argues that the unit of decision is an AI use case at a proposed operating scope. Scope includes the system’s authority (recommend, execute with approval, autonomous capped, autonomous uncapped), the human mediation around it (per-item, sampled or none), transaction limits, volume, environment and the affected population. A refund recommender that a human reads, an agent that executes each refund after per-item approval, and an agent that issues refunds autonomously up to a cap are the same business idea at three scopes with very different risk and evidence needs.

Three decisions follow, shown in Figure 1. Screen the scope: what the system may do during the pilot and under which constraints. Fund the experiment: whether an eligible pilot justifies its cost given projected value, feasibility and what that specific experiment can teach. Authorize expansion: whether both the residual risk is acceptable and there is sufficient evidence of incremental net value at the proposed scope. Whenever scope changes, the framework asks which evidence still applies. This is the conceptual contribution. The comparison with a public AWS sample (§ 6) is an evaluation example only.

Decision framework for one candidate

Figure 1. Decision framework for one candidate (a use case at a proposed operating scope). Screening, funding and expansion are the three decisions; the pilot between funding and expansion produces the evidence that expansion needs. R1–R3 are technical failures, deployment controls and business consequences, R4 is residual risk, V1–V4 are value stages and E1–E4 the evidence levels that support them. Both the risk gate (residual risk, with the expected-loss check where policy requires it) and the value gate must pass at the proposed expansion scope. Evidence applicability links all steps, and a change of scope or evidence (dashed) sends the candidate back to the affected decisions. R-BED and V-BED labels follow a presentation slide; table values, thresholds and appetite are PilotGate prototype policy.

Worked example in brief. The refund candidates in Table 9 are synthetic. RA, the recommender, can run a live pilot with human-approved action and, with evidence at the evaluated scope, passes both gates for expansion (incremental net value about $465,234 per year, interval $220,294 to $710,174). RB, the approval-gated executor, cannot clear the risk gate at its proposed target, and the evidence from its pilot does not carry to the larger scope. Our recommendation is to redesign the scope and then reassess pilot funding, because no EVSI was calculated for a redesigned lower scope. RC, the autonomous executor, may only run in shadow mode, and its shadow run is funded for the error-rate information it yields (a risk decision about the next pilot rung) rather than for value. We carry these three through the paper.

Use casePilot rungResidual risk, pilot scopeResidual risk, target scope
RAHUMAN_APPROVED_LIVELOW [R1 2 | R2 1 | R3 2]HIGH [R1 2 | R2 2 | R3 3]
RBHUMAN_APPROVED_LIVEHIGH [R1 2 | R2 2 | R3 3]CRITICAL [R1 2 | R2 3 | R3 3]
RCSHADOWHIGH [R1 2 | R2 2 | R3 3]CRITICAL [R1 2 | R2 3 | R3 3]

Table 9. The running example at three operating scopes: refund recommender (RA), approval-gated executor (RB) and autonomous capped executor (RC). Residual risk is the R4 level with its R1/R2/R3 inputs (prototype policy, § 2.2).

Contributions. (1) A scope-aware framework with three decisions and an explicit evidence-applicability rule. (2) A risk decision table (R1–R4) with vetoes, scope-specific control requirements and documented precedence, kept separate from quantified loss. (3) A stage-linked value track (V1–V4, E1–E4) in which every transition needs its own evidence, projections are hypotheses, and decision influence is a metric only where value depends on changing decisions. (4) A funding rule using EVPI as an upper bound and EVSI for the designed experiment. (5) A quantified expected-loss check inside the expansion decision (Section 3): required containment, a capacity ceiling, an evidence standard under which a control is credited only on scope-applicable live evidence, and a comparison of control designs. (6) A reproducible Python implementation with 279 passing tests, and synthetic demonstrations, including a synthetic matched-scope fixture and a transferability stress test (value evidence is simulated, not realized).

Framework

Unit of decision and scope

A candidate is a tuple of use case and proposed scope. Authority is ordered RECOMMEND < EXECUTE_WITH_APPROVAL < AUTONOMOUS_CAPPED < AUTONOMOUS_UNCAPPED. A transaction limit is capped, uncapped, not_applicable or unknown. Higher authority and weaker mediation never inherit evidence gathered at a lower authority or stronger mediation; limit and volume may inherit it only through a stated, justified, evaluated transfer assumption within policy caps.

Decision 1: screen the scope

Screening follows the risk track presented as R1 technical failures, R2 deployment controls, R3 business consequences and R4 residual risk (Figure 2). We implement it as a decision table in which tested controls lower the residual assessment relative to the inherent one. R1 is the failure level, from an ordinal judgment or, when a measured 95% upper bound on the error rate covers the scope, from fixed cut points. R2 is the maximum of human-control, exposure and monitoring levels. R3 is the maximum of severity, propagation and recovery levels. Table 1 maps R2 and R3 to a base tier, which R1 adjusts by one step. Hard blockers prohibit the proposed action; vetoes force the RESTRICTED tier before any other rule. An appetite tier per authority then decides whether the residual rating is acceptable (prototype policy: pilot HIGH; scale HIGH, HIGH, MEDIUM, NONE across the four authority levels). Controls count only if tested at the proposed scope (§ 2.5). Three ratings are reported (inherent materiality, inherent before controls, residual) and ordinal judgments are never multiplied or mixed with quantified loss, which is reported on a separate loss view.

Presentation-level structure of the R-BED (risk) and V-BED (value) frameworks

Figure 2. Presentation-level structure of the R-BED (risk) and V-BED (value) frameworks, redrawn from the slide supplied by the author. R-BED and V-BED are proposed frameworks of V. Srivastava; the figure shows their structure only and is not a result of this paper, which adds operating scope, evidence applicability and the quantified check (Figure 1). The slide’s third outcome, “Pilot, with constraints”, is labelled here with the PilotGate action “Extend pilot with constraints”, which continues an existing pilot; the pilot-screen permission is “Start constrained pilot”.

R2 control levelR3 = 1R3 = 2R3 = 3
1LOWLOWMEDIUM
2LOWMEDIUMHIGH
3MEDIUMHIGHCRITICAL

Table 1. PilotGate base risk tier from deployment-control level (R2, rows) and business-consequence level (R3, columns). This table is prototype policy, not a published R-BED table. R1 (technical failure) moves the tier by at most one step; any veto yields RESTRICTED first.

Decision 2: fund the experiment

Eligibility comes from decision 1. Among eligible pilots, funding uses the intake value distribution, feasibility and information value. EVPI, the expected value of perfect information about the expansion decision, is only an upper bound. For each recommendation we also state what the specific experiment can learn: a normal–normal preposterior gives EVSI for a designed pilot of nn cases per arm over a three-year horizon with a stated per-case standard deviation. A shadow run observes errors but not value, so its EVSI on value is zero; it may be funded for risk information or an independent rationale, which we name. An EVSI figure is stated with the experiment it describes: it may be that of a hypothetical design the screen does not permit, or of a scale decision the structural risk rule cannot let reach SCALE at the proposed scope; neither is a funding justification for that scope. If the information case is not supported (EVSI below cost) and no independent rationale is stated, the pilot is not recommended.

Use caseV1/E1V2/E2V3/E3V4/E4LevelDurable
RAmeasuredmeasuredmeasuredmeasuredE4yes
RBmeasuredmeasuredmeasuredmeasuredE4yes
K1measuredmeasuredmeasuredmeasuredE4yes
P1measuredmeasuredweaknoneE2no
RCmeasurednonenonenoneE1no

Table 2. Stage evidence reached by each synthetic use case (E4 economics are simulated). The level is the highest consecutive measured stage; a projection alone is E0. Durability is reported separately (policy thresholds).

Decision 3: authorize expansion

Both gates must pass at the proposed scope. The risk gate uses the R4 residual tier against appetite with control tests applicable to the proposed scope. The value gate uses the stage-linked evidence ladder below.

Value track. Following the presentation slide, V1/E1 comprises benchmarks and task tests, V2/E2 pilots and usage data, V3/E3 controlled comparisons, and V4/E4 realized economic impact. Every transition needs its own evidence, so a level is the highest consecutive stage with measured evidence; an intake projection is a hypothesis (E0). A controlled improvement in an operational outcome can support E3; an annualized estimate still depends on costing and deployment assumptions; E4 requires realized economic impact. A real E4 record needs observed operating costs (invoices or metered usage) and documented savings or redeployed capacity at the scope where they occurred. In this paper E4 is simulated: the run cost is an assumed 105% of the intake estimate, so the dollars verify the logic and are not realized impact. Annualized figures at the proposed scope are forecasts. We treat durability and transferability separately: durability uses configurable stability thresholds (a minimum study size, duration and stability share, which are policy assumptions and not attributed to any source), and transferability is handled by evidence applicability.

Mechanisms and decision influence. Each value claim names a mechanism (decision change, speed, cost, quality or capacity), a baseline, an observable outcome and costing assumptions. Decision influence is measured only for use cases whose value depends on changing decisions. For others (for example approval-gated refunds where the approval decision is unchanged) speed, cost, quality or capacity is measured, so unchanged decisions can coexist with positive value. States are applicable, not_applicable and unknown, and are tested separately. Strong capability and adoption can still yield zero or negative incremental value (K1 below).

Evidence applicability

For each control test, error measurement and value estimate, the scale decision asks whether it applies as measured, needs a stated, justified, evaluated transfer, or must be remeasured. Authority and mediation never transport upward. A control-test failure newer than, or unordered against, a pass revokes the credit; legacy or undated records get no credit at scale; test age is capped at 180 days at the as-of date (policy). Volume transfers only within a ratio cap of 6×6 \times and limits only within a ratio of 1.0, with an effect haircut.

Use caseMechanismDecision influenceObservable outcome
RAdecision_changeapplicableCost per eligible case: handling minutes, leakage from wrong refunds, rework.
RBspeednot_applicableHandling minutes per case after the approval decision, net of approver minutes.
RCcapacitynot_applicableShare of refunds completed without staff time; errors and recovery cost.
P1speednot_applicableMinutes to complete an unchanged reconciliation, before versus after.
K1decision_changeapplicableCost per contact: handling minutes, errors, rework; plus CSAT as guardrail.
D1decision_changeunknownCost per dispute.

Table 3. Value mechanisms. Decision influence is a metric only where value depends on changing a decision (applicable); otherwise it is not_applicable, and an unrecorded state is unknown. RB’s refund approval is unchanged by the tool, yet speed still creates value.

Joint decision and executive actions

Precedence is: a prohibition gives BLOCKED; a scope contradiction or veto gives REDESIGN; an incomplete proposal gives DEFER; contradictory value evidence gives EXTEND_PILOT; value FAIL gives REDESIGN or STOP; risk FAIL gives REDESIGN; both PASS gives SCALE; otherwise EXTEND_PILOT. Executives see four actions: Deploy/Scale, Extend pilot with constraints, Redesign and Stop. "Extend pilot with constraints" continues an existing pilot; permission to start a constrained pilot is the separate outcome of the pilot screen. BLOCKED and DEFER are implementation statuses. A blocker prohibits the proposed action and maps to Stop; because it covers every rung of the ladder, no lower-scope experiment is available. A deferral means evidence or scope information is missing or unresolved; it maps to no action, and a lower-scope experiment is allowed only if that lower scope is itself complete and passes the pilot screen. Table 13 gives the mapping for the example.

Use caseStatusExecutive actionLower-scope experiment?
RASCALEDeploy/Scale-
RBREDESIGNRedesigna redesigned (lower) scope is a new candidate and is screened afresh
K1REDESIGNRedesigna redesigned (lower) scope is a new candidate and is screened afresh
P1EXTEND_PILOTExtend pilot with constraintsthe extended pilot is the lower-scope experiment
RCREDESIGNRedesigna redesigned (lower) scope is a new candidate and is screened afresh

Table 13. Implementation status mapped to executive action, with whether a separately evaluated lower-scope experiment remains open.

Quantifying control investment

Ordinal tiers say how risky a scope is, not whether a control is large enough. Where policy requires it, the expansion decision therefore adds a quantified expected-loss check at the proposed scope. The check is an additional risk check: it never replaces the ordinal rating, the vetoes or the mandatory controls, and a pass satisfies that check only. The value gate stays independent, so no projected value offsets a failed check, and a pass never rescues a value failure. If the check is required and its inputs or applicable evidence are missing, the result is an explicit INSUFFICIENT (the decision is then EXTEND_PILOT), never a silent pass. If policy does not require the check, it is reported for information and no decision changes.

Expected loss, required containment and capacity

Let VV be eligible actions per day and N=365VN = 365V per year (one-year horizon, no discounting). Let qq be the share of actions that are unsafe attempts, ll the loss per escape and dd the share of unsafe attempts contained before execution. Expected annual loss is L=Nq(1−d)lL = Nq(1-d)l, computed separately for ordinary execution error and strategic abuse (mutually exclusive modes). Against an annual appetite AA the required containment is dmin⁡=max⁡{0,1−A/(Nql)}d_{\min} = \max\{0,1-A/(Nql)\}. Containment cannot exceed what the control can deliver. For uniformly sampled review with recall ρ\rho and capacity BB reviews per day, d(V)=ρmin⁡{1,B/V}d(V) = \rho\min\{1,B/V\}, so the modeled ceiling on volume is Vmax⁡=ρB+A/(365ql)V_{\max} = \rho B + A/(365ql) above BB (Appendix A). Mandatory per-item approval behaves differently: above capacity the queue grows by V−BV-B per day and throughput is limited to BB; lower coverage is modeled only if the policy explicitly permits unreviewed overflow. A post-payment audit earns no pre-payment containment, because the payment has already settled. If it recovers funds, the recovery lowers the loss in a separate term and is never counted as containment.

Deterrence is optional and applies only where deliberate abuse is modeled. With abuse payoff GG, penalty CC and honest payoff TT, a risk-neutral actor abuses only if detection is below p∗=(G−T)/(G+C)p^* = (G-T)/(G+C), the break-even of an expected-payoff comparison of the kind used in the economics and finance literature [3]. The threshold follows directly from the stated payoffs. Abuse pays GG if undetected and costs the penalty CC if detected, so its expected payoff at detection probability pp is (1−p)G−pC(1-p)G-pC, and the actor abuses only if this exceeds the honest payoff TT, that is, only if p<(G−T)/(G+C)p < (G-T)/(G+C). Detection is not containment: a detected abuse can still have been paid. Equality at p∗p^{*} is indifference and is not credited as deterrence. In a separate illustration (G=2G = 2, C=3C = 3, T=0.3T = 0.3, so p∗=0.34p^{*} = 0.34) audit detection 0.85⋅600/V0.85 \cdot600/V equals p∗p^{*} at exactly 1,500 cases per day (indifference), strictly exceeds it only below 1,500 and is 31% at 1,644.

Cases/day (VV)Actions/yr (N=365VN = 365V)Expected loss, no controlRequired containmentAchieved, sampled reviewExpected loss, sampled review
300109,500219,000∣0.0219,000 | 0.0% | 85.0% |32,850
1,644600,0601,200,120∣75.01,200,120 | 75.0% | 31.0% |827,820

Table 4. Worked scenario RF (synthetic): 1% unsafe attempts, 200perescape,200 per escape,300,000 per year appetite, 600 reviews per day at 85% recall. At the pilot volume no containment is required; at the proposed volume about 75% is required and sampled review achieves about 31%.

What the required gate accepts as evidence

A synthetic scenario input (a design recall, a flag rate, a recovery rate, an assumed detection probability) is a planning value. It drives the figures, the design comparison and the modeled boundaries, but the required gate never credits a planning value by itself. Every credit that can lower the expected loss is

credited value=min⁡{design value, p}×r,\text{credited value} = \min\{\text{design value},\ p\} \times r,

where pp is the exact one-sided lower confidence bound (level 0.05) on a proportion measured by observed, live seeded attempts whose scope covers the proposed scope, and r≤1r \le1 is the stated risk-side ratio when the evidence is carried to the proposed scope by a transfer assumption. The rule applies to reviewer or control recall, to pipeline containment, to the detection recall behind a post-payment audit, to recovery and to deterrence. A pipeline is credited only on a representative end-to-end test, or on stage records in which the second stage was measured among the attempts that escaped the first; a second-stage rate measured on all attempts earns nothing. Deterrence is credited only if the lower bound on detection strictly exceeds p∗p^{*} and the payoffs GG, CC, TT are a stated, justified and evaluated assumption, since seeded attempts cannot measure them. Self-reported, non-live or inapplicable evidence earns no credit.

Transfer needs its own rule. A transfer record carries a value-side effect ratio for the savings effect, and that ratio is not a recall ratio: it is never applied to, or substituted for, one. Coverage alone, through a transfer assumption that states no risk-side recall ratio, earns no control credit (the check is INSUFFICIENT). A stated ratio, such as 0.85, multiplies the credited recall; 1.0 is an explicit, evaluated claim of no haircut.

The outcome has three branches. Fail if the stated design fails the appetite on its own, or if even the optimistic reading of the evidence, min⁡{design, exact upper bound}\min\{\text{design},\ \text{exact upper bound}\}, fails (for example 0 of 59 caught). Pass if the credited loss is within appetite. Otherwise insufficient: the design would pass but the evidence cannot yet show that it delivers what the design assumes, and the output states how many seeded attempts would be needed. The attempt rate qq and the loss per escape LL remain stated inputs that this gate does not test; they are reported, with stress scenarios that never gate. Appendix B gives the status codes and the three bypasses of an earlier version that this standard closes.

Worked scenario

The synthetic scenario RF has 1% unsafe attempts, $200 per escape, an appetite of 300,000peryear(300,000 per year (821.92 per day), 600 reviews per day at 85% recall, and 300 cases per day in the pilot against 1,644 proposed (Table 4, Figure 3). At 300 cases per day the expected loss with no control is $219,000, below the appetite, so no containment is required. This is the teaching point: a pass at pilot volume does not show that the control can meet the expansion requirement. At 1,644 cases per day about 75% containment is required while the current design achieves about 31%. The modeled ceiling is 920.96 cases per day: 920.9 passes, the ceiling itself passes (equality) and 921.0 fails. At 1,644 cases per day the minimum capacity is 1,450.64 reviews per day, so 1,451 is the smallest whole number, and 1,450 fails. These boundaries hold under the stated point assumptions; whether the control meets them on evidence is the question of Section 3.2.

Table 5 runs the same cases through the decision path. Pilot-scope evidence does not apply at 1,644 cases per day without a stated and evaluated transfer, and when the transfer record states only a value-side effect ratio the check is INSUFFICIENT. With matched-scope evidence the engine reproduces the boundaries: 1,450 reviews per day fail and 1,451 pass. An observed 0 of 59 fails even at 1,451 reviews per day, because the design’s 85% recall is credited only through evidence. Mandatory per-item approval at 600 cases per day without permitted overflow fails as a queue.

Table 5. Engine decisions with the quantified check required. Containment is credited only on scope-applicable live evidence (§ 3.2); “matched” evidence means seeded attempts measured at the proposed scope. The expected loss shown is on the evidence-credited basis. The decision combines the check with RA’s other (synthetic) risk and value evidence, held fixed.

Case (RA at 1,644 cases/day; quantified check required)Quantified checkExpected loss, $/yr (credited) | Decision
no quantified inputsinsufficient (no inputs)–Extend pilot
evidence at pilot scope, no transportinsufficient (evidence not applicable)–Extend pilot
pilot-scope evidence, value-only transport record, 1,451 reviews/dayinsufficient (no risk-side rule)–Extend pilot
pilot-scope evidence, risk-side recall ratio 0.85, 1,451 reviews/dayinsufficient (evidence too weak)$434,826 | Extend pilot
sampled review, 600 reviews/dayfail$827,820 | Redesign
sampled review, 1,450 reviews/dayfail$300,395 | Redesign
sampled review, 1,451 reviews/daypass$299,774 | Scale
sampled review, 1,451 reviews/day, observed 0/59 caughtfail$1,200,120 | Redesign
pipeline d1=1d_1 = 1, no evidenceinsufficient (evidence not applicable)–Extend pilot
pipeline d=0.80d = 0.80, representative end-to-end 59/59pass$240,024 | Scale
mandatory per-item, 600/day, no overflowfail (queue)$180,018 | Redesign
mandatory per-item, 600/day, overflow permittedfail$827,820 | Redesign
post-payment audit, 50% recovery (recovery evidence 59/59)fail$1,013,970 | Redesign

Figure 3. Required and achievable containment against eligible cases per day for the synthetic scenario RF (1% unsafe attempts, $200 per escape, $300,000 per year appetite, 85% recall, 365 days, one-year horizon). Solid: required containment dmin⁡(V)d_{\min}(V); dashed: containment achievable by uniformly sampled review. Shading in (a): volumes at which the modeled expected-loss test passes (V≤920.96V \le920.96). (b) More reviewers (ceilings 1,175.96 and 1,685.96). (c) A lower loss per escape, only if the effect of a limit on loss is defensibly estimated. A pass here is necessary, not sufficient: scale authorization still needs the ordinal risk, evidence, mandatory-control and value conditions.

Comparing control designs

Table 6 compares six designs with one baseline (the current manual process) and one horizon (one year), counting avoided loss once. A change in volume or in the transaction limit is a change of operating scope and is labelled as such. D2, automated pre-payment checks with escalation, has the highest net value ($2,009,787 per year) and fails the expected-loss test: its residual expected loss is $333,033, 11% above the appetite. The failure is narrow. The exact break-even automated-check recall is dmin⁡/ρ=0.8824d_{\min}/\rho= 0.8824: the expected loss is $300,102 just below it and $299,898 just above. Restricting volume to 1,400 cases per day (expected loss $283,605, net value $1,711,497) or adding a $100 limit ($166,517, $1,576,244, with the lower loss per escape an assumption) passes. Figure 4 shows that the highest-value design is on the wrong side of the risk line, which the table alone does not.

Table 6. Design comparison (synthetic RF; manual-process baseline; one-year horizon; avoided loss counted once, inside net value). D0 no added control; D1a per-item approval with unreviewed overflow; D1b per-item approval without overflow (served volume limited to 600/day); D2 automated checks with escalation; D3 post-payment audit (recovery 50% modeled separately, no pre-payment containment); D4 D2 with a $100 limit. “Planning test” applies the $300,000 appetite to the stated design assumptions; “matched evidence” is the same test with containment credited on 59 live seeded attempts, none missed, measured at that scope.

DesignScopeddResidual expected lossControl costFalse blocks/yrNet valuePlanning testMatched evidence
D0proposed scope0.000$1,200,120 | $00$1,200,120 | fail | fail
D1aproposed scope0.310$827,820 | $394,2000$740,220 | fail | fail
D1bscope change: served volume restricted to BB0.850$65,700 | $394,2000-$21,900 | queue | pass
D2proposed scope0.723$333,033 | $42,568594$2,009,787 | fail | fail
D3proposed scope0.000$1,013,970 | $394,2000$992,070 | fail | fail
D4scope change: limit $100 | 0.723 | $166,517$42,568 | 594 | $1,576,244passpass
D2scope change: volume 1,400/day0.723$283,605 | $36,250506$1,711,497 | pass | pass

Figure 4. Residual expected loss and incremental net value of the compared control designs at the proposed scope (synthetic RF, one year, manual-process baseline, avoided loss counted once). The dashed line is the $300,000 appetite. Filled marks pass the expected-loss test applied to the stated design assumptions; whether a design earns a pass on evidence is in Table 7. Squares change the operating scope (served volume, volume or limit). D2 passes at automated-check recall of at least 0.8824; the $100-limit result assumes a loss per escape of $100; the recovery rate, the share of savings kept on reviewed cases and the cap’s effect are planning assumptions.

Sensitivity. Varying recovery from 0 to 75% and loss per escape over $100, $200 and $300, D2 remains the highest-value design in every cell and a post-payment audit is never feasible; recovery moves only D3. The conclusion that the highest-value design fails the expected-loss test holds at $200 and $300 per escape and does not hold at $100, where D2 passes. These are sensitivity ranges, not confidence intervals.

Which design earns a scale pass. Under pilot-scope evidence carried to the proposed scope by a transfer record that states only a value-side ratio (59 of 59 caught at 300 cases per day), no design earns a pass (Table 7): each is insufficient or fails. If 59 live seeded attempts, none missed, are measured at the proposed scope, the following earn a pass on the risk check and the engine’s value gate: per-item approval at a served volume of 600 per day (conditional on the value basis; see below), sampled review at 1,451 reviews per day, D2 at 1,400 cases per day, and D4 with a $100 limit; D2 at the proposed scope earns one only if the automated check’s recall is signed at 0.90 or more. With an observed 0 of 59 none passes. D2 at the proposed scope, the highest-value design, does not earn a pass.

DesignScopeE0: pilot-scope evidence, value-side transfer recordE1: matched scope, 59/59E2: matched scope, 0/59Zero-miss attempts needed
D0proposed scopefail / Redesignfail / Redesignfail / Redesign–
D1aproposed scopeinsufficient (no risk-side rule) / Extend pilotfail / Redesignfail / Redesign–
D1bproposed scopeinsufficient (no risk-side rule) / Extend pilotfail (queue) / Redesignfail (queue) / Redesign–
D1bscope change: served volume 600/dayinsufficient (no risk-side rule) / Extend pilotpass / Scale{sup}†`fail / Redesign–
D1cproposed scopeinsufficient (no risk-side rule) / Extend pilotpass / Scalefail / Redesign19
D2proposed scopeinsufficient (no risk-side rule) / Extend pilotfail / Redesignfail / Redesign–
D2scope change: volume 1,400/dayinsufficient (no risk-side rule) / Extend pilotpass / Scalefail / Redesign9
D3proposed scopeinsufficient (no risk-side rule) / Extend pilotfail / Redesignfail / Redesign–
D4scope change: limit $100 (loss per escape $100 assumed)insufficient (no risk-side rule) / Extend pilotpass / Scalefail / Redesign5
D2*proposed scopeinsufficient (no risk-side rule) / Extend pilotpass / Scalefail / Redesign11

Table 7. Which designs earn a scale pass under the evidence standard. Each cell is the quantified check / the engine decision (RA’s other synthetic evidence held fixed). D1c is sampled review at the 1,451 reviews/day boundary; D2* is D2 with an assumed automated-check recall of 0.90 (above the 0.8824 break-even). The last column is the smallest number of seeded attempts, none missed, that can support the design’s required containment (blank when the stated design fails by itself). † Conditional: the engine’s value gate uses RA’s value basis (fixed 600,000 cases per year, $150,000 run cost) and ignores review cost and served volume; Table 6 gives a net value of −$21,900 for this design on the RF cost basis. See the text below.

Reading the D1b row (Table 6 against Table 7). Per-item approval with served volume restricted to 600 cases per day is a scale pass in Table 7 (matched evidence, 59 of 59) but has a net value of −$21,900 per year in Table 6. The two figures answer different questions on different cost bases. Table 6 uses the scenario RF economics (a $4 saving per case, kept on 50% of reviewed cases, so $2.00; a $1.80 review cost; a residual loss of $0.30 per served case), which gives 438,000 − 394,200 − 65,700 = −21,900. The engine’s value gate instead annualizes RA’s measured pilot effect at a fixed 600,000 cases per year against a fixed $150,000 run cost, whatever control is chosen: it includes neither the review cost nor the served volume. The engine label “SCALE” for D1b at 600 per day is therefore conditional on that value basis, not a recommendation to deploy. In RF terms each served case nets 2.00 − 1.80 − 0.30 = −$0.10 at any queue capacity, so raising throughput does not change the sign (cost and value both scale with volume). The sign changes only if the review cost falls below $1.70 (net zero; $1.50 gives +$43,800), or the share of savings kept on reviewed cases exceeds 52.5%. Before recommending this design, the value estimates must be reconciled on one scope and one cost basis; until then the pass is a statement about the risk check and RA’s value gate, and the value is not shown to be positive.

What seeded attempts can establish

Seeded attempts establish containment for the seeded mix. They do not establish prevalence, representativeness, the false-block rate on legitimate traffic or strategic behavior change; natural traffic establishes those and cannot establish containment of rare attempt types it has not yet seen. With exact one-sided bounds at 95% confidence, 29 attempts with none missed show containment of at least 90% and 59 show at least 95% (28 and 58 do not); Table 8 gives the counts with up to three misses. Expected traffic differs from the traffic needed to {em}observe{/em} enough attempts with a chosen probability: at 1% prevalence, 59 attempts are 5,900 actions in expectation but need 7,211 actions for a 95% chance of seeing at least 59. Simultaneous bounds on qq, dd and an assumed bound on ll are combined by Bonferroni; the bound on ll is an assumption and is labelled so, and conservative input bounds and tail scenarios are reported separately. Tail measures such as conditional value at risk [13] would be a policy extension, not a gate here. EVPI is only a ceiling: in a stated model (prior containment Beta(14,6), a $1.5M operating gain, expansion only if the gate passes on the posterior mean, 59 seeded attempts) EVPI is $1,139,964 and EVSI for that specific experiment is $364,648. Both depend on those synthetic assumptions.

Containment to showAttempts, 0 missed1 missed2 missed3 missedExpected traffic at q=1%q = 1\%Traffic for a 95% chance of enough attempts
90%294661762,9003,834
95%59931241535,9007,211

Table 8. Seeded attempts needed to show containment at one-sided 95% confidence (exact binomial), and the natural traffic needed to see that many unsafe attempts at 1% prevalence. Binomial and negative-binomial calculations agree.

Sources and status of rules

We separate what comes from an authoritative source from what is PilotGate prototype policy. The R-BED and V-BED presentation slide supplied by the author gives the presentation-level structure: the R1–R4 risk flow and the V1–V4 / E1–E4 value stages. We attempted to open the October 2026 R-BED release and crosswalk but could not access it, so no section or version of that release is cited. Earlier author drafts of a multiplicative combination of three value factors disagree about averaging versus maximum, and the slide does not define one; we therefore do not attribute any product of V1, V2 and V3 to R-BED and use no ordinal product. Everything below is a PilotGate policy assumption unless a source is named: the cell values of Table 1, the R1 adjustment, the cut points on the measured error rate, the appetite tiers, evidence level required by projected value size, the durability thresholds (study size, duration, 75% stability share), the transport caps and haircut, the control-test age limit, and the funding hurdle and horizon, and, for the quantified check of § 3, the appetite, the scenario inputs and the evidence standard. The deterrence threshold of that section is derived directly from the stated payoffs, with the economic model of crime [3] as context. All are configurable in fixtures/policy.json. Related work informs only general design: GRADE and evidence-standards ladders for graded evidence [7, 11, 15]; stochastic acceptability analysis and portfolio decision analysis for ranking under uncertainty [9, 14]; value-of-information theory [12]; exact binomial and rule-of-three bounds for error rates [5, 8]; Welch’s unequal-variance comparison [17]; the NIST AI RMF and the EU AI Act for risk-management context [6, 10]; and use-case prioritization from practice [1, 4].

Results on the running example

Screening and funding

Table 9 gives the pilot rung and residual risk. RA and RB reach a human-approved live pilot; RC is limited to shadow mode, and D1 (a vendor model on cardholder data with no lawful basis in the synthetic fixture) is held by a hard blocker. Table 10 shows funding, and its column “EVSI belongs to” says which experiment each EVSI figure describes. For RA, K1 and P1 it is a live pilot that the screen permits, and EVSI exceeds cost. RC. The $86,105 EVSI is the value of the designed $200,000 live pilot, which the pilot screen does not permit (RC is limited to shadow mode); RC’s EVPI bound ($88,629) is below that cost, so the live design is not recommended. The recommended $15,000 shadow run measures no realized value, so its EVSI on value is zero. What it can inform is a risk decision: whether an error-rate bound at the shadow scope could later support the pilot screen for an approval-gated live pilot. We do not model the value of that risk information, and the run has no independent value rationale. RB. The $335,104 EVSI (EVPI $341,766, cost $140,000) is the value of the modeled decision to scale at the proposed scope ($200 limit, 1,644 cases per day). That decision cannot reach SCALE there: even with every required control credited, RB’s best-case risk tier (CRITICAL) exceeds the HIGH appetite, so the structural risk rule fails the risk gate whatever the pilot measures (§ 2.2, Table 12). The figure is therefore not a funding justification for earning approval at the proposed scope. The recommendation for RB is therefore to redesign the scope, then reassess pilot funding; the code’s output for the current scope is the label REDESIGN_SCOPE_THEN_REASSESS, not a pilot approval. The $140,000 pilot (a $50 limit, 150 cases per day) could inform a redesigned lower-scope decision, but we do not compute an EVSI for a lower scope, because the fixture states no value drivers, scope or pilot design for one. Table 11 states what each experiment can teach and the decision it could change.

Use caseDesignEvid.Req.ValueRiskDecisionAnnual value (low / base / high)
RArandomizedE4E3PASSPASSSCALE$220,294 / $465,234 / $710,174
RBrandomizedE4E3INSUFFICIENTFAILREDESIGN$118,887 / $255,999 / $393,111
K1randomizedE4E3FAILINSUFFICIENTREDESIGN$-224,017 / $-161,789 / $-99,561
P1pre_postE2E3INSUFFICIENTINSUFFICIENTEXTEND_PILOT$-235,792 / $-107,026 / $21,740
RCshadowE1E4INSUFFICIENTFAILREDESIGNn/a / n/a / n/a

Table 12. Expansion decisions at the proposed scope. Annual value is the interval of incremental net value per year; Req. is the evidence level the projected value requires.

Expansion decisions

Table 12 shows the expansion decisions. RA passes both gates. RB is INSUFFICIENT on value (its pilot effect does not apply to the proposed scope) and fails risk. K1 shows that strong capability and adoption do not guarantee value: its incremental net value is negative and below the hurdle even at the optimistic end, so the value gate fails. P1 has only E2 evidence where E3 is required, so it extends its pilot. RC has only shadow evidence (E1).

Before and after

Table 14 compares expansion decisions with the 153-test revision. The software was changed (R1–R4 decision table, stage-graded evidence, mechanism model, executive actions, matched-scope case), yet no pilot-screening or expansion decision changed, and one funding recommendation did: RB moved from FUND_PILOT to “redesign the scope, then reassess pilot funding” once the structural risk finding was applied to its funding case (Table 10; see the companion changed-decision table). Table 15 gives the mutation regressions: adding a blocker, making the scope unknown, removing guardrail results, and weakening mediation each stop the expansion. Table 16 shows the scope exercise: raising volume tenfold, raising or removing the limit, sampling review, or raising authority each force reverification.

Table 10: Funding decision. EVPI is an upper bound on what perfect information could be worth. EVSI is for the specific designed experiment named in the column “EVSI belongs to”. RC’s EVSI is for the $200,000 live design, which the pilot screen does not permit; the recommended $15,000 shadow run measures no realized value, so its value-side EVSI is zero, and it informs a risk decision (whether a later approval-gated pilot may pass the screen). RB’s EVSI is the value of the modeled scale decision at the proposed scope, which cannot reach SCALE there (§ 5); it is not a funding justification at that scope. Rows follow the EVSI ranking, which does not account for RB’s structural risk finding; that is why RB is first and yet not recommended for a pilot at its proposed scope.

Use caseCostEVPI boundEVSIEVSI belongs toInformation caseRecommendation
RB$140,000 | $341,766$335,104 | live pilot; the modelled scale decision cannot reach SCALE at the proposed scope | supported | redesign the scope, then reassess pilot funding
K1$70,000 | $227,354$226,455 | live pilot, permitted | supported | fund live pilot
RA$80,000 | $157,211$151,051 | live pilot, permitted | supported | fund live pilot
RC$200,000 | $88,629$86,105 | hypothetical live design; the screen does not permit it | EVPI below cost | shadow run only
P1$60,000 | $111,678$110,154 | live pilot, permitted | supported | fund live pilot
D1$90,000 | n/a | n/a | none | not eligible |
RC shadow run$15,000 | - | $0shadow run: no value measured; informs a risk decisionrisk information onlyshadow run only

Table 10.

Use case153-test revisionThis revisionChanged
RASCALE (value PASS, risk PASS)SCALE (value PASS, risk PASS)no
RBREDESIGN (value INSUFFICIENT, risk FAIL)REDESIGN (value INSUFFICIENT, risk FAIL)no
K1REDESIGN (value FAIL, risk INSUFFICIENT)REDESIGN (value FAIL, risk INSUFFICIENT)no
P1EXTEND_PILOT (value INSUFFICIENT, risk INSUFFICIENT)EXTEND_PILOT (value INSUFFICIENT, risk INSUFFICIENT)no
RCREDESIGN (value INSUFFICIENT, risk FAIL)REDESIGN (value INSUFFICIENT, risk FAIL)no

Table 14. Expansion decisions before and after replacing the product-style rating with the R1–R4 table and stage-graded evidence. No headline decision changed; the explanations and evidence ladder did.

Mutation of the passing caseBeforeAfterExecutive action
none (control)SCALE (value PASS, risk PASS)SCALE (value PASS, risk PASS)Deploy/Scale
1. add a hard blockerSCALE (value PASS, risk PASS)BLOCKED (value PASS, risk PASS)Stop
2. operating scope unknownSCALE (value PASS, risk PASS)DEFER (value PASS, risk PASS)none (deferred)
3. remove all guardrail resultsSCALE (value PASS, risk PASS)EXTEND_PILOT (value INSUFFICIENT, risk PASS)Extend pilot with constraints
4. weaken mediation per-item to sampled, reuse evidenceSCALE (value PASS, risk PASS)REDESIGN (value INSUFFICIENT, risk FAIL)Redesign

Table 15. Regression mutations of the passing RA scale case. Each unsafe change is stopped, deferred, or reverted to an extended pilot or redesign.

Proposed scope change to RADecisionValueRisk
as evaluated (limit $500, 1,644/day, per-item) | SCALE | PASS | PASS
volume x10 (16,440/day)EXTEND_PILOTINSUFFICIENTINSUFFICIENT
limit $1,000 | EXTEND_PILOT | INSUFFICIENT | INSUFFICIENT
uncappedEXTEND_PILOTINSUFFICIENTINSUFFICIENT
sampled reviewREDESIGNINSUFFICIENTFAIL
execute with approvalREDESIGNINSUFFICIENTFAIL

Table 16. Scope exercise: each change to the evaluated scope forces re-verification (full reasons in the companion tables).

Matched-scope scale case and transferability

The matched-scope case is a synthetic matched-scope fixture that verifies the decision logic. Its evidence scope (volume and control-test scope) is set to the proposed 1,644 cases/day and a fresh synthetic sample is drawn; we do not simulate operations or control behavior at that throughput. Because the evidence scope coincides with the proposal, no transfer is needed and both gates pass. The economic result is simulated E4 evidence at the evidence scope, with an assumed run cost, and the annual figure remains a forecast. The transferred case instead takes evidence measured at 300 cases/day and relies on a stated volume-transfer assumption; Table 17 contrasts them. Table 18 stresses that assumption. Removing it, or stating it without evaluating it, gives EXTEND_PILOT. The decision holds for effect haircuts of 1.00 to 0.70 and flips at 0.50. A stricter value threshold of $150,000 still passes; $200,000 does not. The transfer rests on eight weekly observations and should be treated as limited evidence, not a validated assumption. The two gates have different transfer ranges. The risk-side record covers 300 to 1,644 cases per day, the proposed scope. The value-side record for RA covers 300 to 1,700 cases per day, with a 0.85 haircut and a maximum volume ratio of 6.0. Between 1,645 and 1,700 cases per day, with pilot-scope containment evidence only, the value gate can rely on a covered transfer while the risk gate is INSUFFICIENT because the control evidence does not apply; the decision is then EXTEND_PILOT. Above 1,700 neither gate is covered.

CaseDecisionRelies on transferEvidence scope
matched scope (evidence at 1,644/day)SCALEno1,644/day
transferred (evidence at 300/day, proposal 1,644/day)SCALEyes300/day

Table 17. Matched-scope versus transferred evidence for RA. The matched case is a fixture relabelling the evidence scope; its economics are simulated E4 and the annual figure is a forecast.

VariantDecisionValue gateLow end ($/yr)
as evaluated (volume transport, haircut 0.85, hurdle $100,000) | SCALE | PASS | 164750.0
transfer assumption removedEXTEND_PILOTINSUFFICIENT
assumption stated but not evaluatedEXTEND_PILOTINSUFFICIENT
effect haircut 1.00SCALEPASS220294.0
effect haircut 0.85SCALEPASS164750.0
effect haircut 0.70SCALEPASS109206.0
effect haircut 0.50EXTEND_PILOTINSUFFICIENT35147.0
effect haircut 0.30EXTEND_PILOTINSUFFICIENT-38912.0
stricter value threshold $150,000 | SCALE | PASS | 164750.0
stricter value threshold $200,000 | EXTEND_PILOT | INSUFFICIENT | 164750.0
stricter value threshold $250,000 | EXTEND_PILOT | INSUFFICIENT | 164750.0

Table 18. Transferability stress test of the 300 to 1,644 cases/day expansion. The transfer rests on eight weekly observations and is limited evidence.

Evaluation example: a public AWS sample

To show how a single-pass score differs, we reproduce the prioritization in a public AWS sample [2] on a 26-criterion synthetic fixture. As implemented it returns a composite of 4.4, a risk proxy of 25 out of 25 and readiness 5.0, and the verdict is CONDITIONAL GO, because its NO-GO branch needs all three conditions together. Its documentation describes any one condition as sufficient, which gives NO GO. We compare both with the comment-derived OR variant and do not treat either as an authorization (Table 19). Table 20 places these beside the framework: for RC the OR rule says NO GO while the framework gives a shadow pilot at intake and REDESIGN at the larger scope. The point is not that one system is right, but that a use-case score does not say at which scope the use case is acceptable.

VariantCompositeRiskReadinessVerdict
AWS as implemented (NO GO needs all three)4.4255.0CONDITIONAL GO
AWS with documented OR rule (any one)4.4255.0NO GO

Table 19. Reproduction of the AWS sample prioritization on the 26-criterion fixture. Neither verdict is an authorization.

Use caseStateCompositeRiskAWS as implementedAWS OR ruleThis framework
RAintake3.389CONDITIONAL GOCONDITIONAL GOHUMAN_APPROVED_LIVE
RApost_pilot3.566GOGOSCALE
RBintake2.982512CONDITIONAL GOCONDITIONAL GOHUMAN_APPROVED_LIVE
RBpost_pilot3.057512CONDITIONAL GOCONDITIONAL GOREDESIGN
RCintake3.04525CONDITIONAL GONO GOSHADOW
RCpost_pilot3.07525CONDITIONAL GONO GOREDESIGN
P1intake3.274CONDITIONAL GOCONDITIONAL GOHUMAN_APPROVED_LIVE
P1post_pilot3.10516CONDITIONAL GOCONDITIONAL GOEXTEND_PILOT
K1intake3.26754CONDITIONAL GOCONDITIONAL GOHUMAN_APPROVED_LIVE
K1post_pilot3.25259CONDITIONAL GOCONDITIONAL GOREDESIGN
D1intake2.5815CONDITIONAL GOCONDITIONAL GOHOLD

Table 20. Three-way comparison on the synthetic use cases: AWS as implemented, AWS under the documented OR rule, and this framework at the proposed scope.

Verification, scope of claims and limitations

Federal intake. A pinned copy of the 2025 federal inventory [16] has 3,611 rows and 2,901 unique identifiers; 152 are in pre-deployment or pilot stage. A deterministic stratified selection of 12 records (Table 21) is used for intake triage only: agency-reported benefits are treated as intake hypotheses with no evidence level, and rankings driven by proxy fields are exploratory.

Inventory keyUse caseBenefit claimPilot rungMedian rank
VA|VA-25-5578AICESqualitativeSIMULATION11.0
VA|VA-25-3732Modeling to Learn (MTL)qualitativeSIMULATION11.0
VA|VA-25-3480Smart Pension AutomationquantifiedSIMULATION8.0
SBA|SBA-30SBA.gov External Customer ChatbotquantifiedHOLD3.0
SBA|SBA-35AI Incident SummarizationquantifiedHOLD3.0
SBA|SBA-13Generative AI for SBA.gov product trainingqualitativeHOLD5.0
VA|VA-25-1184Ask VA Inquiry Automated Category ClassificationquantifiedSIMULATION8.0
NASA|NASA-744Enhancements to T-Rex and D-RexquantifiedSIMULATION8.0
NASA|NASA-743Ask Tech PortquantifiedSIMULATION8.0
TREAS|REQ-2025-02087AI-assisted Procurement POCqualitativeSIMULATION6.0
EPA|27AI Assistants in Esri ToolsqualitativeSIMULATION2.0
SBA|SBA-23Email, Document, and Composing AI EditorqualitativeSIMULATION2.0

Table 21. Twelve deterministically selected inventory records used for intake triage only. Reported benefits are hypotheses with no evidence level.

What the evidence is. We distinguish three classes of claim.

  • Software verification. The package passes 279 tests with the pinned federal inventory present; without it 275 pass and 4 are skipped, and skipped tests are reported as skipped, not passed (153 in the baseline revision, reproduced before any edit). They cover the decision table, precedence, evidence applicability, mechanisms, executive actions, funding, regressions, and the quantified check, including three regressions in which an unsupported control credit had returned SCALE (Appendix § 11).

  • Synthetic demonstrations. All refund, reconciliation, knowledge-assistant and dispute cases, the assumed 5% run-cost drift (so E4 economics are simulated), and the control tests are synthetic fixtures. They show that the logic behaves as designed, not that any organization’s numbers look like these.

  • Claims requiring field validation. That the R1–R4 cell values, appetite tiers, thresholds and transfer assumptions suit a real organization, and that the stage ladder predicts realized value, are untested here.

Use caseExperimentDecision that could changeIndependent rationale
RAlive pilot at the pilot scope (3000 cases per arm, 8 weeks; a concurrent control is neededscale versus extend, redesign or stop at the proposed scope (the value gate)direct operating benefit during the pilot (assumption)
RBlive pilot at the pilot scope (2000 cases per arm, 8 weeks; a concurrent control is neededa redesigned, lower-scope expansion decision. The pilot cannot earn approval at the proposed scope: best-case risk tier CRITICAL exceeds the HIGH appetite even with every required control crediteddirect operating benefit during the pilot (assumption)
RCshadow run on live traffic: the AI decides, nothing is executedthe pilot screen for the next rung (whether a live approval-gated pilot may be authorized), not the scale decisionnone stated
P1live pilot at the pilot scope (600 cases per arm, 6 weeks; a concurrent control is neededscale versus extend, redesign or stop at the proposed scope (the value gate)direct operating benefit during the pilot (assumption); Regulatory exam finding on reconci
K1live pilot at the pilot scope (6000 cases per arm, 8 weeks; a concurrent control is neededscale versus extend, redesign or stop at the proposed scope (the value gate)direct operating benefit during the pilot (assumption)

Table 11. What each funded experiment can teach and the decision it could change.

Limitations. The risk table and appetite are prototype policy; the October 2026 R-BED release was not accessible; the transfer assumption rests on eight weekly points; the matched-scope case uses a re-scoped control-test set; the quantified check treats the attempt rate and the loss per escape as stated inputs and every scenario parameter is synthetic; the $100-limit result rests on an assumed loss per escape; the pessimistic annualization corner of RA sits at or below the hurdle, and whether that corner should gate SCALE is a policy question we leave open; and funding is deliberately conservative about learning. The framework supports a funding and scoping discussion; it is not a compliance determination.

Fixtures behind the tables. Two synthetic RA pilots are used, and both return SCALE. The canonical pilot (synth.build_all(), 3,044 treated cases, ITT effect ratios 0.6172, 1.0254 and 1.4336) generates the expansion, stage, regression, scope, stress and matched tables (Tables 12, 13, 14, 15, 16, 18 and, against a freshly drawn matched sample, Table 17). The test-helper pilot (3,000 cases per arm, ratios 0.60, 1.00 and 1.40) generates the quantified-engine tables (Tables 5, 7 and 22). The RF design comparison, the worked scenario and the sensitivity tables are closed-form and use no pilot. The fixtures differ only in the low effect ratio, so the gate decision agrees while the extrapolated low end differs: 0.85×0.60×600,000−150,000=$156,0000.85 \times0.60 \times600,000 - 150,000 = \$156,000 for the test-helper pilot and $164,750 for the canonical pilot (Table 18, baseline row).

Reproducibility. The companion package regenerates every table and figure with python reproduce.py and this source, including Figures 3 and 4, with python paper/build_paper.py, which fails if prose diverges from results. Pinned inputs are AWS revision 7a0f137, federal inventory commit 06c7ebe, and fixture hashes in results/environment.json.

Author information and disclosures

Author. Vishal Srivastava, Whiting School of Engineering, Johns Hopkins University, Baltimore, MD 21218, USA (ORCID 0000-0002-5651-8365; [email protected]). R-BED and V-BED are frameworks developed by the author; the rules built on them here are PilotGate policy unless a source is named (§ 4).

Funding. This research received no external funding.

Competing interests. The author declares no competing interests.

Use of AI tools. The author used Claude (Anthropic) to assist with drafting, code, tests and verification of this paper and its companion package. The author reviewed the output and is responsible for the content.

Data and code. All demonstrations use synthetic data, except the public inventories named in § 4. No human subjects were involved. The companion package regenerates every table and figure (see Reproducibility).

References

Derivations

Required containment and volume ceiling. With N=365VN = 365V, L=Nq(1−d)l≤AL = Nq(1-d)l \le A gives d≥1−A/(365Vql)d \ge1 - A/(365Vql), which is zero while V≤A/(365ql)=410.96V \le A/(365ql) = 410.96 cases per day (so no containment is required at 300) and 0.7500 at 1,644. Under sampled review d(V)=ρmin⁡{1,B/V}d(V) = \rho\min\{1, B/V\}. For V≥BV \ge B the test 365ql(V−ρB)≤A365ql(V-\rho B) \le A gives the single ceiling Vmax⁡=ρB+A/(365ql)=510+410.96=920.96V_{\max} = \rho B + A/(365ql) = 510 + 410.96 = 920.96. The expected loss is continuous at V=BV = B and increasing in VV, so there is one ceiling and the test fails above it.

Minimum capacity. At V=1,644V = 1,644 the requirement ρB/V≥dmin⁡\rho B/V \ge d_{\min} gives B≥dmin⁡V/ρ=1,450.64B \ge d_{\min}V/\rho= 1,450.64, so 1,451 is the smallest whole number of reviews per day.

Break-even recall. For D2 the containment is rρr\rho with automated-check recall rr (while flagged items fit within review capacity), so the expected loss equals the appetite at r∗=dmin⁡/ρ=0.8824r^{*} = d_{\min}/\rho= 0.8824.

Exact bounds. If xx of nn seeded attempts are caught, the one-sided lower bound on containment at level α\alpha is α1/n\alpha^{1/n} when x=nx = n and the Clopper–Pearson beta quantile otherwise [5]. Showing containment of at least d0d_0 with none missed needs n≥⌈∣ln⁡α∣/ln⁡d0⌉n \ge\lceil|\ln\alpha|/\ln d_0\rceil attempts: 29 for d0=0.90d_0 = 0.90 and 59 for 0.95. With zero observed failures in general the same logic gives the rule-of-three style bound [8]. When two stages are credited, each bound uses α/2\alpha/2 (Bonferroni); stage independence is never assumed, because the second stage is measured among the escapes of the first.

The evidence standard and its status codes

Table 22 lists the three bypasses found in an independent review of an earlier implementation and what they return under the standard of § 3.2. In the first, a pipeline with first-stage containment 1 and no evidence passed; in the second, point assumptions overrode an observed 0 of 59; in the third, a transfer record’s value-side effect factor 0.85 was ignored, so the gate used the unadjusted recall. The stable status codes and their effect in the decision path are:

  • QL_PASS: this check passes (it satisfies that check only).

  • QL_FAIL_EXPECTED_LOSS, QL_FAIL_CAPACITY_QUEUE: the risk gate fails, so REDESIGN.

  • QL_INSUFFICIENT_NO_SPEC, QL_INSUFFICIENT_NO_APPETITE: required inputs or the appetite are missing.

  • QL_INSUFFICIENT_EVIDENCE_NOT_APPLICABLE: the evidence does not cover the proposed scope.

  • QL_INSUFFICIENT_TRANSPORT_RISK_RULE: coverage rests on a transfer that states no risk-side recall ratio.

  • QL_INSUFFICIENT_EVIDENCE_STRENGTH: the design passes, the evidence is too weak to show it.

Three layers must be kept apart. (i) The quantified check returns one status code. It assesses whether the evidence applies to the proposed scope before it assesses the design, so inapplicable evidence returns QL_INSUFFICIENT_EVIDENCE_NOT_APPLICABLE even where the stated design would fail. (ii) The risk gate combines that code with the ordinal rating, the vetoes and the mandatory controls. A failed mandatory control, a hard veto, a best-case tier above appetite or a quantified FAIL makes the gate FAIL; only if none of these holds does a quantified INSUFFICIENT make the gate INSUFFICIENT. (iii) The final action follows the decision precedence of Section 2: a risk FAIL gives REDESIGN and an INSUFFICIENT gate gives EXTEND_PILOT. A code therefore does not determine the action by itself. In the sampled-review design D1C (600 reviews per day, 1,644 cases per day, appetite $300,000), the stated design’s modeled loss is $827,820 in both of the following cases. With 59 of 59 caught at the pilot scope and no applicable transfer, the check is QL_INSUFFICIENT_EVIDENCE_NOT_APPLICABLE, reports no credited loss, and the decision is EXTEND_PILOT. With 59 of 59 caught at the matched scope the evidence applies, the credited loss still exceeds the appetite, the check is QL_FAIL_EXPECTED_LOSS and the decision is REDESIGN. The statement that every INSUFFICIENT code yields EXTEND_PILOT holds only when no FAIL condition applies elsewhere in the risk gate. This describes the implemented behavior; no rule was changed. A required gate accepts only the evidence basis; point values are for planning and figures, and a required gate configured on them raises an error.

Case (RA at 1,644 cases/day)Before the evidence standardAfter
1 pipeline d1 = 1, evidence = Nonepass / Scaleinsufficient (evidence not applicable) / Extend pilot
2 1,451 reviews/day, observed 0/59 caughtpass / Scalefail / Redesign
3 1,451 reviews/day, pilot-scope 59/59, transport record effect factor 0.85pass / Scaleinsufficient (no risk-side rule) / Extend pilot

Table 22. Three bypasses found in independent review of an earlier implementation of the quantified check, each of which returned SCALE, and their outcomes under the evidence standard.

Synthetic inputs and accepted evidence

  • Synthetic scenario inputs (approved for implementation and sensitivity analysis, not calibrations): qq, ll, AA, ρ\rho, BB, the automated-check recall and flag rate, the share of savings kept on reviewed cases, the recovery rate, the effect of a limit on loss per escape, the EVSI prior and gain, and the deterrence payoffs.

  • Evidence the required gate accepts: observed, live seeded-attempt counts whose scope covers the proposed scope (directly, or through an evaluated transfer that states a risk-side ratio), credited at the exact lower bound and never above the design value. These bound recall, pipeline containment, detection, recovery and (with an evaluated payoff assumption) deterrence.

  • Stated inputs that the gate does not test: the attempt rate qq and the loss per escape ll. Their values drive the result and need their own evidence in an application.

References

  1. [1]appliedAI Initiative. How to find and prioritize AI use cases. Applying ai whitepaper, appliedAI Initiative, jul 2020.
  2. [2]AWS Samples. sample-agentic-value-accelerator, file platform/control plane/backend/src/models/prioritization.py, 2026. Source revision 7a0f137c69ba3542203c36e4218c529d5aba2a4e.
  3. [3]Gary S. Becker. Crime and punishment: An economic approach. Journal of Political Economy, 76 (2):169–217, 1968. doi: 10.1086/259394.DOI
  4. [4]Frank Bodendorf. A data driven use case planning and assessment approach for AI portfolio management. Electronic Markets, 35(1):22, 2025. doi: 10.1007/s12525-025-00759-x. Read as abstract and record only.DOI
  5. [5]C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413, 1934. doi: 10.2307/2331986.DOI
  6. [6]European Parliament and Council of the European Union. Regulation (eu) 2024/1689 laying down harmonised rules on artificial intelligence (artificial intelligence act). Official Journal of the European Union, 13 June 2024, 2024.
  7. [7]Gordon H. Guyatt, Andrew D. Oxman, Gunn E. Vist, Regina Kunz, Yngve Falck-Ytter, Pablo Alonso-Coello, and Holger J. Schünemann. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ, 336(7650):924–926, 2008. doi: 10.1136/bmj.39489.470347.AD.DOI
  8. [8]James A. Hanley and Abby Lippman-Hand. If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA, 249(13):1743–1745, 1983. doi: 10.1001/jama.1983.03330370053031.DOI
  9. [9]Risto Lahdelma, Joonas Hokkanen, and Pekka Salminen. SMAA: stochastic multiobjective acceptability analysis. European Journal of Operational Research, 106(1):137–143, 1998. doi: 10.1016/S0377-2217(97)00163-X.DOI
  10. [10]National Institute of Standards and Technology. Artificial intelligence risk management framework (AI RMF 1.0). Technical Report AI 100-1, NIST, 2023.
  11. [11]Ruth Puttick and Joe Ludlow. Standards of evidence: An approach that balances the need for evidence with innovation. Technical report, Nesta, 2013.
  12. [12]Howard Raiffa and Robert Schlaifer. Applied Statistical Decision Theory. Harvard University, Division of Research, Graduate School of Business Administration, Boston, 1961.
  13. [13]R. Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of Risk, 2(3):21–41, 2000.DOI
  14. [14]Ahti Salo, Jeffrey Keisler, and Alec Morton, editors. Portfolio Decision Analysis: Improved Methods for Resource Allocation. Springer, 2011. ISBN 978-1-4419-9942-9.DOI
  15. [15]Lawrence W. Sherman, Denise C. Gottfredson, Doris L. MacKenzie, John Eck, Peter Reuter, and Shawn D. Bushway. Preventing crime: What works, what doesn’t, what’s promising. Technical report, National Institute of Justice, Report to the U.S. Congress, 1997.

Paper details

vlnc:W_t8kgtx7n
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents