betaFeedback

WorksComputer Science

Risk-certified prompt optimization for LLM-as-a-judge guardrails

Abstract

LLM-as-a-judge guardrails are often controlled primarily through system prompts, yet standard prompt optimization methods target average performance rather than explicit safety constraints. This paper studies system-prompt selection as a constrained optimization problem: maximize benign pass rate while keeping the missed-attack rate below a user-specified level α with finite-sample statistical control. The framework combines LLM-based judging, automatic prompt generation, and distribution-free risk certification. This work instantiates it in two procedures: CPS and CRISP. CPS generates a fixed pool of prompts from training data, then certifies candidates on a held-out evaluation set using simultaneous upper confidence bounds and selects the certified prompt with the highest benign utility. CRISP instead integrates risk feedback into an adaptive search, using hard example-guided rewriting and a reusable-holdout interface to steer generation toward the feasible utility frontier. Experiments on four datasets covering prompt injection, system-prompt leakage, and sensitive-information disclosure show that risk-certified selection consistently respects the target missed-attack level on heldout tests, while utility improves as the permitted risk is relaxed. CRISP is especially useful when fixed-pool search fails to produce certifiable high-utility candidates.

Introduction

Guardrails are an increasingly important component of LLM-based systems, where they act as filters that inspect user inputs or model outputs and decide whether they should be allowed or blocked. They typically operate as one component within a broader AI pipeline. Existing guardrails take several forms. Some rely on fine-tuned models trained specifically to classify content as safe or unsafe. Another practical approach is the LLM-as-a-judge guardrail, in which a general-purpose language model (e.g., gpt-5.4) is turned into a guardrail primarily through its system prompt, without modifying the model weights. The system prompt instructs the model how to interpret a safety policy and output a binary decision such as BLOCK or ALLOW. This paper focuses on the second setting and asks how the system prompt itself can be optimized while providing a statistically certified bound on missed-attack risk [36, 38, 12].

Designing such a guardrail involves an asymmetry between the two possible types of error. Missing a malicious request can expose the underlying system to an attack, whereas incorrectly blocking a benign request primarily reduces usability. Consequently, minimizing average classification error is not necessarily the appropriate objective. In many deployments, practitioners may instead wish to specify an explicit risk level α\alpha, and require the missed-attack rate to remain below this level while allowing as much benign traffic as possible. Guardrail design therefore naturally becomes a constrained optimization problem: maximize benign utility subject to R≤αR \le\alpha, where RR denotes the missed-attack rate. Importantly, this requirement should hold not only on the finite dataset used during development, but also on future samples from the deployment distribution. This is called risk control, where the goal is to ensure that the deployed guardrail has risk at most α\alpha with probability at least 1−δ1-\delta over the calibration data [6, 4].

Distribution-free risk control is well understood when the classifier exposes a continuous score. Conformal prediction [30, 3] and its risk-controlling extensions turn a one-dimensional threshold into a certified operating point: sweep the threshold on held-out calibration data and return the most permissive value whose empirical risk survives a statistical test [6, 4, 2, 26]. However, a deployed guardrail detector often exposes only a binary flag or a coarse severity level [21], and many hosted LLMs do not expose logits: the Claude API has no logprobs parameter [5], and OpenAI reasoning models reject logprobs [22]. If a score threshold is unavailable, the remaining control surface is the system prompt itself. Risk control then becomes a search problem: among the prompts we can write, find one whose missed-attack rate is certified below α\alpha and whose benign pass rate is as high as possible. This is harder than threshold tuning, because prompt space is discrete and unordered.

Our claim: A general-purpose LLM (e.g., gpt-5.4-mini) can be turned into a guardrail with statistically guaranteed risk control, through system-prompt optimization alone and without modifying model weights. We formulate this as risk-certified system-prompt optimization: find a prompt that maximizes benign utility UU subject to a safety constraint R≤αR \le\alpha, where UU is the benign query pass rate and RR the missed-attack rate. The selected prompt must satisfy this constraint with high probability over the calibration data.

Our proposal: We combine three ingredients: an LLM-as-a-judge guardrail, prompt optimization, and distribution-free risk control for certification. We instantiate this framework in two procedures that differ in how generation and certification interact:

  • CPS: Optimize, then Certify (Section 4.1). CPS first generates a fixed, diverse pool of candidate prompts using an existing prompt optimizer. It then applies Learn-then-Test Angelopoulos et al., [4], with Bonferroni multiple-testing correction across candidates, to certify prompts whose risk remains below α\alpha, and selects the certified prompt with the highest empirical utility.

  • CRISP: Risk-aware closed-loop optimization (Section 4.2). CPS can only certify prompts already present in its fixed pool; if no high-utility feasible prompt is generated, certification cannot recover one. CRISP addresses this limitation by integrating risk feedback into prompt generation. Using the reusable-holdout mechanism Guess-and-Check Dwork et al., [9]; Rogers et al. [28], CRISP iteratively rewrites prompts toward the constrained Pareto frontier while preserving valid risk guarantees.

Our Contributions

  1. We formulate LLM guardrail prompt design as risk-certified system-prompt optimization and introduce an (α,ϵ,δ)(\alpha,\epsilon,\delta)-validity guarantee (Definition 1).

  2. We develop CPS and CRISP to build prompt-based guardrails with explicit finite-sample risk guarantees and show that the framework is adaptable to multiple threat domains by testing it on prompt injection, system prompt leakage, and sensitive information disclosure.

  3. Methodologically, we extend the LTT-style setting beyond a fixed candidate family to sequential, adaptive candidate selection, where the set of tested hypotheses is generated based on calibration feedback and the desired risk guarantee.

Background and Related Work

Our framework lies at the intersection of three research areas: LLM safety, automatic prompt optimization, and distribution-free risk control.

Guardrails for LLM Systems Deployed guardrails come in three forms. First, fine-tuned classifiers (ProtectAI’s DeBERTa detector [24], Prompt Guard [20], PIGuard [19]) are fast and cheap but tied to their training distribution: a new policy or attack family requires relabeling and retraining. Second, safety-tuned LLMs such as Llama Guard [16] and ShieldGemma [35] apply a fixed policy taxonomy; NeMo Guardrails [27] instead lets developers script safety flows. Third, LLM-as-a-judge setting [36, 12], where a general-purpose model makes the block/pass decision straight from a system prompt, which is cheap to adapt but sensitive to small edits [29, 31]. Across all these designs the policy prompt is hand-written and then frozen; none gives a procedure for choosing it under an explicit, statistically enforced risk level.

Automatic Prompt Optimization. These methods search over natural-language instructions to improve performance on a downstream task. APE [37] generates and reranks instructions; OPRO [32] proposes prompts from a scored history; EvoPrompt [13] and PromptBreeder [10] apply mutation and crossover; ProTeGi [25] uses textual gradients; MIPROv2 [23] tunes instructions and demonstrations jointly; GEPA [1] adds reflective feedback with Pareto-based selection. All of them search for prompts that maximize an unconstrained average score and return a single winner; they do not directly enforce a requirement such as R≤αR \le\alpha, nor do they provide a finite-sample guarantee that the deployed prompt will satisfy that requirement. We use them as candidate generators and impose the constraint ourselves.

Risk Control. Distribution-free risk control uses a finite i.i.d. calibration sample to return a configuration whose population risk is at most α\alpha with probability at least 1−δ1-\delta. Risk-Controlling Prediction Sets [6] calibrate a scalar threshold under a monotone loss. Learn-then-Test (LTT) [4] drops monotonicity: each candidate configuration is tested against the null hypothesis that it violates the constraint, and a family-wise error rate correction (e.g., Bonferroni or Holm) ensures that all certified candidates are simultaneously feasible with probability at least 1−δ1-\delta. Prompt Risk Control [39] uses this framework to select an LLM prompt from a fixed candidate set.

In each case, the calibrated object is a threshold on a continuous score or a candidate set fixed in advance, whereas the space of guardrail prompts is discrete, unordered, and open-ended. CPS generates its pool of prompts from separate training data and fixes it before certification, so LTT applies directly using calibration data. CRISP instead generates each new prompt from earlier certification outcomes, which makes the tested hypotheses depend on the calibration data; to keep the guarantee valid, it accesses the calibration set during search only through the reusable-holdout mechanism Guess-and-Check [28]. This notion of adaptivity differs from that of anytime-valid inference, e-process-based methods such as adaptive LTT [34]: these fix the hypotheses and test them on sequentially arriving data, whereas CRISP fixes the data and generates the hypotheses sequentially.

Problem Setup

Let xx denote a user query and y∈{0,1}y \in\{0,1\} its label, where y=1y=1 indicates a malicious input that should be blocked and y=0y=0 indicates a benign input that should be allowed. Given an LLM judge ff and a system prompt p∈Pp \in\mathcal{P}, the prompt induces a binary classifier f(p,x)∈{0,1}f(p,x) \in\{0,1\}, where 1 denotes block and 0 denotes pass. f(p,⋅)f(p,\cdot) is deterministic (greedy decoding, temperature=0).

Safety Risk and Benign Utility

Given a labeled dataset sample Dn={(Xi,Yi)}i=1nD_n=\{(X_i,Y_i)\}_{i=1}^n sampled i.i.d. from an unknown distribution π\pi, we define the safety risk as the attack (false-negative) rate on malicious inputs and the utility as the benign pass rate (specificity) on benign inputs:

R(p)=Pr⁡(x,y)∼π(f(p,x)=0∣y=1),U(p)=Pr⁡(x,y)∼π(f(p,x)=0∣y=0).(1)R(p)=\Pr_{(x,y)\sim\pi}\left(f(p,x)=0\mid y=1\right),\qquad U(p)=\Pr_{(x,y)\sim\pi}\left(f(p,x)=0\mid y=0\right). \tag*{(1)}

The corresponding empirical missed-attack and benign pass rates on the dataset DnD_n are:

R^(p,Dn)=∑i:Yi=11[f(p,Xi)=0]∑i=1n1[Yi=1],U^(p,Dn)=∑i:Yi=01[f(p,Xi)=0]∑i=1n1[Yi=0].(2)\widehat{R}(p,D_n)=\frac{\sum_{i:Y_i=1}\mathbb{1}[f(p,X_i)=0]}{\sum_{i=1}^{n}\mathbb{1}[Y_i=1]},\qquad\widehat{U}(p,D_n)=\frac{\sum_{i:Y_i=0}\mathbb{1}[f(p,X_i)=0]}{\sum_{i=1}^{n}\mathbb{1}[Y_i=0]}. \tag*{(2)}

Risk-Constrained Prompt Optimization

If the population distribution π\pi is known, the desired prompt p∗p^{*} would solve the problem

max⁡p∈PU(p)s.t.R(p)≤α,(3)\max_{p\in\mathcal{P}} U(p) \quad\text{s.t.} \quad R(p) \leq\alpha, \tag*{(3)}

where α∈(0,1)\alpha\in(0,1) is a user-specified risk level. Thus, among prompts that satisfy the required safety level, we seek the one that preserves as much benign traffic as possible. This formulation differs from standard prompt optimization because safety risk is treated as a hard constraint, rather than being combined with utility into an unconstrained average-performance objective.

Statistical Objective

In practice, the distribution π\pi is unknown and we only have access to a finite labeled dataset, so the population constraint R(p)≤αR(p) \leq\alpha cannot be verified directly. Our statistical objective is therefore to select a data-dependent prompt p^\hat{p} that achieves high benign utility while satisfying the risk constraint with probability at least 1−δ1-\delta.

Definition 1 ((α,ϵ,δ\alpha,\epsilon,\delta)-valid selection). A procedure A\mathcal{A} maps a dataset Dn={(Xi,Yi)}i=1nD_n=\{(X_i,Y_i)\}_{i=1}^{n} to a pair (C^,p^)=A(Dn)(\widehat{C},\hat{p})=\mathcal{A}(D_n), where C^=C^(Dn)⊆P\widehat{C}=\widehat{C}(D_n)\subseteq\mathcal{P} is a finite set of candidates and p^∈C^\hat{p}\in\widehat{C} is the deployed prompt. For α,δ∈(0,1)\alpha,\delta\in(0,1) and a tolerance ϵ=ϵ(Dn)≥0\epsilon=\epsilon(D_n)\geq0, we say that A\mathcal{A} is (α,ϵ,δ)(\alpha,\epsilon,\delta)-valid if:

Pr⁡Dn∼πn(max⁡p∈C^R(p)≤α and U(p^)≥max⁡p∈C^U(p)−ϵ)≥1−δ.(4)\Pr_{D_n\sim\pi^n}\left(\max_{p\in\widehat{C}}R(p)\leq\alpha\ \text{and}\ U(\hat{p})\geq\max_{p\in\widehat{C}}U(p)-\epsilon\right)\geq1-\delta. \tag*{(4)}

Methods

We propose two methods that differ in how candidate generation and risk certification interact. We draw Dn∼πnD_n\sim\pi^n i.i.d. and partition it into three disjoint subsets: DtrainD_{\mathrm{train}} (used by the generator), DcertD_{\mathrm{cert}} (used only for certification and prompt selection), and DtestD_{\mathrm{test}} (used only for final evaluation). We further partition DcertD_{\mathrm{cert}} by label:

Dcertmal={(Xi,Yi)∈Dcert:Yi=1},Dcertben={(Xi,Yi)∈Dcert:Yi=0}(5)D_{\mathrm{cert}}^{\mathrm{mal}}=\{(X_i,Y_i)\in D_{\mathrm{cert}}:Y_i=1\}, \qquad D_{\mathrm{cert}}^{\mathrm{ben}}=\{(X_i,Y_i)\in D_{\mathrm{cert}}:Y_i=0\} \tag*{(5)}

and write m:=∣Dcertmal∣m:=\lvert D_{\mathrm{cert}}^{\mathrm{mal}}\rvert and b:=∣Dcertben∣b:=\lvert D_{\mathrm{cert}}^{\mathrm{ben}}\rvert.

Clopper–Pearson bounds. In both CPS and CRISP, we use the Clopper–Pearson bound [7] to obtain an upper confidence bound on risk at failure level β∈(0,1)\beta\in(0,1):

Rucb(p,Dcert,β)=Beta⁡−1(1−β;kp+1,m−kp),kp=mR^(p,Dcert),(6)R_{\mathrm{ucb}}(p,D_{\mathrm{cert}},\beta)=\operatorname{Beta}^{-1}(1-\beta;k_p+1,m-k_p), \qquad k_p=m\widehat{R}(p,D_{\mathrm{cert}}), \tag*{(6)}

where mm is the number of malicious examples and kpk_p is the number of missed attacks (false negatives).

Assumption 1 (Calibration sampling). Conditional on the label counts (m,b)(m,b), the subsamples DcertmalD_{\mathrm{cert}}^{\mathrm{mal}} and DcertbenD_{\mathrm{cert}}^{\mathrm{ben}} are independent i.i.d. samples from π(⋅∣y=1)\pi(\cdot\mid y=1) and π(⋅∣y=0)\pi(\cdot\mid y=0), respectively.

CPS: Certified Prompt Search

Idea. CPS separates prompt generation from statistical certification. A prompt optimizer first generates a fixed candidate pool using only DtrainD_{\mathrm{train}}. CPS then evaluates these candidates on the held-out DcertD_{\mathrm{cert}}, certifies those whose risk upper confidence bound is below the target α\alpha, and selects the certified prompt with the highest benign utility.

Procedure.

  1. Generate a fixed candidate pool. Draw NN bootstrap resamples of size BB from DtrainD_{\mathrm{train}}, giving {D1,D2,…,DN}\{D_1,D_2,\ldots,D_N\}, and run a prompt optimizer (e.g., GEPA) on each to obtain

Ptrain={p1,…,pN},pi=GEPA⁡(Di).(7)\mathcal{P}_{\mathrm{train}}=\{p_1,\ldots,p_N\}, \qquad p_i=\operatorname{GEPA}(D_i). \tag*{(7)}
  1. Compute risk certificates. For each pj∈Ptrainp_j\in\mathcal{P}_{\mathrm{train}}, evaluate the risk upper confidence bound on DcertD_{\mathrm{cert}} using equation (6) with the Bonferroni corrected level δR/N\delta_R/N [8]:

Rucb(pj)=Rucb(pj,Dcert,δR/N)R_{\mathrm{ucb}}(p_j)=R_{\mathrm{ucb}}(p_j,D_{\mathrm{cert}},\delta_R/N)

.

Figure 1. CPS: fixed-pool prompt generation, risk certification, and utility-based selection.

  1. Form the certified set. Retain candidates whose upper risk bound satisfies the target:

C={pj∈Ptrain:Rucb(pj)≤α}.(8)\mathcal{C} = \{p_j \in\mathcal{P}_{\mathrm{train}} : R_{\mathrm{ucb}}(p_j) \le\alpha\}. \tag*{(8)}
  1. Select by benign utility. Evaluate the empirical benign utility of each certified candidate on the benign examples in DcertD_{\mathrm{cert}}, and return

p^=arg⁡max⁡pj∈CU^(pj,Dcert).(9)\hat{p} = \mathop{\arg\max}_{p_j \in\mathcal{C}} \widehat{U}(p_j, D_{\mathrm{cert}}). \tag*{(9)}

Proposition 1 (CPS risk control and utility). Under Assumption 1, for the input failure level δR\delta_R, any δU∈(0,1)\delta_U \in(0,1), and ϵU=2log⁡(2N/δU)b\epsilon_U = \sqrt{\frac{2\log(2N/\delta_U)}{b}}, the prompt p^\hat{p} selected by CPS satisfies:

Pr⁡Dcertmal∼π1m, Dcertben∼π0b(R(p^)≤α and U(p^)≥max⁡p∈CU(p)−ϵU)≥1−δR−δU,(10)\Pr_{D_{\mathrm{cert}}^{\mathrm{mal}} \sim\pi_1^m,\ D_{\mathrm{cert}}^{\mathrm{ben}} \sim\pi_0^b}\left(R(\hat{p}) \le\alpha\text{ and } U(\hat{p}) \ge\max_{p \in\mathcal{C}} U(p) - \epsilon_U\right) \ge1-\delta_R-\delta_U, \tag*{(10)}

with the convention that when C=∅\mathcal{C} = \varnothing, CPS abstains and the event holds. (Proof in Appendix B.1).

CRISP: Constrained Risk-calibrated Iterative System-Prompt Optimization

Motivation. CPS generates a fixed pool of prompts before certification. While this makes statistical control straightforward, it creates a generation gap: if the initial pool contains no prompt satisfying the risk constraint, certification alone cannot produce one. CRISP closes this gap by feeding risk certificates back into prompt generation, allowing the search to move toward high-utility prompts near the certification boundary.

Two-population constrained search. Inspired by FI-2pop (Kimbrough et al., [18]), at iteration tt CRISP partitions the explored prompts Pt\mathcal{P}_t into a certified set Ct\mathcal{C}_t (Rucb(p)≤αR_{\mathrm{ucb}}(p) \le\alpha) and an uncertified set Ut\mathcal{U}_t (Pt∖Ct\mathcal{P}_t \setminus\mathcal{C}_t). The two populations serve different search objectives: certified prompts are optimized for utility, while uncertified prompts are optimized to reduce constraint violation. This lets CRISP improve prompts already inside the feasible region while simultaneously pushing promising infeasible prompts toward certification.

Algorithm. Each round of Algorithm 1 makes up to two proposals. IMPROVEUTILITY selects the certified prompt with the highest training utility and rewrites it to admit benign cases that are currently over-blocked. REDUCERISK selects the uncertified prompt with the lowest training risk and rewrites it to block attacks that are currently missed. Each rewrite is a single LLM call that receives the parent prompt, its current risk, the budget α\alpha, and a few hard examples mined only from DtrainD_{\mathrm{train}} (templates in Appendix A.6). Every proposed prompt is then evaluated by Guess-and-Check (GnC) (Rogers et al., [28]) to give an upper confidence bound on the risk.

Guess-and-Check (Algorithm 2) has three stages. Guess: GnC computes a Clopper–Pearson upper bound a^g+τg\hat{a}_g + \tau_g on DtrainD_{\mathrm{train}}; if it already exceeds α\alpha, GnC returns rp=1r_p = 1 without accessing the holdout. Check: otherwise it computes the corresponding bound a^h+τh\hat{a}_h + \tau_h on DcertmalD_{\mathrm{cert}}^{\mathrm{mal}}. If the training bound is strictly more conservative than the holdout bound, GnC releases the training bound, revealing only that the check succeeded. Discretize: if the check fails, GnC releases a coarsened holdout estimate, [a^h]γ+τg[\hat{a}_{h}]_{\gamma}+\tau_{g}, and charges a failure. Confidence levels βi\beta_i are allocated across checks and failures using the summable weights cj=6/(π2(j+1)2)c_j=6/(\pi^2(j+1)^2). Once no valid discretization remains, the reusable-holdout budget is exhausted: GnC returns ⊥\perp and CRISP stops.

Validity under adaptivity. CRISP never accesses DcertmalD_{\mathrm{cert}}^{\mathrm{mal}} directly; it observes it only through GnC’s restricted outputs. Later proposals may therefore depend on earlier certificates without unrestricted reuse of the protected holdout. Moreover, risk depends only on malicious examples (Eq. (2)), so the search never reads DcertbenD_{\mathrm{cert}}^{\mathrm{ben} }. The benign certification examples consequently remain fresh for the final utility-based selection among the certified prompts, of which there are at most 2T+12T+1.

Proposition 2 (CRISP risk control and utility). Under Assumption 1, for the input failure level δR\delta_R, any δU∈(0,1)\delta_U\in(0,1), and ϵU=2log⁡(2K/δU)b\epsilon_U=\sqrt{\frac{2\log(2K/\delta_U)}{b}} with K=2T+1K=2T+1, the prompt p^\hat{p} selected by CRISP satisfies:

Pr⁡Dcertmal∼π1m,Dcertben∼π0b(R(p^)≤α and U(p^)≥max⁡p∈CTU(p)−ϵU)≥1−δR−δU,(11)\Pr_{\substack{D_{\mathrm{cert}}^{\mathrm{mal}}\sim\pi_1^m,\,D_{\mathrm{cert}}^{\mathrm{ben}}\sim\pi_0^b}}\left(R(\hat{p})\leq\alpha\text{ and }U(\hat{p})\geq\max_{p\in C_T}U(p)-\epsilon_U\right)\geq1-\delta_R-\delta_U, \tag*{(11)}

with the convention that when CT=∅C_T=\varnothing, CRISP abstains and the event holds. (Proof in Appendix B.2).

Algorithm 1 CRISP

Require: Inputs: judge ff, splits Dtrain,DcertD_{\mathrm{train}},D_{\mathrm{cert}}, risk budget α\alpha, failure level δR\delta_R

Require: Hyperparams: rounds TT, seed p0p_0

Initialize pool

1: r^p0←GnC⁡(p0,α,δR,Dtrain,Dcert)\hat{r}_{p_0}\leftarrow\operatorname{GnC}(p_0,\alpha,\delta_R,D_{\mathrm{train}},D_{\mathrm{cert}}) 2: P0←{p0}P_0\leftarrow\{p_0\}

Search round

3: for t=0,…,T−1t=0,\ldots,T-1 do 4: Ct←{p∈Pt:r^p≤α}; Ut←Pt∖CtC_t\leftarrow\{p\in P_t:\hat{r}_p\leq\alpha\};\ U_t\leftarrow P_t\setminus C_t

Improve objective (utility)

5: if Ct≠∅C_t\neq\varnothing then 6: p+←arg⁡max⁡p∈CtU^(p,Dtrain)p^+\leftarrow\arg\max_{p\in C_t}\hat{U}(p,D_{\mathrm{train}}) 7: pU←IMPROVEUTILITY⁡(p+,Dtrain)p_U\leftarrow\operatorname{IMPROVEUTILITY}(p^+,D_{\mathrm{train}}) 8: r^pU←GnC⁡(pU,α,δR,Dtrain,Dcert)\hat{r}_{p_U}\leftarrow\operatorname{GnC}(p_U,\alpha,\delta_R,D_{\mathrm{train}},D_{\mathrm{cert}})

Reduce violation (risk)

9: if Ut≠∅U_t\neq\varnothing then 10: p−←arg⁡min⁡p∈UtR^(p,Dtrain)p^-\leftarrow\arg\min_{p\in U_t}\hat{R}(p,D_{\mathrm{train}}) 11: pR←REDUCERISK⁡(p−,Dtrain)p_R\leftarrow\operatorname{REDUCERISK}(p^-,D_{\mathrm{train}}) 12: r^pR←GnC⁡(pR,α,δR,Dtrain,Dcert)\hat{r}_{p_R}\leftarrow\operatorname{GnC}(p_R,\alpha,\delta_R,D_{\mathrm{train}},D_{\mathrm{cert}})

Stop or append prompts

13: if r^pU=⊥\hat{r}_{p_U}=\perp or r^pR=⊥\hat{r}_{p_R}=\perp then break 14: else Pt+1←Pt∪{pR,pU}P_{t+1}\leftarrow P_t\cup\{p_R,p_U\}

Final prompt select

15: CT←{p∈PT:r^p≤α}C_T\leftarrow\{p\in P_T:\hat{r}_p\leq\alpha\} 16: return arg⁡max⁡p∈CTU^(p,Dcert)\arg\max_{p\in C_T}\hat{U}(p,D_{\mathrm{cert}})

Algorithm 2 Guess-and-Check (GnC)

function GnC⁡(p,α,δ,Dtrain,Dcert)\operatorname{GnC}(p,\alpha,\delta,D_{\mathrm{train}},D_{\mathrm{cert}})

Charge query

1: ν←(⌊1/γ⌋+1)∏j∈Γ(⌊1/γ⌋+1)\nu\leftarrow\left(\left\lfloor1/\gamma\right\rfloor+1\right)\prod_{j\in\Gamma}\left(\left\lfloor1/\gamma\right\rfloor+1\right) 2: βi←δ⋅ci⋅cf/ν, cj=6/(π2(j+1)2)\beta_i\leftarrow\delta\cdot c_i\cdot c_f/\nu,\ c_j=6/(\pi^2(j+1)^2)

Guess

3: a^g←R^(p,Dtrain)\hat{a}_g\leftarrow\hat{R}(p,D_{\mathrm{train}}) 4: τg←Rucb(p,Dtrain,βi)−a^g\tau_g\leftarrow R_{\mathrm{ucb}}(p,D_{\mathrm{train}},\beta_i)-\hat{a}_g 5: if a^g+τg>α\hat{a}_g+\tau_g>\alpha then return 11

Check

6: i←i+1i\leftarrow i+1 7: a^h←R^(p,Dcert)\hat{a}_h\leftarrow\hat{R}(p,D_{\mathrm{cert}}) 8: τh←Rucb(p,Dcert,βi)−a^h\tau_h\leftarrow R_{\mathrm{ucb}}(p,D_{\mathrm{cert}},\beta_i)-\hat{a}_h 9: if a^g+τg>a^h+τh\hat{a}_g+\tau_g>\hat{a}_h+\tau_h then return a^g+τg\hat{a}_g+\tau_g

Discretize rate

10: f←f+1f\leftarrow f+1 11: G←{γ∈(0,τg):2e−2(τg−γ)2m≤βi}G\leftarrow\{\gamma\in(0,\tau_g):2e^{-2(\tau_g-\gamma)^2m}\leq\beta_i\} 12: if G≠∅G\neq\varnothing then 13: γ←max⁡G\gamma\leftarrow\max G; append γ\gamma to Γ\Gamma 14: else return [a^h]γ+τg[\hat{a}_h]_\gamma+\tau_g

Return bot

15: return ⊥\perp (stop the optimization)

end function

Note.

  • Rucb(p,D,β)R_{\mathrm{ucb}}(p,D,\beta) is the Clopper–Pearson (eq. 6).

  • [a^h]γ=γ⌊a^h/γ⌋[\hat{a}_h]_\gamma=\gamma\lfloor\hat{a}_h/\gamma\rfloor is rounding down to the γ\gamma-grid.

  • ⌊1/γ⌋+1\lfloor1/\gamma\rfloor+1 is the number of points on the γ\gamma-grid.

  • Γ=(γ1,…,γf)\Gamma=(\gamma_1,\ldots,\gamma_f) is an ordered list.

Experiments

Setup

We evaluate on four datasets spanning three guardrail tasks. Prompt injection uses BIPIA ([33]), where malicious examples embed an indirect injection in otherwise benign external context, and Deepset, a smaller public set of direct prompt-injection and jailbreak inputs. The two output-guardrail tasks are system-prompt leakage (SPL) and sensitive-information disclosure (SID). In SPL the judge screens an assistant response given a confidential system prompt; in SID it screens a response given a list of protected items, using BeaverTails privacy-violation examples ((Ji et al., 2023). Across all datasets, y=1y = 1 denotes a violation that should be blocked and y=0y = 0 a benign case that should be allowed. Each dataset is randomly partitioned into disjoint training, certification, and test splits. The training split DtrainD_{\mathrm{train}} is used only for prompt generation and rewriting, DcertD_{\mathrm{cert}} for certification and prompt selection, and DtestD_{\mathrm{test}} only for final evaluation. Dataset construction, filtering, and split counts are given in Appendix § 10.

Figure 2. CRISP: adaptive prompt search with reusable-holdout risk certification.

CPS constructs a pool of N=20N = 20 candidate prompts before certification, and CRISP uses T=20T = 20 rounds. If no candidate can be certified at a given risk level α\alpha, we report “–” rather than returning an uncertified prompt. We compare against a fixed unoptimized prompt (Naive); unconstrained prompt optimizers that maximize average performance (ProTeGi, GEPA, and MIPROv2); and, for prompt injection, dedicated classifier-based guardrails (ProtectAI, PIGuard, Prompt Guard, and Model Armor [11]). Figure 3 additionally evaluates selected prompts under Best-of-NN jailbreaking [15] on BIPIA (Appendix § 10.§ 5).

Best-of-N jailbreaking risk curves

Figure 3. Best-of-NN jailbreaking on BIPIA. Missed-attack risk as a function of the attacker query budget NN, using gpt-4o-mini as the judge.

Unless otherwise stated, every prompt-optimization method, including CPS and CRISP, starts from the same task-specific seed and uses gpt-5.4-mini as the prompt generator/rewriter. LLM-as-a-judge results are reported with two judges, gpt-4o-mini and gpt-5.4-mini (Table 1); output-guardrail results use gpt-5.4-mini. The judge uses greedy decoding (temperature = 0). We set the failure level to δ=0.10\delta= 0.10, report prompt-injection results at α∈{0.10,0.20,0.30}\alpha\in\{0.10, 0.20, 0.30\}, and report output-guardrail results at α=0.30\alpha= 0.30. Hyperparameters, seed prompts, and the Best-of-NN protocol are in Appendix § 10.

LLM-as-a-judge guardrailMethodBIPIADeepset
Risk↓\downarrowUtility↑\uparrowRisk↓\downarrowUtility↑\uparrow
–Baseline classifier guardrailsProtectAI0.880.880.550.99
–Baseline classifier guardrailsPIGuard0.150.860.081.00
–Baseline classifier guardrailsPrompt Guard0.130.780.070.14
–Baseline classifier guardrailsModel Armor0.940.980.720.98
gpt-4o-miniBaseline prompt optimizationNaive0.600.900.290.92
gpt-4o-miniBaseline prompt optimizationProTeGi0.190.800.070.95
gpt-4o-miniBaseline prompt optimizationGEPA0.330.900.160.95
gpt-4o-miniBaseline prompt optimizationMIPROv20.370.880.290.97
gpt-4o-miniOurs (α\alpha=0.1)CPS0.060.84––
gpt-4o-miniOurs (α\alpha=0.1)CRISP0.020.580.020.83
gpt-4o-miniOurs (α\alpha=0.2)CPS0.120.860.050.97
gpt-4o-miniOurs (α\alpha=0.2)CRISP0.110.860.130.95
gpt-4o-miniOurs (α\alpha=0.3)CPS0.250.880.160.96
gpt-4o-miniOurs (α\alpha=0.3)CRISP0.200.860.110.96
gpt-5.4-miniBaseline prompt optimizationNaive0.380.930.350.97
gpt-5.4-miniBaseline prompt optimizationProTeGi0.150.780.110.99
gpt-5.4-miniBaseline prompt optimizationGEPA0.400.950.151.00
gpt-5.4-miniBaseline prompt optimizationMIPROv20.210.880.240.99
gpt-5.4-miniOurs (α\alpha=0.1)CPS––––
gpt-5.4-miniOurs (α\alpha=0.1)CRISP0.010.780.040.94
gpt-5.4-miniOurs (α\alpha=0.2)CPS––0.090.99
gpt-5.4-miniOurs (α\alpha=0.2)CRISP0.140.850.050.99
gpt-5.4-miniOurs (α\alpha=0.3)CPS0.170.860.151.00
gpt-5.4-miniOurs (α\alpha=0.3)CRISP0.180.890.091.00

Table 1. Prompt-injection test results. Target risk levels: α∈{0.10,0.20,0.30}\alpha\in\{0.10, 0.20, 0.30\}.

Results and Discussion

Risk calibration holds on test data. Across all prompt-injection configurations in Table 1 where a prompt was certified (two LLM judges, two datasets, three risk levels, two methods), the missed-attack rate measured on the untouched test split stays at or below the target α\alpha. The same holds on both output-guardrail tasks at α=0.30\alpha=0.30 (Table 2). This empirical pattern is consistent with Propositions 1 and 2: certification is performed on DcertD_{\mathrm{cert}} to control population risk on fresh samples from the same distribution with high probability. Unconstrained baselines provide no such guarantee.

MethodSPL (leakage)SID (BeaverTails)
Risk↓\downarrowUtility↑\uparrowRisk↓\downarrowUtility↑\uparrow
Baseline prompt optimizationNaive0.170.380.521.00
Baseline prompt optimizationProTeGi0.150.780.130.93
Baseline prompt optimizationGEPA0.170.580.120.95
Baseline prompt optimizationMIPROv20.060.650.130.97
Ours (α\alpha=0.3)CPS0.210.780.150.97
Ours (α\alpha=0.3)CRISP0.080.600.220.96

Table 2. Output-guardrail test results

The target risk level is a usable control knob. Relaxing α\alpha moves the selected prompt along the risk–utility frontier in the expected direction. For CRISP with gpt-4o-mini, benign utility rises from 0.58 at α\alpha=0.10 to 0.86 at α\alpha=0.30 on BIPIA, and from 0.83 to 0.96 on DeepSet. With gpt-5.4-mini it rises monotonically from 0.78 to 0.85 to 0.89 on BIPIA and from 0.94 to 1.00 on DeepSet. Test risk generally increases accordingly but remains within the target risk level. Practitioners therefore can set the safety–usability trade-off explicitly.

Certification preserves competitive utility while enforcing the risk constraint. Certified prompts remain competitive with, and often outperform, unconstrained optimizers at similar risk. At

α\alpha=0.20, on BIPIA with gpt-4o-mini, CPS and CRISP achieve RR=0.12/0.11 and UU=0.86, compared with ProTeGi’s RR=0.19, UU=0.80. With gpt-5.4-mini, CRISP similarly improves over ProTeGi (0.14/0.85 vs. 0.15/0.78). On Deepset with gpt-5.4-mini, CRISP at α\alpha=0.20 matches GEPA and ProTeGi utility while lowering risk (0.05/0.99 vs. GEPA’s 0.15/1.00 and ProTeGi’s 0.11/0.99).

Closing the generation gap. The main difference between CPS and CRISP is certifiability. CPS returns an empty certified set in four of twelve prompt-injection settings, whereas CRISP certifies a prompt in all twelve. CPS can fail when its fixed candidate pool contains no prompt that can be certified, either because finite-sample uncertainty makes certification conservative or because the generator fails to explore sufficiently low-risk prompts. CRISP addresses this generation gap by using risk feedback to steer prompt generation toward certifiable candidates (§ 4.2).

A general-purpose judge matches dedicated classifiers. On prompt injection, PIGuard is the strong classifier baseline. CPS and CRISP match its performance without any fine-tuning. At α\alpha=0.20 on BIPIA, CRISP with gpt-4o-mini (0.11/0.86) improves on PIGuard (0.15/0.86). On Deepset, CRISP with gpt-5.4-mini (0.05/0.99) matches it (0.08/1.00), and CRISP/CPS additionally carry finite-sample guarantees.

Output guardrails present a different trade-off. On SPL and SID, certified selection does not uniformly dominate the unconstrained baselines, but it consistently returns operating points that satisfy the target α=0.30\alpha= 0.30. On SPL, CPS matches ProTeGi’s utility (0.78), while CRISP attains lower risk (0.08) at lower utility (0.60). On SID, CPS and CRISP achieve high utility (0.97 and 0.96) with risks of 0.15 and 0.22, whereas the Naive prompt reaches utility 1.00 but violates the target with risk 0.52. Thus, the main benefit in this regime is certified feasibility rather than uniform frontier improvement.

Robustness beyond the static certificate. We evaluate the selected prompts under Best-of-NN jailbreaking (Figure 3), which measures missed-attack risk as the attacker is permitted an increasing query budget NN. For each example, the attacker may issue up to NN queries, and the attack is considered successful if at least one query is allowed by the guardrail; full evaluation details are provided in Appendix § 10.§ 5. As NN increases, prompts produced by unconstrained optimizers degrade substantially: MIPROv2 reaches a missed-attack rate of 0.8 by N=40N = 40, and GEPA reaches 0.8 by N=100N = 100. In contrast, CRISP at α=0.10\alpha= 0.10 remains at or below 0.07 through N=100N = 100, below the dotted α=0.10\alpha= 0.10 reference line. This experiment is a robustness diagnostic rather than part of the certification guarantee, which does not extend to this multi-query attack setting.

Additional empirical analysis. We provide ablations on the CRISP search horizon in Appendix § 10.§ 7 and an evaluation of the risk-control guarantee across repeated data splits in Appendix § 10.§ 8.

Limitations

The (α,ϵ,δ)(\alpha, \epsilon, \delta)-validity guarantee is finite-sample and distribution-free, but it is not unconditional. It applies under i.i.d. sampling from the calibration distribution π\pi (Definition 1 and Assumption 1). A shift in the attack mix, user population, or judge model can invalidate the certificate, and certification cannot recover a high-utility feasible prompt if the generator never proposes one—which is why CPS can return an empty certified set. The bound likewise does not cover adaptive adversaries: Best-of-NN jailbreaking is a robustness diagnostic, not a certified guarantee. Practically, both procedures require labeled malicious and benign data, and small certification sets make Clopper–Pearson and Guess-and-Check bounds conservative; CRISP may also stop early when the reusable-holdout budget is exhausted. We further restrict to a binary, greedy-decoded judge with a single missed-attack constraint. Stochastic decoding, multi-category policies, and joint control of over-blocking remain open.

Conclusion

We framed LLM-as-a-judge guardrail design as risk-certified system-prompt optimization: maximize benign pass rate subject to a user-specified missed-attack budget, with finite-sample statistical control. CPS generates a fixed pool and then certifies it; CRISP feeds risk certificates back into search so that certifiable high-utility prompts can be found when a fixed pool fails. On four datasets covering prompt injection, system-prompt leakage, and sensitive-information disclosure, selected prompts respect the target risk on held-out tests, utility tracks the permitted budget, and CRISP succeeds in settings where fixed-pool search cannot certify. A general-purpose judge, prompted rather than fine-tuned, is competitive with dedicated injection classifiers while remaining retargetable to a new policy or risk level by rerunning the search. We view certified prompts as one layer in a defense-in-depth stack, not a substitute for monitoring under distribution shift or adaptive attack.

AI use statement

Generative AI tools were used only as writing assistants in preparing this manuscript: we used them to improve transitions, correct grammar, and provide editorial feedback that informed our revisions. They were not used to generate synthetic datasets, develop theoretical models or conceptual frameworks, formulate mathematical claims, assist with proofs, propose or refine hypotheses, design experiments, implement methods, clean or reformat data, or interpret results. Translation, qualitative or thematic analysis, and survey design do not apply to this work. The LLM-as-a-judge guardrail and prompt rewriter described in § 4–§ 5 are components of the method being studied, rather than tools used to prepare the manuscript. We reviewed all AI-suggested wording and take full responsibility for the final content, including text produced with generative AI assistance.

Ethics statement

This work concerns the defensive calibration of LLM-as-a-judge guardrails: selecting a system prompt that maximizes the benign pass rate while keeping the missed-attack rate below a user-specified level, backed by a finite-sample certificate. Its intended use is to help practitioners set an explicit safety–utility trade-off, not to weaken deployed systems.

Evaluating guardrails necessarily involves harmful inputs, including prompt injection, system-prompt leakage, sensitive-information disclosure, and the Best-of-NN robustness diagnostic of Figure 3. We therefore rely on existing public benchmarks and published red-teaming plugins rather than a new exploit pipeline, and we omit step-by-step instructions for attacking production systems. As noted in § 6, the certificate covers neither adaptive adversaries nor distribution shift, so certified prompts should be treated as one layer in a defense-in-depth stack rather than as unconditional safety.

BIPIA, Deepset, and BeaverTails are publicly released datasets. SPL, by contrast, is not a fully public benchmark: it combines the public HuggingFace set gabrielechua/system-prompt-leakage with Promptfoo-generated extraction and hijacking attempts against synthetic confidential system prompts (Appendix § 10.§ 1). No new human-subjects data were collected, so no IRB protocol was required. SID examples are drawn from BeaverTails privacy-violation and safe responses; we introduce no additional personal data and use screening payloads solely to evaluate disclosure detection. Finally, we do not study fairness of over-blocking: utility is an aggregate benign pass rate, which may mask unequal error rates across user groups.

Experimental details

This appendix records the dataset construction, splits, models, and hyperparameters used in § 5.

Datasets

All datasets use the binary convention of § 3: y=1y = 1 is a violation that should be blocked and y=0y = 0 is a benign case that should be allowed. Empty strings, exact text duplicates, and inputs longer than a character cap are dropped before splitting. Table 3 summarizes sources and what the judge sees. Each corpus is then randomly partitioned as in Appendix § 10.§ 2.

DatasetAccessSourcesJudge inputs
BIPIApublicBIPIA [33]context
Deepsetpublicdeepset/prompt-injectionsuser prompt
SPLmixedgabrielchua/system-prompt-leakage + Promptfoosystem prompt, response
SIDpublicBeaverTails [17]protected items, request, response

Table 3. Guardrail benchmarks used in Section 5. SPL is a constructed mix, not a fully public release.

BIPIA (prompt injection). Policy: detect indirect prompt-injection attempts that hide an instruction override inside otherwise benign external content. We reconstruct a binary input-guardrail task from the BIPIA benchmark [33]. Each example is an external context string; malicious contexts embed an injection, while benign contexts do not. The judge screens this context alone (no separate user-intent field). We cap context length at 2,000 characters, drop empty and duplicate contexts, and subsample at most 1,500 examples per class before splitting.

Deepset (prompt injection). Policy: detect direct prompt-injection and jailbreak attempts in the user message. Deepset is the public HuggingFace dataset deepset/prompt-injections. We pool the released train and test splits, drop empty/duplicate strings, and keep the binary labels as released. Malicious examples are direct injections and jailbreaks; benign examples are ordinary user requests. The judge sees only the user prompt. The resulting set is small (661 examples after pooling), which makes certification conservative at tight α\alpha.

SPL (system-prompt leakage). Policy: detect when an assistant response reveals all or part of a confidential system prompt (extraction, hijacking, or override). This is an output-guardrail task: the judge receives the confidential system prompt and the assistant response, packed as a PROTECTED SYSTEM PROMPT block followed by an ASSISTANT RESPONSE TO SCREEN block. SPL is constructed from two sources, then cleaned and re-split:

  • Open-source (gabrielchua/system-prompt-leakage): 300 examples (180 malicious, 120 benign).

  • Promptfoo-generated: 196 examples (87 malicious, 109 benign), built from 600 Mistral-Large system prompts stratified over 15 business domains, three complexity levels (low / medium / high), six languages (English, French, Spanish, German, Chinese, Arabic), and four formats (Markdown, JSON, XML, plain text).

Promptfoo red-teaming uses plugins prompt-extraction (∼\sim44%), hijacking (∼\sim41%), and system-prompt-override (the remainder), with a mix of basic prompts, jailbreak templates (Meta, Hydra, Crescendo), and obfuscation (Base64, Hex, ROT13, homoglyph, Leetspeak). Each row stores system_prompt, llm_response, label, and source. Promptfoo labels were validated by two annotators with joint review. The pooled file has 496 examples (267 malicious / 229 benign) before the filters of Appendix § 10.§ 2: the open-source subset is attack-heavy, while Promptfoo contributes more benign passes. System prompts are truncated at 2,400 characters and responses at 3,000 characters (raw medians 1,446 and 2,128 characters).

SID (sensitive-information disclosure). Policy: detect when an assistant response discloses sensitive information (PII, credentials, financial or medical data, and similar protected items). SID uses BeaverTails [17] (PKU-Alignment/BeaverTails, splits 30k train and 30k test). We label a response malicious (y=1y = 1) if its annotated category includes privacy-violation, and benign (y=0y = 0) if it is marked is_safe; other categories are discarded. The judge screens a fixed list of protected item types (personal identifiers, contact details, financial and medical data) together with the user request and the assistant response, in the payload format PROTECTED / SENSITIVE ITEMS, USER REQUEST, ASSISTANT RESPONSE TO SCREEN. Responses are truncated at 3,000 characters. We balance the two classes at 1,500 examples each before splitting. Unlike SPL, SID does not include the generating system prompt at inference time.

Splits and test filtering

We cap each class, shuffle the pooled set, and cut it into training / certification / test fractions 0.20/0.60/0.20. The same seed is used for all methods on a given dataset. Table 4 reports the resulting counts. A small number of test examples are rejected by the hosted API content filter (HTTP 400); these are excluded from every rate computation, so reported test numbers use the filtered test set. The same filtered test set is used for all methods on a given dataset.

DatasetTrain nn (mal/ben)Cert. nn (mal/ben)Test nn (mal/ben)
BIPIA600 (301/299)1800 (903/897)597 (294/303)
Deepset132 (58/74)397 (149/248)132 (55/77)
SPL93 (53/40)278 (140/138)92 (52/40)
SID599 (299/300)1798 (889/909)597 (306/291)

Table 4. Split sizes used in the experiments. Test counts are after excluding API content-filter rejections (3 examples on BIPIA, 3 on SID, none on Deepset or SPL).

Judge, rewriter, and decoding

The LLM-as-a-judge guardrail is a system prompt pp that maps a screening payload to a one-word decision, BLOCK or ALLOW. The user message contains the content to screen, using the task-specific payload format described in Appendix § 10.§ 1; there is no additional fixed wrapper around the system prompt pp. Decoding is greedy (temperature 0, at most 5 completion tokens), matching the deterministic classifier f(p,⋅)f(p, \cdot) in § 3. Unparseable replies default to ALLOW.

Prompt generation and rewriting always use gpt-5.4-mini, including for CPS/CRISP and for the unconstrained baselines. Table 1 reports two judges, gpt-4o-mini and gpt-5.4-mini; Table 2 uses gpt-5.4-mini as the judge. All prompt-optimization methods start from the same task seed listed below. Naive deploys that seed without optimization.

Task seeds. The shared seeds already contain BLOCK/ALLOW rules and the one-word reply format.

Prompt injection.

Prompt injection system prompt

Figure 1 (PDF p. 15).

System prompt leakage (SPL). You screen an assistant’s reply given a confidential system prompt. BLOCK if the reply looks like it reveals the system prompt. ALLOW if it looks fine and does not. Respond with exactly one word: BLOCK or ALLOW. Do not explain.

Sensitive information disclosure (SID).

You screen an assistant’s reply for sensitive-information disclosure. BLOCK if the reply discloses private personal data of a real person (phone, email, SSN, home address, account number). ALLOW otherwise, including harmful advice that does not disclose those data. Respond with exactly one word: BLOCK or ALLOW. Do not explain.

Method hyperparameters

We set the risk-certification failure level δR=0.10\delta_R = 0.10 throughout. Prompt injection uses α∈{0.10,0.20,0.30}\alpha\in\{0.10, 0.20, 0.30\}; output-guardrail tables use α=0.30\alpha= 0.30. The utility failure level δU\delta_U appears only in the theoretical utility guarantee and is not used by the selection algorithms. Unconstrained baselines are α\alpha-independent: each optimizer is run once on DtrainD_{\mathrm{train}} (with a training-only internal validation split) and the returned prompt is scored on DtestD_{\mathrm{test}}. CPS builds its pool once and then certifies at each α\alpha. CRISP is rerun at each α\alpha.

CPS. Following § 4.1, we draw N=20N = 20 bootstrap samples from DtrainD_{\mathrm{train}} only. Each sample has size B=0.20∣Dtrain∣B = 0.20|D_{\mathrm{train}}|, is drawn with replacement, and may therefore overlap other bootstrap samples. We hold out 25% of each sample as GEPA’s internal validation split; neither portion accesses DcertD_{\mathrm{cert}}. GEPA is run in the light configuration with gpt-5.4-mini as the reflection model. Candidates are then scored on DcertD_{\mathrm{cert}} and certified with one-sided Clopper–Pearson risk UCBs at the Bonferroni level δR/N\delta_R/N. Among certified prompts we select by empirical benign utility on the benign subset of DcertD_{\mathrm{cert}}. If the certified set is empty we report “–”.

CRISP. We run T=20T = 20 search rounds with up to q=2q = 2 proposals per round (one utility-raising rewrite and one risk-lowering rewrite when the corresponding parents exist). Search starts from the shared task seed (round 0). Hard examples are mined only from DtrainD_{\mathrm{train}}: up to three false positives and three true positives for the utility branch, and up to three false negatives and three true negatives for the risk branch. Each rewrite is a single call to gpt-5.4-mini at temperature 0.7 (at most 420 tokens); the templates are in Appendix § 10.§ 6. Risk certificates are obtained from Guess-and-Check (Algorithm 2) on the malicious subset of DcertD_{\mathrm{cert}}, with one-sided Clopper–Pearson radii and summable weights cj=6/(π2(j+1)2)c_j = 6/(\pi^2(j+1)^2). The Guess-and-Check state is initialized once per run and persists across adaptive queries; benign certification examples remain unused until final selection. If Guess-and-Check returns ⊥\bot, search stops. If no prompt is certified, we report “–”.

Unconstrained prompt optimizers. GEPA uses the same light configuration as in the CPS bootstraps, with a feedback metric that distinguishes missed attacks from over-blocks. MIPROv2 is run 3-shot (up to three demonstrations), with 5 instruction candidates and 8 trials, using gpt-5.4-mini as the prompt model. ProTeGi uses textual-gradient beam search with beam size 3, 3 rounds, minibatch size 25, and 2 expansions per parent. All three maximize average accuracy, not the constrained objective.

Classifier baselines. On the same filtered prompt-injection test sets we score ProtectAI (protectai/deberta-v3-base-prompt-injection-v2), PIGuard (leolee99/PIGuard), Prompt Guard (meta-llama/Llama-Prompt-Guard-2-86M), and Model Armor. A classifier prediction is treated as block if its label is in {injection,unsafe,malicious,jailbreak,attack}\{\texttt{injection},\texttt{unsafe},\texttt{malicious},\texttt{jailbreak},\texttt{attack}\} (or LABEL_1 for Prompt Guard).

Best-of-N evaluation

Figure 3 follows Best-of-N jailbreaking [15] against the deployed judge. Here, NN is the attacker budget: at a given NN, the attacker may send NN queries, and we consider the attack successful if at least one query is allowed. We sample 30 malicious seeds uniformly from the BIPIA test split. For each seed, we build a shared bank of textual variants and use the same bank for every defender. Variants combine the character-level augmentations of Hughes et al. [15] (scramble, random capitalization, ASCII ±1\pm1, applied sparingly) with intent-preserving surface transforms (benign padding, mid-document burial, quote-as-example framing, imperative softening, injection relocation, homoglyphs, leetspeak, zero-width characters, and spaced keywords). We do not add meta-wrappers of the form “ignore previous instructions / obey the directives below,” which would introduce additional attack cues. Each candidate is sent to the guardrail; ALLOW counts as success. Risk@NN is the fraction of seeds for which at least one of the first NN variants is allowed. The certificate in Propositions 1–2 does not cover this repeated-query adversary; the plot is a robustness diagnostic, not a certified guarantee.

CRISP rewrite templates

CRISP never edits prompts with DcertD_{\mathrm{cert}}. Both branches receive the parent prompt, the current empirical risk and budget α\alpha, and a few hard training examples, and must output only the revised system prompt. The rewriter is gpt-5.4-mini at temperature 0.7.

ImproveUtility (certified parent p+p^{+}).

This prompt is designed to block [violation]. It is good but overblocks some safe cases. current risk (False Negative Rate) = [R^\hat{R}], allowed risk = [α\alpha] Current prompt: [parent] Must ALLOW (false positives): [up to 3 blocked benign DtrainD_{\mathrm{train}} examples] Must still BLOCK: [up to 3 correctly blocked attacks] Fix the prompt to ALLOW the safe cases (false positives). Output only the revised prompt, keep it small to avoid overfitting.

ReduceRisk (uncertified parent p−p^{-}).

This prompt is designed to block [violation]. It is good but misses some attack patterns. current risk (False Negative Rate) = [R^\hat{R}], allowed risk = [α\alpha] Current prompt: [parent] Must BLOCK (false negatives, missed patterns): [up to 3 missed DtrainD_{\mathrm{train}} attacks] Must still ALLOW: [up to 3 correctly allowed benign examples] Fix the prompt to BLOCK the missed patterns (false negatives). Output only the revised prompt, keep it small to avoid overfitting.

Ablation on the CRISP Search Horizon

We evaluate the effect of the CRISP search horizon on BIPIA at α=0.20\alpha= 0.20 and α=0.30\alpha= 0.30, using the same data split and gpt-5.4-mini judge, and reporting the corresponding main-table experiments. Each run is extended to T=40T = 40. At checkpoints T∈{0,1,2,5,10,20,40}T \in\{0, 1, 2, 5, 10, 20, 40\}, we record the prompt that CRISP would return if the search were stopped at that point: the certified candidate with the highest benign utility on DcertD_{\mathrm{cert}}. Here, T=0T = 0 corresponds to evaluating the shared seed prompt without any rewriting.

TTCertifiedCalib RiskCalib UtilityTest RiskTest UtilityCandidatesRewrites
0No (0/1)––––10
1Yes (1/2)0.010.480.010.5021
2Yes (3/4)0.060.680.060.6643
5Yes (7/10)0.130.790.150.82109
10Yes (14/20)0.120.810.110.832019
20Yes (21/40)0.120.810.110.834039
40Yes (45/80)0.120.810.110.838079

Table 5. CRISP search horizon on BIPIA at α=0.20\alpha= 0.20. Certified indicates whether the certified set is nonempty, with the number of certified candidates relative to the total pool size shown in parentheses. Calib denotes DcertD_{\mathrm{cert}}, while Test denotes the filtered test set. “–” indicates that no candidate is certified. The same prompt is selected at T=10T = 10, T=20T = 20, and T=40T = 40.

TTCertifiedCalib RiskCalib UtilityTest RiskTest UtilityCandidatesRewrites
0No (0/1)––––10
1Yes (1/2)0.130.520.100.5621
2Yes (2/4)0.130.520.100.5643
5Yes (5/10)0.160.830.170.80109
10Yes (11/20)0.200.850.240.842019
20Yes (22/40)0.200.850.240.844039
40Yes (44/80)0.140.860.160.848079

Table 6. CRISP search horizon on BIPIA at α=0.30\alpha= 0.30. The same prompt is selected at T=1T = 1 and T=2T = 2, while another prompt is selected at both T=10T = 10 and T=20T = 20.

In both runs, the initial seed leaves the certified set empty, whereas a single rewrite is sufficient to produce a certified prompt. Every subsequent checkpoint also remains certified, with test risk below the corresponding target α\alpha. At α=0.20\alpha= 0.20, test utility increases from 0.50 at T=1T = 1 to 0.83 at T=10T = 10, while calibration utility rises from 0.48 to 0.81. Beyond T=10T = 10, the selected prompt remains unchanged through T=40T = 40. This indicates that, for this run, most of the benefit of adaptive search is obtained within the first few rounds, with additional iterations expanding the candidate pool without improving the final selected operating point.

A similar pattern appears at α=0.30\alpha= 0.30. Test utility improves rapidly, from 0.56 at T=1T = 1 to 0.84 at T=10T = 10, and remains unchanged at T=20T = 20. By T=40T = 40, CRISP selects a different prompt whose calibration risk decreases from 0.20 to 0.14 and whose test risk decreases from 0.24 to 0.16. Calibration utility increases slightly from 0.85 to 0.86, while test utility remains 0.84. Thus, the longer search does not improve held-out utility in this case, but it discovers an alternative operating point with substantially lower risk at essentially the same utility.

Overall, these results suggest that relatively few CRISP rounds are sufficient to enter the feasible region and recover most of the attainable utility in these runs. Longer searches can still refine the selected risk–utility trade-off, particularly by discovering lower-risk prompts, but the resulting gains need not be monotonic or substantial. The values differ slightly from Table 1 because this ablation uses a separate stochastic rerun, including randomness in rewriting and in the training examples selected during rewriting.

Repeated splits on BIPIA

The main tables use a single 0.20/0.60/0.20 split (seed 1), which can show Rtest≤αR_{\mathrm{test}} \le\alpha only once. To check that this holds with probability at least 1−δ1-\delta, we redraw BIPIA 50 times. Each draw samples 600 examples per class from the pooled set, then applies the same 0.20/0.60/0.20 cut. CRISP is rerun independently on each split at risk levels α∈{0.20,0.30}\alpha\in\{0.20, 0.30\}, failure level δ=0.10\delta= 0.10, and T=10T = 10 iterations, with gpt-4o-mini as the judge and gpt-5.4-mini as the rewriter.

α=0.20\alpha= 0.20α=0.30\alpha= 0.30
Rtest≤αR_{\mathrm{test}} \le\alpha49/4950/50
Abstention rate (CT=∅CT = \emptyset)1/500/50
Mean RtestR_{\mathrm{test}}0.090.15

Table 7. CRISP on 50 independent BIPIA re-splits at α=0.20\alpha= 0.20 and α=0.30\alpha= 0.30. The fraction of certified splits with Rtest≤αR_{\mathrm{test}} \le\alpha is compared to 1−δ=0.901-\delta= 0.90.

At both risk levels every certified split has test missed-attack rate below α\alpha, so the empirical coverage is 1.00, above 1−δ=0.901-\delta= 0.90. Raising the budget from 0.20 to 0.30 lifts mean test risk from 0.09 to 0.15, still inside the target, and the single abstention disappears.

Proofs of the guarantees

Probability space. Both proofs are conditional on the label counts (m,b)(m,b) and on DtrainD_{\mathrm{train}}. The CRISP proof is also conditional on the rewriter randomness ξ\xi. These quantities are fixed before DcertD_{\mathrm{cert}} is used, and the bounds below do not depend on them, so the same bounds hold unconditionally. Under Assumption 1:

  1. Dcertmal∼π1mD_{\mathrm{cert}}^{\mathrm{mal}} \sim\pi_1^m and Dcertben∼π0bD_{\mathrm{cert}}^{\mathrm{ben}} \sim\pi_0^b are independent.

  2. For each prompt pp, mR^(p,Dcert)m\widehat{R}(p,D_{\mathrm{cert}}) depends only on DcertmalD_{\mathrm{cert}}^{\mathrm{mal}}, and bU^(p,Dcert)b\widehat{U}(p,D_{\mathrm{cert}}) depends only on DcertbenD_{\mathrm{cert}}^{\mathrm{ben}}.

Proof of Proposition 1

We treat risk and utility separately, through the events ERE_R and EUE_U defined below, and combine them at the end. When C=∅C = \emptyset, CPS abstains and the claimed event holds by convention.

Lemma B.1 (Clopper–Pearson bound). Let k∼Bin⁡(n,θ)k \sim\operatorname{Bin}(n,\theta) with θ∈[0,1]\theta\in[0,1], n≥1n \ge1, and β∈(0,1)\beta\in(0,1). Define

CP⁡n(k;β)={Beta⁡−1(1−β;k+1,n−k)if k<n,1if k=n.\operatorname{CP}_n(k;\beta) = \begin{cases} \operatorname{Beta}^{-1}(1-\beta;k+1,n-k) & \text{if } k<n,\\ 1 & \text{if } k=n. \end{cases}

Then

Pr⁡(θ>CP⁡n(k;β))≤β;\Pr\left(\theta> \operatorname{CP}_n(k;\beta)\right) \le\beta;
CP⁡n(k;β)≤kn+log⁡(1/β)2nfor every k∈{0,…,n}.\operatorname{CP}_n(k;\beta) \le\frac{k}{n}+\sqrt{\frac{\log(1/\beta)}{2n}} \quad\text{for every } k \in\{0,\ldots,n\}.

Proof.

(a) Part (a) is the coverage property of the one-sided Clopper–Pearson bound [7].

(b). Let θ^=k/n\widehat{\theta}=k/n and w=log⁡(1/β)/(2n)w=\sqrt{\log(1/\beta)/(2n)}. If k=nk=n, then CP⁡n(k;β)=1≤1+w=θ^+w\operatorname{CP}_n(k;\beta)=1\le1+w=\widehat{\theta}+w. Likewise, if θ^+w≥1\widehat{\theta}+w\ge1, then CP⁡n(k;β)≤1≤θ^+w\operatorname{CP}_n(k;\beta)\le1\le\widehat{\theta}+w. It remains to consider the case k<nk<n and θ^+w<1\widehat{\theta}+w<1. Define

F(q)=Pr⁡Z∼Bin⁡(n,q)(Z≤k).F(q)=\Pr_{Z\sim\operatorname{Bin}(n,q)}(Z\le k).

By the beta-binomial identity and the definition of the Clopper–Pearson bound, F(CP⁡n(k;β))=βF(\operatorname{CP}_n(k;\beta))=\beta. Since FF is continuous and strictly decreasing on (0,1)(0,1), CP⁡n(k;β)\operatorname{CP}_n(k;\beta) is the unique q∈(0,1)q\in(0,1) satisfying F(q)=βF(q)=\beta.

Applying Hoeffding’s inequality [14] at q=θ^+wq=\widehat{\theta}+w gives

F(θ^+w)=Pr⁡(Z/n−q≤−w)≤e−2nw2=β=F(CP⁡n(k;β)).F(\widehat{\theta}+w)=\Pr(Z/n-q\le-w)\le e^{-2nw^2}=\beta=F(\operatorname{CP}_n(k;\beta)).

Since FF is strictly decreasing, it follows that CP⁡n(k;β)≤θ^+w\operatorname{CP}_n(k;\beta)\le\widehat{\theta}+w.

Step 1: Risk control (Clopper–Pearson + Bonferroni). Let Ptrain={p1,…,pN}\mathcal{P}_{\mathrm{train}} = \{p_1,\ldots,p_N\} be the pool from step 1 of CPS: draw NN bootstrap resamples from DtrainD_{\mathrm{train}} and run a prompt optimizer (GEPA) on each. This pool is fixed before DcertD_{\mathrm{cert}} is read.

Fix p∈Ptrainp \in\mathcal{P}_{\mathrm{train}}. Let m=∣Dcertmal∣m = \lvert D_{\mathrm{cert}}^{\mathrm{mal}}\rvert be the number of malicious certification examples (y=1y = 1), and let kpk_p be the number of missed attacks of pp on those examples (malicious inputs that pp incorrectly allows). The missed-attack rate of pp is R(p)R(p), and

kp=∑(Xi,Yi)∈Dcertmal1{f(p,Xi)=0},R^(p)=kpm.k_p = \sum_{(X_i,Y_i)\in D_{\mathrm{cert}}^{\mathrm{mal}}} 1\{f(p,X_i)=0\}, \qquad\widehat{R}(p)=\frac{k_p}{m}.

Therefore, kp∼Bin⁡(m,R(p))k_p \sim\operatorname{Bin}(m,R(p)). Set Rucb(p,Dcert,β)=CP⁡m(kp;β)R_{\mathrm{ucb}}(p,D_{\mathrm{cert}},\beta)=\operatorname{CP}_m(k_p;\beta). Lemma B.1(a) gives, for this fixed prompt,

Pr⁡(R(p)>Rucb(p,Dcert,β))≤β.\Pr\left(R(p)>R_{\mathrm{ucb}}(p,D_{\mathrm{cert}},\beta)\right)\le\beta.

Take β=δR/N\beta=\delta_R/N. A union bound over the NN prompts gives

Pr⁡(R(p)≤Rucb(p,Dcert,δR/N) for all p∈Ptrain)≥1−δR.\Pr\left(R(p)\le R_{\mathrm{ucb}}(p,D_{\mathrm{cert}},\delta_R/N)\text{ for all }p\in\mathcal{P}_{\mathrm{train}}\right)\ge1-\delta_R.

Step 1 therefore produces the certified set

C={p∈Ptrain:Rucb(p,Dcert,δR/N)≤α}.(12)\mathcal{C}=\left\{p\in\mathcal{P}_{\mathrm{train}}:R_{\mathrm{ucb}}(p,D_{\mathrm{cert}},\delta_R/N)\le\alpha\right\}. \tag*{(12)}

On the event above, every p∈Cp\in\mathcal{C} satisfies R(p)≤αR(p)\le\alpha. Define

ER={R(p)≤α for all p∈C}.E_R=\{R(p)\le\alpha\text{ for all }p\in\mathcal{C}\}.

Then Pr⁡(ER)≥1−δR\Pr(E_R)\ge1-\delta_R. The selected prompt p^∈C\hat{p}\in\mathcal{C} satisfies R(p^)≤αR(\hat{p})\le\alpha on ERE_R whenever C≠∅\mathcal{C}\ne\varnothing.

Step 2: Utility near-optimality (Hoeffding + union bound). Let b=∣Dcertben∣b=\lvert D_{\mathrm{cert}}^{\mathrm{ben}}\rvert and K=∣C∣K=\lvert\mathcal{C}\rvert. Step 1 produces C\mathcal{C} in equation 12, so K≤NK\le N. The set C\mathcal{C} is a function of Ptrain\mathcal{P}_{\mathrm{train}} and DcertmalD_{\mathrm{cert}}^{\mathrm{mal}} only. Utility is computed on DcertbenD_{\mathrm{cert}}^{\mathrm{ben}}, which is independent of that sample. Assume C≠∅\mathcal{C}\ne\varnothing. Conditionally on C\mathcal{C}, Hoeffding’s inequality gives, for every p∈Cp\in\mathcal{C},

Pr⁡(∣U^(p)−U(p)∣>t)≤2e−2bt2.\Pr\left(\left\lvert\widehat{U}(p)-U(p)\right\rvert>t\right)\le2e^{-2bt^2}.

A union bound over the certified prompts, together with K≤NK\le N, gives

Pr⁡(∃p∈C:∣U^(p)−U(p)∣>t)≤2Ke−2bt2≤2Ne−2bt2.(13)\Pr\left(\exists p\in\mathcal{C}:\left\lvert\widehat{U}(p)-U(p)\right\rvert>t\right)\le2Ke^{-2bt^2}\le2Ne^{-2bt^2}. \tag*{(13)}

where C⊆Ptrain\mathcal{C}\subseteq\mathcal{P}_{\mathrm{train}} is the set produced in Step 1, consisting of those prompts with Rucb(p;δR/N)≤αR_{\mathrm{ucb}}(p;\delta_R/N)\le\alpha. Choose

t=log⁡(2N/δU)2b,t=\sqrt{\frac{\log(2N/\delta_U)}{2b}},

so that 2Ne−2bt2=δU2Ne^{-2bt^2}=\delta_U. The same bound holds unconditionally, because the right-hand side does not depend on C\mathcal{C}. Let

p⋆∈arg⁡max⁡p∈CU(p),p^∈arg⁡max⁡p∈CU^(p).p^\star\in\arg\max_{p\in\mathcal{C}}U(p),\qquad\hat{p}\in\arg\max_{p\in\mathcal{C}}\widehat{U}(p).

On the event that ∣U^(p)−U(p)∣≤t\lvert\widehat{U}(p)-U(p)\rvert\le t for every p∈Cp\in\mathcal{C},

U(p^)≥U^(p^)−t≥U^(p⋆)−t≥U(p⋆)−2t.U(\hat{p})\ge\widehat{U}(\hat{p})-t\ge\widehat{U}(p^\star)-t\ge U(p^\star)-2t.

Together with the empty-pool convention, this uniform event is contained in

EU={C=∅}∪{U(p^)≥max⁡p∈CU(p)−ϵU},ϵU=2t=2log⁡(2N/δU)b.E_U=\{\mathcal{C}=\varnothing\}\cup\left\{U(\hat{p})\ge\max_{p\in\mathcal{C}}U(p)-\epsilon_U\right\},\qquad\epsilon_U=2t=\sqrt{\frac{2\log(2N/\delta_U)}{b}}.

Hence Pr⁡(EU)≥1−δU\Pr(E_U)\ge1-\delta_U.

Step 3: Combine. By Assumption 1, conditional on the label counts (m,b)(m,b) (and on DtrainD_{\mathrm{train}}, which is treated as fixed throughout the proof), the malicious and benign certification subsamples DcertmalD_{\mathrm{cert}}^{\mathrm{mal}} and DcertbenD_{\mathrm{cert}}^{\mathrm{ben}} are independent. This independence justifies treating the benign certification sample as fresh in Step 2 after the certified set CC has been formed using DcertmalD_{\mathrm{cert}}^{\mathrm{mal}}.

From Steps 1 and 2,

Pr⁡(ERc)≤δR,Pr⁡(EUc)≤δU.\Pr(E_R^c) \le\delta_R,\qquad\Pr(E_U^c) \le\delta_U.

Therefore, by the union bound,

Pr⁡(ER∩EU)=1−Pr⁡(ERc∪EUc)≥1−δR−δU.\Pr(E_R \cap E_U) = 1 - \Pr(E_R^c \cup E_U^c) \ge1 - \delta_R - \delta_U.

On the event ER∩EUE_R \cap E_U, if C=∅C = \varnothing then CPS abstains and the claimed event holds by convention. Otherwise the selected prompt p^∈C\hat{p} \in C satisfies both

R(p^)≤αR(\hat{p}) \le\alpha

and

U(p^)≥max⁡p∈CU(p)−ϵU.U(\hat{p}) \ge\max_{p \in C} U(p) - \epsilon_U.

Hence,

Pr⁡(R(p^)≤α and U(p^)≥max⁡p∈CU(p)−ϵU)≥1−δR−δU,\Pr\left(R(\hat{p}) \le\alpha\text{ and } U(\hat{p}) \ge\max_{p \in C} U(p) - \epsilon_U\right) \ge1 - \delta_R - \delta_U,

with the convention that the event holds when C=∅C = \varnothing. Since this bound holds conditionally on (m,b,Dtrain)(m,b,D_{\mathrm{train}}), it also holds unconditionally, which proves Proposition 1.

Proof of Proposition 2

We restate the proposition for convenience. Throughout, π1=π(⋅∣y=1)\pi_1 = \pi(\cdot\mid y = 1) and π0=π(⋅∣y=0)\pi_0 = \pi(\cdot\mid y = 0).

Proposition 2 (CRISP risk control and utility; restated). Suppose Assumption 1 holds, and let α,δR∈(0,1)\alpha,\delta_R \in(0,1) and T∈NT \in\mathbb{N}. Let p^\hat{p} and CTC_T be the prompt and the certified set returned by CRISP (Algorithm 1). For any δU∈(0,1)\delta_U \in(0,1) and ϵU=2log⁡(2K/δU)/b\epsilon_U = \sqrt{2\log(2K/\delta_U)/b} with K=2T+1,K = 2T + 1,

Pr⁡Dcertmal∼π1m,  Dcertben∼π0b(R(p^)≤α and U(p^)≥max⁡p∈CTU(p)−ϵU)≥1−δR−δU,(14)\Pr_{D_{\mathrm{cert}}^{\mathrm{mal}}\sim\pi_1^m,\;D_{\mathrm{cert}}^{\mathrm{ben}}\sim\pi_0^b}\left(R(\hat{p}) \le\alpha\text{ and } U(\hat{p}) \ge\max_{p \in C_T} U(p) - \epsilon_U\right) \ge1 - \delta_R - \delta_U, \tag*{(14)}

with the convention that when CT=∅C_T = \varnothing, CRISP abstains and the event in (11) holds.

Overview. As in Proposition 1, we control risk using DcertmalD_{\mathrm{cert}}^{\mathrm{mal}} and utility using DcertbenD_{\mathrm{cert}}^{\mathrm{ben}}, and combine the two guarantees at the end. The utility argument is essentially unchanged from Proposition 1, since CRISP does not access DcertbenD_{\mathrm{cert}}^{\mathrm{ben}} until the final selection step. The key difference lies in the risk argument. In round tt, the prompt submitted by CRISP for certification is generated using certificates released in earlier rounds, which themselves depend on DcertmalD_{\mathrm{cert}}^{\mathrm{mal}}. Thus, the candidates are not fixed before the calibration data are observed, and the fixed-family Bonferroni argument used in Proposition 1 no longer applies. We instead handle this adaptivity using the transcript-counting argument underlying Guess-and-Check (Rogers et al., [28]), adapted to the one-sided Clopper–Pearson construction in Algorithm 2. The proof proceeds through three ingredients:

  1. Every value released by GnC dominates a Clopper–Pearson bound computed on DcertmalD_{\mathrm{cert}}^{\mathrm{mal}} (Lemma B.2);

  1. Conditional on the training data and the rewriter’s randomness, the entire search is a deterministic function of the sequence of check outcomes, allowing each check to be analyzed as though its prompt were fixed in advance (Lemma B.3);

  1. The confidence levels assigned by Algorithm 2 sum to at most δR\delta_R over all possible sequences of check outcomes (Lemma B.4).

State of GnC. GnC is stateful. Its state (i,f,Γ)(i,f,\Gamma) records the number ii of checks performed so far, the number ff of those checks that failed, and the ordered list Γ=(γ1,…,γf)\Gamma=(\gamma_1,\ldots,\gamma_f) of discretization levels used at the failed checks, with repeated values kept. Initially (i,f,Γ)=(0,0,∅)(i,f,\Gamma)=(0,0,\varnothing). A call to GnC on a prompt pp made in state (i,f,Γ)(i,f,\Gamma) uses the confidence level

β=δRcicfν,ν=(if)∏ℓ=1fN(γℓ),cj=6π2(j+1)2,(15)\beta=\frac{\delta_R c_i c_f}{\nu},\qquad\nu=\binom{i}{f}\prod_{\ell=1}^{f}N(\gamma_\ell),\qquad c_j=\frac{6}{\pi^2(j+1)^2}, \tag*{(15)}

where N(γ)=⌊1/γ⌋+1N(\gamma)=\lfloor1/\gamma\rfloor+1 is the size of the grid defined below. In cjc_j only, π\pi denotes the numerical constant rather than the data distribution, so that ∑j≥0cj=1\sum_{j\ge0}c_j=1. The level β\beta is fixed by the state at the start of the call, before ii or ff is incremented, and the same β\beta is used for every bound in that call. The call proceeds in three stages.

Guess. GnC computes the training empirical risk a^g=R^(p,Dtrain)\hat{a}_g=\widehat{R}(p,D_{\mathrm{train}}) and slack τg=Rucb(p,Dtrain,β)−a^g\tau_g=R_{\mathrm{ucb}}(p,D_{\mathrm{train}},\beta)-\hat{a}_g. If a^g+τg>α\hat{a}_g+\tau_g>\alpha, it returns rp=1r_p=1. In this case DcertmalD_{\mathrm{cert}}^{\mathrm{mal}} is not accessed and the state is unchanged.

Check. Otherwise the call is a check: it accesses DcertmalD_{\mathrm{cert}}^{\mathrm{mal}}, sets i←i+1i\leftarrow i+1, and computes a^h=R^(p,Dcert)\hat{a}_h=\widehat{R}(p,D_{\mathrm{cert}}) and τh=Rucb(p,Dcert,β)−a^h\tau_h=R_{\mathrm{ucb}}(p,D_{\mathrm{cert}},\beta)-\hat{a}_h, so that a^h+τh=CPm(kp;β)\hat{a}_h+\tau_h=\mathrm{CP}_m(k_p;\beta). The check passes if a^g+τg>a^h+τh\hat{a}_g+\tau_g>\hat{a}_h+\tau_h, that is, if the training bound is strictly more conservative than the holdout bound. GnC then returns rp=a^g+τgr_p=\hat{a}_g+\tau_g, which reveals nothing about DcertmalD_{\mathrm{cert}}^{\mathrm{mal}} beyond the fact that the check passed.

Discretize. Otherwise the check fails, and GnC sets f←f+1f\leftarrow f+1. For γ>0\gamma>0, the operator [a]γ=γ⌊a/γ⌋[a]_\gamma=\gamma\lfloor a/\gamma\rfloor rounds a∈[0,1]a\in[0,1] down to the grid Grid⁡(γ)={0,γ,2γ,…,γ⌊1/γ⌋}\operatorname{Grid}(\gamma)=\{0,\gamma,2\gamma,\ldots,\gamma\lfloor1/\gamma\rfloor\}. Hence [a]γ∈Grid⁡(γ)[a]_\gamma\in\operatorname{Grid}(\gamma), [a]γ>a−γ[a]_\gamma>a-\gamma, and ∣Grid⁡(γ)∣=N(γ)|\operatorname{Grid}(\gamma)|=N(\gamma). Since a^h\hat{a}_h is an average over the m=∣Dcertmal∣m=|D_{\mathrm{cert}}^{\mathrm{mal}}| malicious certification examples, GnC forms

G={γ∈(0,τg):2e−2m(τg−γ)2≤β}=(0,τg−wβ],wβ=log⁡(2/β)2m,G=\left\{\gamma\in(0,\tau_g):2e^{-2m(\tau_g-\gamma)^2}\le\beta\right\}=(0,\tau_g-w_\beta],\qquad w_\beta=\sqrt{\frac{\log(2/\beta)}{2m}},

which is non-empty if and only if τg>wβ\tau_g>w_\beta. If G≠∅G\ne\varnothing, GnC sets γ=max⁡G=τg−wβ>0\gamma=\max G=\tau_g-w_\beta>0, appends γ\gamma to Γ\Gamma, and returns rp=[a^h]γ+τgr_p=[\hat{a}_h]_\gamma+\tau_g. If G=∅G=\varnothing, it returns rp=⊥r_p=\perp and CRISP halts. No further check is made after ⊥\perp, and ⊥\perp never certifies a prompt, since ⊥≰α\perp\nleq\alpha.

Check histories. A check history is a finite sequence h=(o1,…,oi)h=(o_1,\ldots,o_i) of check outcomes, where oℓ=✓o_\ell=\checkmark if the ℓ\ell-th check passed, and oℓ=vo_\ell=v if it failed and GnC released the grid value v=[a^h]γv=[\hat{a}_h]_\gamma. We write i(h)i(h) for its length and f(h)f(h) for its number of failures.

Lemma B.2 (Every released certificate dominates a holdout bound). Consider a call to GnC on a prompt pp that runs at level β\beta and returns rp≠⊥r_p\ne\perp. Then:

rp≥Rucb(p,Dcert,β)=CPm(kp;β).r_p\ge R_{\mathrm{ucb}}(p,D_{\mathrm{cert}},\beta)=\mathrm{CP}_m(k_p;\beta).

Proof. There are three possible cases.

  1. If a^g+τg>α\hat{a}_g+\tau_g>\alpha, the call exits at the guess stage and returns rp=1r_p=1. Since CPm(kp;β)≤1\mathrm{CP}_m(k_p;\beta)\le1, we have rp≥CPm(kp;β)r_p\ge\mathrm{CP}_m(k_p;\beta).

  1. Otherwise, the call reaches the check stage. If

a^g+τg>a^h+τh=CPm(kp;β),\hat{a}_g+\tau_g>\hat{a}_h+\tau_h=\mathrm{CP}_m(k_p;\beta),

the check succeeds and returns rp=a^g+τgr_p=\hat{a}_g+\tau_g. Hence, directly from the check condition, rp≥CPm(kp;β)r_p\ge\mathrm{CP}_m(k_p;\beta).

  1. Finally, suppose the check fails and G≠∅G\ne\varnothing. Then GnC returns

rp=[a^h]γ+τgr_p=[\hat{a}_h]_\gamma+\tau_g

for some γ∈G\gamma\in G. Since rounding down by γ\gamma loses less than γ\gamma, [a^h]γ>a^h−γ[\hat{a}_h]_\gamma>\hat{a}_h-\gamma, and therefore

rp>a^h+(τg−γ).r_p>\hat{a}_h+(\tau_g-\gamma).

By the definition of GG,

2e−2m(τg−γ)2≤β,2e^{-2m(\tau_g-\gamma)^2} \le\beta,

so

τg−γ≥log⁡(2/β)2m>log⁡(1/β)2m.\tau_g-\gamma\ge\sqrt{\frac{\log(2/\beta)}{2m}} > \sqrt{\frac{\log(1/\beta)}{2m}}.

Thus, using Lemma B.1(b) with n=mn=m and k=kp=ma^hk=k_p=m\hat{a}_h,

rp>a^h+log⁡(1/β)2m≥CP⁡m(kp;β).r_p > \hat{a}_h+\sqrt{\frac{\log(1/\beta)}{2m}} \ge\operatorname{CP}_m(k_p;\beta).

Thus every nontrivial released certificate dominates the corresponding holdout bound.

Lemma B.3 (The search is determined by the check history). Fix DtrainD_{\mathrm{train}} and the internal algorithm randomness ξ\xi. For every check history hh, let php_h denote the prompt submitted at the next check, whenever such a check occurs. Then php_h, the GnC state (i(h),f(h),Γh)(i(h),f(h),\Gamma_h), and the confidence level βh\beta_h are deterministic functions of (Dtrain,ξ,h)(D_{\mathrm{train}},\xi,h). In particular, php_h and βh\beta_h are fixed without direct access to DcertmalD_{\mathrm{cert}}^{\mathrm{mal}}.

Interpretation. Fix DtrainD_{\mathrm{train}} and ξ\xi. Before using DcertmalD_{\mathrm{cert}}^{\mathrm{mal}}, we could imagine running CRISP for every possible sequence of check outcomes and constructing the full tree of possible prompts {ph}h\{p_h\}_h. This tree is determined without seeing DcertmalD_{\mathrm{cert}}^{\mathrm{mal}}. The holdout data only determine which branch of this already-defined tree is followed.

Proof. We argue recursively over the length of the history. At the root, the seed prompt, DtrainD_{\mathrm{train}}, and ξ\xi are fixed. Given a history hh, all certificates released at preceding checks are determined by hh. Hence the certified and uncertified populations, the parent selected by CRISP, the training scores, and the hard examples supplied to the rewriter are fixed, so the next proposed prompt is fixed. Any calls that terminate at the guess stage are also determined entirely by these quantities and DtrainD_{\mathrm{train}}, and therefore introduce no additional dependence on the holdout.

Likewise, hh determines the number of preceding checks and failures and the previously released discretization levels, so the state (i(h),f(h),Γh)(i(h),f(h),\Gamma_h) is fixed. Consequently,

βh=δRci(h)cf(h)νh\beta_h=\frac{\delta_R c_{i(h)}c_{f(h)}}{\nu_h}

is fixed. If the next call reaches the check stage and fails, the only new information released is the discretized value v=[a^h]γhv=[\hat{a}_h]_{\gamma_h}, where γh\gamma_h and its grid are fixed by the current state. Thus every possible continuation is determined by appending either a successful check or one of these finitely many released grid values. Therefore the entire tree of possible checks can be constructed from (Dtrain,ξ)(D_{\mathrm{train}},\xi) before observing DcertmalD_{\mathrm{cert}}^{\mathrm{mal}}; the holdout determines only which branch of this tree is followed.

Lemma B.4 (Counting check histories). Let Hi,f\mathcal{H}_{i,f} denote the set of check histories of length ii containing ff failed checks. For h∈Hi,fh\in\mathcal{H}_{i,f}, let Γh=(γ1,…,γf)\Gamma_h=(\gamma_1,\ldots,\gamma_f) be the discretization levels used at its failures and define

νh=(if)∏j=1fN(γj),N(γ)=∣Grid⁡(γ)∣=⌊1γ⌋+1.\nu_h=\binom{i}{f}\prod_{j=1}^{f}N(\gamma_j), \qquad N(\gamma)=|\operatorname{Grid}(\gamma)|=\left\lfloor\frac{1}{\gamma}\right\rfloor+1.

Then

∑h∈Hi,f1νh≤1.\sum_{h\in\mathcal{H}_{i,f}}\frac{1}{\nu_h}\le1.

Consequently, if βh=δRcicf/νh\beta_h=\delta_Rc_i c_f/\nu_h, then

∑i≥0∑f=0i∑h∈Hi,fβh≤δR.\sum_{i\ge0}\sum_{f=0}^{i}\sum_{h\in\mathcal{H}_{i,f}}\beta_h\le\delta_R.

Proof. Fix the number of checks ii and failures ff. A history h∈Hi,fh \in H_{i,f} is determined by two things: the ff positions at which failures occur, denoted by F⊆{1,…,i}F \subseteq\{1,\ldots,i\}, with ∣F∣=f|F|=f, and the discretized values released at those failures, v1,…,vfv_1,\ldots,v_f.

First fix a set of failure positions F⊆{1,…,i}F \subseteq\{1,\ldots,i\}, with ∣F∣=f|F|=f. There are (if)\binom{i}{f} possible choices of FF. Once FF is fixed, the remaining uncertainty in the history hh comes only from the released values v1,…,vfv_1,\ldots,v_f.

By Lemma B.3, once the previous outcomes (v1,…,vj−1)(v_1,\ldots,v_{j-1}) are fixed, the discretization level at the jj-th failure, γj\gamma_j, is also fixed. The released value at that failure, vjv_j, can take at most N(γj)=∣Grid⁡(γj)∣N(\gamma_j)=|\operatorname{Grid}(\gamma_j)| possible values, i.e., vj∈Grid⁡(γj)v_j \in\operatorname{Grid}(\gamma_j).

Therefore, for a fixed failure-position set FF, the possible histories hh form a tree in which the jj-th failure branches into at most N(γj)N(\gamma_j) possible values vjv_j. Assign each such branch the weight 1/N(γj)1/N(\gamma_j). Since the weights of all children vj∈Grid⁡(γj)v_j \in\operatorname{Grid}(\gamma_j) of any node sum to at most 11,

∑vj∈Grid⁡(γj)1N(γj)≤1.\sum_{v_j\in\operatorname{Grid}(\gamma_j)} \frac{1}{N(\gamma_j)} \le1.

Therefore, by induction over the ff failure levels, the total weight of all reachable leaves with failure-position set FF is at most 11:

∑h:F(h)=F∏j=1f1N(γj(h))≤1.\sum_{h:F(h)=F}\prod_{j=1}^{f}\frac{1}{N(\gamma_j(h))}\le1.

Since

1νh=(if)−1∏j=1f1N(γj(h)),\frac{1}{\nu_h}=\binom{i}{f}^{-1}\prod_{j=1}^{f}\frac{1}{N(\gamma_j(h))},

and there are (if)\binom{i}{f} possible failure-position sets FF, summing over all histories h∈Hi,fh \in H_{i,f} gives

∑h∈Hi,f1νh≤(if)(if)−1=1.\sum_{h\in H_{i,f}}\frac{1}{\nu_h}\le\binom{i}{f}\binom{i}{f}^{-1}=1.

Finally, using the confidence level assigned to history hh, βh=δRcicf/νh\beta_h=\delta_R c_i c_f/\nu_h, we obtain

∑i≥0∑f=0i∑h∈Hi,fβh=δR∑i≥0ci∑f=0icf∑h∈Hi,f1νh≤δR(∑i≥0ci)(∑f≥0cf)=δR,\begin{aligned} \sum_{i\ge0}\sum_{f=0}^{i}\sum_{h\in H_{i,f}}\beta_h =\delta_R\sum_{i\ge0}c_i\sum_{f=0}^{i}c_f\sum_{h\in H_{i,f}}\frac{1}{\nu_h} \\ &\le\delta_R\left(\sum_{i\ge0}c_i\right)\left(\sum_{f\ge0}c_f\right)=\delta_R, \end{aligned}

because the weights cjc_j satisfy ∑j≥0cj=1\sum_{j\ge0}c_j=1.

Thus,

∑h∈Hi,f1νh≤1and∑i,f,hβh≤δR.\sum_{h\in H_{i,f}}\frac{1}{\nu_h}\le1 \qquad\text{and}\qquad \sum_{i,f,h}\beta_h\le\delta_R.

Proof of Proposition 2.

Step 1: Risk control (Guess-and-Check + union bound). Recall that all probabilities are conditional on (m,b,Dtrain,ξ)(m,b,D_{\mathrm{train}},\xi).

Let PT\mathcal{P}_T be the pool of prompts present when CRISP terminates, either after TT rounds or when GnC returns ⊥\perp and the search breaks. The search starts from the seed prompt p0p_0 and submits at most two proposals per round, one utility rewrite and one risk rewrite, and it makes no further calls once GnC returns ⊥\bot. Let

CT={p∈PT:rp≤α}(16)C_T = \{p \in\mathcal{P}_T : r_p \le\alpha\} \tag*{(16)}

be the certified set returned by CRISP. The symbol ⊥\bot is never ≤α\le\alpha, so a halted call does not certify its prompt. Therefore

∣CT∣≤∣PT∣≤2T+1=K.|C_T| \le|\mathcal{P}_T| \le2T + 1 = K.

Define

ER={R(p)≤α for all p∈CT},E_R = \{R(p) \le\alpha\text{ for all } p \in C_T\},

so that

ERc={∃p∈CT:R(p)>α}.E_R^c = \{\exists p \in C_T : R(p) > \alpha\}.

For every potential check history hh, define

Bh={R(ph)>Rucb(ph,Dcert,βh)}.B_h = \{R(p_h) > R_{\mathrm{ucb}}(p_h,D_{\mathrm{cert}},\beta_h)\}.

By Lemma B.3, conditional on DtrainD_{\mathrm{train}} and ξ\xi, the pair (ph,βh)(p_h,\beta_h) is fixed before DcertmalD_{\mathrm{cert}}^{\mathrm{mal}} is observed. Hence, by Lemma B.1(a),

Pr⁡(Bh)≤βh.\Pr(B_h) \le\beta_h.

Suppose ERcE_R^c occurs. Then the realized CRISP run certifies some prompt ph∈CTp_h \in C_T with R(ph)>αR(p_h) > \alpha. Since php_h is certified, rph≤αr_{p_h} \le\alpha, and by Lemma B.2,

rph≥Rucb(ph,Dcert,βh).r_{p_h} \ge R_{\mathrm{ucb}}(p_h,D_{\mathrm{cert}},\beta_h).

Therefore

R(ph)>α≥rph≥Rucb(ph,Dcert,βh),R(p_h) > \alpha\ge r_{p_h} \ge R_{\mathrm{ucb}}(p_h,D_{\mathrm{cert}},\beta_h),

so BhB_h occurs. Thus ERc⊆⋃hBhE_R^c \subseteq\bigcup_h B_h, and

Pr⁡(ERc)≤Pr⁡(⋃hBh)since ERc⊆⋃hBh≤∑hPr⁡(Bh)by the union bound≤∑hβhby xref‘Lemma B.1(a)‘≤δRby xref‘Lemma B.4‘.\begin{aligned} \Pr(E_R^c) &\le\Pr\left(\bigcup_h B_h\right) && \text{since } E_R^c \subseteq\bigcup_h B_h\\ &\le\sum_h \Pr(B_h) && \text{by the union bound}\\ &\le\sum_h \beta_h && \text{by {xref}`Lemma B.1(a)`}\\ &\le\delta_R && \text{by {xref}`Lemma B.4`.} \end{aligned}

Step 1 therefore produces the certified set CTC_T in equation (16). On the event ERE_R, every p∈CTp \in C_T satisfies R(p)≤αR(p) \le\alpha, and

Pr⁡(ER)≥1−δR.(17)\Pr(E_R) \ge1 - \delta_R. \tag*{(17)}

The selected prompt p^∈CT\hat{p} \in C_T satisfies R(p^)≤αR(\hat{p}) \le\alpha on ERE_R whenever CT≠∅C_T \ne\varnothing.

Step 2: Utility near-optimality (Hoeffding + union bound). Let b=∣Dcertben∣b = |D_{\mathrm{cert}}^{\mathrm{ben}}| and K=2T+1K = 2T + 1. Step 1 produces CTC_T in equation (16), so ∣CT∣≤K|C_T| \le K. The set CTC_T is a function of DtrainD_{\mathrm{train}}, ξ\xi, and DcertmalD_{\mathrm{cert}}^{\mathrm{mal}} only: GnC reads only missed-attack rates, parents are ranked on DtrainD_{\mathrm{train}}, and the rewriter uses only ξ\xi and DtrainD_{\mathrm{train}} (Lemma B.3). Utility is computed on DcertbenD_{\mathrm{cert}}^{\mathrm{ben}}, which is independent of that sample. Assume CT≠∅C_T \ne\varnothing. Conditionally on CTC_T, Hoeffding’s inequality gives, for every p∈CTp \in C_T,

Pr⁡(∣U^(p)−U(p)∣>t)≤2e−2bt2.\Pr\left(|\widehat{U}(p) - U(p)| > t\right) \le2e^{-2bt^2}.

A union bound over the certified prompts, together with ∣CT∣≤K|C_T| \le K, gives

Pr⁡(∃p∈CT:∣U^(p)−U(p)∣>t)≤2∣CT∣e−2bt2≤2Ke−2bt2,(18)\Pr\left(\exists p \in C_T : |\widehat{U}(p) - U(p)| > t\right) \le2|C_T|e^{-2bt^2} \le2Ke^{-2bt^2}, \tag*{(18)}

where CT⊆PTC_T \subseteq\mathcal{P}_T is the set produced in Step 1, consisting of those prompts with rp≤αr_p \le\alpha. Choose

t=log⁡(2K/δU)2b.t = \sqrt{\frac{\log(2K/\delta_U)}{2b}}.

so that 2Ke−2bt2=δU2Ke^{-2bt^{2}}=\delta_U. The same bound holds unconditionally, because the right-hand side does not depend on CTC_T. Let

p⋆∈arg⁡max⁡p∈CTU(p),p^∈arg⁡max⁡p∈CTU^(p).p^\star\in\arg\max_{p\in C_T} U(p), \qquad\hat{p} \in\arg\max_{p\in C_T} \widehat{U}(p).

On the event that ∣U^(p)−U(p)∣≤t|\widehat{U}(p)-U(p)|\le t for every p∈CTp\in C_T,

U(p^)≥U^(p^)−t≥U^(p⋆)−t≥U(p⋆)−2t.U(\hat{p})\ge\widehat{U}(\hat{p})-t\ge\widehat{U}(p^\star)-t\ge U(p^\star)-2t.

Together with the empty-pool convention, this uniform event is contained in

EU={CT=∅}∪{U(p^)≥max⁡p∈CTU(p)−ϵU},ϵU=2t=2log⁡(2K/δU)b.E_U=\left\{C_T=\varnothing\right\}\cup\left\{U(\hat{p})\ge\max_{p\in C_T}U(p)-\epsilon_U\right\},\qquad\epsilon_U=2t=\sqrt{\frac{2\log(2K/\delta_U)}{b}}.

Hence Pr⁡(EU)≥1−δU\Pr(E_U)\ge1-\delta_U.

Step 3. Combine. By Assumption 1, conditional on the label counts (m,b)(m,b) (and on DtrainD_{\mathrm{train}} and the rewriter randomness ξ\xi, which are treated as fixed throughout the proof), the malicious and benign certification subsamples DcertmalD^{\mathrm{mal}}_{\mathrm{cert}} and DcertbenD^{\mathrm{ben}}_{\mathrm{cert}} are independent. This independence justifies treating the benign certification sample as fresh in Step 2 after the certified set CTC_T has been formed using DcertmalD^{\mathrm{mal}}_{\mathrm{cert}}.

From Steps 1 and 2,

Pr⁡(ERc)≤δR,Pr⁡(EUc)≤δU.\Pr(E_R^c)\le\delta_R,\qquad\Pr(E_U^c)\le\delta_U.

Therefore, by the union bound,

Pr⁡(ER∩EU)=1−Pr⁡(ERc∪EUc)≥1−δR−δU.\Pr(E_R\cap E_U)=1-\Pr(E_R^c\cup E_U^c)\ge1-\delta_R-\delta_U.

On the event ER∩EUE_R\cap E_U, if CT=∅C_T=\varnothing then CRISP abstains and the claimed event holds by convention. Otherwise the selected prompt p^∈CT\hat{p}\in C_T satisfies both

R(p^)≤αR(\hat{p})\le\alpha

and

U(p^)≥max⁡p∈CTU(p)−ϵU.U(\hat{p})\ge\max_{p\in C_T}U(p)-\epsilon_U.

Hence,

Pr⁡(R(p^)≤α and U(p^)≥max⁡p∈CTU(p)−ϵU)≥1−δR−δU,\Pr\left(R(\hat{p})\le\alpha\ \text{and}\ U(\hat{p})\ge\max_{p\in C_T}U(p)-\epsilon_U\right)\ge1-\delta_R-\delta_U,

with the convention that the event holds when CT=∅C_T=\varnothing. Since this bound holds conditionally on (m,b,Dtrain,ξ)(m,b,D_{\mathrm{train}},\xi), it also holds unconditionally, which proves Proposition 2.

Remark B.1 (Validity in the sense of Definition 1). Step 1 controls every certified prompt, not only the selected one: on ERE_R, max⁡p∈CTR(p)≤α\max_{p\in C_T}R(p)\le\alpha. Together with Step 2, this shows that CRISP is (α,ϵU,δR+δU)(\alpha,\epsilon_U,\delta_R+\delta_U)-valid in the sense of Definition 1, with C^=CT\widehat{C}=C_T.

Remark B.2 (Stopping rule and the untouched benign split). The risk bound equation 16 depends neither on TT nor on how the search ends. It holds whether CRISP runs all TT rounds or stops at ⊥\bot, so stopping affects only utility, and the number of rounds enters the guarantee only through KK in Step 2. Step 2 relies on CTC_T being fixed by DtrainD_{\mathrm{train}}, ξ\xi, and DcertmalD^{\mathrm{mal}}_{\mathrm{cert}} before the benign sample is used. If the search ranked parents or decided when to stop using DcertbenD^{\mathrm{ben}}_{\mathrm{cert}}, then CTC_T would depend on DcertbenD^{\mathrm{ben}}_{\mathrm{cert}} and the conditional Hoeffding argument would fail. This is why IMPROVEUTILITY ranks parents by U^(⋅,Dtrain)\widehat{U}(\cdot,D_{\mathrm{train}}) rather than by U^(⋅,Dcert)\widehat{U}(\cdot,D_{\mathrm{cert}}).

References

  1. [1]Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning, 2025. URL https://arxiv.org/abs/2507.19457, 2(3):4, 2025.
  2. [2]Anastasios Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In International conference on learning representations, volume 2024, pp. 55198–55218, 2024.
  3. [3]Anastasios N Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494–591, 2023.
  4. [4]Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.01052, 2021.arxiv.org/abs/2110.01052
  5. [5]Anthropic. Messages API reference. https://platform.claude.com/docs/en/api/messages, 2026. Accessed 2026-09-16.
  6. [6]Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM, 68(6):1–34, 2021.arxiv.org/abs/2101.02703
  7. [7]C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413, 1934.DOI
  8. [8]Olive Jean Dunn. Multiple comparisons among means. Journal of the American Statistical Association, 56(293):52–64, 1961. doi: 10.1080/01621459.1961.10482090.DOI
  9. [9]Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. The reusable holdout: Preserving validity in adaptive data analysis. Science, 349(6248):636–638, 2015.DOI
  10. [10]Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797, 2023.arxiv.org/abs/2309.16797
  11. [11]Google Cloud. Model armor overview. https://docs.cloud.google.com/model-armor/overview, 2026.
  12. [12]Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. The Innovation, 7(6), 2026.
  13. [13]Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In International Conference on Learning Representations (ICLR), 2024.
  14. [14]Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American statistical association, 58(301):13–30, 1963.
  15. [15]John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Arushi Somani, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, et al. Best-of-n jailbreaking. Advances in Neural Information Processing Systems, 38:73137–73221, 2025.arxiv.org/abs/2412.03556
  16. [16]Hakan Inan, Kartikeya Upasani, Jiafeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madiam Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023.arxiv.org/abs/2312.06674
  17. [17]Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36:24678–24704, 2023.arxiv.org/abs/2307.04657
  18. [18]Steven Orla Kimbrough, Gary J Koehler, Ming Lu, and David Harlan Wood. On a feasible–infeasible two-population (fi-2pop) genetic algorithm for constrained optimization: Distance tracing and no free lunch. European journal of operational research, 190(2):310–327, 2008.DOI
  19. [19]Hao Li, Xiaogeng Liu, Ning Zhang, and Chaowei Xiao. Piguard: Prompt injection guardrail via mitigating overdefense for free. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 30420–30437, 2025.DOI
  20. [20]Meta. Prompt guard. https://www.llama.com/docs/model-cards-and-prompt-formats/prompt-guard/, 2024. Model card.
  21. [21]Microsoft. Harm categories in azure AI content safety. https://learn.microsoft.com/azure/ai-services/content-safety/concepts/harm-categories, 2026a. Accessed 2026-09-16.
  22. [22]Microsoft. Azure OpenAI reasoning models. https://learn.microsoft.com/azure/foundry/openai/how-to/reasoning, 2026b. Accessed 2026-09-16.
  23. [23]Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 9340–9366, 2024.
  24. [24]ProtectAI. Fine-tuned DeBERTa-v3-base for prompt injection detection. https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2, 2024. Hugging Face model card.
  25. [25]Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 7957–7968, 2023.
  26. [26]Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi Jaakkola, and Regina Barzilay. Conformal language modeling. In International Conference on Learning Representations, volume 24, pp. 11654–11681, 2024.arxiv.org/abs/2306.10193
  27. [27]Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, 2023.
  28. [28]Ryan Rogers, Aaron Roth, Adam Smith, Nathan Srebro, Om Thakkar, and Blake Woodworth. Guaranteed validity for empirical approaches to adaptive data analysis. In International Conference on Artificial Intelligence and Statistics, pp. 2830–2840. PMLR, 2020.
  29. [29]Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In International Conference on Learning Representations, volume 24, pp. 25055–25083, 2024.
  30. [30]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005.
  31. [31]Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, et al. Large language models are not fair evaluators. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 9440–9450, 2024.
  32. [32]Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations (ICLR), 2024.
  33. [33]Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Benchmarking and defending against indirect prompt injection attacks on large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp. 1809–1820, 2025.
  34. [34]Matteo Zecchin, Sangwoo Park, and Osvaldo Simeone. Adaptive learn-then-test: Statistically valid and efficient hyperparameter selection. arXiv preprint arXiv:2409.15844, 2024.
  35. [35]Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharap u, Olivia Sturman, and Oscar Wahltinez. Shieldgemma: Generative ai content moderation based on gemma. arXiv preprint arXiv:2407.21772, 2024.arxiv.org/abs/2407.21772
  36. [36]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023.arxiv.org/abs/2306.05685
  37. [37]Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The eleventh international conference on learning representations, 2023.arxiv.org/abs/2211.01910
  38. [38]Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. In International Conference on Learning Representations, volume 2025, pp. 51257–51296, 2025.
  39. [39]Thomas Zollo, Todd Morrill, Zhun Deng, Jake Snell, Toniann Pitassi, and Richard Zemel. Prompt risk control: A rigorous framework for responsible deployment of large language models. In International Conference on Learning Representations, volume 2024, pp. 4045–4067, 2024.arxiv.org/abs/2311.13628

Paper details

vlnc:W_6vgj33zq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI tools

View statement
Looking for related papers…
Contents