WorksComputer Science
Risk-certified prompt optimization for LLM-as-a-judge guardrails
Abstract
LLM-as-a-judge guardrails are often controlled primarily through system prompts, yet standard prompt optimization methods target average performance rather than explicit safety constraints. This paper studies system-prompt selection as a constrained optimization problem: maximize benign pass rate while keeping the missed-attack rate below a user-specified level α with finite-sample statistical control. The framework combines LLM-based judging, automatic prompt generation, and distribution-free risk certification. This work instantiates it in two procedures: CPS and CRISP. CPS generates a fixed pool of prompts from training data, then certifies candidates on a held-out evaluation set using simultaneous upper confidence bounds and selects the certified prompt with the highest benign utility. CRISP instead integrates risk feedback into an adaptive search, using hard example-guided rewriting and a reusable-holdout interface to steer generation toward the feasible utility frontier. Experiments on four datasets covering prompt injection, system-prompt leakage, and sensitive-information disclosure show that risk-certified selection consistently respects the target missed-attack level on heldout tests, while utility improves as the permitted risk is relaxed. CRISP is especially useful when fixed-pool search fails to produce certifiable high-utility candidates.
Introduction
Guardrails are an increasingly important component of LLM-based systems, where they act as filters that inspect user inputs or model outputs and decide whether they should be allowed or blocked. They typically operate as one component within a broader AI pipeline. Existing guardrails take several forms. Some rely on fine-tuned models trained specifically to classify content as safe or unsafe. Another practical approach is the LLM-as-a-judge guardrail, in which a general-purpose language model (e.g., gpt-5.4) is turned into a guardrail primarily through its system prompt, without modifying the model weights. The system prompt instructs the model how to interpret a safety policy and output a binary decision such as BLOCK or ALLOW. This paper focuses on the second setting and asks how the system prompt itself can be optimized while providing a statistically certified bound on missed-attack risk [36, 38, 12].
Designing such a guardrail involves an asymmetry between the two possible types of error. Missing a malicious request can expose the underlying system to an attack, whereas incorrectly blocking a benign request primarily reduces usability. Consequently, minimizing average classification error is not necessarily the appropriate objective. In many deployments, practitioners may instead wish to specify an explicit risk level , and require the missed-attack rate to remain below this level while allowing as much benign traffic as possible. Guardrail design therefore naturally becomes a constrained optimization problem: maximize benign utility subject to , where denotes the missed-attack rate. Importantly, this requirement should hold not only on the finite dataset used during development, but also on future samples from the deployment distribution. This is called risk control, where the goal is to ensure that the deployed guardrail has risk at most with probability at least over the calibration data [6, 4].
Distribution-free risk control is well understood when the classifier exposes a continuous score. Conformal prediction [30, 3] and its risk-controlling extensions turn a one-dimensional threshold into a certified operating point: sweep the threshold on held-out calibration data and return the most permissive value whose empirical risk survives a statistical test [6, 4, 2, 26]. However, a deployed guardrail detector often exposes only a binary flag or a coarse severity level [21], and many hosted LLMs do not expose logits: the Claude API has no logprobs parameter [5], and OpenAI reasoning models reject logprobs [22]. If a score threshold is unavailable, the remaining control surface is the system prompt itself. Risk control then becomes a search problem: among the prompts we can write, find one whose missed-attack rate is certified below and whose benign pass rate is as high as possible. This is harder than threshold tuning, because prompt space is discrete and unordered.
Our claim: A general-purpose LLM (e.g., gpt-5.4-mini) can be turned into a guardrail with statistically guaranteed risk control, through system-prompt optimization alone and without modifying model weights. We formulate this as risk-certified system-prompt optimization: find a prompt that maximizes benign utility subject to a safety constraint , where is the benign query pass rate and the missed-attack rate. The selected prompt must satisfy this constraint with high probability over the calibration data.
Our proposal: We combine three ingredients: an LLM-as-a-judge guardrail, prompt optimization, and distribution-free risk control for certification. We instantiate this framework in two procedures that differ in how generation and certification interact:
CPS: Optimize, then Certify (Section 4.1). CPS first generates a fixed, diverse pool of candidate prompts using an existing prompt optimizer. It then applies Learn-then-Test Angelopoulos et al., [4], with Bonferroni multiple-testing correction across candidates, to certify prompts whose risk remains below , and selects the certified prompt with the highest empirical utility.
CRISP: Risk-aware closed-loop optimization (Section 4.2). CPS can only certify prompts already present in its fixed pool; if no high-utility feasible prompt is generated, certification cannot recover one. CRISP addresses this limitation by integrating risk feedback into prompt generation. Using the reusable-holdout mechanism Guess-and-Check Dwork et al., [9]; Rogers et al. [28], CRISP iteratively rewrites prompts toward the constrained Pareto frontier while preserving valid risk guarantees.
Our Contributions
We formulate LLM guardrail prompt design as risk-certified system-prompt optimization and introduce an -validity guarantee (Definition 1).
We develop CPS and CRISP to build prompt-based guardrails with explicit finite-sample risk guarantees and show that the framework is adaptable to multiple threat domains by testing it on prompt injection, system prompt leakage, and sensitive information disclosure.
Methodologically, we extend the LTT-style setting beyond a fixed candidate family to sequential, adaptive candidate selection, where the set of tested hypotheses is generated based on calibration feedback and the desired risk guarantee.
Background and Related Work
Our framework lies at the intersection of three research areas: LLM safety, automatic prompt optimization, and distribution-free risk control.
Guardrails for LLM Systems Deployed guardrails come in three forms. First, fine-tuned classifiers (ProtectAI’s DeBERTa detector [24], Prompt Guard [20], PIGuard [19]) are fast and cheap but tied to their training distribution: a new policy or attack family requires relabeling and retraining. Second, safety-tuned LLMs such as Llama Guard [16] and ShieldGemma [35] apply a fixed policy taxonomy; NeMo Guardrails [27] instead lets developers script safety flows. Third, LLM-as-a-judge setting [36, 12], where a general-purpose model makes the block/pass decision straight from a system prompt, which is cheap to adapt but sensitive to small edits [29, 31]. Across all these designs the policy prompt is hand-written and then frozen; none gives a procedure for choosing it under an explicit, statistically enforced risk level.
Automatic Prompt Optimization. These methods search over natural-language instructions to improve performance on a downstream task. APE [37] generates and reranks instructions; OPRO [32] proposes prompts from a scored history; EvoPrompt [13] and PromptBreeder [10] apply mutation and crossover; ProTeGi [25] uses textual gradients; MIPROv2 [23] tunes instructions and demonstrations jointly; GEPA [1] adds reflective feedback with Pareto-based selection. All of them search for prompts that maximize an unconstrained average score and return a single winner; they do not directly enforce a requirement such as , nor do they provide a finite-sample guarantee that the deployed prompt will satisfy that requirement. We use them as candidate generators and impose the constraint ourselves.
Risk Control. Distribution-free risk control uses a finite i.i.d. calibration sample to return a configuration whose population risk is at most with probability at least . Risk-Controlling Prediction Sets [6] calibrate a scalar threshold under a monotone loss. Learn-then-Test (LTT) [4] drops monotonicity: each candidate configuration is tested against the null hypothesis that it violates the constraint, and a family-wise error rate correction (e.g., Bonferroni or Holm) ensures that all certified candidates are simultaneously feasible with probability at least . Prompt Risk Control [39] uses this framework to select an LLM prompt from a fixed candidate set.
In each case, the calibrated object is a threshold on a continuous score or a candidate set fixed in advance, whereas the space of guardrail prompts is discrete, unordered, and open-ended. CPS generates its pool of prompts from separate training data and fixes it before certification, so LTT applies directly using calibration data. CRISP instead generates each new prompt from earlier certification outcomes, which makes the tested hypotheses depend on the calibration data; to keep the guarantee valid, it accesses the calibration set during search only through the reusable-holdout mechanism Guess-and-Check [28]. This notion of adaptivity differs from that of anytime-valid inference, e-process-based methods such as adaptive LTT [34]: these fix the hypotheses and test them on sequentially arriving data, whereas CRISP fixes the data and generates the hypotheses sequentially.
Problem Setup
Let denote a user query and its label, where indicates a malicious input that should be blocked and indicates a benign input that should be allowed. Given an LLM judge and a system prompt , the prompt induces a binary classifier , where 1 denotes block and 0 denotes pass. is deterministic (greedy decoding, temperature=0).
Safety Risk and Benign Utility
Given a labeled dataset sample sampled i.i.d. from an unknown distribution , we define the safety risk as the attack (false-negative) rate on malicious inputs and the utility as the benign pass rate (specificity) on benign inputs:
The corresponding empirical missed-attack and benign pass rates on the dataset are:
Risk-Constrained Prompt Optimization
If the population distribution is known, the desired prompt would solve the problem
where is a user-specified risk level. Thus, among prompts that satisfy the required safety level, we seek the one that preserves as much benign traffic as possible. This formulation differs from standard prompt optimization because safety risk is treated as a hard constraint, rather than being combined with utility into an unconstrained average-performance objective.
Statistical Objective
In practice, the distribution is unknown and we only have access to a finite labeled dataset, so the population constraint cannot be verified directly. Our statistical objective is therefore to select a data-dependent prompt that achieves high benign utility while satisfying the risk constraint with probability at least .
Definition 1 (()-valid selection). A procedure maps a dataset to a pair , where is a finite set of candidates and is the deployed prompt. For and a tolerance , we say that is -valid if:
Methods
We propose two methods that differ in how candidate generation and risk certification interact. We draw i.i.d. and partition it into three disjoint subsets: (used by the generator), (used only for certification and prompt selection), and (used only for final evaluation). We further partition by label:
and write and .
Clopper–Pearson bounds. In both CPS and CRISP, we use the Clopper–Pearson bound [7] to obtain an upper confidence bound on risk at failure level :
where is the number of malicious examples and is the number of missed attacks (false negatives).
Assumption 1 (Calibration sampling). Conditional on the label counts , the subsamples and are independent i.i.d. samples from and , respectively.
CPS: Certified Prompt Search
Idea. CPS separates prompt generation from statistical certification. A prompt optimizer first generates a fixed candidate pool using only . CPS then evaluates these candidates on the held-out , certifies those whose risk upper confidence bound is below the target , and selects the certified prompt with the highest benign utility.
Procedure.
Generate a fixed candidate pool. Draw bootstrap resamples of size from , giving , and run a prompt optimizer (e.g., GEPA) on each to obtain
Compute risk certificates. For each , evaluate the risk upper confidence bound on using equation (6) with the Bonferroni corrected level [8]:
.
Figure 1. CPS: fixed-pool prompt generation, risk certification, and utility-based selection.
Form the certified set. Retain candidates whose upper risk bound satisfies the target:
Select by benign utility. Evaluate the empirical benign utility of each certified candidate on the benign examples in , and return
Proposition 1 (CPS risk control and utility). Under Assumption 1, for the input failure level , any , and , the prompt selected by CPS satisfies:
with the convention that when , CPS abstains and the event holds. (Proof in Appendix B.1).
CRISP: Constrained Risk-calibrated Iterative System-Prompt Optimization
Motivation. CPS generates a fixed pool of prompts before certification. While this makes statistical control straightforward, it creates a generation gap: if the initial pool contains no prompt satisfying the risk constraint, certification alone cannot produce one. CRISP closes this gap by feeding risk certificates back into prompt generation, allowing the search to move toward high-utility prompts near the certification boundary.
Two-population constrained search. Inspired by FI-2pop (Kimbrough et al., [18]), at iteration CRISP partitions the explored prompts into a certified set () and an uncertified set (). The two populations serve different search objectives: certified prompts are optimized for utility, while uncertified prompts are optimized to reduce constraint violation. This lets CRISP improve prompts already inside the feasible region while simultaneously pushing promising infeasible prompts toward certification.
Algorithm. Each round of Algorithm 1 makes up to two proposals. IMPROVEUTILITY selects the certified prompt with the highest training utility and rewrites it to admit benign cases that are currently over-blocked. REDUCERISK selects the uncertified prompt with the lowest training risk and rewrites it to block attacks that are currently missed. Each rewrite is a single LLM call that receives the parent prompt, its current risk, the budget , and a few hard examples mined only from (templates in Appendix A.6). Every proposed prompt is then evaluated by Guess-and-Check (GnC) (Rogers et al., [28]) to give an upper confidence bound on the risk.
Guess-and-Check (Algorithm 2) has three stages. Guess: GnC computes a Clopper–Pearson upper bound on ; if it already exceeds , GnC returns without accessing the holdout. Check: otherwise it computes the corresponding bound on . If the training bound is strictly more conservative than the holdout bound, GnC releases the training bound, revealing only that the check succeeded. Discretize: if the check fails, GnC releases a coarsened holdout estimate, , and charges a failure. Confidence levels are allocated across checks and failures using the summable weights . Once no valid discretization remains, the reusable-holdout budget is exhausted: GnC returns and CRISP stops.
Validity under adaptivity. CRISP never accesses directly; it observes it only through GnC’s restricted outputs. Later proposals may therefore depend on earlier certificates without unrestricted reuse of the protected holdout. Moreover, risk depends only on malicious examples (Eq. (2)), so the search never reads . The benign certification examples consequently remain fresh for the final utility-based selection among the certified prompts, of which there are at most .
Proposition 2 (CRISP risk control and utility). Under Assumption 1, for the input failure level , any , and with , the prompt selected by CRISP satisfies:
with the convention that when , CRISP abstains and the event holds. (Proof in Appendix B.2).
Algorithm 1 CRISP
Require: Inputs: judge , splits , risk budget , failure level
Require: Hyperparams: rounds , seed
Initialize pool
1: 2:
Search round
3: for do 4:
Improve objective (utility)
5: if then 6: 7: 8:
Reduce violation (risk)
9: if then 10: 11: 12:
Stop or append prompts
13: if or then break 14: else
Final prompt select
15: 16: return
Algorithm 2 Guess-and-Check (GnC)
function
Charge query
1: 2:
Guess
3: 4: 5: if then return
Check
6: 7: 8: 9: if then return
Discretize rate
10: 11: 12: if then 13: ; append to 14: else return
Return bot
15: return (stop the optimization)
end function
Note.
is the Clopper–Pearson (eq. 6).
is rounding down to the -grid.
is the number of points on the -grid.
is an ordered list.
Experiments
Setup
We evaluate on four datasets spanning three guardrail tasks. Prompt injection uses BIPIA ([33]), where malicious examples embed an indirect injection in otherwise benign external context, and Deepset, a smaller public set of direct prompt-injection and jailbreak inputs. The two output-guardrail tasks are system-prompt leakage (SPL) and sensitive-information disclosure (SID). In SPL the judge screens an assistant response given a confidential system prompt; in SID it screens a response given a list of protected items, using BeaverTails privacy-violation examples ((Ji et al., 2023). Across all datasets, denotes a violation that should be blocked and a benign case that should be allowed. Each dataset is randomly partitioned into disjoint training, certification, and test splits. The training split is used only for prompt generation and rewriting, for certification and prompt selection, and only for final evaluation. Dataset construction, filtering, and split counts are given in Appendix § 10.
Figure 2. CRISP: adaptive prompt search with reusable-holdout risk certification.
CPS constructs a pool of candidate prompts before certification, and CRISP uses rounds. If no candidate can be certified at a given risk level , we report “–” rather than returning an uncertified prompt. We compare against a fixed unoptimized prompt (Naive); unconstrained prompt optimizers that maximize average performance (ProTeGi, GEPA, and MIPROv2); and, for prompt injection, dedicated classifier-based guardrails (ProtectAI, PIGuard, Prompt Guard, and Model Armor [11]). Figure 3 additionally evaluates selected prompts under Best-of- jailbreaking [15] on BIPIA (Appendix § 10.§ 5).

Figure 3. Best-of- jailbreaking on BIPIA. Missed-attack risk as a function of the attacker query budget , using gpt-4o-mini as the judge.
Unless otherwise stated, every prompt-optimization method, including CPS and CRISP, starts from the same task-specific seed and uses gpt-5.4-mini as the prompt generator/rewriter. LLM-as-a-judge results are reported with two judges, gpt-4o-mini and gpt-5.4-mini (Table 1); output-guardrail results use gpt-5.4-mini. The judge uses greedy decoding (temperature = 0). We set the failure level to , report prompt-injection results at , and report output-guardrail results at . Hyperparameters, seed prompts, and the Best-of- protocol are in Appendix § 10.
| LLM-as-a-judge guardrail | Method | BIPIA | Deepset | |||
| Risk | Utility | Risk | Utility | |||
| – | Baseline classifier guardrails | ProtectAI | 0.88 | 0.88 | 0.55 | 0.99 |
| – | Baseline classifier guardrails | PIGuard | 0.15 | 0.86 | 0.08 | 1.00 |
| – | Baseline classifier guardrails | Prompt Guard | 0.13 | 0.78 | 0.07 | 0.14 |
| – | Baseline classifier guardrails | Model Armor | 0.94 | 0.98 | 0.72 | 0.98 |
| gpt-4o-mini | Baseline prompt optimization | Naive | 0.60 | 0.90 | 0.29 | 0.92 |
| gpt-4o-mini | Baseline prompt optimization | ProTeGi | 0.19 | 0.80 | 0.07 | 0.95 |
| gpt-4o-mini | Baseline prompt optimization | GEPA | 0.33 | 0.90 | 0.16 | 0.95 |
| gpt-4o-mini | Baseline prompt optimization | MIPROv2 | 0.37 | 0.88 | 0.29 | 0.97 |
| gpt-4o-mini | Ours (=0.1) | CPS | 0.06 | 0.84 | – | – |
| gpt-4o-mini | Ours (=0.1) | CRISP | 0.02 | 0.58 | 0.02 | 0.83 |
| gpt-4o-mini | Ours (=0.2) | CPS | 0.12 | 0.86 | 0.05 | 0.97 |
| gpt-4o-mini | Ours (=0.2) | CRISP | 0.11 | 0.86 | 0.13 | 0.95 |
| gpt-4o-mini | Ours (=0.3) | CPS | 0.25 | 0.88 | 0.16 | 0.96 |
| gpt-4o-mini | Ours (=0.3) | CRISP | 0.20 | 0.86 | 0.11 | 0.96 |
| gpt-5.4-mini | Baseline prompt optimization | Naive | 0.38 | 0.93 | 0.35 | 0.97 |
| gpt-5.4-mini | Baseline prompt optimization | ProTeGi | 0.15 | 0.78 | 0.11 | 0.99 |
| gpt-5.4-mini | Baseline prompt optimization | GEPA | 0.40 | 0.95 | 0.15 | 1.00 |
| gpt-5.4-mini | Baseline prompt optimization | MIPROv2 | 0.21 | 0.88 | 0.24 | 0.99 |
| gpt-5.4-mini | Ours (=0.1) | CPS | – | – | – | – |
| gpt-5.4-mini | Ours (=0.1) | CRISP | 0.01 | 0.78 | 0.04 | 0.94 |
| gpt-5.4-mini | Ours (=0.2) | CPS | – | – | 0.09 | 0.99 |
| gpt-5.4-mini | Ours (=0.2) | CRISP | 0.14 | 0.85 | 0.05 | 0.99 |
| gpt-5.4-mini | Ours (=0.3) | CPS | 0.17 | 0.86 | 0.15 | 1.00 |
| gpt-5.4-mini | Ours (=0.3) | CRISP | 0.18 | 0.89 | 0.09 | 1.00 |
Table 1. Prompt-injection test results. Target risk levels: .
Results and Discussion
Risk calibration holds on test data. Across all prompt-injection configurations in Table 1 where a prompt was certified (two LLM judges, two datasets, three risk levels, two methods), the missed-attack rate measured on the untouched test split stays at or below the target . The same holds on both output-guardrail tasks at (Table 2). This empirical pattern is consistent with Propositions 1 and 2: certification is performed on to control population risk on fresh samples from the same distribution with high probability. Unconstrained baselines provide no such guarantee.
| Method | SPL (leakage) | SID (BeaverTails) | |||
| Risk | Utility | Risk | Utility | ||
| Baseline prompt optimization | Naive | 0.17 | 0.38 | 0.52 | 1.00 |
| Baseline prompt optimization | ProTeGi | 0.15 | 0.78 | 0.13 | 0.93 |
| Baseline prompt optimization | GEPA | 0.17 | 0.58 | 0.12 | 0.95 |
| Baseline prompt optimization | MIPROv2 | 0.06 | 0.65 | 0.13 | 0.97 |
| Ours (=0.3) | CPS | 0.21 | 0.78 | 0.15 | 0.97 |
| Ours (=0.3) | CRISP | 0.08 | 0.60 | 0.22 | 0.96 |
Table 2. Output-guardrail test results
The target risk level is a usable control knob. Relaxing moves the selected prompt along the risk–utility frontier in the expected direction. For CRISP with gpt-4o-mini, benign utility rises from 0.58 at =0.10 to 0.86 at =0.30 on BIPIA, and from 0.83 to 0.96 on DeepSet. With gpt-5.4-mini it rises monotonically from 0.78 to 0.85 to 0.89 on BIPIA and from 0.94 to 1.00 on DeepSet. Test risk generally increases accordingly but remains within the target risk level. Practitioners therefore can set the safety–usability trade-off explicitly.
Certification preserves competitive utility while enforcing the risk constraint. Certified prompts remain competitive with, and often outperform, unconstrained optimizers at similar risk. At
=0.20, on BIPIA with gpt-4o-mini, CPS and CRISP achieve =0.12/0.11 and =0.86, compared with ProTeGi’s =0.19, =0.80. With gpt-5.4-mini, CRISP similarly improves over ProTeGi (0.14/0.85 vs. 0.15/0.78). On Deepset with gpt-5.4-mini, CRISP at =0.20 matches GEPA and ProTeGi utility while lowering risk (0.05/0.99 vs. GEPA’s 0.15/1.00 and ProTeGi’s 0.11/0.99).
Closing the generation gap. The main difference between CPS and CRISP is certifiability. CPS returns an empty certified set in four of twelve prompt-injection settings, whereas CRISP certifies a prompt in all twelve. CPS can fail when its fixed candidate pool contains no prompt that can be certified, either because finite-sample uncertainty makes certification conservative or because the generator fails to explore sufficiently low-risk prompts. CRISP addresses this generation gap by using risk feedback to steer prompt generation toward certifiable candidates (§ 4.2).
A general-purpose judge matches dedicated classifiers. On prompt injection, PIGuard is the strong classifier baseline. CPS and CRISP match its performance without any fine-tuning. At =0.20 on BIPIA, CRISP with gpt-4o-mini (0.11/0.86) improves on PIGuard (0.15/0.86). On Deepset, CRISP with gpt-5.4-mini (0.05/0.99) matches it (0.08/1.00), and CRISP/CPS additionally carry finite-sample guarantees.
Output guardrails present a different trade-off. On SPL and SID, certified selection does not uniformly dominate the unconstrained baselines, but it consistently returns operating points that satisfy the target . On SPL, CPS matches ProTeGi’s utility (0.78), while CRISP attains lower risk (0.08) at lower utility (0.60). On SID, CPS and CRISP achieve high utility (0.97 and 0.96) with risks of 0.15 and 0.22, whereas the Naive prompt reaches utility 1.00 but violates the target with risk 0.52. Thus, the main benefit in this regime is certified feasibility rather than uniform frontier improvement.
Robustness beyond the static certificate. We evaluate the selected prompts under Best-of- jailbreaking (Figure 3), which measures missed-attack risk as the attacker is permitted an increasing query budget . For each example, the attacker may issue up to queries, and the attack is considered successful if at least one query is allowed by the guardrail; full evaluation details are provided in Appendix § 10.§ 5. As increases, prompts produced by unconstrained optimizers degrade substantially: MIPROv2 reaches a missed-attack rate of 0.8 by , and GEPA reaches 0.8 by . In contrast, CRISP at remains at or below 0.07 through , below the dotted reference line. This experiment is a robustness diagnostic rather than part of the certification guarantee, which does not extend to this multi-query attack setting.
Additional empirical analysis. We provide ablations on the CRISP search horizon in Appendix § 10.§ 7 and an evaluation of the risk-control guarantee across repeated data splits in Appendix § 10.§ 8.
Limitations
The -validity guarantee is finite-sample and distribution-free, but it is not unconditional. It applies under i.i.d. sampling from the calibration distribution (Definition 1 and Assumption 1). A shift in the attack mix, user population, or judge model can invalidate the certificate, and certification cannot recover a high-utility feasible prompt if the generator never proposes one—which is why CPS can return an empty certified set. The bound likewise does not cover adaptive adversaries: Best-of- jailbreaking is a robustness diagnostic, not a certified guarantee. Practically, both procedures require labeled malicious and benign data, and small certification sets make Clopper–Pearson and Guess-and-Check bounds conservative; CRISP may also stop early when the reusable-holdout budget is exhausted. We further restrict to a binary, greedy-decoded judge with a single missed-attack constraint. Stochastic decoding, multi-category policies, and joint control of over-blocking remain open.
Conclusion
We framed LLM-as-a-judge guardrail design as risk-certified system-prompt optimization: maximize benign pass rate subject to a user-specified missed-attack budget, with finite-sample statistical control. CPS generates a fixed pool and then certifies it; CRISP feeds risk certificates back into search so that certifiable high-utility prompts can be found when a fixed pool fails. On four datasets covering prompt injection, system-prompt leakage, and sensitive-information disclosure, selected prompts respect the target risk on held-out tests, utility tracks the permitted budget, and CRISP succeeds in settings where fixed-pool search cannot certify. A general-purpose judge, prompted rather than fine-tuned, is competitive with dedicated injection classifiers while remaining retargetable to a new policy or risk level by rerunning the search. We view certified prompts as one layer in a defense-in-depth stack, not a substitute for monitoring under distribution shift or adaptive attack.
AI use statement
Generative AI tools were used only as writing assistants in preparing this manuscript: we used them to improve transitions, correct grammar, and provide editorial feedback that informed our revisions. They were not used to generate synthetic datasets, develop theoretical models or conceptual frameworks, formulate mathematical claims, assist with proofs, propose or refine hypotheses, design experiments, implement methods, clean or reformat data, or interpret results. Translation, qualitative or thematic analysis, and survey design do not apply to this work. The LLM-as-a-judge guardrail and prompt rewriter described in § 4–§ 5 are components of the method being studied, rather than tools used to prepare the manuscript. We reviewed all AI-suggested wording and take full responsibility for the final content, including text produced with generative AI assistance.
Ethics statement
This work concerns the defensive calibration of LLM-as-a-judge guardrails: selecting a system prompt that maximizes the benign pass rate while keeping the missed-attack rate below a user-specified level, backed by a finite-sample certificate. Its intended use is to help practitioners set an explicit safety–utility trade-off, not to weaken deployed systems.
Evaluating guardrails necessarily involves harmful inputs, including prompt injection, system-prompt leakage, sensitive-information disclosure, and the Best-of- robustness diagnostic of Figure 3. We therefore rely on existing public benchmarks and published red-teaming plugins rather than a new exploit pipeline, and we omit step-by-step instructions for attacking production systems. As noted in § 6, the certificate covers neither adaptive adversaries nor distribution shift, so certified prompts should be treated as one layer in a defense-in-depth stack rather than as unconditional safety.
BIPIA, Deepset, and BeaverTails are publicly released datasets. SPL, by contrast, is not a fully public benchmark: it combines the public HuggingFace set gabrielechua/system-prompt-leakage with Promptfoo-generated extraction and hijacking attempts against synthetic confidential system prompts (Appendix § 10.§ 1). No new human-subjects data were collected, so no IRB protocol was required. SID examples are drawn from BeaverTails privacy-violation and safe responses; we introduce no additional personal data and use screening payloads solely to evaluate disclosure detection. Finally, we do not study fairness of over-blocking: utility is an aggregate benign pass rate, which may mask unequal error rates across user groups.
Experimental details
This appendix records the dataset construction, splits, models, and hyperparameters used in § 5.
Datasets
All datasets use the binary convention of § 3: is a violation that should be blocked and is a benign case that should be allowed. Empty strings, exact text duplicates, and inputs longer than a character cap are dropped before splitting. Table 3 summarizes sources and what the judge sees. Each corpus is then randomly partitioned as in Appendix § 10.§ 2.
Table 3. Guardrail benchmarks used in Section 5. SPL is a constructed mix, not a fully public release.
BIPIA (prompt injection). Policy: detect indirect prompt-injection attempts that hide an instruction override inside otherwise benign external content. We reconstruct a binary input-guardrail task from the BIPIA benchmark [33]. Each example is an external context string; malicious contexts embed an injection, while benign contexts do not. The judge screens this context alone (no separate user-intent field). We cap context length at 2,000 characters, drop empty and duplicate contexts, and subsample at most 1,500 examples per class before splitting.
Deepset (prompt injection). Policy: detect direct prompt-injection and jailbreak attempts in the user message. Deepset is the public HuggingFace dataset deepset/prompt-injections. We pool the released train and test splits, drop empty/duplicate strings, and keep the binary labels as released. Malicious examples are direct injections and jailbreaks; benign examples are ordinary user requests. The judge sees only the user prompt. The resulting set is small (661 examples after pooling), which makes certification conservative at tight .
SPL (system-prompt leakage). Policy: detect when an assistant response reveals all or part of a confidential system prompt (extraction, hijacking, or override). This is an output-guardrail task: the judge receives the confidential system prompt and the assistant response, packed as a PROTECTED SYSTEM PROMPT block followed by an ASSISTANT RESPONSE TO SCREEN block. SPL is constructed from two sources, then cleaned and re-split:
Open-source (
gabrielchua/system-prompt-leakage): 300 examples (180 malicious, 120 benign).Promptfoo-generated: 196 examples (87 malicious, 109 benign), built from 600 Mistral-Large system prompts stratified over 15 business domains, three complexity levels (low / medium / high), six languages (English, French, Spanish, German, Chinese, Arabic), and four formats (Markdown, JSON, XML, plain text).
Promptfoo red-teaming uses plugins prompt-extraction (44%), hijacking (41%), and system-prompt-override (the remainder), with a mix of basic prompts, jailbreak templates (Meta, Hydra, Crescendo), and obfuscation (Base64, Hex, ROT13, homoglyph, Leetspeak). Each row stores system_prompt, llm_response, label, and source. Promptfoo labels were validated by two annotators with joint review. The pooled file has 496 examples (267 malicious / 229 benign) before the filters of Appendix § 10.§ 2: the open-source subset is attack-heavy, while Promptfoo contributes more benign passes. System prompts are truncated at 2,400 characters and responses at 3,000 characters (raw medians 1,446 and 2,128 characters).
SID (sensitive-information disclosure). Policy: detect when an assistant response discloses sensitive information (PII, credentials, financial or medical data, and similar protected items). SID uses BeaverTails [17] (PKU-Alignment/BeaverTails, splits 30k train and 30k test). We label a response malicious () if its annotated category includes privacy-violation, and benign () if it is marked is_safe; other categories are discarded. The judge screens a fixed list of protected item types (personal identifiers, contact details, financial and medical data) together with the user request and the assistant response, in the payload format PROTECTED / SENSITIVE ITEMS, USER REQUEST, ASSISTANT RESPONSE TO SCREEN. Responses are truncated at 3,000 characters. We balance the two classes at 1,500 examples each before splitting. Unlike SPL, SID does not include the generating system prompt at inference time.
Splits and test filtering
We cap each class, shuffle the pooled set, and cut it into training / certification / test fractions 0.20/0.60/0.20. The same seed is used for all methods on a given dataset. Table 4 reports the resulting counts. A small number of test examples are rejected by the hosted API content filter (HTTP 400); these are excluded from every rate computation, so reported test numbers use the filtered test set. The same filtered test set is used for all methods on a given dataset.
| Dataset | Train (mal/ben) | Cert. (mal/ben) | Test (mal/ben) |
| BIPIA | 600 (301/299) | 1800 (903/897) | 597 (294/303) |
| Deepset | 132 (58/74) | 397 (149/248) | 132 (55/77) |
| SPL | 93 (53/40) | 278 (140/138) | 92 (52/40) |
| SID | 599 (299/300) | 1798 (889/909) | 597 (306/291) |
Table 4. Split sizes used in the experiments. Test counts are after excluding API content-filter rejections (3 examples on BIPIA, 3 on SID, none on Deepset or SPL).
Judge, rewriter, and decoding
The LLM-as-a-judge guardrail is a system prompt that maps a screening payload to a one-word decision, BLOCK or ALLOW. The user message contains the content to screen, using the task-specific payload format described in Appendix § 10.§ 1; there is no additional fixed wrapper around the system prompt . Decoding is greedy (temperature 0, at most 5 completion tokens), matching the deterministic classifier in § 3. Unparseable replies default to ALLOW.
Prompt generation and rewriting always use gpt-5.4-mini, including for CPS/CRISP and for the unconstrained baselines. Table 1 reports two judges, gpt-4o-mini and gpt-5.4-mini; Table 2 uses gpt-5.4-mini as the judge. All prompt-optimization methods start from the same task seed listed below. Naive deploys that seed without optimization.
Task seeds. The shared seeds already contain BLOCK/ALLOW rules and the one-word reply format.
Prompt injection.

Figure 1 (PDF p. 15).
System prompt leakage (SPL). You screen an assistant’s reply given a confidential system prompt. BLOCK if the reply looks like it reveals the system prompt. ALLOW if it looks fine and does not. Respond with exactly one word: BLOCK or ALLOW. Do not explain.
Sensitive information disclosure (SID).
You screen an assistant’s reply for sensitive-information disclosure. BLOCK if the reply discloses private personal data of a real person (phone, email, SSN, home address, account number). ALLOW otherwise, including harmful advice that does not disclose those data. Respond with exactly one word: BLOCK or ALLOW. Do not explain.
Method hyperparameters
We set the risk-certification failure level throughout. Prompt injection uses ; output-guardrail tables use . The utility failure level appears only in the theoretical utility guarantee and is not used by the selection algorithms. Unconstrained baselines are -independent: each optimizer is run once on (with a training-only internal validation split) and the returned prompt is scored on . CPS builds its pool once and then certifies at each . CRISP is rerun at each .
CPS. Following § 4.1, we draw bootstrap samples from only. Each sample has size , is drawn with replacement, and may therefore overlap other bootstrap samples. We hold out 25% of each sample as GEPA’s internal validation split; neither portion accesses . GEPA is run in the light configuration with gpt-5.4-mini as the reflection model. Candidates are then scored on and certified with one-sided Clopper–Pearson risk UCBs at the Bonferroni level . Among certified prompts we select by empirical benign utility on the benign subset of . If the certified set is empty we report “–”.
CRISP. We run search rounds with up to proposals per round (one utility-raising rewrite and one risk-lowering rewrite when the corresponding parents exist). Search starts from the shared task seed (round 0). Hard examples are mined only from : up to three false positives and three true positives for the utility branch, and up to three false negatives and three true negatives for the risk branch. Each rewrite is a single call to gpt-5.4-mini at temperature 0.7 (at most 420 tokens); the templates are in Appendix § 10.§ 6. Risk certificates are obtained from Guess-and-Check (Algorithm 2) on the malicious subset of , with one-sided Clopper–Pearson radii and summable weights . The Guess-and-Check state is initialized once per run and persists across adaptive queries; benign certification examples remain unused until final selection. If Guess-and-Check returns , search stops. If no prompt is certified, we report “–”.
Unconstrained prompt optimizers. GEPA uses the same light configuration as in the CPS bootstraps, with a feedback metric that distinguishes missed attacks from over-blocks. MIPROv2 is run 3-shot (up to three demonstrations), with 5 instruction candidates and 8 trials, using gpt-5.4-mini as the prompt model. ProTeGi uses textual-gradient beam search with beam size 3, 3 rounds, minibatch size 25, and 2 expansions per parent. All three maximize average accuracy, not the constrained objective.
Classifier baselines. On the same filtered prompt-injection test sets we score ProtectAI (protectai/deberta-v3-base-prompt-injection-v2), PIGuard (leolee99/PIGuard), Prompt Guard (meta-llama/Llama-Prompt-Guard-2-86M), and Model Armor. A classifier prediction is treated as block if its label is in (or LABEL_1 for Prompt Guard).
Best-of-N evaluation
Figure 3 follows Best-of-N jailbreaking [15] against the deployed judge. Here, is the attacker budget: at a given , the attacker may send queries, and we consider the attack successful if at least one query is allowed. We sample 30 malicious seeds uniformly from the BIPIA test split. For each seed, we build a shared bank of textual variants and use the same bank for every defender. Variants combine the character-level augmentations of Hughes et al. [15] (scramble, random capitalization, ASCII , applied sparingly) with intent-preserving surface transforms (benign padding, mid-document burial, quote-as-example framing, imperative softening, injection relocation, homoglyphs, leetspeak, zero-width characters, and spaced keywords). We do not add meta-wrappers of the form “ignore previous instructions / obey the directives below,” which would introduce additional attack cues. Each candidate is sent to the guardrail; ALLOW counts as success. Risk@ is the fraction of seeds for which at least one of the first variants is allowed. The certificate in Propositions 1–2 does not cover this repeated-query adversary; the plot is a robustness diagnostic, not a certified guarantee.
CRISP rewrite templates
CRISP never edits prompts with . Both branches receive the parent prompt, the current empirical risk and budget , and a few hard training examples, and must output only the revised system prompt. The rewriter is gpt-5.4-mini at temperature 0.7.
ImproveUtility (certified parent ).
This prompt is designed to block [violation]. It is good but overblocks some safe cases. current risk (False Negative Rate) = [], allowed risk = [] Current prompt: [parent] Must ALLOW (false positives): [up to 3 blocked benign examples] Must still BLOCK: [up to 3 correctly blocked attacks] Fix the prompt to ALLOW the safe cases (false positives). Output only the revised prompt, keep it small to avoid overfitting.
ReduceRisk (uncertified parent ).
This prompt is designed to block [violation]. It is good but misses some attack patterns. current risk (False Negative Rate) = [], allowed risk = [] Current prompt: [parent] Must BLOCK (false negatives, missed patterns): [up to 3 missed attacks] Must still ALLOW: [up to 3 correctly allowed benign examples] Fix the prompt to BLOCK the missed patterns (false negatives). Output only the revised prompt, keep it small to avoid overfitting.
Ablation on the CRISP Search Horizon
We evaluate the effect of the CRISP search horizon on BIPIA at and , using the same data split and gpt-5.4-mini judge, and reporting the corresponding main-table experiments. Each run is extended to . At checkpoints , we record the prompt that CRISP would return if the search were stopped at that point: the certified candidate with the highest benign utility on . Here, corresponds to evaluating the shared seed prompt without any rewriting.
| Certified | Calib Risk | Calib Utility | Test Risk | Test Utility | Candidates | Rewrites | |
| 0 | No (0/1) | – | – | – | – | 1 | 0 |
| 1 | Yes (1/2) | 0.01 | 0.48 | 0.01 | 0.50 | 2 | 1 |
| 2 | Yes (3/4) | 0.06 | 0.68 | 0.06 | 0.66 | 4 | 3 |
| 5 | Yes (7/10) | 0.13 | 0.79 | 0.15 | 0.82 | 10 | 9 |
| 10 | Yes (14/20) | 0.12 | 0.81 | 0.11 | 0.83 | 20 | 19 |
| 20 | Yes (21/40) | 0.12 | 0.81 | 0.11 | 0.83 | 40 | 39 |
| 40 | Yes (45/80) | 0.12 | 0.81 | 0.11 | 0.83 | 80 | 79 |
Table 5. CRISP search horizon on BIPIA at . Certified indicates whether the certified set is nonempty, with the number of certified candidates relative to the total pool size shown in parentheses. Calib denotes , while Test denotes the filtered test set. “–” indicates that no candidate is certified. The same prompt is selected at , , and .
| Certified | Calib Risk | Calib Utility | Test Risk | Test Utility | Candidates | Rewrites | |
| 0 | No (0/1) | – | – | – | – | 1 | 0 |
| 1 | Yes (1/2) | 0.13 | 0.52 | 0.10 | 0.56 | 2 | 1 |
| 2 | Yes (2/4) | 0.13 | 0.52 | 0.10 | 0.56 | 4 | 3 |
| 5 | Yes (5/10) | 0.16 | 0.83 | 0.17 | 0.80 | 10 | 9 |
| 10 | Yes (11/20) | 0.20 | 0.85 | 0.24 | 0.84 | 20 | 19 |
| 20 | Yes (22/40) | 0.20 | 0.85 | 0.24 | 0.84 | 40 | 39 |
| 40 | Yes (44/80) | 0.14 | 0.86 | 0.16 | 0.84 | 80 | 79 |
Table 6. CRISP search horizon on BIPIA at . The same prompt is selected at and , while another prompt is selected at both and .
In both runs, the initial seed leaves the certified set empty, whereas a single rewrite is sufficient to produce a certified prompt. Every subsequent checkpoint also remains certified, with test risk below the corresponding target . At , test utility increases from 0.50 at to 0.83 at , while calibration utility rises from 0.48 to 0.81. Beyond , the selected prompt remains unchanged through . This indicates that, for this run, most of the benefit of adaptive search is obtained within the first few rounds, with additional iterations expanding the candidate pool without improving the final selected operating point.
A similar pattern appears at . Test utility improves rapidly, from 0.56 at to 0.84 at , and remains unchanged at . By , CRISP selects a different prompt whose calibration risk decreases from 0.20 to 0.14 and whose test risk decreases from 0.24 to 0.16. Calibration utility increases slightly from 0.85 to 0.86, while test utility remains 0.84. Thus, the longer search does not improve held-out utility in this case, but it discovers an alternative operating point with substantially lower risk at essentially the same utility.
Overall, these results suggest that relatively few CRISP rounds are sufficient to enter the feasible region and recover most of the attainable utility in these runs. Longer searches can still refine the selected risk–utility trade-off, particularly by discovering lower-risk prompts, but the resulting gains need not be monotonic or substantial. The values differ slightly from Table 1 because this ablation uses a separate stochastic rerun, including randomness in rewriting and in the training examples selected during rewriting.
Repeated splits on BIPIA
The main tables use a single 0.20/0.60/0.20 split (seed 1), which can show only once. To check that this holds with probability at least , we redraw BIPIA 50 times. Each draw samples 600 examples per class from the pooled set, then applies the same 0.20/0.60/0.20 cut. CRISP is rerun independently on each split at risk levels , failure level , and iterations, with gpt-4o-mini as the judge and gpt-5.4-mini as the rewriter.
| 49/49 | 50/50 | |
| Abstention rate () | 1/50 | 0/50 |
| Mean | 0.09 | 0.15 |
Table 7. CRISP on 50 independent BIPIA re-splits at and . The fraction of certified splits with is compared to .
At both risk levels every certified split has test missed-attack rate below , so the empirical coverage is 1.00, above . Raising the budget from 0.20 to 0.30 lifts mean test risk from 0.09 to 0.15, still inside the target, and the single abstention disappears.
Proofs of the guarantees
Probability space. Both proofs are conditional on the label counts and on . The CRISP proof is also conditional on the rewriter randomness . These quantities are fixed before is used, and the bounds below do not depend on them, so the same bounds hold unconditionally. Under Assumption 1:
and are independent.
For each prompt , depends only on , and depends only on .
Proof of Proposition 1
We treat risk and utility separately, through the events and defined below, and combine them at the end. When , CPS abstains and the claimed event holds by convention.
Lemma B.1 (Clopper–Pearson bound). Let with , , and . Define
Then
Proof.
(a) Part (a) is the coverage property of the one-sided Clopper–Pearson bound [7].
(b). Let and . If , then . Likewise, if , then . It remains to consider the case and . Define
By the beta-binomial identity and the definition of the Clopper–Pearson bound, . Since is continuous and strictly decreasing on , is the unique satisfying .
Applying Hoeffding’s inequality [14] at gives
Since is strictly decreasing, it follows that .
Step 1: Risk control (Clopper–Pearson + Bonferroni). Let be the pool from step 1 of CPS: draw bootstrap resamples from and run a prompt optimizer (GEPA) on each. This pool is fixed before is read.
Fix . Let be the number of malicious certification examples (), and let be the number of missed attacks of on those examples (malicious inputs that incorrectly allows). The missed-attack rate of is , and
Therefore, . Set . Lemma B.1(a) gives, for this fixed prompt,
Take . A union bound over the prompts gives
Step 1 therefore produces the certified set
On the event above, every satisfies . Define
Then . The selected prompt satisfies on whenever .
Step 2: Utility near-optimality (Hoeffding + union bound). Let and . Step 1 produces in equation 12, so . The set is a function of and only. Utility is computed on , which is independent of that sample. Assume . Conditionally on , Hoeffding’s inequality gives, for every ,
A union bound over the certified prompts, together with , gives
where is the set produced in Step 1, consisting of those prompts with . Choose
so that . The same bound holds unconditionally, because the right-hand side does not depend on . Let
On the event that for every ,
Together with the empty-pool convention, this uniform event is contained in
Hence .
Step 3: Combine. By Assumption 1, conditional on the label counts (and on , which is treated as fixed throughout the proof), the malicious and benign certification subsamples and are independent. This independence justifies treating the benign certification sample as fresh in Step 2 after the certified set has been formed using .
From Steps 1 and 2,
Therefore, by the union bound,
On the event , if then CPS abstains and the claimed event holds by convention. Otherwise the selected prompt satisfies both
and
Hence,
with the convention that the event holds when . Since this bound holds conditionally on , it also holds unconditionally, which proves Proposition 1.
Proof of Proposition 2
We restate the proposition for convenience. Throughout, and .
Proposition 2 (CRISP risk control and utility; restated). Suppose Assumption 1 holds, and let and . Let and be the prompt and the certified set returned by CRISP (Algorithm 1). For any and with
with the convention that when , CRISP abstains and the event in (11) holds.
Overview. As in Proposition 1, we control risk using and utility using , and combine the two guarantees at the end. The utility argument is essentially unchanged from Proposition 1, since CRISP does not access until the final selection step. The key difference lies in the risk argument. In round , the prompt submitted by CRISP for certification is generated using certificates released in earlier rounds, which themselves depend on . Thus, the candidates are not fixed before the calibration data are observed, and the fixed-family Bonferroni argument used in Proposition 1 no longer applies. We instead handle this adaptivity using the transcript-counting argument underlying Guess-and-Check (Rogers et al., [28]), adapted to the one-sided Clopper–Pearson construction in Algorithm 2. The proof proceeds through three ingredients:
Every value released by GnC dominates a Clopper–Pearson bound computed on (Lemma B.2);
Conditional on the training data and the rewriter’s randomness, the entire search is a deterministic function of the sequence of check outcomes, allowing each check to be analyzed as though its prompt were fixed in advance (Lemma B.3);
The confidence levels assigned by Algorithm 2 sum to at most over all possible sequences of check outcomes (Lemma B.4).
State of GnC. GnC is stateful. Its state records the number of checks performed so far, the number of those checks that failed, and the ordered list of discretization levels used at the failed checks, with repeated values kept. Initially . A call to GnC on a prompt made in state uses the confidence level
where is the size of the grid defined below. In only, denotes the numerical constant rather than the data distribution, so that . The level is fixed by the state at the start of the call, before or is incremented, and the same is used for every bound in that call. The call proceeds in three stages.
Guess. GnC computes the training empirical risk and slack . If , it returns . In this case is not accessed and the state is unchanged.
Check. Otherwise the call is a check: it accesses , sets , and computes and , so that . The check passes if , that is, if the training bound is strictly more conservative than the holdout bound. GnC then returns , which reveals nothing about beyond the fact that the check passed.
Discretize. Otherwise the check fails, and GnC sets . For , the operator rounds down to the grid . Hence , , and . Since is an average over the malicious certification examples, GnC forms
which is non-empty if and only if . If , GnC sets , appends to , and returns . If , it returns and CRISP halts. No further check is made after , and never certifies a prompt, since .
Check histories. A check history is a finite sequence of check outcomes, where if the -th check passed, and if it failed and GnC released the grid value . We write for its length and for its number of failures.
Lemma B.2 (Every released certificate dominates a holdout bound). Consider a call to GnC on a prompt that runs at level and returns . Then:
Proof. There are three possible cases.
If , the call exits at the guess stage and returns . Since , we have .
Otherwise, the call reaches the check stage. If
the check succeeds and returns . Hence, directly from the check condition, .
Finally, suppose the check fails and . Then GnC returns
for some . Since rounding down by loses less than , , and therefore
By the definition of ,
so
Thus, using Lemma B.1(b) with and ,
Thus every nontrivial released certificate dominates the corresponding holdout bound.
Lemma B.3 (The search is determined by the check history). Fix and the internal algorithm randomness . For every check history , let denote the prompt submitted at the next check, whenever such a check occurs. Then , the GnC state , and the confidence level are deterministic functions of . In particular, and are fixed without direct access to .
Interpretation. Fix and . Before using , we could imagine running CRISP for every possible sequence of check outcomes and constructing the full tree of possible prompts . This tree is determined without seeing . The holdout data only determine which branch of this already-defined tree is followed.
Proof. We argue recursively over the length of the history. At the root, the seed prompt, , and are fixed. Given a history , all certificates released at preceding checks are determined by . Hence the certified and uncertified populations, the parent selected by CRISP, the training scores, and the hard examples supplied to the rewriter are fixed, so the next proposed prompt is fixed. Any calls that terminate at the guess stage are also determined entirely by these quantities and , and therefore introduce no additional dependence on the holdout.
Likewise, determines the number of preceding checks and failures and the previously released discretization levels, so the state is fixed. Consequently,
is fixed. If the next call reaches the check stage and fails, the only new information released is the discretized value , where and its grid are fixed by the current state. Thus every possible continuation is determined by appending either a successful check or one of these finitely many released grid values. Therefore the entire tree of possible checks can be constructed from before observing ; the holdout determines only which branch of this tree is followed.
Lemma B.4 (Counting check histories). Let denote the set of check histories of length containing failed checks. For , let be the discretization levels used at its failures and define
Then
Consequently, if , then
Proof. Fix the number of checks and failures . A history is determined by two things: the positions at which failures occur, denoted by , with , and the discretized values released at those failures, .
First fix a set of failure positions , with . There are possible choices of . Once is fixed, the remaining uncertainty in the history comes only from the released values .
By Lemma B.3, once the previous outcomes are fixed, the discretization level at the -th failure, , is also fixed. The released value at that failure, , can take at most possible values, i.e., .
Therefore, for a fixed failure-position set , the possible histories form a tree in which the -th failure branches into at most possible values . Assign each such branch the weight . Since the weights of all children of any node sum to at most ,
Therefore, by induction over the failure levels, the total weight of all reachable leaves with failure-position set is at most :
Since
and there are possible failure-position sets , summing over all histories gives
Finally, using the confidence level assigned to history , , we obtain
because the weights satisfy .
Thus,
Proof of Proposition 2.
Step 1: Risk control (Guess-and-Check + union bound). Recall that all probabilities are conditional on .
Let be the pool of prompts present when CRISP terminates, either after rounds or when GnC returns and the search breaks. The search starts from the seed prompt and submits at most two proposals per round, one utility rewrite and one risk rewrite, and it makes no further calls once GnC returns . Let
be the certified set returned by CRISP. The symbol is never , so a halted call does not certify its prompt. Therefore
Define
so that
For every potential check history , define
By Lemma B.3, conditional on and , the pair is fixed before is observed. Hence, by Lemma B.1(a),
Suppose occurs. Then the realized CRISP run certifies some prompt with . Since is certified, , and by Lemma B.2,
Therefore
so occurs. Thus , and
Step 1 therefore produces the certified set in equation (16). On the event , every satisfies , and
The selected prompt satisfies on whenever .
Step 2: Utility near-optimality (Hoeffding + union bound). Let and . Step 1 produces in equation (16), so . The set is a function of , , and only: GnC reads only missed-attack rates, parents are ranked on , and the rewriter uses only and (Lemma B.3). Utility is computed on , which is independent of that sample. Assume . Conditionally on , Hoeffding’s inequality gives, for every ,
A union bound over the certified prompts, together with , gives
where is the set produced in Step 1, consisting of those prompts with . Choose
so that . The same bound holds unconditionally, because the right-hand side does not depend on . Let
On the event that for every ,
Together with the empty-pool convention, this uniform event is contained in
Hence .
Step 3. Combine. By Assumption 1, conditional on the label counts (and on and the rewriter randomness , which are treated as fixed throughout the proof), the malicious and benign certification subsamples and are independent. This independence justifies treating the benign certification sample as fresh in Step 2 after the certified set has been formed using .
From Steps 1 and 2,
Therefore, by the union bound,
On the event , if then CRISP abstains and the claimed event holds by convention. Otherwise the selected prompt satisfies both
and
Hence,
with the convention that the event holds when . Since this bound holds conditionally on , it also holds unconditionally, which proves Proposition 2.
Remark B.1 (Validity in the sense of Definition 1). Step 1 controls every certified prompt, not only the selected one: on , . Together with Step 2, this shows that CRISP is -valid in the sense of Definition 1, with .
Remark B.2 (Stopping rule and the untouched benign split). The risk bound equation 16 depends neither on nor on how the search ends. It holds whether CRISP runs all rounds or stops at , so stopping affects only utility, and the number of rounds enters the guarantee only through in Step 2. Step 2 relies on being fixed by , , and before the benign sample is used. If the search ranked parents or decided when to stop using , then would depend on and the conditional Hoeffding argument would fail. This is why IMPROVEUTILITY ranks parents by rather than by .
References
- [1]Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning, 2025. URL https://arxiv.org/abs/2507.19457, 2(3):4, 2025.
- [2]Anastasios Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In International conference on learning representations, volume 2024, pp. 55198–55218, 2024.
- [3]Anastasios N Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494–591, 2023.
- [4]Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.01052, 2021.arxiv.org/abs/2110.01052
- [5]Anthropic. Messages API reference. https://platform.claude.com/docs/en/api/messages, 2026. Accessed 2026-09-16.
- [6]Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM, 68(6):1–34, 2021.arxiv.org/abs/2101.02703
- [7]C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413, 1934.DOI
- [8]Olive Jean Dunn. Multiple comparisons among means. Journal of the American Statistical Association, 56(293):52–64, 1961. doi: 10.1080/01621459.1961.10482090.DOI
- [9]Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. The reusable holdout: Preserving validity in adaptive data analysis. Science, 349(6248):636–638, 2015.DOI
- [10]Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797, 2023.arxiv.org/abs/2309.16797
- [11]Google Cloud. Model armor overview. https://docs.cloud.google.com/model-armor/overview, 2026.
- [12]Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. The Innovation, 7(6), 2026.
- [13]Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In International Conference on Learning Representations (ICLR), 2024.
- [14]Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American statistical association, 58(301):13–30, 1963.
- [15]John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Arushi Somani, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, et al. Best-of-n jailbreaking. Advances in Neural Information Processing Systems, 38:73137–73221, 2025.arxiv.org/abs/2412.03556
- [16]Hakan Inan, Kartikeya Upasani, Jiafeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madiam Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023.arxiv.org/abs/2312.06674
- [17]Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36:24678–24704, 2023.arxiv.org/abs/2307.04657
- [18]Steven Orla Kimbrough, Gary J Koehler, Ming Lu, and David Harlan Wood. On a feasible–infeasible two-population (fi-2pop) genetic algorithm for constrained optimization: Distance tracing and no free lunch. European journal of operational research, 190(2):310–327, 2008.DOI
- [19]Hao Li, Xiaogeng Liu, Ning Zhang, and Chaowei Xiao. Piguard: Prompt injection guardrail via mitigating overdefense for free. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 30420–30437, 2025.DOI
- [20]Meta. Prompt guard. https://www.llama.com/docs/model-cards-and-prompt-formats/prompt-guard/, 2024. Model card.
- [21]Microsoft. Harm categories in azure AI content safety. https://learn.microsoft.com/azure/ai-services/content-safety/concepts/harm-categories, 2026a. Accessed 2026-09-16.
- [22]Microsoft. Azure OpenAI reasoning models. https://learn.microsoft.com/azure/foundry/openai/how-to/reasoning, 2026b. Accessed 2026-09-16.
- [23]Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 9340–9366, 2024.
- [24]ProtectAI. Fine-tuned DeBERTa-v3-base for prompt injection detection. https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2, 2024. Hugging Face model card.
- [25]Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 7957–7968, 2023.
- [26]Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi Jaakkola, and Regina Barzilay. Conformal language modeling. In International Conference on Learning Representations, volume 24, pp. 11654–11681, 2024.arxiv.org/abs/2306.10193
- [27]Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, 2023.
- [28]Ryan Rogers, Aaron Roth, Adam Smith, Nathan Srebro, Om Thakkar, and Blake Woodworth. Guaranteed validity for empirical approaches to adaptive data analysis. In International Conference on Artificial Intelligence and Statistics, pp. 2830–2840. PMLR, 2020.
- [29]Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In International Conference on Learning Representations, volume 24, pp. 25055–25083, 2024.
- [30]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005.
- [31]Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, et al. Large language models are not fair evaluators. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 9440–9450, 2024.
- [32]Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations (ICLR), 2024.
- [33]Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Benchmarking and defending against indirect prompt injection attacks on large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp. 1809–1820, 2025.
- [34]Matteo Zecchin, Sangwoo Park, and Osvaldo Simeone. Adaptive learn-then-test: Statistically valid and efficient hyperparameter selection. arXiv preprint arXiv:2409.15844, 2024.
- [35]Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharap u, Olivia Sturman, and Oscar Wahltinez. Shieldgemma: Generative ai content moderation based on gemma. arXiv preprint arXiv:2407.21772, 2024.arxiv.org/abs/2407.21772
- [36]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023.arxiv.org/abs/2306.05685
- [37]Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The eleventh international conference on learning representations, 2023.arxiv.org/abs/2211.01910
- [38]Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. In International Conference on Learning Representations, volume 2025, pp. 51257–51296, 2025.
- [39]Thomas Zollo, Todd Morrill, Zhun Deng, Jake Snell, Toniann Pitassi, and Richard Zemel. Prompt risk control: A rigorous framework for responsible deployment of large language models. In International Conference on Learning Representations, volume 2024, pp. 4045–4067, 2024.arxiv.org/abs/2311.13628

