betaFeedback

WorksComputer Science

Verified but Vacuous: Three Failure Surfaces of LLM-Driven Verified Code Generation

Abstract

What should ``verified'' mean when evaluating LLM-generated code? Many existing pipelines report a single verification number. We argue this collapses three distinct properties measured in a conditional audit sequence: verifier acceptance (Surface A), \emph{syntactic specification binding (Surface B: does the proof obligation bind to the spec?), and semantic specification fidelity (Surface C: does the spec capture the informal requirement?). Two empirical demonstrations show the surfaces separate. \textbf{First (A≠BA \neq B), via Li@Opus (a same-model reproduction of \citet{li2025dafny}'s Dafny-as-intermediate-representation (IR) pipeline under Claude Opus 4.7) on three Python benchmarks}: it reaches 99.1% verifier acceptance on HumanEval while only 1.9% carry a certificate binding to a spec-defined predicate or function. \textbf{Second (B≠CB \neq C), via VCG on two Rust benchmarks}: we use VCG (which is our own in-house Dafny-as-intermediate pipeline) on Rust because the Rust target lets us enforce fully-verifiable output (every component ships as a Dafny certificate, ensures-binding gate passing by construction). Even inside that construction, 11.48% (MBPP-Rust) and 14.82% (VeriContest LC slice) of structurally-verified samples still fail hidden tests. On the LC slice of VeriContest, VCG's reported End2End rate is 7.79% (535/6870). The cited external rates are 5.51% (GPT-5.5) and 1.88% (single-shot Opus 4.7). These give descriptive differences of 2.3 percentage points and 4.14×\times, respectively; task-budget, repair and harness differences prevent a matched-performance claim.

Introduction

What should “verified” mean when evaluating LLM-generated code? Large language models can generate code that passes tests most of the time. Frontier models solve 70–95% of HumanEval-style problems ([3]) and 60–90% of LeetCode-style benchmarks (newfacade, 2024; [8]). Passing a test suite tells us that no tested input triggered a bug; it does not tell us that the implementation satisfies the requirements on inputs the tests did not visit. For code shipped into production (payment processors, medical dose calculators, security-sensitive parsers), that distinction is crucial.

Formal verification systems (Dafny (Leino, [10]), Lean (de Moura & Ullrich, [5]), F* (Swamy et al., [21]), Verus (Lattuada et al., [9])) can machine-check that an implementation satisfies its formal specification. Recent work has explored LLMs as authors of specs (Ma et al., [14]), impls (Misu et al., [15]), proof tactics (Yang et al., [27]), and complete verified programs (Xie et al., [24]; Li et al., [11]), with promising but basic-level empirical evidence. Existing evaluations of such pipelines typically report a single verification number alongside pass@k: dafny verify accepted N% of the samples, or verus returned zero on N% of them. This paper argues that a single verification number collapses three properties that are distinct and empirically separable. We measure them on conditional populations; their semantic properties are not logically nested.

We came to this decomposition while building VCG, an in-house Dafny-as-intermediate pipeline for LLM-driven verified code generation. VCG’s pre-launch audit surfaced a class of dafny verify-accepted components whose ensures clauses did not reference any spec-defined predicate/function. This violated VCG’s intended external-specification convention. Reference absence alone does not establish semantic vacuity: meaningful inline postconditions can fail this gate, and a named tautology can pass it. Dafny still establishes the obligations generated by the actual program and assumptions; the gate measures a separate structural property. We added a static ensures-binding integrity check to VCG’s verify-repair loop to reject such components. That check turned out to be the simplest instance of a distinct evaluation surface we had been conflating with dafny verify acceptance.

Applying the same static check to a same-model reproduction of Li et al. [11]’s Dafny-as-intermediate pipeline (under Claude Opus 4.7) makes the case concretely: dafny verify accepts 99.1% of samples on HumanEval, but only 1.9% carry a certificate that structurally binds its proof obligation to a spec-defined predicate/function. Fully 94.7% ship no ensures clause at all; another 3.4% contain an ensures clause that does not bind to a spec-defined predicate/function. This is not a defect of Li et al.’s pipeline: their few-shot prompt (Figure 6 of Li et al. [11], reproduced verbatim in our few-shot) is a bodyless method {testEntry} with no postcondition, plus a Main() full of expect statements. Dafny assumes those expectations (not proves them) and checks them at runtime; on this input shape, verifier acceptance establishes the remaining static proof obligations (well-formedness, termination, local method-body proof) under those fixture assumptions, but does not prove the fixture expectations themselves or a general functional postcondition. The mismatch is a defect of the metric we use to summarize the pipeline, not of the pipeline itself.

Contributions. This paper isolates and develops the methodology contribution that motivated our own pipeline design:

  1. A decomposition of verification quality into three conditionally measured, empirically separable failure surfaces (§ 2): Surface A (verifier acceptance), Surface B (syntactic specification binding), and Surface C (semantic specification fidelity). We contribute the ensures-binding gate (§ 3) as a syntactic instance of Surface B for Dafny pipelines that emit distinguishable specification and implementation artifacts: a static AST walk that runs in <10<10 ms per Dafny component.

  1. A three-level analytical decomposition of Surface B, and an empirical A→BA \to B gap of roughly fortyfold on Li@Opus (§ 5). To avoid overclaiming what one syntactic gate establishes, we report certificate quality at three observable levels: Level 1 = verifier accepted (Surface A); Level 2 = Impl.dfy contains at least one ensures clause (architecture-agnostic postcondition presence, which does not require the clause to reference any spec-defined symbol); Level 3 = at least one ensures clause binds to a spec-defined predicate/function (our gate; a Surface-B instance for pipelines that separate spec and impl artifacts). On a same-model reproduction of Li et al.’s Dafny-as-IR pipeline applied to three Python benchmarks of simple coding tasks (HumanEval, LeetCodeDataset, LiveCodeBench v6; LeetCode-style), Level 1 stays at 92–99% while Level 3 stays in a narrow 1.9–2.7% band which has a roughly fortyfold (36–52×) gap. The intermediate Level 2 climbs from 5.3% on HumanEval to 10.61% on LCB v6 (1,032/9,731; 10.9% on the 9,448 test-evaluated subset): the pipeline writes more postconditions on harder problems, but they do not get bound to spec-defined symbols. Level 2 measures postcondition presence without requiring an external symbol, whereas Level 3 measures the external-specification convention. The reported pattern does not show that inline postconditions are semantically weaker, or isolate prompt convention from task difficulty. A controlled comparison is needed to distinguish those explanations.

3. Semantic-fidelity finding: Surface B →\to Surface C also leaks (§ 6). Even inside VCG, which passes our external-spec-binding Surface-B criterion on 100% of shipped verified components by construction (other conceivable Surface B instances may still fail on this population), 11.48% (MBPP-Rust; Rust) and 14.82% (VeriContest LC; Rust) of structurally-verified samples still fail hidden tests. Under the standard trusted-toolchain and correct-oracle assumptions, this is an operational proxy for requirement fidelity, not a unique diagnosis of specification failure.

The sharper claim is that verification quality must be decomposed, and that many existing evaluations report only its weakest surface (AxDafny [2] and VeriContest’s Post2Exe check [24 §4.1] are notable exceptions; Table 6 in Appendix § 14). § 9 closes with the concrete methodological ask that follows: any pipeline shipping certificates should report at least one of Surface B or Surface C alongside its verifier-acceptance rate.

SystemSurface ASurface B (syntactic binding)Surface C (semantic fidelity)
Li et al. [11]YesNot reportedRuntime fixtures only
VeriContest (Xie et al., [24]) end-to-endYesNot reportedPost2Exe on postcondition
AxDafny (Breen et al., [2])YesMissing/trivial ensures, {:axiom}, LLM reviewerNot reported
VCG (our own)YesEnsures-binding gateHidden-test proxy on Rust runs; Post2Exe applied on VeriContest LC (7.79% End2End @1, 535/6870)

Table 6. What existing pipelines and evaluations report on each surface. Most systems that report Surface A do not report either companion surface.

Figure 1 (Appendix § 11) visualises the cascade: Li@Opus’s collapse at A→BA \to B vs. VCG’s collapse at B→CB \to C, with bar widths proportional to the fraction of attempted samples surviving each surface.

Cascade diagram comparing Li@Opus and VCG pipelines

Figure 1. Two pipelines fall out of the cascade at different surfaces. Bar widths are proportional to the fraction of attempted samples surviving each surface (HumanEval: 1640 attempts; VeriContest LC: 6870 attempts). Li@Opus’s collapse is at A→BA \to B (99.1% verifier acceptance, but only 1.9% carry a certificate binding to a spec-defined predicate). VCG closes B on 100% of A by construction; its collapse is at B→CB \to C (22.49% dafny verify passing to 7.79% End2End @1). A pipeline reporting only Surface A would show 99.1% and 22.49% respectively for Li@Opus and VCG and hide the shape of failure in both cases.

Three failure surfaces of LLM-driven verified code generation

LLM-driven formal verification can fail at three logically distinct, empirically separable surfaces. Our audit sequence is conditional: Surface B is evaluated on the population that passes Surface A; Surface C is evaluated on the population that passes Surface B. These population filters do not establish logical implications among the underlying properties.

Surface A: Verifier acceptance. Did the verifier return zero on the emitted proof? For Dafny targets, dafny verify returning zero. For Verus targets, verus returning zero. Necessary but weakest, as we will show in this paper.

Surface B: Syntactic specification binding. Of the certificates the verifier accepts, what fraction actually bind the proof to something meaningful? We propose the ensures-binding gate (§ 3) as a concrete syntactic instance for Dafny: the method must contain at least one ensures clause, and at least one such clause must reference an identifier defined as a predicate or function in the paired specification artifact. This is a syntactic external-reference heuristic for this architecture, not a necessary condition for semantic adequacy. It rejects missing external references, including methods with no ensures clause. A meaningful inline postcondition can also be rejected; reference absence alone is not semantic vacuity. It does not catch semantically-vacuous predicates such as predicate P(x: int) { true }, nor does it establish that the predicate correctly captures the informal requirement; those failure modes live in Surface C.

Surface C: Semantic specification fidelity. Of the specs that survive Surface B, what fraction actually capture the requirement the user wanted? This is what VeriContest’s Post2Exe check [24 §4.1] addresses: compile the emitted postcondition to an executable predicate and run it against positive and negative test cases. If the predicate admits outputs a negative test would reject, the spec is incomplete. Post2Exe is a semantic check complementary to our syntactic Surface B.

Concretely:

  • Surface A fails when the verifier rejects the proof.

  • Surface B fails when the verifier accepts a proof of a certificate lacking the required external-specification reference (§ 5 measures this on Li@Opus, Python). This structural result alone does not establish a semantic failure.

  • Surface C fails when the verifier accepts a proof of a well-formed spec that nonetheless doesn’t capture the intended behaviour (§ 6 measures this on VCG, Rust, which passes our external-spec-binding Surface-B criterion by construction).

Illustrative failure modes. Two minimal Dafny snippets on a canonical “return the maximum of nums” task make the separation concrete. Surface A passes, Surface B fails: the verifier accepts, but no ensures clause binds to the spec predicate.

SurfaceCheckCatchesBlind spot (deferred)
A: verifier acceptancedafny verify (or verus) exits zeroproof gaps, type errors, missing lemmasvacuous or absent ensures clauses
B: syntactic spec bindingsome ensures clause references a spec-defined predicate/functionmissing references to the paired spec artifact, including meaningful inline contractssemantically weak predicates that bind but under-specify
C: semantic spec fidelitypostcondition accepts positive tests and rejects negatives (Post2Exe)weak specs whose predicates admit incorrect outputsrequirements no test suite happens to exercise

Table 1. The audit stages use conditional populations. Verifier acceptance, external-reference checking and finite behavioral probes measure distinct properties; the stages are not a logical hierarchy.

predicate MaxCorrect(nums: seq<int>, r: int)
  requires |nums| > 0
{ (forall i :: 0 <= i < |nums| ==> nums[i] <= r)
  && (exists i :: 0 <= i < |nums| && nums[i] == r) }

method find_max(nums: seq<int>) returns (r: int)
  requires |nums| > 0
{ r := 0; }                         // dafny verify exits 0 (no ensures)

The method has no functional postcondition to prove; other Dafny obligations remain. The strong MaxCorrect predicate is left dangling. Surface A passes; the ensures-binding gate rejects.

Surface A and Surface B pass, Surface C fails: the ensures clause binds to a named spec predicate, but the predicate is too weak to constrain the implementation.

predicate MaxCorrect(nums: seq<int>, r: int)
  requires |nums| > 0
{ r >= 0 }                          // under-specified spec

method find_max(nums: seq<int>) returns (r: int)
  requires |nums| > 0
  ensures MaxCorrect(nums, r)       // syntactic binding satisfied
{ r := 0; }

The ensures clause references MaxCorrect, so the gate accepts. But MaxCorrect accepts any non-negative output; on input [5,3,7] the shipped method returns 0 and the postcondition still evaluates to true. Post2Exe (Surface C) catches this by running the postcondition against a negative test case for which 0≥00 \ge0 falsely accepts a value the requirement rules out.

A pipeline shipping certificates ought to report at least one of Surface B or Surface C alongside Surface A. Many existing evaluations report only Surface A.

The ensures-binding gate: a syntactic instance of Surface B

Let MM be a Dafny method with ensures E1,…,EnE_1, \ldots, E_n, and let SS be the paired specification (Spec.dfy plus any transitively-included support files). Define Preds(S) as the set of predicate and function symbols declared in SS, and Ids(E_i) as the set of identifiers referenced in ensures clause EiE_i. The ensures-binding gate accepts MM iff:

n≥1∧∃i,∃p∈Preds⁡(S):p∈Ids⁡(Ei)n \ge1 \land\exists i, \exists p \in\operatorname{Preds}(S) : p \in\operatorname{Ids}(E_i)

Equivalently: the method has at least one ensures clause, and at least one of those clauses names an identifier that refers to a spec-defined predicate/function. The n≥1n \ge1 conjunct is explicit so the condition is not vacuously satisfied by methods with zero postcondition clauses. The existential form (some clause binds, not every clause) matches the reference implementation (Appendix C) and preserves certificates that pair one binding functional postcondition with auxiliary sanity postconditions that need not independently reference the named predicate. The check is a static AST walk, cheap to run (<10<10 ms per component in practice), and is drop-in for Dafny pipelines that emit distinguishable specification and implementation artifacts (the architectural regime this paper measures against).

What the gate rejects. Certificates with no ensures-clause reference to a specification-defined predicate or function. Some such certificates also fail the intended requirement. A canonical example (from our pre-launch audit of an earlier draft of our own pipeline):

predicate ComputeXCorrect(input: int, result: int)
  requires input >= 0
{ result >= 0 && result == input * 2 }

method compute_x(input: int) returns (result: int)
  requires input >= 0
  ensures result >= 0             // trivially provable, unbound to spec
{ result := 0; }                  // dafny verify returns zero

The ensures result >=0>= 0 clause is provable against result :=0:= 0 without invoking ComputeXCorrect; the shipped code is wrong. The gate rejects this statically at zero SMT cost. It does not catch vacuous specs whose predicate itself does not capture the requirement (Surface C) nor vacuously-easy predicates like predicate P(x: int) { true }; those live in § 6.

Empirical anchor. In VCG’s pre-launch internal audit (across three Python benchmarks), ∼27%\sim27\% of components that had passed dafny verify failed this gate. This is a reported structural rejection rate, not an independently adjudicated semantic-vacuity prevalence. The original per-component records and sampling denominator are needed to validate that estimate. § 5 shows that on a pipeline that is not trying to do that (i.e., Li et al.’s Dafny-as-intermediate approach reproduced on Claude Opus 4.7), the failure rate is not 27% but ∼98%\sim98\%.

Experimental setup

The two demonstrations in § 5 and § 6 share a common backend and evaluation protocol. This section consolidates the setup so those sections can report results without re-introducing it.

Benchmarks. Five benchmarks across two target languages (Table 2). The three Python benchmarks (HumanEval, LeetCodeDataset, LiveCodeBench v6) are LeetCode-style function-completion tasks and drive the A→BA \to B measurement in § 5. The two Rust benchmarks (MBPP-Rust, VeriContest LC slice) drive the B→CB \to C measurement in § 6, where the Rust target lets us enforce fully-verifiable output.

Benchmark# problemsSamples/prob.TargetSource
HumanEval / HumanEval+16410Python[3, 13]
LeetCodeDataset22810Python(newfacade, 2024)
LiveCodeBench v61,05510Python[8]
MBPP-Rust20810Rust[6]
VeriContest LC slice68710Rust[24]

Table 2. Benchmarks used in this study. Samples/prob. is 10 across all VCG, BareLLM, and Li@Opus runs; the VeriContest authors’ reported per-slice Opus 4.7 End2End LC number (1.88%; our same-backend comparator in § 6) is single-shot per problem.

Pipelines compared. Three pipelines share Claude Opus 4.7 as the sole backend. BareLLM is the same-backend minimal baseline: Opus 4.7 emits target-language code directly with no spec, no proof, and no verification scaffolding (used for the pass-rate comparison in § 7). Li@Opus is a same-model reproduction of Li et al. [11]’s Dafny-as-intermediate-representation pipeline; the few-shot prompt is verbatim from Figure 6 of Li et al. [11] (reproduction details in Appendix § 12). VCG is our own seven-agent pipeline: Planner decomposes the problem into components; SpecAgent authors Dafny specs; ImplementationAgent authors implementations; ProverAgent drives a verify-repair loop with dafny verify and the ensures-binding gate; verified components transpile to the target language via dafny build -target:{py,rs}. The two LC-slice External comparators we cite, the VeriContest authors’ per-slice same-backend Opus 4.7 End2End LC number (1.88%) and the strongest previously published End2End LC number (GPT-5.5’s 5.51%), come directly from Xie et al. [24] and are not re-run by us.

Model, sampling, and verifier. All LLM calls hit Anthropic Claude Opus 4.7 via Google Vertex AI. VCG’s verifier stack is Dafny 4.10 with Z3 4.13. Every VCG-shipped verified component passes: (i) dafny verify; (ii) the ensures-binding gate (§ 3); (iii) an extracted-example gate on visible test cases from the problem description; and (iv) dafny build to the target language.

Statistical treatment. pass@1 and pass@10 are computed per Chen et al. [3]’s unbiased estimator over the 10 samples/problem. Reported rates are point estimates; we do not report confidence intervals in the main body. The three primary pipelines use the same reported backend; this does not equalise prompts, generation or repair budgets. The external 1.88% comparator is single-shot while VCG uses ten samples/problem. For equal sample counts, the mean per-problem pass@1 estimator equals the pooled success fraction; ten samples do not inherently inflate it. Unequal procedures still limit the external 4.14× comparison.

Evidence and reproducibility. Across the Python benchmarks, the verifier-accepted counts in Table 3 (2,217 on LeetCodeDataset; 9,731 on LiveCodeBench v6) and the test-evaluated counts in Table 7 (2,208 and 9,448, respectively) differ by 9 and 283 samples that failed downstream Dafny-to-Python compilation (209 on LiveCodeBench v6) or test-harness shimming (9 on LeetCodeDataset; 74 on LiveCodeBench v6) prior to test execution. On LiveCodeBench v6, the 1,058 verifier-accepted samples carrying an ensures clause (10.87% of 9,731) yield 1,032 test-evaluated samples (10.92% of 9,448; 10.61% of 9,731) after 26 compilation/shimming exclusions.

HumanEvalLeetCodeDatasetLCB v6
Total samples (attempted)1,6402,28010,550
Level 1: verifier-accepted (denom.)1,6252,2179,731 *
Level 2: any ensures clause5.3%9.6%10.61%*
Level 3: ensures-binding gate passes1.9%2.7%2.3%

Table 3. Certificate quality on Li@Opus, Claude Opus 4.7, three Python benchmarks. Level-2 and Level-3 rates are conditional on verifier acceptance (Surface A); raw dafny verify pass rates are 99.1% / 97.2% / 92.2%.

Benchmark (Python)With ensuresWithout ensuresDelta
HumanEval (base tests)80.2% (n=86n = 86)94.2% (n=1,539n = 1{,}539)−14.0 pp
LeetCodeDataset59.6% (n=213n = 213)66.1% (n=1,995n = 1{,}995)−6.5 pp
LiveCodeBench v653.2% (n=1,032n = 1{,}032)62.6% (n=8,416n = 8{,}416)−9.4 pp

Table 7. Pass rate by presence of ensures clause on test-evaluated samples, Li@Opus, three Python benchmarks (excluding 9 LeetCodeDataset and 283 LiveCodeBench v6 compilation/shimming dropouts). Samples with an ensures clause pass hidden tests less often than samples without one on every benchmark; this observational association among self-selected outputs does not identify a causal effect of writing specifications.

Verifier acceptance does not imply syntactic specification binding: the Li@Opus A → B gap

We apply the ensures-binding gate to every verifier-accepted sample from a same-model reproduction of Li et al. [11]’s Dafny-as-intermediate pipeline (“Li@Opus”; reproduction details in Appendix § 12). Certificate quality is reported at the three observable levels introduced in Contribution 2: Level 1 (verifier accepted; Surface A), Level 2 (any ensures clause; architecture-agnostic presence check, an inline ensures r == 2 * x counts), and Level 3 (an ensures clause binds to a spec-defined predicate/function; our gate). Level 3 is an external-spec-binding check for pipelines that separate spec and impl artifacts, not a universal test of postcondition meaning, as we will show in this paper.

9,448 samples evaluated for test execution in Table 7 (283 compilation/shimming exclusions), Level 2 prevalence is 10.92% (1,032/9,448). Evaluated against all 9,731 verifier-accepted samples, the 1,032 shipped postcondition samples represent 10.61% (or 1,058/9,731 = 10.87% including the 26 postcondition-bearing exclusions).

Level 2 alone already exposes the gap, architecture-agnostically. Nine out of ten Li@Opus “verified” samples have no ensures clause anywhere in the generated Impl.dfy (94.7% / 90.4% / 89.4% respectively across all accepted samples; 89.1% on the test-evaluated subset). The “specification” is a bodyless method {:testEntry} signature with no postcondition, plus a Main() full of expect statements that Dafny assumes (not proves) at verify time. What the verifier certifies is that these assumed fixtures are consistent enough to admit a local method-body proof. It is not functional correctness against a specification. This is faithful to Li et al.’s benchmarked pipeline, not a replication defect: Figure 6 of Li et al. [11], reproduced verbatim in our few-shot prompt, has exactly this shape.

Level 3 is roughly fortyfold smaller than Level 1 (36–52× across the three benchmarks), and roughly benchmark-invariant. The gate-pass rate stays in a narrow 1.9–2.7% band across all three benchmarks despite level-2 prevalence doubling from HumanEval (5.3%) to LCB v6 (10.61%) as benchmark difficulty rises. Additional ensures clauses at higher difficulty add inline postconditions not tied to a named symbol, rather than externalising specifications. The narrow range suggests the prompt/spec convention is a stronger limiter than benchmark difficulty; whether it persists on further benchmarks is a testable prediction. Other structural (Surface B) and semantic (Surface C) checks are possible (see §3, §8, Table 6 in Appendix § 14).

Syntactic binding does not imply semantic fidelity: the B→CB \to C leak in VCG

The A→BA \to B gap in §5 is stark because Li@Opus’s pipeline puts essentially no structural pressure on the spec. What about a pipeline that does pass our external-spec-binding instance of Surface B by construction? Does that guarantee Surface C?

We measure this on our own architecture VCG, evaluated on two Rust benchmarks (MBPP-Rust and VeriContest LC). VCG is a seven-agent Dafny-as-intermediate pipeline that runs the ensures-binding gate inside the prover’s verify-repair loop and downgrades any component whose certificate would fail the gate. Across five benchmarks and two target languages evaluated in this study, VCG ships 18,008 verified components, 100% of which pass the ensures-binding gate, i.e. 100% pass our external-spec-binding Surface-B criterion (other conceivable Surface B instances may still fail on this population; we do not check those).

Yet some of those shipped-and-structurally-verified samples still fail hidden tests. This provides operational evidence that Surface C (semantic specification fidelity) is a distinct failure surface, under the trusted-toolchain and correct-oracle assumptions stated below.

Surface C operationalised on the two Rust benchmarks. Under the VCG Rust configuration, every component becomes a verification target and no unverified glue is shipped. Under the standard trusted-toolchain and correct-oracle assumptions (Dafny transpilation is sound; the Rust harness runs the intended oracle; the wrapper/marshalling layer preserves semantics), hidden-test failure among structurally-verified all-component samples is a Surface-C proxy. Assigning an individual failure to specification inadequacy additionally requires excluding implementation, composition, translation, wrapper, oracle and resource causes. A sample is fully-verified iff every component in its build package shipped as a gate-passing Dafny certificate.

Two Surface-C instances reported in this section. Both condition on samples that have already cleared Surfaces A and B, but they ask different questions and can catch different failures (Table 4). Table 5 reports the first (hidden-test failure) on both Rust benchmarks; the Post2Exe End2End @1 rate is discussed later in this section on VeriContest.

(i) Hidden-test failure(ii) Post2Exe
Question askedDoes the implementation match the requirement?Does the postcondition itself match the requirement?
MechanismRun the shipped code against hidden testsTranslate the postcondition to an executable predicate; check it accepts positives and rejects negatives
Reported inTable 5 (both Rust benchmarks)End2End @1 (VeriContest LC)

Table 4. Two Surface-C instances reported in this paper. Both condition on samples that have already cleared Surfaces A and B. They ask different questions and can catch different failure modes.

Benchmark (Rust)Total attemptsFully-verifiedB→CB \to C failsB→CB \to C failure rate
MBPP-Rust [6]2,0801,08012411.48%
VeriContest LC [24]6,8701,54522914.82%

Table 5. VCG on the two Rust benchmarks. Fully-verified samples pass our Surface-B (ensures-binding) criterion by construction and ship a dafny verify-accepted certificate on every component. B→CB \to C failure rate is the fraction of those that nonetheless fail hidden tests, a Surface-C proxy under the trusted-toolchain and correct-oracle assumptions. Unconditional failure rate is the same failure count expressed as a share of all attempts (10 samples per problem: 5.96% on MBPP-Rust, 3.33% on VeriContest LC).

The Rust rates on the first operationalisation are in Table 5.

Expressed as a share of ALL attempts (not conditional), the Rust residue is small: 5.96% of MBPP-Rust samples (124/2080) and 3.33% of VeriContest-LC samples (229/6870) are fully-verified but fail tests. These are the two Surface-C proxies reported in this paper; because hidden-test failure can in principle also reflect transpiler unsoundness, harness/oracle disagreement, or benchmark-oracle errors, spec inadequacy is one candidate explanation; these aggregates do not rank the competing causes.

Why the leak exists even with the gate enforced. The ensures-binding gate is an architecture-specific reference check: it verifies that the certificate mentions the spec’s predicates. It cannot verify that those predicates capture the requirement. A spec that names the right predicates but constrains them incompletely will still ship a structurally-gated certificate whose impl passes the proof but does not generalise to hidden tests. Closing that gap requires a semantic check (Post2Exe [24 §4.1] is one option; surface-grounded spec strengthening, lifting concrete examples from the problem description into formal lemmas the spec must satisfy, is another we are pursuing).

Post2Exe as Surface C, applied to VCG on the VeriContest LC slice. We ran Post2Exe (the semantic-fidelity check from Xie et al. 2026) on all 1,316 VCG-verified+passing samples on the VeriContest LC slice (687 problems, 10 samples/problem). 535 pass all four End2End checks (precond-equivalence, postcond-Post2Exe, CodeGen, ProofGen), giving an End2End @1 of 7.79% (535/6870). The cited external LC rates are 5.51% (GPT-5.5) and 1.88% (single-shot Opus 4.7), giving descriptive contrasts of 2.3 percentage points and 4.14×. They are external reported comparisons, not a matched-budget or matched-repair SOTA demonstration. The Post2Exe keep-rate among verified-and-passing LC samples is 40.7% (535 / 1,316). CF-slice evaluation is left to future work.

Postcondition presence and observed pass rates

Both VCG and Li@Opus show the same pattern in the runs measured here: samples that carry any ensures clause pass tests less often than samples without one.

The effect is quantified in Table 7 (Appendix E): −14.0 pp on HumanEval (n=86n = 86 vs. 1,539), −6.5 pp on LeetCodeDataset (n=213n = 213 vs. 1,995), and −9.4 pp on LiveCodeBench v6 (n=1,032n = 1,032 vs. 8,416); samples with any ensures clause vs. samples without, respectively.

These are observational associations conditional on generated postcondition presence. Presence does not establish that a contract is nontrivial, and the groups need not match in problem difficulty, generation failures or repair history. The aggregate table therefore does not identify a causal pass-rate cost of writing specifications. VCG’s reported 3–5 percentage-point difference from BareLLM on Rust runs is a separate architectural comparison with multiple changing components, not an identified instance of the same mechanism.

Reporting pass@k alongside structural binding and task outcomes makes the measured differences visible. A causal explanation requires a controlled intervention with comparable tasks, model, information, sampling and repair budgets, and complete failure accounting.

Related work

LLM-driven verified code generation. The closest prior work is Li et al. [11] (Dafny as intermediate language for LLM-driven verified Python) and VeriContest (Xie et al., [24] (a 946-problem verifiable-Rust benchmark with a Post2Exe semantic check, our closest Surface-C analogue). AutoSpec (Wen et al., [23] and Clover (Sun et al., [20] investigate LLM-generated Dafny specs specifically. VerusBench (Yang et al., [25], RepoVBench (Zhong et al., [28], VeruSAGE (Yang et al., [26], and VeriStruct (Sun et al., [19] evaluate LLM proof generation in Verus at different scales but do not evaluate end-to-end NL →\to verified code in the same shape as Li et al. [11] or VeriContest (Xie et al., [24].

Certificate integrity and vacuous verification. Prior work has noted that verifier acceptance is insufficient: Dafny’s {:axiom} and {:verify false} attributes can bypass proof obligations entirely (Dafny Reference Manual, [4], and community folklore), and any pipeline that ships specs must decide how to handle those attributes. Our paper generalises: the “obvious” vacuity of {:axiom} sits alongside the subtler class of no ensures clause at all (a certificate that proves what Dafny checks even without any postcondition), which the ensures-binding gate catches statically. AxDafny (Breen et al., [2] independently converges on a related family of Surface-B-style checks (missing ensures, trivially-true postconditions, bypass constructs, plus an LLM reviewer for other cheating patterns), see Table 6 in Appendix § 14 for a side-by-side positioning. Clover (Sun et al., [20] proposes closed-loop consistency checks among code, spec, and tests as a related but semantic-rather-than-syntactic integrity mechanism. VeriContest’s Post2Exe check Xie et al., [24] is the closest prior Surface-C instance: compile the emitted postcondition to an executable predicate and run it against positive/negative tests. Ensures-binding and Post2Exe are complementary, syntactic and cheap vs. semantic and expensive; a pipeline shipping certificates ought to report at least one.

Pass@k-only evaluations. Codex (Chen et al., [3], AlphaCode (Li et al., [12], and self-consistency / reflection / multi-agent follow-ups Wang et al., [22]; Shinn et al. [18]; Qian et al. [17] established the pass@k evaluation methodology on HumanEval Chen et al., [3], MBPP Austin et al., [1], and LeetCode-style benchmarks. Code-efficiency metrics Huang et al., [7] also lie on the functional-correctness axis. Our three-surface decomposition is additive to these, not a replacement.

Our contribution is a metric decomposition, not a new pipeline: the three-surface decomposition is formalism-agnostic; our ensures-binding gate is a Dafny-specific Surface-B instance; Post2Exe Xie et al., [24] is a complementary semantic Surface-C instance in Verus. The community is independently converging on structural (Surface B) checks; this paper names the surface and reports the paired A-vs-B measurement on a pipeline that does not enforce a Surface B check itself. Table 6 (Appendix § 14) places the closest prior pipelines and evaluations on the three-surface hierarchy.

Conclusion, limitations, and methodological ask

“Verified” is not a single property. Our three-surface audit separates acceptance, external reference and requirement fidelity. In our same-model reproduction of Li et al.’s pipeline, 99.1% Dafny verifier acceptance on HumanEval corresponds to only 1.9% external-spec binding, a roughly fortyfold gap holding across three Python benchmarks of differing difficulty. On our own architecture that closes this gap by construction, an 11.48% (MBPP-Rust) / 14.82% (VeriContest LC) B →\to C gap remains. On the VeriContest LC slice VCG’s End2End @1 is 7.79% (535/6870), a reported result with descriptive external contrasts of +2.3 pp against 5.51% and 4.14× against 1.88%; budgets and repair procedures are not matched. The methodological ask: any pipeline shipping certificates should report at least one of Surface B (e.g. the ensures-binding gate) or Surface C (e.g. Post2Exe) alongside Surface A; where feasible, both, since Surface B alone leaves the semantic blind spot we demonstrate on VCG, and Surface C alone leaves the syntactic blind spot we demonstrate on Li@Opus.

Limitations. Single model family: all primary results use Claude Opus 4.7. Prompt convention is a plausible explanation of the gap (Li et al.’s few-shot is a bodyless method {testEntry} whose Main() runs expect statements at runtime; the model reproduces that shape), but the reported comparisons do not isolate it from model capability or other pipeline differences. Cross-model generalisation (Sonnet 4.7, GPT-5.5, Gemini 3.1 Pro, open-source models) is open work. One Surface-B instance: the ensures-binding gate is one concrete Surface-B check, not the only one; it does not catch predicate P(x: int) { true } trivialities, {:axiom}-bypassed proofs, or Post2Exe-detectable semantic gaps [24 §4.1]. AxDafny [2] converges on a related Surface-B check that would also work. The methodological ask is “report some Surface-B or Surface-C instance,” not specifically this gate.

AI disclosure section

In this work, we used generative AI tools for assisting with implementing methods, supporting qualitative and thematic data analysis, creating or editing software code and figures, drafting and editing prose, and checking arithmetic and references. We have reviewed all AI-assisted work; LLM-generated code was verified and tested for correctness by the authors, and we take responsibility for the final content of this work.

Cascade figure: A B vs. B C collapse

Figure 1 shows the two-pipeline cascade referenced from § 1.

Li@Opus reproduction details

Codebase. The reproduction consists of prompt files (verbatim from Figure 6 of Li et al. [11]), the verify-repair loop implementation, the compile/shim scripts, and per-benchmark harness adapters.

Adaptations per benchmark.

  • HumanEval / HumanEval+: no adaptation; Li et al.’s prompts assume this shape.

  • LeetCodeDataset: problems reshaped into function-plus-docstring form (adapter in benchmarks.py); Li’s prompts unchanged.

  • LiveCodeBench v6: 611 stdin/stdout problems wrapped as solve(stdin: str) -> str; 444 function-style problems required no wrapping. This is an adaptation of Li’s pipeline, not something the paper specifies.

Numbers reported (Claude Opus 4.7 via Vertex; Vertex rejects the temperature parameter for Opus 4.7 so all samples use default settings).

  • HumanEval base tests: 92.6% pass@1 / 97.0% pass@10 (1,519 / 1,640 samples).

  • HumanEval+ plus tests: 77.4% pass@1 / 84.8% pass@10 (1,270 / 1,640 samples).

  • LeetCodeDataset: 63.4% pass@1 / 79.4% pass@10 (1,445 / 2,280 samples).

  • LiveCodeBench v6: 55.2% pass@1 / 76.5% pass@10 (5,819 / 10,550 samples).

Ensures-binding gate implementation

The gate is a static AST walk over each generated Impl.dfy file. Methods with zero ensures clauses fail the gate immediately (matching the n≥1n \ge1 conjunct in § 3). For every method declaration with at least one ensures clause, the gate:

  1. Parses Impl.dfy with Dafny’s Python-accessible parser and collects every ensures clause as a Dafny expression AST.

  2. Locates the paired Spec.dfy (same component workspace) and any support files transitively include’d from it. Enumerates every top-level predicate and function declaration in these spec files to build the set Preds(S) of spec-defined predicate/function symbols.

  1. For each ensures clause EiE_i, walks its identifier set and checks whether any identifier resolves to a member of Preds⁡(S)\operatorname{Preds}(S). If any single identifier does, the clause is bound; if none of the ensures clauses on the method binds, the certificate is rejected.

Edge cases handled: transitive include chains; function methods that are both spec and impl symbols (only the spec-side declaration counts for Preds⁡(S)\operatorname{Preds}(S)); qualified references (Spec.PredicateName); and forall/exists bindings whose bound variables don’t count against the binding requirement.

Runtime: under 10 ms per component on our runs.

Existing systems on the three-surface audit

Referenced from § 8.

Pass rates by presence of ensures clause

Companion to § 7.

References

  1. [1]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.arxiv.org/abs/2108.07732
  2. [2]Blair Breen, Alexander Letson, Begoña Requena Pozo, and Luca Sarra. AxDafny: Agentic verified code generation in Dafny. arXiv preprint arXiv:2606.32007, 2026. Validation checks include missing ensures, trivially-true postconditions, proof-bypass constructs, and an LLM reviewer for other cheating patterns.arxiv.org/abs/2606.32007
  3. [3]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
  4. [4]Dafny Reference Manual. Dafny documentation: {:axiom}, {:verify false}, {:extern} attributes. https://dafny.org/latest/DafnyRef/DafnyRef, 2024.
  5. [5]Leonardo de Moura and Sebastian Ullrich. The lean 4 theorem prover and programming language. In Conference on Automated Deduction (CADE-28), 2021.DOI
  6. [6]Emergence AI. MBPP-Rust: Rust port of the MBPP dataset, 208 problems covering task ids 11–510, 2026. Released alongside this paper’s evaluation artifacts.
  7. [7]Dong Huang et al. EffiBench: Benchmarking the efficiency of automatically generated code. arXiv preprint arXiv:2402.02037, 2024.arxiv.org/abs/2402.02037
  8. [8]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.arxiv.org/abs/2403.07974
  9. [9]Andrea Lattuada, Travis Hance, Chanhhee Cho, Matthias Brun, Isitha Subasinghe, Yi Zhou, Jon Howell, Bryan Parno, and Chris Hawblitzel. Verus: Verifying Rust programs using linear ghost types. Proceedings of the ACM on Programming Languages (OOPSLA), 2024.DOI
  10. [10]K. Rustan M. Leino. Dafny: An automatic program verifier for functional correctness. In Logic for Programming, Artificial Intelligence, and Reasoning (LPAR-16), 2010.
  11. [11]Yue Li, Georg Zetzsche, and Siva Somayajula. Dafny as verification-aware intermediate language for code generation. In Dafny Workshop, co-located with POPL, 2025. arXiv:2501.06283.arxiv.org/abs/2501.06283
  12. [12]Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, et al. Competition-level code generation with AlphaCode. Science, 2022.DOI
  13. [13]Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems (NeurIPS), 2023. EvalPlus / HumanEval+.arxiv.org/abs/2305.01210
  14. [14]Yifei Ma et al. Specification generation for neural programs via large language models. arXiv preprint arXiv:2402.11897, 2024.
  15. [15]Md. Rakib Hossain Misu et al. Towards AI-assisted synthesis of verified Dafny methods. arXiv preprint arXiv:2402.00247, 2024.DOI
  16. [17]Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. In Association for Computational Linguistics (ACL), 2024.arxiv.org/abs/2307.07924
  17. [18]Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2303.11366
  18. [19]Chuyue Sun, Yifan Sun, Dorsa Amrollahi, Eric Zhang, Shuvendu Lahiri, Shan Lu, David Dill, and Clark Barrett. VeriStruct: AI-assisted automated verification of data-structure modules in Verus. arXiv preprint arXiv:2510.25015, 2025.arxiv.org/abs/2510.25015
  19. [20]Chuyue Sun et al. Clover: Closed-loop verifiable code generation. arXiv preprint arXiv:2310.17807, 2023.arxiv.org/abs/2310.17807
  20. [21]Nikhil Swamy, Cătălin Hrițcu, Chantal Keller, Aseem Rastogi, Antoine Delignat-Lavaud, Simon Forest, Karthikeyan Bhargavan, Cédric Fournet, Pierre-Yves Strub, Markulf Kohlweiss, Jean-Karim Zinzindohoue, and Santiago Zanella-Béguelin. Dependent types and multi-monadic effects in F*. In Principles of Programming Languages (POPL), 2016.
  21. [22]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2203.11171
  22. [23]Cheng Wen et al. Enchanting program specification synthesis by large language models using static analysis and program verification. arXiv preprint arXiv:2404.00762, 2024.arxiv.org/abs/2404.00762
  23. [24]Zhiyuan Xie, Mridul Pawagi, Yifei Liu, Ashutosh Rai, Lianmin Shao, Jason Berberian Jr., Sicheng Che, and Wenting Wang. VeriContest: A competitive-programming benchmark for verifiable code generation. arXiv preprint arXiv:2605.08553, 2026.arxiv.org/abs/2605.08553
  24. [25]Chenyuan Yang, Arina Gladshtein, et al. AutoVerus: Automated proof generation for rust code. arXiv preprint arXiv:2409.13082, 2024. Introduces VerusBench.DOI
  25. [26]Chenyuan Yang, Norrie Neamtu, Chris Hawblitzel, Jacob R. Lorch, and Shuvendu Lu. VeruSAGE: A study of agent-based verification for Rust systems. arXiv preprint arXiv:2512.18436, 2025. Introduces VeruSAGE-Bench.
  26. [27]Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. LeanDojo: Theorem proving with retrieval-augmented language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2306.15626
  27. [28]Sicheng Zhong, Jiaqi Zhu, Yifan Tian, and Xujie Si. RAG-Verus: Repository-level program verification with LLMs using retrieval augmented generation. arXiv preprint arXiv:2502.05344, 2025. Introduces RepoVBench.arxiv.org/abs/2502.05344

Paper details

vlnc:W_y3pnck63
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents