betaFeedback

WorksComputer Science

Beyond Correctness: Attribution of Natural-Language Documentation for Formal Libraries

Abstract

Formal languages are widely used in software engineering to specify and verify program properties. Because their mathematical notation and logic are hard to read, developers often rely on natural-language documentation to understand the functions in formal libraries. Yet a documentation statement can be provably true and still hold after the function's behavior changes. We call a statement's dependence on the function's specific behavior attribution, and we give a method to check it. We design and implement DocAuth, a documentation-generation method that uses attribution to decide what to publish and attaches to each published sentence the formal property recovered from it, together with checkable evidence. On 150 functions from the Community Books library of the ACL2 theorem prover, three large language models produce proved but non-attributable properties for about a third of the functions. DocAuth matches or exceeds the attribution rate of a mutation-based filter while delivering 37% more attributable sentences. Compared with a pipeline that checks truth alone, it raises the share of published sentences that a second model confirms as attributable from 82.9% to 94.7%. These results show that provable truth does not guarantee that documentation reflects what a function actually does. Attribution complements truth as a criterion for generating and evaluating documentation for formal libraries.

Introduction

Establishing that a program behaves as intended is a fundamental task in software engineering, especially for systems with stringent reliability requirements. Formal languages let developers state the properties a program should satisfy in precise mathematical notation and establish them by mechanized proof, and they have been applied to processors and systems software [17]. Theorem provers such as ACL2 [21], Lean [9], Coq [The Coq Development Team 2025], and Isabelle/HOL [26] have accumulated large libraries of formally defined functions and proved theorems. The notation, however, is hard to read: to work out what a function does, a reader often has to unfold nested definitions, trace recursion, and piece together behavior that several theorems characterize only jointly. Formal libraries therefore need natural-language documentation alongside their definitions and proofs, so that developers can understand and reuse their functions [8]. Now that large language models (LLMs) generate documentation and properties at scale, the generated content is usually checked for correctness: whether it can be proved, or whether it is consistent with the code [25, 30].

A documentation statement can, however, be provably true and still say nothing about the target function’s specific behavior. Figure 1 shows two examples. For implode from the ACL2 Community Books, the statement “Under ACL2’s len measure, the result of implode is no longer than the input” is backed by a proved theorem. But implode returns a string, and len returns 0 for strings, so every implementation that returns a string satisfies the statement. The statement that the Lean function canon maps equal inputs to equal outputs follows from equality substitution alone and holds for every function of the same type. Both statements paraphrase their theorems accurately, yet one conveys only a return type and the other only a general fact.

Proved properties that hold regardless of the target’s implementation

Figure 1. (a) ACL2: Restating the declared return type

(define implode ((x character-listp)) :returns (str stringp) ...) (defthm implode-no-longer (<= (len (implode x)) (len x)))

“Under ACL2’s len measure, the result of implode is no longer than the input.”

(b) Lean: An instance of congruence

def canon (s : String) : String := ... theorem canon_congr {a b : String} (h : a = b) : canon a = canon b := by rw [h]

“canon maps equal inputs to equal outputs.”

We call a statement’s dependence on the target function’s specific behavior attribution. To check it, we hold the function’s public interface (its input constraints and return type) and the background knowledge fixed and vary only the target’s behavior. A statement that holds for the target is attributable if it fails for at least one alternative implementation that satisfies the same interface; such an implementation, together with an input on which the statement fails, is a witness. A statement that follows from the interface and background knowledge alone is non-attributable: even when formally proved, it tells the reader nothing about the target’s behavior beyond the interface.

Attribution thus complements truth. Truth confirms that a statement holds for the target; attribution confirms that it describes behavior beyond the interface. Since callers can already see the interface, a statement about the target’s behavior should say more than the interface does, which makes attribution a basic requirement for such statements. It is one quality dimension, orthogonal and complementary to importance, completeness, and readability, which together determine whether documentation is useful to developers; we focus on attribution.

Attribution answers a question that no existing measure answers: does a statement tell the reader anything about behavior beyond the interface (Table 1)? It does so by measuring each statement against the function’s public interface. Every alternative implementation must satisfy this interface, so the alternatives range over precisely the behaviors a caller cannot rule out without reading the function body. Existing methods also test a conclusion by changing something, but none of them bounds the change by the interface. Mutation-based evaluation of specifications [1, 12] changes too little: it tries only a handful of edits to the function body, and these edits often leave the behavior a property describes untouched and need not satisfy the interface. Vacuity checking and semantic dependence [2, 28] change too much: they replace a part with an unconstrained value and thus break the declared interface well, so a statement that merely restates the return type would be “falsified” even though it adds nothing beyond the interface. Anchoring the check to the interface also backs both verdicts with checkable evidence. Non-attribution is established by a proof that covers every implementation satisfying the interface, which no finite set of mutants can provide; attribution is confirmed by a witness that itself satisfies the interface, which readers can check for themselves.

MeasureUnit of decisionReferenceQuestion answeredCan reject Figure 1 statements
Provability [25]Individual propertyOriginal targetCan the property be proved?✗
Mutation discrimination [1, 12, 16]Property or whole specificationSelected code mutantsWhich tested implementations does it rule out?◐
Vacuity and semantic dependence [2, 4, 28]Property or program partChanges to a subformula or program partDoes the conclusion depend on the changed part?✗
Document–code equivalence [11, 34]Entire document or specificationOverall behavior of the original targetDo the two describe the same behavior?✗
Source support and consistency [27, 29, 30]Sentence or commentSource text or codeIs the statement supported and consistent?✗
Attribution (ours)Individual statementImplementations under the fixed interfaceDoes it add behavioral information beyond the interface?✓

Table 1. Attribution compared with existing measures. ✓ guaranteed; ◐ depends on the selected variants; ✗ not guaranteed.

Turning attribution into a check and using it for documentation poses three challenges. First, non-attribution is a claim about every implementation that satisfies the interface, including implementations not definable in the logic. We reduce it to a single proof in an erased environment, where the target is replaced by a fresh symbol constrained only by its interface (Lemma 1). Second, a witness must itself satisfy the interface, so arbitrary edits do not qualify. DocAuth builds alternative implementations from fixed templates over the function’s arguments and has ACL2 prove that each one satisfies the interface. Third, documentation is prose, and rendering a property as a sentence can drop precisely the part that depended on the implementation. DocAuth therefore recovers a formal property from each rendered sentence, checks it again, and publishes only the sentences that pass, each with its recovered property and evidence.

On 150 functions from the ACL2 Community Books, three LLMs produce proved but non-attributable properties for about a third of the functions. Attributable properties also catch changes their check never saw: under interface-preserving code changes that differ from the witness templates, 64.8% of attributable properties detect a change, compared with 15.3% of the proved properties for which no witness is found and none of the non-attributable ones. At the sentence level, DocAuth raises the share of published sentences confirmed as attributable from 82.9% pipeline that checks truth alone, to 94.7%, and stays 9.3 points ahead when the reassessment model cannot see the code. It also delivers 37% more attributable sentences than a mutation-based filter. The two checks divide the work: rechecking the rendered sentences accounts for most of the rate gain, and selecting attributable candidates raises the number of attributable sentences.

This paper makes three contributions:

  • A definition of attribution and a sound check for it. Relative to the public interface, we separate what the interface already states from what an individual statement adds. An erasure proof establishes non-attribution for all conforming implementations, and an interface-conforming witness confirms attribution.

  • DocAuth, a documentation-generation method. It applies the check to candidate properties and to the properties recovered from rendered sentences, and publishes each sentence with its formal property and evidence.

  • An empirical evaluation. Non-attributable properties occur in about a third of the functions, and attributable properties detect interface-preserving changes that the others miss. Pooled over the three generator models, DocAuth delivers the most attributable sentences at the highest attribution rate among the methods compared.

Attribution

Consider str::base64-encode-list from the ACL2 Community Books, one of our evaluation targets, which encodes a character list in Base64. Among the candidate properties an LLM proposed for it are these two:

(character-listp (base64-encode-list a1))
(equal (base64-encode-list nil) nil)

ACL2 proves both, but only the second says something about this encoder. The first repeats the :returns declaration and holds for any implementation that returns a character list. Replacing the implementation exposes the difference: an implementation that always returns a one-character list still satisfies the interface, and the first property remains true while the second fails.

Formal Definition

Definition 1 (Attribution, non-attribution, and witnesses within a search scope). Let II be the public interface of the target ff, and let the background environment BB contain, in place of ff, only a placeholder symbol constrained by II, with no fact about that symbol other than II. Let C(I)C(I) be the set of all functions that can be substituted for the placeholder and satisfy II, including functions not definable in ACL2; ff satisfies II, so f∈C(I)f \in C(I). The property φ\varphi is relativized to the input constraints of II; VφV_{\varphi} is the set of assignments to its free variables over the domain of discourse; and [ ⁣[φ] ⁣]g∈{T,⊥}[\![\varphi]\!]_{g} \in\{\mathsf{T},\bot\} denotes the truth value of φ\varphi in BB when gg is substituted for the placeholder and vv for the free variables. If [ ⁣[φ] ⁣]f=T[\![\varphi]\!]_{f} = \mathsf{T} for every v∈Vφv \in V_{\varphi}, we write Valid⁡(f,φ)\operatorname{Valid}(f,\varphi), and we let

W(φ)={(g,v)∈C(I)×Vφ∣[ ⁣[φ] ⁣]g=⊥}.W(\varphi) = \{(g,v) \in C(I) \times V_{\varphi} \mid[\![\varphi]\!]_{g} = \bot\}.

Under Valid⁡(f,φ)\operatorname{Valid}(f,\varphi), we call φ\varphi attributable if W(φ)≠∅W(\varphi) \ne\varnothing and non-attributable if W(φ)=∅W(\varphi) = \varnothing. We call each (g,v)∈W(φ)(g,v) \in W(\varphi) a witness; by Valid⁡(f,φ)\operatorname{Valid}(f,\varphi), every witness has g≠fg \ne f. Within a finite search scope S⊆C(I)×VφS \subseteq C(I) \times V_{\varphi}, we confirm attribution if S∩W(φ)≠∅S \cap W(\varphi) \ne\varnothing; if S∩W(φ)=∅S \cap W(\varphi) = \varnothing and φ\varphi has not been established as non-attributable, we say that no witness is found for φ\varphi within that scope.

A target proof and a single witness suffice to confirm attribution, but a finite search that finds no witnesses cannot establish non-attribution. Our checker uses S=A(I)×XφS = A(I) \times X_{\varphi}, where A(I)⊆C(I)A(I) \subseteq C(I) and Xφ⊆VφX_{\varphi} \subseteq V_{\varphi} are finite sets generated by fixed rules.

Checking Method

The check is anchored to the interface II published in the source code: proofs about the original function and the witness search over alternative implementations both assume the input constraints of II (the guard).

The erased environment EfE_f implements the background environment BB of Definition 1. Rather than deleting the consequences of ff from the original environment, we build EfE_f by loading three parts in order: the events that precede the definition of ff in its source file, a set of standard background libraries, and finally a fresh symbol ufu_f. The only constraint on ufu_f is the return contract in II (the declared return type); the input constraints become premises of the property. Because ufu_f is introduced after every other event, no fact in EfE_f other than the return contract mentions it, and no lemma needs to be filtered out. During checking, occurrences of ff in a property are rewritten to ufu_f. If the property also uses other functions that call ff, their meaning still depends on ff, so the property is left undecided in EfE_f and recorded as unresolved.

Lemma 1 (Soundness of erasure verdicts). Suppose that no function in φ\varphi other than ff depends on ff. If φ[uf/f]\varphi[u_f/f] is provable in EfE_f, then W(φ)=∅W(\varphi)=\varnothing.

Proof sketch. Every event in EfE_f is a definition or has been proved by ACL2, so it holds in the standard model. Take any g∈C(I)g \in C(I): the only fact in EfE_f that mentions ufu_f is the return contract, which gg satisfies, so interpreting ufu_f as gg keeps every fact of EfE_f true. By soundness, φ[uf/f]\varphi[u_f/f] is true under this interpretation; by the premise, φ\varphi involves ff only at the rewritten positions, so ⟦φ[ν]⟧g=⊤\llbracket\varphi[\nu]\rrbracket_g=\top for every ν∈Vφ\nu\in V_\varphi. When gg is definable in ACL2, this argument coincides with ACL2’s functional instantiation rule [22]. □

Checking starts in EfE_f; properties not proved there move on to the original environment TT, which consists of the same background libraries plus the library containing ff. A property refuted by a counterexample in TT is recorded as false for the target and receives no attribution verdict. Every other property whose check completes receives exactly one of the following outcomes.

  • ❶ Attributable. φ\varphi is proved in TT, and A(I)×XφA(I) \times X_\varphi contains an alternative implementation gg satisfying the interface and an assignment ν\nu that together falsify φ\varphi. The target proof and the witness form the attribution evidence.

  • ❷ Non-attributable. φ\varphi remains provable in EfE_f; that is, II and the background already entail φ\varphi. By Lemma 1, W(φ)=∅W(\varphi)=\varnothing, and since f∈C(I)f \in C(I), Valid(f,φ)(f,\varphi) follows without a proof in TT.

  • ❸ No witness found. φ\varphi is not proved in EfE_f but is proved in TT, and A(I)×XφA(I) \times X_\varphi, searched in full, contains no witness.

If a dependency is unavailable, a proof or evaluation does not complete, or no sampled input satisfies the property’s premises, the outcome is unresolved. Figure 2 shows how the two properties above receive outcomes ❷ and ❶, respectively.

The erased environment and the attribution check for the running example

Figure 2. Erased environment. A fresh constrained symbol replaces the target str::base64-encode-list; the symbol carries only the target’s public interface :returns (x-enc character-listp).

(encapsulate (((u-56c775 *) => *)) (local (defun u-56c775 (a1) (declare (ignore a1)) nil)) (defthm u-56c775-contract (character-listp (u-56c775 a1)) :rule-classes :rewrite))

Outcomes. Both properties are rewritten over this symbol, with the input constraint as a premise. The first is still provable, because the :returns declaration entails it (non-attributable); the second is not, and a witness that satisfies the interface falsifies it (attributable).

(implies (character-listp a1) (character-listp (u-56c775 a1))) ; after erasure: proved (implies (character-listp nil) (equal (u-56c775 nil) nil)) ; after erasure: not proved (defun g1-21f79 (a1) '(#\a)) ; witness: satisfies the interface and falsifies the property

DocAuth

Attribution is decided on formal properties, but developers read natural-language sentences. DocAuth therefore selects documentation content at the formal level, renders it as sentences, and decides attribution again for what each sentence actually says (Figure 3). The workflow has three steps. An LLM proposes candidate properties φ\varphi for the target function, and the candidate check keeps only the attributable ones. Each kept property is rendered as a sentence ss, from which DocAuth recovers a formal property ψ\psi; ψ\psi is checked again, and ss is published only if ψ\psi is attributable. Both checks run Algorithm 1, with φ\varphi and ψ\psi, respectively, as the property of Definition 1.

Figure 3. Erase target semantics; Candidate Generation: Function, LLM, Candidate property; Normalize; Candidate Check: No erasure proof, Target proof, Conforming witness; Rendering & Recheck: Render, Back-translate, No erasure proof, Target proof, Conforming witness, Published sentence; Developer; Discarded; if all pass; if any fails. The DocAuth workflow

The two checks play different roles. The candidate check decides what to write, so that the limited sentence budget is spent on attributable content. The sentence recheck decides what publish, because readers read sentences, not properties. Both checks let through only content confirmed as attributable; nothing is published when the outcome is unresolved or no witness is found. Every published sentence comes with the property recovered from it and that property’s attribution evidence, which readers can check for themselves.

Candidate Generation

Candidate generation is an ordinary property-generation step. The model sees the formal source, including the complete function body, so that it can summarize implemented behavior worth documenting. The prompt asks for the shape of the result, relationships between the result and the inputs, boundary cases, and behavior under composition with standard functions; it mentions neither attribution nor erasure (Figure 4). All content filtering is left to the later checks, so the generator can be swapped for another model or property-generation method. Identifiers defined in the corpus are replaced with opaque names so that the model describes behavior from the definitions rather than from names; ACL2 built-in vocabulary is kept. The model is queried once per function for as many candidates as the candidate budget specifies. Responses are normalized to single ACL2 terms (stripping wrappers such as defthm, removing duplicates, and restoring the original names) and stored in the order the model returns them.

Prompt for candidate-property generation

Figure 4. Prompt for candidate-property generation

Implementing the Check

Before generating candidates, DocAuth extracts the target’s explicit guard and any separately declared return contract from the source as the interface II and freezes it. The reference is thus the interface that the library’s authors have already published, not one chosen by DocAuth or the model. The erased environment EfE_f and the original environment TT are built as in § 2.2, and all checks for a given target share the same interface and environments. Checked properties are not registered as proof rules, so checks of different properties do not affect one another. Algorithm 1 gives the complete procedure, where [g/f][g/f] replaces the target ff with gg and [ν][\nu] substitutes the concrete inputs ν\nu.

Algorithm 1 Determining whether a property is attributable

Require: Original environment TT, target ff, property φ\varphi, fixed alternative templates and input-generation rules

Ensure: Attribution evidence (target proof and witness), or a reason to withhold publication

  1. I←I \gets public interface of ff

  2. Ef←E_f \gets source events preceding the target definition and standard background libraries

  3. In EfE_f, replace ff with a fresh symbol ufu_f constrained only by II ▹ Lemma 1

  4. φ^←φ\hat{\varphi} \gets\varphi with the interface input constraints added ▹ Relativized form in Definition 1

  5. if φ^[uf/f]\hat{\varphi}[u_f/f] is proved in EfE_f then

  6. &nbsp;&nbsp;return ❷ Non-attributable ▹ Entailed by the interface and background

  7. end if

  8. c←c \gets proof of φ^\hat{\varphi} in TT; if no proof is obtained, return false for the target or unresolved

  9. Construct alternative implementations A(I)A(I) and input assignments XφX_\varphi using the fixed rules

  10. for all g∈A(I)g \in A(I) such that ACL2 accepts gg and proves that it satisfies II do

  11. &nbsp;&nbsp;if some ν\nu in XφX_\varphi satisfies the premises and makes φ^[g/f][ν]=⊥\hat{\varphi}[g/f][\nu] = \bot then

  12. &nbsp;&nbsp;&nbsp;&nbsp;return (c,(g,ν))(c,(g,\nu)) ▹ True for the original target, false for an alternative

  13. &nbsp;&nbsp;end if

  14. end for

  15. if evaluations are incomplete or no sampled input satisfies the premises then

  16. &nbsp;&nbsp;return unresolved

  17. end if

  18. return ❸ No witness found ▹ Search within scope completed without a witness

The alternative implementation in a witness must satisfy the interface; otherwise it would falsify even a property that merely restates the return type. DocAuth builds alternative implementations from the interface rather than from the function body, using two kinds of templates over the function’s arguments: one returns a fixed constant, and the other builds the result from the arguments by returning an argument, transforming one, or swapping argument positions. The templates depend only on the function’s signature, not on its body, and the witnesses they yield are short enough for readers to inspect directly, as with the constant implementation in Figure 2. A template instance enters the search only after ACL2 admits it and proves that it satisfies II. The system takes the first 16 instances in a fixed order and keeps those that pass as A(I)A(I). The assignments XφX_{\varphi} are generated from variable types and property premises, and the search stops at the first witness. The full templates, search order, and input-generation rules are in the replication package.

Rendering and Rechecking

The candidate check confirms only φ\varphi, but what is published is the sentence rendered from φ\varphi. Rendering can change a property’s meaning or scope, and the sentence need not preserve the implementation-dependent part of φ\varphi, so DocAuth checks the sentence itself. Properties that pass the candidate check are taken in stored order, up to the sentence budget, and each is rendered as one sentence ss. A separate model call then back-translates ss into a formal property ψ\psi, given ss and the target’s definition; because this call never sees φ\varphi, ψ\psi reflects only what the sentence says. The recovered property ψ\psi then goes through the full check of Algorithm 1: if it is attributable, the sentence is published with ψ\psi and its evidence; otherwise the sentence is discarded. The recheck does not require ψ\psi to equal φ\varphi: readers rely on the sentence, and the evidence delivered with it concerns ψ\psi, the property the sentence expresses. The rendering and back-translation prompts are in the replication package.

For the encoder of § 2, the property (equal (base64-encode-list nil) nil) can be rendered as “Encoding an empty character list yields an empty character list.” When the same property is recovered from this sentence, the recheck again obtains a target proof and the one-character-list witness, and DocAuth delivers the sentence with the recovered property and its evidence. If instead the rendering says only “The encoded result is a character list,” the recovered property merely restates the return type and remains provable in the erased environment, so the recheck classifies it as non-attributable and the sentence is not published.

Evaluation

Experimental Setup

From the 2,054 functions in 11 subcorpora of the ACL2 Community Books, we select 162 functions that have public, fully extractable definitions, take at least one argument, do not access state, support safe renaming and erasure checking, and have at least three proved library theorems. Before generation, we order them pseudo-randomly by a hash of their names and take the first 150 as targets.

Properties are generated by qwen2.5-coder:7b, deepseek-v4-flash, and gpt-5.6-terra, hereafter Qwen2.5-7B, DeepSeek-v4-flash, and GPT-5.6-terra. All three models use the same targets, prompts, and checking configuration, run at temperature 0, and are queried once per function for k=15k = 15 properties. The human-written group consists of the proved theorems that library authors wrote for the same functions, checked under the same configuration.

In the sentence experiment, the three models serve only as sentence writers, and all candidate properties come from the GPT-5.6-terra pool. Each publication path writes at most 4 sentences per function, and a function with nothing published counts as zero. Back-translation during publication and LLM judging both use GPT-5.6-terra.

Properties were generated on 25–27 August 2026 (UTC); the sentence experiment and timing measurements ran on 10–25 September. Qwen2.5-7B ran locally under Ollama (digest dae161e27b0e); the other models were accessed by model name. On 10 September, DeepSeek replaced V4-Flash with V4.1-Flash, so its properties come from V4-Flash and its sentences from V4.1-Flash, both labelled DeepSeek-v4-flash.

Our main measure is the attribution rate, the share of published sentences that a cross-model reassessment confirms as attributable. Throughout, counts of attributable sentences refer to sentences confirmed by this reassessment. The reassessment uses claude-sonnet-5, which plays no part in publication: it recovers formal properties afresh from the opaque definitions and the published sentences and checks them with Algorithm 1; failed back-translations and unresolved checks count as not confirmed. All 95% intervals are percentile bootstrap intervals [10] over 10,000 resamples of target functions, which keep each function’s observations together [3]. Differences between rates are computed from the rates as printed, to one decimal place, whereas intervals are computed from unrounded data.

RQ1, RQ2, and RQ5 analyze the properties from the four sources, RQ3 analyzes the sentence experiment, and RQ4 reuses truncated records from both. The five research questions are:

  • RQ1 (Prevalence): How Common Are Non-Attributable Properties?

  • RQ2 (Criterion Validity): Do Attributable Properties Detect Interface-Preserving Implementation Changes?

  • RQ3 (Method Effectiveness): Does DocAuth Improve the Attribution of Generated Documentation?

  • RQ4 (Parameter Effects): How Do Budget Settings Affect the Yield of Attributable Content?

  • RQ5 (Coverage and Cost): How Many Functions Are Covered, and at What Cost?

RQ1 (Prevalence): How Common Are Non-Attributable Properties?

All three generator models produce non-attributable properties, and these properties occur in about a third of the functions: among functions with at least one completed check, 35.0%, 32.3%, and 29.9% have at least one non-attributable property for Qwen2.5-7B, DeepSeek-v4-flash, and GPT-5.6-terra, respectively. Table 2 gives the check outcomes for each source. Percentages are relative to properties whose check completed; unresolved checks are analyzed in RQ5.

Qwen2.5-7BDeepSeek-v4-flashGPT-5.6-terraHuman-written
Properties checked1,7432,0302,0901,068
Checks completed9391,2151,371466
❶ Attributable461 (49.1%)921 (75.8%)1,209 (88.2%)409 (87.8%)
False for the target373 (39.7%)143 (11.8%)27 (2.0%)0 (0.0%)
❷ Non-attributable80 (8.5%)77 (6.3%)72 (5.3%)20 (4.3%)
❸ No witness found25 (2.7%)74 (6.1%)63 (4.6%)37 (7.9%)

Table 2. Outcomes of the properties checked on 150 functions

Stronger models mainly reduce errors, not non-attribution: from Qwen2.5-7B to GPT-5.6-terra, the share of properties false for the target falls from 39.7% to 2.0%, whereas the share of non-attributable ones falls only from 8.5% to 5.3%. Identifying non-attributable properties therefore takes a check that goes beyond truth.

Of the proved theorems written by library authors, 87.8% are attributable, on par with GPT-5.6-terra. Nearly all of the 20 non-attributable ones are lemmas that serve the prover, such as (equal (perm (true-list-fix x) y) (perm x y)), rather than documentation for readers.

RQ2 (Criterion Validity): Do Attributable Properties Detect Interface-Preserving Implementation Changes?

Most attributable properties (64.8%) detect interface-preserving implementation changes that played no role in their check, compared with only 15.3% of the properties that also hold for the target but for which no witness was found. Non-attributable properties detect none (0/48). This follows from what non-attribution means: such properties are implied by the interface alone, hold for every implementation that preserves it, and carry no information about the implementation. The detection rate of attributable properties, by contrast, is not fixed by the definition.

We apply single-point edits to the function definitions using common mutation-testing operators [18]: accessor or comparison replacement, argument swaps, constant changes, and conjunct deletion. These edits differ from the witness templates (§ 3.2). We keep only edits that ACL2 accepts, that change behavior on sampled inputs, and that satisfy the declared return contract; 37 of the 67 functions with parseable definitions have at least one usable edit. A property detects an edit if it evaluates to false on some sampled input, and rates are computed over properties with at least one usable edit.

Pooled over the four groups, 614 of 948 attributable properties detect at least one change, with per-group rates ranging from 55.3% to 67.1% (Table 3). The pooled rate is 49.5 percentage points above that of no-witness properties (95% CI [+31.9, +63.8]), and the gap is still 47.9 points on the 18 functions that have properties of both kinds (63.7% versus 15.8%, 95% CI [+25.5, +66.7]). The 9 no-witness properties that do detect a change are in fact attributable: the edits that detect them are witnesses outside the search scope. Filtering by attribution sacrifices almost no detection power: of the 507 property–change failures among GPT-5.6-terra’s proved properties, 503 (99.2%) involve attributable properties.

Property sourceAttributableNo witness foundNon-attributable
Qwen2.5-7B68/123 (55.3 %)1/6 (16.7 %)0/16
DeepSeek-v4-flash204/304 (67.1 %)4/21 (19.0 %)0/14
GPT-5.6-terra264/401 (65.8 %)4/19 (21.1 %)0/10
Human-written78/120 (65.0 %)0/13 (0.0 %)0/8
Total614/948 (64.8 %)9/59 (15.3 %)0/48

Table 3. Change detection by property source and check outcome

RQ3 (Method Effectiveness): Does DocAuth Improve the Attribution of Generated Documentation?

Pooled over the three generator models, DocAuth achieves the highest attribution rate of the six methods under all three reassessment conditions and delivers the most attributable sentences. Its edge over truth checking comes mainly from the sentence recheck, while its edge over mutation filtering lies mainly in yield.

We compare six methods on the same functions and generator models, built around three main paths. The baseline writes sentences directly from the opaque source code without any check (Figure 5). Truth checking renders properties proved to hold for the target, and DocAuth renders attributable properties. These two back-translate every sentence in the same way and differ only in the criterion used for candidate selection and sentence rechecking.

Three variants complete the comparison. LLM judging is added after the baseline or truth checking: a judge reads the definition and the sentence and decides whether the sentence describes behavior specific to the target. Mutation filtering follows nl2postcond [12] and MutDafny [1]: at both candidate selection and sentence rechecking in the truth-checking path, it keeps only properties falsified by at least one mutant. The mutants are all single-point edits of the function body under the RQ2 operators and, like the witness templates, must be proved by ACL2 to satisfy the return contract. Mutation filtering and DocAuth thus differ mainly in whether alternative implementations come from the function body or from the interface.

Image not extracted

Figure 5. Prompt for direct documentation generation in the baseline

Baseline (formal source → sentence, without intermediate properties). You are documenting a function in an ACL2 library so that other developers can understand what it does without reading its implementation. Write short, factual English sentences describing its behavior. Each sentence should state one point about the function’s inputs and result: the shape of the result, the relationship between the result and inputs, or what happens in boundary cases. Describe what it computes, not how it is implemented; do not name other library functions or describe internal steps or helper routines. Output only a numbered list with one sentence per line.

Overall effect. DocAuth has the highest cross-model attribution rate of the six methods for every generator model (Table 4, Figure 6). Pooled over the three models, it reaches 94.7%, which is 11.8 percentage points above truth checking (95% CI [+8.5, +15.3]) and 45.0 points above the baseline. It publishes fewer sentences than truth checking (1,238 versus 1,301) yet delivers 94 more attributable ones (1,172 versus 1,078, 95% CI [+55, +136]). LLM judging helps little: it raises the baseline by 4.2 points (95% CI [+2.9, +5.5]) and truth checking by a non-significant 1.6 points (95% CI [−0.1, +3.8]).

Sentence attribution rates, pooled over the three models

Figure 6. Sentence attribution rates, pooled over the three models

Generator modelMethodPublishedAttributableCross-modelInterface-onlyInterface-only, equivalent to ψ\psi
Qwen2.5-7BBaseline57524743.0%26.1%20.2%
Qwen2.5-7B+ LLM judge40421753.7%34.2%26.5%
Qwen2.5-7BTruth check37629979.5%62.5%43.9%
Qwen2.5-7B+ LLM judge35929080.8%63.5%44.8%
Qwen2.5-7B+ mutation filter26522986.4%64.5%44.2%
Qwen2.5-7BDocAuth35931788.3%68.0%46.8%
DeepSeek-v4-flashBaseline57730352.5%37.1%28.9%
DeepSeek-v4-flash+ LLM judge45425355.7%38.3%29.1%
DeepSeek-v4-flashTruth check45838483.8%67.5%48.3%
DeepSeek-v4-flash+ LLM judge38232885.9%66.8%44.8%
DeepSeek-v4-flash+ mutation filter32030394.7%75.0%54.4%
DeepSeek-v4-flashDocAuth43241796.5%77.3%52.8%
GPT-5.6-terraBaseline57230753.7%39.2%29.2%
GPT-5.6-terra+ LLM judge51126852.4%36.6%25.6%
GPT-5.6-terraTruth check46739584.6%68.5%46.3%
GPT-5.6-terra+ LLM judge37232386.8%67.5%40.9%
GPT-5.6-terra+ mutation filter34232193.9%75.1%48.8%
GPT-5.6-terraDocAuth44743898.0%80.3%57.3%

Table 4. Attributable sentences and attribution rates by generator model and reassessment condition

Roles of the two checks. The sentence recheck raises the attribution rate, whereas the candidate check raises the yield. Switching only the recheck criterion of the truth-checking path to attribution lifts its rate from 82.9% to 93.9% (+11.0 points, 95% CI [+7.9, +14.4]); 7.4 points of this gain [+4.5, +10.6]) come from removing non-attributable and no-witness sentences. The same recheck also lifts the baseline from 49.7% to 86.7%. Under the same recheck, DocAuth and this truth-checking path have similar rates (94.7% versus 93.9%, 95% CI [−0.8, +2.3]), but DocAuth delivers 103 more attributable sentences (95% CI [+63, +145]): sentences written from attributable properties pass the recheck more often (87.7%, versus 78.4% for truth checking and 50.1% for the baseline).

Comparison with mutation filtering. Mutation filtering comes close to the attribution rate of DocAuth but delivers far fewer attributable sentences. Its rate is 92.0%, 9.1 points above truth checking (95% CI [+5.8, +12.7]), and DocAuth is another 2.7 points higher (95% CI [+0.6, +4.8]). Mutation filtering, however, publishes only 927 sentences, 853 of them attributable, so DocAuth delivers 319 more (95% CI [+236, +404]). The gap arises in candidate selection. The mutants falsify no non-attributable property, but they also falsify only 60.4% of the attributable ones, so mutation filtering selects 374 properties (DocAuth: 481) and none at all for 52 functions (DocAuth: 27). A single-point edit often leaves the behavior a property describes untouched, whereas alternative implementations built from the interface are not limited by the function body.

Per-function delivery. Each of the three generator models documents each function once, giving 450 documents. DocAuth delivers at least one attributable sentence in 80.9% of them, on par with the baseline’s 83.3% (95% CI of the difference [−7.1, +2.0]). It delivers four attributable sentences in 40.9% of documents, versus 8.0% for the baseline, and 2.60 per document on average, compared with 2.40 for truth checking and 1.90 for the baseline (Table 5). The 18.7% of documents in which DocAuth publishes nothing stem mainly from the 27 functions whose property pool contains no attributable property.

MethodNothing published≥1\ge1 attributable≥2\ge2 attributable4 attributableAttributable per documentAttribution rate
Baseline0.7%83.3%65.8%8.0%1.9049.7%
Truth check16.9%79.3%72.0%30.9%2.4082.9%
+ mutation filter36.4%62.4%55.6%26.4%1.9092.0%
DocAuth18.7%80.9%74.7%40.9%2.6094.7%

Table 5. Per-function delivery

Composition of published content. The attribution criterion tilts published content only slightly toward point and boundary facts, and this shift explains almost none of the rate gain. We classify each published sentence, by the syntax of the ψ\psi delivered with it, as a point fact, a boundary case (an argument takes a base value, or a premise restricts it to its base case), a type or shape fact, or a general relation (rules in the replication package). Point and boundary facts make up 38.4% of DocAuth’s sentences and 35.1% of truth checking’s (95% CI of the difference [+1.3, +5.4]). Both exceed the 20.0% in the property pool, because both paths take properties from the front of the list, where models tend to write base cases. The 3.3-point difference comes mainly from incomplete checking: among properties proved true of the target, 15.0% of general relations remain unresolved or have no witness found, versus 1.7% of point and boundary facts; this is the same limitation that bounds coverage in RQ5. A Kitagawa decomposition (no interaction residue) assigns only 0.4 of DocAuth’s 11.8-point lead over truth checking to the class mix, and on general relations alone DocAuth is still 16.3 points higher (90.1% versus 73.8%, 95% CI [+11.1, +21.5]). The classes do not measure importance, but attributable boundary facts are not trivial either: in RQ2 they detect changes about as often as general relations do (60.9% versus 68.9%).

Reassessment without code. The cross-model reassessment reads the function body and may use it to fill in what a sentence leaves unsaid. We therefore reassess all published sentences under two stricter conditions. In the first, the reassessment model sees only the interface (function name, formal parameters, input constraint, and return contract); the second additionally requires the recovered property to be proved equivalent in ACL2 to the ψ\psi delivered with the sentence. Pooled over the three models, DocAuth remains the highest of the six methods (75.7% and 52.7%), 9.3 and 6.4 points above truth checking (95% CIs [+6.0, +12.7] and [+3.4, +9.6]). Its rate differences from mutation filtering are not significant (+3.6 and +3.3 points, 95% CIs [-0.1, +7.3] and [-1.2, +7.6]), but DocAuth delivers 269 and 194 more attributable sentences (95% CIs [+198, +341] and [+130, +258]). Without the code, 20.5% of the sentences originally confirmed for DocAuth and 21.2% of those for truth checking are no longer confirmed. The losses are similar, so DocAuth’s advantage does not depend on the reassessment model reading the code.

Sensitivity and stability of the recheck. The sentence recheck is sensitive to changes in a sentence’s meaning, and its results are stable. A model that sees only the sentence rewrote 230 confirmed-attributable sentences, stratified across the three paths. When a rewrite changes one result, premise, or constant, 70.0% of the rewritten sentences fail the recheck and 45.2% yield a property that is false for the target; when it changes only the wording, 86.5% still pass. Of 337 sentences reassessed twice, 94.1% (DocAuth) and 91.3% (truth checking) yield equivalent properties in both runs.

RQ4 (Parameter Effects): How Do Budget Settings Affect the Yield of Attributable Content?

Larger candidate budgets and search scopes yield more attributable content, with diminishing returns, and writing more sentences per function barely lowers the attribution rate. Instead of regenerating, we truncate the saved records to vary the number of candidates per function kk (Figure 7a), the witness-search scope (Figure 7b), and the number of sentences per function mm (Figure 7c,Figure 7d).

Four-panel line chart showing attributable share, properties still without a witness, attribution rate, and attributable sentences as functions of candidate budget, search scope, and sentences per function

Figure 7. Effects of the candidate budget, the search scope, and the number of sentences per function

Candidate budget. When each function’s candidates are truncated to the first k∈{3,5,10,15}k \in\{3, 5, 10, 15\}, the number of attributable properties grows with kk while their share falls (Figure 7a). For GPT-5.6-terra, the count rises from 311 to 1,209 and the share drops from 73.9% to 57.8%. The other three groups follow the same trend; candidates later in the list more often turn out false for the target or unresolved. The change-detection rate of attributable properties stays between 55% and 70% at every kk. The number of functions with enough attributable properties for four sentences saturates quickly: for GPT-5.6-terra, it is 113 at k=10k = 10, and k=15k = 15 adds only 5 more.

Search scope. The witness search tries alternative implementations in a fixed order and stops at the first witness. The number of properties still without a witness drops most sharply at the second alternative, from 690 to 324 for GPT-5.6-terra; after the 16th, 63 remain, which is the no-witness count in Table 2 (Figure 7b). The curves are still falling at that point, so a larger scope would find witnesses for some of these properties too.

Sentences per function. When each function’s published sentences are truncated to the first m∈{1,2,3,4}m \in\{1, 2, 3, 4\}, DocAuth’s attribution rate falls only from 96.7% to 94.7%, while its attributable sentences grow from 354 to 1,172. At every mm, its rate is 9.4–11.8 points above that of truth checking (Figure 7c,Figure 7d).

RQ5 (Coverage and Cost): How Many Functions Are Covered, and at What Cost?

The four property sources cover 70.0%–84.7% of the functions, and coverage is limited mainly by unresolved checks rather than by the witness search. Running DocAuth end to end takes a median of 219.8 seconds per function, of which only 14.1 seconds are spent in the prover, and costs about $0.048 in model calls.

Function coverage. A function counts as covered if it obtains at least one attributable property, and the denominator is all 150 targets (Table 6). Coverage depends on how attributable properties are distributed across functions, not only on their number: GPT-5.6-terra obtains more attributable properties than DeepSeek-v4-flash (1,209 versus 921) but covers fewer functions (123 versus 127).

Property sourceProperties checkedFunctions coveredCoverage
Qwen2.5-7B1,743114/15076.0%
DeepSeek-v4-flash2,030127/15084.7%
GPT-5.6-terra2,090123/15082.0%
Human-written1,068105/15070.0%

Table 6. Function coverage by property source

What limits coverage. Unresolved outcomes account for 34.4%–56.4% of all checked properties, far more than the 1.4%–3.6% with no witness found (Figure 8a). For generated properties, most unresolved outcomes stem from missing target proofs or evaluation failures (Figure 8b). Human-written theorems have the highest unresolved share, and 43.0% of their unresolved outcomes are due to missing dependencies or erasure-environment problems: these theorems often involve the target indirectly, through other library functions that call it, and such properties are left undecided in the erased environment (§ 2.2).

Check outcomes and reasons for unresolved outcomes by property source

Figure 8. Check outcomes and reasons for unresolved outcomes by property source

Runtime cost. We time an end-to-end run of DocAuth from fresh candidates with GPT-5.6-terra on all 150 functions (15 candidates per function, a single process on an Intel i7-14700HX). The model service’s content filter rejects one function’s definition; the remaining 149 complete the measurement (Figure 9). The median total time per function is 219.8 seconds (p10p_{10}–p90p_{90}: 91.8–324.1), of which 197.5 seconds are spent on model calls and 14.1 seconds in the prover. The median input and output token counts are 2,054 and 3,551. At the list price of GPT-5.6-terra ($2 and $12 per million input and output tokens), the median model-call cost is about $0.048 per function, or about $7.3 in total for the 149 functions. The 27 functions with no published sentence stop after candidate checking, with a median total time of only 90.1 seconds. Timings depend on API response speed, and measurements taken at different times can vary considerably.

Figure 9. Distribution of token usage and total time per function

Discussion

Attribution fills a gap in how documentation is evaluated. In software engineering, a test is judged not only by whether it passes but also by whether it fails on faulty implementations [18]. Generated documentation and specifications, by contrast, are mostly judged by the first criterion alone: whether they can be proved or are consistent with the code [25, 30]. For statements that describe a function’s behavior, attribution imposes the second criterion: a true statement should also fail for at least one implementation that satisfies the same interface. Since stronger models mainly reduce errors rather than non-attribution (RQ1), and LLM judging helps little (RQ3), evaluations of documentation- and specification-generation methods can report the attribution rate alongside correctness measures.

Unlike scores based on textual similarity to reference comments or on human or LLM ratings, every attribution verdict comes with checkable evidence: a target proof and a witness confirm attribution, and an erasure proof establishes non-attribution; for a sentence, the verdict concerns the property ψ\psi recovered from it. Because the measure is relative to the library’s published interface, a stronger interface leaves fewer statements attributable, so attribution rates of different methods should be compared under the same interface. The measure also applies to existing documentation: when it is drawn from library theorems, lemmas written for the prover can be set aside first (RQ1).

We see three directions for future work. First, attribution can be extended to documentation comments in mainstream languages. There, executable alternative implementations that satisfy a signature or contract can supply witnesses, whereas proving that a statement holds for the target and establishing non-attribution still require a verifier. Second, attribution can be combined with measures of importance, completeness, and readability into an overall assessment of documentation quality, and developer studies can test how this assessment relates to the usefulness of documentation. Third, studies of real code evolution can test whether attributable statements also serve as regression checks; RQ2 already provides preliminary evidence for single-point edits.

Threats to Validity

Construct validity and developer feedback. The attribution rate measures whether statements meet the basic requirement of describing behavior beyond the interface. Whether documentation is useful to developers also depends on importance, completeness, and readability, which need their own measures. We did not conduct a developer study, so it remains untested whether attributable documentation helps developers understand and maintain code, and how much effort it takes to inspect the attached evidence.

Measurement. The formal meaning of each sentence is recovered by an LLM, and the reassessment uses the same checker as publication. The interface-only reassessment and the meaning-change experiment in RQ3 reduce the risk that recovery relies on the code, but they do not replace human judgment of what a sentence means.

Experimental scope. The experiments cover stateless functions with extractable definitions from the ACL2 Community Books and three LLMs, each run once at temperature 0; other prompts, sampling settings, or model versions may yield different candidate properties. Programs with mutable state, external interaction, or concurrency place different demands on interface modeling and alternative implementations. Other provers need their own interface extraction and erasure environment; there, the natural counterpart is a bodyless function constrained by its declared specification.

Related Work

Can specifications detect incorrect implementations? nl2postcond [12] generates postconditions from natural-language requirements and checks both whether they are correct and whether they distinguish correct from incorrect code. MutDafny [1] mutates code to find specifications too weak to notice the change, and IronSpec [16] tests whether specifications match developer intent. Like these methods, we change the implementation and observe whether a property still holds. Unlike them, we measure against the public interface, judge individual statements already known to be true, and can prove non-attribution by erasure instead of merely failing to find a distinguishing implementation. Vacuity checking and semantic dependence likewise change one part of an artifact to decide whether a conclusion depends on it [2, 4, 28]; we instead change the whole function’s behavior while keeping its published interface.

Does documentation describe code accurately? LLMs are already used to generate code comments and summaries [15, 31], which are mostly evaluated by textual similarity to reference comments or by the correctness of their content, that is, by whether the documentation says anything wrong. Panthapakkal et al. [2021] detect comments that become outdated as code changes, Clover [30] checks consistency among code, documentation, and formal annotations, and CASCADE [24] generates tests from documentation and runs them on the code to find documentation that contradicts it. Attribution in natural language generation is a different notion: it requires generated text to be supported by an identifiable source [29]. Truth, source support, and consistency with the code do not rule out the sentences in Figure 1, which are consistent with the code and well supported yet only restate the interface or a general fact; this is precisely what our notion of attribution checks. Documentary [11] and VeriEquivBench [34] check whether an entire document or specification is behaviorally equivalent to the code. Their unit of decision is overall behavior, whereas we examine the behavioral information that a single statement provides. Khatib et al. [2026] change the code, regenerate summaries, and observe whether the summaries change, which evaluates the summarization model itself; we do not regenerate, but check whether each written sentence still holds for alternative implementations.

Property generation and documentation maintenance. Tools such as QuickSpec [6], Daikon [14], and SpecGen [25] automatically discover or generate function properties, and some of them prune redundant results. QuickSpec does not report equations that follow from ones it has already found; IsaCoSy [19], HipSpec [5], and Hipster [20] discard conjectures that follow from known lemmas during theory exploration; and Daikon suppresses invariants implied by other invariants [13]. Such pruning is relative to the other properties already obtained, and a pruned property may still describe implementation-specific behavior. Attribution, by contrast, is relative to the library’s published interface. It judges each true statement on its own, asking whether it says anything about behavior beyond the interface, and it confirms attribution with a witness and establishes non-attribution with an erasure proof, providing a basis for selecting documentation content. DocAuth’s recheck recovers formal properties from natural language and is therefore related to autoformalization [7, 33]; rather than aiming for a faithful sentence-by-sentence translation, however, we check whether the recovered property is still attributable. XDoc [8] organizes and publishes ACL2 library documentation, which complements our content selection.

Conclusion

A provably true documentation statement may still do no more than restate the interface or a general fact. This paper turns that distinction into a checkable condition: relative to the public interface, a statement that holds for the target is attributable exactly when it fails for some alternative implementation that satisfies the same interface, and an erasure proof establishes when no such implementation exists. Non-attributable properties occur in about a third of the functions we studied, whereas most attributable properties detect interface-preserving implementation changes. By adding attribution on top of truth checking, DocAuth delivers more attributable sentences than both truth checking and mutation filtering, at a higher attribution rate, and publishes each sentence with the formal property recovered from it and that property’s evidence. Just as a test should not only pass but also fail on faulty implementations, a documentation statement that describes a function’s behavior should not only be true but also say more than the interface does. Attribution checks this basic requirement. It complements correctness, can guide documentation generation, and can serve as a measure for evaluating documentation and specifications.

Data Availability

The replication package for this paper will be deposited in a permanent archive. It contains source-code snapshots, experimental inputs and outputs, verification evidence, prompts and model configurations, and scripts for offline recomputation, and it retains the records of selection, exclusion, and unresolved cases. The package also describes the execution environment and the configuration of external services.

References

  1. [1]Isabel Amaral, Alexandra Mendes, and José Campos. 2026. MutDafny: A Mutation-Based Approach to Assess Dafny Specifications. arXiv:2511.15403 [cs.SE] https://arxiv.org/abs/2511.15403 ICSE 2026 Research Track.
  2. [2]Ilan Beer, Shoham Ben-David, Cindy Eisner, and Yoav Rodeh. 2001. Efficient Detection of Vacuity in Temporal Model Checking. Formal Methods in System Design 18, 2 (2001), 141–163. doi:10.1023/A:1008779610539DOI
  3. [3]A. Colin Cameron, Jonah B. Gelbach, and Douglas L. Miller. 2008. Bootstrap-Based Improvements for Inference with Clustered Errors. The Review of Economics and Statistics 90, 3 (2008), 414–427. doi:10.1162/rest.90.3.414DOI
  4. [4]Hana Chockler, Joseph Y. Halpern, and Orna Kupferman. 2008. What Causes a System to Satisfy a Specification? ACM Transactions on Computational Logic 9, 3 (2008), 20:1–20:26. doi:10.1145/1352582.1352588DOI
  5. [5]Koen Claessen, Moa Johansson, Dan Rosén, and Nicholas Smallbone. 2013. Automating Inductive Proofs Using Theory Exploration. In Automated Deduction – CADE-24, 24th International Conference on Automated Deduction (Lecture Notes in Computer Science, Vol. 7988). Springer, Lake Placid, NY, USA, 392–406. doi:10.1007/978-3-642-38574-2_27DOI
  6. [6]Koen Claessen, Nicholas Smallbone, and John Hughes. 2010. QuickSpec: Guessing Formal Specifications Using Testing. In Tests and Proofs, 4th International Conference, TAP 2010 (Lecture Notes in Computer Science, Vol. 6143). Springer, Málaga, Spain, 6–21. doi:10.1007/978-3-642-13977-2_3DOI
  7. [7]Matthias Cosler, Christopher Hahn, Daniel Mendoza, Frederik Schmitt, and Caroline Trippel. 2023. nl2spec: Interactively Translating Structured Natural Language to Temporal Logics with Large Language Models. In Computer Aided Verification (CAV 2023) (Lecture Notes in Computer Science, Vol. 13965). Springer. doi:10.1007/978-3-031-37703-7_18DOI
  8. [8]Jared Davis and Matt Kaufmann. 2014. Industrial-Strength Documentation for ACL2. In Proceedings Twelfth International Workshop on the ACL2 Theorem Prover and Its Applications (Electronic Proceedings in Theoretical Computer Science, Vol. 152). Open Publishing Association, Vienna, Austria, 9–25. doi:10.4204/EPTCS.152.2DOI
  9. [9]Leonardo de Moura, Soonho Kong, Jeremy Avigad, Floris van Doorn, and Jakob von Raumer. 2015. The Lean Theorem Prover (System Description). In International Conference on Automated Deduction (CADE). Springer, Berlin, Heidelberg, 378–388. doi:10.1007/978-3-319-21401-6_26DOI
  10. [10]Bradley Efron and Robert J. Tibshirani. 1993. An Introduction to the Bootstrap. Number 57 in Monographs on Statistics and Applied Probability. Chapman & Hall, New York.
  11. [11]Aryaz Eghbali, Zhongxin Liu, and Michael Pradel. 2026. Natural Language-Focused Software Engineering via Code-Documentation Equivalence. Proceedings of the ACM on Software Engineering 3, FSE, Article FSE204 (2026), 22 pages. doi:10.1145/3808211DOI
  12. [12]Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions? Proceedings of the ACM on Software Engineering 1, FSE (2024), 1889–1912. doi:10.1145/3660791DOI
  13. [13]Michael D. Ernst, Adam Czeisler, William G. Griswold, and David Notkin. 2000. Quickly Detecting Relevant Program Invariants. In Proceedings of the 22nd International Conference on Software Engineering (ICSE ’00). ACM, Limerick, Ireland, 449–458. doi:10.1145/337180.337240
  14. [14]Michael D. Ernst, Jeff H. Perkins, Philip J. Guo, Stephen McCamant, Carlos Pacheco, Matthew S. Tschantz, and Chen Xiao. 2007. The Daikon System for Dynamic Detection of Likely Invariants. Science of Computer Programming 69, 1–3 (2007), 35–45. doi:10.1016/j.scico.2007.01.015DOI
  15. [15]Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. 2024. Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE 2024). ACM. doi:10.1145/3597503.3608134DOI
  16. [16]Eli Goldweber, Weixin Yu, Seyed Armin Vakil Ghahani, and Manos Kapritsos. 2024. IronSpec: Increasing the Reliability of Formal Specifications. In 18th USENIX Symposium on Operating Systems Design and Implementation. USENIX Association, Santa Clara, CA, 875–891. https://www.usenix.org/conference/osdi24/presentation/goldweber
  17. [17]Warren A. Hunt, Matt Kaufmann, J Strother Moore, and Anna Slobodova. 2017. Industrial Hardware and Software Verification with ACL2. Philosophical Transactions of the Royal Society A 375, 2104 (2017), 20150399. doi:10.1098/rsta.2015.0399DOI
  18. [18]Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. doi:10.1109/TSE.2010.62DOI
  19. [19]Moa Johansson, Lucas Dixon, and Alan Bundy. 2011. Conjecture Synthesis for Inductive Theories. Journal of Automated Reasoning 47, 3 (2011), 251–289. doi:10.1007/s10817-010-9193-yDOI
  20. [20]Moa Johansson, Dan Rosén, Nicholas Smallbone, and Koen Claessen. 2014. Inpster: Integrating Theory Exploration in a Proof Assistant. In Intelligent Computer Mathematics - International Conference, CICM 2014 (Lecture Notes in Computer Science, Vol. 8543). Springer, Coimbra, Portugal, 108–122. doi:10.1007/978-3-319-08434-3_9DOI
  21. [21]Matt Kaufmann, Panagiotis Manolios, and J Strother Moore. 2000. Computer-Aided Reasoning: An Approach. Kluwer Academic Publishers, Boston, MA, USA. doi:10.1007/978-1-4615-4449-4DOI
  22. [22]Matt Kaufmann and J Strother Moore. 2001. Structured Theory Development for a Mechanized Logic. Journal of Automated Reasoning 26, 2 (2001), 161–203. doi:10.1023/A:1026517200045DOI
  23. [24]Tobias Kiecker, Jan Arne Sparka, Martin Reuter, Albert Ziegler, and Lars Grunske. 2026. CASCADE: Detecting Inconsistencies between Code and Documentation with Automatic Test Generation. Proceedings of the ACM on Software Engineering 3, FSE (2026), 3816–3838. doi:10.1145/3808175DOI
  24. [25]Lezhi Ma, Shangqing Liu, Yi Li, Xiaofei Xie, and Lei Bu. 2025. SpecGen: Automated Generation of Formal Program Specifications via Large Language Models. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering. IEEE Press, Piscataway, NJ, USA, 16–28. doi:10.1109/ICSE55347.2025.00129DOI
  25. [26]Tobias Nipkow, Lawrence C. Paulson, and Markus Wenzel. 2002. Isabelle/HOL: A Proof Assistant for Higher-Order Logic. Lecture Notes in Computer Science, Vol. 2283. Springer, Berlin, Heidelberg. doi:10.1007/3-540-45949-9DOI
  26. [27]Sheena Panthaplackel, Junyi Jessy Li, Milos Gligoric, and Raymond J. Mooney. 2021. Deep Just-In-Time Inconsistency Detection Between Comments and Source Code. Proceedings of the AAAI Conference on Artificial Intelligence 35, 1 (2021), 427–435. doi:10.1609/aaai.v35i1.16119DOI
  27. [28]Andy Podgurski and Lori A. Clarke. 1990. A Formal Model of Program Dependences and Its Implications for Software Testing, Debugging, and Maintenance. IEEE Transactions on Software Engineering 16, 9 (1990), 965–979. doi:10.1109/32.57884DOI
  28. [29]Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. Measuring Attribution in Natural Language Generation Models. Computational Linguistics 49, 4 (2023), 777–840. doi:10.1162/coli_a_00486DOI
  29. [30]Chuyue Sun, Ying Sheng, Oded Padon, and Clark Barrett. 2024. Clover: Closed-Loop Verifiable Code Generation. In AI Verification – First International Symposium, SAIV 2024. Springer, Montreal, Canada, 134–155. doi:10.1007/978-3-031-65112-0_7DOI
  30. [31]Weisong Sun, Chunrong Fang, Yudu You, Yun Miao, Yi Liu, Yuekang Li, Gelei Deng, Shenghan Huang, Yuchen Chen, Quanjun Zhang, Hanwei Qian, Yang Liu, and Zhenyu Chen. 2023. Automatic Code Summarization via ChatGPT: How Far Are We? arXiv:2305.12865. arXiv:2305.12865arxiv.org/abs/2305.12865
  31. [33]Yuhui Wu, Albert Qiaochu Jiang, Wenda Li, Markus N. Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. 2022. Autoformalization with Large Language Models. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022).
  32. [34]Lingfei Zeng, Fengdi Che, Xuhan Huang, Fei Ye, Xu Xu, Binhang Yuan, and Jie Fu. 2026. VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Code. arXiv:2510.06296 [cs.PL] https://proceedings.iclr.cc/paper_files/paper/2026/hash/ebfa4297cd6419f64efe86f657ba49d0-Abstract-Conference.html International Conference on Learning Representations (ICLR 2026).arxiv.org/abs/2510.06296

Paper details

vlnc:W_4xqygm7c
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents