WorksComputer Science
VerusWeave: A Benchmark for Proof Repair in Concurrent Verus Programs
Abstract
Concurrent software underpins critical infrastructure, including operating systems and databases, yet thread interactions can have bugs that bypass conventional testing. Verus extends Rust with formal verification capabilities, enabling developers to specify, implement and prove the correctness of concurrent code in the same language. Although coding agents have been explored for automated proof construction, existing Verus benchmarks offer limited coverage of concurrency-specific examples that connect atomic operations to abstract protocols. To address this gap, we introduce VerusWeave, the first benchmark dedicated to concurrent proof repair, derived from four open-source projects. It comprises 75 focused tasks and 17 full-module variants that exercise features used to verify low-level concurrent code. Tasks remove selected proof annotations and require agents to fill gaps, while preserving executable code and specifications. The evaluation harness validates repairs using Verus and a change checker, recording solutions, execution traces, and resource usage. An initial evaluation of four general-purpose coding-agent harnesses achieved 75-85% success on focused tasks and 29-47% on full-module variants, highlighting the challenge of autonomous module-wide repairs. VerusWeave provides a reusable foundation for evaluating automated proof construction for concurrent software.
Introduction
Large Language Models (LLMs) and autonomous programming agents are transforming software development, but generated code is not necessarily correct, and its growing volume increases verification demands. Providing stronger correctness guarantees without prohibitive development effort is therefore increasingly important.
Formal verification addresses this need by mathematically establishing that programs satisfy their specifications, unlike testing, which examines finitely many inputs. However, its adoption remains limited by the expertise, cost, and maintenance effort required to construct and evolve proofs [1], [2]. LLMs can automate specification generation, invariant inference, and proof construction [3, 4, 5, 6], while integrating verification into coding agents could further ease the development and maintenance of verified software.
Concurrent software presents a particularly challenging target for this automation. It underpins critical infrastructure, including operating-system kernels, databases, and distributed services, yet bugs may depend on rare thread interleavings that testing struggles to expose [7]. Proving correctness also requires reasoning about inter-thread interference, shared-state invariants, and memory-access permissions, making concurrency a demanding setting for automated proof generation.
Verus [8] provides a foundation for addressing this challenge by extending Rust with specifications and proof constructs for machine-checked verification. Its applications to operating systems and security modules demonstrate the practical relevance of this approach [9, 10, 11]. Meanwhile, coding agents can iteratively modify code, invoke tools, and respond to compiler and verifier feedback. This makes proof repair a natural target for automation: agents can refine proof annotations until verification succeeds, without changing the executable implementation or specification. Successful verification provides machine-checkable evidence of correctness relative to the encoded specification and verifier assumptions.
Despite growing interest in LLM-assisted verification, existing Verus benchmarks provide limited coverage of the low-level concurrency primitives and proof patterns needed to verify concurrent systems. Consequently, coding agents’ ability to repair such proofs, the difficulties they encounter, and the effects of agent and harness design remain underexplored.
To address this gap, we introduce VerusWeave, a benchmark for evaluating agents on concurrent Verus proof repair. Derived from existing Verus projects, it comprises 75 focused tasks and 17 full-module variants exercising low-level concurrency primitives. VerusWeave targets researchers developing automated software repair methods and studying agents’ effectiveness, failure modes, and resource usage. The dataset contains standalone Rust tasks alongside logs, statistics, and solutions from an initial evaluation. We include an evaluation harness to run the benchmark, validate candidates and inspect run-level JSON records containing validation outcomes, resource statistics, and solution paths.
Benchmark Design and Construction
The benchmark targets two proof-obligation families in Verus’s low-level concurrency support [12]: ghost-resource manipulation around atomic operations and inductiveness proofs for tokenized state machines. These families address different, but complementary, parts of a concurrent correctness argument. The first connects an executable synchronization event to a change in abstract state; the second establishes that the abstract change is permitted by the protocol.

Figure 1. Atomic-operation task from OnceImpl::get. The added ghost code recovers the permission to the initialized cell; executable code is unchanged.

Figure 2. Reader–writer lock state-machine task. The added proof establishes zero reader counts and no pending or active readers at initialization.
| Module | A | T | F |
| Verified Node Replication | |||
| context | 2 | 0 | ✓ |
| cyclicbuffer | 0 | 6 | ✓ |
| flat_combine | 0 | 8 | ✓ |
| log | 5 | 0 | ✓ |
| replica | 4 | 0 | ✓ |
| unbounded_log | 0 | 6 | ✓ |
| Verus examples | |||
| count_to_two | 2 | 0 | ✓ |
| cuckoo | 3 | 1 | ✓ |
| dist_rwlock | 0 | 4 | ✓ |
| fifo | 2 | 5 | ✓ |
| helping | 3 | 0 | ✓ |
| statics | 1 | 0 | – |
| Verified Memory Allocator | |||
| free | 2 | 0 | ✓ |
| linked_list | 6 | 0 | ✓ |
| tokens | 0 | 3 | ✓ |
| types | 2 | 0 | ✓ |
| VOSTD | |||
| once | 2 | 0 | ✓ |
| rwlock | 7 | 0 | ✓ |
| wait | 1 | 0 | – |
| Total (19 modules) | 42 | 33 | 17 |
Table 1. Source modules and repair tasks in VerusWeave. A: atomic-operation tasks; T: state-machine inductiveness tasks; F: full-module variant. Module names omit .rs.
Task Families
Atomic-operation tasks require the agent to restore ghost code associated with a fixed atomic operation, such as load, compare-and-exchange, or fetch-and-add. This code tracks abstract protocol state and resource ownership without affecting execution; invariants connect that state and ownership to the shared atomic value. The omitted code appears within an atomic_with_ghost! block or another proof context that permits ghost-resource updates. To reconstruct it, the agent must reason about the operation’s effect on the atomic value and its return value to determine how to update the ghost state. A task is complete when the restored ghost updates are verified to preserve the invariants.
State-machine tasks require the agent to restore an inductiveness proof for an abstract concurrent protocol defined with Verus’s tokenized_state_machine! macro, that specifies the program’s possible states, initialization, transitions, invariants, and ghost-resource ownership. Each task retains these definitions but removes the inductive proof lemma for a selected transition. The agent must show that, if the invariants hold before the transition, they also hold afterward.
Figures 1 and 2 illustrate representative tasks from the atomic-operation and state-machine families, respectively.
Source Projects and Selection
The corpus draws on open-source Verus code from Verified Memory Allocator [13], Verified Node Replication [14], and vostd [15], the verification library used by Asterinas [16]. Due to the scarcity of public projects exercising the target concurrency mechanisms, we also include examples from the Verus repository [17]. These sources cover logical atomicity demonstrations, reader–writer locks, concurrent queues, memory allocation, and replicated logs. Table I summarizes source provenance and task counts.
The wait task proof was developed during this work and subsequently contributed to vostd, where it was merged in pull request 702.1
Task Extraction
We selected source modules containing the target proof-obligation families and recursively inlined local dependencies into standalone Rust files. Focused tasks remove ghost code associated with selected atomic operations or invariant openings, or the body of a selected lemma, while preserving executable code and specifications. Each task targets one function, except wait and cuckoo_inductive, which group functions with short, similar proof gaps.
This process yielded 75 focused tasks: 42 atomic-operation tasks and 33 state-machine tasks. For 17 source modules, we also removed all selected proofs simultaneously to create full-module variants requiring multiple repairs. These variants reuse the focused tasks’ proof obligations, providing complementary evaluations of isolated and combined repairs rather than 92 independent obligations.
Evaluation Framework
The tasks are accompanied by an automated evaluation framework used to collect the results presented in Section IV. Every agent must follow the same proof-repair contract. It may add proof code, but it must preserve all executable code, specification code, and existing function contracts. In particular, the agent may not weaken a precondition or postcondition, replace executable statements, introduce an axiom, or mark an existing function as trusted. The use of admit and assume is also prohibited. This way, agents cannot obtain a passing result by moving, removing, or replacing existing behavior.
Execution
Each task runs in a temporary directory inside a dedicated Docker container with the agent’s CLI setup and a fixed base image built with the latest or another specified Verus version, its standard library (vstd), and Lynette [18, 19], a cheat checker which we extended to support the corpus’s proof forms. The prompt specifies input/output paths and edit restrictions, instructing the agent to verify its changes with Verus until a candidate passes verification and the cheat checker.
Since public repositories may expose reference proofs, external search and download tools are disabled or blocked, while model-provider access remains enabled. Session logs support manual inspection to identify and repeat runs that accessed external solutions.
Agents are given 20 minutes for each task. When this limit is reached, the agent container is stopped and the task marked as timed out. If the agent produces the expected output file, the evaluation framework extracts and evaluates it; otherwise, the task records that no candidate solution was produced.
Solution Validation and Recorded Outputs
Each candidate is validated in a fresh container built from the benchmark base image and accepted only if it passes Verus verification and Lynette.
The harness preserves generated solutions and complete session logs. Agent-specific parsers extract estimated model cost, agent turns, tool calls, and session duration. These statistics, validation results, timeout status, and solution paths are recorded incrementally in a run-level JSON file.
Infrastructure failures, including rate limits, abnormal exits, and invalid logs, leave tasks pending for resumption without repeating completed work.
Experimental Evaluation
We evaluate four coding-agent harnesses on VerusWeave to assess repair effectiveness, challenging proof patterns, and resource consumption: Codex [20], OpenCode [21], GitHub Copilot [22], and Pi [23]. All use medium reasoning effort and otherwise default configurations.
a) Proof-repair effectiveness: Agents repaired 56–64 of 75 focused tasks (75–85%), versus 5–8 of 17 full-module tasks (29–47%) with GPT-5.6 Luna [24]. A follow-up using Claude Sonnet 4.6 [25] with OpenCode and Pi solved five of the seven focused tasks that all initial agents had failed, but only one of the seven remaining unsolved full-module tasks. These baselines suggest that coordinating multiple repairs within a module is harder than completing isolated proof gaps.
b) Proof difficulty: All seven focused tasks initially unsolved by every agent involved atomic operations. Repair success declined with larger proof omissions and more atomic sites. Qualitative analysis revealed difficulties preserving arithmetic bounds, applying existing lemmas, and avoiding unproductive responses to verifier feedback. These findings illustrate the benchmark’s potential for analysing proof difficulty and guiding concurrency-aware repair.
c) Resource efficiency: The study compared repair success with token consumption, tool calls, and monetary cost. Pi achieved success rates among the highest observed while maintaining the lowest mean cost per attempt in both task sets. Across the evaluated configurations, greater resource consumption did not yield more successful repairs. The benchmark, therefore, supports comparisons of repair effectiveness and resource efficiency under a common evaluation protocol.
Related Work
Table 2 compares existing Verus benchmarks by size, synthesis objectives, and coverage of program domains and Verus primitives. Despite their diversity, concurrency-specific proof obligations remain underrepresented. VerusWeave addresses this gap by evaluating agents’ ability to repair proofs involving two largely unexplored Verus concurrency primitives: permissioned atomics and tokenized state machines.
| Benchmark | Size | Main task | Focus |
| Verus-Bench [18] | 150 tasks | Proof repair | Small algorithm programs |
| VeruSAGE-Bench [19] | 849 tasks | Proof repair | Real systems code |
| VeriStruct [26] | 11 modules | Specification + proof synthesis | Data structures |
| AlgoVeri [27] | 77 problems | Code + proof generation | Textbook algorithms |
| VeriContest [28] | 946 problems | Specification, code, proof generation | Competitive programming |
| LeetCode-Verus [29] | 37 examples | Verified program collections | LeetCode algorithms |
| HumanEval-Verus [30] | 85 function tasks | Verified program collections | Small programming problems |
| VerusWeave | 75 tasks / 17 modules | Concurrent proof repair | Atomics and tokenized state machines |
Table 2. Comparison of Verus benchmarks.
Threats to Validity and Future Work
VerusWeave samples four projects rather than representative Rust concurrency code, reflecting the scarcity of public projects using these interfaces. As tasks originate from public repositories and recent models are evaluated, prior exposure to source proofs cannot be excluded.
Future extensions include larger proof holes and logical atomicity and weak-memory reasoning. Agent involvement could also expand beyond proof repair to executable-code generation alongside proofs, guided by behavioural specifications.
Conclusion
We presented VerusWeave, the first Verus benchmark focused entirely on concurrent code, with 75 focused proof-repair tasks and 17 full-module variants from four open-source projects. It covers ghost-resource reasoning around atomic operations and invariant preservation in tokenized state machines. Its evaluation framework validates repairs and records agent behaviour and resource use. Initial results show higher success on focused tasks than full-module variants, motivating further evaluation of extended concurrency proof repairs. VerusWeave provides a reusable basis for evaluating and improving automated proof construction and verification tools for concurrent software.
Acknowledgements
This work was co-funded by the European Union through the Lisboa 2030 Programme (ERDF) and by national funds through FCT, I.P., under project no. 15018 (project/ SafeIaC, DOI: 10.54499/2023.18089.ICDT).
References
References
- [1]L. S. Brugger, X. Denis, and P. Müller, “Toward practical deductive verification: Insights from a qualitative survey in industry and academia,” 2025. [Online]. Available: https://arxiv.org/abs/2510.20514
- [2]F. Oliveira, A. Mendes, and C. Carreira, “What challenges do developers face when using verification-aware programming languages?” in 36th IEEE International Symposium on Software Reliability Engineering (ISSRE), 2025.arxiv.org/abs/2506.23696
- [3]Á. F. Silva, A. Mendes, and J. F. Ferreira, “Leveraging large language models to boost Dafny’s developers’ productivity,” 2024 IEEE/ACM 12th International Conference on Formal Methods in Software Engineering (FormaliSE), pp. 138–142, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:266725333arxiv.org/abs/2401.00963
- [4]Á. Silva, A. Mendes, and R. Martins, “Inferring multiple helper dafny assertions with llms,” arXiv preprint arXiv:2511.00125, 2025.
- [5]K. Thompson, N. Saavedra, P. Carrott, K. Fisher, A. Sanchez-Stern, Y. Brun, J. F. Ferreira, S. Lerner, and E. First, “Rango: Adaptive retrieval-augmented proving for automated software verification,” 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pp. 347–359, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:274710349arxiv.org/abs/2412.14063
- [6]H. Tu, H. Zhao, Y. Song, M. Zafar, R. Meng, and A. Roychoudhury, “Agentic verification of software systems,” Proc. ACM Softw. Eng., vol. 3, no. FSE, Jun. 2026. [Online]. Available: https://doi.org/10.1145/3808164
- [7]M. Musuvathi, S. Qadeer, T. Ball, G. Basler, P. A. Nainar, and I. Neamtiu, “Finding and reproducing Heisenbugs in concurrent programs,” in Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’08. USA: USENIX Association, 2008, p. 267–280.
- [8]A. Lattuada, T. Hance, C. Cho, M. Brun, I. Subasinghe, Y. Zhou, J. Howell, B. Parno, and C. Hawblitzel, “Verus: Verifying Rust programs using linear ghost types,” Proceedings of the ACM on Programming Languages, vol. 7, no. OOPSLA1, pp. 286–315, 2023. [Online]. Available: https://doi.org/10.1145/3568037
- [9]J. Zhang, X. Xu, Z. Zou, Z. Tang, X. Wan, K. Hu, S. Wang, W. Xu, D. Wang, H. Chen, L. Huang, S. Yan, Y. Tamir, Y. Luo, X. Wang, H. Yu, Z. Wang, H. Tian, and D. Zhou, “CortenMM: Efficient memory management with strong correctness guarantees,” in Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, ser. SOSP ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 1082–1098. [Online]. Available: https://doi.org/10.1145/3731569.3764836
- [10]Z. Zhou, Anjali, W. Chen, S. Gong, C. Hawblitzel, and W. Cui, “VeriSMO: A verified security module for confidential VMs,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 599–614. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/zhou
- [11]P. Singh, J. Gancher, and B. Parno, “OwlC: Compiling security protocols to verified, secure, high-performance libraries,” Cryptology ePrint Archive, Paper 2025/1092, 2025. [Online]. Available: https://eprint.iacr.org/2025/1092DOI
- [12]T. Hance, “Verifying concurrent systems code,” Ph.D. dissertation, Carnegie Mellon University, Aug. 2024.DOI
- [13]Verus Project Contributors, “verified-memory-allocator, based on mimalloc,” https://github.com/verus-lang/verified-memory-allocator, 2024, accessed: 2026-09-07.
- [14]——, “verified-node-replication,” https://github.com/verus-lang/verified-node-replication, 2024, accessed: 2026-09-07.
- [15]VOSTD Project Contributors, “vostd,” https://github.com/asterinas/vostd, 2025, accessed: 2026-09-07.
- [16]Asterinas Project, “Asterinas.” [Online]. Available: https://github.com/asterinas/asterinas
- [17]Verus Project Contributors, “verus,” https://github.com/verus-lang/verus, 2024, accessed: 2026-09-07.
- [18]C. Yang, X. Li, M. R. H. Misu, J. Yao, W. Cui, Y. Gong, C. Hawblitzel, S. Lahiri, J. R. Lorch, S. Lu, F. Yang, Z. Zhou, and S. Lu, “AutoVerus: Automated proof generation for Rust code,” Proceedings of the ACM on Programming Languages, vol. 9, no. OOPSLA2, p. 3454–3482, Oct. 2025. [Online]. Available: http://dx.doi.org/10.1145/3763174
- [19]C. Yang, N. Neamtu, C. Hawblitzel, J. R. Lorch, and S. Lu, “VeruSAGE: A study of agent-based verification for Rust systems,” 2025. [Online]. Available: https://arxiv.org/abs/2512.18436
- [20]OpenAI, “OpenAI Codex CLI,” https://github.com/openai/codex, 2026, accessed: 2026-09-02.
- [21]OpenCode, “OpenCode: The open source AI coding agent,” https://opencode.ai/en/docs, 2026, accessed: 2026-09-02.
- [22]GitHub, “GitHub Copilot,” https://docs.github.com/en/copilot/get-started/what-is-github-copilot, 2026, accessed: 2026-09-02.
- [23]Earendil Inc. and Contributors, “Pi: A minimal agent harness,” https://pi.dev/, 2026, accessed: 2026-09-02.
- [24]OpenAI, “GPT-5.6 Luna Model,” https://developers.openai.com/api/docs/models/gpt-5.6-luna, 2026, accessed: 2026-09-08.
- [25]Anthropic, “Introducing Claude Sonnet 4.6,” https://www.anthropic.com/news/claude-sonnet-4-6, Feb. 2026, accessed: 2026-09-08.
- [26]C. Sun, Y. Sun, D. Amrollahi, E. Zhang, S. Lahiri, S. Lu, D. Dill, and C. Barrett, “VeriStruct: AI-assisted automated verification of data-structure modules in Verus,” in Proceedings of the 32nd International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS), 2026.arxiv.org/abs/2510.25015
- [27]H. Zhao, Z. Yang, J. Li, M. He, Z. Li, C. Jin, V. Veeravalli, A. Gupta, and S. Arora, “AlgoVeri: An aligned benchmark for verified code generation on classical algorithms,” in Proceedings of the 43rd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, T. Zhang, M. Dudík, M. Jaggi, A. Agarwal, S. Li, D. Schuurmans, J. Zhu, F. Berkenkamp, H. Dong, and A. Bietti, Eds., vol. 306. PMLR, 06–11 Jul 2026, pp. 162621–162652. [Online]. Available: https://proceedings.mlr.press/v306/zhao26bm.htmlarxiv.org/abs/2602.09464
- [28]Z. Xie, M. Pawagi, Y. Liu, A. Rai, L. Shao, J. B. Jr., S. Che, and W. Wang, “VeriContest: A competitive-programming benchmark for verifiable code generation,” 2026. [Online]. Available: https://arxiv.org/abs/2605.08553
- [29]W. Dai and Contributors, “Verus study cases: LeetCode,” https://github.com/Com/WeituoDAI/verus-study-cases-leetcode, 2025, gitHub repository, accessed: 2026-09-20.
- [30]A. Bai, J. Bosamiya, E. Fernando, M. R. Hossain, J. R. Lorch, S. Lu, N. Neamtu, B. Parno, A. Shah, and E. Tang, “HumanEval-Verus: Hand-written examples of verified Verus code derived from HumanEval,” https://github.com/secure-foundations/human-eval-verus, 2024, gitHub repository.