betaFeedback

WorksComputer Science

Can Coding Agents Write and Transpile Unit Tests

Abstract

Coding assistants built on Large Language Models (LLMs) have become a staple of modern software development, increasingly trusted to execute workflows and modernize code end-to-end. Much of this trust rests on passing tests, which are often written by the coding agents themselves. In this work, we examine the ability of these agents to both write reliable tests and translate them between programming languages. We benchmark three popular coding agents, each evaluated with a native frontier model and high reasoning effort. Verifying across a broad suite of metrics, we find that coding agents far exceed human written suites on traditional measures of coverage. This advantage, however, is uneven: agent test quality degrades sharply with project size, and much of the advantage lies in a greater number of written tests. We also show that agentic test translation—while seemingly strong in aggregate—remains inconsistent and unreliable.

Introduction

Coding agents have become integral to modern software development, increasingly trusted with tasks as ambitious as generating and transpiling (translating) entire codebases. Verifying this work, however, requires well-constructed tests, which are often missing in realistic code generation settings. Translation is similarly afflicted: even when source language suites are reliable, producing an equally reliable suite in the target language is difficult, as each language introduces distinct translation edge cases and failure modes.

Prior work shows that LLMs can generally write unit tests for small, isolated tasks [14, 32]. By contrast, the quality of generated tests at the scale of full codebases, across different programming languages, and using state-of-the-art coding assistants has received considerably less attention. Moreover, even as agents are increasingly tasked with porting legacy systems to newer languages [5, 11, 25], their ability to faithfully translate tests remains underexplored.

In this work, we assemble a dataset of 240 open source projects with human written tests across C++, Python, and Rust. We then use this dataset to evaluate the test generation and translation abilities of three popular coding frameworks: Claude Code, Codex, and Cursor. Combining a broad suite of metrics, agent trace analyses, and controlled experiments, we characterize the strengths and weaknesses of these frameworks. Our contributions are as follows:

  1. We conduct a detailed investigation of the capabilities and failure modes of modern agentic test generation, finding that coding agents write far more comprehensive suites than their measured human counterparts. However, they often write less thorough individual tests.

40th Conference on Neural Information Processing Systems (NeurIPS 2026).

Experimental Design Overview

Figure 1. Experimental Design Overview. (Top): We use Cursor, Claude Code, and Codex to generate unit tests for 240 projects across three languages, measuring unit testing capabilities. (Bottom) Additionally, we assess test translation from C++ to Rust, evaluating the model’s ability to translate existing C++ test suites.

  1. We extend our analysis to test transpilation, showing that agents are inconsistent and unfaithful test transpilers in a way that aggregate metrics lack the power to detect.

  1. We perform an analysis of agentic reasoning traces, exposing pitfalls that sabotage unit test generation and revealing avenues for future work.

Data

Two datasets underpin our study: a corpus of 240 permissively licensed GitHub projects for the test generation task, and a 50-project subset enhanced for transpilation. Both datasets are described below.

Unit testing data

We collect 240 projects, stratified uniformly across Python, C++, and Rust. Within each language, we further stratify by size: 30 small projects (0-1,000 lines of source code), 30 medium (1k-5k), and 20 large (5k-10k). All collected projects meet the following criteria:

  • Permissive licensing for redistribution (Apache 2.0 or MIT)

  • A fully passing and present test suite

  • C++ datasets: a CMake build (CMakeLists.txt) and no external dependencies

We place additional restrictions on C++ projects since they will be translated in our test transpilation experiment, and many dependencies do not have an obvious Rust counterpart.

Our preprocessing pipeline refactors all projects into a uniform format. It then anonymizes them by applying semantic preserving changes to naming schemes and deleting non-docstring documentation. This serves to differentiate the data from that which may have been seen at training time, thus reducing replication of memorized human tests [13]. The dataset statistics are provided in Table 1.

LanguageNNSmallMediumLargeLOC med. [IQR]Files med.
Python803030201,656 [738, 5,111]8
C++803030201,408 [615, 5,122]5
Rust803030202,526 [734, 5,012]8.5
Total2409090601,688 [700, 5,015]8

Table 1. Unit Test generation corpus: 240 frozen, anonymized repositories, stratified by language and size.

Transpilation Data

From the C++ corpus, we transpile 50 projects to Rust and release both the source and transpiled artifacts. These projects map cleanly, having platform-agnostic builds and no dependencies that might lack a Rust equivalent. Our final dataset consists of 23 small projects, 20 medium projects, and seven large projects.

Metrics and measurement

We score every test suite against a fixed per-project checklist: the production lines that exist, the mutants (small bugs) we inject, and the project’s public methods. The first two anchor our traditional software testing metrics (line coverage, mutation score, and vacuous test rate) which capture whether tests execute the code and whether they can detect basic faults, but not whether they validate edge cases and core behaviors. For that, we introduce an LLM-as-judge protocol that scores suites against a catalog of important behaviors and edge cases, enabling finer-grained semantic comparison.

We measure the project’s own tests once, then score agent tests against the same reference. When a score would divide by zero, we omit that project from the average.

Line coverage. Let UU be the set of all measurable production lines in a given project. A line is considered "covered" only if it is run by a test. We only keep tests that pass, and we drop tests that crash, fail spuriously, or only pass when run in conjunction with other tests. If the suite never runs, or no test passes, the score is 0:

C=∣Cover⁡(P)∣∣U∣.(1)C = \frac{|\operatorname{Cover}(P)|}{|U|}. \tag*{(1)}

Cover⁡(P)\operatorname{Cover}(P) is the lines the remaining passing tests PP ran.

Mutation score. We make small automatic mutations (bug-inducing edits) to the production code, such as flipping a comparison, swapping + and −, negating a condition, or changing a number cc to c+1c+1 or 0. Each project gets a fixed sample of at most 120 such edits; those that do not compile are thrown out. The tests kill a mutation if some test fails, hangs, or crashes when running against it.

MS=∣K∣∣V∣.(2)\mathrm{MS} = \frac{|K|}{|V|}. \tag*{(2)}

KK is the set of mutants the tests kill. VV is the set of edits that compiled.

Vacuous-test rate. We replace each function body with a do-nothing stub (return None in Python, an empty, error raising, or default return in C++ and Rust). We then define the vacuous test rate by the fraction of tests that continue to pass when run against the stubbed implementation (and were thus not validating behavior in the first place):

Rvac=∣A∖Sens∣∣A∣.(3)R_{\mathrm{vac}} = \frac{|A \setminus Sens|}{|A|}. \tag*{(3)}

Where AA is the set of tests that pass on the original (non-stubbed) implementation. SensSens is the subset of AA that then fail, hang, or crash when run against the stubbed implementation.

LLM-as-judge. Once per project, a judge model (Kimi K3) reads the public functions and writes a list of important behaviors and edge case inputs for these behaviors. Every suite is scored against that same list. A behavior counts as covered if a simple text pattern for it appears in the tests. Edge cases need a second model pass: tests often build such inputs in variables, and thus may be missed by a fixed pattern.

Cbeh=∣Bhit∣∣B∣,Cedge=∣Ehit∣∣E∣.(4)C_{\mathrm{beh}} = \frac{|\mathcal{B}_{\mathrm{hit}}|}{|\mathcal{B}|}, \qquad C_{\mathrm{edge}} = \frac{|\mathcal{E}_{\mathrm{hit}}|}{|\mathcal{E}|}. \tag*{(4)}

B\mathcal{B} and E\mathcal{E} are the lists of behaviors and edge cases. The hit sets are the ones the suite covered.

All remaining metric definitions can be found in Appendix § 12.

Experiments

We run three containerized and sandboxed experiments, building each project inside a clean container in accordance with its build instructions. Unless otherwise stated, we run all models with high reasoning and unlimited token budgets for testing. Our prompts—along with other shared settings—are not specialized to any agent and can be found in Appendix § 10. Individual experiments are detailed in the following sections.

Unit testing

To measure the ability of modern coding frameworks to write unit tests, we remove all human tests and instruct the three coding agents to repopulate them. A coding assistant may run the tests and observe output – it is bound only by a lack of network access so that it cannot look up the projects online. For each coding framework, we leverage the most recent frontier LLM: GPT-5.6-sol for Codex, Claude-Opus-5 for Claude Code, and Cursor-Grok-4.6 for Cursor.

Test transpilation

Beyond generating unit tests from scratch, we examine coding agents’ ability to translate existing test suites across languages. We center evaluations on C++ to Rust, a pairing of particular interest in code modernization [8, 20]. Using the transpiled Rust project corpus detailed in § 2.2, the agent is presented with a full C++ project, the native C++ tests, and the transpiled Rust source. It is then prompted to transpile the C++ test suite to Rust.

As in the previous section, we leverage the most recent frontier LLM for each coding agent. We also perform unconditional generation of a Rust suite (providing only transpiled Rust, without the source code and source test suite) as a means of comparison.

Interpretive analysis

The Interpretive experiment repeats the unit testing setup in 4.1 on a 225 project subset. We leverage a Codex harness and GPT 5.6, investigating reasoning traces to identify core issues in Codex’s test generation workflow. We establish three fundamental failure modes and track them across projects by analyzing local reasoning traces, tool calls, and additional harness data.

Results

Unit testing

Holistically, our results show that the agentic test suites were far more comprehensive than their human counterparts. Figure 2 shows statistically significant improvements in mutant kill rate over the human baseline, and the pattern repeats across line coverage, redundancy, behavioral coverage, and edge case coverage.

Mutation score by author, kill probability by language, and mutation score by project size

Figure 2. Coding agents outperform human-written test suites on average, but the lead narrows on larger projects. (a) Mutation score by suite author; all agents exceed human baseline. (b) Mutant kill probability by language. Measured suites perform best in Python and worst in C++ across the board. (c) Mutation score by project-size. Agent scores decay faster than human scores; on large projects only Claude is far ahead. Error bands are 95% CIs.

Stratification, however, exposes clear weaknesses. Agents cover Python capably, but struggle with C++ across the board (Figure 2b). Every agent experiences a dropoff of at least 14% relative to its Python performance.

Test quality also decays significantly with project size, and disproportionately so in agents. On large projects, Codex essentially matches the quality of human written suites, in stark contrast to small projects where agents dominate. Full results appear in Appendix § 9.

Analyzing suite quality, moreover, conceals a major testing efficiency disparity (Figure 3). On a per-test basis, humans are more efficient authors than two of the three coding agents: each human-written test covers 8.5 production lines, against 3.7 for Cursor and 2.2 for Claude Code, with only Codex ahead at 11.6. (Figure 3a). Claude Code in particular reaches its coverage primarily through volume, typically writing roughly over seven times as many tests as the human baseline. This volume carries some waste. The vacuous-test rate (tests that still pass even once the tested function they cover is stubbed out) climbs steadily from Python through C++ to Rust, and is highest for Claude Code on average (Figure 3b).

Agents achieve similar test quality at very different costs

Figure 3. Agents achieve similar test quality at very different costs. (a) Production lines covered per test written: Codex is the most economical author; Claude reaches comparable coverage with roughly five times as many tests. (b) Vacuous-test rate by language, rising from Python to Rust; Claude and Cursor fare worst throughout. (c) Spend per mutant killed, relative to Codex: Claude uses far more tokens per mutant killed than other suites. Error bars are 95% CIs.

The pattern also extends to resources: per mutant killed, Claude Code spends 7.3×7.3 \times the input tokens, 6.4×6.4 \times the output tokens, and 8.5×8.5 \times the wall-clock time of Codex (Figure 3c). Cursor sits near 2 to 5×5 \times on all three measures. Taken together, the figures suggest that large codebases and lower-resource languages remain harder for agents to unit test; and where agents do hold an advantage, much of it result from simply writing more tests.

Test transpilation

We evaluate three coding agents—Claude Code, Codex, and Cursor—as test transpilers and place full corpus results in Table 2. The agents appear to be decent transpilers in aggregate, with translated suites roughly maintaining the mutation scores of their C++ baselines. The pointwise picture is different. As seen in Figure 4, the agent-transpiled suite underperforms the source suite in edge case coverage 58% of the time, with dropoffs reaching 45% in the absolute. In 55% of projects, the agent kills fewer mutants altogether (Figure 5). In the worst case, the mutant coverage of the source suite experiences nearly a 70% relative dropoff in all agentic transpilations.

MetricC++ humanClaude Opus 5Codex gpt-5.6Cursor grok-4.6
Mutation score0.559±0.0330.559 \pm0.0330.514±0.0330.514 \pm0.0330.489±0.0350.489 \pm0.0350.513±0.0330.513 \pm0.033
Edge case coverage0.340±0.0350.340 \pm0.0350.297±0.0310.297 \pm0.0310.287±0.0320.287 \pm0.0320.289±0.0310.289 \pm0.031
Line coverage0.747±0.0340.747 \pm0.0340.651±0.0320.651 \pm0.0320.612±0.0360.612 \pm0.0360.638±0.0340.638 \pm0.034

Table 2. Aggregate metrics for human source test suites and transpiled Rust across all three coding agents. Transpiled test suites lag behind C++ source suites in every major indicator, in particular in line coverage. Values are mean ±\pm sample standard deviation across three runs.

Edge case coverage scores for human source vs transpiled suites

Figure 4. Edge case coverage scores for human source test suites, transpiled Rust test suites, and test suites independently generated from transpiled Rust source.

Mutation scores for human source vs transpiled suites

Figure 5. Mutation scores across human, transpiled, and independently generated test suites. Transpilation performance is upper bounded by independently generated performance, and transpiled test suites lag meaningfully behind their human sources on several projects.

This gap is partially masked by independent test generation. Figure 5 makes this visible by also plotting suites generated from the final Rust implementation alone, with the C++ source withheld. The comparison reveals a surprising ceiling, showing that free-form generation nearly always upper-bounds the translated suites. In effect, the agent interpolates between translation and generation, sometimes producing tests that far exceed the source suite. However, the coupling cuts both ways. Where unconditional generation is weak, translation is weak and unfaithful. In some instances the translated suite even falls catastrophically below both the free-form suite and the human source.

Notably, the three agents’ translated suites cluster tightly on a significant majority of projects—these failure modes appear to be properties of the task, not of any particular assistant. Taken together, the results raise concerns about the reliability of modern coding agents on transpilation tasks.

Interpretive analysis

We examined 675 test-generation trajectories produced by Codex GPT-5.6 Sol across 225 projects, with three runs per project. The traces reveal failures in repository scope and target selection and in oracle formation, alongside gaps in coverage-feedback utilization. All percentages below use the 675 trajectories as their denominator.

The weakest runs share the following failure modes: skipping relevant code, forming trivial or implementation-anchored oracles, and failing to implement feedback effectively. These modes appeared even when the traces showed apparent prior knowledge: the agent identified the upstream library or recalled its structure, but still produced a weak suite (Listings C.12–C.17).

Repository scope and target selection. In 103 trajectories (15.3%), we found evidence that the agent either did not read relevant source code or knowingly omitted a supported testing objective. The traces show relevant implementation files left unread despite their inclusion in the build and their connection to the tested public behavior (Listings C.1–C.3). They also contain explicit decisions to exclude identified behavior without retaining an equivalent test.

Oracle formation. Seventy-five trajectories (11.1%) exhibited evidence of a retained oracle-formation failure. These included preserving behavior the agent recognized as erroneous, accepting incidental output from malformed input, and weakening assertions until they no longer established the intended property (Listings C.4–C.9). The final suite could therefore pass while retaining assertions that preserved an implementation error or no longer checked the intended validation behavior.

Feedback utilization. In 376 trajectories (55.7%), no coverage-measurement attempt was recorded, either through an explicit agent command or through inherited test infrastructure (Table 12). The traces show repeated test execution and repair without a corresponding coverage measurement. In these runs, the execution feedback established whether the generated suite passed, but did not measure which implementation paths remained untested (Listings C.10–C.11 and Table 11).

EventFailedPassedExit code
item_2216771
item_283901
item_301921
item_340930

Table 11. Recorded test-execution outcomes for geometer, repetition C.

Recorded coverage signalTrajectoriesOf all 675
No measurement attempt37655.7%
Measurement attempted21131.3%
Successful measurement documented9914.7%

Table 12. Selected coverage observations, including inherited instrumentation. Rows are not an exhaustive partition; successful measurement is a subset of attempted measurement. No weak-outcome classification is used.

We tested these findings using a failure-aware prompt (Listing B.3) to run Codex GPT-5.6 Sol on 41 projects spanning multiple languages and sizes. The failure-aware prompt improves most metrics, at the cost of slightly increased redundancy (Table 15).

MetricStandard promptFailure-aware promptDifference
Line coverage92.21%96.43%+4.22 pp
Mutation score69.98%73.61%+3.63 pp
Redundancy18.01%20.80%+2.79 pp
Vacuous-test rate3.86%3.62%−0.24 pp

Table 15. Test-quality metrics on the matched 41-project cohort. The standard-prompt value is the mean of three runs; the failure-aware prompt uses one run.

Related Works

Datasets and benchmarks. Existing test generation benchmarks are predominantly in Python or Java. TestEval (210 Python LeetCode problems) [35] and TestBench (108 functions from nine real-world Java projects) [38] provide curated unit-level tasks, but are limited in scale and complexity. TestGenEval [17] and SWT-Bench [26] are substantially larger and more realistic, but are derived from SWE-Bench [18] repository histories. Their target behavior is therefore defined relative to an issue-resolving patch and may span multiple components, making these tasks closer to repository-level regression or issue reproduction than isolated unit testing.

Recent studies also suggest that data contamination and memorized surface patterns can inflate apparent test-generation performance. Huang et al. [16] introduce ULT (UnLeakedTestBench), a benchmark of real world Python functions whose corresponding tests do not appear in The Stack v2 [23], alongside PLT (PreLeakedTestBench), which contains functions with potentially leaked tests. Across 12 models, average Pass@1 falls from 48.42% on PLT to 12.69% on ULT. Haroon et al. [13] reach a related conclusion: even semantics-preserving program transformations substantially reduce generated-test pass rates.

Our dataset instead targets isolated, human written unit tests from 240 real world projects spanning Python, C++, and Rust, with a range of project sizes and complexities. We additionally apply transformations such as identifier renaming to reduce the likelihood that models can rely on memorized public source code or tests.

LLMs for test generation. LLMs have been used throughout the unit test generation pipeline. Prior work generates tests from function context with execution feedback [6, 32], augments existing suites at industrial scale [2], reproduces bug reports as tests [19], and improves assertions using retrieved demonstrations [27]. Hybrid methods combine LLMs with search-based testing [9, 22] or mutation testing [7]. Others guide generation with program analysis, including method slices [36], symbolic path constraints [31], coverage reports [30], and uncovered control-flow paths [12]. ASTER extends this style of pipeline across languages [29].

Empirical studies in JavaScript [32], Java [33, 37], Python [35], and oracle generation [24] collectively show that LLMs can produce plausible tests for simple methods but degrade substantially on more complex, real world code. Controlled studies identify several recurring failure modes. Generated tests can anchor on an incorrect implementation [15], inherit mistakes from self-written code [21], or be emptied to make them pass [4]. Test quality also degrades with program complexity [1] and can remain superficial despite high coverage [34].

Most of these studies evaluate single model calls or fixed pipelines, often in a single language. Coding agents have received far less attention. SWT-Bench [26] shows that repair agents outperform dedicated test generators at reproducing GitHub issues, but only in Python. The only direct comparison of production assistants we are aware of evaluates 99 LeetCode problems [14]. Whether agents’ harnesses, tools, and instructions change these outcomes on real, multilingual repositories remains open.

Test migration and cross-language translation. LLM code translation is known to be unreliable, with correct translation rates of ≤47%\le47\% across models and benchmarks [28]. Test translation has received less attention. Even within a single language, results remain poor: LLMs fail on roughly half of Python unittest-to-pytest migrations [3]. Cross-language migration has so far succeeded only with traditional program analysis and a human in the loop. MUT [10] translates 550 tests through AST rewriting, of which maintainers accepted 253. We are unaware of prior work that measures semantic preservation—including edge cases, assertions, and fault-catching power—when translating tests across languages. .

Conclusion

In this paper, we evaluated three coding agents on unit test generation for 240 real-world projects across Python, C++, and Rust. On average, agents beat human suites on traditional metrics, but the gains are uneven: performance degrades with project size and lags on C++. Much of the agents’ advantage emanates from writing more tests as opposed to making them stronger individually.

We also study C++→RustC++ \to Rust test transpilation. While aggregate scores look fair, translated suites frequently diverge from their sources and regress on about 55% of projects. We notice this issue across both the software testing metrics we use and the semantic edge case metrics we define.

Our reasoning-trace analysis suggests three causes for these observations: agents read repositories incompletely, anchor oracles to the implementation, and rarely use coverage feedback. We thus conclude that agent-written, and especially translated, test suites remain unreliable in many scenarios and should be treated as drafts requiring careful review and measurement.

Limitations and future work. There are several present limitations to our work. First, our corpus consists of open-source repositories and does not include enterprise-scale codebases. Enterprise test suites are often more mature than the baselines we measure here, which could narrow agents’ apparent advantage while further stress-testing the reliability of agent-written suites. Building representative enterprise benchmarks, with appropriate privacy and licensing constraints, is an important direction for future work.

Furthermore, we test agentic harnesses with their native models, and do not isolate the individual impact of either the harness or models themselves. This is an important follow-up direction, and could aid practitioners working to implement harness and model level improvements.

Our interpretive analysis is also limited. Reasoning traces come from a single model (GPT) in a single harness (Codex); no comparable traces exist for the other frontier agents in our main experiments. Whether the same failure modes hold across agents, models, harnesses, and project types remains open.

Finally, we document frequent semantic regressions in C++→RustC++ \to Rust test transpilation but do not propose a remedy. Future progress likely requires stronger harnesses and evaluation loops, such as verifiers and targeted behavioral checks, alongside post-training tailored to semantic preservation.

Acknowledgments

We thank Alecander Gonzalez for his great help in editing and revising several drafts of this paper.

Unit Testing Experiment Metrics

MetricClaudeCursorCodexHumanAgent mean − Human [95% CI]
Mutation score0.8180.7760.7240.6250.147 [0.120, 0.174]
Line coverage0.9260.9130.8730.7620.142 [0.116, 0.168]
Coverage at pass0.9260.9130.8730.7620.142 [0.116, 0.168]
Mutants killed per test0.330.481.598.40−7.63 [−10.01, −5.25]
Assertion density0.2400.2890.2800.1700.099 [0.082, 0.116]
Test redundancy0.1600.1750.1500.253−0.096 [−0.129, −0.063]
Invalid-test rate0.00000.00170.00000.00000.0006 [−0.0002, 0.0013]
Stub sensitivity0.8650.8770.9190.8930.005 [−0.026, 0.036]
Vacuous test rate0.1350.1230.0810.107−0.005 [−0.036, 0.026]
Tests written29916247263−94 [−348, 159]
Test lines of code227715755361485−22 [−333, 289]
Test lines added22771575536––
Assertions written543447154217164 [124, 204]
Mutants killed per added test0.0920.1260.373––
Mutants killed per added test line0.01010.01200.0288––
Generation seconds1209611121––
Tool calls686718––
Output tokens977744998412945––
Input tokens64993781901918763534––
Mutation score per minute0.0700.0930.445––
Mutation score per tool call0.02260.01560.0529––
Suite runtime seconds3.241.940.816.86−4.87 [−8.16, −1.57]
Behavioral coverage0.8960.8880.8250.5990.271 [0.232, 0.310]
Edge-case coverage0.8850.8410.7180.4610.354 [0.319, 0.388]
Behaviors per test0.0470.0840.2601.064−0.937 [−1.321, −0.552]
Edge cases per test0.1210.1990.5511.610−1.323 [−1.858, −0.787]

Table 3. Overall projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.

MetricClaudeCursorCodexHumanAgent mean − Human [95% CI]
Mutation score0.8910.8410.7910.7060.134 [0.082, 0.187]
Line coverage0.9810.9630.9310.8440.115 [0.072, 0.158]
Coverage at pass0.9810.9630.9310.8440.115 [0.072, 0.158]
Mutants killed per test0.230.391.111.41−0.84 [−1.42, −0.25]
Assertion density0.2610.3180.2790.1990.087 [0.060, 0.115]
Test redundancy0.1600.1750.1500.253−0.096 [−0.129, −0.063]
Invalid-test rate0.00000.00000.00000.00000.0000 [−0.0000, 0.0000]
Stub sensitivity0.8990.9260.9610.9210.019 [−0.039, 0.077]
Vacuous test rate0.1010.0740.0390.079−0.019 [−0.077, 0.039]
Tests written51424476679−401 [−1163, 361]
Test lines of code226216405992151−650 [−1335, 35]
Test lines added22621640599––
Assertions written58951717237155 [−25, 134]
Mutants killed per added test0.0520.0870.227––
Mutants killed per added test line0.01140.01250.0296––
Generation seconds1202663133––
Tool calls877320––
Output tokens964055567014414––
Input tokens84830352277637920344––
Mutation score per minute0.0780.0900.436––
Mutation score per tool call0.01940.01520.0515––
Suite runtime seconds3.853.051.259.70−6.98 [−13.02, −0.95]
Behavioral coverage0.9150.9090.8580.6230.271 [0.212, 0.330]
Edge-case coverage0.9320.8860.7640.5250.336 [0.273, 0.399]
Behaviors per test0.0340.0730.2100.210−0.104 [−0.240, 0.032]
Edge cases per test0.0960.1830.4620.281−0.034 [−0.119, 0.051]

Table 4. Python projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.

MetricClaudeCursorCodexHumanAgent mean − Human [95% CI]
Mutation score0.7460.6820.6510.5370.156 [0.106, 0.207]
Line coverage0.8940.8690.8470.7420.129 [0.077, 0.180]
Coverage at pass0.8940.8690.8470.7420.129 [0.077, 0.180]
Mutants killed per test0.380.531.9621.28−20.56 [−26.87, −14.25]
Assertion density0.2570.3110.3140.1450.149 [0.117, 0.182]
Test redundancy–––––
Invalid-test rate0.00010.00000.00000.00000.0000 [−0.0000, 0.0001]
Stub sensitivity0.8550.8520.9220.899−0.004 [−0.091, 0.082]
Vacuous test rate0.1450.1480.0780.1010.004 [−0.082, 0.091]
Tests written219123314974 [?, ?]
Test lines of code24591406494144310 [−594, 613]
Test lines added24581406494––
Assertions written623434157140264 [?, ?]
Mutants killed per added test0.1350.1760.591––
Mutants killed per added test line0.01250.01590.0387––
Generation seconds1384551110––
Tool calls686116––
Output tokens1122684319411763––
Input tokens64371141524567607467––
Mutation score per minute0.0580.0950.451––
Mutation score per tool call0.02040.01530.0571––
Suite runtime seconds4.902.020.617.98−5.47 [−13.15, 2.20]
Behavioral coverage0.8870.8720.7960.5990.253 [0.185, 0.321]
Edge-case coverage0.8640.8030.6950.3850.402 [0.343, 0.462]
Behaviors per test0.0560.1020.3332.747−2.612 [−3.688, −1.536]
Edge cases per test0.1400.2310.6824.091−3.781 [−5.270, −2.292]

Table 5. C++ projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.

MetricClaudeCursorCodexHumanAgent mean − Human [95% CI]
Mutation score0.8160.8040.7300.6320.151 [0.114, 0.188]
Line coverage0.9040.9060.8410.7010.182 [0.141, 0.224]
Coverage at pass0.9040.9060.8410.7010.182 [0.141, 0.224]
Mutants killed per test0.390.511.712.51−1.64 [−2.36, −0.92]
Assertion density0.2000.2390.2480.1680.061 [0.036, 0.087]
Test redundancy–––––
Invalid-test rate0.00000.00500.00000.00000.0017 [−0.0007, 0.0040]
Stub sensitivity0.8480.8550.8860.8680.001 [−0.031, 0.033]
Vacuous test rate0.1520.1450.1140.132−0.001 [−0.033, 0.031]
Tests written165118336046 [17, ?]
Test lines of code21101678515859575 [?, ?]
Test lines added21101678515––
Assertions written416391132139174 [?, ?]
Mutants killed per added test0.0880.1160.301––
Mutants killed per added test line0.00640.00760.0181––
Generation seconds1039618119––
Tool calls496818––
Output tokens846485108712658––
Input tokens45779851903550762791––
Mutation score per minute0.0730.0950.448––
Mutation score per tool call0.02810.01640.0499––
Suite runtime seconds0.970.750.562.91−2.15 [−4.30, 0.01]
Behavioral coverage0.8840.8830.8190.5700.292 [0.213, 0.371]
Edge-case coverage0.8560.8320.6920.4760.317 [0.259, 0.375]
Behaviors per test0.0510.0780.2370.237−0.115 [−0.179, −0.050]
Edge cases per test0.1280.1820.5080.457−0.184 [−0.316, −0.052]

Table 6. Rust projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.

MetricClaudeCursorCodexHumanAgent mean − Human [95% CI]
Mutation score0.8630.8520.8370.6740.176 [0.134, 0.218]
Line coverage0.9300.9270.9350.8080.123 [0.088, 0.158]
Coverage at pass0.9300.9270.9350.8080.123 [0.088, 0.158]
Mutants killed per test0.360.441.6611.28−10.56 [−14.96, −6.17]
Assertion density0.2340.2770.2530.1870.068 [0.038, 0.097]
Test redundancy0.2540.2660.2180.303−0.059 [−0.130, 0.012]
Invalid-test rate0.00000.00240.00000.00000.0008 [−0.0008, 0.0023]
Stub sensitivity0.8660.8960.9170.8890.015 [−0.037, 0.067]
Vacuous test rate0.1340.1040.0830.111−0.015 [−0.067, 0.037]
Tests written152124323765 [42, 89]
Test lines of code1091933317299482 [388, 575]
Test lines added1091933317––
Assertions written2552558251147 [130, 164]
Mutants killed per added test0.0860.1080.356––
Mutants killed per added test line0.01110.01280.0334––
Generation seconds56944487––
Tool calls294212––
Output tokens44099351508668––
Input tokens1987904796060295296––
Mutation score per minute0.1190.1370.676––
Mutation score per tool call0.03800.02410.0793––
Suite runtime seconds1.531.250.731.64−0.47 [−1.37, 0.44]
Behavioral coverage0.9350.9280.8930.7090.209 [0.138, 0.281]
Edge-case coverage0.9350.9170.8610.4900.414 [0.350, 0.479]
Behaviors per test0.0340.0430.1441.375−1.312 [−2.066, −0.558]
Edge cases per test0.0990.1240.4081.659−1.458 [−2.190, −0.727]

Table 7. Small projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.

Table 8: Medium projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.

MetricClaudeCursorCodexHumanAgent mean − Human [95% CI]
Mutation score0.8170.7860.7310.6160.162 [0.116, 0.208]
Line coverage0.9370.9230.8900.7380.179 [0.132, 0.225]
Coverage at pass0.9370.9230.8900.7380.179 [0.132, 0.225]
Mutants killed per test0.360.541.728.83−7.95 [−12.22, −3.68]
Assertion density0.2430.3030.2840.1650.112 [0.083, 0.141]
Test redundancy0.1260.1440.1260.240−0.108 [−0.149, −0.068]
Invalid-test rate0.00010.00210.00000.00000.0007 [−0.0007, 0.0021]
Stub sensitivity0.8670.8510.9210.916−0.031 [−0.063, 0.001]
Vacuous test rate0.1330.1490.0790.0840.031 [−0.001, 0.063]
Tests written3091675416215 [−100, 129]
Test lines of code224216385941312180 [−228, 587]
Test lines added22421638594––
Assertions written535485168202194 [130, 257]
Mutants killed per added test0.1100.1470.445––
Mutants killed per added test line0.01170.01410.0315––
Generation seconds1149665126––
Tool calls616818––
Output tokens935075406213900––
Input tokens59925661893812715974––
Mutation score per minute0.0510.0770.372––
Mutation score per tool call0.01750.01300.0467––
Suite runtime seconds3.121.780.745.01−3.13 [−5.66, −0.61]
Behavioral coverage0.8980.8880.8320.5360.336 [0.274, 0.398]
Edge-case coverage0.8860.8500.7110.4550.361 [0.309, 0.414]
Behaviors per test0.0500.0880.2700.967−0.831 [−1.364, −0.298]
Edge cases per test0.1230.2070.5531.757−1.462 [−2.511, −0.414]

Table 8.

MetricClaudeCursorCodexHumanAgent mean − Human [95% CI]
Mutation score0.7520.6470.5460.5660.083 [0.029, 0.137]
Line coverage0.9050.8750.7550.7300.115 [0.057, 0.172]
Coverage at pass0.9050.8750.7550.7300.115 [0.057, 0.172]
Mutants killed per test0.240.441.293.43−2.78 [−5.31, −0.24]
Assertion density0.2430.2860.3170.1540.128 [0.100, 0.157]
Test redundancy0.0710.0870.0820.206−0.125 [−0.192, −0.059]
Invalid-test rate0.00000.00000.00000.00000.0000 [0.0000, 0.0000]
Stub sensitivity0.8620.8630.9210.8640.049 [−0.027, 0.126]
Vacuous test rate0.1380.1370.0790.136−0.049 [−0.126, 0.027]
Tests written50321058752−495 [−1504, 514]
Test lines of code411024427773522−1079 [−2133, −26]
Test lines added41082442777––
Assertions written986679238488147 [18, ?]
Mutants killed per added test0.0740.1210.290––
Mutants killed per added test line0.00620.00770.0179––
Generation seconds2258780164––
Tool calls13610527––
Output tokens1846856611717928––
Input tokens1402680935728651537231––
Mutation score per minute0.0240.0530.212––
Mutation score per tool call0.00750.00700.0229––
Suite runtime seconds6.003.231.0117.48−14.06 [−26.59, −1.54]
Behavioral coverage0.8410.8340.7200.5390.260 [0.194, 0.326]
Edge-case coverage0.8150.7230.5330.4310.259 [0.199, 0.320]
Behaviors per test0.0620.1400.4180.745−0.538 [−1.244, 0.168]
Edge cases per test0.1510.2970.7611.316−0.913 [−1.919, 0.093]

Table 9. Large projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.

Prompts

Listing B.1: Test suite generation prompt (C++).

Write a unit test suite for this project.

The production code is under ‘src/‘. Put your tests under
‘tests/‘. Do not change anything under ‘src/‘: the
task is to test the code as it is, and an edit there invalidates the run.

‘tests/‘ holds one file, ‘tests/CMakeLists.txt‘, and it belongs to
the harness rather than to you: the build system names it, so deleting it or
writing tests into it stops the project building. Leave it exactly as it is and
add your tests in new files beside it.

Every ‘.cpp‘ file you add under ‘tests/‘ is compiled into its own test
executable, and every ‘TEST(...)‘ in it is registered with CTest for you,
including in a subdirectory you create. You do not need to add anything to
‘tests/CMakeLists.txt‘. GoogleTest and GoogleMock are available and
linked for you, including their ‘main‘, so write ‘TEST(...)‘ and do not define
‘main‘ yourself.

Dependencies are already installed, and the project already builds. There is no
network: nothing can be installed, fetched, or looked up, so write the suite
from the code in front of you. Run your test suite with:

    cmake --build build && HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= \
    ctest --test-dir build --no-tests=error

Write the suite a maintainer would accept. Cover the public behavior of each
module, including error paths and edge cases, and make every assertion
one that would fail if the behavior it describes changed. Do not write tests
that assert nothing, or that only assert a call did not raise.

Stop when you have finished writing your tests and the whole suite passes.

Listing B.2: Transpilation task prompt.

This directory contains a rust project whose source was translated from
the project in ‘source_proj/‘. Translate each test from ‘source_proj/‘
into rust.

The production code is under ‘src/‘. Put your tests under
‘tests/‘. Do not change anything under ‘src/‘ or
‘source_proj/‘: the task is to test the code as it is, and an edit there
invalidates the run.

‘tests/‘ is empty.

Cargo builds an integration test from ‘tests/<name>.rs‘ and from
‘tests/<dir>/main.rs‘, and from nowhere else. A file in a
subdirectory is a module, not a target, so a test written to
‘tests/unit/parser.rs‘ is never compiled and never runs unless a
‘main.rs‘ beside it declares it.

Dependencies are already installed, and the project already builds. There is no
network: nothing can be installed, fetched, or looked up, so write the suite
from the code in front of you. Run your test suite with:

    HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= \
    cargo nextest run --workspace

Write the suite a maintainer would accept. Reproduce the behavior of each test
in ‘source_proj/‘ as faithfully as you can. You may skip any tests whose
translation is impossible because the translated rust implementation
differs from the original behavior. Do not write tests that assert
nothing, or that only assert a call did not raise.

Stop when you have translated each test into rust and the whole suite
passes.

Listing B.3: Failure-aware instructions used in the treatment prompt.

Use this workflow:

1. Inventory all production source files and public entry points before choosing what to test. Do not limit inspection to a few representative modules.

2. For each selected behavior, construct inputs that actually reach the intended path. If setup is difficult, inspect the implementation and dependencies and repair the setup rather than silently replacing the behavior with an easier one.

3. Derive expected results from an explicit contract, invariant, or independent calculation. Do not change an expected value solely to match observed output. If implementation behavior conflicts with an explicit contract, do not encode that behavior as correct merely to make the suite pass.

4. Once the suite is green, measure source coverage with:

<language-specific coverage command>

Inspect the file-level report and missing lines or regions, then add meaningful tests for important uncovered behavior. Rerun both the suite and coverage after revisions. Passing tests alone are not evidence that the suite is complete.

Stop when the whole suite passes and you have used the coverage report to check for important behavior that remains untested.

Trajectory evidence

The examples below illustrate the findings using runs from the 675 GPT trajectories analyzed above. Each example names the project and identifies which of its three runs is shown. We reproduce the relevant agent statements, commands, execution output, source code, and generated tests so that readers can follow the evidence for each finding. Event identifiers locate the excerpts within the recorded run.

The excerpts preserve the recorded content. Long lines are wrapped to fit the page, and test-code excerpts omit the leading + symbols that marked added lines in the recorded diff.

Unread implementation: minisketch (repetition C)

The recorded source reads cover the public interface, dispatch, shared sketch algorithms, field configuration, and utility code. The complete command inventory contains no read of the eight generic field-backend implementation bodies. Their relevance is established by the recorded build configuration and the final tests, which exercise all eight storage widths, rather than inferred from a low metric. This supports a finding of omitted source inspection; it does not establish that these files were wholly untested.

Listing C.1: Minisketch’s first implementation-reading command.

/bin/sh -lc "sed -n ’1,260p’ src/include/minisketch.h && sed -n ’1,320p’ src/minisketch.cpp && sed -n ’1,280p’ src/sketch.h && sed -n ’1,360p’ src/sketch_impl.h"
EventOther recorded source reads (analyst summary)
item_3Further public-header and dispatcher reads; false_positives.h; project, source, and test CMake files.
item_6Remaining shared sketch code; fielddefines.h, util.h, and int_utils.h.
item_7Public header read from HEAD.

Table 10. Additional source inspection in the minisketch trajectory, summarized from the recorded commands.

Minisketch: final-test evidence (continued)

Listing C.2: Captured build wiring includes all eight generic field-backend files.

add_library(minisketch_field_sources INTERFACE)
target_sources(minisketch_field_sources
  INTERFACE
    fields/generic_1byte.cpp
    fields/generic_2bytes.cpp
    fields/generic_3bytes.cpp
    fields/generic_4bytes.cpp
    fields/generic_5bytes.cpp
    fields/generic_6bytes.cpp
    fields/generic_7bytes.cpp
    fields/generic_8bytes.cpp
)

Listing C.3: The final minisketch tests exercise the corresponding storage widths.

TEST(Decoding, WorksAcrossEveryFieldStorageWidth)
{
    const uint32_t representative_bits[] = {8, 16, 24, 32, 40, 48, 56, 64};
    for (uint32_t bits : representative_bits) {
        for (uint32_t implementation = 0;
             implementation <= minisketch_implementation_max(); ++implementation) {
            if (!minisketch_implementation_supported(bits, implementation)) continue;
            auto sketch = MakeSketch(bits, implementation, 3);
            ASSERT_NE(nullptr, sketch) << bits << "/" << implementation;
            const uint64_t high = bits == 64 ? (uint64_t{1} << 63)
                                             : (uint64_t{1} << (bits - 1));
            minisketch_add_uint64(sketch.get(), 1);
            minisketch_add_uint64(sketch.get(), high);
            minisketch_add_uint64(sketch.get(), high + 7);
            EXPECT_EQ(std::vector<uint64_t>({1, high, high + 7}),
                      Decode(sketch.get(), 3))
                << bits << "/" << implementation;
        }
    }
}

The source also links minisketch_field_sources into the minisketch library. Conditional CLMUL backends were disabled in this build; the finding concerns the active generic backends and does not depend on CLMUL. We checked the complete inventory of recorded reads to establish the omission; Listing C.1 shows one selected command.

Retained area-formula error: geometer (repetition B)

The agent explicitly identifies the circle-area formula as erroneous. The final test nevertheless requires that formula’s result. For radius 2, the mathematical area is 4π4\pi, while the retained assertion requires 8π8\pi.

Listing C.4: Recorded recognition of the circle-area error.

There’s this known issue with the calculation of a circle’s area -- it returns 2πr² instead of the
    expected πr². I should confirm this in the public behavior documentation and ensure it passes quality
    checks.

Listing C.5: The final assertion retains the recognized erroneous result.

def test_circle_lie_coordinates_intersection_angle_area_and_validation():
    unit = CircularCurve()
    np.testing.assert_allclose(unit.lie_coordinates, [0, 1, 0, 0])
    assert unit.intersection_angle(unit) == pytest.approx(0)
    assert CircularCurve(radius=2).area == pytest.approx(8 * np.pi)

The final run reports 104 passing tests (item_51), but the area assertion does not establish the intended formula. The task required a passing suite and prohibited edits to production source code. Those constraints provide context for the choice without changing what the retained assertion requires.

Weakened missing-input validation: pyflwdir (repetition C)

The initial test expects a ValueError for missing inputs, and the captured implementation contains the corresponding guard. Execution instead fails later during arithmetic on arrays containing None. The agent changes the final test to require that incidental TypeError.

Listing C.6: The original assertion requires the missing-input validation error.

with pytest.raises(ValueError, match='"zs" and "rivdst"'):
>       flw.river_depth(q, width)

Listing C.7: The captured source contains the intended validation guard.

if rivslp is None:
    if zs is None or rivdst is None:
        raise ValueError('"zs" and "rivdst" are required if "rivslp" is None.')
    dz = zs - self.downstream(zs)

Listing C.8: The observed exception is an incidental subtraction error.

E       TypeError: unsupported operand type(s) for -: 'NoneType' and 'NoneType'

src/pyflwdir/flwdir.py:780: TypeError

Listing C.9: The final test accepts the incidental error instead of requiring the validation guard.

def test_river_depth_manning_and_argument_errors(flw):
    q = np.full(flw.shape, 100.0)
    width = np.full(flw.shape, 10.0)
    slope = np.full(flw.shape, 0.001)
    depth = flw.river_depth(q, width, rivslp=slope, manning=0.03)
    expected = ((0.03 * 100) / (np.sqrt(0.001) * 10)) ** (3 / 5)
    np.testing.assert_allclose(depth, expected)
    with pytest.raises(ValueError, match="Method unknown"):
        flw.river_depth(q, width, rivslp=slope, method="other")
    with pytest.raises(TypeError, match="unsupported operand"):
        flw.river_depth(q, width)

In this trajectory, accepting the output from malformed input also weakens the intended validation oracle. The finding rests on the change in the assertion and does not require an unrecorded admission of a bug. The source guard supports the intended exception; the docstring describes the required inputs but does not independently specify an exception class. The final suite reports 59 passing tests (item_47).

Execution without coverage measurement: geometer (repetition C)

This run is a different repetition from the oracle example above. The record contains four completed suite executions using the same command. The number of failures falls until the suite passes. Review of the complete trajectory identifies no coverage-measurement attempt or inherited instrumentation.

Listing C.10: The command used for each of the four geometer suite executions.

/bin/sh -lc "HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= PYTHONPATH=/work/src
    PYTHONDONTWRITEBYTECODE=1 python -m pytest tests -p no:cacheprovider -o addopts= -o log_file= -o
    python_files=test*.py *_test.py *_tests.py’ --continue-on-collection-errors"

Listing C.11: The final recorded pytest summary.

============================== 93 passed in 0.39s ==============================

The initial file inventory lists coverage executables, but listing an executable is not a measurement attempt. To assess whether coverage was measured, we reviewed all recorded commands, configuration, and instrumentation; the selected pytest commands alone would not establish its absence. This example documents the feedback process without establishing weak final coverage or a causal effect on test quality.

Stated familiarity

In Listings C.12–C.17, the agent describes familiarity with the libraries while planning tests. These statements document claimed familiarity; they do not independently establish prior knowledge. We do not assess a relationship between these statements and test quality here.

In Cleo C, betfairlightweight B, pycasbin B, and PyJWT B and C, the statements precede the first recorded read of implementation or test contents. The preceding commands inspect file inventories, repository metadata, or project configuration. The betfairlightweight, pycasbin, and PyJWT B excerpts specifically claim knowledge of upstream tests, whereas Cleo C and PyJWT C describe familiarity with the project.

Listing C.12: Stated familiarity with Cleo, repetition C.

It seems like Cleo is an open-source project I’m familiar with.

Listing C.13: Stated familiarity with upstream tests in betfairlightweight, repetition B, before implementation or test-content inspection.

I’m familiar with some tests from the betfairlightweight GitHub repository, but there’s no network access.

Listing C.14: Stated knowledge of pycasbin tests, repetition B, before implementation or test-content inspection.

I might also leverage what I know about pycasbin tests.

Listing C.15: Stated knowledge of upstream tests in PyJWT, repetition B, before implementation or test-content inspection.

I could leverage what I know from the upstream tests.

Listing C.16: Stated existing familiarity with PyJWT, repetition C, before implementation or test-content inspection.

It looks like I could use the PyJWT version 2.13 since I have some existing familiarity with it.

The more-itertools C statement below follows source inspection. It documents a claim of familiarity, but the trace does not establish whether that familiarity predates the run.

Listing C.17: Stated familiarity and plans to draw on that knowledge in more-itertools, repetition C.

I’ll inspect implementations in manageable chunks and leverage the library I’m familiar with. Writing tests based on this knowledge will be critical.

Formulas Used

SymbolMeaning
UUProduction lines we measure.
Cover⁡(P)\operatorname{Cover}(P)Those lines that passing tests execute.
K,VK, VKilled mutants and viable mutants.
RRViable mutants the suite actually reaches.
n,nlocn, n_{\mathrm{loc}}Tests written, and test lines written.
TtT_tLines test tt covers, after removing shared setup.
A,SensA, \mathrm{Sens}Tests that pass on the original code; and the subset of those that then fail, hang, or crash against the stub.
B,E\mathcal{B}, \mathcal{E}Catalog behaviors and edge cases.

Table 13. Notation.

MetricFormula
Line coverage ↑\uparrow∣Cover⁡(P)∣/∣U∣\lvert\operatorname{Cover}(P)\rvert/\lvert U\rvert
Mutation score ↑\uparrow∣K∣/∣V∣\lvert K\rvert/\lvert V\rvert
Kill rate (reached) ↑\uparrow∣K∣/∣R∣\lvert K\rvert/\lvert R\rvert
Mutants not reached ↓\downarrow1−∣R∣/∣V∣1-\lvert R\rvert/\lvert V\rvert
Tests written −-nn
Test lines −-nlocn_{\mathrm{loc}}
Assertion density ↑\uparrowassertions /nloc/n_{\mathrm{loc}}
Stub sensitivity ↑\uparrow∣Sens∣/∣A∣\lvert\mathrm{Sens}\rvert/\lvert A\rvert
Redundancy ↓\downarrowmean overlap of TtT_t across pairs of tests
Vacuous-test rate ↓\downarrow∣A∖Sens∣/∣A∣\lvert A\setminus\mathrm{Sens}\rvert/\lvert A\rvert
Invalid-test rate ↓\downarrowtests that do not pass / all tests
Kills per added test ↑\uparrownew kills /n/n
Kills per added line ↑\uparrownew kills /nloc/n_{\mathrm{loc}}
Faults caught per test ↑\uparrow∣K∣/n\lvert K\rvert/n
Behavioral coverage ↑\uparrowbehaviors hit /∣B∣/\lvert\mathcal{B}\rvert
Edge case coverage ↑\uparrowedge cases hit /∣E∣/\lvert\mathcal{E}\rvert
Behaviors per test ↑\uparrowbehaviors hit /n/n
Edge cases per test ↑\uparrowedge cases hit /n/n
Generation time ↓\downarrowwall-clock seconds
Tool calls ↓\downarrowreported tool-call count
Output tokens ↓\downarrowreported completion tokens
Cost (USD) ↓\downarrowreported dollar cost
Mutation per minute ↑\uparrowmutation score / generation minutes
Mutation per tool call ↑\uparrowmutation score / tool calls

Table 14. Reported metrics. ↑\uparrow higher is better; ↓\downarrow lower is better.

Failure aware prompt experiment

Compute Used

All experiments were run on 7 AWS EC2 c8a.16xlarge CPU instances for approximately 5 days, corresponding to roughly 840 instance-hours of compute. Each instance provided 64 AMD x86-64 CPU cores (64 vCPUs, single-threaded with no SMT) and 128 GiB of memory, for a maximum concurrent allocation of 448 CPU cores and 896 GiB of memory. No GPUs were used.

Data Provenance

ProjectRepositoryCommitLicenseCitation
C++ projects
AHRSwgrand/AHRSa9f989d07bf1acc75774f5e595c2652eed9385f9MITWilliam Grand. AHRS. https://github.com/wgrand/AHRS (commit a9f989d07bf1acc75774f5e595c2652eed9385f9).
artificial-neural-networkcamchambers/artificial-neural-networkf74dc79cda62ff7f049488cb3cf00c25fdf63e0eMITCam. artificial-neural-network. https://github.com/camchambers/artificial-neural-network (commit f74dc79cda62ff7f049488cb3cf00c25fdf63e0e).
base64tobiaslocker/base648d96a2a737ac1396304b1de289beb3a5ea0cb752MITTobias Locker. base64. https://github.com/tobiaslocker/base64 (commit 8d96a2a737ac1396304b1de289beb3a5ea0cb752).
biscuitlioncash/biscuit805c1756dd602e50bd4f57e772998ab3ad91df7fMITLioncash/Lioncache. biscuit. https://github.com/lioncash/biscuit (commit 805c1756dd602e50bd4f57e772998ab3ad91df7f).
Bit-ManipulationZhuagenborn/Bit-Manipulation365848f2bc366dafc2e9d008bea69322bd3d186fMITZhenshuo Chen and Guowen Liu. Bit Manipulation. 2024. https://github.com/Zhuagenborn/Bit-Manipulation (commit 365848f2bc366dafc2e9d008bea69322bd3d186f).
blazecsvjuanchristensen/blazecsv021eaa6841f202434884ab971e4a376700ea98c5MITBlazeCSV Contributors. blazecsv. https://github.com/juanchristensen/blazecsv (commit 021eaa6841f202434884ab971e4a376700ea98c5).
boolean-parserKronuz/boolean-parser018594354b11e199843e3de33e7055ddc70b29fdMITDubalu LLC. boolean-parser. https://github.com/Kronuz/boolean-parser (commit 018594354b11e199843e3de33e7055ddc70b29fd).
Cachemarcizhu/Cache5b983fd2e793b38275ac6555b8893f876df60e61MITMarc Izquierdo. Cache. https://github.com/marcizhu/Cache (commit 5b983fd2e793b38275ac6555b8893f876df60e61).
cacheflowAnjalihere/cacheflow102b7bde6bb715feae5d661a61971cf182759d06MITCacheFlow contributors. cacheflow. https://github.com/Anjalihere/cacheflow (commit 102b7bde6bb715feae5d661a61971cf182759d06).
cinisuconbu/cini4655ca8aefba2d455fa1d634274fe6894a275a14MITD.Miwa. cini. https://github.com/suconbu/cini (commit 4655ca8aefba2d455fa1d634274fe6894a275a14).
clippmuellan/clipp2c32b2f1f7cc530b1ec1f62c92f698643bb368dbMITAndre Muller. clipp. https://github.com/muellan/clipp (commit 2c32b2f1f7cc530b1ec1f62c92f698643bb368db).
Colorful_IOvaeleborne/Colorful_IOe4dfe37e52c7f0cb9e9250e11fe06f0857a57c9bMITDylan Morgan. Colorful_IO. https://github.com/vaeleborne/Colorful_IO (commit e4dfe37e52c7f0cb9e9250e11fe06f0857a57c9b).
cpp-algorithmsbreadtitor/cpp-algorithmsaf8c610c47 5a31ffff4c dbec171805 457faa16deMITQiaoxi Guo. cpp-algorithms. https://github.com/breadtitor/cpp-algorithms (commit af8c610c475a31ffff4cdbec171805457faa16de).
cpp-algorithms-collection0baidUllah-10/cpp-algorithms-collection77765561faa03bb2aacae8e68d27ef2e1f49c0d9MITObaid Ullah. cpp-algorithms-collection. https://github.com/ObaidUllah-10/cpp-algorithms-collection (commit 77765561faa03bb2aacae8e68d27ef2e1f49c0d9).
cpp-projectsadonayjunior/cpp-projects1be17024b46d691728d645bc3a97a91f834e8e1dMITAdonay Junior. cpp-projects. https://github.com/adonayjunior/cpp-projects (commit 1be17024b46d691728d645bc3a97a91f834e8e1d).
cpp-ringbufferDevTuwkan4ik/cpp-ringbuffer7dacb7797b5658c0c40b45124614120c52ae44aaMITTuwkan4ik. cpp-ringbuffer. https://github.com/DevTuwkan4ik/cpp-ringbuffer (commit 7dacb7797b5658c0c40b45124614120c52ae44aa).
cpp-work-stealing-schedulerzenith19/cpp-work-stealing-scheduler8fd0a187f2a6fa2cd0df16e41498acMITZenit Bushwas. cpp-work-stealing-scheduler. https://github.com/zenith19/cpp-work-stealing-scheduler (commit 80e60921218fd0a187f2a6fa2cd0df16e41498ac).
cpp_chess_botAgentMax05/cpp_chess_botdf10d5d329823197494ada3f9d1c25376240aea9MITAgentMax05. cpp_chess_bot. https://github.com/AgentMax05/cpp_chess_bot (commit df10d5d329823197494ada3f9d1c25376240aea9).
ProjectRepositoryCommitLicenseCitation
cpu_featuresgoogle/cpu_features5514d935a6ff2d10dc2aa07c2fb61db677ab4033Apache-2.0The Android Open Source Project. cpu_features. https://github.com/google/cpu_features (commit 5514d935a6ff2d10dc2aa07c2fb61db677ab4033).
csvGaspardKirira/csv2c5211a7dc9bcf7233ac4532671d9d4d1e389f16MITG-kirira. csv. https://github.com/GaspardKirira/csv (commit 2c5211a7dc9bcf7233ac4532671d9d4d1e389f16).
CustomerSlip-CLISagarBiswas-MultiHAT/CustomerSlip-CLI08b49bbcae20d99d72ed4f6c793f63c193d0a6a2MITSagar Biswas. CustomerSlip-CLI. https://github.com/SagarBiswas-MultiHAT/CustomerSlip-CLI (commit 08b49bbcae20d99d72ed4f6c793f63c193d0a6a2).
CxxUrlchmike/CxxUrle81b86e8779dcd393ed791c1928690e57a83c544MITChristophe Meessen. CxxUrl. https://github.com/chmike/CxxUrl (commit e81b86e8779dcd393ed791c1928690e57a83c544).
DataStructuresAnd-AlgorithmsGiorgiBeruashvili4/DataStructuresAndAlgorithmsa1532217cd26ff26dbad42ea3050f3be1f929a24MITGiorgi Beruashvili. DataStructuresAndAlgorithms. https://github.com/GiorgiBeruashvili4/DataStructuresAndAlgorithms (commit a1532217cd26ff26dbad42ea3050f3be1f929a24).
DB25-sql-tokenizerspace-rf-org/DB25-sql-tokenizerf0100dc949e5f5052dc7e83012058a9757aca74eMITChiradip Mandal, Space-RF.org. DB25-sql-tokenizer. https://github.com/space-rf-org/DB25-sql-tokenizer (commit f0100dc949e5f5052dc7e83012058a9757aca74e).
delaunay32morishuz/delaunay3216157f4a093bf177a417fc472a80291106e930ebMITDelaunay32 contributors. delaunay32. https://github.com/morishuz/delaunay32 (commit 16157f4a093bf177a417fc472a80291106e930eb).
diffGaspardKirira/diffcd6a3c5410448e526e3fe664acc60e496612cbc4MITG-kirira. diff. https://github.com/GaspardKirira/diff (commit cd6a3c5410448e526e3fe664acc60e496612cbc4).
ecsmartinstarkov/ecsc0769a2e5db91045cbb3a3c688fb5c6383b13b94MITMartin Starkov. ecs. https://github.com/martinstarkov/ecs (commit c0769a2e5db91045cbb3a3c688fb5c6383b13b94).
edlibMartinsos/edlib0ddc23ea06abd95ab6227442acb4c27bd7607b68MITMartin Sosic. edlib. https://github.com/Martinsos/edlib (commit 0ddc23ea06abd95ab6227442acb4c27bd7607b68).
ef-containerserik-fryscok/ef-containerse4d63123874a66035ff098c3277ddc264aa5a9ceMITErik Fryscok. ef-containers. https://github.com/erik-fryscok/ef-containers (commit e4d63123874a66035ff098c3277ddc264aa5a9ce).
emailGaspardKirira/email4dae7b2ed6d27e0da5d8e4b554805b807092299eMITG-kirira. email. https://github.com/GaspardKirira/email (commit 4dae7b2ed6d27e0da5d8e4b554805b807092299e).
filesystemgulrak/filesystem2b4d7f239e0e749315fca1de7b2cf7438cf94a9aMITSteffen Schumann. filesystem. https://github.com/gulrak/filesystem (commit 2b4d7f239e0e749315fca1de7b2cf7438cf94a9a).
formatdacap/format69d87625342f498f33796077e64695dc975b365dMITDavid Capelo. format. https://github.com/dacap/format (commit 69d87625342f498f33796077e64695dc975b365d).
GraphicsEngineAdrianParry-17/GraphicsEngine30064df5bc5b676e65e5798411c1404c6cc9b6b3MITAdrian Parry. GraphicsEngine. https://github.com/AdrianParry-17/GraphicsEngine (commit 30064df5bc5b676e65e5798411c1404c6cc9b6b3).
HammingArchiverW-y-l-t-/HammingArchiver3e8ddf65953e8fb1429f9a143b67970b96f8baeMITDmitry Semenov. HammingArchiver. https://github.com/W-y-l-t-/HammingArchiver (commit 3e8ddf65953e8fb1429f9a143b67970b96f8bae).
hexGaspardKirira/hex74d278d8ead559ab64ded97bd2a4b5b5e14ee1730MITG-kirira. hex. https://github.com/GaspardKirira/hex (commit 74d278d8ead559ab64ded97bd2a4b5b5e14ee1730).
hooda-langttema4/hooda-lang85da03687d57536613673e614e791bae2a76ed66MITHooDA Language Contributors. hooda-lang. https://github.com/ttema4/hooda-lang (commit 85da03687d57536613673e614e791bae2a76ed66).
iirlberndporr/iirlbb21b64ea2daa9eda92e57763e21c0e322411f34MITBernd Porr. iirl. https://github.com/berndporr/iirl (commit bb21b64ea2daa9eda92e57763e21c0e322411f34).

Table 1 (PDF p. 28).

ProjectRepositoryCommitLicenseCitation
inicppDeguiLiu/inicpp52c061b5df23f3f23880843c2b47b68dc1f6fb06MITinicpp contributors. inicpp. https://github.com/DeguiLiu/inicpp (commit 52c061b5df23f3f23880843c2b47b68dc1f6fb06).
iowahills_dsphayguen/iowahills_dspd0303977b7ad4208b2d63e9b8839f6dc5f1acfe0MIThayati ayguen. iowahills_dsp. https://github.com/hayguen/iowahills_dsp (commit d0303977b7ad4208b2d63e9b8839f6dc5f1acfe0).
ip_utilsGaspardKirora/ip_utilsdb952a8a2c84bf4dfd3de45913d006a77c3bcd8cMITG-kirira. ip_utils. https://github.com/GaspardKirora/ip_utils (commit db952a8a2c84bf4dfd3de45913d006a77c3bcd8c).
jkdsjkomyno/jkds94ffdf3e1f1cf7a413756a7096c749e675ca7b0aMITAlberto Schiabel. jkds. https://github.com/jkomyno/jkds (commit 94ffdf3e1f1cf7a413756a7096c749e675ca7b0a).
json-parseralevarn/json-parserffef66c6ade70b15ccea023db58d060ff62fae97MITMarcus Alevarn. json-parser. https://github.com/alevarn/json-parser (commit ffef66c6ade70b15ccea023db58d060ff62fae97).
knetkibaamor/knete9cbf08caa701f17e172decdd699b4ba45da4136MITKiba Amor. knet. https://github.com/kibaamor/knet (commit e9cbf08caa701f17e172decdd699b4ba45da4136).
limit-order-book-matching-engineAayushChudiwal/limit-order-book-matching-enginec0e9382e77a09382afed02a7265b25d2e96c36e4MITAayush Chudiwal. limit-order-book-matching-engine. https://github.com/AayushChudiwal/limit-order-book-matching-engine (commit c0e9382e77a09382afed02a7265b25d2e96c36e4).
louvreAlessandro-Salerno/louvree293844c0dad3a07b070c02ba9009a345cd6ec87Apache-2.0Alessandro Salerno. louvre. https://github.com/Alessandro-Salerno/louvre (commit e293844c0dad3a07b070c02ba9009a345cd6ec87).
lru-cache-cppgeoggrigori/lru-cache-cpp238d1a0104f4fa5167dd420176aa3c57e469db5dMITGeovana Grigorio. lru-cache-cpp. https://github.com/geoggrigori/lru-cache-cpp (commit 238d1a0104f4fa5167dd420176aa3c57e469db5d).
lugjwtowner/lugd64953c263071aafe66decffd7b7508f79329287MITJesse W. Towner and Bjoern Hoehrmann. lug. https://github.com/jwtowner/lug (commit d64953c263071aafe66decffd7b7508f79329287).
LuminaAbhijitDoesCoding/Lumina323d19576e8a24ca94efc7bfbbe26f94211dd196MITAbhijit Balpande. Lumina. https://github.com/AbhijitDoesCoding/Lumina (commit 323d19576e8a24ca94efc7bfbbe26f94211dd196).
MatrixUtilevrhel/MatrixUtil10115c0ffaa56bd3f56513423c63aa4d6183cefeMITEthan Vrhel. MatrixUtil. https://github.com/evrhel/MatrixUtil (commit 10115c0ffaa56bd3f56513423c63aa4d6183cefe).
mau-mau1kookieh/mau-mau918adb08797d75ff29134ae10eb2aa7f9f6b19c0MITIgor. mau-mau. https://github.com/1kookieh/mau-mau (commit 918adb08797d75ff29134ae10eb2aa7f9f6b19c0).
microECSviktorfejes/microECSa5aad63ca0d878f28dd146bd75f1421bc64907faMITViktor Fejes. microECS. https://github.com/viktorfejes/microECS (commit a5aad63ca0d878f28dd146bd75f1421bc64907fa).
mimRinzii/mim577b1326a3b620ab712345572e7504dfc98343ecMITIan Pike. mim. https://github.com/Rinzii/mim (commit 577b1326a3b620ab712345572e7504dfc98343ec).
MiniDBHenrikGharagyozyan/MiniDB5d173fa0cf096fd3a55c42a8829d4fb268d17bMITHenrik Gharagyozyan. MiniDB. https://github.com/HenrikGharagyozyan/MiniDB (commit 5d173fa0cf096fd3a55c42a8829d4fb268d17b).
minisketchbitcoin-core/minisketch4a179c61e3cbe3ac2b3c027764ce8eb5183155e1MITPieter Wuille, Greg Maxwell, Gleb Naumenko. minisketch. https://github.com/bitcoin-core/minisketch (commit 4a179c61e3cbe3ac2b3c027764ce8eb5183155e1).
miovimpunk/mio8b6b7d878c89e81614d05edca7936de41ccdd2daMITvimpunk. mio. https://github.com/vimpunk/mio (commit 8b6b7d878c89e81614d05edca7936de41ccdd2da).
mpmc_mapliulilittle/mpmc_mape9c04095db85fa7e7266fa5704f1d92138653b8fMITKyou. mpmc_map. https://github.com/liulilittle/mpmc_map (commit e9c04095db85fa7e7266fa5704f1d92138653b8f).
ProjectRepositoryCommitLicenseCitation
mpmc_queueliulilittle/mpmc_queue53825e333a29bb81599f328f2f46fd67e2c05e3cMITKyou. mpmc_queue. https://github.com/liulilittle/mpmc_queue (commit 53825e333a29bb81599f328f2f46fd67e2c05e3c).
mstrkatahiromz/mstr1f97b1e8d2fae9bdb67e0f8200d5b85ecd20668dMITKatayama Hirofumi MZ. mstr. https://github.com/katahiromz/mstr (commit 1f97b1e8d2fae9bdb67e0f8200d5b85ecd20668d).
pamyatVirusAid/pamyat2da9eca35a6ea1056a5d475af7df305c923d836fMITD3F0LT (VirusAid). pamyat. https://github.com/VirusAid/pamyat (commit 2da9eca35a6ea1056a5d475af7df305c923d836f).
PC-ALEAlex-Pennington/PC-ALE913edbd9bf9b436daf40caf54c3b72630e8630d3MITPC-ALE 2.0 Contributors. PC-ALE. https://github.com/Alex-Pennington/PC-ALE (commit 913edbd9bf9b436daf40caf54c3b72630e8630d3).
randomilqvya/random4271e1c82f1c954d128289b86fa068f76e3b9c76MITilqvya (Illia Polishchuk). random. https://github.com/ilqvya/random (commit 4271e1c82f1c954d128289b86fa068f76e3b9c76).
sgp4dnwrnr/sgp4661e057a5d369d5ee424676cf1d69cbead95ff2cApache-2.0Daniel Warner. sgp4. https://github.com/dnwrnr/sgp4 (commit 661e057a5d369d5ee424676cf1d69cbead95ff2c).
simplesongregjesl/simpleson5de8b8138f6f2e5f64a918d92fb37d43dbb255eeMITGregory Eslinger. simpleson. https://github.com/gregjesl/simpleson (commit 5de8b8138f6f2e5f64a918d92fb37d43dbb255ee).
smathslendidev/smathc390ef3761e96626b37485c6ae3ff0ebe073dfc6Apache-2.0SlendiDev. smath. https://github.com/slendidev/smath (commit c390ef3761e96626b37485c6ae3ff0ebe073dfc6).
splitKronuz/split07e75813190490799a0dee9e71ee8aee5aa178eeMITDubalu LLC. split. https://github.com/Kronuz/split (commit 07e75813190490799a0dee9e71ee8aee5aa178ee).
SPSCQueuerigtorp/SPSCQueue1053918dbd251fbff69b24ef27fa5d51c29ec2afMITErik Rigtorp. SPSCQueue. https://github.com/rigtorp/SPSCQueue (commit 1053918dbd251fbff69b24ef27fa5d51c29ec2af).
SQLhskl18/SQL857b2cb90408e5398c1c47d433c67b198f4314adMITHanbin Tan. SQL. https://github.com/hskl18/SQL (commit 857b2cb90408e5398c1c47d433c67b198f4314ad).
Srlnight-shift/Srl2da759a6a2e0430b96cb44ddd927ca9f9e863b37MITAndreas Samoljuk. Srl. https://github.com/night-shift/Srl (commit 2da759a6a2e0430b96cb44ddd927ca9f9e863b37).
statlibirfandemir00/statlib6d936857679c9838952880764ffc986fddadca3eMITstatlib contributors. statlib. https://github.com/irfandemir00/statlib (commit 6d936857679c9838952880764ffc986fddadca3e).
StringPooldanielkrupinski/StringPool4dd2bc2d02322578e1efc387674059b75b5cce87MITDaniel Krupinski. StringPool. https://github.com/danielkrupinski/StringPool (commit 4dd2bc2d02322578e1efc387674059b75b5cce87).
suggestionvigdail/suggestionc2c85bc81d396a5a580d020edeecd0a9a3b5323MITvigdail. suggestion. https://github.com/vigdail/suggestion (commit c2c85bc81d396a5a580d020edeecd0a9a3b5323).
templatingGaspardKirira/templating7507ad09e2e96d9b386f41921908b630575766fMITG-kirira. templating. https://github.com/GaspardKirira/templating (commit 7507ad09e2e96d9b386f41921908b630575766f).
ThreadSafe-Queue_CPPSayedAdil72/ThreadSafeQueue_CPP8e3f77ab6acaecb1845399481b01b9d4760306fMITSayedAdil72. ThreadSafeQueue_CPP. https://github.com/SayedAdil72/ThreadSafeQueue_CPP (commit 8e3f77ab6acaecb1845399481b01b9d4760306f).
urlGaspardKirira/urla33ef106bfd7bdcc4294c7dc4b2294c0e5599cffbMITG-kirira. url. https://github.com/GaspardKirira/url (commit a33ef106bfd7bdcc4294c7dc4b2294c0e5599cffb).
validatorGaspardKirira/validator738651056a0e97e4630d9014d22abb9416ef8e34MITG-kirira. validator. https://github.com/GaspardKirira/validator (commit 738651056a0e97e4630d9014d22abb9416ef8e34).

Table 16 (PDF p. 30).

ProjectRepositoryCommitLicenseCitation
waitfreequeuemarcusspangenberg/waitfreequeue46bd721d246420164de0644c82b659a384350209MITMarcus Spangenberg. waitfreequeue. https://github.com/marcusspangenberg/waitfreequeue (commit 46bd721d246420164de0644c82b659a384350209).
wsjcpp-diff-textwsjcpp/wsjcpp-diff-text495040dabf0a8303c705adf2b4d5bc3b372a9dc7MITwsjcpp. wsjcpp-diff-text. https://github.com/wsjcpp/wsjcpp-diff-text (commit 495040dabf0a8303c705adf2b4d5bc3b372a9dc7).
z80suzukiplan/z80e3926769a790fab0af1c34a5540e317f8d4f0ddcMITYoji Suzuki. z80. https://github.com/suzukiplan/z80 (commit e3926769a790fab0af1c34a5540e317f8d4f0ddc).
zernoVirusAid/zernoc40a912a49fc99caacbbf5cc3efdad00214fed95MITD3F0LT (VirusAid). zerno. https://github.com/VirusAid/zerno (commit c40a912a49fc99caacbbf5cc3efdad00214fed95).
zpp_bitseyalz800/zpp_bits4a57ecd68a481c2cd6e7ee94aaf83282a735fdedMITEyal Z. zpp_bits. https://github.com/eyalz800/zpp_bits (commit 4a57ecd68a481c2cd6e7ee94aaf83282a735fded).
Python projects
argon2-cffihynek/argon2-cffica682ee81aea0294ba3f187731b3236989feb507MITHynek Schlawack and the argon2-cffi contributors. argon2-cffi. https://github.com/hynek/argon2-cffi (commit ca682ee81aea0294ba3f187731b3236989feb507).
astprettyasottile/astpretty0fe8cc6c1cacd4a40fe4f95431f6ecb03035d149MITAnthony Sottile. astpretty. https://github.com/asottile/astpretty (commit 0fe8cc6c1cacd4a40fe4f95431f6ecb03035d149).
astralsffjunkie/astralac23ab5c0c69837d8fa1a5bb184c2d6d125b26b3Apache-2.0Simon Kennedy. astral. https://github.com/sffjunkie/astral (commit ac23ab5c0c69837d8fa1a5bb184c2d6d125b26b3).
base58checkjoeblackwaslike/base58checkeed1e43d395af3c68799540f9323e07147d66b14MITJoseph Black. base58check. https://github.com/joeblackwaslike/base58check (commit eed1e43d395af3c68799540f9323e07147d66b14).
bcodeflying-sheep/bcodea50996aa1741685c2daba6a9b4893692f377695aMITPhil Schaf. bcode. https://github.com/flying-sheep/bcode (commit a50996aa1741685c2daba6a9b4893692f377695a).
beancount-parserLaunchPlatform/beancount-parsere7e9c4b66ce86cfb03421f3290d7b8b14c40f43fMITLaunch Platform LLC. beancount-parser. https://github.com/LaunchPlatform/beancount-parser (commit e7e9c4b66ce86cfb03421f3290d7b8b14c40f43f).
betfairbetcode-org/betfaira8736c033e226a592feb20b7248b0ff93f045a95MITLiam Pauling and [email protected]. betfair. https://github.com/betcode-org/betfair (commit a8736c033e226a592feb20b7248b0ff93f045a95).
biblibaclements/biblibab0e857b9198fe425ec9b02fcc293b5d9fd0c406MITAustin Clements. biblib. https://github.com/aclements/biblib (commit ab0e857b9198fe425ec9b02fcc293b5d9fd0c406).
binaryofek/binary5a924cc7b106d1c1ba338088c61663e1f762d534Apache-2.0Ofek Lev and [email protected]. binary. https://github.com/ofek/binary (commit 5a924cc7b106d1c1ba338088c61663e1f762d534).
bitsetsxflr6/bitsets8393910175 6d8edb57bf f357b765ec 3ab45a84b4MITSebastian Bank. bitsets. https://github.com/xflr6/bitsets (commit 83939101756d8edb57bff357b765ec3ab45a84b4).
cachetoolstkem/cachetools01af8e5b7c e44432b357 e26c7d67eb 7fa055ae72MITThomas Kemmer. cachetools. https://github.com/tkem/cachetools (commit 01af8e5b7ce44432b357e26c7d67eb7fa055ae72).
c algebraashenfad/c algebra476c3e65da 58bf6e0641 3ae588adf5 196d9f7818MITAdam Ashenfelter. c algebra. https://github.com/ashenfad/c algebra (commit 476c3e65da58bff6e6413ae588adf5196d9f7818).
casbin-pycasbinapache/casbin-pycasbinbf5a94be89 9c3eb14e9d 9509904a3b 38d9f2cf71Apache-2.0The Apache Software Foundation. casbin-pycasbin. https://github.com/apache/casbin-pycasbin (commit bf5a94be899c3eb14e9d9509904a3b38d9f2cf71).
ProjectRepositoryCommitLicenseCitation
ChemFormulamolshape/ChemFormulaa4fc868ef09b8e3dbecbd5acf2bd33a155383ea3MITAxel Muller. ChemFormula. https://github.com/molshape/ChemFormula (commit a4fc868ef09b8e3dbecbd5acf2bd33a155383ea3).
cleopython-poetry/cleo8c3f1bbf82d918fc137ad40db65c446e14564aacMITSebastien Eustace. cleo. https://github.com/python-poetry/cleo (commit 8c3f1bbf82d918fc137ad40db65c446e14564aac).
doatools.pymorriswmz/doatools.py9469db201e0418aef6b97583ef54b6fec2769502MITMianzhi Wang. doatools.py. https://github.com/morriswmz/doatools.py (commit 9469db201e0418aef6b97583ef54b6fec2769502). Associated citation note: If you find my code helpful. You are welcomed to cite my papers http://research.wmz.ninja/research.html.
edit-distancebelambert/edit-distance4e23134ed4577a3347df25bd4daf1b5fa5d0ea6Apache-2.0Ben Lambert. edit-distance. https://github.com/belambert/edit-distance (commit 4e23134ed4577a3347df25bd4daf1b5fa5d0ea6).
funcparserlibvlasovskikh/funcparserlib18c0a99dcdb427e35226c74b7cc2617223c8e1faMITAndrey Vlasovskikh. funcparserlib. https://github.com/vlasovskikh/funcparserlib (commit 18c0a99dcdb427e35226c74b7cc2617223c8e1fa).
geometerjan-mue/geometerdcea8db304b52c8da175f2c3127be9f78062ed0bMITJan Muller. geometer. https://github.com/jan-mue/geometer (commit dcea8db304b52c8da175f2c3127be9f78062ed0b).
gpxpytkrajina/gpxpy09fc46b3cad16b5bf49edf8e7ae873794a959620Apache-2.0Tomo Krajina. gpxpy. https://github.com/tkrajina/gpxpy (commit 09fc46b3cad16b5bf49edf8e7ae873794a959620).
heavylightlewisfogden/heavylight35a7fdcea6ff05ea688140b04d7f207c6dd2c729MITLewis Fogden. heavylight. https://github.com/lewisfogden/heavylight (commit 35a7fdcea6ff05ea688140b04d7f207c6dd2c729).
hirolabwoodsend/hirola2199ba4bdee056563618cd6dc3735ebfa55812b9MITBrenainn Woodsend. hirola. https://github.com/bwoodsend/hirola (commit 2199ba4bdee056563618cd6dc3735ebfa55812b9).
hyperboliccduck/hyperbolic9b41d6f798fbff5c1438f047f67c123d99dc69b4MITCasey Duckering. hyperbolic. https://github.com/cduck/hyperbolic (commit 9b41d6f798fbff5c1438f047f67c123d99dc69b4).
immutabledictcorenting/immutabledict8763951343bea968833207b63af0ae6181781392MITCorentin Garcia. immutabledict. https://github.com/corenting/immutabledict (commit 8763951343bea968833207b63af0ae6181781392).
janusaio-libs/janus431248c05c63d24911e66f4f9a6fedc905c39562Apache-2.0Andrew Svetlov and aio-libs team. janus. https://github.com/aio-libs/janus (commit 431248c05c63d24911e66f4f9a6fedc905c39562).
json-fivespyoungtech/json-fivef95e527c712113674c3621d8244c9c7162a0348Apache-2.0Spencer Phillips-Jones. json-five. https://github.com/spyoungtech/json-five (commit f95e527c712113674c3621d8244c9c7162a0348).
json5kittusharsadhwani/json5kit7e592968ba0dec8a7131e822c653fe669a319ee9MITTushar Sadhwani. json5kit. https://github.com/tusharsadhwani/json5kit (commit 7e592968ba0dec8a7131e822c653fe669a319ee9).
kingdontBuLi/kingdon9f763e6f85d5d8dbfaea8881d812c0MITMartin Roelfs. The Willing Kingdon Cliffson Algebra Library. Lecture Notes in Computer Science, 2025.
larklark-parser/lark9bae96161b45a5bf70011c75b1ef41228d3f4caaMITErez Shinan. lark. https://github.com/lark-parser/lark (commit 9bae96161b45a5bf70011c75b1ef41228d3f4caa).
markdown-it-pyexecutablebooks/markdown-it-pybff75edcd7e6ce68f417803361d6e9f1223ad373MITExecutableBookProject. markdown-it-py. https://github.com/executablebooks/markdown-it-py (commit bff75edcd7e6ce68f417803361d6e9f1223ad373).
marshmallowmarshmallow-code/marshmallowc7b559a1fab3a57ca6dba0ab336841c5038a782MITSteven Loria and [email protected]. marshmallow. https://github.com/marshmallow-code/marshmallow (commit c7b559a1fab3a57ca6dba0ab336841c5038a782).
mashumaroFatal1ty/mashumaro0ad3eaed34f09e4a361613b607aa78e9a51d9999Apache-2.0Alexander Tikhonov. mashumaro. https://github.com/Fatal1ty/mashumaro (commit 0ad3eaed34f09e4a361613b607aa78e9a51d9999).

Table 1 (PDF p. 32).

ProjectRepositoryCommitLicenseCitation
mistletoemiyuchina/mistletoe028fdb017b464804c6c706acf04c884b7d36ab93MITMi Yu. mistletoe. https://github.com/miyuchina/mistletoe (commit 028fdb017b464804c6c706acf04c884b7d36ab93).
more-itertoolsmore-itertools/more-itertools2fe1b2eeb9d75f994113fe3ac76d14b6bcd6fb10MITErik Rose. more-itertools. https://github.com/more-itertools/more-itertools (commit 2fe1b2eeb9d75f994113fe3ac76d14b6bcd6fb10).
mwparserfromhellearwig/mwparserfromhell343bfee5f1eb35ae9b9238df5f83d6b9a5638af1MITBen Kurtovic. mwparserfromhell. https://github.com/earwig/mwparserfromhell (commit 343bfee5f1eb35ae9b9238df5f83d6b9a5638af1).
netdiffopenwisp/netdiff5a33cfe5de608d5337c89a9fc740b91b40d2035bMITNinux.org - Wireless Network Community. netdiff. https://github.com/openwisp/netdiff (commit 5a33cfe5de608d5337c89a9fc740b91b40d2035b).
numericalunitssbyrnes321/numericalunitseab32cb572247c9bb11afb284ec329d29ec649d2MITSteven J. Byrnes. numericalunits. https://github.com/sbyrnes321/numericalunits (commit eab32cb572247c9bb11afb284ec329d29ec649d2).
odata-v4-querydaireto/odata-v4-query88239a687af15c5d2f673584cdd3129e88c72d1dMITDairo Mosquera. odata-v4-query. https://github.com/daireto/odata-v4-query (commit 88239a687af15c5d2f673584cdd3129e88c72d1d).
paradictpyrustic/paradictd161c79a5fd6666e1ecaa8c9b3b4f22e83f1d71bMITPyrustic Architect. paradict. https://github.com/pyrustic/paradict (commit d161c79a5fd6666e1ecaa8c9b3b4f22e83f1d71b).
parser1chardj0n3s/parse8059e320eccf40d750843006dd2ef25197bbff74MITRichard Jones. parse. https://github.com/r1chardj0n3s/parse (commit 8059e320eccf40d750843006dd2ef25197bbff74).
parseratordatamade/parseratore9bed77ac75524084b2d8864985857a6c87b3746MITdatamade. parserator. https://github.com/datamade/parserator (commit e9bed77ac75524084b2d8864985857a6c87b3746).
parsypython-parsy/parsy03deafa98a17adc27b1f650241701b2d21902b3eMITJeanine Adkisson. parsy. https://github.com/python-parsy/parsy (commit 03deafa98a17adc27b1f650241701b2d21902b3e).
pdfsyntaxdesgeeko/pdfsyntax475d42babeecdc42ffe978dca45f107fbc4f9a7fMITMartin D. pdfsyntax. https://github.com/desgeeko/pdfsyntax (commit 475d42babeecdc42ffe978dca45f107fbc4f9a7f).
pingparsingthombashi/pingparsing5c9d51d322298c5c0777088af504e044b0896d37MITTsuyoshi Hombashi. pingparsing. https://github.com/thombashi/pingparsing (commit 5c9d51d322298c5c0777088af504e044b0896d37).
py-markdown-tablehvalev/py-markdown-table63b91b3a2a2f30ed03a0c4bf71484081e2abaf40MIThvalev. py-markdown-table. https://github.com/hvalev/py-markdown-table (commit 63b91b3a2a2f30ed03a0c4bf71484081e2abaf40).
pycodemathcybersora9/pycodemath7098891ae8d0d5de7a4b366b499d4e8b6bae85MITcybersora. pycodemath. https://github.com/cybersora9/pycodemath (commit 7098891ae8d0d5de7a4b366b499d4e8b6bae85).
pyflwdirDeltares/pyflwdirbe5deb79b05064fa05028c7802bf7d41563e7c04MITDeltares. pyflwdir. https://github.com/Deltares/pyflwdir (commit be5deb79b05064fa05028c7802bf7d41563e7c04).
pyglsopenlawlibrary/pyglse0fc705e4a36cced330967dc4967bdc3f092dcdApache-2.0Open Law Library. pygls. https://github.com/openlawlibrary/pygls (commit e0fc705e4a36cced330967dc4967bdc3f092dcd).
pyjwtjpadilla/pyjwt7144e4534c34810f4525dc4578a32addd8212cffMITJose Padilla. pyjwt. https://github.com/jpadilla/pyjwt (commit 7144e4534c34810f4525dc4578a32addd8212cff).
pykwalifyGrokzen/pykwalifyba66b488dd0ccf3acac97e42b21254abd2134181MITJohan Andersson. pykwalify. https://github.com/Grokzen/pykwalify (commit ba66b488dd0ccf3acac97e42b21254abd2134181).
pyliftoverkonstantint/pyliftover10b422fa8bb1a683eb2cc1d897472b2d2e1797b1MITKonstantin Tretyakov. pyliftover. https://github.com/konstantint/pyliftover (commit 10b422fa8bb1a683eb2cc1d897472b2d2e1797b1).

Table 1 (PDF p. 34).

ProjectRepositoryCommitLicenseCitation
pynmea2Knio/pynmea2fcd90dccac7909afe2025f6a7512b38b3bf25e83MITTom Flanagan. pynmea2. https://github.com/Knio/pynmea2 (commit fcd90dccac7909afe2025f6a7512b38b3bf25e83).
pyorbitalpytroll/pyorbitalc6a21204dcbf7c5938a8f27f4388364ec414a73bApache-2.0Pytroll. pyorbital. https://github.com/pytroll/pyorbital (commit c6a21204dcbf7c5938a8f27f4388364ec414a73b).
pyprobablesbarrust/pyprobables7671cc8fa38792793f51172ddcd082dc02d69b0fMITTyler Barrus. PyProbables. 2026. https://github.com/barrust/pyprobables (commit 7671cc8fa38792793f51172ddcd082dc02d69b0f, version 0.7.0).
pyproject-metadatapypa/pyproject-metadataa434728f7ea7306e66c79a8913c8fecfa8663dd4MITFilipe Lains. pyproject-metadata. https://github.com/pypa/pyproject-metadata (commit a434728f7ea7306e66c79a8913c8fecfa8663dd4).
python-cwtdajiaji/python-cwt9932ad0fadb359a6a1aa0901a950cfd21388415cMITAjitomi Daisuke. python-cwt. https://github.com/dajiaji/python-cwt (commit 9932ad0fadb359a6a1aa0901a950cfd21388415c).
python-diceborntyping/python-dice742ce6af27ee46c077628a4be5c19e6b151abaccMITSam Clements, 2017 Caleb Johnson. python-dice. https://github.com/borntyping/python-dice (commit 742ce6af27ee46c077628a4be5c19e6b151abacc).
python-pcapngrshk/python-pcapng33e722f6d5cc41154fda56d8dff62e2970078fd5Apache-2.0Sam Santi. python-pcapng. https://github.com/rshk/python-pcapng (commit 33e722f6d5cc41154fda56d8dff62e2970078fd5).
python-unidiffmatiasb/python-unidiff93c68088d7c50557cbf9cd89a1faed36157bf414MITMatias Bordese. python-unidiff. https://github.com/matiasb/python-unidiff (commit 93c68088d7c50557cbf9cd89a1faed36157bf414).
python-utils-cliKebiLab/python-utils-cli8c2443ddd7b1185c49646e058400496d79535fc2MITKebiLab. python-utils-cli. https://github.com/KebiLab/python-utils-cli (commit 8c2443ddd7b1185c49646e058400496d79535fc2).
QuantumCalcuhsr/QuantumCalc22594de804135a6051bd4bc788748e09aa036937MITUHSR. QuantumCalc. https://github.com/uhsr/QuantumCalc (commit 22594de804135a6051bd4bc788748e09aa036937).
razdelnatasha/razdel668dbe191a5cfd94bebf9155e2ffa5f94ff3fe33MITnatasha. razdel. https://github.com/natasha/razdel (commit 668dbe191a5cfd94bebf9155e2ffa5f94ff3fe33).
sacremoseshplt-project/sacremoses53ce2d93d81434dc9d0f5d631fcbfc1280d52e17MITalvations. sacremoses. https://github.com/hplt-project/sacremoses (commit 53ce2d93d81434dc9d0f5d631fcbfc1280d52e17).
simple-ddl-parserxnuinside/simple-ddl-parser08d9f1bab70c683c5d2a6f227d3fefcce2442378MITIuliia Volkova. simple-ddl-parser. https://github.com/xnuinside/simple-ddl-parser (commit 08d9f1bab70c683c5d2a6f227d3fefcce2442378).
simple-file-checksumsashsinha/simple-file-checksum8d1d067b4a26130c6cf9c3c5fa4c1e8d14f5535a5MITSash Sinha. simple-file-checksum. https://github.com/sashsinha/simple-file-checksum (commit 8d1d067b4a26130c6cf9c3c5fa4c1e8d14f5535a5).
sqlitedictpiskvorky/sqlitedict96e81621fd6ab094efdd86e70fd57efe9d40ca12Apache-2.0Radim Rehurek. sqlitedict. https://github.com/piskvorky/sqlitedict (commit 96e81621fd6ab094efdd86e70fd57efe9d40ca12).
srtcdown/srtff96458d16e8e9f28315687fbed2bb23b013c6b0MITChris Down. srt. https://github.com/cdown/srt (commit ff96458d16e8e9f28315687fbed2bb23b013c6b0).
syntokfnl/syntok371e6ca0d4e307271eaf439f912fadefdaa8e9MITFlorian Leitner. syntok. https://github.com/fnl/syntok (commit 371e6ca0d4e307271eaf439f912fad94efdaa8e9).
tokenize-rtasottile/tokenize-rt51d76ae32ce66963b401b2b71e03babba0962301eMITAnthony Sottile. tokenize-rt. https://github.com/asottile/tokenize-rt (commit 51d76ae32ce66963b401b2b71e03baba0962301e).
tokenstreamvberlier/tokenstream02dcf78104d09c5587e086c7094966c614a91d6eMITValentin Berlier. tokenstream. https://github.com/vberlier/tokenstream (commit 02dcf78104d09c5587e086c7094966c614a91d6e).

Table 1 (PDF p. 35).

ProjectRepositoryCommitLicenseCitation
ttoksimonw/ttok9ad9394c6b886d2ca26e53ea27205192325f3744Apache-2.0Simon Willison. ttok. https://github.com/simonw/ttok (commit 9ad9394c6b886d2ca26e53ea27205192325f3744).
ulidahawker/ulid06289583e9de4286b4d80b4ad000d137816502caApache-2.0Andrew R. Hawker. ulid. https://github.com/ahawker/ulid (commit 06289583e9de4286b4d80b4ad000d137816502ca).
uniqpathJulienRabault/uniqpath47ac8a08ff227a2d61ddd1031bafbbc2f3dd6a18MITJulien Rabault. uniqpath. https://github.com/JulienRabault/uniqpath (commit 47ac8a08ff227a2d61ddd1031bafbbc2f3dd6a18).
usaddressdatamade/usaddressaa7699b53a0843fc443f9e87285b88cbd9eaf50aMITThe Atlanta Journal Constitution. usaddress. https://github.com/datamade/usaddress (commit aa7699b53a0843fc443f9e87285b88cbd9eaf50a).
vcfpybihealth/vcfpya9353cd8351d77966b066279e3a92ab579622d04MITManuel Holtgrewe. vcfpy. https://github.com/bihealth/vcfpy (commit a9353cd8351d77966b066279e3a92ab579622d04).
vdfValvePython/vdfd76292623e326fb165fe3bdb684832cdf30959d4MITRossen Georgiev. vdf. https://github.com/ValvePython/vdf (commit d76292623e326fb165fe3bdb684832cdf30959d4).
vondraktom-spalding/vondrake093de4105b085a6e2251b027fa917027e5a6299MITTom Spalding and available under the MIT license:. vondrak. https://github.com/tom-spalding/vondrak (commit e093de4105b085a6e2251b027fa917027e5a6299).
xphylejdidion/xphyle3fb5e097f177536d176de8a47c11b2db1dd2d785MITJohn Didion. xphyle. https://github.com/jdidion/xphyle (commit 3fb5e097f177536d176de8a47c11b2db1dd2d785).
zernikejacopoantonello/zernikee81e48b07d706bf924d7b5098ac190a06a3a7471Apache-2.0Jacopo Antonello. zernike. https://github.com/jacopoantonello/zernike (commit e81e48b07d706bf924d7b5098ac190a06a3a7471).
zipflysandes/zipflyc44bfed9190926b94c7b15c8ee04ceb0f91fbb0aMITCardallot, Inc. zipfly. https://github.com/sandes/zipfly (commit c44bfed9190926b94c7b15c8ee04ceb0f91fbb0a).
Rust projects
atomrust-syndication/atome7f31696e402ae77c5fc1283216546ad167b9af2Apache-2.0rust-syndication. atom. https://github.com/rust-syndication/atom (commit e7f31696e402ae77c5fc1283216546ad167b9af2).
biblatextypst/biblatex66c76abb952f73ee3889dd05f7bdfade16b15787Apache-2.0Martin Haug. biblatex. https://github.com/typst/biblatex (commit 66c76abb952f73ee3889dd05f7bdfade16b15787).
bitpackingquickwit-oss/bitpacking13744e122742b204dfd3037b0f6882650e029abdMITPaul Masurel. bitpacking. https://github.com/quickwit-oss/bitpacking (commit 13744e122742b204dfd3037b0f6882650e029abd).
byteandylokandy/byte369bbbadd1b83612a11a9563a86ac522aa4b768MITAndy Lok. byte. https://github.com/andylokandy/byte (commit 369bbbadd1b83612a11a9563a86ac522aa4b768).
coitreesdcjones/coitrees05a282d6be9923d01120fb6eda45c03051d3bd93MITDaniel C. Jones. coitrees. https://github.com/dcjones/coitrees (commit 05a282d6be9923d01120fb6eda45c03051d3bd93).
coloursumticky/coloursum3e1f4dbbb441139a0a8cfa329bf5dc9f50f5d7a1MITJessica Stokes**. coloursum. https://github.com/ticky/coloursum (commit 3e1f4dbbb441139a0a8cfa329bf5dc9f50f5d7a1).
cronzslayton/croncf2dc1d519246de9d230d3de2b7cd65e30305eApache-2.0Zack Slayton. cron. https://github.com/zslayton/cron (commit cf2dc1d519246de9d230d3de2b7cd65e30305e).
daachorsedaac-tools/daachorsed65e30305ed6ef8abdb641ebbc4b1911e4fd18a0a507be0cffApache-2.0daac-tools. daachorse. https://github.com/daac-tools/daachorse (commit d65e30305ed6ef8abdb641ebbc4b1911e4fd18a0a507be0cff).
ProjectRepositoryCommitLicenseCitation
data-encodingia0/data-encodingd936082cefb89c427c670ba6dfb7c5cb709e8cb5MITJulien Cretin and Google Inc. data-encoding. https://github.com/ia0/data-encoding (commit d936082cefb89c427c670ba6dfb7c5cb709e8cb5).
di-rsNercury/di-rs4e46bc5461b1df378e28b62fff5ceb094e0ad856Apache-2.0Rust DI Developers and Nerijus Arlauskas. di-rs. https://github.com/Nercury/di-rs (commit 4e46bc5461b1df378e28b62fff5ceb094e0ad856).
doip-codecsamp-reston/doip-codec40c996ab58b8834897cbb3aaf432ee6f789225ecMITSamuel Preston. doip-codec. https://github.com/samp-reston/doip-codec (commit 40c996ab58b8834897cbb3aaf432ee6f789225ec).
enumsetLymia/enumsetfde6b36448e50e8904ac7a80cfef4e f7dedfbd51Apache-2.0Rimi Kanokawa. enumset. https://github.com/Lymia/enumset (commit fde6b36448e50e8904ac7a80cfef4ef7dedfbd51).
exmexbertiqwerty/exmex3ef98341d5e7cd484c0c862498f8c19f384dba8eApache-2.0Behrang Shafei. exmex. https://github.com/bertiqwerty/exmex (commit 3ef98341d5e7cd484c0c862498f8c19f384dba8e).
getargsj-tai/getargse55c7b8917bc9afa60c6adf16c26700f581da717MITJasmine Tai. getargs. https://github.com/j-tai/getargs (commit e55c7b8917bc9afa60c6adf16c26700f581da717).
gumdropmurarth/gumdrop31dfd31aeb59aedd774b507c86943c89ee04cb47Apache-2.0Murarth. gumdrop. https://github.com/murarth/gumdrop (commit 31dfd31aeb59aedd774b507c86943c89ee04cb47).
httparseseanmonstar/httparsea0fa552e4e0fa9a5914a736ef3a556748fef709bApache-2.0Sean McArthur. httparse. https://github.com/seanmonstar/httparse (commit a0fa552e4e0fa9a5914a736ef3a556748fef709b).
httpdatepyfisch/httpdate53df52094835dd85aca70e23b9f384abcf843f7fApache-2.0Pyfisch. httpdate. https://github.com/pyfisch/httpdate (commit 53df52094835dd85aca70e23b9f384abcf843f7f).
include-jsondtolnay/include-jsond12b33da705611c2851d55c465f228ce4d47741fApache-2.0David Tolnay. include-json. https://github.com/dtolnay/include-json (commit d12b33da705611c2851d55c465f228ce4d47741f).
interavldomodwyer/interavl3032d86be1d1d438073aaa5d988f1c5d5153cb11Apache-2.0Dom. interavl. https://github.com/domodwyer/interavl (commit 3032d86be1d1d438073aaa5d988f1c5d5153cb11).
intervallumptal/intervallumcdd2d37cf07367c3ccc6ed55743f1bd72e8f8b50Apache-2.0Pierre Talbot. intervallum. https://github.com/ptal/intervallum (commit cdd2d37cf07367c3ccc6ed55743f1bd72e8f8b50).
ip4sumfereidani/ip4sum219b34a32b06908e3b37369eda935b775ba75871MITKhashayar Fereidani. ip4sum. https://github.com/fereidani/ip4sum (commit 219b34a32b06908e3b37369eda935b775ba75871).
itertools-numbluss/itertools-num6a24eeda60e2ced6952c601036028f0b88cad6f9Apache-2.0bluss. itertools-num. https://github.com/bluss/itertools-num (commit 6a24eeda60e2ced6952c601036028f0b88cad6f9).
kd-treeu1roh/kd-treeb478e2a881f2bcca2f16fcac27fc6a6bbef81215MITTerada Yuichiro (u1roh). kd-tree. https://github.com/u1roh/kd-tree (commit b478e2a881f2bcca2f16fcac27fc6a6bbef81215).
linux-perf-datam stange/linux-perf-data6f911d5fcf315610649ea90932ef7cc4cb96414fApache-2.0Markus Stange. linux-perf-data. https://github.com/mstange/linux-perf-data (commit 6f911d5fcf315610649ea90932ef7cc4cb96414f).
literal-escaperrust-lang/literal-escaperccd015ab0b388fb5e1c0c1085307aedd8210c7c8Apache-2.0The Rust Programming Language. literal-escaper. https://github.com/rust-lang/literal-escaper (commit ccd015ab0b388fb5e1c0c1085307aedd8210c7c8).
luhnpacak/luhn4ef876bb6c0314273af455174baaae d26ba42a25MITMichael Baykov. luhn. https://github.com/pacak/luhn (commit 4ef876bb6c60314273af455174baaaed26ba42a25).
lz4_flexPSeitz/lz4_flexf6251928e30af958243a3ef8f22e3071d241543dMITPascal Seitz. lz4_flex. https://github.com/PSeitz/lz4_flex (commit f6251928e30af958243a3ef8f22e3071d241543d).

Table 1 (PDF p. 36).

ProjectRepositoryCommitLicenseCitation
lz4rippaddor/lz4ripec564ab78ba83e22534b0593fd30bc85f93af282MITPascal Seitz and Patrik Wenger. lz4rip. https://github.com/paddor/lz4rip (commit ec564ab78ba83e22534b0593fd30bc85f93af282).
machinerust-bakery/machine4588a2192d5ca9b60f02c9f710b3ff8058384d6bApache-2.0Geoffroy Couprie. machine. https://github.com/rust-bakery/machine (commit 4588a2192d5ca9b60f02c9f710b3ff8058384d6b).
mathcoreNonanti/mathcore78141f35f20a572783fbe60cb4baf5bd0b165358MITBerkant. mathcore. https://github.com/Nonanti/mathcore (commit 78141f35f20a572783fbe60cb4baf5bd0b165358).
minimadCanop/minimad0c84eb561786647be962f8e0ee053da1685aa73dMITDenys Seguret. minimad. https://github.com/Canop/minimad (commit 0c84eb561786647be962f8e0ee053da1685aa73d).
monkey-rshatashiro/monkey-rs2e1370ec9c2fb3711b02e79c4db086864359c892MITHyunje Jun. monkey-rs. https://github.com/hatashiro/monkey-rs (commit 2e1370ec9c2fb3711b02e79c4db086864359c892).
multiset-hashcronokirby/multiset-hashabc86fc5195446fb7df8faba5aac11d55e9bba98MITLucas Meier. multiset-hash. https://github.com/cronokirby/multiset-hash (commit abc86fc5195446fb7df8faba5aac11d55e9bba98).
noditripytide/nodit93caf14ffe23f1b87f30af800a8aa96e8fca1624MITJames Forster. nodit. https://github.com/ripytide/nodit (commit 93caf14ffe23f1b87f30af800a8aa96e8fca1624).
nom-sqlms705/nom-sql23e2b18c600a3537dcc553f7efdda8fd2cd5f20bMITMalte Schwarzkopf. nom-sql. https://github.com/ms705/nom-sql (commit 23e2b18c600a3537dcc553f7efdda8fd2cd5f20b).
num-traitsrust-num/num-traits1166177ec398601d6dd4d88891cf7674f3978888Apache-2.0The Rust Project Developers. num-traits. https://github.com/rust-num/num-traits (commit 1166177ec398601d6dd4d88891cf7674f3978888).
numeric_castNugine/numeric_cast3e9bc428b51b60761443ae2d0883e0f779d6d453MITNugine. numeric_cast. https://github.com/Nugine/numeric_cast (commit 3e9bc428b51b60761443ae2d0883e0f779d6d453).
ordsearchjonhoo/ordsearchadf72d1c648aa91f6649ed4a3dc71568c78a036cApache-2.0Jon Gjengset. ordsearch. https://github.com/jonhoo/ordsearch (commit adf72d1c648aa91f6649ed4a3dc71568c78a036c).
os-releasepop-os/os-release5d42b3083a070c0f132ee9a561f5a78bb90541c5MITSystem76. os-release. https://github.com/pop-os/os-release (commit 5d42b3083a070c0f132ee9a561f5a78bb90541c5).
parse-sizekennytm/parse-size8f2bc5a8635abc3db760f0eb857b77fd78b3d08cMITkennytm. parse-size. https://github.com/kennytm/parse-size (commit 8f2bc5a8635abc3db760f0eb857b77fd78b3d08c).
parsidateparsicore/parsidate8f735de1e38a845e41dd56f0922c30d325750d3bApache-2.0ParsiCore - Rust. parsidate. https://github.com/parsicore/parsidate (commit 8f735de1e38a845e41dd56f0922c30d325750d3b).
pico-argsRazrFalcon/pico-argsee9e334fdd5bd5dac16ec882ec4bced1d6c11bc9MITYevhenii Reizner. pico-args. https://github.com/RazrFalcon/pico-args (commit ee9e334fdd5bd5dac16ec882ec4bced1d6c11bc9).
pomJ-F-Liu/pom0fd011c736ea77b06c62d8af851c7f7140087a719MITJunfeng Liu. pom. https://github.com/J-F-Liu/pom (commit 0fd011c736ea77b06c62d8af851c7f7140087a719).
quartz_nbtRusty-Quartz/quartz_nbtb8faaa4ef7e55b47c4c1bc8045e7ffc0cbc3b1eMITCassy343, maddy makesgames. quartz_nbt. https://github.com/Rusty-Quartz/quartz_nbt (commit b8faaa4ef7e55b47c4c1bc8045e7ffc0cbc3b1e).
range_map_vecmicrosoft/range_map_vecb449acc86d92e3aa6fe946b1ff2576b1dc5b7e6dMITMicrosoft. range_map_vec. https://github.com/microsoft/range_map_vec (commit b449acc86d92e3aa6fe946b1ff2576b1dc5b7e6d).
repair-jsonm1guelfp/repair-json08f14cc37fc0824a965387235d367d82cf7006aeMITMiguel Piedrafita. repair-json. https://github.com/m1guelfp/repair-json (commit 08f14cc37fc0824a965387235d367d82cf7006ae).

Table 1 (PDF p. 38).

ProjectRepositoryCommitLicenseCitation
robustgeorust/robust654f34cb8cdb3ae21bf59ef3472f92652942cd74Apache-2.0The Spade Developers and The GeoRust Developers. robust. https://github.com/georust/robust (commit 654f34cb8cdb3ae21bf59ef3472f92652942cd74).
rssgensebastienrousseau/rssgenc6916ce1418359e4bfa4693e978bac2923831555Apache-2.0Sebastien Rousseau. rssgen. https://github.com/sebastienrousseau/rssgen (commit c6916ce1418359e4bfa4693e978bac2923831555).
rust-algorithmsEbTech/rust-algorithms2073670c540935960849f365e1895f558fa4319dMITAram Ebtekar. rust-algorithms. https://github.com/EbTech/rust-algorithms (commit 2073670c540935960849f365e1895f558fa4319d).
rust-crc32fastsrijs/rust-crc32fastd5c123d00d5a0236b92acc56f984a5a8a5c7baf5Apache-2.0Sam. rust-crc32fast. https://github.com/srijs/rust-crc32fast (commit d5c123d00d5a0236b92acc56f984a5a8a5c7baf5).
rust-elias-fanotomarrell/rust-elias-fano9bf616b75ada7a3bb4c542d7b89213fccd00196dMITTom Arrell. rust-elias-fano. https://github.com/tomarrell/rust-elias-fano (commit 9bf616b75ada7a3bb4c542d7b89213fccd00196d).
rust-fsmeugene-babichenko/rust-fsm6d029fbb60f1499b88124a7a224e94d283655fbdMITYevhenii Babichenko. rust-fsm. https://github.com/eugene-babichenko/rust-fsm (commit 6d029fbb60f1499b88124a7a224e94d283655fbd).
rust-hash-ringmattnenterprise/rust-hash-ringa4180b319b93b2cfddbea728071d5f3f91f976aeMITMatt McCoy. rust-hash-ring. https://github.com/mattnenterprise/rust-hash-ring (commit a4180b319b93b2cfddbea728071d5f3f91f976ae).
rust-lappersstadick/rust-lapperc545d40ac5b232456c37a17221a4bee47c670e88MITSeth M. Stadick. rust-lapper. https://github.com/sstadick/rust-lapper (commit c545d40ac5b232456c37a17221a4bee47c670e88).
rust-line-endingjzombie/rust-line-endinga51774df6b7cbcad941dec8103d6a888060807d4MITJeremy Harris. rust-line-ending. https://github.com/jzombie/rust-line-ending (commit a51774df6b7cbcad941dec8103d6a888060807d4).
rust-pegkevinmehall/rust-peg3453b34e8dcf0482c650f917e60d43ed944a71e2MITKevin Mehall. rust-peg. https://github.com/kevinmehall/rust-peg (commit 3453b34e8dcf0482c650f917e60d43ed944a71e2).
rust-sugiyamapaddison/rust-sugiyama75f31ef9ab7082950ef2f709d3aa05150c0515dfMITpaddison. rust-sugiyama. https://github.com/paddison/rust-sugiyama (commit 75f31ef9ab7082950ef2f709d3aa05150c0515df).
rust-sunrisenathan-osman/rust-sunrise2e7d1fd5d77ca753bd18ca914594ac391ccbe953MITNathan Osman. rust-sunrise. https://github.com/nathan-osman/rust-sunrise (commit 2e7d1fd5d77ca753bd18ca914594ac391ccbe953).
rustc-hashrust-lang/rustc-hash6ced6cb4235295509102d05f8915e0bb0d8cfe54Apache-2.0The Rust Programming Language. rustc-hash. https://github.com/rust-lang/rustc-hash (commit 6ced6cb4235295509102d05f8915e0bb0d8cfe54).
RustFFTawelkie/RustFFTa34d4f15c905a6b94eee0cbdad7ac95b6ff5481bApache-2.0Allen Welkie and Elliott Mahler. RustFFT. https://github.com/awelkie/RustFFT (commit a34d4f15c905a6b94eee0cbdad7ac95b6ff5481b).
semtoolsrun-llama/semtoolsfe7f6d260fdc0128768fcfe a0e8d62MITLlamaIndex. semtools. https://github.com/run-llama/semtools (commit fe7f6d260fdc0128768fcfea0e8d624d6f304739).
semverdtolnay/semver280ebc6edac3aa4cdc545dbff8a26c5ac4861feApache-2.0David Tolnay. semver. https://github.com/dtolnay/semver (commit 280ebc6edac3aa4cdc545dbff8a26c5ac4861fe).
sgmlishmernen/sgmlish2f50fbf82340d8a82e581114f1bd259695ae30deMITDaniel Luz. sgmlish. https://github.com/mernen/sgmlish (commit 2f50fbf82340d8a82e581114f1bd259695ae30de).
smolbitsetserenity-rs/smolbitset894d03c39ae2289bc89ad302726b1cd8b740edeaMITFlorian Gebhardt. smolbitset. https://github.com/serenity-rs/smolbitset (commit 894d03c39ae2289bc89ad302726b1cd8b740edea).
speedatepydantic/speedate6fafc2c60b5cbe91378934a9b1b670fb8fc88102MITSamuel Colvin. speedate. https://github.com/pydantic/speedate (commit 6fafc2c60b5cbe91378934a9b1b670fb8fc88102).
ProjectRepositoryCommitLicenseCitation
srtlibGavalasDev/srtlibcc0d3560057f3cd5ec257bdf4ce0cdd50875bf81Apache-2.0Konstantinos Gavalas. srtlib. https://github.com/GavalasDev/srtlib (commit cc0d3560057f3cd5ec257bdf4ce0cdd50875bf81).
sscanfmich101mich/sscanfa5075444e3aeb9b5b833c12dfa25a2859fce4ac5Apache-2.0Malte Hillmann. sscanf. https://github.com/mich101mich/sscanf (commit a5075444e3aeb9b5b833c12dfa25a2859fce4ac5).
substrait-rssubstrait-io/substrait-rse16e99d9b4d8e0ca25ab27b0e557a99b558a50d3Apache-2.0Substrait. substrait-rs. https://github.com/substrait-io/substrait-rs (commit e16e99d9b4d8e0ca25ab27b0e557a99b558a50d3).
texting_robotsSmerity/texting_robots8e52f3f4126cd88b23833317b4dadd91d3fcd1d3Apache-2.0Stephen Merity. texting_robots. https://github.com/Smerity/texting_robots (commit 8e52f3f4126cd88b23833317b4dadd91d3fcd1d3).
tibitsetbestouff/tibitset1bf65a461ee845244ffd22e5476422fcf246afc80Apache-2.0Xavier Bestel. tibitset. https://github.com/bestouff/tibitset (commit 1bf65a461ee845244ffd22e5476422fcf246afc80).
tinyb36lmammino/tinyb361f4e68cec59164d18c2855da54970a2df490c0daMITLuciano Mammino. tinyb36. https://github.com/lmammino/tinyb36 (commit 1f4e68cec59164d18c2855da54970a2df490c0da).
tinybitsetmoia-oss/tinybitsetac302ca09dc199d8220952374ed4d6d3ec7c4721Apache-2.0MOIA GmbH - Open Source. tinybitset. https://github.com/moia-oss/tinybitset (commit ac302ca09dc199d8220952374ed4d6d3ec7c4721).
tls-parserrusticata/tls-parser6554155918278531370e7d0addbd5d759e3a4cc9Apache-2.0Pierre Chifflier. tls-parser. https://github.com/rusticata/tls-parser (commit 6554155918278531370e7d0addbd5d759e3a4cc9).
tomllibjoelself/tomllib8c2aecc689c5fc3167a687dd6c41eca8a59605d0MITJoel Self. tomllib. https://github.com/joelself/tomllib (commit 8c2aecc689c5fc3167a687dd6c41eca8a59605d0).
ubyteSergioBenitez/ubyted8ff5088d1a68e9ee6c50b302af185904726f507Apache-2.0Sergio Benitez. ubyte. https://github.com/SergioBenitez/ubyte (commit d8ff5088d1a68e9ee6c50b302af185904726f507).
urdf-rsopenrr/urdf-rs9eeb4a4f762c9f5293c87d7fabec259e96267aa4Apache-2.0openrr. urdf-rs. https://github.com/openrr/urdf-rs (commit 9eeb4a4f762c9f5293c87d7fabec259e96267aa4).
vectrixrossmacarthur/vectrix758967efe0215e9b682117f2e81494fe8b4de557Apache-2.0Ross MacArthur. vectrix. https://github.com/rossmacarthur/vectrix (commit 758967efe0215e9b682117f2e81494fe8b4de557).
watermill.rsonline-ml/watermill.rs48e1f819298f4158b4c5eb634d06ae1f485bf57dMITAdil Zouitine. watermill.rs. https://github.com/online-ml/watermill.rs (commit 48e1f819298f4158b4c5eb634d06ae1f485bf57d).
yapjsdw/yap5f791d2ae05c67de624b6c019fa74aed300fda03MITJames Wilson. yap. https://github.com/jsdw/yap (commit 5f791d2ae05c67de624b6c019fa74aed300fda03).
zlatlong-rsdgryski/zlatlong-rsbdb58ae807815bc53b927fea87993e92e5980a07MITDamian Gryski. zlatlong-rs. https://github.com/dgryski/zlatlong-rs (commit bdb58ae807815bc53b927fea87993e92e5980a07).

Table 1 (PDF p. 39).

Statement of LLM Usage

We used LLM-based tools for limited assistance during the preparation of this work. A coding agent was leveraged to assist with aspects of implementation after the experimental design and intended behavior had been specified by the authors. LLMs were also used as editorial aids to review drafts for clarity, concision, and phrasing. All methodological decisions, experimental design, analysis, and scientific conclusions were made by the authors, who reviewed and verified the resulting code and manuscript.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

  • You should answer [Yes], [No], or [N/A].

  • [N/A] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

  • Please provide a short (1–2 sentence) justification right after your answer (even for [N/A]).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will also be asked to include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While [Yes] is generally preferable to [No], it is perfectly acceptable to answer [No] provided a proper justification is given (e.g., error bars are not reported because it would be too computationally expensive” or “we were unable to find the license for the dataset we used”). In general, answering [No] or [N/A] is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

  • Delete this instruction block, but keep the section heading “NeurIPS Paper Checklist”,

  • Keep the checklist subsection headings, questions/answers and guidelines below.

  • Do not modify the questions and only use the provided macros for your answers.

  1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: The abstract and conclusion clearly sum up the 3 primary results in the paper: aggregate results showing coding agents outperform humans while unit testing (but struggle in long-context settings), results showing transpilation can be inconsistent in modern coding agents, and an analyses of traces reveals several test generation failure modes in Codex that may generalize.

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

  1. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See limitations and future work section under conclusion header!

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [N/A]

Justification: No theoretical proofs in the paper.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] Justification: Code and data will be released for the paper, meaning it will be reproducible. Further, beyond the specific repositories used (which will also be released), all other scoring information was provided.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We plan to release code and data

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: There are no training details, but we detail the data we evaluated on.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars and confidence intervals in our primary figure and results table.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide a section detailing the cloud compute we used, including instance and memory. Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: No item of the code of ethics was violated in the writing of this paper.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [N/A]

Justification: The work has no societal impact - it is an evaluation study of existing technology.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: The dataset we plan to release is composed of pre-existing, highly permissively licensed projects that could have easily been used prior. The paper poses no misuse risks.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All papers are credited and license + terms of use are respected. See data provenance section.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [N/A]

Justification: We do not release new data assets.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: The paper does not involve crowdsourcing, nor research human subjects.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: Our paper does not involve any crowdsourcing work, nor research with human subjects.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: We include a declaration of LLM Usage in the Appendix.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [1]Azat Abdullin, Pouria Derakhshanfar, and Annibale Panichella. Test wars: A comparative study of SBST, symbolic execution, and LLM-based approaches to unit test generation. In Proceedings of the 18th IEEE International Conference on Software Testing, Verification and Validation (ICST), pages 221–232, 2025. doi: 10.1109/ICST62969.2025.10989033.
  2. [2]Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. Automated unit test improvement using large language models at meta. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE Companion), pages 185–196, 2024.arxiv.org/abs/2402.09171
  3. [3]Altino Alves, João Eduardo Montandon, and Andre Hora. Testing framework migration with large language models. In Proceedings of the 7th ACM/IEEE International Conference on Automation of Software Test (AST), pages 12–22, 2026. doi: 10.1145/3793654.3793749.DOI
  4. [4]Vekil Bekmyradov, Noah C. Pütz, and Thomas Bartz-Beielstein. LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDB, 2026.arxiv.org/abs/2604.14437
  5. [5]Gaurav Bhatnagar. Modernization of enterprise payment infrastructure: A case study on LLM-assisted migration of legacy distributed systems. Array, 30:100806, 2026. doi: 10.1016/j.array.2026.100806.DOI
  6. [6]Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuigang Deng, and Jianwei Yin. Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE Companion), pages 572–576, 2024.arxiv.org/abs/2305.04764
  7. [7]Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Desmarais. Effective test generation using pre-trained large language models and mutation testing. Information and Software Technology, 171:107468, 2024. doi: 10.1016/j.infsof.2024.107468.DOI
  8. [8]DARPA. TRACTOR: Translating all C to Rust. https://www.darpa.mil/research/programs/translating-all-c-to-rust, 2024.
  9. [9]Amirhossein Deljouyi, Roham Koohestani, Maliheh Izadi, and Andy Zaidman. Leveraging large language models for enhancing the understandability of generated unit tests. In Proceedings of the 47th International Conference on Software Engineering (ICSE), pages 392–404, 2025.arxiv.org/abs/2408.11710
  10. [10]Yi Gao, Xing Hu, Tongtong Xu, Xin Xia, David Lo, and Xiaohu Yang. MUT: Human-in-the-loop unit test migration. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE), pages 229:1–229:12, 2024. doi: 10.1145/3597503.3639124.
  11. [11]Swen E. Gaudl. A case study in re-engineering and porting a legacy codebase using LLM pair-programming: A first look. In Joint Proceedings of the ACM UMAP 2026 Workshops, volume 4206 of CEUR Workshop Proceedings, pages 253–256. CEUR-WS.org, 2026. URL https://ceur-ws.org/Vol-4206/ParaAdapt-3.pdf.
  12. [12]Sijia Gu, Noor Nashid, and Ali Mesbah. LLM test generation via iterative hybrid program analysis. In Proceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE), 2026. doi: 10.1145/3744916.3764553.DOI
  13. [13]Sabaat Haroon, Mohammad Taha Khan, and Muhammad Ali Gulzar. Evaluating LLM-based test generation under software evolution, 2026.arxiv.org/abs/2603.23443
  14. [14]Moavia Hassan, Zhengli Liu, Bangchao Wang, Bing Li, and Jian Wang. Python unit tests generation: A comparative study on Claude Code, Cursor, GitHub Copilot and Amazon Q. SSRN Electronic Journal, 2026. Preprint.DOI
  15. [15]Dong Huang, Jie M. Zhang, Mingzhe Du, Mark Harman, and Heming Cui. Rethinking the influence of source code on test case generation, 2024.
  16. [16]Dong Huang, Jie M. Zhang, Mark Harman, Qianru Zhang, Mingzhe Du, and See-Kiong Ng. Benchmarking LLMs for unit test generation from real-world functions, 2025.arxiv.org/abs/2508.00408
  17. [17]Kush Jain, Gabriel Synnaeve, and Baptiste Rozìere. TestGenEval: A real world unit test generation and test completion benchmark. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.00752
  18. [18]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024. arXiv:2310.06770.arxiv.org/abs/2310.06770
  19. [19]Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In Proceedings of the 45th International Conference on Software Engineering (ICSE), pages 2312–2323, 2023.arxiv.org/abs/2209.11515
  20. [20]Anirudh Khatry, Robert Zhang, Jia Pan, Ziteng Wang, Qiaochu Chen, Greg Durrett, and Isil Dillig. CRUST-Bench: A comprehensive benchmark for C-to-safe-Rust transpilation. arXiv preprint arXiv:2504.15254, 2025.arxiv.org/abs/2504.15254
  21. [21]Michael Konstantinou, Florian Tambon, and Mike Papadakis. On the risk of coding before testing: An empirical study on LLM-based test generation workflow, 2026.arxiv.org/abs/2607.05139
  22. [22]Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. CodaMOSA: Escaping coverage plateaus in test generation with pre-trained large language models. In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE), pages 919–931, 2023. doi: 10.1109/ICSE48619.2023.00085.DOI
  23. [23]Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Aо Tang, Dmytro Pyakhtar, Jiawei Liu, Yuxiang Wei, et al. StarCoder 2 and The Stack v2: The next generation, 2024. arXiv:2402.19173.arxiv.org/abs/2402.19173
  24. [24]Davide Molinelli, Luca Di Grazia, Alberto Martin-Lopez, Michael D. Ernst, and Mauro Pezzè. Do LLMs generate useful test oracles? an empirical study with an unbiased dataset. In Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 278–290, 2025. doi: 10.1109/ASE63991.2025.00031.DOI
  25. [25]Zahra Moti, Heydar Soudani, and Jonck van der Kogel. LegacyTranslate: LLM-based multi-agent method for legacy code translation, 2026.arxiv.org/abs/2603.14054
  26. [26]Niels Müllnder, Mark Niklas Müller, Jingxuan He, and Martin Vechev. SWT-bench: Testing and validating real-world bug-fixes with code agents. In Advances in Neural Information Processing Systems, volume 37, pages 81857–81887, 2024. doi: 10.52202/079017-2601.DOI
  27. [27]Noor Nashid, Mifta Sintaha, and Ali Mesbah. Retrieval-based prompt selection for code-related few-shot learning. In Proceedings of the 45th International Conference on Software Engineering (ICSE), pages 2450–2462, 2023.DOI
  28. [28]Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassі, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. Lost in translation: A study of bugs introduced by large language models while translating code. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE), pages 995–1007, 2024. doi: 10.1145/3597503.3639226.DOI
  29. [29]Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. Aster: Natural and multi-language unit test generation with llms, 2025. arXiv:2409.03093.arxiv.org/abs/2409.03093
  30. [30]Juan Altmayer Pizzorno and Emery D. Berger. Coverup: Coverage-guided llm-based test generation, 2024. arXiv:2403.16218.
  31. [31]Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. Code-aware prompting: A study of coverage-guided test generation in regression setting using llm. Proceedings of the ACM on Software Engineering, 1(FSE), 2024. Article 43.arxiv.org/abs/2402.00097
  32. [32]Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1):85–105, 2024. doi: 10.1109/TSE.2023.3334955.DOI
  33. [33]Mohammed Latif Siddiq, Joanna Cecilia Da Silva Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinícius Carvalho Lopes. Using large language models to generate junit tests: An empirical study. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE), pages 313–322, 2024.DOI
  34. [34]Yuxuan Sun, Yuze Zhao, Yufeng Wang, Yao Du, Zhiyuan Ma, Jinbo Wang, Mengdi Zhang, Kai Zhang, and Zhenya Huang. SWE-mutation: Can LLMs generate reliable test suites in software engineering?, 2026.
  35. [35]Wenhan Wang, Chenyuаn Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. TestEval: Benchmarking large language models for test case generation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3547–3562, 2025. doi: 10.18653/v1/2025.findings-naacl.197.DOI
  36. [36]Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. Hits: High-coverage llm-based unit test generation via method slicing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1258–1268, 2024.arxiv.org/abs/2408.11324
  37. [37]Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, et al. On the evaluation of large language models in unit test generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1607–1619, 2024.arxiv.org/abs/2406.18181
  38. [38]Quanjun Zhang, Ye Shang, Chunrong Fang, Siqi Gu, Jianyi Zhou, and Zhenyu Chen. TestBench: Evaluating class-level test case generation capability of large language models, 2024.arxiv.org/abs/2409.17561

Paper details

vlnc:W_s6mejmzc
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents