betaFeedback

WorksComputer Science

Ephyra: A Property-Based Testing Framework for the LLM Era

Abstract

Coding agents now write a growing share of software tests; however, property-based testing (PBT) libraries are designed with historical considerations. We present Ephyra, a PBT library in Rust built on three concepts: customizability, observability and generator synthesis. Ephyra lets users compose their own testing loops from generation, mutation and shrinking, and synthesize these components to satisfy functional predicates. Applying Ephyra to 14 open source systems, we have found 349 bugs and reported 300 of them, in database management systems (SQLite, Turso, PostgreSQL, DuckDB, DataFusion), the verified policy engine (Cedar), the SQL equivalence prover (QED), the URL parser used in Node.js (ada-url), reimplementations of existing open source systems in Rust (coreutils, brush, gitoxide), and compilers (binaryen, OCaml, and OxCaml).

Introduction

Large language models (LLMs) have significantly affected every phase of the software development life cycle (SDLC). Testing is no exception; the way software tests are read, written, and leveraged has been changing especially due to the growing popularity of spec-driven development [1] and the ability to run CEGIS-style [32] program synthesis loops [2, 33, 41] relying on comprehensive test suites. This abstract addresses two questions—predominantly the latter question—in the midst of these changes:

  1. Should, and how can, we leverage LLMs in property-based testing workflows?

  2. How should the design of PBT libraries change with respect to the changing software engineering practices?

The first question is increasingly investigated in the literature and practice of PBT. The main interest is in using models to write property-based tests [10, 18, 36] with further research on LLM-guided generation [5, 39] and LLM-authored test-case generators [3, 37, 40, 43] and test case shrinkers [44]. The second question, although naturally explored by using PBT libraries through agentic workflows, is currently under-researched. Similar questions have been posed in the context of dependency management [31, 34, 38] and data management systems domains [17]. In the context of PBT, the most significant effort to date is PBT skills [9, 35] for LLMs. The core hypothesis behind the proposed library in § 3 is two-fold. First, we posit that the future of property-based testing is hybrid. We hypothesize that instead of simply sampling random inputs from a constructed distribution, the future PBT workloads will interleave generation, enumeration and mutation depending on the domain and system under test. With this hypothesis, we develop a programmable PBT [12] library that exposes primitives for constructing testing loops.

Second, program synthesis for verifiable domains has shown promising results with the growing capabilities of LLMs [11, 23, 25]. Existing work on generator derivation strategies includes type-based generators [22, 27], specification-based generators [15] derived from inductive specifications, and the randomness-parsing approach for deriving shrinkers and mutators from a given generator [6, 19, 20]. However, these approaches are incompatible with arbitrary functional predicates, and randomness-parsing derives specific shrinkers/mutators that are immutable. Our library includes a synthesis tool that can optimize generators for a given set of metrics while adhering to the validity predicates as well as correct-by-construction shrinker/mutator synthesis that is validated against the generator.

Under these hypotheses, we propose three design principles for our new library, Ephyra1:

  1. Verifiable components should be synthesized.

  2. Customizability should be a first-class concern.

  3. Observability should be a first-class primitive.

Background

This section presents existing literature and design space for the three principles outlined in § 1.

Components in PBT

A component is a PBT abstraction, such as a generator, mutator, shrinker or printer, that defines an interface operating on the input type of the property. The classical property runner defined in QuickCheck [4] can be decomposed into the following property runner diagram:

Abstract representation of QuickCheck runner

Figure 1. Abstract representation of QuickCheck runner taken from Programmable PBT [12].

The implementation of these components is a significant part of the PBT development; the quality of the test case generator has a significant effect on bug finding performance [14], and further so, the design of the libraries themselves is affected by this effort trade-off. Many modern libraries use the “randomness-parsing” [21] approach to test case generation, in which they generate random bytes that are then interpreted into structures, citing the benefit of this approach in automatically producing correct-by-construction shrinkers and mutators [20] without additional effort. Other libraries provide “derivations”, compile-time code generation via macros to produce generators and shrinkers based on a type [22, 27], or in the case of QuickChick, inductive specifications [26]. Derivations are immensely useful in their convenience; derived generators can remove the hurdle of tediously writing and maintaining many small generators that could be trivial to write but time-consuming; derived shrinkers can be as good as the correct-by-construction shrinkers the randomness parsing libraries provide [13]. Most of these derivations, however, are not long term testing strategies but starting points that lower the cost of property-based testing a new system. One notable exception is the derivation of generators that satisfy inductive specifications [15], giving rise to unlimited theoretical complexity, however limited in its applicability to testing programs due to the lack of such inductive specifications in practice. In Ephyra, we close this gap by “synthesizing” generators that satisfy functional specifications like the validity of a SQLite file (PRAGMA integrity_check) while optimizing the generator in a hill-climbing loop with respect to statistics we collect from the system under test.

Customizability in PBT

A property-based testing framework typically provides a language for writing properties, a set of APIs for writing generators, automations for deriving generators and shrinkers from the input types, one or more testing loops with limited configurability for testing the properties by sampling the provided generators, and some observability hooks for measuring statistics computed from the tests. Programmable PBT [12] is a new paradigm for PBT library design that gives the library users the primitives for constructing their own testing loops, a new axis in customization. The original work presents two design choices, (1) a property data structure in Rocq that carries a dependently-typed list of input values, and (2) a property data structure in Racket that carries a dynamically-typed context, both of which allow for inspecting the shape of the property. Leveraging the capability of inspecting the property, the authors then build four APIs—building blocks—for constructing existing property runners in the literature such as below:

Definition runLoop (fuel : nat) (cprop : Prop ∅) :=
  let fix runLoop' (fuel : nat) (cprop : Prop ∅)
        (passed : nat) (discards : nat) : G Result :=
    match fuel with
    | O => ret (mkResult discards false passed [])
    | S fuel' =>
        input <- gen cprop (log2 (passed + discards));;
        res <- run cprop input ;;
        match res with
        | Normal seed false => (* Fails *)
            let shrunk := shrinker 10 cprop seed in
            let printed := print cprop 0 shrunk in
            ret (mkResult discards true (passed + 1) printed)
        | Normal _ true => (* Passes *)
            runLoop' fuel' cprop (passed + 1) discards
        | Discard _ _ => (* Discard *)
            runLoop' fuel' cprop passed (discards + 1)
        end in
  runLoop' fuel cprop 0 0.
QuickCheck-style property runner written in Rocq

Figure 2. QuickCheck-style property runner written in Rocq taken from Programmable PBT [12].

The programmability of Ephyra is limited compared to the original design; we inherit the four building blocks generate, mutate, shrink, and run but keep the property representation opaque. This hybrid design allows for the users to write their own property runners, but not the building blocks themselves. For instance, the users can arbitrarily interleave generation and mutation, but never at the granularity of individual input variables, only for the whole set of input variables to the property. We present the capabilities of the library in § 3.

Observability in PBT

The first question one must ask once they have written a property-based test is: What does the test actually do? The typical mechanisms PBT libraries provide for understanding the generators are (1) sampling a number of inputs from the generators, and (2) computing aggregate statistics via collect callbacks. Tyche [7] and the OpenPBTStats [7] format have advanced the state of PBT observability by turning plain textual histograms computed by the libraries into dynamically rendered graphs while enriching the types of feedback libraries can emit during the testing runs. In Ephyra, observability, particularly through code coverage computed via LLVM coverage counters, is used extensively to guide the evolution of the generator. Generators are not treated as concrete programs that are written, but distributions to be shaped by the reviewers that rely on the observability to understand the underlying tests. We have built prototype interfaces for guiding the generators through the observations collected from the test runs, and we plan to keep developing them in the future.

Ephyra: A Programmable PBT Framework with Synthesizable Components

Ephyra, the ancient Greek city now known as Corinth, is the home of Sisyphus, a man forever cursed to carry a boulder up a hill, only to keep going in a metaphorical Möbius strip, never approaching the end. The name of the library is an homage to the most Sisyphean task in modern software development, testing; particularly random testing where we roll not one, but millions of rocks through a variety of software, and Ephyra is the home of all such efforts. The library follows the programmable PBT tradition of publishing existing property-runners; a simple generational runner, a fuzzing runner and a targeted runner are all within the current implementation. The core innovation, synthesized components, is managed via the accompanying command line interface, and constraints on the generators are denoted via Rust macros as shown in Figure 3.

Definition of a generator for the ada-url case study

Figure 3. Definition of a generator for the ada-url case study.

struct Canonical;
impl Gen for Canonical {
    type Value = String;
    fn generate(&self, g: &mut Source) -> String { ... }
    fn shrink(&self, s: &String, visit: ...) -> ControlFlow<()> { ... }
    fn mutate(&self, s: &String, g: &mut Source) -> String { ... }
}
// Validity predicate: ada parses the string to itself.
fn is_canonical(s: &String) -> bool {
    canonicalize(s).as_deref() == Some(s.as_str())
}
// Diversity metrics to optimize for the generator distributions
fn canonical_metrics(s: &String) -> Vec<(&'static str, f64)> {
    let b = |x: bool| f64::from(u8::from(x)); // true -> 1.0, false -> 0.0
    vec![("chars", s.chars().count() as f64), ("has query", b(s.contains('?'))), ...]
}
#[ephyra::generator(is_canonical, metrics = canonical_metrics)]
fn canonical_urls() -> impl Gen<Value = String> { Canonical }

#[ephyra::test(gen = canonical_urls(), tests = 3000)]
fn canonical_url_is_reference_canonical(url: String) {
    let r = whatwg_url::run(&Case::parse(url.clone())); // reference implementation
    assert!(r.can_parse);
    assert_eq!(r.href(), Some(url.as_str()));
}

A validity predicate is an arbitrary Rust function of type fn(&T) -> bool, generators mark both their validity predicates as well as the set of metrics to be computed over the distribution of the inputs. For each annotated generator, ephyra check performs a full Ephyra run over 10,000 inputs by default and verifies, per input, that (1) the generated value satisfies the predicate; (2) the first 64 shrink candidates differ from the value and satisfy the predicate; (3) repeatedly taking the first shrink candidate stays valid, never revisits a value, and settles within 1000 steps; (4) four mutants stay valid; and (5) every metric is finite. We use the result of these checks to drive the component synthesis for repairing broken generators, mutators and shrinkers as well as improving accepted ones against the set of computed metrics, complemented by code coverage.

Preliminary Experimental Results

We applied Ephyra to fourteen real systems in two modes that differ in the steering of the generator synthesis. Appendix § 7 lists every bug with its upstream report. In the open-loop style, we have run interactive coding-agent sessions that write Ephyra generators, predicates and properties for one target, tune the generators against a metric, run campaigns, and triage what they find. In the closed-loop style, the steering judgement is left up to ephyra-agent with automated patches within the loop to strengthen the ability to find new bugs.

Open-Loop Case Studies

We ran open-loop studies on six open-source systems: SQLite (63), Turso (72), PostgreSQL (16), ada-url (12), Cedar (2), and QED (2); 167 confirmed bugs in total. We have used classic properties from the literature such as the metamorphic database testing oracles [28 b], simpler model implementations for model-based testing and differential testing against reference implementations. The ability to generate against arbitrary predicates has given us the opportunity to experiment with new styles of generators, such as directly generating SQLite files from scratch as opposed to the classic approach of running multi-round stateful tests against the database.

Closed-Loop Campaign

The closed-loop experiment ran ephyra-agent (Figure 4b) on five Rust and C++ systems in parallel, for about eight hours, one 16-CPU workspace and a $100 budget each. Table 1 lists, per target, the number of verified bugs, the region coverage of the target’s source code, and the agent spend. After the initial campaign, we kept running it informally, finding 72 more bugs; a newer trial on compilers has led to 30 bugs and 11 issues in binaryen, OCaml, and OxCaml.

Two experiment modes: open loop and closed loop

Figure 4. Two experiment modes: rounded, tinted boxes are LLM agents; square boxes are deterministic tools; the dashed grey box is a user.

TargetPropertyBugsCov.Setup ($) | Triage ($)Write-up ($) | Total ($)
uutils coreutilsdifferential vs. GNU coreutils3033.4%11502587
brushdifferential vs. bash1735.7%13421267
gitoxidedifferential vs. git1427.3%11241145
DuckDBTLP, NoREC, optimizer toggles1122.4%20411274
DataFusionTLP, NoREC, rule toggles834.9%1237957
Total806719568330

Table 1. Closed-loop campaign. Setup (intake and harness), triage and write-up are agent costs in $.

Discussion and Future Work

The main contribution we presented in the experimental results is not unique to Ephyra; it is possible to set up any PBT library with a similar optimization loop, and intuitively expect similar results. We see our experiments as a starting point for agent-authored optimized generators, and Ephyra’s design choices as a good set of primitives for such explorations. Evaluations are immensely more complex to design when LLMs are part of the evaluated system; the classical design of ETNA [14] was not sufficient for our measurements, so we plan to look into existing coding agent evaluation frameworks such as Harbor [8] for more controlled experimental settings that only modify the generator, keeping the rest of the system constant to understand the effect in isolation. One of the crucial problems we encountered was reward hacking. Given a metric, an agent can optimize solely the metric rather than the testing power it stands for. Designing metrics that resist gaming, and detecting the reward hacking remains an open problem for agent-driven generator synthesis. The future work we imagine is multifaceted: we would like to work on turning Tyche [7] into an interactive tool for shaping generators instead of just understanding them; instead of simply measuring the generators, we would like to prove properties about their soundness and completeness with respect to the desired distribution; we would like to explore the effects of different property-runners in more detail and understand how we can utilize source-level structured mutations more effectively than we do today. Finally, the optimization architecture we built is a very simple hill-climbing loop. Researchers have shown a variety of different strategies for autonomous optimization loops [16, 23, 30, 42] that we plan to evaluate on generator optimization and adopt when we observe their benefits.

References

Bug Reports

Table 2 lists the 349 bugs found in the case studies of § 4: 167 in the open-loop studies, 80 in the closed-loop campaign, 72 in a later closed-loop run, and 30 in a later trial on compilers. Duplicate means an existing upstream issue or pull request already covers the bug; won’t report means we decided not to report the bug; not yet reported means its write-up is ready but not filed; declined means upstream closed our report as not planned.

BugReportStatus
SQLite (open loop, 63)
ROLLBACK TO after DROP TABLE/INDEX commits a stale root page (incremental vacuum)forum 3bc3c8adfafixed
DELETE leaves an unaccounted byte on an interior page (WITHOUT ROWID, 3-byte key)–fixed
x IN (SELECT ...) in HAVING or the result converts the GROUP BY term x–fixed
Blob over 4000 bytes read from overflow pages is converted as UTF-8 in a UTF-16 databaseforum 48c64d8babfixed
LEFT JOIN with b = NULL AND c IN (...) on index (b, c) also returns a matched rowforum 78bc86a164fixed
SELECT * over a USING join left of a RIGHT/FULL JOIN fails with ambiguous column nameforum 8be1215e2creported
Flattened UNION ALL subquery with a RIGHT/FULL JOIN after its first SELECT loses WHERE termsforum d098617140fixed
Window query drops the left row of a correlated LEFT JOIN whose ON has a bare outer columnforum 7f4ae01b58fixed
UNION ALL subquery/CTE with inner join left of a RIGHT/FULL JOIN returns only its last armforum 50dd78a44bfixed
GROUP BY/ORDER BY over a subquery ordered the other direction skips the outer sortforum 5cd4b3b27afixed
json_valid() returns 0 for canonical JSON made from JSON5 input (stale cached parse)forum 1339f93b28fixed
Join drops rows with an expression index over a column pinned by col = constantforum 529732a774fixed
MULTI-INDEX OR over a partial index WHERE a=const fails with internal query planner errorforum 3ede79ce42fixed
Partial index WHERE col = constant makes col read back as the constant with col’s affinityforum 35992eec46partly fixed
Bloom filter before an INTEGER PRIMARY KEY lookup drops the REAL join key -2632^{63}forum 403ca95409fixed
Constant propagation leaves REAL -2632^{63} a REAL and misses an INTEGER -2632^{63} rowforum 9bbaa34eeapartly fixed
sum(b ORDER BY a) aggregates the ORDER BY term instead of the argumentforum b53a65d1fcfixed
TEXT over 4000 bytes read from overflow pages loses a leading BOM in a UTF-16 databaseforum 36a73e8c8afixed
JoinMiss optimization reuses a pruned automatic index across correlated subquery runsforum 8e5c07be86fixed
LIKE/GLOB optimization’s two-range index scan is treated as ordered under IN/skip-scanforum 8cbaf327c7fixed
Planner ignores NOT INDEXED on WITHOUT ROWID tables (plan only)forum cd41fceaa0fixed
x IN (SELECT y COLLATE NOCASE ...) seeks a BINARY index and misses rowsforum f505c400cffixed
ADD COLUMN blob DEFAULT is read as UTF-8 when cast to text in a UTF-16 databaseforum a9c654a346fixed
Index on an untyped VIRTUAL generated column over REAL returns integral REALs as INTEGERforum 6973ad169efixed
WITHOUT ROWID ORDER BY added column NULLS LAST with PK IN: malformed error or wrong rowsforum eee345f931fixed

Table 2. Bugs found by the case studies, with their upstream reports.

BugReportStatus
group_concat() window drops a separator after an inverse step leaves the text emptyforum b02f0a01e0fixed
WITHOUT ROWID row-value range into an added column seeks the PK: malformed or missing rowsforum eee345f931 (reply)reported
concat()/concat_ws() take a BLOB’s size before UTF-8 conversion in a UTF-16 databaseforum c621e963a9duplicate
Row-value IN list driving a join reuses the full IN’s ephemeral table, repeating rowsforum 764df92dafixed
Row-value IN (SELECT a, b) over WITHOUT ROWID with added column b probes the PK: malformedforum eee345f931 (reply)reported
Compound with ORDER BY COLLATE applies the last SELECT’s GROUP BY/HAVING/DISTINCT to allforum 3859ed06b6fixed
OR in a LEFT JOIN’s ON with the no-op join table omitted: wrong rows or SIGSEGVforum 16c7c8892bfixed
Btree delete keeps the deleted cell’s cached CellInfo; a later write reports malformedforum 3e17701ba5fixed
Compound merge removes duplicates under one operand’s collation, not the compound’sforum a3d1c48f31fixed
RENAME COLUMN of a TEMP table rewrites temp triggers and views twice, breaking themforum c75ba2bcd0fixed
FTS5 ft = ’q’: constant propagation breaks auxiliary functions (no such cursor)forum a27cfbee6creported
FTS5 MATCH with rowid = non-integer returns every matching rowforum 422031d9f4fixed
Virtual table created in a savepoint skips xRollbackTo; FTS transaction then failsforum 9454b36e0afixed
FTS5 detail=none/column tokenizes UNINDEXED columns; snippet() reports malformedforum a3eee51a91fixed
Geopoly rowid = non-integer value finds the truncated rowid; UPDATE/DELETE hit the wrong rowforum 6d40716f60fixed
FTS5 locale=1, contentless_unindexed=1 with an UNINDEXED column: every INSERT is malformedforum c7ab2c9250reported
FTS5 tokendata=1 query for a term missing from a segment reads another page: corruption errorforum 8a9e1dca7efixed
FTS5 contentless_unindexed table: DROP leaves %_content behind and RENAME skips itforum 87d4bdefc0fixed
Geopoly UPDATE of the rowid without _shape fails with an invalid polygon errorforum 1b9d323951fixed
R-Tree UPDATE SET rowid / INSERT with rowid ignore the new rowidforum 26da7cd9e2fixed
FTS5 secure-delete: stale doclist-index jump returns rows of the next term†fixed
FTS5 detail=none: secure-delete of a rowid replaced with secure-delete off corrupts the indexforum e380b6876creported
fts5vocab term range constraints and ORDER BY term use UTF-8 order in a UTF-16 databaseforum b0c03988e8reported
Compound rows change with ORDER BY when a later member sets the collation (since 3.53)forum 2fb4734a27reported
FTS5 secure-delete: a merge drops a delete marker, so a later secure delete revives old entry†forum 737cb495eareported
DROP CONSTRAINT of a foreign key with SET DEFAULT/NOT DEFERRABLE/MATCH fails or leaves part†forum 5e80e94512fixed
DROP NOT NULL removes NOT DEFERRABLE instead, or drops a DEFERRABLE clause after NOT NULL†forum 28af38d7c3fixed
FTS5: phrases of a NEAR group that does not match a row report stale instances for itforum 9fa08e2dbdreported
FTS5: wrong instances below a non-matching AND, and right of a NOT whose left side is empty†forum 5cffabd8e9reported

Table 3. Bug Report Status

BugReportStatus
fts5vocab rowids restart in every scan, so MULTI-INDEX OR and RIGHT JOIN return wrong rowsforum d29f636f37reported
RENAME COLUMN fails on an INTEGER PRIMARY KEY column named in a UNIQUE table constraintforum b088eb5b02reported
MULTI-INDEX OR on a WITHOUT ROWID table misses rows when an OR term uses OP_SeekScanforum f554fe3656fixed
FTS5 trigram: LIKE and GLOB on an UNINDEXED column return no rowsforum be50ab5692reported
WHERE OR on a table left of a RIGHT JOIN drives a MULTI-INDEX OR and is never tested–won’t report
FTS5 tokendata=1: prefix query then term query in an open transaction reads freed memoryforum 5c53f75fbbfixed
FTS5 trigram LIKE pushdown reads one byte past the pattern (sqlite3Fts5ExprPattern)†forum 402c7177cbfixed
In a UTF-16 database a zeroblob() used as text is converted from UTF-8, so CAST(zeroblob(4) AS TEXT) has 8 bytes and depends on the planforum 420911fdacfixed
UPDATE re-applies affinity to unassigned columns: REAL -2632^{63} becomes INTEGER in the table but not in untouched indexes (trunk regression)–won’t report
Turso (open loop, 72)
X -> X / X -> X with identical operands panics (index out of bounds)#9836reported
Date/time modifiers accept trailing text#9837fixed
json_each/json_tree with a NULL root path errors instead of returning no rows#9838reported
json_each/json_tree error on an integer beyond i64 instead of returning a real#9839fixed
Negative-zero REAL becomes -0.0 in JSON; SQLite writes 0.0#9840reported
json_error_position at end of input after whitespace: internal error or off by one#9841fixed
Unused CTE is resolved and its errors reported when the statement has any subquery#9842fixed
json_patch with a repeated key (object then null) gives malformed JSONB or wrong text#9843reported
json_set/jsonb_set/json_remove mishandle NULL results, inputs and paths#9844reported
->/->> abbreviated, 32-bit integer and BLOB paths return NULL#9845reported
RIGHT/FULL JOIN with a WHERE subquery over a table-valued function panics#9847fixed
MATERIALIZED CTE with a self-referencing subquery in a nested query: internal error#9848fixed
Text-to-real parsing for REAL affinity and sum/avg differs from CAST beyond exponent ~99#9849reported
RIGHT JOIN then LEFT JOIN whose ON subquery reads the left table: panic or wrong rows#9850reported
Subqueries and constants hoisted before the loop raise errors SQLite never hits#9851reported
FULL/RIGHT JOIN onto a WITHOUT ROWID table with a secondary index panics#9852fixed
HAVING terms on GROUP BY columns are not applied before aggregating#9880fixed
Hash join returns a whole-number REAL stored as an integer as an INTEGER#9881fixed
MATERIALIZED CTE first filled inside a nested subquery’s loop panics#9882fixed
min/max window with a frame and outer ORDER BY rowid fails with table not found#9883fixed
strftime() ignores the julian-day range check#9884fixed
json_pretty of an empty object/array adds a newline#9885reported
timediff skips day>28 normalization#9886fixed

Table 1 (PDF p. 11).

BugReportStatus
HAVING predicate that is a GROUP BY term reads the column after the scan#9887fixed
strftime %J / REAL formatting prints 17 digits#9888reported
-0.0 in a REAL column keeps its sign#9889fixed
+HH:MM modifier with hour 24 adds a day#9921fixed
json_insert at the root path replaces the document#9924fixed
-9223372036854775808 and -9.223372036854776e18 treated as the same expression#9925fixed
BETWEEN on a partial index’s column panics (covering index misjudged)#9926fixed
JSON infinity rendering by global string replace corrupts JSON strings#9927fixed
UPDATE through a covering index with a rowid-reading partial index: panic or duplicate rowid#9932fixed
UPDATE with a BEFORE UPDATE trigger builds an index key without REAL affinity: corrupt index#9933fixed
Partial index + join panics: column not stored in the index#9934fixed
Date/time functions don’t stop at an embedded NUL#9935fixed
JSON subtype lost when a result column contains a GROUP BY expression#9966fixed
JSON cache treats a REAL 1.0 as the INTEGER 1 seen just before#9967fixed
Date/time functions reject a trailing vertical tab after the time value#9968fixed
strftime() prints an overflowing day as is instead of normalizing it#9969fixed
CREATE INDEX rejects double-quoted string literals SQLite accepts#10337reported
likelihood() rejects a float literal with an exponent (625e-4) or in parentheses#10338reported
pragma_quick_check / pragma_integrity_check name their column message#10339reported
ANALYZE cannot reach the temp schema or TEMP tables and indexes#10340fixed
VACUUM INTO ’:memory:’ writes a file named :memory:; VACUUM INTO ” rejected#10341fixed
json_each/json_tree declare column types, so comparisons apply the wrong affinity#10342fixed
JSON5 // comment ends only at LF, not at CR, U+2028 or U+2029#10343fixed
group_concat over a sliding window frame panics or drops an empty string#10352reported
Column qualified with sqlite_master not found when FROM names sqlite_schema#10353reported
PRAGMA index_xinfo reports the rowid entry’s name as ” instead of NULL#10354reported
ANALYZE on a database without tables does not create sqlite_stat1#10355reported
soundex() returns ’?000’ for input with a non-letter#10356reported
json_patch/jsonb_patch decode escapes in added keys; jsonb_patch writes a malformed key#10370reported
json_error_position one character off for a bad escape or a number without digits#10378reported
RENAME COLUMN of a WITHOUT ROWID primary-key column moves it to the end of the key#10387reported
Correlated subquery reads NULL for a hash-build table when the probe side is a LEFT JOIN#10388reported
subquery not found for a WHERE subquery reading a table outside a materialized hash build#10389reported
RIGHT JOIN unmatched-rows pass panics (cursor is None) with a nested FROM subquery#10390reported
ADD COLUMN ... DEFAULT -9223372036854775808 reads as REAL for existing rows#10391reported
STRICT type check runs before NOT NULL, so INSERT OR IGNORE fails on a row SQLite skips#10392reported
UPSERT DO UPDATE is compiled when the INSERT cannot conflict; its errors fail the INSERT#10393reported

Table 4.

BugReportStatus
concat_ws() drops BLOB arguments and returns NULL when the separator is a BLOB#10394reported
Compound ORDER BY x ISNULL does not match a result column written x IS NULL#10395reported
schema.alias.column naming an outer query’s table is not found in a subquery#10396fixed
PRAGMA table_info/table_list keep a view’s old columns after its table changes#10397reported
sqlite_schema.sql holds a re-serialized statement instead of the original text#10398reported
PRAGMA index_list/index_info/index_xinfo omit a WITHOUT ROWID table’s primary-key index#10399fixed
unhex(X, Y) returns NULL when Y is an INTEGER, REAL or BLOB#10400fixed
UPDATE of a child table fails with foreign key mismatch when no FK column is assigned#10401fixed
replace(X, ”, NULL) returns NULL instead of X#10426fixed
OR over an index and the rowid misses rows: a seek leaves an earlier DeferredSeek pending–not yet reported
JSON functions take a non-JSONB BLOB for JSONB and skip leading whitespace bytes–not yet reported
UPDATE on a parent table fails with foreign key mismatch while foreign keys are off–not yet reported
PostgreSQL (open loop, 16)
Nested BETWEEN grows exponentially in parse analysis and ignores statement_timeoutBUG #19758reported
Hashed set operations and recursive UNION pre-size hash tables without the hash_mem capBUG #19759reported
CLUSTER/REPACK drops a live row updated in a rolled-back subtransactionBUG #19760reported
btree_gist date index: inserting infinity fails with cannot subtract infinite datesBUG #19761reported
EXPLAIN ANALYZE of a Parallel Append sharing a SubPlan fails an assert in shm_toc_insertBUG #19766reported
ORDER BY/GROUP BY of a negated constant deparses as a column position; dumps do not restoreBUG #19767reported
GROUP BY with ROLLUP and ndistinct statistics fails with corrupt MVNDistinct entryBUG #19772reported
Cross-partition UPDATE of a self-referencing table with ON UPDATE CASCADE fails with 23503BUG #19781reported
SET CONSTRAINTS in a released savepoint survives ROLLBACK TO an outer savepointBUG #19782reported
Sorted GROUPING SETS leave a grouping column un-nulled (PG 19 regression)BUG #19783reported
min/max optimization costing = ANY over max(col) fails with no relation entry for relid 1BUG #19784reported
JSON_TABLE: subquery in PASSING fails with XX000; PASSING evaluated once per columnBUG #19786reported
could not find pathkey item to sort: Gather Merge over merge join below reordered LEFT JOINsBUG #19787reported
MCV statistics with a domain-over-bool column: expression not found in statistics objectBUG #19789reported
XX000 cannot start a subtransaction with unprocessed inval messages after ALTER DOMAINBUG #19790reported
ALTER COLUMN TYPE of an INCLUDE column of an index backing a foreign key fails (2BP01)BUG #19791reported

Table 5.

BugReportStatus
ada-url (open loop, 12)
Drive-letter reference against a file: base marks the URL as having an opaque pathada#1278reported
url_search_params keeps ill-formed UTF-8 instead of decoding it to U+FFFDada#1279declined
URLPattern test/exec/match resolve a relative string input against an empty base URLada#1280fixed
URLPattern special-scheme pathname does not canonicalize a backslash to /ada#1281fixed
set_host(host:port) over the max input length applies the host, drops the port, returns trueada#1282reported
canonicalize_protocol indexes an empty string_view for an escaped-colon protocol patternada#1283reported
URLPattern opaque pathname: a leading // is parsed as an authority and droppedada#1284reported
URLPattern search canonicalization uses the non-special query encode setada#1285reported
url_aggregator host setters exceed the max input length with IDNA-expanding hostsada#1326reported
generate_pattern_string drops braces around a part whose prefix starts with / but is longerada#1327fixed
IDNA combining-mark, bidi and normalization tables predate the Unicode 17 mapping tableidna#97fixed
x86-64 builds run SSSE3 instructions without a runtime CPU check–fixed
Cedar (open loop, 2)
upsert_entities keeps a stale ancestor when the batch names the same entity twicecedar#2606fixed
add_entities compares parents with the stored closure: drops parents, refuses identicalscedar#2629reported
QED (open loop, 2)
Real constants parsed as f32, so distinct literals are proved equal (wide ones panic)qed#13reported
Panic on INTEGER vs DECIMAL operands in IN (subquery) and UNIONqed#14reported
uutils coreutils (closed loop, 45)
comm compares lines including their newline, rejecting sorted input#15439fixed
comm --check-order --total still prints the total after an order failure#15254duplicate
cksum with a legacy algorithm omits the file name for an explicit -#15254duplicate
csplit --suppress-matched rejects a line number one past the end of input#15584fixed
csplit /regex/ after a line-number pattern never tests the line the split stopped at#15440reported
expand rejects --tabs=2,/2 and --tabs=4,+2 as not ascending#15579fixed
dd status=none still warns that it cannot skip on a short stdinPR #13115duplicate
fmt puts one space instead of two after a sentence-ending first word at line end#15585reported
expand misplaces tab stops after a file without a trailing newline and past 8 KiB#15656reported
fold counts control characters such as ESC and DEL as zero columns#15571reported
join orders a missing join field before an empty one#15254won’t report
join treats an empty or blank-only line as one empty field#15254won’t report
nl pads unnumbered lines with width+1 spaces, ignoring the -s length#15591reported
od -S with -Ax prints 7-digit hex offsets instead of 6#15592fixed
sort -V ignores -i, -d and -f#15587reported
seq -w pads to the width of the increment, which is never printed#15572fixed
sort ignores a key’s own r when the key is the whole line (-k1r)#15570fixed

Table 6.

BugReportStatus
sort treats form feed and carriage return as blanks#15588fixed
numfmt --invalid: one invalid field leaves the whole line unconverted#15575reported
split --verbose prints nothing with -n/--number#15583reported
split accepts a suffix start too large for the suffix length on empty input#15569reported
join: trailing blanks start an extra field; -i prints the first line’s join field#15576fixed
split -a 0 with -n N uses a 1-character suffix instead of the default#15581fixed
tr validates string2 constructs against the wrong mode#15589reported
numfmt --to=si/iec ignores --round for values that get no suffix#15574reported
sort -h ignores -f, so lowercase suffixes are not folded#15577reported
expand counts a NUL byte as one column, so the next tab stop comes early#15580reported
nl resets the header/footer section to body at each input file#15586fixed
numfmt --round changes the last digit of a number that needs no rounding–won’t report
sort: global -i is not overridden by -d#15578reported
basenc --base64url -d rejects input whose last group has no = padding*#15664reported
cat -s drops a single blank line after a line continued from an earlier file or read*#15655reported
chmod rejects modes such as -=r, -+x and -,u+x as unknown options*#15665reported
chmod --reference processes the first FILE operand last*#15666reported
csplit hangs on /REGEX/-N after a line-number pattern that stopped at a regex line*#15653reported
cut --output-delimiter prints the last field twice when the line has no terminator*#15654reported
dd conv=swab pairs bytes within each input block, so output depends on bs= and reads*#15646reported
numfmt prints -0 for a negative number that rounds to zero*#15661reported
printf %e/%g round an exact halfway value up instead of to even*#15662reported
printf %.s accepts a * precision above 2147483647 instead of failing#15663reported
split -l parses its count like a byte SIZE (-l 010 means 8 lines, -l 1k accepted)*#15660reported
expand/unexpand take a bare - operand as an obsolete -N tab option, skipping stdin*#15658reported
unexpand handles a line continuing from one FILE into the next as two lines*#15657reported
uniq -f treats carriage return and form feed as field separators*#15659reported
unexpand counts a NUL byte as one column, so blanks after it hit the wrong tab stops*PR #15627duplicate

Table 7 (PDF p. 15).

BugReportStatus
brush (closed loop, 27)
+= on a declare -i variable panics on i64 overflow (overflow-checked builds)#1577reported
${s/pat/rep}: $1, $0, ${name} in the replacement are read as regex group references–fixed
Negative subscript on a sparse array counts from the element count, not the highest index#1570fixed
printf invalid-number messages ignore 2> redirection, and the exit status is 0#1589reported
break/continue do not track how many loops enclose them#1595reported
Unexpanded ~01 loses its leading zero and prints as ~1#1592reported
Error inside a compound command with 2> ignores that redirect#1596reported
Command substitution hangs when a builtin writes over 64 KiB after a nested $() | #1576 | reported
Glob "**" (empty quotes before the leading dot) does not match dot files#1584reported
An error on one line of a multi-line $() does not end the command substitution | #1593 | reported
${v=word} expands to word, not v’s new value, under declare -u/-l/-c/-i | #1585 | reported
Brace expansion is not applied to assignment arguments of declare/local/-export#1569reported
A shell error in one pipeline stage aborts the parent shellPR #1538duplicate

Table 8 (PDF p. 15).

BugReportStatus
${a[@]+x} with an empty array gives one empty word instead of none | #1568 | fixed
~- expands to an empty string when OLDPWD is declared but unset#1591reported
An assignment error inside a function does not abandon the rest of the calling line#1594reported
nullglob removes a non-pattern word like [a/b] (bracket expression containing /)#1582reported
echo -n -n hi fails with a usage error, and -e wins over a later -E*#1588reported
[[ -z == x ]] is accepted as a string comparison; bash rejects it*#1587reported
shopt -s globstar is accepted, but ** does not match recursively*#1581reported
printf with a large * width aborts and drops the rest of the output*#1590reported
read drops control characters from a pipe or file and treats ^C/^D as keystrokes*#1579reported
declare -f indents lines inside multi-line strings and here-documents*#1580reported
Glob path component without metacharacters keeps backslashes from an expansion*#1583reported
${v/pat/rep}: unquoted & in the replacement is not replaced by the match* | #1586 | reported
A syntax error later in a script or -c string stops every earlier command from running*#1578reported
( case a in a) echo y;; esac ) is a syntax error: esac ) parsed as another case item*PR #1529duplicate
gitoxide (closed loop, 31)
@{push} fails when the branch has no configured remote; git falls back to origin#3096reported
@{u}@{1} and @{push}@{...} are rejected with unconsumed input#3098reported
rev-parse of a short name fails when an earlier DWIM candidate is a dangling symref#3090reported
<name>@{n} fails when the branch name looks like a short hash#3099reported
@{-N} peels an annotated tag that git returns unpeeled#3095reported
<ref>@{n} on an unborn HEAD or dangling symref returns a reflog entry; git fails#3089reported
@{-1} resolves to a deleted branch whose name matches a dangling symref#3090reported
SHA-256 repositories: loose objects are never found by an abbreviated id#3101reported
<rev>^{/regex} on an unpeelable ref (unborn HEAD) searches all refs instead of failing#3100reported
<ref>@{n} resolves to the null object id when the entry records a ref deletion#3089reported
A short object prefix fails when a dangling symref has the same name#3090reported
<ref>@{0} returns the newest reflog entry instead of the ref’s current value#3089reported
<ref>@{N} fails when the reflog has exactly N entries (oldest has a previous value)#3089reported
@{push} with push.default=upstream and a separate push remote maps through the wrong remote#3096reported
Tree merge deep-asserts: one side renames and edits a file, the other makes the same edit*#3091reported
Commit::describe() ignores annotated tags that point to another tag*#3103reported
topo traversal returns a tip that an end hides (Parents::First, or tip also an end)*#3105reported
rev_walk().with_hidden() returns ancestors of hidden commits under skewed commit times*#3104reported
<rev>::file/ resolves to the blob; git rejects a trailing slash on a non-directory*#3102reported
Tree merge keeps the ancestor when one side edits a file, the other makes it a symlink*#3107reported
Tree merge: same content on both sides plus one-sided chmod +x keeps the ancestor*#3097reported

Table 9.

BugReportStatus
status() fails when HEAD’s branch is a symbolic ref to an unborn branch*#3112reported
status() misses a staged deletion when the index entry is intent-to-add*#3115reported
status() lists a file that replaced a submodule as untracked as well as a type change*#3113reported
Tree merge reports a false dir/file conflict when the other side deletes part of the dir*#3106reported
status ignores a gitlink whose path is not listed in .gitmodules*#3114reported
status() reports an executable replaced by a symlink as Modified, not TypeChange*#3111reported
Tree merge loses a file renamed onto a directory that the other side modified*#3109reported
Tree merge treats d as renamed to y after a move of d/e to y/z*#3108reported
Text merge reports no conflict when both sides append different lines at the end*#3110reported
Tree merge drops a file added on both sides when only one side makes it executable*#3097reported
DuckDB (closed loop, 29)
Outer join with ON-clause subqueries under an inequality join: DataChunk::Slice assertion#27717reported
SEMI join ON true replaced by a cross product leaves a stale projection map (bind error)#27452reported
Nested loop join ignores the projection map after an outer join with an ON subquery#27717reported
Piecewise merge join ignores the projection map after an outer join with an ON subquery#27717reported
dense_rank() over an empty local sort collection segfaults#27448fixed
Hash join with a zero-column probe side fails: tuple width should not be 0#27450fixed
Join order optimizer drops a relation-free filter (WHERE NULL returns a row)#27451fixed
Statistics propagation table filter loses a column reference (assertion)#27447reported
Join elimination asserts on a conditionless join from a correlated aggregate subquery#27445reported
Join elimination leaves a stale parent projection map (missing child column)#27449reported
Statistics propagation folds an aggregate with no expressions (assertion exprs.size() > 0)#27446fixed
SEMI JOIN on a right-only condition returns no rows with filter_pushdown disabled*#27633reported
Caching-operator flush continues into a finished operator: join_hashtable assertion*#27722reported
LEFT JOIN ON false AND x IN (SELECT ...) fails: Failed to bind column reference*#27719reported
Join with an ungrouped aggregate over an empty table: LIMIT 1 returns 3 rows*#27714reported
Table filter found to be true-or-NULL keeps rows where the predicate is NULL*#27715reported
Lateral reference in a FULL JOIN ON subquery: INTERNAL !ContainsDependentJoin*#27716reported
CROSS JOIN + LEFT JOIN ON (SELECT true) + WHERE: assertion "normalized" in join order*#27723reported
x IN (correlated scalar subquery, x) treated as NULL-propagating: drops a row*#27712reported
FULL JOIN with an ON subquery over a CROSS JOIN: Vector::Reference type mismatch*#27726reported
abs() statistics on FLOAT/DOUBLE are backwards when the range ends at 0 (wrong result)*#27711reported

Table 17.

BugReportStatus
Join condition statistics narrowed by a sibling condition (debug_verify_stats mismatch){sup}*#27725reported
Lambda inside an aggregate inside a lambda: BoundReferenceExpression#27720reported
INTERNAL error{sup}*#27724reported
SIMILAR TO / regexp filter above a FULL JOIN fails a string_stats assertion{sup}*#27724reported
IN subquery in JOIN ON plus LEFT JOIN ON subquery: index out of range#27606duplicate
INTERNAL error{sup}*
Outer join max cardinality set to the product: LIMIT 1 returns 2 rows, EXISTS is false{sup}*#27641duplicate
Correlated IN/NOT IN over mostly-NULL subquery values takes quadratic time{sup}*–won’t report
Query with streaming delim CTEs deadlocks with all threads idle (regression){sup}*#27737reported
EXPLAIN (SQL) exports a CSE-hoisted constant as a quantile argument (binder error){sup}*#27776reported
DataFusion (closed loop, 20)
UNION with an EXISTS arm fails: physical input schema differs (stale nullability)PR #26201duplicate
GROUP BY c, CAST(c AS BIGINT) fails in eliminate_group_by_constant (duplicate names)PR #26356duplicate
ORDER BY ... LIMIT 0 panics in TopK (k > 0) when eliminate_limit does not run#26431reported
Correlated EXISTS/IN selecting CAST of its correlated column fails (duplicate names)#26033duplicate
ORDER BY col, scalar subquery LIMIT over a RIGHT JOIN fails SanityCheckPlan (regression)#26572reported
= ANY (subquery) in a JOIN ON over an aliased table fails: No field named#26573reported
Join equivalence drops or keeps side constants wrongly; ORDER BY skipped after RIGHT JOIN#26036duplicate
Filter on a constant column is pushed below a global aggregate, returning a wrong row#26545duplicate
SanityCheckPlan with sort-merge join when a filter makes a sort or join key constant{sup}*#26624reported
SanityCheckPlan for two window functions with target_partitions = 1{sup}*#26626reported
push_down_filter substitutes the wrong expression when col and CAST(col) share a name{sup}*#26619reported
count() OVER () sees only the LIMIT rows under ORDER BY ... LIMIT over another window{sup}#26617reported
Panic: two window functions sharing ORDER BY a + 1 get different sort keys (debug){sup}*#26620reported
ORDER BY <double> LIMIT n drops -0.0 rows when 0.0 arrives earlier (TopK filter){sup}*#26618reported
SanityCheckPlan with sort-merge join when a filter says a + b = a{sup}*#26623reported
SanityCheckPlan for UNION ALL ... ORDER BY when each arm selects a column twice{sup}*#26625reported
GROUP BY CAST(a AS VARCHAR), a with a common subexpression fails in optimize_projections{sup}*#26621reported
x > ALL (subquery) in the SELECT list fails with a schema error unless aliased{sup}*#26622reported
ORDER BY a * 3 > 1 LIMIT n over a join fails SanityCheckPlan (Null-typed range){sup}*#26430duplicate
Sort-merge join claims the right side’s ordering when the left key is constant{sup}*#25052duplicate

Table 10.

BugReportStatus
binaryen (compiler trial, 18)
--translate-to-exnref produces invalid IR for a delegate target ending in a none-typed instrbinaryen#9291reported
Binary reader: element segment indexes off by one after a declarative segment–not yet reported
Text parser leaves declarative element segments out of the elem index space–not yet reported
Text parser rejects a table init expression written as plain instructions–not yet reported
memory.copy between 32-bit and 64-bit memories aborts the reader–not yet reported
Readers reject an if with an unreachable condition targeted by a branch inside it–not yet reported
Readers reject an if with block parameters after unreachable–not yet reported
Readers abort with length_error on a multivalue if with an unreachable condition–not yet reported
Readers hit an assertion in fixExtraOutput for a loop with params–not yet reported
Text parser rejects a labeled if whose condition sets a non-nullable local–not yet reported
OptimizeInstructions treats two identical suspends as equal (silent miscompile)–not yet reported
--outlining treats instructions with different operand types as equal–not yet reported
--outlining outlines multi-result sequences without multivalue–not yet reported
--outlining aborts on unreachable atomic.rmw, struct.get, call_ref–not yet reported
--outlining asserts in StackSignature on repeated code with unreachable operands–not yet reported
--translate-to-exnref drops a typed try’s type on try~delegate–not yet reported
--llvm-nontrapping-fptoint-lowering asserts on an unreachable trunc_sat operand–not yet reported
--memory64-lowering makes an unreachable memory.grow/table.grow invalid–not yet reported
OCaml (compiler trial, 2)
Non-exhaustive integer match crashes instead of raising Match_failureocaml#15185reported
amd64 -linscan: Atomic.compare_and_set returns true without storing the valueocaml#15189reported
OxCaml (compiler trial, 10)
-Oclassic crashes on a division by constant zero inside a tryoxcaml#7668fixed
-Oclassic crashes on a float# field read in an unreachable match armoxcaml#7683reported
Reaper with phantom lets: an unused local in a closure breaks compilationoxcaml#7684reported
Binary emitter encodes 32-bit bswap of r8d-r15d without REX.Boxcaml#7700reported
-use-ssa: validator rejects x≤1x \le1 against x<2x < 2 for the same switchoxcaml#7701reported
-use-ssa: a call before an unused pure op becomes a tail call in one pipeline onlyoxcaml#7702reported
-flambda2-expert-inline-effects-in-cmm: duplicate binding order on a twice-used floatoxcaml#7704reported
-flambda2-expert-inline-effects-in-cmm: Array.iter2 crash in emit_expr_raiseoxcaml#7705declined
-no-cfg-eliminate-dead-trap-handlers: regalloc check fails on a nested try–won’t report
-no-cfg-eliminate-dead-trap-handlers: dead loop around a try kept as a cycle–won’t report

Table 11.

References

  1. [1]Services, A. W. (2025). Kiro. <a href="https://doi.org/https://kiro.dev/" rel="noopener noreferrer" target="_blank">https://doi.org/https://kiro.dev/</a>
  2. [2]Carlini, N. (2026). Building a C compiler with a team of parallel Claudes. Anthropic Engineering Blog. <a href="https://doi.org/https://www.anthropic.com/engineering/building-c-compiler" rel="noopener noreferrer" target="_blank">https://doi.org/https://www.anthropic.com/engineering/building-c-compiler</a>
  3. [3]Chuyang Chen, Brendan Dolan-Gavitt, and Zhiqiang Lin. 2025. ELFuzz: Efficient Input Generation via LLM-driven Synthesis Over Fuzzer Space. In 34th USENIX Security Symposium (USENIX Security 25). USENIX Association, Seattle, WA, 6279–6298. https://www.usenix.org/conference/usenixsecurity25/presentation/chen-chuyangusenix.org/conference/usenixsecurity25/presentation/chen-chuyang
  4. [4]Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming (ICFP). ACM, New York, NY, USA, 268–279. doi:10.1145/351240.351266DOI
  5. [5]Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM, 423–435. doi:10.1145/3597926.3598067DOI
  6. [6]Harrison Goldstein, Samantha Frohlich, Meng Wang, and Benjamin C. Pierce. 2023. Reflecting on Random Generation. Proceedings of the ACM on Programming Languages 7, ICFP (2023), 322–355. doi:10.1145/3607842DOI
  7. [7]Harrison Goldstein, Jeffrey Tao, Zac Hatfield-Dodds, Benjamin C. Pierce, and Andrew Head. 2024. Tyche: Making Sense of PBT Effectiveness. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST). ACM, 1–16. doi:10.1145/3654777.3676407DOI
  8. [8]Harbor Framework Team. 2026. Harbor: A Framework for Evaluating and Optimizing Agents and Models in Container Environments. https://github.com/harbor-framework/harbor. doi:10.5281/zenodo.20953922 Accessed 2026-10-20.DOI
  9. [9]developers, H. (n.d.). hegel-skill: An Agent Skill for Writing Hegel Tests. Retrieved https://github.com/hegeldev/hegel-skillgithub.com/hegeldev/hegel-skill
  10. [10]developers, H. (n.d.). hegel-zoo: Property-Based Tests for Open-Source Projects, in Every Language Hegel Supports. Retrieved https://github.com/hegeldev/hegel-zoogithub.com/hegeldev/hegel-zoo
  11. [11]Andrej Karpathy. 2026. autoresearch: AI Agents Running Research on Single-GPU nanochat Training Automatically. https://github.com/karpathy/autoresearch. Accessed 2026-10-20.DOI
  12. [12]Alperen Keles, Justine Frank, Ceren Mert, Harrison Goldstein, and Leonidas Lampropoulos. 2026a. Programmable Property-Based Testing. arXiv:2602.18545 [cs.SE]. https://arxiv.org/abs/2602.18545
  13. [13]Alperen Keles, George Miao, and Leonidas Lampropoulos. 2026b. Evaluating Shrinking (Experience Report). arXiv:2608.09935 https://arxiv.org/abs/2608.09935 To appear in Haskell ’26: Proceedings of the 19th ACM SIGPLAN International Haskell Symposium.
  14. [14]Alperen Keles, Jessica Shi, Nikhil Kamath, Tin Nam Liu, Ceren Mert, Harrison Goldstein, Benjamin C. Pierce, and Leonidas Lampropoulos. 2026c. Etna: An Evaluation Platform for Property-Based Testing. Journal of Functional Programming 36 (2026). doi:10.46429/jfp.17867DOI
  15. [15]Leonidas Lampropoulos, Zoe Paraskevopoulou, and Benjamin C. Pierce. 2018. Generating Good Generators for Inductive Relations. Proceedings of the ACM on Programming Languages 2, POPL (Jan. 2018), 30 pages. doi:10.1145/3158133DOI
  16. [16]Lange, R. T., Lakshminarayanan, R., & Cetin, E. (2025). Think in Haskell: Towards Open-Ended And Sample-Efficient Program Evolution. arXiv:2509.19349
  17. [17]Liu, S., Ponnampalli, S., Shankar, S., Zeighami, S., Zhu, A., Agarwal, S., Chen, R., Suwito, S., Yuan, S., Zorzi, I. S., Zaharia, M., Cheung, A., Crooks, J. E., Gonzalez, J. E., & Parameswaran, A. G. (2026). Supporting Our AI Overlords: Redesigning Data Systems to be Agent-First. Conference on Innovative Data Systems Research (CIDR). arXiv:2509.00997
  18. [18]Maaz, M., DeMoe, L., Hatfield-Dodds, Z., & Carlini, N. (2025). Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem. arXiv:2510.09907
  19. [19]MacIver, D. R. (2026). Hypothesis, Antithesis, synthesis. Antithesis Blog. https://antithesis.com/blog/2026/hegel/antithesis.com/blog/2026/hegel
  20. [20]MacIver, D. R., & Dolsak, A. F. (2020). Test-Case Reduction via Test-Case Generation: Insights from the Hypothesis Reducer (Tool Insights Paper). 34th European Conference on Object-Oriented Programming (ECOOP 2020) (LIPIcs, Vol. 166), 13:1–13:27. https://doi.org/10.4230/LIPIcs.ECOOP.2020.13
  21. [21]David R. MacIver, Zac Hatfield-Dodds, and Many Contributors. 2019. Hypothesis: A New Approach to Property-Based Testing. Journal of Open Source Software 4, 43 (2019), 1891. doi:10.21105/joss.01891DOI
  22. [22]Agustín Mista, Alejandro Russo, and John Hughes. 2018. Branching Processes for QuickCheck Generators. In Proceedings of the 11th ACM SIGPLAN International Symposium on Haskell. ACM, 1–13. doi:10.1145/3242744.3242747DOI
  23. [23]Novikov, A., Vũ, N., Eisenberger, M., Dupont, E., Huang, P.-S., Wagner, A. Z., Shirobokov, S., Kozlovskii, B., Ruiz, F. J. R., Mehrabian, A., Kumar, M. P., See, A., Chaudhuri, S., Holland, G., Davies, A., Nowozin, S., Kohli, P., & Balog, M. (2025). AlphaEvolve: A Coding Agent for.
  24. [25]Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. 2025. KernelBench: Can LLMs Write Efficient GPU Kernels? arXiv:2502.10517 https://arxiv.org/abs/2502.10517
  25. [26]Zoe Paraskevopoulou, Aaron Eline, and Leonidas Lampropoulos. 2022. Computing Correctly with Inductive Relations. In Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation (PLDI). ACM, 966–980. doi:10.1145/3519939.3523707DOI
  26. [27]contributors, P. (2026). proptest-derive: Custom-Derive for the Arbitrary Trait of proptest. <a href="https://doi.org/https://docs.rs/proptest-derive/latest/proptest_derive/" rel="noopener noreferrer" target="_blank">https://doi.org/https://docs.rs/proptest-derive/latest/proptest_derive/</a>
  27. [28]Manuel Rigger and Zhendong Su. 2020a. Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 1140–1152. doi:10.1145/3368089.3409710DOI
  28. [30]Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. 2023. Mathematical discoveries from program search with large language models. Nature 625, 7995 (2023), 468–475. doi:10.1038/s41586-023-06924-6DOI
  29. [31]Ronacher, A. (2025). Build It Yourself. <a href="https://doi.org/https://lucumr.pocoo.org/2025/1/24/build-it-yourself/" rel="noopener noreferrer" target="_blank">https://doi.org/https://lucumr.pocoo.org/2025/1/24/build-it-yourself/</a>
  30. [32]Armando Solar-Lezama, Liviu Tancau, Rastislav Bodík, Sanjit Seshia, and Vijay Saraswat. 2006. Combinatorial Sketching for Finite Programs. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). ACM, 404–415. doi:10.1145/1168857.1168907DOI
  31. [33]Sumner, J. (2026). Rewriting Bun in Rust. <a href="https://doi.org/https://bun.com/blog/bun-in-rust" rel="noopener noreferrer" target="_blank">https://doi.org/https://bun.com/blog/bun-in-rust</a>
  32. [34]Tomlinson, J. (2026). Open Source Libraries Are Evolving into Reference Implementations. <a href="https://doi.org/https://jacobtomlinson.dev/posts/2026/open-source-libraries-are-evolving-into-reference-implementations/" rel="noopener noreferrer" target="_blank">https://doi.org/https://jacobtomlinson.dev/posts/2026/open-source-libraries-are-evolving-into-reference-implementations/</a>
  33. [35]Bits, T. of . (2026). Property-Based Testing Skill. <a href="https://doi.org/https://github.com/trailofbits/skills/tree/main/plugins/property-based-testing" rel="noopener noreferrer" target="_blank">https://doi.org/https://github.com/trailofbits/skills/tree/main/plugins/property-based-testing</a>
  34. [36]Vasudev Vikram, Caroline Lemieux, Joshua Sunshine, and Rohan Padhye. 2024. Can Large Language Models Write Good Property-Based Tests? arXiv:2307.04346 [cs.SE] https://arxiv.org/abs/2307.04346
  35. [37]Vasudev Vikram and Rohan Padhye. 2026. Fuzzing with Agents? Generators Are All You Need. arXiv:2604.01442 https://arxiv.org/abs/2604.01442
  36. [38]Willison, S. (2026). My answers to the questions I posed about porting open source code with LLMs.. <a href="https://doi.org/https://simonwillison.net/2026/Jan/11/answers/" rel="noopener noreferrer" target="_blank">https://doi.org/https://simonwillison.net/2026/Jan/11/answers/</a>
  37. [39]Xia, C. S., Paltenghi, M., Tian, J. L., Pradel, M., & Zhang, L. (2024). Fuzz4All: Universal Fuzzing with Large Language Models. Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE), 1–13. https://doi.org/10.1145/3597503.3639121
  38. [40]Chenyuan Yang, Yinlin Deng, Runyu Lu, Jiayi Yao, Jiawei Liu, Reyhaneh Jabbarvand, and Lingming Zhang. 2024. WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models. Proceedings of the ACM on Programming Languages 8, OOPSLA2 (2024), 709–735. doi:10.1145/3689736DOI
  39. [41]Yang, J., Lieret, K., Ma, J., Thakkar, P., Pedchenko, D., Sootla, S., McMilin, E., Yin, P., Hou, R., Synaeev, G., Yang, D., & Press, O. (2026). ProgramBench: Can Language Models Rebuild Programs From Scratch?. https://arxiv.org/abs/2605.03546
  40. [42]Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. 2025a. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents. arXiv:2505.22954 https://arxiv.org/abs/2505.22954
  41. [43]Zhang, K., Dayou, Z., Wu, W., Wang, S., Zhao, Y., & Xie, Z. (2025). Low-Cost and Comprehensive Non-textual Input Fuzzing with LLM-Synthesized Input Generators. https://arxiv.org/abs/2501.19282
  42. [44]Mengxiao Zhang, Yongqiang Tian, Zhenyang Xu, Yiwen Dong, Shin Hwei Tan, and Chengnian Sun. 2024. LPR: Large Language Models-Aided Program Reduction. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM, 261–273. doi:10.1145/3650212.3652126DOI

Paper details

vlnc:W_bc33parc
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents