betaFeedback

WorksComputer Science

A Multi-Domain Study on the Effectiveness of LLMs at Writing Property-Based Tests

Abstract

A property-based test (PBT) checks a property of a program on many generated inputs, but writing one takes a property, a generator of valid inputs, and an oracle. LLM-based PBT writers produce all three, yet they have been evaluated only on Python libraries. We study how two of them, Proptest-AI and Agentic-PBT, generalize to ten programs in five domains: deep learning libraries, databases, system software, compilers, and LLM serving engines. Most of these programs take text in their own language, such as queries, configurations, and programs. Agentic-PBT covers more branches than Proptest-AI on seven of the ten programs with Opus 5 and on all ten with GLM-5.3-Flash. Domain-specific fuzzers cover more of the compilers and of PostgreSQL and less of PyTorch than Agentic-PBT. We filed 180 bug reports, 44 of which duplicate reports that others filed after the models' knowledge cutoffs; maintainers have confirmed 65 of the others, in all five domains, and fixed 23. We draw lessons for building and evaluating such writers: low observability hurts them, and mature fuzzers paired with LLM-written properties could check that each feature behaves as documented.

Introduction

A property is a predicate over software behavior that must hold for every input in a declared input space [10, 18, 30]. A property-based test (PBT) [10] checks a property on many generated inputs: a generator draws inputs from the input space, and an oracle checks the property on each input and the program’s output. PBT libraries such as QuickCheck [10] and Hypothesis [49] run the test and automatically reduce a failing input into a small counterexample. They do not write the test itself. Writing a PBT takes a property worth checking, a generator that respects the constraints the program places on its inputs, and an oracle that knows what the program must compute.

Properties can be mined from a project’s artifacts [5], such as its tests, documentation, and commit history. Techniques for mining properties go back more than two decades [2, 20, 23, 24, 37, 46, 47, 53, 70, 84, 92], and many tools turn mined properties into tests that find bugs [6, 7, 25, 27, 31, 59, 77, 78, 88, 89]. For example, Toradocu [27] mines API documentation, MR-Scout [89] existing tests, and DeepREL [16] the signatures of deep learning APIs. Each of these tools is built for one source, the kind of artifact it mines, and one domain, a class of programs such as deep learning libraries. Moving it to a new source or domain means rebuilding its mining logic and generator.

Large Language Models (LLMs) can read any kind of software artifact, which would replace the per-source mining logic, and they can write the generator and the oracle that a new program needs. We call a tool that uses an LLM to write PBTs an LLM-based PBT writer. Two general-purpose ones have been proposed. Proptest-AI [79] prompts an LLM with the documentation of one Python API method. Agentic-PBT [48] prompts a coding agent, an LLM that can run commands and edit files, to read a Python module’s code, write PBTs, run them, revise a test whose failure it judges a false alarm rather than a bug, and report the remaining failures as bugs. Both tools were evaluated only on Python libraries, which take Python values as inputs and need no harness, the code that starts or feeds the program under test.

Authors’ Contact Information: M M Abid Naziri, NC State University, Raleigh, NC, USA, [email protected]; Feiran (Alex) Qin, NC State University, Raleigh, NC, USA, [email protected]; Khush Patel, NC State University, Raleigh, NC, USA, [email protected]; Marcelo D’Amorim, NC State University, Raleigh, NC, USA, [email protected]. Their evaluations therefore leave open whether an LLM can write PBTs for programs whose inputs are text in the program’s own language, such as queries, configuration files, or programs. Such an input is valid only under the program’s semantics, and a test usually reaches the program through a process or a server. For such programs, a developer must write a generator that builds valid inputs, and an oracle that either predicts the result or checks it against a second computation. A PBT writer that generalizes would bring property-based testing to database servers, compilers, and system software. Compilers and databases already have mature testing tools [36, 41, 55, 67, 68, 69, 73, 93, 98], and such a writer would complement them with one tool that needs only a small harness per domain. We therefore ask: how well does an LLM-based PBT writer generalize across domains, and where does it fall short? To our knowledge, no study has evaluated such a writer on these programs or compared its tests across domains.

Our Work. We answer this question with a study of five domains: deep learning libraries, databases, system software, compilers, and LLM serving engines. We chose widely used domains that differ in how a test reaches the program and in the inputs it must build (§ 5). We study two programs per domain, ten programs in total: JAX [8] and PyTorch [58] (deep learning libraries), PostgreSQL [63] and ArangoDB [4] (databases), OpenSSH [54] and cronie [12] (system software), GCC [26] and Clang [45] (compilers), and vLLM [33] and SGLang [97] (LLM serving engines). Databases, system software, and compilers are reached only through a process or a server, which no prior evaluation covered. For these domains, we give the tools a harness that only runs the program and returns its output, e.g., a module that starts a database server and returns the rows of a query. We write one harness per domain, in about 30 minutes with an LLM’s help, and the tests of its two programs share it. The harness does not check the output. The LLM writes all generators, properties, and oracles. Deep learning libraries serve as a control: a test calls them directly with Python values and needs no harness, as in the prior evaluations. LLM serving needs no harness either: tests call the engines’ Python APIs. Besides the harness, we change only the 32 of Agentic-PBT’s 290 prompt lines that locate and read the target, and none of Proptest-AI’s prompts. Each test targets a focal point, one feature of a program, such as an SQL function or a compiler builtin.

The study has four research questions. RQ1 assesses performance and selects the writer that RQ2 to RQ4 study. We compare the two LLM-based PBT writers with each other on all ten programs, with two LLMs from different vendors, Claude Opus 5 and GLM-5.3-Flash. We measure branch coverage and examine failure reports. RQ2 studies the selected writer, Agentic-PBT, per domain: the oracles of its tests and the sources of their properties. RQ3 counts the confirmed bugs that the tests reveal, those that maintainers accept as real. Because an LLM may have seen a bug report during training, a previously reported bug counts only if its first public report postdates the knowledge cutoff of the model that found it. RQ4 compares Agentic-PBT with domain-specific tools in three domains, in branch coverage and in the crashes that each reveals: Centaur [52] and FlashFuzz [65] on PyTorch, MetaMut [55] and GrayC [21] on GCC and Clang, and SQLancer [68] on PostgreSQL.

Results. Agentic-PBT covers more branches than Proptest-AI on seven of the ten programs with Opus 5 and on all ten with GLM-5.3-Flash. Over 12 hours, the domain-specific tools cover more of the compilers and of PostgreSQL, where Agentic-PBT reaches 73% to 85% of the leading tool’s coverage, and Agentic-PBT covers 1.1 to 1.4 times as much of PyTorch as FlashFuzz and Centaur. On the compilers, MetaMut and GrayC find 23 and 4 crashes and Agentic-PBT 4, none in common. The documentation states the property of 78% of the tests, and 41% of their oracles compare the output with another run or another implementation rather than check a stated condition. We filed 180 bug reports from Agentic-PBT’s tests. Among them are 44 duplicates: bugs that others reported independently after the knowledge cutoff of the model that found them. Of the other 136 reports, maintainers have confirmed 65, in every domain and all ten programs, fixed 23, and rejected 3; the rest await a response. Most duplicates are in LLM serving, 36 of the 44.

Key Lessons. We highlight two of the lessons of § 7.1. First, low observability hurts PBT writers. A property can be documented and still be hard to check when the program does not show the behavior that the property concerns, which affects system-level tests most. A cronie test, for example, has to exceed one of cronie’s limits to reveal how cronie reads a crontab line. Second, mature fuzzers and LLM-based writers contribute different parts of a test. The fuzzers reach more of the compilers and of PostgreSQL, but they check crashes or, in SQLancer’s case, one relation between queries. The writers take most of their properties from the documentation, and most of the bugs Agentic-PBT found are wrong results that only such a property detects. Future work could have MetaMut or SQLancer check properties that an LLM extracts from the documentation.

We make three contributions: (1) a study of general-purpose LLM-based PBT writers on ten programs in five domains, most of which take text in their own language, which measures the coverage, failures, oracles, and bugs of their tests (§ 6); (2) lessons for building and evaluating such writers (§ 7.1); and (3) 65 new bugs confirmed by maintainers, and an artifact with the harnesses, prompts, scripts, tests, and bug reports of the study, so that others can repeat it with other models and writers (Data Availability).

Background

Property-Based Testing

Definition. Property-based testing checks that a program satisfies a property on many generated inputs [10]. A program f:X→Yf : X \to Y usually accepts only part of its input space XX, the inputs that satisfy its precondition prepre; we call them valid. Given a property PP, a property-based test approximates ∀x∈X:pre(x)⇒P(x,f(x))\forall x \in X : pre(x) \Rightarrow P(x, f(x)) by checking it on sampled inputs, because XX is usually too large to enumerate. A property-based test (PBT) has two parts. A generator samples inputs xx from XX; it should draw only valid ones. An oracle evaluates P(x,f(x))P(x, f(x)) on each sampled input xx and the program’s output f(x)f(x), usually as an assertion, and decides whether the execution is acceptable [5]. The test fails when the oracle rejects an input, and that input is a counterexample to the property. A failure on an invalid input, or on a property the program never promised, is usually a false alarm, not a bug.

PBT libraries. PBT libraries provide building blocks for generators and a runner that executes a test on many inputs; the user combines these building blocks into a generator and writes the oracle. Examples are QuickCheck [10] for Haskell and Hypothesis [49] for Python. In Hypothesis, a generator is called a strategy. The library provides strategies for common types, such as st.integers() and st.lists(), which can be combined into strategies for structured inputs. The @given decorator turns a test function with an assertion into a property-based test that runs the function on inputs drawn from the strategies. The @settings decorator configures the run; for example, @settings(max_examples = N) sets how many inputs to draw, 100 by default. When the test fails, Hypothesis shrinks the failing input to a smaller one that still fails.

Example. Figure 1 shows a PBT in Hypothesis for a function quicksort that sorts a list of integers. The decorator @given tells Hypothesis to draw inputs from the strategy st.lists(st.integers()), which generates lists of integers, including the empty list and lists with duplicate or negative elements. The test calls quicksort on each drawn list xsxs, and the oracle checks that the output ysys is ordered, a property of every correct sorting implementation. If quicksort misplaces an element, the assertion fails, and Hypothesis reports a shrunk failing list as the counterexample. This oracle is partial. A quicksort that drops elements still passes it; catching that bug also requires checking that ysy_s is a permutation of xsx_s.

A Hypothesis test that checks that quicksort’s output is ordered

Figure 1. A Hypothesis test that checks that quicksort’s output is ordered.

LLM-Based PBT Writers

Proptest-AI [79] and Agentic-PBT [48] are two recent LLM-based PBT writers for Python. Figure 2 shows their prompts.

Prompts of Proptest-AI and Agentic-PBT

Figure 2. The prompts of Proptest-AI [79], from its artifact [80], and Agentic-PBT [48], verbatim but abridged. [...] marks cuts; shading marks the parts filled in per target.

Stage 1 prompt

You are a world class Python programmer. Please review the following API documentation for {function_name}. Documentation: {api_documentation}. Extract a list of maximum five properties that you would use for property-based testing. Specifically, give properties on the output of the function, which may or may not depend on the input arguments. [...] Do not write any tests. Only write this list of five properties in natural language.

Stage 2 prompt, sent after the LLM’s reply

Now, write a Python property-based test suite using the library Hypothesis testing the properties you described. [...] In your generation strategy, be aware of very large inputs and overflows.

-------------- Output Format --------------

from hypothesis import given, strategies as st import {module_name}

@given(st.data()) def test_{function_name}_property(): (...)

End program

(a) Proptest-AI (two-prompt mode).

Property-Based Testing Bug Hunter

You are a bug-hunting agent focused on finding genuine bugs through property-based testing with Hypothesis. [...]

Your Todo List

  1. Analyze target 2. Understand the target

  2. Propose properties 4. Write tests

  3. Test execution and bug triage

  4. Report or conclude

Core Process

2. Understand the target [...]

3. Propose properties

  • Invariants - Round-trip properties

  • Inverse operations - Multiple implementations

  • Metamorphic properties [...]

Only test properties that the code is explicitly claiming to have. [...]

4. Write tests

  • sound: tests only inputs expected by the code

  • complete: tests all inputs expected by the code

If soundness and completeness are in conflict, prefer writing sound but incomplete properties.

5. Test execution and bug triage

Run your tests with pytest. For test failures, apply this bug triage rubric: [...] - Do callers of this code make assumptions that prevent this input? [...] If false alarm detected: Return to Step 4 [...]

6. Bug Reporting [report template] [...]

Now analyze the targets: $ARGUMENTS

(b) Agentic-PBT.

Proptest-AI. Proptest-AI [79] prompts an LLM with the documentation of one Python API method and asks for PBTs of that method. Its authors propose two prompting modes. The one-prompt mode uses one prompt, which asks for a list of properties and a test that checks them in the same answer. The two-prompt mode (Figure 2a) splits this work into two prompts, sent in sequence in one conversation. The Stage 1 prompt gives the method’s name ({function_name}) and its documentation from the library’s website ({api_documentation}), and asks the LLM to list up to five properties of the method’s output. The Stage 2 prompt, sent after the LLM answers, asks for a Hypothesis test suite that checks the properties the LLM just listed. It ends with a template that imports the method’s module ({module_name}) and names the test function. The authors evaluate Proptest-AI on 40 deterministic API methods from ten Python libraries, such as numpy, pandas, networkx, and datetime, with GPT-4, Claude-3-Opus, and Gemini-1.5-Pro. In their best configuration, GPT-4 with two prompts, 41.74% of the tests from Stage 2 run without error and pass on the library, against 31.04% for the next best, Claude-3-Opus with two prompts.

Agentic-PBT. Agentic-PBT [48] is a natural-language prompt (Figure 2b) that directs a coding agent, an LLM that runs commands and edits files, to write property-based tests for Python code, run them, and report bugs. The prompt takes one argument ($ARGUMENTS), the target: a Python module, file, or function, such as numpy.linalg. It lists six steps. The agent reads the target’s source code and documentation (steps 1–2). It proposes properties that the code claims in its docstrings, comments, or callers, and reads the callers to learn which inputs the target expects (step 3). It writes a Hypothesis test for each property (step 4). A strategy should draw only inputs the code expects; when it cannot also draw all of them, the prompt prefers missing valid inputs to drawing invalid ones. The agent then runs the tests with pytest and triages each failure with a rubric in the prompt. The rubric asks whether the failure reproduces, whether callers can pass the failing input, and whether the code claims the property (step 5). When the agent judges a failure a false alarm, it revises the test, its strategy or its property, and reruns it. It reports each remaining failure with the failing test, the shrunk input, a reproducing script, and, when possible, a patch (step 6). The authors run Agentic-PBT with Claude Opus 4.1 on 933 modules of 100 Python packages, from the standard library to numpy and pandas. The agent reports 984 failures as likely bugs. The authors review 50 of these alarms, sampled from the 80% that an LLM ranks highest, and judge 28 to be real bugs.

Differences. Table 1 compares the two tools. The key difference is that only Agentic-PBT runs its tests and revises those behind false alarms; Proptest-AI leaves both to the user. They also differ in where they find properties: Proptest-AI receives the documentation in its prompt, whereas Agentic-PBT searches the code, docstrings, comments, and callers itself.

Proptest-AI [79]Agentic-PBT [48]
TargetOne Python API methodA Python module, file, or function
Property sourceDocumentation, given in the promptCode, docstrings, comments, and callers it finds
LLM useOne prompt, or two in sequenceAgent loop with shell and file tools
Runs its testsNoYes, with pytest
Revises its testsNoYes, when a triage rubric flags a false alarm
OutputHypothesis test suiteHypothesis test suite, plus a written report for each failure it judges a bug
Evaluation40 methods, 10 libraries; GPT-4, Claude-3-Opus, Gemini-1.5-Pro933 modules, 100 packages; Claude Opus 4.1
Reported result41.74% of tests run and pass (best configuration)28 of 50 sampled alarms judged real by the authors

Table 1. Proptest-AI and Agentic-PBT as reported by their authors, whose results are not comparable.

Testing the Domains with PBTs

A domain is a class of programs that take the same kind of input and that a test controls and observes the same way [1], such as through a function call or a process. This section describes how a PBT reaches and checks the programs of each of the five domains we study, and shows example tests that Agentic-PBT wrote.

Test Characteristics

Table 2 describes the tests of each domain by three characteristics, which the domain and its focal points fix. Level is where a test meets the program: an API-level test calls a function, and a system-level test runs a program or sends input to a running one. Input language is the form of the test’s input, from Python values to text in the program’s own language, such as SQL, C, or a configuration file. Harness is the code, beyond the generator and oracle, that builds, starts, or feeds the program under test. API-level tests call the program directly and have no harness.

DomainLevelInput languageHarness
Prior evaluationsAPIPython valuesNone
Deep learning librariesAPIPython values (tensors)None
DatabasesSystemSQL; AQLStarts a server, sends queries
System softwareSystemConfiguration textRuns a command; starts sshd
CompilersSystemC and C++ programsCompiles and runs a program
LLM servingAPIPython values, prompts, and configurationsNone

Table 2. How a test reaches the programs of each domain.

What separates our domains from the prior evaluations are the three characteristics of Table 2. Deep learning libraries match the prior evaluations in all three. LLM serving also uses API calls without a harness, with inputs ranging from Python values to prompts and configurations. Databases, system software, and compilers differ in all three. In these three domains, the harness is one more function to call, since we write it and the writer reads its code, but the input is text in the program’s own language, valid only under the program’s semantics, and the oracle must know what that text means. The prior evaluations cover up to 933 Python modules [48], all called with Python values; our ten programs are fewer, but a sharper test of whether a writer generalizes.

The tests of Section 3.2 rely on two further notions. A constraint on the inputs is relational when it relates one input to another. Because Hypothesis’s built-in strategies draw each value on its own, a relational constraint usually requires a custom generator: built-in strategies combined with filter or map, or a function written with @st.composite. We name oracle families after the survey of Barr et al. [5], whose “assertions and contracts” and “pseudo-oracles” are our first two. A contract oracle checks one call against a condition on its outcome, including an expected output that the test computes from the input. A differential oracle compares the output with that of an independent implementation; it can be another program, another function or a different mode of the same program. A metamorphic oracle relates the outputs of two calls whose inputs differ in a known way. RQ2 (§ 6.2) adds two rarer families from the same survey.

Studied Domains

We select the five domains below for their popularity and so that they vary in Level, Input language, and Harness. For each of the three domains that differ from the prior evaluations, a figure shows one test that Agentic-PBT wrote, because these tests show what the writer must produce beyond a Python call: a query, a crontab, or a C++ program, and an oracle for it. Each figure omits the test’s imports, helper code, assertion messages, and checks that the program neither crashed nor timed out, and renames its generator custom_gen. A generator marked simplified omits values that the test never uses (a partition column in PostgreSQL’s rows, a flag in Clang’s tuple) and some forms of input: frames without an EXCLUDE clause in PostgreSQL, and month and weekday names (e.g., jan) and lists with an asterisk (e.g., *, 5) in cronie. For the other two domains, whose tests resemble those of the prior evaluations, we describe a test in prose. Each of the three tests shown in a figure checks a constraint stated in the program’s documentation, needs a custom generator to meet it, and fails on the program.

Deep Learning Libraries. JAX [8] and PyTorch [58] are Python libraries for building and training neural networks, and many techniques test such libraries [15, 16, 39, 60, 85, 88, 95]. Their APIs take as inputs tensors, which are multi-dimensional arrays with a shape and an element type. A test calls an API directly. For example, Agentic-PBT’s test of jax.lax.pmean, which averages a tensor over groups of devices, must draw groups that “cover all axis indices exactly once” [76], with a randomly drawn list of lists of device IDs. Its custom generator draws one group label per device and groups the devices with the same label, and its oracle computes the mean of each group in NumPy and compares it with the output of the API.

Databases. A database server answers queries over its tables, and testing database servers has its own line of research [67, 68, 69, 73, 98]. We study PostgreSQL [63], queried in SQL, and ArangoDB [4], queried in AQL; a test writes a query as text, sends it to a running server, and checks the rows. Figure 3 shows a test of PostgreSQL’s nth_value(v, n), which returns the value in the nn-th row of a window frame, a range of rows around each row. The test fills a query template with a table, a sort order, a frame clause, and a row index nn. Built-in strategies draw all but the frame clause, which needs a custom generator, custom_gen, because its two ends constrain each other. The generator orders the kinds of the two ends, as the manual requires when it states that “the

Property-based test for nth_value

Figure 3. PostgreSQL (database). Agentic-PBT’s test for nth_value. run sends a query to the server through the harness, and values_clause writes the rows as a VALUES list. Each row holds a sort key k and a value v.

# Property-based test
@settings(max_examples=400, deadline=None)
@given(rows=tables, frame=custom_gen(), order=orderings, n=st.integers(1, 6))
def test_nth_value_matches_frame(rows, frame, order, n):
    r = run(f"SELECT nth_value(v, {n}) OVER w, array_agg(v) OVER w "
            f"FROM ({values_clause(rows)}) t(k, v) WINDOW w AS (ORDER BY {order} {frame})")
    assume(r.ok)
    for nv, agg in r.rows:
        reference = agg[n - 1] if len(agg or []) >= n else None
        assert nv == reference
# Custom generator (simplified)
@st.composite
def custom_gen(draw):
    kinds = ["UNBOUNDED PRECEDING", "{} PRECEDING", "CURRENT ROW", "{} FOLLOWING", "UNBOUNDED FOLLOWING"]
    start = draw(st.integers(0, 3))
    end = draw(st.integers(1, 4).filter(lambda e: e >= start))
    mode = draw(st.sampled_from(["ROWS", "RANGE", "GROUPS"]))
    excl = "EXCLUDE " + draw(st.sampled_from(["NO OTHERS", "CURRENT ROW", "GROUP", "TIES"]))
    lo, hi = kinds[start].format(draw(st.integers(0, 4))), kinds[end].format(draw(st.integers(0, 4)))
    return f"{mode} BETWEEN {lo} AND {hi} {excl}"
keys, vals = st.one_of(st.integers(0, 4), st.none()), st.one_of(st.integers(-9, 9), st.none())
tables = st.lists(st.tuples(keys, vals), min_size=1, max_size=9)
orderings = st.sampled_from(["k", "k DESC", "k NULLS FIRST", "k DESC NULLS LAST"])
# Example input (reveals a failure)
rows = [(0, None), (0, 0), (None, 0)]
frame = "ROWS BETWEEN 1 FOLLOWING AND UNBOUNDED FOLLOWING EXCLUDE TIES"
order, n = "k", 1
# Property-based test
@settings(max_examples=300, deadline=None)
@given(custom_gen, st.integers(1_000_000_000, 2_000_000_000), st.integers(60, 5 * 24 * 3600))
def test_all_jobs_in_interval_match_documented_semantics(fields, start, window):
    r = cmd_harness.run("cronnext", ["-t", str(start), "-q", str(start + window), "-f", "tab"],
                        files={"tab": " ".join(fields) + " echo job\n"})
    reference = Entry(*fields).matches_in((start + 59) // 60 * 60, (start + window) // 60 * 60)
    assert sorted(int(m) for m in NEXT_RE.findall(r.stdout)) == reference
# Custom generator (simplified)
def field(lo, hi):
    rng = st.tuples(st.integers(lo, hi), st.integers(lo, hi)).filter(lambda ab: ab[0] <= ab[1])
    rng = rng.map(lambda ab: "%d-%d" % ab)
    stepped = st.tuples(rng, st.integers(2, hi - lo)).map(lambda rs: "%s/%d" % rs)
    plain = st.one_of(st.integers(lo, hi).map(str), rng, stepped)
    star = st.one_of(st.just("*"), st.integers(2, hi - lo).map(lambda d: "*/%d" % d))
    return st.one_of(star, st.lists(plain, min_size=1, max_size=3).map(", ".join))
custom_gen = st.tuples(field(0, 59), field(0, 23), field(1, 31), field(1, 12), field(0, 7))
# Example input (reveals a failure)
fields = ("*", "*", "*", "*", "*/2")
start, window = 1_000_000_000, 80000

frame_end choice cannot appear earlier […] than the frame_start choice does [64]. Sort keys draw from five values and NULL, so rows often tie, as in the example input (that reveals a failure); ties matter, in particular because the frame’s EXCLUDE options drop the current row or the rows tied with it. The test discards queries the server rejects, so it cannot detect a server that wrongly rejects a valid query. Its oracle is differential: it computes array_agg(v), another function of the server, over the same frame in the same query and checks that nth_value equals its n-th element, or NULL when the frame holds fewer than n rows. The test fails for n=1n = 1 and the frame ROWS BETWEEN 1 FOLLOWING AND UNBOUNDED FOLLOWING EXCLUDE TIES: for the first of two tied rows that a third row follows, array_agg returns the third row’s value, but nth_value returns NULL [62].

# Property-based test
@settings(max_examples=100, deadline=None)
@given(custom_gen())
def test_ptr_align_constfold_matches_runtime(c):
    ba, al, sz, off, op = c
    reference = (off + al - 1) // al * al if op == "up" else off // al * al
    assume(0 <= reference < sz)
    r = cc_harness.compile_and_run(PROG %
    assume(r.compiled)
    ct_diff, ct_read, rt_diff, rt_read = map(int, r.run_stdout.split())
    assert (rt_diff, rt_read) == (reference, reference %
    assert (ct_diff, ct_read) == (rt_diff, rt_read)
# Custom generator (simplified)
@st.composite
def custom_gen(draw):
    ba = 1 << draw(st.integers(0, 7))
    al = 1 << draw(st.integers(0, 7))
    sz = draw(st.integers(1, 4)) * ba
    off = draw(st.integers(0, sz - 1))
    return ba, al, sz, off, draw(st.sampled_from(["up", "down"]))
# Example input (reveals a failure)
ba, al, sz, off, op = 2, 2, 2, 1, "down"

// Failing input, rendered: the program that PROG yields for the values above
struct S { alignas(2) unsigned char d[2]; };
constexpr S g = mk();                         // g.d[i] == i
constexpr const unsigned char *cp = __builtin_align_down(g.d + 1, 2);
constexpr int ct_diff = (int)(cp - g.d);
constexpr int ct_read = (int)*cp;
__attribute__((noinline)) const unsigned char *rt(const unsigned char *p, unsigned long a) {
    return __builtin_align_down(p, a);
}
volatile int voff = 1;
int main() {
    const unsigned char *p = rt(g.d + voff, 2);
    printf("%
}

System Software. OpenSSH [54] and cronie [12] are Unix programs for remote login and job scheduling, and system software of this kind has been a target of fuzzing research [61]. Each program reads a configuration file, such as sshd_config or a crontab, in a format that its manual page defines. A test writes a configuration file, runs a program on it, and parses what the program prints. Figure 4 shows a test of cronie’s program cronnext, which lists the times in a window at which a crontab’s job runs. The test fills a crontab line template with five generated time fields and draws the window with built-in strategies. A field can be an asterisk, a number, a range, a step, or a list of these, and crontab(5) states that in a range “the first number must be less than or equal to the second one” [14]. The test therefore uses a custom generator, custom_gen, that draws each field within its bounds, keeps only a range whose ends are in order, and draws a step no larger than the field’s span; the figure ends with the draw on which the test fails. The test writes the line to a file and runs cronnext on it through the harness. Its oracle is a contract, because code of the test computes the expected output: the test compares the times that cronnext lists with those that a 92-line Python model of crontab(5) computes, which AGENTIC-PBT also wrote. The test fails on the line * * * * */2. crontab(5) states that “Step values are also permitted after an asterisk,” so */2 in the day-of-week field names every second day from Sunday: Sunday, Tuesday, Thursday, and Saturday. The job must therefore not run on a Monday, but for a window that starts on a Sunday, cronnext also lists midnight of the following Monday [13].

Property-based test for cron command lines

Figure 4. cronie (system software). Agentic-PBT’s test for the focal point “cron command lines,” a section of crontab(5). Entry is the model of crontab(5), and its method matches_in lists the minutes at which the job runs; NEXT_RE matches a time in the output of croniext, and cmd_harness is the harness.

Compilers. Clang [45] and GCC [26] compile C and C++ programs, and many techniques generate programs to test compilers [9, 36, 41, 93]. A test compiles and runs a small program, which must be free of undefined behavior, because the C and C++ standards allow any output for such a program. Figure 5 shows a test of Clang’s __builtin_align_up and __builtin_align_down, which round a pointer to a multiple of an alignment and, the manual states, “are evaluatable inside constant expressions” [75]. The test fills a C++ program template, PROG, with five generated values: the alignment and size of an array whose byte ii holds ii modulo 256, an offset into it, the alignment to round to, and the direction. The program rounds the pointer at compile time and at run time and prints, for each, the result’s offset and the byte it points to. The manual states that if the alignment “is not a power of two at run time, the behavior of these builtins is undefined,” and that the result “must be within the same underlying allocation or one past the end” [75]. The custom generator, custom_gen, therefore draws both alignments as powers of two, which alignas also requires of the array’s, the size as a multiple of the array’s alignment, and an offset inside the array. The test discards a draw whose result leaves the array, even one past the end, because the program reads through the pointer. It also discards a program that fails to compile, so it cannot detect a compiler that wrongly rejects a valid program. Its oracle is differential: the test requires the compile-time results to equal the run-time ones. It also computes the rounded offset in Python and requires the run-time offset and byte to match it, which is a contract. The test fails when baba, alal, and szsz are 2, offoff is 1, and opop is down: both offsets are 0 and the run-time read returns byte 0, but the compile-time read returns byte 1 [44].

Clang compiler test code

Figure 5. Clang (compiler). AGENTIC-PBT’s test for “Alignment builtins” in Clang’s manual, and below, the program that the template PROG yields for the failing input (#include and mk’s body omitted).

LLM Serving. An LLM serving engine answers prompts with a model: it batches requests, applies the fine-tuned weights that a request asks for, and streams each response as it is generated. Testing serving engines is a recent line of work [40, 96]. We study vLLM [33] and SGLang [97], whose tests call Python APIs of the engines and their components or start a whole engine. For example, Agentic-PBT checks that a response does not depend on the other requests that the engine handles [83], that streaming does not change it [72], and that the engine applies the fine-tuned weights that it accepts [82]. Because a response has no single correct value, these properties compare two runs whose responses must be equal, as with different batches or streaming modes, or must differ, as with and without the fine-tuned weights.

Evaluation

We evaluate LLM-based PBT writers with the following research questions:

  • RQ1 [Performance]: How do the two LLM-based PBT writers compare in coverage and failures?

  • RQ2 [Characterization]: What characterizes the property-based tests that AGENTIC-PBT writes?

  • RQ3 [Usefulness]: How useful is AGENTIC-PBT in finding developer-confirmed bugs?

  • RQ4 [Domain-Specific Tools]: How much do AGENTIC-PBT’s tests and domain-specific fuzzers cover, and which crashes does each find that the other does not?

RQ1 evaluates the performance of the writers across the five domains and selects one to study further. RQ2 characterizes the tests by the families of oracles they check and the artifacts that state their properties. RQ3 measures usefulness by the bugs that developers confirm. RQ4 compares AGENTIC-PBT with domain-specific tools in branch coverage and in the crashes that each reveals.

Targets and focal points. Table 3 lists the ten programs of § 5.2, two per domain, which we call targets, and the version of each that we test; RQ3 tests older versions of the same targets (Table 6). A writer tests a target one focal point at a time. Its kind depends on the domain: an API method or class of a Python library, a function of a query language, an option or keyword of a configuration file, a construct of the crontab syntax, or a language extension of a compiler. We extract each target’s focal points from the documentation in the table’s Source column, and the last column gives their number.

DomainTargetVersionSourceExampleNumber
Deep learningJAX0.11.2API referencejax.numpy.isin1,558
PyTorch2.14.0API referencetorch.cumsum4,107
LLM servingvLLM0.30.0API referenceRepetitionDetectionParams4,399
SGLang0.5.20docstringsSiglip2Encoder5,716
DatabasePostgreSQL18.6function referencejsonb_each_text634
ArangoDB3.12.12function referenceDATE_ISOWEEK255
System softwareOpenSSH10.5p1manual pagesAddressFamily167
cronie1.7.2manual page@yearly39
CompilerGCC17 (dc301f9)manualnested functions1,000
Clang24 (4057f8e)manualfallthrough487

Table 3. The ten targets of RQ1 and RQ2 (RQ4 uses four of them), the documentation from which we extract their focal points, one example, and their number.

DomainTargetVersionReportedConfirmedRejectedPendingAwaitingFixedDuplicate
Deep learningJAX0.10.138281914113
PyTorch2.12.0128-4413
LLM servingvLLM0.28.03327-64320
SGLang0.5.183317115-116
DatabasePostgreSQL18.691-8-1-
ArangoDB3.12.4235-18-41
System softwareOpenSSH8.9p151-4-1-
cronie1.7.21313--121-
CompilerGCC17 (33097f2)53113--
Clang22.1.896-35-1
Total180109368422344

Table 6. Bugs Agentic-PBT found and we reported so far, by domain, target, and release.

RQ1: Performance of PBT Writers

RQ1 compares PROPTEST-AI and AGENTIC-PBT on ten programs in five domains by their tests, cost, branch coverage, and failures, and selects the writer that RQ2 to RQ4 study.

Setup. Sampling focal points. For each target, we sample 30 of its documented focal points (Table 3) uniformly at random; on PyTorch, the 30 are the first 30 of the 50 APIs that RQ4 draws uniformly from the 368 that both of its fuzzers support. Both writers test the same focal points, which the artifact lists. Writers. We run both writers with two LLMs: Claude Opus 5, a proprietary frontier model, and GLM-5.3-Flash [94], an open-weights model from another vendor. We chose GLM-5.3-Flash because it comes from a vendor other than Anthropic, whose researchers developed AGENTIC-PBT, because its weights are open, so others can rerun the study without a proprietary API, and because it costs much less to run. The second model tests whether the comparison between the writers holds with another model. It does not test whether the writers perform equally well with both. PROPTEST-AI receives the documentation of each focal point. Following its original evaluation [79], we use its two-prompt mode (§ 4.3) and repeat the procedure five times per focal point.

Harness. On the six targets outside Python, both writers use the harness of the target’s domain (Table 2); for PROPTEST-AI, we append a description of the harness to the documentation, without changing its prompts. We wrote each harness in about 30 minutes with the help of an LLM. The three harnesses total 511 lines of Python, 1.8% of the test code that AGENTIC-PBT writes for these targets.

Execution environment. Each session, one run of the agent on one focal point, runs in its own container. The agent can read the target’s source code, build, and harness, and can write only to its working directory. RQ1 thus compares the writers as shipped: AGENTIC-PBT holds strictly more information than PROPTEST-AI, since it reads the code and the harness and runs its tests with execution feedback, whereas PROPTEST-AI sees only the documentation and the harness description, in five two-prompt runs. Budget. Each writer fuzzes its tests of a target for 30 minutes on 20 CPU cores, split evenly among its tests, so both writers spend the same fuzzing time on each target. We rerun each PBT with a new seed after it finishes or fails, until its allocated time ends. Each generation session is capped at 15 minutes. We do not cap the LLM cost, and we report it.

Number of Tests Generated and Cost. Table 4 gives, per target, the valid tests that each writer produces (property-based tests that pytest can run) and the LLM cost of producing them, with each LLM. For Proptest-AI, we sum the valid PBT functions across its five responses per focal point. With Opus 5, Agentic-PBT writes about a third as many tests as Proptest-AI, 2,352 against 6,853, at a similar cost, $670.6 against $645.5, or about $2.2 per focal point for each. With GLM-5.3-Flash, Agentic-PBT writes 1,633 tests for $12.9 and Proptest-AI 5,479 for $1.1. With Opus 5, Agentic-PBT also writes example-based tests for 41 of the 300 focal points, such as bug reproductions.

TargetOpus 5GLM-5.3-Flash
Agentic-PBTProptest-AIAgentic-PBTProptest-AI
Tests∣Tests∣| Tests |Tests∣Tests∣| Tests |
JAX31551.767741.72251.15490.1
PyTorch43654.766130.92410.76100.1
vLLM20262.570045.11501.04820.1
SGLang19168.254049.61731.23640.1
PostgreSQL25673.073375.31731.36120.1
ArangoDB24459.569951.61171.46180.1
OpenSSH20464.770287.01702.06170.1
cronie14266.771690.81351.55610.1
GCC21887.772276.91351.45090.1
Clang14481.970396.81141.35570.1
Total2,352670.66,853645.51,63312.95,4791.1

Table 4. Valid PBTs and LLM cost ($) per writer and LLM.

Coverage. We measure the branch coverage of each target as a whole, on coverage-instrumented builds (coverage.py for Python code, gcov or llvm-cov for C and C++). Figure 6 plots the coverage of each target over the fuzzing time, with the branches covered on the y-axis and the minutes on the x-axis. It has one column per domain, with the domain’s two targets one above the other. The color of a line gives the writer, and its style gives the LLM, solid for Claude Opus 5 and dashed for GLM-5.3-Flash. A dot ends each line and is labeled with its final coverage. With Opus 5, Agentic-PBT covers more branches than Proptest-AI on seven of the ten targets: by 33% to 136% on the four API-level targets, and by 2% to 8% on PostgreSQL, ArangoDB, and OpenSSH, where the margin on OpenSSH is barely above the variation between fuzzing runs reported below. Proptest-AI covers 5% to 9% more on GCC, Clang, and cronie. On GCC, switching to GLM-5.3-Flash reduces Proptest-AI’s coverage from 296k to 169k branches, against 282k to 251k for Agentic-PBT. With GLM-5.3-Flash, both writers cover less, and Agentic-PBT covers more than Proptest-AI on all ten targets.

Figure 6. Branch coverage over time, Opus 5 solid and GLM-5.3-Flash dashed; a band spans four fuzzing runs. Titles give each target’s total branches (Python code for JAX, vLLM, and SGLang; libtorch_cpu for PyTorch).

Each curve is one generation run. Generating the tests of both writers with Opus 5 costs about $1,300 in LLM fees (Table 4), so we do not repeat the generation. We instead repeat the fuzzing, which costs only machine time, and report the margins between the writers. We use the second LLM to examine how the comparison changes with model choice. We rerun the fuzzing of every target’s tests three times with new seeds, and the bands in Figure 6 span the four runs. These repetitions measure fuzzing variability for the generated tests. With Opus 5, a writer’s final coverage varies by at most 1.7% on eight targets, by 4.8% on vLLM, and by 10.1% on PostgreSQL, where one run of Agentic-PBT covers 9% less than the other three; the writers’ ranges overlap only on PostgreSQL. With GLM-5.3-Flash, a writer’s coverage varies by at most 1.2% on nine targets and by 6.6% on PostgreSQL, and no ranges overlap.

Failures. We examine failures from the initial fuzzing run with Opus 5, which gives both writers higher coverage. Within each PBT, we deduplicate failures by exception type and error message. We replace variable parts of messages, such as numbers and memory addresses, with #. For example, rows: [(1.0,), (0.0,)] becomes rows: [(#,), (#,)]. For crashes, we compare signals and stack traces. After setting the criteria, two authors independently classify 30 randomly sampled unique failures per writer as true or false positives. Their labels show substantial agreement (κ=0.69\kappa= 0.69) [35]. We resolve disagreements by discussing the evidence for each label. We judge a failure to be a true positive when it violates documented behavior, exposes a security defect, or disagrees with another implementation where both specify the same behavior. We exclude incorrect test expectations, including expectations about behavior left unspecified by the documentation. A true-positive example is torch.special.hermite_polynomial_he, which violates its documented recurrence. We use the final labels to calculate the true-positive rates. Table 5 reports the unique failures and sampled true-positive rates: 9 of 30 for Agentic-PBT and 4 of 30 for Proptest-AI. The 95% Wilson confidence intervals are 16.7–47.9% and 5.3–29.7%, respectively [86]. Scaling the rates and their intervals by the unique failure counts gives about 190 (104–300) and 170 (69–385) true positives, respectively.

WriterUniqueTP (%)
Agentic-PBT62730
Proptest-AI1,29813

Table 5. Unique failures and sampled true-positive (TP) rates, with Opus 5; 30 failures sampled per writer.

Answer to RQ1: Agentic-PBT covers more branches than Proptest-AI on all ten targets with GLM-5.3-Flash and on seven with Opus 5, where Proptest-AI leads by 5–9% on GCC, Clang, and cronie. With Opus 5, Agentic-PBT writes about a third as many tests. Its sampled true-positive rate is 30%, against 13% for Proptest-AI, but the small samples cannot confirm a difference. We select Agentic-PBT for its higher coverage on most targets and because it alone runs and triages its tests, and Opus 5 for its higher coverage than GLM-5.3-Flash.

RQ2: Characteristics of the Property-Based Tests

RQ2 characterizes the property-based tests that Agentic-PBT writes with Opus 5, the configuration that RQ1 selects. We measure two characteristics of a test: the family of its primary oracle, which says what the test checks, and the artifact that states its property, which says whether the program’s documentation or its code states the property.

Setup. The population is the 2,622 property-based test functions of this configuration: the 2,352 tests of RQ1 (Table 4) and 270 tests on 20 further PyTorch focal points of RQ4 (§ 6.4). We sample 500 tests, 50 per target. We visit a target’s focal points in a seeded random order and take the first test of each, then the second test of each, and so on until we have 50, so no test is drawn twice. This draw underrepresents a focal point with many tests, so every share weights a sampled test by the number of tests of its focal point divided by the number we sampled from it [11]. A target’s weight thus sums to its number of tests, and a share over all tests weights each target by its tests. For the 95% confidence interval of a share, we use a normal interval with Kish’s effective sample size (∑w)2/∑w2\sum w)^2 / \sum w^2 over the weights ww of the sampled tests [32], which treats the sampled tests as independent draws. The interval is at most ±12.4% within a domain and ±6.0% over all tests, so a difference between two domains is significant only when it exceeds 17 points.

Labels. The primary oracle of a test checks its intended property, as opposed to checks such as the type of an output the harness expects. Its family is contract, differential, or metamorphic, which § 5.1 defines after Barr et al. [5]. Two rarer families of the same survey also occur: algebraic (an operation followed by its inverse returns the original value) and implicit (the program does not crash). The primary source of a test’s property is the artifact that states it most directly: documentation (manuals and docstrings), code (the code under test and related code, such as a caller), the project’s tests, or another artifact, such as a standard or a reference implementation. When a test has several checks, its primary oracle is the one that the test’s name and docstring describe, or else its last assertion on the program’s output. The cronie test of Figure 4, for example, has a contract as its primary oracle, because the expected times come from a model that the test itself computes, not from another implementation.

Methods. An LLM agent (Claude Opus 5.5) labels the 500 tests. A second agent from another vendor (Gemini 3.1 Pro) labels an audit sample of 100, a seeded random draw of ten tests per target, without seeing the first agent’s labels. An author decides each disagreement. For the oracle, an agent picks one of ten kinds of check from a closed list. A script maps the kind to its family, so no labeler chooses a family directly; the artifact lists the ten kinds and their families. The two agents agree on the family for 82% of the audit sample (κ=0.72\kappa= 0.72). For the source, a label names the artifact, the location, and the passage that states the property. A script finds that passage in the writer’s transcript for 473 of the 478 tests whose label names an artifact; the other 5 passages appear only in the test itself, and the remaining 22 tests name no artifact. The agents agree on the source class for 75% (κ=0.49\kappa= 0.49). In 15 of the 25 disagreements the second agent finds no artifact, and the author sides with the first agent in 21 and with the second in 4. The agreement shows that the two LLMs label consistently, not that the labels are correct, and the confidence intervals above cover sampling error only, not errors that both LLMs make.

Results. Figure 7 shows the families and the sources per domain. Contract oracles are the most common family in every domain, 54% of all tests, from 40% in the databases to 72% in LLM serving. Of all tests, 47.5% check an invariant of one output, 6.1% check that the program rejects an invalid input, and 0.8% check a documented value. The other 46% of the tests have a primary oracle of another family: metamorphic oracles relate two runs (21.3%), differential oracles compare with another implementation (19.3%), algebraic oracles check a round trip (3.5%), and implicit oracles check only that the program does not crash (1.5%). A test can also check more than its primary oracle, and Figure 7 counts only the primary one. The metamorphic and differential oracles together are the tests that compare a run with another run or implementation, 41% of all tests. They are fewest in LLM serving (22%) and most in the deep learning libraries (51%) and the databases (51%). What they compare with depends on the domain. In the deep learning libraries, tests compare with a reference library such as NumPy (27% differential). In system software, they check equivalences that the manual states (24% metamorphic), such as between a crontab line starting with @weekly and one starting with 0 0 * * 0.

Figure 7. Oracle families (a) and property sources (b), in weighted % of tests.

Documentation states the property of 78% of all tests: a docstring or text the program prints for 41%, and a separate file for 37%. Code states 14%. The labels name no artifact for 5%: for 3.2% the labeler found no passage that states the property, for 0.9% it judged that the writer used general knowledge, and 0.7% check only that valid inputs do not crash. LLM serving is the exception where documentation states 56% of the properties and code 38%. In the other four domains documentation states 78% to 89% and code 8% to 14%.

Answer to RQ2: AGENTIC-PBT’s tests check what the program documents: documentation states 78% of the properties, and 54% of the primary oracles check a condition on one outcome. Most of the rest compare with a second run or implementation: the deep learning tests compare with a reference library, and the system software tests check equivalences that the manuals state. The exception is LLM serving, where the code states 38% of the properties.

RQ3: Developer-Confirmed Bugs

Setup. Sessions. We run Agentic-PBT with Opus 5, the configuration that RQ1 selects. The target versions are mostly older than RQ1 and RQ2, because we began the bug hunting campaigns earlier; Table 6 also counts the 30 bugs that RQ1’s sessions revealed on the versions of Table 3. We run one session per focal point, and we choose the focal points from the target’s documentation, its test files, and the pull requests that fixed bugs. The agent receives only the focal point’s name, such as jax.numpy.histogram. Outside Python it also receives the harness of RQ1 (§ 6.1). Bug hunting uses a longer session cap than RQ1, so that a session can explore its focal point further: 20 minutes. LLM serving engines are capped at 1 hour to give the engine time to start. The agent reads a shallow checkout of the pinned commit, with no Git history, in a container whose only network access is the model’s API. These controls block online bug reports and the target’s history. The one attempt to fetch a web page was denied.

Fuzzing and triage. After generation, we fuzz every test without an LLM for one hour per focal point on one CPU core. A failure is a test that violates its oracle or crashes. We deduplicate crashes by their top two stack frames and other failures by test and exception type. An LLM agent discards false positives and merges the rest into distinct candidate bugs. We read a candidate ourselves, reproduce it on the newest release of the target (on the compilers, the development head), and report it with a standalone reproducer. A reported candidate is a bug, which developers confirm, reject, or leave pending. A confirmed bug is a duplicate if someone outside this project reported it before a session found it, and new otherwise; new bugs are fixed or await a fix. As in § 1, a previously reported bug counts only if its first public report postdates the training cutoff of the model that found it. When Opus 5 finds a bug that was first reported before its cutoff (May 2026), we rerun the focal point in an isolated session with Opus 4.8, whose cutoff is January 2026. We count the bug only if Opus 4.8 finds it again and the report postdates January 2026, and otherwise we set the bug aside.

Measures. We report new confirmed bugs per focal point, leaving duplicates out to measure the ability to find new bugs, and the rejection rate per domain, the share of reported bugs that developers rejected. Candidates we have not read or reported, and pending reports, make the confirmed rate a lower bound. Because the focal points’ sources and the targets’ versions differ by target, the rates describe each domain’s run and do not rank the domains against one another.

Bugs. Table 6 shows the bugs we reported per domain and target, by status: confirmed, rejected, or pending. It splits the confirmed ones into those awaiting a fix, those fixed, and the duplicates. We have reported 180 bugs on 10 targets: 109 are confirmed, 68 pending, and 3 rejected. Of the confirmed bugs, 44 are duplicates, which leaves 136 reports not known to duplicate an earlier one; of these, maintainers confirmed 65 new bugs, of which 23 are fixed. Only 24 of the 180 reported bugs are crashes, uncaught exceptions, or hangs. The other 156 are wrong results, which a test detects only through its oracle. Our experiments cover 742 focal points in total: 180 in the deep learning libraries, 153 in LLM serving, 240 in the databases, 77 in system software, and 92 in the compilers.

ArangoDB’s SUBSTITUTE

Figure 8a. ArangoDB’s SUBSTITUTE.

Clang’s vectorcall attribute

Figure 8b. Clang’s vectorcall attribute.

We read and reported 0.24 (180/742) bugs per focal point, and the developers have confirmed 0.09 (65/742) new bugs per focal point, or 0.15 (109/742) counting the duplicates. In LLM serving, 36 of the 44 confirmed bugs are duplicates, which leaves 0.05 (8/153) new bugs per focal point, the second-lowest rate after the databases’ 0.02 (5/240); system software has the highest, 0.18 (14/77). The developers rejected 2% (3/180) of the reports: 7% (1/14) in the compilers, 2% (1/50) in the deep learning libraries, 2% (1/66) in LLM serving, and none in the databases and system software. In the databases, 26 of 32 reports are still pending (8 of PostgreSQL’s 9), so their 0.02 new confirmed bugs per focal point is a lower bound, which rises to 0.13 if maintainers confirm every pending report as new.

Examples. Figure 8 shows two of these bugs as minimal reproducers. Figure 8a shows a bug in ArangoDB’s AQL function SUBSTITUTE(value, search, replace), which, as its manual states, replaces “all occurrences of search” in value with replace. The test draws the three strings from lowercase letters, runs the query through the harness, and requires the result to equal that of Python’s str.replace, without an exception and without a warning. It fails when value is empty: SUBSTITUTE returns null with a warning instead of the empty string. The developers confirmed the bug, merged a fix the next day, and released it in ArangoDB 3.12.12 [3].

Two minimal reproducers

Figure 8. Two bugs that Agentic-PBT found, as minimal reproducers.

Figure 8b shows a bug in Clang’s vectorcall attribute, which, as Clang’s manual states, “changes the calling convention of a function,” i.e., the way a call passes the function’s arguments. The test draws a return type and up to nine parameter types from scalar, vector, and structure types. It then builds a program that defines the same function with and without the attribute. Through the harness, it compiles and runs the program, and requires that both functions return the same value without crashing. It fails when a parameter is a long double: Clang crashes with an assertion failure during code generation instead of compiling the function. An LLVM developer labeled our report as confirmed [43].

Answer to RQ3: Developers confirmed bugs in all five domains. Of the 180 bugs we reported, 109 are confirmed, 65 of them new and 23 fixed, and 3 rejected (2%). Only 24 are crashes, uncaught exceptions, or hangs; the rest are wrong results that only an oracle detects. New confirmed bugs per focal point range from 0.02 in the databases and 0.05 in LLM serving, where most confirmed bugs were already known, to 0.18 in system software.

RQ4: Comparison with Domain-Specific Tools

RQ4 asks how much Agentic-PBT’s tests and the fuzzers built for three of our domains cover, and which crashes each finds that the other does not. The fuzzers’ oracles differ from a property-based test’s, so other failures are not comparable: MetaMut and GrayC detect compiler crashes, Centaur and FlashFuzz PyTorch crashes, and SQLancer crashes and one fixed relation between queries.

Setup. Tools and targets. We compare with MetaMut [55] and GrayC [21] on GCC and Clang, with SQLancer [68] on PostgreSQL, and with Centaur [52] and FlashFuzz [65] on PyTorch. We omit JAX, which the evaluated versions of Centaur and FlashFuzz do not support, and ArangoDB, whose support SQLancer has removed [74]. Each tool runs in its default configuration, and the artifact lists the versions. MetaMut and GrayC start from the seed corpora of their releases. Budget. Agentic-PBT uses Opus 5 and RQ1’s generation setup. Each tool then runs once for 12 hours on 20 CPU cores on the instrumented builds of RQ1; the 12 hours cover fuzzing only, and Agentic-PBT’s test generation (Table 4) precedes them. Agentic-PBT’s tests share the budget equally and run one after another on each core, so its early coverage is lower than in RQ1 (Figure 6). Coverage. On PyTorch, all three tools test the same 50 APIs, sampled from those that both fuzzers support, and we measure each tool’s branch coverage only during calls to these APIs, as Centaur’s evaluation does [52]. On the other targets, we measure the coverage of the whole program. Crashes. We replay the inputs that Agentic-PBT, MetaMut, and GrayC saved on the evaluated compiler builds, replay FlashFuzz’s inputs on PyTorch, and check the child process’s signal for Agentic-PBT’s PyTorch crash. We count the crashes that reproduce, whether or not they were reported before, and deduplicate them per compiler and tool by the first two function names in the stack trace.

Coverage. Figure 9 shows the coverage over 12 hours. MetaMut covers the most branches on GCC and Clang; GrayC reaches 82–86% of its coverage and Agentic-PBT 73–79%. SQLancer leads on PostgreSQL with 61,984 branches, of which Agentic-PBT reaches 85%. Agentic-PBT leads on PyTorch with 6,670 branches, 1.1 times FlashFuzz’s coverage and 1.4 times Centaur’s. On the compilers and PostgreSQL, Agentic-PBT’s tests target 30 focal points per program while the fuzzers exercise the whole program, so whole-program coverage favors the fuzzers; on PyTorch, all three tools are confined to the same 50 APIs.

Figure 9. Branch coverage over 12 hours, one run per tool.

Crashes. Table 7 gives the reproduced crashes per tool and target. On the compilers, MetaMut finds the most, 7 on GCC and 16 on Clang, against Agentic-PBT’s 2 and 2, and no crash signature of Agentic-PBT matches one of MetaMut’s or GrayC’s. Agentic-PBT’s tests do not seek crashes; its four are property tests that crash the compiler instead of failing. On PyTorch, Agentic-PBT and FlashFuzz find the same crash, in PyTorch’s GRUCell module; Centaur finds none. No tool crashes PostgreSQL. Agentic-PBT’s two Clang crashes include the vectorcall crash of RQ3 (Figure 8b), whose test an RQ1 session wrote; the counts also include crashes reported before our runs, such as GrayC’s crash on huge by-value arguments [42].

ToolGCCClangPyTorchPostgreSQL
Agentic-PBT2210
MetaMut716––
GrayC13––
FlashFuzz––1–
Centaur––0–
SQLancer–––0
Shared0010

Table 7. Reproduced crashes per tool, and those shared with Agentic-PBT (–: not run on the target).

Answer to RQ4: The domain-specific fuzzers cover more of GCC, Clang, and PostgreSQL, where Agentic-PBT reaches 73–85% of the leader’s coverage; Agentic-PBT covers more of the sampled PyTorch APIs. MetaMut and GrayC find 23 and 4 compiler crashes to Agentic-PBT’s 4, and none of Agentic-PBT’s matches one of theirs. The comparison covers crashes, the one failure both sides detect; the oracle-detected bugs of RQ3 lie outside it.

Discussion

Lessons Learned

We draw three lessons from our results: two for building and evaluating PBT writers, and one on how such writers relate to the fuzzers that already test these domains.

(1) Low observability hurts PBT writers. Observability is how much of a program’s behavior its outputs reveal [22]. A property can be documented and still be hard to check, because the program does not show the behavior that the property concerns. This affects system-level tests most, which see only what the program prints. crontab(5) states which lines of a crontab are environment settings and which are jobs, but cronie’s output does not say how it read a given line. An Agentic-PBT test works around this with cronie’s limit of 1,000 environment settings: it writes 1,000 settings and then the line under test, so cronie reports an error if it reads that line as a setting and accepts the file if it reads it as a job. The test thus depends on knowledge beyond the property, which the LLM supplied. A writer facing low observability needs either the knowledge, from the LLM or from the user through a prompt that asks for it, or a way to raise observability, for example probes that expose the program’s internal decisions, neither of which we tried.

(2) Evaluate a writer’s reports as well as its final tests. While it writes its tests, a writer can find a failure and then stop generating the inputs that reveal it. On torch.fft.ifftshift, the session reported a failure for calls with no dimensions to rearrange, but its final test no longer generates such calls. Agentic-PBT at least keeps such failures in its reports. Evaluations should therefore include a writer’s reports, and writers should record the inputs they exclude.

(3) Fuzzers supply inputs, and LLM-based writers supply properties. The documentation states the property of 78% of Agentic-PBT’s tests, and source code that of 14% (§ 6.2). Only 24 of the 180 bugs are crashes or hangs; the rest are wrong results that only an oracle detects (§ 6.3). The domain-specific tools of RQ4 cover more of the compilers and of PostgreSQL than Agentic-PBT, but their oracles check crashes and hangs or, in SQLancer’s case, one fixed relation between a query and its partitions; none checks the documented behavior of a feature. Given properties that an LLM mines from documentation and code, a fuzzer could also check what each feature should compute.

Threats to Validity

Internal validity. Each writer generates its tests once per configuration; the repeated fuzzing measures the variability of those tests, not of their generation. RQ4 runs each tool only once. We wrote the harnesses, which could favor one writer, but both use the same harness, which only runs the program and returns its output. Humans judge RQ1’s sampled failures, but an LLM agent triages RQ3’s failures and LLM agents label RQ2’s tests; a second vendor’s agent relabels an audit sample and an author decides each disagreement, and the labels with low agreement remain a threat. The triage agent of RQ3 and the primary labeler of RQ2 are Claude models, from the vendor of the writer’s main LLM; the Gemini audit mitigates this for RQ2, and for RQ3 we read every candidate we report, although we do not review the failures the triage discards. A bug may be in a model’s training data, so we exclude bugs reported before the model’s cutoff and recheck earlier ones with a model whose cutoff is earlier. Construct validity. Branch coverage of a whole target includes code that every test reaches, such as start-up code, and coverage does not measure fault finding. Unique failures depend on how we deduplicate them, by error message or stack trace, which can merge distinct bugs or split one. Confirmed bugs depend on maintainers, and pending reports make our rates lower bounds. External validity. We study two programs in each of five domains, two writers, and two LLMs, so our results may not hold for other programs, domains, writers, or models. We sample 30 focal points per target, and on PyTorch RQ4 samples only the APIs that Centaur and FlashFuzz support. All tests use Hypothesis, on selected versions of each target.

Related Work

LLM-based PBT writers. Proptest-AI [79] and Agentic-PBT [48] (§ 4.3) are the two general-purpose writers we know of, and both were evaluated only on Python libraries. He et al. [29] use LLM-written PBTs to validate and refine LLM-generated code, and Gentoo [81] has a coding agent write input generators for Java fuzz targets, without oracles.

Mining properties and oracles. Before LLMs, each tool mined one kind of property from one source. Daikon [20] infers likely invariants from execution traces; Toradocu [27], MeMo [7], and CallMeMaybe [6] extract exception, metamorphic, and temporal oracles from Javadoc comments; MR-Scout [89] mines metamorphic relations from existing tests; and DeepREL [16] infers relations between the APIs of deep learning libraries. TOGA [17] and ChatAssert [28] generate assertions for a given test with a neural model or an LLM, and Endres et al. [19] have an LLM turn natural-language intent into postconditions. An LLM-based PBT writer instead produces the generator, the property, and the oracle together, from whatever artifact states the property.

Test generation and fuzzing. Property-based testing has been joined with coverage-guided fuzzing before LLMs: JQF and Zest [56, 57], FuzzChick [34], and RLCheck [66] steer a generator toward valid and diverse inputs with coverage feedback or reinforcement learning, but the property and the generator come from the user. LLMs also write unit tests [38, 71] and fuzz inputs, either for one domain, as TitanFuzz [15] does for deep learning libraries, or for many languages at once, as Fuzz4All [87] does; WhiteFox [90], KernelGPT [91], and ChatAFL [51] have an LLM read a compiler’s, a kernel’s, or a protocol’s source or specification to steer a fuzzer’s inputs. Unit tests check one input each, fuzzers detect mostly crashes, and LLM-based test generators can discard the tests that reveal bugs [50].

Domain-specific testing. The system-level domains of our study have mature testing tools: Csmith [93], EMI [36], YARPGen [41], MetaMut [55], and GrayC [21] for compilers, and RAGS [73], PQS [69], NoREC [67], TLP [68], and Squirrel [98] for databases; deep learning libraries have Centaur [52] and FlashFuzz [65], and testing LLM serving engines is just beginning [40, 96]. RQ4 compares Agentic-PBT with five of them.

Conclusion

We asked how far LLM-based PBT writers, built and evaluated on Python libraries, carry over to compilers, databases, system software, and LLM serving engines. They carry over, with two limits. Agentic-PBT wrote property-based tests in all five domains, took most of their properties from the documentation, and revealed 65 new bugs that maintainers confirmed in all ten programs; most of the bugs we reported are wrong results that only an oracle detects. It covered more than Proptest-AI on seven of the ten programs with Opus 5 and on all ten with GLM-5.3-Flash. The first limit is coverage: the generators reach 73 to 85% of a mature fuzzer’s coverage on the compilers and PostgreSQL, where MetaMut finds 23 compiler crashes against Agentic-PBT’s 4, none in common; on PyTorch, Agentic-PBT covers more than its fuzzers. The second is observability: some documented properties are hard to check because the program’s output does not reveal the behavior they concern. The coverage gap suggests a remedy. The fuzzers check for crashes or, in SQLancer’s case, for one fixed relation between queries, while the writer supplies the documented property; a fuzzer’s generator steered toward a property’s focal point could reach the inputs that the writer’s generator does not.

Data Availability

Our artifacts are publicly available at https://anonymous.4open.science/r/pbt-study-artifact-6A0B.

References

References

  1. [1]Paul Ammann and Jeff Offutt. 2016. Introduction to Software Testing (2nd ed.). Cambridge University Press, USA.
  2. [2]Glenn Ammons, Rastislav Bodík, and James R. Larus. 2002. Mining Specifications. In POPL. 4–16.
  3. [3]ArangoDB developers. 2026. AQL SUBSTITUTE() returns null for an empty input string. https://github.com/arangodb/arangodb/issues/23293. Accessed Sep 2026.
  4. [4]ArangoDB GmbH. 2026. ArangoDB. https://arangodb.com. Accessed Sep 2026.
  5. [5]Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Trans. Softw. Eng. 41, 5 (2015), 507–525. https://doi.org/10.1109/TSE.2014.2327875
  6. [6]Arianna Blasi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezzè. 2022. Call Me Maybe: Using NLP to Automatically Generate Unit Test Cases Respecting Temporal Constraints. In ASE. https://doi.org/10.1145/3551349.3556961
  7. [7]Arianna Blasi, Alessandra Gorla, Michael D. Ernst, Mauro Pezzè, and Antonio Carzaniga. 2021. How Automatically identifying metamorphic relations in Javadoc comments for test automation. J. Syst. Softw. 181 (2021), 111041.
  8. [8]James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Doug Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018. JAX: composable transformations of Python+NumPy programs. http://github.com/jax-ml/jax
  9. [9]Junjie Chen, Yibesh Patra, Michael Pradel, Yixuan Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A Survey of Compiler Testing. ACM Comput. Surv. 53, 1, Article 4 (2020), 36 pages.DOI
  10. [10]Koen Claessen and John Hughes. 2000. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. In ICFP. 268–279.
  11. [11]William G. Cochran. 1977. Sampling Techniques (3 ed.). John Wiley & Sons, New York.
  12. [12]cronie developers. 2026. cronie: cron daemon. https://github.com/cronie-crond/cronie. Accessed Sep 2026.
  13. [13]cronie developers. 2026. cronnext ignores a stepped day-of-week and day-of-month field. https://github.com/cronie-crond/cronie/issues/215. Accessed Sep 2026.
  14. [14]cronie developers. 2026. crontab(5): files used to schedule the execution of programs. https://man7.org/linux/man-pages/man5/crontab.5.html. Accessed Sep 2026.
  15. [15]Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models. In ISSTA. 423–435. https://doi.org/10.1145/3597926.3598067
  16. [16]Yinlin Deng, Chenyuan Yang, Anjiang Wei, and Lingming Zhang. 2022. Fuzzing Deep-Learning Libraries via Automated Relational API Inference. In ESEC/FSE. 44–56.arxiv.org/abs/2207.05531
  17. [17]Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K. Lahiri. 2022. TOGA: A Neural Method for Test Oracle Generation. In ICSE. 2130–2141.
  18. [18]Matthew B. Dwyer, George S. Avrunin, and James C. Corbett. 1999. Patterns in Property Specifications for Finite-State Verification. In ICSE. 411–420. https://doi.org/10.1145/302405.302672
  19. [19]Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions? Proc. ACM Softw. Eng. 1, FSE (2024), 1889–1912. https://doi.org/10.1145/3660791
  20. [20]Michael D. Ernst, Jake Cockrell, William G. Griswold, and David Notkin. 2001. Dynamically Discovering Likely Program Invariants to Support Program Evolution. IEEE Trans. Softw. Eng. 27, 2 (2001), 99–123.
  21. [21]Karine Even-Mendoza, Arindam Sharma, Alastair F. Donaldson, and Cristian Cadar. 2023. GrayC: Greybox Fuzzing of Compilers and Analysers for C. In ISSTA. 1219–1231. https://doi.org/10.1145/3597926.3598130
  22. [22]Roy S. Freedman. 1991. Testability of Software Components. IEEE Trans. Softw. Eng. 17, 6 (1991), 553–564.DOI
  23. [23]Mark Gabel and Zhendong Su. 2008. Javert: Fully Automatic Mining of General Temporal Properties from Dynamic Traces. In FSE. 339–349.
  24. [24]Samuele Germiniani, Daniele Nicoletti, and Graziano Pravadelli. 2025. A Systematic Literature Review on Mining LTL Specifications. IEEE Access 13 (2025), 48950–48998. https://doi.org/10.1109/ACCESS.2025.3551607
  25. [25]Johann Glock, Clemens Bauer, and Martin Pinzger. 2025. Testarier: Semantics-Based Test Generalization from Conventional Unit Tests to Property-Based Tests. arXiv:2512.14475arxiv.org/abs/2512.14475
  26. [26]GNU project. 2026. GCC, the GNU Compiler Collection. https://gcc.gnu.org. Accessed Sep 2026.
  27. [27]Alberto Goffi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezzè. 2016. Automatic Generation of Oracles for Exceptional Behaviors. In ISSTA. 213–224.
  28. [28]Ishrak Hayet, Adam Scott, and Marcelo d’Amorim. 2025. ChatAssert: LLM-Based Test Oracle Generation with External Tools Assistance. IEEE Trans. Softw. Eng. 51, 1 (2025), 305–319. https://doi.org/10.1109/TSE.2024.3519159
  29. [29]Lehan He, Zeren Chen, Zhe Zhang, Xiang Gao, and Lu Sheng. 2026. PGS: Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback. In ICML. 41125–41150.arxiv.org/abs/2506.18315
  30. [30]Robert M. Hierons, Kirill Bogdanov, Jonathan P. Bowen, Rance Cleaveland, John Derrick, Jeremy Dick, Marian Gheorghe, Mark Harman, Kalpesh Kapoor, Paul Krause, Gerald Lüttgen, Anthony J. H. Simons, Sergiy Vilkomir, Martin R. Woodward, and Hussein Zedan. 2009. Using formal specifications to support testing. ACM Comput. Surv. 41, 2, Article 9 (2009), 76 pages. https://doi.org/10.1145/1459352.1459354
  31. [31]Hong Jin Kang and David Lo. 2021. Adversarial Specification Mining. ACM Trans. Softw. Eng. Methodol. 30, 2, Article 16 (2021), 40 pages. https://doi.org/10.1145/3424307
  32. [32]Leslie Kish. 1965. Survey Sampling. John Wiley & Sons, New York.
  33. [33]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In SOSP. 611–626. https://doi.org/10.1145/3600006.3613165
  34. [34]Leonidas Lampropoulos, Michael Hicks, and Benjamin C. Pierce. 2019. Coverage Guided, Property Based Testing. Proc. ACM Program. Lang. 3, OOPSLA, Article 181 (2019), 29 pages.DOI
  35. [35]J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159–174. https://doi.org/10.2307/2529310
  36. [36]Vu Le, Merhdad Afshari, and Zhendong Su. 2014. Compiler Validation via Equivalence Modulo Inputs. In PLDI. 216–226.
  37. [37]Claire Le Goues and Westley Weimer. 2009. Specification Mining With Few False Positives. In TACAS. 292–306.
  38. [38]Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023. CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models. In ICSE. 919–931.DOI
  39. [39]Jiawei Liu, Jinkun Lin, Fabian Ruffy, Cheng Tan, Jinyang Li, Aurojit Panda, and Lingming Zhang. 2023. NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers. In ASPLOS. 530–543. https://doi.org/10.1145/3575693.3575707
  40. [40]Mugeng Liu, Siqi Zhong, Weichen Bi, Yixuan Zhang, Zhiyang Chen, Zhenpeng Chen, Xuanzhe Liu, and Yun Ma. 2026. A First Look at Bugs in LLM Inference Engines. ACM Trans. Softw. Eng. Methodol. 35, 9, Article 284 (2026), 34 pages. https://doi.org/10.1145/3788873
  41. [41]Vsevolod Livinskii, Dmitry Babokin, and John Regehr. 2020. Random Testing for C and C++ Compilers with YARPGen. Proc. ACM Program. Lang. 4, OOPSLA, Article 196 (2020), 25 pages.DOI
  42. [42]LLVM developers. 2024. Clang crashes with huge byval arguments are unsupported. https://github.com/llvm/llvm-project/issues/118207. Accessed Oct 2026.
  43. [43]LLVM developers. 2026. clang __attribute__((vectorcall)) crashes on any long double parameter (x86_64 Linux). https://github.com/llvm/llvm-project/issues/227448. Accessed Oct 2026.
  44. [44]LLVM developers. 2026. constexpr __builtin_align_down/up on a pointer into an array reads the original element. https://github.com/llvm/llvm-project/issues/27548. Accessed Oct 2026.
  45. [45]LLVM project. 2026. Clang: a C language family frontend for LLVM. https://clang.llvm.org. Accessed Sep 2026.
  46. [46]David Lo, Siau-Cheng Khoo, Chao Liu, and Jiawei Han. 2011. Specification Mining: A Concise Introduction. In Mining Software Specifications: Methodologies and Applications. CRC Press, 1–27.
  47. [47]Davide Lorenzoli, Leonardo Mariani, and Mauro Pezzè. 2008. Automatic Generation of Software Behavioral Models. In ICSE. 501–510.DOI
  48. [48]Muhammad Maaz, Liam DeVoe, Zac Hatfield-Dodds, and Nicholas Carlini. 2025. Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem. In Deep Learning for Code Workshop at NeurIPS. arXiv:2510.09907arxiv.org/abs/2510.09907
  49. [49]David R. MacIver, Zac Hatfield-Dodds, and Many Other Contributors. 2019. Hypothesis: A new approach to property-based testing. J. Open Source Softw. 4, 43 (2019), 1891. https://doi.org/10.21105/joss.01891
  50. [50]Noble Saji Mathews and Meiyappan Nagappan. 2024. Design choices made by LLM-based test generators prevent them from finding bugs. arXiv preprint arXiv:2412.14137 (2024). https://doi.org/10.48550/arXiv.2412.14137
  51. [51]Ruijie Meng, Martin Mirchev, Marcel Böhme, and Abhik Roychoudhury. 2024. Large Language Model Guided Protocol Fuzzing. In NDSS. https://doi.org/10.14722/ndss.2024.24556
  52. [52]M. M. Abid Nazir, Shinhae Kim, Feiran Alex Qin, Saikat Dutta, and Marcelo d’Amorim. 2026. Testing Deep Learning Libraries via Neurosymbolic Constraint Learning. In ICSE. 3586–3598. https://doi.org/10.1145/3744916.3787800
  53. [53]Tung Thanh Nguyen, Hoan Anh Nguyen, Nam H. Pham, Jafar M. Al-Kofahi, and Tien N. Nguyen. 2009. Graph-Based Mining of Multiple Object Usage Patterns. In ESEC/FSE. 383–392.DOI
  54. [54]OpenBSD project. 2026. OpenSSH. https://www.openssh.com. Accessed Sep 2026.
  55. [55]Xianfei Ou, Cong Li, Yanyan Jiang, and Chang Xu. 2024. The Mutators Reloaded: Fuzzing Compilers with Large Language Model Generated Mutation Operators. In ASPLOS. 298–312. https://doi.org/10.1145/3622773.3674171
  56. [56]Rohan Padhye, Caroline Lemieux, and Koushik Sen. 2019. JQF: Coverage-Guided Property-Based Testing in Java. In ISSTA (Tool Demonstrations). 398–401.DOI
  57. [57]Rohan Padhye, Caroline Lemieux, Koushik Sen, Mike Papadakis, and Yves Le Traon. 2019. Semantic Fuzzing with Zest. In ISSTA. 329–340.DOI
  58. [58]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. NeurIPS 32 (2019).arxiv.org/abs/1912.01703
  59. [59]Hila Peleg, Dan Rasin, and Eran Yahav. 2018. Generating Tests by Example. In VMCAI. 406–429. https://doi.org/10.1007/978-3-319-73721-8_19
  60. [60]Hung Viet Pham, Thibaud Lutellier, Weihzen Qi, and Lin Tan. 2019. CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries. In ICSE. 1027–1038. https://doi.org/10.1109/ICSE.2019.00107
  61. [61]Van-Thuan Pham, Marcel Böhme, and Abhik Roychoudhury. 2020. AFLNet: A Greybox Fuzzer for Network Protocols. In ICST. 460–465.DOI
  62. [62]PostgreSQL developers. 2026. BUG #19732: first_value/last_value/nth_value return NULL with EXCLUDE TIES when the current row is outside its frame. https://www.postgresql.org/message-id/[email protected]. Accessed Oct 2026.
  63. [63]PostgreSQL Global Development Group. 2026. PostgreSQL. https://www.postgresql.org. Accessed Sep 2026.
  64. [64]PostgreSQL Global Development Group. 2026. PostgreSQL Documentation, Section 4.2.8: Window Function Calls. https://www.postgresql.org/docs/current/sql-expressions.html. Accessed Sep 2026.
  65. [65]Feiran Alex Qin, M. M. Abid Naziri, Hengyu Ai, Saikat Dutta, and Marcelo d’Amorim. 2026. How Effective Is Coverage-Guided Fuzzing to Test Deep Learning Library APIs?. In ICST. 88–99. https://doi.org/10.1109/ICST69053.2026.000023
  66. [66]Sameer Reddy, Caroline Lemieux, Rohan Padhye, and Koushik Sen. 2020. Quickly Generating Diverse Valid Test Inputs with Reinforcement Learning. In ICSE. 1410–1421.DOI
  67. [67]Manuel Rigger and Zhendong Su. 2020. Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction. In ESEC/FSE. 1140–1152.DOI
  68. [68]Manuel Rigger and Zhendong Su. 2020. Finding Bugs in Database Systems via Query Partitioning. Proc. ACM Program. Lang. 4, OOPSLA, Article 211 (2020), 30 pages.DOI
  69. [69]Manuel Rigger and Zhendong Su. 2020. Testing Database Engines via Pivoted Query Synthesis. In OSDI. 667–682.arxiv.org/abs/2001.04174
  70. [70]Martin P. Robillard, Eric Bodden, David Kawrykow, Mira Mezini, and Tristan Ratchford. 2013. Automated API Property Inference Techniques. IEEE Trans. Softw. Eng. 39, 5 (2013), 613–637.DOI
  71. [71]Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Trans. Softw. Eng. 50, 1 (2024), 85–105. https://doi.org/10.1109/TSE.2023.3334955
  72. [72]SGLang developers. 2026. deepseekv32/deepseekv4 tool-call parsers drop calls when the streamed output has two chunks. https://github.com/sgl-project/sglang/issues/35563. Accessed Oct 2026.
  73. [73]Donald R. Slutz. 1998. Massive Stochastic Testing of SQL. In VLDB. 618–622.
  74. [74]SQLancer developers. 2024. Remove NoSQL databases. https://github.com/sqlancer/sqlancer/pull/915. Pull request 915. Accessed Oct 2026.
  75. [75]The Clang Team. 2026. Clang Language Extensions: Alignment Builtins. https://clang.llvm.org/docs/LanguageExtensions.html. Accessed Sep 2026.
  76. [76]The JAX Authors. 2026. jax.lax.pmean. https://docs.jax.dev/en/latest/_autosummary/jax.lax.pmean.html. Accessed Sep 2026.
  77. [77]Deepika Tiwari, Yogya Gamage, Martin Monperrus, and Benoit Baudry. 2024. PROZE: Generating Parameterized Unit Tests Informed by Runtime Data. In SCAM. 166–176. https://doi.org/10.1109/SCAM63643.2024.00025
  78. [78]Keita Tsukamoto, Yuta Maezawa, and Shinichi Honiden. 2018. AutoPUT: An Automated Technique for Retrofitting Closed Unit Tests Into Parameterized Unit Tests. In SAC. 1944–1951. https://doi.org/10.1145/3167132.3167340
  79. [79]Vasudev Vikram, Caroline Lemieux, Joshua Sunshine, and Rohan Padhye. 2023. Can Large Language Models Write Good Property-Based Tests? arXiv preprint arXiv:2307.04346 (2023).arxiv.org/abs/2307.04346
  80. [80]Vasudev Vikram, Caroline Lemieux, Joshua Sunshine, and Rohan Padhye. 2024. Proptest AI Anonymous Dataset. Zenodo. https://doi.org/10.5281/zenodo.10967488
  81. [81]Vasudev Vikram and Rohan Padhye. 2026. Predicate-Guided Synthesis of Input Generators via Gentoo. In ASE. To appear.DOI
  82. [82]vLLM developers. 2026. LoRA adapter targeting no configured modules silently serves the base model. https://github.com/vllm-project/vllm/issues/55193. Accessed Oct 2026.
  83. [83]vLLM developers. 2026. [ROCm][gfx1151] ROCM_ATTN returns different outputs for the same greedy request after other requests. https://github.com/vllm-project/vllm/issues/57493. Accessed Oct 2026.
  84. [84]Andrzej Wasylkowski, Andreas Zeller, and Christian Lindig. 2007. Detecting Object Usage Anomalies. In ESEC/FSE. 35–44.DOI
  85. [85]Anjiang Wei, Yinlin Deng, Chenyuan Yang, and Lingming Zhang. 2022. Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source. In ICSE. 995–1007. https://doi.org/10.1145/3510003.3510041
  86. [86]Edwin B. Wilson. 1927. Probable Inference, the Law of Succession, and Statistical Inference. J. Amer. Statist. Assoc. 22, 158 (1927), 209–212. https://doi.org/10.1080/01621459.1927.10502953
  87. [87]Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. Fuzz4All: Universal Fuzzing with Large Language Models. In ICSE. 1–13.
  88. [88]Danning Xie, Yitong Li, Mijung Kim, Hung Viet Pham, Lin Tan, Xiangyu Zhang, and Michael W. Godfrey. 2022. DocTer: documentation-guided fuzzing for testing deep learning API functions. In ISSTA. 176–188. https://doi.org/10.1145/3533767.3534220
  89. [89]Congying Xu, Valerio Terragni, Hengcheng Zhu, Jiarong Wu, and Shing-Chi Cheung. 2024. MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases. ACM Trans. Softw. Eng. Methodol. 33, 6 (2024). https://doi.org/10.1145/3656340
  90. [90]Chenyuan Yang, Yinlin Deng, Runyu Lu, Jiayi Yao, Jiawei Liu, Reyhaneh Jabbarvand, and Lingming Zhang. 2024. WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models. Proc. ACM Program. Lang. 8, OOPSLA2 (2024), 709–735. https://doi.org/10.1145/3689736
  91. [91]Chenyuan Yang, Zijiye Zhao, and Lingming Zhang. 2025. KernelGPT: Enhanced Kernel Fuzzing via Large Language Models. In ASPLOS. 560–573. https://doi.org/10.1145/3676641.3716022
  92. [92]Jinlin Yang, David Evans, Deepali Bhardwaj, Thirumalash Bhat, and Manuvvir Das. 2006. Peracotta: Mining Temporal API Rules from Imperfect Traces. In ICSE. 282–291.
  93. [93]Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and Understanding Bugs in C Compilers. In PLDI. 283–294.
  94. [94]Z.ai. 2024. GLM-5.3-Flash. https://huggingface.co/zai-org/GLM-5.3-Flash. Open weights. Accessed Sep 2026.
  95. [95]Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2022. Machine Learning Testing: Survey, Landscapes and Horizons. IEEE Trans. Softw. Eng. 48, 1 (2022), 1–36. https://doi.org/10.1109/TSE.2019.2962027
  96. [96]Yunze Zhao, Yibo Zhao, Yuchen Zhang, Zaoxing Liu, and Michelle L. Mazurek. 2026. Continuous Discovery of Vulnerabilities in LLM Serving Systems with Fuzzing. arXiv:2605.11202 [cs.CR]arxiv.org/abs/2605.11202
  97. [97]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In NeurIPS. https://doi.org/10.5220/079017-2000
  98. [98]Rui Zhong, Yongheng Chen, Hong Hu, Hangfan Zhang, Wenke Lee, and Dinghao Wu. 2020. SQUIRREL: Testing Database Management Systems with Language Validity and Coverage Feedback. In CCS. 955–970.arxiv.org/abs/2006.02398

Paper details

vlnc:W_x9wyteyv
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

an LLM

View statement
Looking for related papers…
Contents