betaFeedback

WorksComputer Science

Teralizer: A Tool for Generalizing JUnit Tests into Property-Based jqwik Tests

Abstract

Background: Property-based testing validates properties across many generated inputs and can thereby detect regressions that conventional unit tests miss. However, writing property-based tests is challenging for practitioners, which limits their adoption. Tool: Teralizer automatically generalizes JUnit tests into property-based jqwik tests for regression detection. Through single-path symbolic analysis, it extracts from each test execution a path condition, which characterizes the inputs that follow the same execution path, and a symbolic output expression, which computes the expected output for these inputs. The generalized tests generate inputs that satisfy the path condition and use the symbolic output expression as the oracle. Results: In our evaluation, generalization increased the mutation scores of EvoSuite-generated test suites by 0.8 to 3.9 percentage points. On the EqBench benchmark, generalizing the test suites that EvoSuite generated with 1 s per class reached the mutation score of EvoSuite's 60 s test suites with 31.9% less total runtime. Availability: Teralizer is available as open source at https://github.com/glockyco/Teralizer.

Introduction

Unit tests are widely used to detect regressions, i.e., unintended changes in program behavior. Since a conventional unit test validates the method under test (MUT) only for specific input-output pairs, regressions that affect other inputs remain undetected. Figure 1 shows an example. The method approvedAmount approves loans of up to FULL for applicants with a credit score of at least 80, and the unit test fullApproval validates this behavior for a score of 90 and a requested amount of 20. A refactoring that merges the two calls of Math.min unintentionally changes the condition score >= 80 to score > 80. Applicants with a score of 80 are thereby approved for at most PARTIAL instead of FULL, but the unit test fullApproval still passes.

Test generalization with Teralizer on the running example

Figure 1. Test generalization with Teralizer on the running example. (1) The unit test validates a single input and passes before and after a refactoring that unintentionally changes the condition score >= 80 to score > 80. (2) Teralizer extracts the path condition and the symbolic output expression of the tested execution path. (3) The generalized test validates the assertion for generated inputs that satisfy the path condition and thereby detects the regression.

Property-based testing (PBT) [1] addresses this limitation by validating properties across many inputs that are produced by generators. In the example, a property-based test that validates the approved amount for all scores of at least 80 detects the regression. However, writing property-based tests requires developers to identify suitable properties and to define generators with appropriate constraints, which practitioners find challenging and which limits the adoption of PBT [2].

In this paper, we present Teralizer, a tool that automatically generalizes JUnit tests into property-based jqwik [3] tests based on the approach of our previous work [4]. Through single-path symbolic analysis with Symbolic PathFinder (SPF) [5], Teralizer extracts from the execution of a unit test a path condition, which characterizes all inputs that follow the same execution path as the original inputs, and a symbolic output expression, which computes the expected output for these inputs. It then creates a property-based test that generates inputs satisfying the path condition and uses the symbolic output expression as the expected value of the original assertion. In Figure 1, this generalized test detects the regression that the unit test misses.

The approach and its evaluation are described in detail in [4], while this paper focuses on the implementation and the usage of the tool. TERALIZER is open source under the MIT license,1 with a replication package on Zenodo [6].

Overview of Teralizer

TERALIZER is implemented in Java and takes a Maven or Gradle project with JUnit 3, 4, or 5 tests as input, to which it adds the generalized tests as jqwik test classes. Figure 2 shows the five stages of its fully automated pipeline [4] and their results for the three unit tests of the running example. Tests and assertions that a stage cannot process are excluded from the subsequent stages, and the reasons are recorded.

Five stages of TERALIZER and results for three unit tests

Figure 2. The five stages of TERALIZER with the tools that they use, and their results for the three unit tests of the running example.

Stages 1 and 2: TERALIZER executes the test suite and analyzes all passing tests with Spoon [7]. Stage 1 identifies the assertions that TERALIZER can generalize, which currently are equality, Boolean, and exception assertions of JUnit and equality assertions of Hamcrest. Stage 2 identifies the MUT call of each assertion, i.e., the method call whose result the assertion validates. For this purpose, a data-flow analysis traces the asserted value back to the method call that produced it. If the analysis does not yield a unique call, a ranking heuristic selects one of the preceding calls based on its parameter and return types, its class, its name, and its distance to the assertion. Arguments of primitive types, their boxed types, and strings become the inputs of the generalized test.

Stage 3: TERALIZER executes each test with SPF using single-path symbolic analysis, which treats these arguments as symbolic values while it follows the execution path of the original inputs and therefore requires no constraint solving. SPF adds the condition of each taken branch that depends on a symbolic value to the path condition. When the MUT returns, TERALIZER records the path condition and the output as the input/output specification. For fullApproval, the output is the symbolic expression amount, whereas SPF provides only the concrete outputs 5 and 0 for the other two tests.

Stage 4: TERALIZER creates a jqwik test class with a property-based version of the original test (Figure 1), in which the arguments of the MUT call are replaced with generated parameters and the expected value of the assertion with the output of the specification. The output therefore has to be a valid oracle for all inputs of the input partition, which holds for a symbolic output expression. A concrete output is not necessarily constant for the input partition, because SPF also provides only a concrete output when it cannot track the dependence of the output on the inputs, e.g., for outputs that are accumulated in input-dependent loops [4]. TERALIZER therefore uses a concrete output only if the MUT directly returns a literal, as in rejection (return 0). In partialApproval, the constant is returned via Math.min, which the current implementation does not recognize as constant, so that the assertion is excluded.

The parameters are generated according to one of two strategies. NAIVE generates random values and applies the path condition as a filter. IMPROVED additionally encodes constraints that compare a parameter with a constant or with another parameter in the generators, which reduces the number of rejected inputs and enables jqwik to generate inputs at the boundaries of the input partition, such as the score of 80 that reveals the regression in Figure 1. The figure shows these constraints as jqwik annotations, whereas the generated code implements them in a supplier class. A generalized test first executes the original inputs and then generated inputs, up to a configurable number of inputs per test (tries). Generalized tests that fail for the current implementation are excluded.

Stage 5: This optional stage applies mutation testing with PIT [8] and retains a generalized test only if it detects at least one mutant that the original test suite does not detect. In the example, this holds for fullApproval, whose generalized test detects the mutant that changes score >= 80 to score > 80, i.e., the fault that the refactoring of Figure 1 introduces.

Usage

TERALIZER is executed in a Docker container. On a machine with Bash, Git LFS, and Docker, the following commands build the image in a clone of the repository and apply TERALIZER to the running example:

git lfs pull
git submodule update --init --recursive jpf-symbc
docker build -t teralizer .
scripts/try-example.sh

The script try-example.sh runs the pipeline on a copy of the example and prints the results (Figure 3). It reports for each assertion that reaches Stage 4 whether a generalized test was created or why the assertion was excluded, the mutants that the original and the generalized test suite detect, and the paths of the generalized test classes. It then applies the refactoring of Figure 1 and executes all tests. The three original tests pass, whereas the generalized test of fullApproval fails at the fourth try and reports the failing input together with the expected and the actual output. Processing the example takes less than two minutes on a laptop (Intel Core Ultra 7 265H, 16 GB of memory assigned to Docker).

Output of scripts/try-example.sh for the example

Figure 3. Output of scripts/try-example.sh for the example (abridged).

For another project, the script takes the project directory as its argument. The project has to build with Java 8, which SPF requires, and use Maven or Gradle with JUnit 3, 4, or 5. The script works on a copy of the project and uses the configuration file of the running example, which specifies the generalization strategy and the number of tries per generalized test:

teralizer.generalizations.IMPROVED_200_TRIES {
  algorithm = "IMPROVED", jqwik.tries = 200
}

TERALIZER adds each generalized test as a separate jqwik test class to the package of the original test class and does not modify existing tests. The generalized tests are executed on the JUnit Platform. In the running example, adding jqwik as a test dependency is sufficient to execute them together with the original tests via mvn test. The results of all stages and the reasons for all exclusions are stored in a PostgreSQL database.

Evaluation

The evaluation in [4] uses test suites that EvoSuite [9] generates with search budgets of 1 s, 10 s, and 60 s per class for the EqBench benchmark [10] and for numeric utility classes of Apache Commons [11], the developer-written tests of the same classes, and 584 open-source projects from the RepoReapers dataset [12]. Mutation scores are measured with PIT.

Effectiveness: Generalization increases the mutation scores of the EvoSuite-generated test suites by 1.22–3.93 percentage points for EqBench and by 0.82–1.33 percentage points for Apache Commons, depending on the search budget, the generalization strategy, and the number of tries. For the developer-written tests, which already achieve a mutation score of 80.35%, the increases are 0.05–0.07 percentage points. The detection rate increases most for mutants of arithmetic operations and, with IMPROVED, for mutants of conditional boundaries, such as the mutant of the running example.

Efficiency: For EqBench, EvoSuite with a search budget of 1 s followed by TERALIZER (A in Figure 4) achieves a 0.1 percentage points higher mutation score than EvoSuite with a search budget of 60 s in 31.9% less total runtime. With a search budget of 10 s (B), this combination achieves a 2.2 percentage points higher mutation score in 12.4% less time. For Apache Commons, all test suites of EvoSuite are Pareto-optimal, but generalization achieves higher mutation scores than EvoSuite alone (59.5% vs. 58.1%). Stage 5 takes 59.1–95.7% of the processing time of TERALIZER and retains 1,555 of the 21,478 generalized tests (7.2%).

Mutation scores and total runtimes of EvoSuite alone and followed by TERALIZER

Figure 4. Mutation scores and total runtimes of EvoSuite alone and followed by TERALIZER [4]. Lines connect each EvoSuite test suite with the Pareto-optimal test suites that TERALIZER creates from it (pp: percentage points).

Applicability: The current prototype of TERALIZER is less successful for real-world projects. Of the 584 RepoReapers projects, 85 (14.6%) complete all five stages, which yields 1,435 generalized tests. Across all projects, 3.8% of the assertions pass the first three stages, compared with 47.8% for EqBench and Apache Commons. The other assertions are excluded mainly because no MUT call can be identified or because the parameters or the return value of the MUT have unsupported types. In Stage 4, 61.6% of the generalization attempts are excluded for a lack of a valid oracle, 77.4% of them because of a concrete output that the MUT does not return as a literal, as in partialApproval.

Limitations and Future Work

Since the oracles of the generalized tests are derived from the implementation, generalized tests detect behavioral changes in later versions, but not faults that are already present in the analyzed version. Moreover, single-path symbolic analysis sis requires pure MUTs, i.e., deterministic MUTs without side effects that depend only on their parameters [4].

The applicability of TERALIZER is mainly limited by the underlying symbolic analysis. SPF requires Java 8. Since it constructs exact symbolic models for constraint solving [5], it supports only inputs and outputs of primitive types, their boxed types, and strings. Test generalization has weaker requirements than constraint solving, because a generator that covers only a subset of an input partition reduces the number of tested inputs, but does not cause spurious test failures [4]. We therefore plan to extend the symbolic analysis beyond what is required for constraint solving, e.g., with approximate models for arrays, objects, and further string operations.

So far, we have not focused on the presentation of the generalized tests. For example, the generated test class of fullApproval comprises 412 lines of code, most of which implement the generators and record the tested inputs for the evaluation, compared to six lines of the simplified version in Figure 1. In future work, we plan to make the generalized tests easier to understand and to maintain for practitioners.

Related Work

Test generalization is a form of test amplification [13], which enhances existing tests. Test generation tools such as EvoSuite [9] instead infer test scenarios and assertions from the observed behavior of the implementation, which risks overfitting the implementation [14]. The setup code and assertions of developer-written tests encode knowledge about the intended behavior that the implementation itself does not contain, which makes their generalized tests less likely to overfit the implementation. Both techniques can be combined by generalizing generated tests, as in Section IV.

To our knowledge, JARVIS [15] is the only other tool that automatically generalizes unit tests into property-based tests. It infers properties from groups of similar tests based on predefined abstraction templates and does not analyze the implementation. TERALIZER creates generalized tests in 9 of the 12 projects of the JARVIS evaluation, whereas the published JARVIS results cover 2 projects [4]. Approaches that generate parameterized unit tests start from tests without assertions [16] or use values observed at runtime [17] and do not generalize oracles beyond these values. Large language models generate property-based tests without existing tests, but their generators and assertions can be invalid or unsound [18].

Conclusion

We presented TERALIZER, a tool that automatically generalizes JUnit tests into property-based jqwik tests based on single-path symbolic analysis. The generalized tests validate the original assertions across the tested input partitions and thereby detect regressions that the original tests miss.

Acknowledgment

This research was funded in whole or in part by the Austrian Science Fund (FWF) 10.55776/P36698. For open access purposes, the author has applied a CC BY public copyright license to any author accepted manuscript version arising from this submission. The research reported in this paper has been partly funded by BMMI, BMWET, and the State of Upper Austria in the frame of the SCCH competence center INTEG-RATE (FFG grant no. 892418) part of the FFG COMET Competence Centers for Excellent Technologies Programme. We used OpenAI GPT, Anthropic Claude, and Google Gemini models to assist with writing, coding, and literature search. The authors reviewed and refined the generated material and take full responsibility for the final manuscript.

References

  1. [1]K. Claessen and J. Hughes, “QuickCheck: A lightweight tool for random testing of Haskell programs,” in Proc. 5th ACM SIGPLAN Int. Conf. Functional Programming (ICFP). ACM, 2000, pp. 268–279.
  2. [2]H. Goldstein, J. W. Cutler, D. Dickstein, C. Pierce, and A. Head, “Property-based testing in practice,” in Proc. 46th IEEE/ACM Int. Conf. Softw. Eng. (ICSE). ACM, 2024, pp. 187:1–187:13.DOI
  3. [3]jqwik-team, “jqwik: Property-based testing in Java.” [Online]. Available: https://jqwik.net/
  4. [4]J. Glock, C. Bauer, and M. Pinzger, “Teralizer: Semantics-based test generalization from conventional unit tests to property-based tests,” arXiv preprint arXiv:2512.14475, 2025.arxiv.org/abs/2512.14475
  5. [5]C. S. Păsăreanu, W. Visser, D. H. Bushnell, J. Geldenhuys, P. C. Mehlitz, and N. Rungta, “Symbolic PathFinder: Integrating symbolic execution with model checking for Java bytecode analysis,” Automated Software Engineering, vol. 20, no. 3, pp. 391–425, 2013.DOI
  6. [6]J. Glock, C. Bauer, and M. Pinzger, “Replication package for: ‘Teralizer: Semantics-based test generalization from conventional unit tests to property-based tests’,” Zenodo, 2026. [Online]. Available: https://doi.org/10.5281/zenodo.17950380
  7. [7]R. Pawlak, M. Monperrus, N. Petitprez, C. Noguera, and L. Seinturier, “Spoon: A library for implementing analyses and transformations of Java source code,” Software: Practice and Experience, vol. 46, no. 9, pp. 1155–1179, 2016.DOI
  8. [8]H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque, “PIT: A practical mutation testing tool for Java (demo),” in Proc. 25th ACM SIGSOFT Int. Symp. Softw. Testing and Analysis (ISSTA). ACM, 2016, pp. 449–452.
  9. [9]G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” in Proc. 19th ACM SIGSOFT Symp. Foundations of Softw. Eng. and 13th European Softw. Eng. Conf. (ESEC/FSE). ACM, 2011, pp. 416–419.
  10. [10]S. Badihi, Y. Li, and J. Rubin, “EqBench: A dataset of equivalent and non-equivalent program pairs,” in Proc. 18th IEEE/ACM Int. Conf. Mining Softw. Repositories (MSR). IEEE, 2021, pp. 610–614.DOI
  11. [11]The Apache Software Foundation, “Apache Commons.” [Online]. Available: https://commons.apache.org/
  12. [12]N. Munaiah, S. Kroh, C. Cabrey, and M. Nagappan, “Curating GitHub for engineered software projects,” Empirical Software Engineering, vol. 22, no. 6, pp. 3219–3253, 2017.
  13. [13]B. Danglot, O. Vera-Perez, Z. Yu, A. Zaidman, M. Monperrus, and B. Baudry, “A snowballing literature study on test amplification,” Journal of Systems and Software, vol. 157, p. 110398, 2019.DOI
  14. [14]E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, 2015.DOI
  15. [15]H. Peleg, D. Rasin, and E. Yahav, “Generating tests by example,” in Proc. 19th Int. Conf. Verification, Model Checking, and Abstract Interpretation (VMCAI). Springer, 2018, pp. 406–429.DOI
  16. [16]G. Fraser and A. Zeller, “Generating parameterized unit tests,” in Proc. 20th ACM SIGSOFT Int. Symp. Softw. Testing and Analysis (ISSTA). ACM, 2011, pp. 364–374.DOI
  17. [17]D. Tiwari, Y. Gamage, M. Monperrus, and B. Baudry, “PROZE: Generating parameterized unit tests informed by runtime data,” in Proc. 24th IEEE Int. Conf. Source Code Analysis and Manipulation (SCAM). IEEE, 2024, pp. 166–176.DOI
  18. [18]V. Vikram, C. Lemieux, J. Sunshine, and R. Padhye, “Can large language models write good property-based tests?” arXiv preprint arXiv:2307.04346, 2024.arxiv.org/abs/2307.04346

Paper details

vlnc:W_pf2ghvjb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

GPT (multiple versions), Anthropic Claude, Gemini (multiple versions)

View statement
Looking for related papers…
Contents