betaFeedback

WorksComputer Science

MLIR-MRacle: Memory-Model-Aware Metamorphic Fuzzing of Concurrent MLIR

Abstract

Multi-level compilers built on MLIR lower concurrent programs through many stages towards targets with potentially drastically different memory models, where an optimisation which appears reasonable, can silently change program behaviour. Testing these lowerings is hard: a concurrent program often has no single expected outcome, and existing MLIR fuzzers often assume determinism. We present MLIR-MRacle, a memory-model-aware metamorphic fuzzer for concurrent MLIR programs. MLIR-MRacle applies metamorphic relations gated by the target memory model, so that both the original and transformed programs remain well defined - executing both repeatedly, and comparing their outcome sets or distributions withone of two statistical oracles. A coverage guided campaign driver built on libafl is used to perform continuous fuzzing. MRs, memory models, oracles, and execution backends all provide extension points for future researchers and compiler engineers to build off this package. In a case study on Armv8 hardware, MLIR-MRacle ran over 145k trials across four configurations, reaching up to 93.6k compiler coverage edges, and found a bug in the OpenMP dialect verifier that has since been fixed upstream. Video: https://youtu.be/lKtGk0H97Ts?si=dWLUHl2CJ24ep6JY

Video: https://youtu.be/lKtGk0H97Ts?si=dWLUHl2CJ24ep6JY

Introduction

MLIR [1] has become the standard shared infrastructure for building multi-level compilers. Its concurrency dialects, such as OpenMP and async, are lowered through many stages towards targets whose memory models differ from one another [2, 3]. Optimisations that appear reasonable can break concurrent programs under weak memory models [4, 5]; such bugs surface as silent miscompilations rather than crashes. Nor is the infrastructure itself immune: whilst developing MLIR-MRacle, we identified a bug in the OpenMP dialect verifier, which failed to reject invalid nested critical operations, allowing programs that hang indefinitely to compile. The bug has since been fixed [6].

Testing these lowerings is hard. Concurrency amplifies the oracle problem [7]: a program’s result is a set, or distribution, of outcomes rather than a single value, increasing the amount of reasoning needed. Which outcomes are legal depends on the target, and some arise only rarely – after many runs or under specific stress conditions [8, 9]. MLIR’s in-tree tests, by

contrast, check each program against a singular expected output. Existing tools do not close this gap. MLIR fuzzers [10, 11, 12, 13, 14] rely on crash, differential or semantic oracles under the assumption of deterministic programs, and their mutations are not memory-model-aware. Where they do produce concurrent programs, these may be racy or have undefined behaviour, so a change in outcome cannot be attributed to the compiler. Concurrency-focused compiler testing [5, 15, 16] targets C/C11 and specific backends, and often requires a formal model to judge outcomes [2]. Metamorphic and EMI-based compiler testing [17, 18, 19] assumes a deterministic output to compare against. To the best of our knowledge, no tool tests concurrency semantics at the level of MLIR without either assuming determinism or requiring a formal model.

We present MLIR-MRacle, a metamorphic testing [20], [21] tool for concurrent MLIR programs. Each metamorphic relation (MR) is gated by the target memory model, so that both the original and transformed programs remain well defined. MLIR-MRacle judges the outcomes of both programs with a statistical oracle, using either set relations over agitated executions or a comparison of outcome distributions [22]. A coverage-guided campaign driver, built on libafl [23], turns this into a continuous fuzzing campaign. MLIR-MRacle is aimed at MLIR dialect and lowering developers, and at testing researchers who wish to add memory models, MRs, or execution backends; for example, a simulator backend would allow the same campaigns to test system simulators [24, 25, 26].

In summary, our contributions are:

  1. MLIR-MRacle, an extensible tool for metamorphic testing of concurrent MLIR, with four extension points: MRs, memory models, oracle modes, and execution backends.

  1. Campaign infrastructure providing coverage feedback, failure replay, and reproducible runs on Armv8 hardware.

  1. A case study demonstrating MLIR-MRacle at scale on Armv8 hardware, alongside an uncovered bug in MLIR’s OpenMP dialect

MLIR-MRacle: Architecture and Design

MLIR-MRacle is a layered metamorphic fuzzer (Figure 1). A Rust campaign driver manages the corpus, schedules trials and records results, while a native C++ engine handles MLIR-specific logic, parsing, MR selection, memory-model gating, lowering, execution, and oracle judgement. The two communicate through a narrow C ABI, so the driver is independent of MLIR and LLVM internals. A trial is fully described by the tuple ⟨seed,MR,memory model,backend,oracle mode⟩\langle\text{seed}, \text{MR}, \text{memory model}, \text{backend}, \text{oracle mode}\rangle; each element is chosen independently, so the same program pair can be judged under different backends or oracles without changing its MR.

MLIR-MRacle architecture

Figure 1. MLIR-MRacle architecture. A campaign driver schedules trials over a corpus; each trial calls the native engine. Teal boxes are pluggable: MRs, execution backends and oracle modes are selected independently per trial; agitation applies only under the outcome oracle. Dashed boxes are extension points, e.g. memory models or a simulator backend.

a) Metamorphic relations: MRs are implemented as rewrites within a dedicated MLIR pass. Each MR in the registry declares the memory model under which it is valid, its rewrite, and the relation it expects between the outcomes of the original and transformed programs: equality, subset, or superset. For each trial, the pass filters the registry by the selected memory model, picks an applicable MR at random, and falls back to another if the rewrite cannot be applied. MRs are either generic (i.e., valid under any model) or model specific. Generic MRs include those intended to increase program diversity, such as insert-random-arith, which insert a random-length chain of MLIR arith operations into the program, and concurrency-related MRs that hold under all memory models, such as insert-fence, a subset relation that inserts a fence anywhere valid in a program. In contrast, model-specific MRs are valid only for the memory model they target, such as commute-relaxed-read-write, where

relaxed atomic accesses are commuted where the model [3] permits it, which may be unsound under another memory model [4].

b) Agitation and execution: A single run of a concurrent program is weak evidence of a fault, or lack thereof, especially since many outcomes are rare [8]. Under the outcome oracle, MLIR-MRacle therefore agitates both programs before execution. Each program is cloned (10 clones by default), and each clone is perturbed independently: random scf.while jitter at the MLIR level, basic block shuffling after lowering to LLVM IR, varied optimisation levels, OpenMP settings and thread counts inspired by traditional litmus testing approaches [27]. These perturbations are drawn from the same distribution for both programs, so the relation the MR predicts is preserved. Each clone is then executed repeatedly (2,000 runs by default), and its outcomes are pooled. A cheap single-threaded run of the source program acts as an early exit for deterministic failures. Execution sits behind a backend interface; MLIR-MRacle currently provides an in-process LLJIT backend, and a native-process backend. A hardware or system simulator adapter fits the same interface which would enable the same campaigns to test simulators such as gem5 [24, 25, 26].

c) Oracles: MLIR-MRacle offers two oracle modes. The outcome oracle checks the expected relation over outcome sets, using a Poisson bound to separate outcomes too rare to support a verdict from genuine violations; inconclusive trials are rerun with freshly agitated clones until they pass, fail, or exhaust a rerun budget. The adaptive oracle compares pooled distributions by squared Hellinger distance against a noise envelope obtained by permuting clones between the two sides, so its threshold adapts to the scheduling noise observed. adaptive is an instance of statistical metamorphic testing [22].

d) Campaign driver: The driver is built on libafl [23]. Each trial runs in a separate worker process that calls the engine, so a crash or hang in MLIR is recorded as a finding rather than ending the campaign. Feedback combines edge coverage from instrumented MLIR builds with semantic feedback: the MR applied, its expected relation, IR features and the set of observed outcomes. Raw outcome counts are deliberately excluded from novelty, since sampling noise alone would otherwise grow the corpus. Every failure is archived with its transformed program, outcomes, verdict and build and environment metadata, and a replay tool reruns any archived trial outside the campaign.

e) Implementation: MLIR-MRacle comprises 8,594 lines of C++, and 1,583 lines of Rust. A single Cargo build fetches and builds a pinned LLVM/MLIR (373c54d), the engine and its sanitizer variants.

Usage

a) Installation: MLIR-MRacle runs on Linux and builds with cargo build --locked --release (Rust 1.93.1, Clang 21 with libomp, CMake and Ninja). The first build fetches the pinned LLVM/MLIR and builds it with coverage instrumentation, and as such, is expensive; later builds reuse it. A Dockerfile packages the same toolchain for users who prefer a container, and prebuilt Docker images are archived on Zenodo [28].

b) Running a campaign: A campaign is launched with mracle-fuzz, choosing a seed corpus, the memory model whose MRs are enabled (--model; generic MRs are always included), the execution backend (--backend) and the oracle mode (--comparison):

target/release/mracle-fuzz \
  --input corpus/seeds --output runs/demo \
  --model armv8 --backend jit \
  --comparison adaptive --duration-secs 86400

The output directory records the campaign configuration, build and environment metadata, an event log, and a summary, alongside the queue (with provenance for every generated program) and one directory per trial holding its logs and coverage. When a trial fails, its directory also keeps the source and transformed MLIR, the lowered LLVM IR and the oracle’s run information.

c) Inspecting a finding: Any trial can be rerun outside the campaign from its saved outputs, which writes a fresh artefact directory:

target/release/mracle-replay \
  runs/demo/runs/00000000/request.json \
  --output replay-00000000

Budgets such as repetitions, run counts, and timeouts can be overridden, e.g. to confirm a rare outcome with more executions.

d) Extending MLIR-MRacle: Built-in MRs are functions wrapped in a FunctionTransform (listing 1). The function returns true only if the mutation was applied successfully; the entry names the MR, its memory model and its expected relation, which the oracle consumes. MRs that need state can instead subclass MetamorphicTransform and call registerTransform().

bool tryMyRewrite(func::FuncOp op, RewriterBase &rw,
                  std::mt19937 &rng) {
  // precondition -> safety check -> rewrite
}
const FunctionTransform kMyModelTransforms[] = {
  FunctionTransform("my-rewrite", "my-model",
                    &tryMyRewrite,
                    OutcomeRelation::Equality),
};

A memory model is simply a named table of this form, exposed through a getter declared in Transforms.h and added to the registry in MetamorphicTransform.cpp; it then becomes selectable via --model. Oracle modes and backends are added in the same way at both layers: a variant in the driver’s Comparison or Backend enum, its counterpart in the engine, and a case in the pipeline’s dispatch. New seeds are valid OpenMP .mlir programs placed in corpus/seeds, though this can similarly be extended to other dialects with a Backend extension, such a use would be beneficial for dialect developers wishing to test their dialect’s memory model conformance.

OracleBackendTrials/hEdges (k)CorpusFindings
outcomeJIT2,55793.61,15810,784
outcomenative84682.27149,450
adaptiveJIT47183.34555,699
adaptivenative16272.0267667

Table 1. Results per configuration over three 12-hour runs. Trials/h, edges and corpus size are means; findings are totals.

Case Study

We ran a small campaign to show that MLIR-MRacle operates at scale on real Armv8 hardware, and to give practitioners a sense of the trade-offs between its configurations.

a) Setup: We ran all campaigns on six Armv8 m400 nodes of the Cloudlab testbed [29], each with eight 64-bit ARMv8 (Atlas/A57) cores, and 64GB system memory, with up to four concurrent jobs per node. We crossed the two oracle modes with both backends, giving four configurations, and ran each for 12 hours, three times, with the Armv8 and generic MRs enabled, and a corpus of 46 seeds, which were manually translated to OpenMP from the C4 dataset [15].

b) Results: Table I summarises the 12 runs. Across them, MLIR-MRacle attempted 145,296 trials, of which 118,696 completed as normal executions, and 26,600 were flagged as findings. The outcome oracle with the JIT backend was the clear leader, at about 2,560 trials per hour, 5–16× the rate of the adaptive oracle; it also reached the most coverage (93.6k edges) and by far the largest corpus (1,158 programs, against 267–455 for the adaptive oracle). The native backend was around three times slower than JIT under every oracle and reached about 11k fewer edges, although this comparison is not strictly like for like: the JIT backend runs more of the compiler in process, so some of its additional coverage can be attributed to the code required to manage the runtime, rather than the intended exploration.

c) Findings: Flag rates differ between the two oracles, and the pattern reverses across backends. Roughly 34% of adaptive/JIT trials were flagged, against 12% for outcome/JIT; on the native backend, the adaptive oracle flagged roughly 11%, against 31% for outcome. Each finding is a candidate for further triage rather than a confirmed bug: its saved artefacts and replayable output allow a developer to inspect the transformed program and its outcomes directly. Triage of these findings is ongoing; the OpenMP verifier bug [6] was found during earlier ad-hoc running of the tool during its development.

d) Limitations: Three 12-hour repetitions on one platform cannot support statistical claims [30], and coverage is not directly comparable across backends, so we present these results as an indication of use cases rather than an evaluation of effectiveness.

Applications

Because MRs, memory models, oracles and backends vary independently, MLIR-MRacle can be adapted to several groups of users.

a) MLIR maintainers: Short outcome/JIT campaigns could run alongside changes to the OpenMP dialect and its lowerings, with archived trials replayed as self-contained reproducers for bug reports.

b) Dialect developers: Developers of new concurrency dialects can supply seeds in their dialect and an execution backend, then use the generic MRs to check that their constructs lower to the behaviour their memory model intends. The JIT backend, which runs lowering and code generation in process, gives fast feedback at this stage.

c) Target developers: Teams deploying to targets with weaker or less well-documented memory models can add that model as a table of MRs (Section III) and use the native backend, which links against the real OpenMP runtime, to test the toolchain as deployed. A simulator backend would extend the same campaigns to architectural simulators.

These uses compose: a high-throughput outcome/JIT campaign can generate candidate findings whose inputs seed a slower adaptive campaign, separating likely faults from scheduling noise. We have not evaluated this workflow here.

Conclusion

We presented MLIR-MRacle, a memory-model-aware metamorphic fuzzer for concurrent MLIR. By gating MRs on the target memory model and judging outcomes with statistical oracles, it tests concurrency lowering without assuming determinism or requiring complex formal models. Its MRs, memory models, oracles and backends are independent extension points; future work will add further memory models and a simulator backend, and extend MLIR-MRacle beyond the OpenMP dialect.

Availability: MLIR-MRacle, its seed corpus, campaign scripts and results are available at https://github.com/dakaidan/MLIR-MRacle-Tool, and archived on Zenodo [28].

References

  1. [1]C. Lattner, M. Amini, U. Bondhugula, A. Cohen, A. Davis, J. Pienaar, R. Riddle, T. Shpeisman, N. Vasilache, and O. Zinenko, “MLIR: Scaling compiler infrastructure for domain specific computation,” in CGO, 2021, pp. 2–14.DOI
  2. [2]J. Alglave, L. Maranget, and M. Tautschnig, “Herding cats: Modelling, simulation, testing, and data mining for weak memory,” ACM TOPLAS, vol. 36, no. 2, pp. 1–74, 2014.
  3. [3]C. Pulte, S. Flur, W. Deacon, J. French, S. Sarkar, and P. Sewell, “Simplifying arm concurrency: multicopy-atomic axiomatic and operational models for armv8,” PACMPL, vol. 2, no. POPL, pp. 1–29, 2017.
  4. [4]V. Vafeiadis, T. Balabonski, S. Chakraborty, R. Morisset, and F. Zappa Nardelli, “Common compiler optimisations are invalid in the c11 memory model and what we can do about it,” in POPL, 2015, pp. 209–220.
  5. [5]R. Morisset, P. Pawar, and F. Zappa Nardelli, “Compiler testing via a theory of sound optimisations in the c11/c++ 11 memory model,” ACM SIGPLAN Notices, vol. 48, no. 6, pp. 187–196, 2013.
  6. [6]Gabayoyo, “[MLIR][OpenMP] verification of CriticalOp does not check for nested CriticalOps,” GitHub issue, LLVM Project #217354, Aug. 2026. [Online]. Available: https://github.com/llvm/llvm-project/issues/217354
  7. [7]E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The Oracle Problem in Software Testing: A Survey,” IEEE TSE, vol. 41, no. 5, pp. 507–525, May 2015.DOI
  8. [8]J. Kirkham, T. Sorensen, E. Tureci, and M. Martonosi, “Foundations of empirical memory consistency testing,” Proc. ACM Program. Lang., vol. 4, no. OOPSLA, pp. 226:1–226:29, Nov. 2020.DOI
  9. [9]J. Alglave, M. Batty, A. F. Donaldson, G. Gopalakrishnan, J. Ketema, D. Poetzl, T. Sorensen, and J. Wicker­son, “GPU Concurrency: Weak Behaviours and Programming Assumptions,” SIGARCH Comput. Archit. News, vol. 43, no. 1, pp. 577–591, Mar. 2015.
  10. [10]H. Wang, J. Chen, C. Xie, S. Liu, Z. Wang, Q. Shen, and Y. Zhao, “Mlirsmir: Random program generation for fuzzing mlir compiler infrastructure,” in ASE. IEEE, 2023, pp. 1555–1566.DOI
  11. [11]C. Suo, J. Chen, S. Liu, J. Jiang, Y. Zhao, and J. Wang, “Fuzzing mlir compiler infrastructure via operation dependency analysis,” in ISSTA, 2024, pp. 1287–1299.DOI
  12. [12]B. Lim­panukorn, J. Wang, H. J. Kang, Z. Zhou, and M. Kim, “Fuzzing mlir compilers with custom mutation synthesis,” in ICSE. IEEE, 2025, pp. 217–229.
  13. [13]P. Yu, N. Wu, and A. F. Donaldson, “Ratte: Fuzzing for miscompilations in multi-level compilers using composable semantics,” in ASPLOS, 2025, pp. 966–981.DOI
  14. [14]J. Liang, S. Huang, and T. Su, “Finding bugs in mlir compiler infrastructure via lowering space exploration,” in ASE. IEEE, 2025, pp. 636–647.DOI
  15. [15]M. Windsor, A. F. Donaldson, and J. Wicker­son, “C4: the c compiler concurrency checker,” in ISSTA, 2021, pp. 670–673.DOI
  16. [16]L. Geeson and L. Smith, “Compiler testing with relaxed memory models,” in CGO. IEEE, 2024, pp. 334–348.arxiv.org/abs/2310.12337
  17. [17]V. Le, M. Afshari, and Z. Su, “Compiler validation via equivalence modulo inputs,” in PLDI. Edinburgh United Kingdom: ACM, Jun. 2014, pp. 216–226.
  18. [18]A. F. Donaldson and A. Lascu, “Metamorphic testing for (graphics) compilers,” in MET, 2016, pp. 44–47.DOI
  19. [19]A. F. Donaldson, H. Evrard, A. Lascu, and P. Thomson, “Automated testing of graphics shader compilers,” PACMPL, vol. 1, no. OOPSLA, pp. 1–29, 2017.DOI
  20. [20]T. Y. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. Tse, and Z. Q. Zhou, “Metamorphic testing: A review of challenges and opportunities,” ACM CSUR, vol. 51, no. 1, pp. 1–27, 2018.
  21. [21]S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-Cortés, “A survey on metamorphic testing,” IEEE TSE, vol. 42, no. 9, pp. 805–824, 2016.DOI
  22. [22]R. Guderlei and J. Mayer, “Statistical metamorphic testing testing programs with random output by means of statistical hypothesis tests and metamorphic testing,” in QSIC. IEEE, 2007, pp. 404–409.DOI
  23. [23]A. Fiordalisi, D. C. Maier, D. Zhang, and D. Balzarotti, “Libafl: A framework to build modular and reusable fuzzers,” in CCS, 2022, pp. 1051–1065.
  24. [24]K. Even-Mendoza, H. D. Menéndez, W. B. Langdon, A. Dakhama, J. Petke, and B. R. Bruce, “Search+ llm-based testing for arm simulators,” in ICSE. IEEE, 2025, pp. 469–480.
  25. [25]A. Dakhama, K. Even-Mendoza, W. B. Langdon, H. Menendez, and J. Petke, “SearchGem5: Towards reliable gem5 with search based software testing and large language models,” in SSBSE. Springer Nature Switzerland Cham, 2023, pp. 160–166.DOI
  26. [26]A. Dakhama, K. Even-Mendoza, W. B. Langdon, H. D. Menéndez, and J. Petke, “Enhancing search-based testing with LLMs for finding bugs in system simulators,” Autom. Softw. Eng., vol. 32, no. 2, p. 63, Jul. 2025.DOI
  27. [27]J. Alglave, L. Maranget, S. Sarkar, and P. Sewell, “Litmus: Running tests against hardware,” in TACAS. Springer, 2011, pp. 41–44.
  28. [28]A. Gabayoyo, A. Dakhama, and A. Rajan, “MLIR-MRacle: Memory-Model-Aware Metamorphic Fuzzing of Concurrent MLIR — Artifact,” 2026. [Online]. Available: https://doi.org/10.5281/zenodo.23545733
  29. [29]D. Duplyakin, R. Ricci, A. Maricq, G. Wong, J. Duerig, E. Eide, L. Stoller, M. Hibler, D. Johnson, K. Webb et al., “The design and operation of {CloudLab},” in USENIX ATC, 2019, pp. 1–14.
  30. [30]G. Klees, A. Ruef, B. Cooper, S. Wei, and M. Hicks, “Evaluating fuzz testing,” in CCS, 2018, pp. 2123–2138.arxiv.org/abs/1808.09700

Paper details

vlnc:W_b6kcmdxq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents