betaFeedback

WorksPhysics and Astronomy

Noise-Aware Testing of NISQ Quantum Programs: Self-Baseline Oracle with a Controlled False-Positive Rate

Abstract

Testing quantum programs on Noisy Intermediate-Scale Quantum (NISQ) devices faces a test-oracle problem: hardware noise and finite-shot sampling make the runs of a correct program vary as much as those of a faulty one, so oracles with a fixed acceptance threshold produce false positives that depend on the program and the backend. We present a noise-aware testing framework built on self-baseline calibration: each run of the program under test is compared with repeated executions of a trusted reference program on the same backend, and the acceptance threshold is the (1−α)(1-α) quantile of the finite-shot null distribution of the Jensen-Shannon divergence, so the oracle's false-positive rate is a level set in advance. A noncentral chi-square model predicts detection power, a homogeneity test and a two-run sentinel detect drift, a mutation engine labels every mutant exactly within a three-class noise-aware taxonomy, and zero-noise extrapolation attributes rejections to faults or noise. In simulation on nine core benchmarks, 32 QASMBench programs, ten real bugs, seven noise regimes and three device snapshots, the oracle's false-positive rate is 0.0011 at α=10−3α= 10^{-3} over 9,000 correct runs, against 36-74% for fixed thresholds, 8.9% for a QOIN-style filter and 10.8% for a globally calibrated threshold, while attaining the highest F1 on non-equivalent faults (0.941). On an IBM Heron device, using about 14 minutes of QPU time, it rejected none of 82 correct runs and detected 120 of 121 faults and semantic mutants, and a two-run sentinel reduced drift-induced false positives from 27.4% to 1.8%.

Introduction

A quantum program can be correct and still fail its test. On today’s Noisy Intermediate-Scale Quantum (NISQ) hardware, every gate is imperfect, every measurement is stochastic, and the resulting run-to-run variation can be statistically indistinguishable from a software fault [1, 2, 3, 4]. This is the normal operating condition of quantum software, and it challenges an assumption that has long underpinned software testing: that a correct program produces a recognizable output, subject at most to a small and uniform tolerance. Traditional test oracles interpret deviations from an expected output as evidence of defects; on NISQ hardware, legitimate hardware-induced variation produces the same observable behavior, and empirical studies report failures that arise from hardware behavior rather than implementation errors [4, 5]. The central challenge is therefore deciding whether an observed execution should be regarded as correct.

The oracle problem is harder in quantum computing than in classical software: the no-cloning theorem prevents complete inspection of quantum states, measurement collapses the state observed, and the exponential state space precludes exhaustive verification. Noise adds a further layer: the variability of an execution becomes a property of the backend, so correctness decisions must separate deviations caused by faults from those caused by device noise and finite-shot sampling. Yet practitioners often resort to deterministic assertions, fixed seeds and approximate equality [6], and evaluation practices remain largely inherited from classical software engineering [7].

Existing research has addressed this problem through three broad classes of correctness oracles. The first relies on fixed-threshold decision rules, where executions are accepted whenever the observed divergence remains below a predetermined global tolerance. Such thresholds inevitably become unsound once hardware noise exceeds the chosen limit; here and throughout, we call an oracle sound in a setting if it rarely rejects correct runs there (at its nominal level for an oracle that has one, and at most 5% otherwise). The second class adopts noise-filtering strategies, exemplified by approaches such as QOIN [8], which estimate hardware noise and compensate for it before comparing program behavior with the ideal noiseless output. These methods are conceptually related to quantum error mitigation [9, 10] but require an explicit or learned representation of the underlying noise characteristics, making them dependent on particular hardware configurations. A third line of work employs metamorphic, property-based or statistical testing formulations [11, 12, 13, 14, 15], reducing dependence on ideal outputs by expressing correctness through programmer-specified relations or statistical properties. While effective in some settings, these approaches shift the burden of correctness from the program itself to externally supplied specifications.

Fixed-threshold and noise-filtering techniques evaluate executions against a reference the device cannot realize, the noiseless ideal output; metamorphic approaches replace it with specifications that the developer must supply. None constructs the oracle from the behavior of a trusted version of the program on the target backend, and none of the threshold-based oracles states the false-positive rate its threshold delivers under noise.

This paper adopts a different perspective for the setting in which a trusted reference version of the program is available, which is the setting of mutation analysis and regression testing: we decide whether a program under test P′P' behaves like the reference PP on the same backend. Instead of comparing every execution against an ideal reference that the hardware cannot faithfully produce, we compare it against a baseline that PP establishes about itself through repeated execution on that backend. We refer to this principle as self-baseline calibration. Its key observation is that two correct executions are independent multinomial samples of one output distribution, so the distribution of their divergence under the hypothesis “P′P' behaves like PP” is known from finite-shot statistics. The resulting noise-aware oracle accepts a run whenever its divergence from the self-baseline lies below the (1−α)(1-\alpha) quantile of that null distribution. The false-positive rate is therefore a nominal level α\alpha chosen by the tester, detection power follows from a noncentral chi-square model, the calibration runs additionally serve to detect drift through a homogeneity test, and a short sentinel before each test batch decides when to recalibrate. The oracle requires neither the ideal output, nor a parametric noise model, nor a learned noise representation; it does require the trusted reference program.

Across nine core benchmarks and five noise regimes, the observed false-positive rate (FPR) of the noise-aware oracle is 0.0011 at α=10−3\alpha= 10^{-3} over 9,000 correct runs (0.0013 on 32 QASMBench programs), whereas fixed thresholds reach 36–74%, a QOIN-style filter 8.9% and a threshold calibrated globally on the device’s noise model 10.8%. On non-equivalent injected faults, the oracle attains F1F_1 0.941; it rejects 93.8% of semantic mutants and detects 96.4% of the runs of observable real bugs from Bugs4Q. On an IBM Heron device, it rejected none of 82 correct runs and detected 120 of 121 faults and semantic mutants. The evaluation rests on exact semantic labels: every mutant and fault is classified as equivalent or not by statevector simulation before any noisy execution, so detection is never credited for, or penalized on, programs indistinguishable from the original.

Research gap and problem formulation

Four methodological limitations remain. First, quantum mutation testing largely inherits classical, noise-oblivious semantics, so a killed mutant may reflect hardware variation rather than a fault, and the rate at which quantum operators produce equivalent mutants, a classical concern [16, 17], has not been quantified. Second, oracles rely on fixed thresholds or hardware-specific noise representations, neither sound across heterogeneous noise, and none sets its false-positive rate in advance or accounts for the false negatives it trades for soundness. Third, mutation taxonomies introduce categories without establishing that they are statistically distinguishable under noise. Fourth, failure attribution has seen little evaluation against labeled ground truth and typically relies on a noiseless simulator, which is unavailable on hardware. We therefore investigate the following problem:

How can a noise-aware testing framework for NISQ quantum programs provide a correctness oracle with a controlled false-positive rate under heterogeneous noise, while supporting mutation testing with validated mutants, an empirically grounded mutation taxonomy, quantified fault-detection capability, and attribution of observed failures to software defects or hardware noise?

We decompose it into five research questions, each answered on simulated noise, device snapshots and IBM Quantum hardware, in the order in which a tester relies on the framework: the oracle must be sound (RQ1); mutation analysis depends on its verdicts (RQ2); attribution explains ambiguous verdicts (RQ3); sensitivity quantifies what soundness costs (RQ4); and drift and cost determine practicality (RQ5).

RQ1. Does the noise-aware oracle control its false-positive rate at the nominal level across heterogeneous noise conditions and device noise models, and how does it compare with fixed-threshold, envelope-based, noise-filtering and noise-model-calibrated oracles?

RQ2. How often do quantum mutation operators produce equivalent mutants, is the proposed noise-aware mutation taxonomy empirically supported, and to what extent does mutation testing under the noise-aware oracle detect semantic mutants?

RQ3. When the oracle reports a deviation, can the failure be attributed to a software fault or to hardware noise without a noiseless simulator?

RQ4. What fault-detection capability (recall, precision, F1) does the noise-aware oracle achieve, how well does the analytic power model predict it, and how does it scale with circuit size?

RQ5. How does the oracle behave under drift between calibration and testing, how should recalibration be triggered, and what does calibration cost in executions?

Contributions

This paper makes four contributions.

  1. A noise-aware oracle with a nominal false-positive level. We formulate the NISQ test oracle problem in the differential setting as a two-sample test against a self-baseline on the same backend, and design the oracle around it: the acceptance threshold is the (1−α)(1-\alpha) quantile of the finite-shot null distribution of the Jensen–Shannon divergence, estimated by parametric bootstrap with a correction for unobserved outcomes;

pooling KK calibration runs raises sensitivity, with a single-run variant for tight execution budgets; a homogeneity test checks that the baseline is valid; a noncentral chi-square model predicts recall and the shot budget for a target power; and a sentinel policy decides when to recalibrate. The underlying test statistics are classical [18]; the contribution is the oracle design and the evidence that fixed, filtered and globally calibrated thresholds cannot keep the false-positive rate at a level set in advance across programs and noise levels.

(2) A noise-aware mutation testing methodology with exact semantic labels. We provide a mutation engine that places operators before measurement and labels every mutant as equivalent or not on the tested inputs by statevector simulation, a quantification of equivalent-mutant rates for quantum mutation operators and fault injectors, and a three-class mutation taxonomy (semantic, noise-amplifying, noise-neutral) validated with stated hypotheses, tests that account for shared calibrations, effect sizes and a dose-response model of noise amplification.

(3) A hardware-executable bug–noise attribution procedure. We attribute a reported deviation to a fault or to noise by zero-noise extrapolation of the output distributions of both programs using global unitary folding, and evaluate it against exact, non-circular labels.

(4) A comprehensive empirical evaluation. We evaluate the framework on nine core benchmarks and 32 QASMBench programs, ten real bugs from Bugs4Q, five depolarizing and two coherent-error noise regimes, three device-snapshot backends, circuits of up to 16 qubits and IBM Quantum hardware, against fixed-threshold, envelope-based (statistical process control), QOIN-style, noise-model-calibrated and re-execution baselines, reporting full confusion matrices, cluster-robust confidence intervals and variance-based sensitivity indices, and characterize where the approach stops working.

The underlying principle, deriving the acceptance threshold from a system’s own observed variability, may also apply to other software observed through a noisy channel, such as flaky tests in continuous integration [19, 20, 21] and machine-learning inference under numerical nondeterminism [22]. Every experiment and quantitative claim in this paper, however, concerns quantum programs on NISQ hardware or models of it; § 6.4 discusses this scope.

Paper organization. § 2 reviews related work, § 3 presents the framework, § 4 the study design, and § 5 the results. § 6 discusses implications, § 7 threats to validity, and § 8 concludes.

Background and Related Work

Existing quantum software testing techniques respond to NISQ noise in three ways: they filter it out via error mitigation or learned noise models, tolerate it with a fixed divergence threshold, or restructure the correctness check via metamorphic relations, assertions, property-based testing or richer measurement specifications [5, 11, 23].

Terminology. Noise denotes all stochastic deviations of a physical execution from the ideal unitary semantics of a program: gate, decoherence and readout errors and their drift. It is distinct from finite-shot sampling variability, which arises even on a noiseless simulator. A fault (or bug) is a defect that changes the program’s ideal semantics.

Quantum Software Testing Tools and Adequacy

Assertion-based approaches such as AssertsQ [12] verify properties of quantum states; MorpH [13] and QDiff [24] apply metamorphic and differential testing [25]; QMutPy [26, 27] and Muskit [28] adapt mutation analysis to Qiskit programs; and Pontillo et al. [15] adapt property-based testing to noisy quantum computers. Most of these techniques assume near-ideal execution or rely on manually selected thresholds whose robustness under heterogeneous noise has not been established. Test-generation frameworks such as Quito [29], QuCAT [30] and QuSBT [31] address input generation rather than oracle soundness, mutation adequacy or failure attribution.

Coverage criteria for quantum programs include input and output coverage [32] and gate branch coverage [33]; they measure structural and behavioral exercise but do not address whether deviations arise from faults or noise, and we treat them as complementary to the adequacy metrics introduced here.

Mutation Testing for Quantum Programs

Quantum mutation tools such as QMutPy and Muskit define quantum-specific operators (gate insertion, deletion, replacement and measurement manipulation) and report mutation scores, treating observable deviations as kills [26, 28]. They inherit classical assumptions: behavioral differences are interpreted as faults, although under NISQ execution noise can itself induce deviations or mask genuine faults. The closest work to ours is the empirical study of Fortz et al. [34], which examines mutation analysis of 41 quantum programs under noisy simulators of three IBM devices. They report that noise substantially changes the behavioral distance between programs and their mutants, making equivalent mutants harder to separate from genuine faults; that density-matrix metrics discriminate best but are unavailable on hardware, whereas output-distribution metrics reach up to 73% accuracy; that thresholds set for the noise profile outperform noiseless thresholds; and that noise effects depend more on the algorithm and circuit than on the mutation type. Our work is complementary. Fortz et al. compare distance metrics and threshold-selection strategies; we derive the threshold of an output-distribution oracle from finite-shot statistics so that its false-positive rate is set in advance, label mutant equivalence exactly so that detection is measured only on non-equivalent mutants, and validate a mutation taxonomy and an attribution procedure on top of that oracle. Their observation that noise effects depend more on the circuit than on the mutation type is consistent with our finding that the two semantic operator families are not statistically separable (§ 5.2). We also evaluate their threshold construction as a baseline (§ 4.5).

Pan et al. [4] analyze the influence of noise on program behavior, and Virani et al. [35] propose statistical techniques for distinguishing bug-induced failures from noise; neither redefines mutation adequacy for NISQ execution.

Noise-Aware Testing and Oracles

QOIN [8] learns backend- and circuit-specific noise characteristics by transfer learning and filters noise from execution results before applying an existing oracle. Such approaches recover an approximation of the ideal behavior but preserve the underlying oracle semantics.

A complementary line of research investigates improvements to distribution-based correctness oracles by redesigning the measurements used to characterize quantum program behavior. Oldfield et al. [36] observe that treating quantum programs purely as classical output distributions suffers from two fundamental limitations: the exponential growth of the output space and the inability of computational-basis measurements to detect certain classes of quantum faults, such as relative phase errors. They address these limitations through reduced program specifications and projective measurements in multiple bases. Related work on assertion-based verification [12, 37] and specification-aware test generation [38] similarly focuses on improving fault detectability by enriching program specifications or measurement strategies. Bensoussan et al. [39] propose live noise fingerprinting: device noise is characterized continuously from empirical observations with classical shadow tomography, and fingerprints of different platforms are compared to flag programs likely to behave differently across them. Noise fingerprints characterize the device; our oracle characterizes the variability of a program on a device, and the drift test of § 3.2 detects when that variability changes. The two are natural complements for deciding when recalibration is needed.

Less attention has been given to oracles that judge correctness directly under noisy execution, with a false-positive level set in advance, without an ideal reference or explicit noise mitigation.

Statistical Foundations: From Process Control to Self-Baseline Calibration

Estimating a process mean and dispersion from repeated observations and flagging deviations beyond a μ+kσ\mu+ k\sigma control limit forms the basis of Shewhart statistical process control (SPC) [40], while detecting distributional changes over time has been extensively studied in sequential change detection and CUSUM methods [41, 42]. Similar baselining principles appear in classical flaky-test analysis and performance regression testing, where current executions are compared against historical execution baselines. A control-limit envelope τ=max⁡(μ+kσ,τmin⁡)\tau= \max(\mu+ k\sigma, \tau_{\min}) over pairwise divergences of repeated runs is the natural transfer of this idea to quantum output distributions, and we include it as a baseline (the SPC envelope). It has two weaknesses in this setting: the pairwise divergences of KK runs are dependent and skewed, so kk carries no coverage guarantee, and a floor τmin⁡\tau_{\min} is needed to prevent degenerate limits on concentrated outputs, which then dominates the decision. Our oracle instead uses the known sampling distribution of divergence statistics between multinomial samples: the Jensen–Shannon divergence is a member of the power-divergence family whose scaled form is asymptotically chi-square distributed under the null hypothesis of equal distributions [18, 43], so a threshold at a nominal false-positive level can be computed for each program and backend.

The statistical machinery is therefore classical, and we do not claim a new test statistic. Statistical tests have been used in quantum software testing before: statistical assertions check properties of intermediate states with chi-square tests [14], and property-based testing has been adapted to noisy hardware [15]. These approaches test a program against a specification of its expected behavior. What has been missing is a differential oracle that tests a program against the noisy behavior of a trusted version of itself on the same backend, together with an account of what such an oracle needs to be usable in a testing process: a threshold whose false-positive rate is chosen rather than tuned, a sensitivity that can be planned, a check that the baseline is still valid, labels that separate detectable from undetectable mutants, and a way to explain the rejections that are caused by noise. Our contribution is this oracle design and its empirical characterization, including the evidence that the thresholds in current use, fixed, filtered or calibrated globally on a noise model, cannot keep the false-positive rate at a level set in advance across programs and noise levels.

Methodological Gaps

Under noise, a killed mutant no longer implies test effectiveness: noise may amplify small syntactic changes so that a correct program or an equivalent mutant fails, or mask genuine faults, yet QMutPy and Muskit evaluate mutants with noise-oblivious oracles [26, 28], which may overestimate adequacy [5, 11]. Noise-aware simulation and benchmarking studies confirm that noise dominates observable behavior as depth, entanglement and gate density grow, even for correct programs [4, 44, 45, 46]. Distribution-based oracles typically use a manually selected threshold that is fixed across programs and backends [12, 13], and no fixed threshold states the false-positive rate it delivers. Mutation taxonomies are motivated by operator semantics but not evaluated for statistically indistinguishable behavior under noise, and the equivalent-mutant problem, undecidable in general for classical software [16, 17], is decidable for quantum programs of moderate width via statevector simulation, which existing tools do not exploit. Failure attribution matters for debugging, fault localization and repair [47, 48, 49, 50], which assume that failures are genuine defects, but attribution mechanisms [35] are evaluated on limited cases, and those that need a noiseless simulation cannot run on hardware beyond classical simulation. Methodological analyses also report inconsistent experimental design, oracle selection and noise handling across empirical studies [7].

Table 1 summarizes representative approaches across four dimensions: noise-aware oracles, noise-sensitive mutation models, adequacy metrics for noisy execution and failure attribution. Existing approaches address one or two in isolation; to our knowledge, none integrates them in one methodology evaluated under a unified protocol.

WorkNoise-aware oracleNoise-sensitive mutationsNoise-aware metricsFailure attribution
AssertsQ [12]✓✗✗✗
MorphQ [13]✗✗✗✗
QDiff [24]✗✗✗✗
QMutPy [26]✗✗✗✗
Muskit [28]✗✗✗✗
QOIN [8]✓✗✗✗
Pontolillo et al. [15]✓✗✗✗
Fortz et al. [34]✓✗✗✗
Virani et al. [35]✗✗✗✓
This work✓✓✓✓

Table 1. Comparison of representative quantum software testing approaches. A ✓ marks a dimension the approach treats as a first-class concern: noise-aware oracle, the correctness decision accounts for hardware noise; noise-sensitive mutations, mutation operators categorized by their interaction with noise; noise-aware metrics, adequacy metrics defined under noisy execution; failure attribution, a mechanism separating bug-induced from noise-induced failures.

The Noise-Aware Testing Framework

The framework has three components (Figure 1): a noise-aware oracle whose threshold is derived from the finite-shot sampling distribution of the divergence between repeated executions of the reference program (§ 3.2), a mutation engine whose mutants carry exact semantic labels within a three-class taxonomy (§ 3.4), and a bug–noise attribution procedure based on zero-noise extrapolation (§ 3.6). A power model (§ 3.3) and adequacy metrics (§ 3.5) complete it.

Workflow of the framework

Figure 1. Workflow of the framework. The reference program PP is executed KK times on the target backend; the pooled counts form the self-baseline reference, the homogeneity test checks the calibration runs for drift, and the null-quantile threshold τα\tau_\alpha is computed from the reference counts. Each program under test P′P' (a mutant, an injected fault, or a new version) is executed once and accepted if its divergence from the reference does not exceed τα\tau_\alpha. Rejected programs can be passed to zero-noise-extrapolation attribution, which labels the deviation as a fault or as a noise effect. Before each batch of tests, a sentinel of two executions of PP checks that the reference is still valid and triggers recalibration if it is not (§ 3.2). Exact semantic labels (statevector) are used only to construct and evaluate mutants and faults, never by the oracle itself.

Setting and Problem Statement

Self-baseline calibration is the general principle: judging a program’s behavior against a baseline established by repeated execution of a trusted reference on the same backend, rather than against a fixed tolerance or an ideal output. This paper instantiates it with Jensen–Shannon divergences and their finite-shot null distribution; the noise-aware oracle is the resulting artifact. The oracle is differential: it decides whether a program under test P′P' behaves like a trusted reference PP on the same backend B\mathcal{B}, as in mutation analysis (P′P' a mutant) and regression testing (PP the last accepted version). The correctness of PP itself must be established by other means, such as specification-based or property-based testing on small instances.

Definition 3.1 (NISQ testing problem). Given a reference program PP, a program under test P′P' and a noisy backend B\mathcal{B}, the NISQ testing problem is to decide whether the output distribution of P′P' on B\mathcal{B} differs from that of PP on B\mathcal{B} by more than finite-shot sampling explains, and, if it does, whether the difference is caused by a change of the program’s ideal semantics (a fault) or only by its interaction with hardware noise (a noise effect). The attribution procedure of § 3.6 returns one of {Bug, Noise}.

The framework accepts a Qiskit circuit ending in computational-basis measurement of some or all qubits (variational parameters fixed), a backend (a noisy simulator, a Qiskit fake backend or an IBM Quantum device), and a configuration (shots NN, calibration size KK, level α\alpha, seed).

Observability limitation. A change that leaves the measured distribution unchanged for the tested input cannot be detected by any threshold on that distribution; relative phase errors immediately before measurement are the canonical example. Equivalence and detection are therefore defined with respect to the measured output distribution, and injected faults that change the unitary but not the measured output are reported separately. Richer measurement specifications [36] address this limitation and are orthogonal to the threshold construction.

Noise-Aware Oracle

Divergence statistic. Let pp and qq be empirical output distributions over measured bitstrings. We use the base-2 Jensen–Shannon divergence [51]

D(p,q)=12KL⁡(p∥m)+12KL⁡(q∥m),m=12(p+q),(1)D(p,q)=\tfrac{1}{2}\operatorname{KL}(p\mathbin{\|}m)+\tfrac{1}{2}\operatorname{KL}(q\mathbin{\|}m),\qquad m=\tfrac{1}{2}(p+q), \tag*{(1)}

which is symmetric and bounded in [0,1][0,1]. The two distributions are aligned over the union of observed bitstrings, additively smoothed with ε=10−12\varepsilon=10^{-12}, and renormalized before DD is computed. We state this explicitly because the Jensen–Shannon distance D\sqrt{D} and the divergence DD are frequently conflated; all thresholds in this paper are on the divergence scale.

Null distribution of the divergence. Let the reference be estimated from nrefn_{\mathrm{ref}} shots and the program under test from ntestn_{\mathrm{test}} shots. Under the null hypothesis H0H_0: “P′P' has the same output distribution π\pi as PP on B\mathcal{B}”, the two count vectors are independent multinomial samples of π\pi. A second-order expansion of DD around π\pi gives Dln⁡2≈18∑x(p^x−q^x)2/πxD\ln2\approx\frac{1}{8}\sum_x(\hat{p}_x-\hat{q}_x)^2/\pi_x, and the multinomial central limit theorem gives the Pearson-type limit

T=8ln⁡2 D(p^,q^)1/nref+1/ntest→dχd−12,(2)T=\frac{8\ln2\,D(\hat{p},\hat{q})}{1/n_{\mathrm{ref}}+1/n_{\mathrm{test}}}\xrightarrow{d}\chi^2_{d-1}, \tag*{(2)}

where dd is the number of outcomes with πx>0\pi_x>0. This is the two-sample instance of the classical result for power-divergence statistics [18, 43]. Two consequences matter for testing. First, the divergence between two correct runs does not depend on how noisy the backend is, only on how many outcomes the noisy distribution occupies and on the shot budget: noise matters through dd. Second, a threshold at the nominal level α\alpha follows directly, valid asymptotically:

ταχ2=χd−1,1−α21/nref+1/ntest8ln⁡2.(3)\tau_\alpha^{\chi^2}=\chi^2_{d-1,1-\alpha}\frac{1/n_{\mathrm{ref}}+1/n_{\mathrm{test}}}{8\ln2}. \tag*{(3)}

Bootstrap threshold. Equation (3) requires dd and assumes large counts per outcome. The oracle therefore uses a parametric bootstrap [52] from the reference counts. The plug-in estimate π^\hat{\pi} is corrected for outcomes not yet observed: with f1f_1 and f2f_2 the numbers of outcomes observed once and twice in nn reference shots, S^0=f12/(2f2)\hat{S}_0=f_1^2/(2f_2) unseen outcomes (bias-corrected to f1(f1−1)/2f_1(f_1-1)/2 when f2=0f_2=0, and capped by the number of possible outcomes) [53] share the Good–Turing unseen mass f1/nf_1/n [54], and the observed probabilities are scaled by 1−f1/n1-f_1/n. We draw BB pairs (B=4000B=4000 in the main false-positive evaluation of RQ1 and in the hardware analyses, 2000 or 3000 in the other experiments; § 4.8) of multinomial samples of sizes (nref,ntest)(n_{\mathrm{ref}},n_{\mathrm{test}}) from π^\hat{\pi}, compute DD for each pair, and set τα\tau_\alpha to the empirical (1−α)(1-\alpha) quantile. Equation (3) is reported alongside as a closed-form check.

Pooled reference. The reference is the pool of KK calibration executions of PP, so nref=KNn_{\mathrm{ref}}=KN and the term 1/nref1/n_{\mathrm{ref}} shrinks with KK. Pooling yields a more precise reference, which lowers the threshold and raises power (§ 3.3), and several independent runs, which make the drift test below possible. We call the oracle with a pooled reference (K=8K=8 by default) the noise-aware oracle (NAO), and its single-reference variant (K=1K=1) NAO-1. NAO-1 forgoes both benefits for a calibration cost of one execution and is intended for stable, compiled-once backends; NAO is the recommended default, and NAO-1 is evaluated alongside it to measure the contribution of pooling and the drift test.

Algorithm 1 Noise-aware oracle (NAO)

Require: reference program PP, backend B\mathcal{B}, shots NN, calibration size KK, level α\alpha, bootstrap size BB, drift level αdrift\alpha_{\mathrm{drift}} and widening factor kk, sentinel size mm and level αs\alpha_s; program under test P′P'

Ensure: verdict (Pass/Fail), divergence DD, threshold τ\tau, drift flag

  1. if reference not cached then

  2. execute PP on B\mathcal{B} KK times with NN shots →c1,…,cK\to c_1,\ldots,c_K {one transpilation, fixed layout}

  3. cref←∑icic_{\mathrm{ref}} \leftarrow\sum_i c_i; π^←\hat{\pi} \leftarrow plug-in of crefc_{\mathrm{ref}} with unseen-outcome correction

  4. τ←(1−α)\tau\leftarrow(1-\alpha) quantile of DD over BB bootstrap pairs of sizes (KN,N)(KN,N) drawn from π^\hat{\pi}

  5. drift ←\leftarrow false

  6. if homogeneity test on c1,…,cKc_1,\ldots,c_K rejects at αdrift\alpha_{\mathrm{drift}} then

  7. τ←max⁡(τ,δˉ+ksδ)\tau\leftarrow\max(\tau,\bar{\delta}+k s_{\delta}); drift ←\leftarrow true

  8. end if

  9. end if

  10. if a new batch of tests begins then

  11. execute PP on B\mathcal{B} mm times with NN shots →s1,…,sm\to s_1,\ldots,s_m; s←∑jsjs \leftarrow\sum_j s_j {sentinel}

  12. τs←\tau_s \leftarrow threshold of lines 4–8 with level αs\alpha_s and test size mNmN, i.e. pairs of sizes (KN,mN)(KN,mN)

  13. if D(cref,s)>τsD(c_{\mathrm{ref}},s)>\tau_s then

  14. discard the cached reference and repeat lines 2–8 {recalibrate}

  15. end if

  16. end if

  17. execute P′P' on B\mathcal{B} with NN shots, on the initial layout of P→c′P \to c'

  18. return (D(cref,c′)≤τ,D(cref,c′),τ,drift)(D(c_{\mathrm{ref}},c')\leq\tau,D(c_{\mathrm{ref}},c'),\tau,\mathrm{drift})

Drift detection. Equation (2) holds only if the backend is stationary while PP and P′P' are executed. Under stationarity, the KK calibration runs are multinomial samples of one distribution, so the oracle applies a Pearson chi-square test of homogeneity to their K×dK \times d contingency table, merging outcomes with expected count below five per run [43]. If homogeneity is rejected at αdrift=0.01\alpha_{\mathrm{drift}}=0.01 (for example, because the noise drifted or the circuit was recompiled onto other qubits), the oracle reports drift and widens the threshold to max⁡(τα,δˉ+ksδ)\max(\tau_\alpha,\bar{\delta}+k s_\delta), where δˉ\bar{\delta} and sδs_\delta are the mean and standard deviation of the pairwise divergences among the calibration runs and k=4k=4. Drift after calibration is checked by a sentinel: before a batch of tests, PP is executed mm times, and the pooled runs are compared with the reference by the oracle’s own two-sample test at level αs\alpha_s; if it rejects, the reference is recalibrated. Because a false recalibration costs only KK executions, αs\alpha_s can be less strict than α\alpha, which raises the sentinel’s power. RQ5 evaluates m∈{1,2,4}m \in\{1,2,4\} and αs∈{10−3,10−2}\alpha_s \in\{10^{-3},10^{-2}\}; we use m=2m=2 and αs=10−2\alpha_s=10^{-2}.

Algorithm 1 summarizes the oracle.

What the level means. The level α\alpha is nominal, not an unconditional finite-sample guarantee. The rejection probability of a correct program equals α\alpha only to the extent that four approximations hold: the plug-in estimate π^\hat{\pi}, with its unseen-outcome correction, represents the support and shape of π\pi; the bootstrap quantile is close to the true null quantile (at α=10−3\alpha=10^{-3} and B=4000B=4000, only about four draws lie beyond it, so the level holds on average over windows, not in each window); the calibration and test runs are samples of one distribution (stationary backend, identical compilation); and there are enough shots per outcome to estimate the null from the reference. The first two are statistical approximations whose accuracy RQ1 measures; the last two are assumptions about the test setting, of which the drift test and the sentinel check stationarity. Our claim is therefore empirical: under the conditions evaluated in RQ1, the observed false-positive rate is consistent with α\alpha where these assumptions hold, and RQ1 reports where it is not.

Definition 3.2 (Noise-aware oracle). For a reference program PP, a program under test P′P', a backend B\mathcal{B} and a level α\alpha, the noise-aware oracle returns Pass if and only if D(p^ref,p^P′)≤τα(P,B)D(\hat{p}_{\mathrm{ref}},\hat{p}_{P'}) \le\tau_{\alpha}(P,\mathcal{B}), where p^ref\hat{p}_{\mathrm{ref}} is the pooled calibration distribution of PP on B\mathcal{B}, p^P′\hat{p}_{P'} the empirical distribution of one execution of P′P' on B\mathcal{B}, and τα\tau_{\alpha} the threshold of Algorithm 1. If P′P' has the same output distribution as PP and the backend is stationary, the oracle rejects with probability approximately α\alpha; the accuracy of the approximation depends on the number of shots per outcome (see What the level means above).

Detection Power and Shot Budget

If P′P' differs from PP, the scaled statistic TT of Equation (2) is approximately noncentral chi-square with d−1d-1 degrees of freedom and noncentrality λ=8ln⁡2 δ/(1/nref+1/ntest)\lambda= 8 \ln2 \, \delta/(1/n_{\mathrm{ref}} + 1/n_{\mathrm{test}}), where δ\delta is the divergence between the two true (noisy) output distributions [43]. The probability that the oracle detects the difference is therefore

power⁡(δ)=Pr⁡[χd−12(λ)>χd−1,1−α2].(4)\operatorname{power}(\delta) = \Pr\left[\chi^2_{d-1}(\lambda) > \chi^2_{d-1,1-\alpha}\right]. \tag*{(4)}

Equation (4) makes the soundness–sensitivity trade-off explicit and quantitative. It yields the minimum detectable divergence at a given shot budget and power, and conversely the shot budget needed to detect a fault of a given size. It also shows how noise reduces sensitivity: noise both shrinks δ\delta (it pulls the faulty and the correct distributions towards the maximally mixed distribution) and increases dd.

Noise-Aware Mutation Testing

Placement and exact labels. Operators that insert gates place them at a uniformly random position before the first measurement; operators that remove or rewrite gates act on gates before the first measurement. Gates placed after the final measurement cannot influence the measured output, and § 5.2 shows that such placement yields equivalent mutants in every case. For each generated mutant, the framework computes the exact ideal output distribution of the original and of the mutant by statevector simulation and records their divergence DidealD_{\mathrm{ideal}}; the mutant is equivalent if Dideal<10−10D_{\mathrm{ideal}} < 10^{-10} for the tested input. By default the input is ∣0⋯0⟩\lvert0 \cdots0 \rangle, the initial state of every benchmark; the framework can prepend preparation of arbitrary computational-basis inputs, and § 5.2 reports equivalence over four inputs. The exact labels require simulating the circuit, which is feasible for the program widths at which mutation analysis is performed in this paper; at larger widths they can be computed on the reference version or on reduced instances.

Taxonomy. Definition 3.3 organizes mutants by two properties: whether the mutation changes the program’s ideal semantics, and how it changes the program’s physical execution.

Definition 3.3 (Noise-aware mutation taxonomy). A mutant P′P' of a program PP belongs to one of three classes:

  • Semantic (MS\mathcal{M}_{S}): Dideal(P,P′)>0D_{\mathrm{ideal}}(P,P') > 0. The mutation changes the ideal output. Two operator families generate semantic mutants: additive (gates are inserted) and subtractive (gates are removed).

  • Noise-amplifying (MA\mathcal{M}_{A}): Dideal(P,P′)=0D_{\mathrm{ideal}}(P,P') = 0 and the circuit of P′P' compiled for the target backend contains more physical operations than that of PP. The ideal output is unchanged, but the mutant is exposed to more gate error than the original; whether its output distribution changes measurably depends on the program, the noise and the shot budget (§ 5.2).

  • Noise-neutral (MN\mathcal{M}_{N}): Dideal(P,P′)=0D_{\mathrm{ideal}}(P,P') = 0 and the circuit of P′P' compiled for the target backend contains no more physical operations than that of PP; gates may be reordered or re-expressed.

The semantic criterion is a property of the program; the physical criterion is a property of the program and the compilation target, because which operations are physical (rather than virtual, such as RzR_z on IBM devices) and how a circuit is routed depend on the target. The operators of Table 2 are designed for one class each: the noise-amplifying operators insert gates whose product is the identity, protected from cancellation by barriers, and the noise-neutral operators reorder commuting gates or re-express a gate in an equivalent form, for example H⋅CZ⋅HH \cdot CZ \cdot H, which compiles back to a CNOT on a device with a native CNOT. Whether a mutant meets the physical criterion of its operator’s class on a given target is verified by comparing the gate counts of the compiled mutant and original (§ 5.2).

Class (family)OperatorDescription
Semantic (additive)insert_cnotinsert a CNOT between two distinct random qubits
insert_tinsert a T gate on a random qubit
insert_xinsert an X gate on a random qubit
insert_rotationinsert Rx/y/z(θ)R_{x/y/z}(\theta), ∣θ∣∼U[π/8,π/2]|\theta| \sim U[\pi/8,\pi/2]
Semantic (subtractive)remove_gateremove a randomly selected gate
remove_1q_gateremove a randomly selected single-qubit gate
rotation_to_identityremove a randomly selected rotation gate
Noise-amplifyinginverse_pair_1qinsert GG†G G^\dagger, G∈{H,X,Y,Z,S,T}G \in\{H,X,Y,Z,S,T\}
cx_pairinsert two identical CNOTs
rotation_pairinsert R(θ)R(−θ)R(\theta)R(-\theta), R∈{Rx,Ry}R \in\{R_x,R_y\}
Noise-neutralcommute_swapswap two adjacent gates acting on disjoint qubits
cx_to_h_cz_hreplace a CNOT by H⋅CZ⋅HH \cdot CZ \cdot H on its target
x_to_h_z_hreplace an X gate by H⋅Z⋅HH \cdot Z \cdot H in place

Table 2. Mutation operators by taxonomy class. Insertions are placed at a uniformly random position before the first measurement; inserted pairs are separated by barriers so that the transpiler cannot cancel them.

Semantic mutants are the faults a test suite should detect. Noise-amplifying and noise-neutral mutants are negative controls: an oracle that rejects them reports a fault that does not exist. A noise-amplifying mutant is a particularly informative control, because a sensitive oracle can correctly detect that its output distribution differs from the original’s (it is noisier) although its semantics are unchanged; such rejections are the cases that attribution must resolve. Table 2 lists the operators and Figure 2 summarizes the taxonomy.

Figure 2. The three mutation classes. The semantic class contains the faults to be detected; the two semantics-preserving classes are negative controls. Noise-amplifying mutants preserve the ideal output but add physical gate error, so a sensitive oracle may observe a genuine distributional difference; attribution decides whether such a difference is a noise effect. § 5.2 tests whether the three classes are empirically distinguishable.

Generation. The engine samples an operator uniformly within the requested family and applies it to a fresh copy of the original (one mutation per mutant), driven by a seeded generator; an inapplicable operator is redrawn, so every mutant is labeled with the operator that produced it. For semantic families, the engine redraws (up to 60 times) until it obtains a non-equivalent mutant and records the equivalent draws it rejected; for the semantics-preserving families, it verifies that every mutant is equivalent. Every program’s engine is seeded with the same value.

Mutation Adequacy Metrics

Definition 3.4 (Noise-aware mutation score). Within an evaluation cell (one benchmark, one noise level and one oracle calibration), let the original program pass the oracle. The noise-aware mutation score is the fraction of semantic mutants rejected by the oracle. NAMS =∣{m∈Ms:oracle rejects m}∣/∣Ms∣= |\{m \in\mathcal{M}_s : \text{oracle rejects }m\}|/|\mathcal{M}_s|. Cells in which the original program fails the oracle are reported and excluded.

Because equivalent mutants are excluded from MS\mathcal{M}_S by construction, NAMS is not deflated by mutants that cannot be killed. The rejection rates of the two semantics-preserving classes are reported as false-rejection rates. Detection of injected faults is reported with the full confusion matrix, recall, precision, F1 and false-positive rate, computed only on faults whose ideal measured output on the tested input differs from the original’s; rejections of equivalent faults are reported as a separate rate.

Bug–Noise Attribution

When the oracle rejects P′P', the attribution procedure decides whether the deviation is caused by a change of the ideal semantics (Bug) or by hardware noise (Noise), using only executions on the same backend. It relies on zero-noise extrapolation (ZNE) [10, 55]: if the deviation is semantic, it persists as noise is removed; if it is a noise effect, it vanishes.

  1. Noise scaling. Both circuits are folded globally, U↦U(U†U)mU \mapsto U(U^{\dagger}U)^m, giving noise scale factors s=2m+1∈{1,3}s = 2m + 1 \in\{1, 3\} (and 5 for the quadratic variant); barriers between folds prevent the transpiler from undoing them.

  2. Measurement. At each scale, PP and P′P' are each executed once with NN shots.

(3) Extrapolation. Every outcome probability of PP and of P′P' is extrapolated to s=0s = 0 by Richardson extrapolation (linear through s=1,3s = 1, 3; quadratic through s=1,3,5s = 1, 3, 5), clipped at zero and renormalized, and the Jensen–Shannon distance r0r_0 between the two extrapolated distributions is computed.

(4) Decision. A null distribution of r0r_0 is obtained by redrawing, at every scale, both count vectors from their pooled distribution (no difference between PP and P′P'), repeating the extrapolation 2000 times. The case is labeled Bug if r0r_0 exceeds the null 0.99 quantile and Noise otherwise.

The null of step 4 represents sampling variation only: it assumes that the extrapolated outputs of PP and P′P' coincide. A noise effect satisfies this only to the extent that the extrapolation model holds. If the folded circuits saturate towards the maximally mixed distribution, if PP and P′P' decay at different rates, or if the realized noise scale differs from the nominal factor (for example because a folded circuit is routed differently), a systematic residual remains that the null does not contain, and a noise effect is labeled Bug. The procedure is therefore a conditional diagnostic; RQ3 measures its reliability.

The linear variant costs four executions per case and the quadratic variant six. We also evaluate exponential extrapolation [55] under a global depolarizing model p(s)=u+(p0−u)fsp(s) = u + (p_0 - u)f^s, with ff estimated per circuit from s=1,3s = 1, 3, which removes the bias of a linear fit when the circuits decay at different rates at the price of amplifying sampling noise. No variant requires a noiseless simulator. The ground truth for evaluating attribution is exact: a case is a Bug if and only if the ideal measured outputs of PP and P′P' differ on the tested input.

Study Design

Table 3 gives an overview of how each research question is evaluated. Every configuration value stated here is recorded with its results.

RQSubjectsEnvironmentsOracles / methodsMetrics
RQ19 benchmarks, correct programs only; 40 calibration windows × 5 runs per cell; 32 QASMBench programs (extended suite); sampled supports of 8 to 16,384 outcomes; bootstrap stability (27 cells, BB up to 40,000)5 synthetic noise levels; 2 coherent-error levels; 3 device snapshots × 2 compilation policies; IBM deviceNAO, NAO-1 (α∈{10−3,10−2}\alpha\in\{10^{-3}, 10^{-2}\}), SPC envelope, fixed thresholds, QOIN-style filter, noise-calibrated global threshold (extended suite)FPR (≈ α\alpha, ↓ for baselines) with cluster-bootstrap CI; dispersion ratio
RQ212 operators × 9 benchmarks × 100 draws (census); 15 mutants per family per cell, 27 cells; amplification dose study (1,620 mutants); extended suite (5,760 mutants)noiseless (census); low, medium, high noiseexact statevector labels; NAO, NAO-1, SPC envelopeequivalent-mutant rate; NAMS ↑; false-rejection rate ↓; class contrasts; predicted vs observed rejection
RQ31,062 labeled cases (mutants, faults, unmodified programs), 27 cells; 392 cases on snapshotslow, medium, high noise; 2 device snapshots; IBM deviceZNE (linear, quadratic, exponential); re-execution triageaccuracy ↑, Cohen’s κ\kappa ↑, bug precision and recall ↑, noise recall ↑
RQ450 non-equivalent faults and 1,350 correct runs, 45 cells; 163 faults in the extended suite; 10 real bugs (Bugs4Q); GHZ and QAOA up to 16 qubits; global threshold on held-out programs5 synthetic noise levels; 2 coherent-error levels; device snapshots; η=0.005\eta= 0.005 (scaling); IBM deviceNAO, NAO-1, SPC envelope, fixed thresholds, QOIN-style filter, noise-calibrated global threshold; marginal NAO; ablation (six configurations)recall ↑, precision ↑, F1 ↑, FPR ↓; predicted vs observed power
RQ5GHZ-3, QAOA-3, Bell-8, QAOA-8; 25 windows per drift levelcalibration at η0\eta_0, testing at η0(1+r)\eta_0(1 + r); IBM device sessionsNAO with recalibration policies (none, sentinel, co-scheduled); homogeneity testFPR ↓, drift-detection rate ↑, executions per campaign ↓

Table 3. Study design per research question. Cells are (benchmark, noise level) pairs. ↑: higher is better; ↓: lower is better; “≈ α\alpha”: the target is the nominal level.

Subject Programs and Inputs

The nine core benchmarks (Table 4) span 2 to 8 qubits: five small benchmarks covering the canonical primitives of prior quantum software testing work, and four larger ones in which noise has more opportunity to corrupt outputs. QAOA-8 is the most demanding (depth 27, 16 entangling gates); Bell-8 and GHZ-8 are shallower but have highly concentrated outputs. For RQ4, a scaling study adds GHZ circuits of 10 to 16 qubits and one-layer QAOA-MaxCut ring circuits of 10 and 12 qubits at medium noise, with eight windows of four correct runs (32 per circuit) and one run of each applicable non-equivalent fault of five injectors.

BenchmarkQubitsDepth2-qubit gatesDescriptionFamily
Bell231two-qubit entangled statestate preparation
GHZ-3342three-qubit GHZ statestate preparation
QFT-3374three-qubit quantum Fourier transformarithmetic kernel
QAOA-33126one-layer QAOA on a 3-qubit ringvariational
VQE-3352hardware-efficient ansatz, fixed parametersvariational
Bell-88978-qubit chain-entangled statestate preparation
GHZ-88978-qubit GHZ state, star topologystate preparation
QAOA-882716one-layer QAOA-MaxCut on an 8-qubit ringvariational
HWE-661310two-layer hardware-efficient ansatz, fixed parametersvariational

Table 4. Benchmark circuits. Depth is the logical depth before transpilation, including the measurement layer.

Extended suite. RQ1, RQ2 and RQ4 are replicated on 32 programs from QASMBench [46]: every small-scale program with at most 10 qubits that is a static circuit ending in measurement (excluding mid-circuit measurement, reset, classical control, unparsable programs and QAOA-3, which duplicates a core benchmark). The suite covers arithmetic, error correction, oracle algorithms, phase estimation, linear solvers (HHL), quantum walks, simulation, machine learning (DNN), variational circuits and state preparation, from 2 to 10 qubits and up to depth 551; several programs measure only some qubits. In RQ1 and RQ4, ten windows per program and noise level each judge five correct runs and one run of every applicable fault (8,000 correct runs; 163 non-equivalent faults); in RQ2, 15 mutants per family per program at three noise levels (5,760 mutants). The noise-calibrated global threshold for this suite is computed from 30 separate runs of each program.

Inputs. All noisy executions start from ∣0⋯0⟩|0 \cdots0\rangle. To check that equivalence labels do not depend on this choice, the census of RQ2 also evaluates every mutant and fault on three further random computational-basis states, prepared by prepended X gates; a mutant is equivalent over the four inputs only if its ideal output equals the original’s on all of them.

Mutants and Injected Faults

Equivalence census (RQ2). Each operator of Table 2 is applied 100 times to every benchmark on which it is applicable, without redrawing, and every raw draw is labeled by statevector simulation. Eleven operators are applicable to at least one core benchmark; x_to_h_z_h is not, because no core benchmark contains an X gate, and is exercised by the extended suite. A placement ablation applies the three insertion operators insert_cnot, insert_t and insert_x 50 times per benchmark both before and after the final measurement.

Mutation analysis (RQ2). For each benchmark, the engine generates 15 mutants per family (semantic additive, semantic subtractive, noise-amplifying, noise-neutral; seed 42), so each of the 27 cells (nine benchmarks at low, medium and high noise) contains 30 semantic mutants and 15 mutants of each semantics-preserving class, 1,215 mutants in total. Each mutant is executed once and judged by all oracles on the same counts.

Noise-amplification dose (RQ2). To test whether the rejections of noise-amplifying mutants are explained by the gate error they add, a dose-response study inserts kk barrier-protected inverse pairs at independent random positions before measurement: k∈{1,2,4}k \in\{1, 2, 4\} CNOT pairs or k∈{1,4,16}k \in\{1, 4, 16\} single-qubit pairs, ten mutants per (benchmark, pair type, kk), nine benchmarks at low, medium and high noise (1,620 mutant executions). Each mutant is judged by NAO and also executed with 200,000 shots to estimate its true noisy divergence from the original, from which Equation (4) predicts its rejection probability.

Injected faults (RQ4). Eight deterministic fault injectors model common programming errors: removing an entangling gate (remove_entanglement), applying the wrong measurement basis (wrong_basis), swapping the operands of a two-qubit gate (swap_qubits), inserting a spurious X gate (extra_x), dropping the last gate before measurement (drop_last_gate), inserting a Z gate (phase_flip), perturbing a rotation angle (wrong_angle) and retargeting a two-qubit gate (wrong_target). Of the 66 applicable injections on the nine benchmarks, 16 are equivalent on ∣0⋯0⟩\lvert0\cdots0\rangle and are excluded from recall, leaving 50 non-equivalent faults. Every fault is executed in 10 calibration windows at each of the five noise levels (2,500 fault tests), and each window also judges three fresh correct runs (1,350 matched negatives). The fixed thresholds are evaluated at τ∈{0.01,0.02,0.05,0.1}\tau\in\{0.01, 0.02, 0.05, 0.1\}. To validate the power model, each fault’s true noisy divergence is estimated from 200,000-shot runs of the program and the fault, and, independently of the benchmarks, synthetic distributions with d∈{4,16,64,256}d \in\{4, 16, 64, 256\} outcomes are perturbed by 14 fault sizes and tested 300 times each.

Real bugs (RQ4). We use real bugs from Bugs4Q [56], reproducible bugs in Qiskit programs collected from GitHub issues, Stack Overflow and Quantum Computing Stack Exchange. We include every bug whose buggy and fixed programs differ in the circuit itself (gates, operands, measurement mapping or initial state) and run as static circuits ending in measurement, and exclude API, version, visualization and simulator-configuration bugs, bugs whose fix replaces the whole program, and dynamic circuits. Ten bugs qualify (Table 24); we transcribe them to the current Qiskit version without changing their circuits. The fixed program is the reference PP and the buggy program P′P'. Each bug is evaluated at the five noise levels and on FakeManilaV2 and FakeLagosV2, with ten windows per environment, each judging three runs of the fixed and one run of the buggy program; rejections in the first three windows of each environment are attributed by linear ZNE.

Table 24. RQ4: real bugs from Bugs4Q [56]. JSD: ideal divergence between fixed and buggy program on ∣0⋯0⟩\lvert0 \cdots0 \rangle (0: not observable on this input). Detection: fraction of buggy runs rejected over 70 windows (five noise levels and two device snapshots). Higher detection is better for observable bugs.

Bug (Bugs4Q source)KindQubitsJSDNAOSPCFixed 0.05
Aer #664measurement mapping31.0001.001.001.00
Stack Overflow 63283443qubit order31.0000.970.861.00
Stack Overflow 62661255wrong gate20.3111.000.891.00
Stack Overflow, inverse QFTwrong subroutine31.0000.910.861.00
Stack Exchange 9measurement mapping31.0001.001.001.00
Stack Exchange 15wrong subroutine40.5810.860.711.00
Stack Exchange (p. 1–25) 9missing gates40.8290.970.860.86
Stack Exchange (p. 1–25) 19wrong initial state21.0001.001.001.00
Terra #5580qubit order301.000.440.60
Terra #5098control state300.640.000.71
Observable bugs: recall / precision / F10.964 / 0.994 / 0.9790.896 / 1.000 / 0.9450.982 / 0.330 / 0.494
FPR on runs of the fixed programs ↓0.00140.00000.5314

Table 24.

Attribution cases (RQ3). Each of the 27 cells contributes ten noise-amplifying and five noise-neutral mutants and the unmodified program (ground truth Noise), and eight additive and eight subtractive semantic mutants and every non-equivalent injected fault (ground truth BUG), 1,062 cases in total (582 BUG, 480 Noise). To compare extrapolation methods on identical data, the same cases are executed once more at s=1,3,5s = 1, 3, 5 with the counts stored, and every method is applied to these counts; this comparison is repeated on two device snapshots (FakeManilaV2 and FakeLagosV2, five small benchmarks, 392 cases), whose noise is not depolarizing.

Execution Environments

​Synthetic noise. Synthetic noise is parameterized by a single error level η\eta: single-qubit gates receive a depolarizing channel with parameter η\eta, two-qubit gates a depolarizing channel with parameter min⁡(10η,0.5)\min(10\eta, 0.5), and readout a symmetric bit-flip channel with flip probability min⁡(2η,0.5)\min(2\eta, 0.5). Five levels are evaluated: very low (η=0.0005\eta= 0.0005), low (0.001), medium (0.005), high (0.02) and very high (0.05, where the two-qubit channel saturates at the cap). The levels form a controlled stress scale rather than models of particular devices; realistic device noise is represented by the device snapshots and the IBM device below.

​Coherent noise. Depolarizing noise is incoherent: it moves every output distribution towards the uniform one. Real devices also have coherent errors, systematic over-rotations that can shift output probabilities in either direction. To test the oracle under them, a second synthetic model follows every single-qubit gate by RX(ϵ)R_X(\epsilon) and every two-qubit gate by RZZ(5ϵ)R_{ZZ}(5\epsilon) on the qubits of the gate, composed with depolarizing error at η=0.001\eta= 0.001 and symmetric readout error 2η2\eta, for ϵ∈{0.02,0.05}\epsilon\in\{0.02, 0.05\} rad. On the nine core benchmarks, 20 calibration windows per benchmark and ϵ\epsilon each judge five correct runs and one run of every applicable injected fault (1,800 correct runs and 2,000 non-equivalent fault tests), with the oracles of the fault-detection design on identical executions.

​Device snapshots. Three Qiskit fake backends are calibration snapshots of IBM Quantum devices: FakeManilaV2 (5 qubits), FakeLagosV2 (7 qubits) and FakeKolkataV2 (27 qubits). Each snapshot includes measured relaxation and dephasing times, gate error rates, readout assignment errors and the coupling map. The small benchmarks run on all three backends and the 8-qubit benchmarks additionally on FakeKolkataV2, with 15 calibration windows of five correct runs per (benchmark, backend) pair. For detection (RQ4), eight mutants per family and every applicable injected fault are judged under the fixed policy. Because the transpiler’s layout and routing choices change the noise a circuit experiences, two compilation policies are evaluated. All executions are transpiled by the Qiskit transpiler at optimization level 1. Under the fixed policy, every circuit is transpiled once with a fixed seed and every program under test is placed on the initial layout of the reference; this is the recommended practice and the default of the framework. Under the random policy, every execution is transpiled afresh, so that layout and routing may change between runs; this emulates a pipeline that recompiles on every submission.

IBM Quantum hardware. The hardware evaluation consists of a probe, a mini campaign (HM) and three campaigns (H1 to H3), executed through the Qiskit Runtime sampler with 4096 shots under the fixed compilation policy (Table 5). In every job, the circuits are interleaved in a seeded random order and placed on one initial layout per program, chosen by the success probability estimated from the device’s reported errors and kept for every later job on that program. Raw counts, transpiled circuits, job identifiers and a backend snapshot are archived. The probe executed Bell and GHZ-3 on ibm_fez (IBM Heron) on 16 June 2026: per circuit, eight calibration runs, one reference run, three test runs and three runs for each of three fixed thresholds.

Table 5. Sources of evidence. Executions: runs of a circuit with 4096 shots. Windows: calibrations from which a reference and a threshold are built. Simulated experiments are listed by research question (over 199,000 executions in total).

SourceBackendsProgramsRQsExecutionsWindows
Synthetic noise (simulation)Aer, 5 depolarizing levels9 core, 32 QASMBench, 10 real bugs, scaling to 16 qubitsRQ1–RQ5see caption
Coherent noise (simulation)Aer, 2 coherent-error levels9 coreRQ1, RQ4see caption
Device snapshots (simulation)FakeManilaV2, FakeLagosV2, FakeKolkataV2core benchmarks, real bugsRQ1–RQ4see caption
Hardware: probeibm_fez, 1 sessionBell, GHZ-3RQ1422
Hardware: HM, first job2ibm_kingston, 2 jobsBell, GHZ-3, QFT-3, QAOA-3, VQE-3RQ1–RQ41325
Hardware: HM, second jobibm_kingston, 1 jobsameRQ53453
Hardware: HM, third jobibm_kingston, 1 jobBell-8, GHZ-8, QAOA-8, HWE-6; 8 real bugsRQ1, RQ416412
Hardware: HM, fourth jobibm_kingston, 1 jobas first jobRQ5745
Hardware: H1 and H3ibm_kingston, 2 jobsGHZ-3, QAOA-3, Bell-8RQ1–RQ42063
Hardware: H2ibm_kingston, 8 jobsGHZ-3, QAOA-3RQ512816

Table 5. Sources of evidence. Executions: runs of a circuit with 4096 shots. Windows: calibrations from which a reference and a threshold are built. Simulated experiments are listed by research question (over 199,000 executions in total).

HM covers all five research questions and is sized to fit the free IBM Quantum allowance, so that the hardware evaluation can be reproduced without paid access. Its first job executes, per benchmark (Bell, GHZ-3, QFT-3, QAOA-3, VQE-3), eight calibration runs, one single-reference run, four correct runs, up to three non-equivalent faults, one additive and one subtractive semantic mutant, two noise-amplifying cx_pair mutants, one noise-neutral mutant, and the reference and the four non-neutral mutants at noise scale 3. The second job, 24.6 hours later, executes two sentinel runs, three correct runs and two faults per benchmark, judged against the first job’s calibrations. The third job extends HM to the four larger core benchmarks (eight calibration runs, four correct runs, three faults and two semantic mutants each) and to the eight observable real bugs (eight calibration runs and two runs of the fixed program, and two runs of the buggy program, each). The fourth job, 48.1 hours after the first, repeats the second with eight new calibration runs, so that the sentinel policy is evaluated with a full recalibration. The SPC envelope is calibrated on the nine runs of PP of the first job. HM ran on ibm_kingston (IBM Heron) between 21 and 23 October 2026 and used 446 seconds of QPU time.

H1 to H3 use GHZ-3, QAOA-3 and Bell-8 with more runs per benchmark. H1 (detection) executes, per benchmark, eight calibration runs, ten correct runs, twenty semantic and ten semantics-preserving mutants and every non-equivalent fault. H2 (drift) repeats eight calibration runs of GHZ-3 and QAOA-3 in eight sessions over two calendar days. H3 (attribution) executes five noise-amplifying mutants (physical pairs only, since ZZ, SS and TT are virtual on IBM devices) and five semantic mutants of GHZ-3 and QAOA-3, with the reference, at noise scales 1 and 3. They ran on ibm_kingston on 22 and 23 October 2026 and used 378 seconds. Simulation provides the rates and intervals on which the answers to the research questions rest; the hardware runs test whether the same behavior appears on a physical device.

Drift Scenarios (RQ5)

Drift is modeled as a change of the synthetic noise level, on GHZ-3, QAOA-3, Bell-8 and QAOA-8. Drift after calibration: the oracle is calibrated at η0=0.005\eta_0 = 0.005 and tested on correct runs at η0(1+r)\eta_0(1+r), r∈{0,0.1,0.25,0.5,1.0}r \in\{0, 0.1, 0.25, 0.5, 1.0\}, with 25 windows of five test runs. Recalibration policies are compared on paired executions for r∈{0,0.1,0.25,0.5}r \in\{0, 0.1, 0.25, 0.5\}: no recalibration; a sentinel with m∈{1,2,4}m \in\{1, 2, 4\} runs tested at αs∈{10−3,10−2}\alpha_s \in\{10^{-3}, 10^{-2}\}, followed by recalibration (K=8K = 8 runs at the new noise level) if it rejects; and co-scheduled calibration, in which the reference is always built in the same session as the tests. Each window executes one calibration at η0\eta_0, four sentinel runs, one recalibration set and five test runs, and every policy uses the subset it needs. Drift during calibration: the noise level of the eight calibration runs increases linearly from η0\eta_0 to η0(1+r)\eta_0(1+r), and the rate at which the homogeneity test flags drift is recorded over 25 windows. Execution cost is computed per campaign of MM programs under test.

Baselines

SPC envelope. The envelope oracle accepts a run if its divergence from a single reference run is at most τ=max⁡(μ+kσ,τmin⁡)\tau= \max(\mu+ k\sigma, \tau_{\min}), where μ\mu and σ\sigma are the mean and standard deviation of the (K2)\binom{K}{2} pairwise divergences among $K calibration runs of PP. This is the classical statistical-process-control envelope [40] applied to the self-baseline, with K=12K = 12, k=4k = 4 and τmin⁡=0.02\tau_{\min} = 0.02. It shares the self-baseline reference with NAO but sets its threshold heuristically, which isolates the contribution of the null-quantile threshold.

Fixed threshold. The fixed-threshold oracle rejects when the divergence between the test run and the ideal output distribution, estimated by a noiseless simulated execution of PP with NN shots, exceeds a constant τ∈{0.005,0.01,0.02,0.05,0.10}\tau\in\{0.005, 0.01, 0.02, 0.05, 0.10\} (subsets of these values in the experiments with many oracles). This is the decision rule that noise-oblivious practice, and tools such as QMutPy and Muskit, effectively implement.

QOIN-style filter. A reproducible stand-in for the noise-filtering strategy of QOIN [8]: a per-outcome multiplicative noise factor is estimated from Kc=5K_c = 5 executions of the correct circuit (additive smoothing 10−310^{-3}), its inverse is applied to each test run, and the corrected distribution is compared with the ideal output at τ=0.05\tau= 0.05. We do not reproduce QOIN’s learned filter (a neural network fine-tuned per circuit by transfer learning); the comparison is at the level of the strategy, not the implementation.

Re-execution triage (RQ3). The attribution baseline compares PP and P′P' under three re-executions: with four times the shots, under the noise model scaled by 0.5, and on a noiseless simulator. It labels the case Bug if the noiseless divergence exceeds 0.1, Noise if it does not and at least one noisy re-execution also stays within that tolerance, and Inconclusive otherwise (which did not occur). It needs a scalable noise model and a noiseless simulator, so it cannot run on hardware; it quantifies what the hardware-executable ZNE procedure gains or loses against simulator-dependent triage.

Noise-calibrated global threshold (RQ4). Following Fortz et al. [34], this baseline compares the Jensen–Shannon distance between a run and the exact ideal output with one threshold per noise model, T=Q87.5(μi)+Q87.5(SEi)T = Q_{87.5}(\mu_i) + Q_{87.5}(SE_i), where μi\mu_i and SEiSE_i are the mean and standard error of the distance over 30 noisy executions of correct program ii and Q87.5Q_{87.5} is the 87.5th percentile over programs. The thresholds are computed from executions separate from the evaluation, which follows the fault-detection design with NAO judging the same executions, at 4096 shots for both oracles (Fortz et al. use 10,000). Because the percentile is taken over the programs to which the threshold is applied, a separate experiment recomputes it without each program in turn and applies it to fresh runs of that program.

NAO-1 and marginal NAO. NAO-1 (§ 3.2) is the low-cost variant of NAO; it differs only in pooling and the drift test. For the scaling study, marginal NAO applies the threshold of Equation (3) to every two-qubit marginal at level α/(n2)\alpha/\binom{n}{2} (Bonferroni) and rejects if any marginal exceeds its threshold.

Ablation (RQ4). The comparisons above change more than one component at a time, so the ablation builds six oracles that add one component each: (A) ideal reference with fixed $\tau = 0.02; (B) one self-baseline run with the same threshold; (C) one self-baseline run with the SPC envelope; (D) one self-baseline run with the null-quantile threshold (NAO-1); (E) the pooled reference without the drift test; and (F) NAO. All six judge the same executions in two settings: the nine core benchmarks at five noise levels (10 windows per cell), and the five small benchmarks on the three snapshots under the random compilation policy (10 windows each), where the drift test is designed to matter. Each window judges five correct runs and one run of every applicable fault, on separate seeds.

Information and executions available to each oracle. Table 6 lists what each oracle uses. The self-baseline oracles need no simulation of the program but spend executions of the reference on the backend; the oracles that compare with the ideal output need no calibration executions but need a noiseless simulation, whose cost grows exponentially with the qubit count, and the global threshold also a device noise model and runs of a set of programs. All oracles judge identical test executions, but not at equal calibration budget (NAO 8, SPC envelope 13); NAO-1, with one calibration execution, and the ablation control for this. Execution costs per campaign are in Table 28.

OracleReferenceClassical computation or models neededCalibrationPer test
NAOpool of 8 runs of PPnone81
NAO-11 run of PPnone11
SPC envelope1 run of PPnone; threshold from 12 further runs of PP131
Fixed thresholdideal output of PPnoiseless simulation of PP01
QOIN-style filterideal output of PPnoiseless simulation of PP; correction from 5 runs of PP51
Global thresholdexact ideal output of PPstatevector simulation of PP; device noise model; 30 noisy executions of each calibration program01
ZNE attribution (linear)runs of PP and P′P' at scales 1, 3none4 per attributed case
Re-execution triageoutputs of PP and P′P'noiseless simulator; scalable noise model3 settings for PP and P′P'

Table 6. Information and executions used by each oracle for one (program, backend) pair. Calibration: executions of the reference program on the backend before testing; per test: executions of the program under test.

OracleM=10M = 10M=100M = 100M=1000M = 1000
Fixed threshold (unsound)101001000
NAO-1111011001
QOIN-style filter (Kc=5K_c = 5)161061006
NAO (K=8K = 8)181081008
NAO (K=8K = 8), sentinel (2, 10−210^{-2}) every 50 testsa^{a}201121048
SPC envelope (K=12K = 12)231131013

Table 28. RQ5: executions per test campaign of MM programs under test, for one (program, backend) pair. Sentinel (m,αs)=(2,10−2)(m,\alpha_s) = (2, 10^{-2}): before every 50 tests, m=2m = 2 runs of the reference program are pooled and tested against the reference at level αs=10−2\alpha_s = 10^{-2}. Lower is better.

Calibration Windows and Independence

A calibration window is one set of calibration executions of PP from which every oracle builds its reference and threshold. In RQ1 and RQ4, a window has 13 executions of PP: the first eight form NAO’s pooled reference, the thirteenth is the single reference of NAO-1 and the SPC envelope, and the first twelve calibrate the envelope; the QOIN-style filter uses the first five. Within a window, all oracles judge the same test executions (paired design). Test executions are independent multinomial samples given the window, but tests of one window share its calibration, so if the calibration underestimates the variability, all of them are affected together. False-positive rates are therefore reported with 95% cluster-bootstrap intervals that resample whole windows (2000 resamples) [57], and RQ1 draws 40 independent windows per cell. In RQ2, all mutants of a cell share one window, so the class contrasts are also tested conditionally on the window and at the level of cells.

Statistical Procedures

Table 7 lists each test with its hypotheses and interpretation. All tests are two-sided; when a family of tests is reported, the Bonferroni-corrected level is stated with the result.

ProcedureUsed inHypothesesInterpretation
Wilson interval [58]all proportions(interval estimate)95% interval for a rate; valid near 0 and 1
Cluster bootstrap [52, 57]RQ1, RQ4(interval estimate)95% interval that accounts for tests sharing a calibration window
Calibration checkRQ1H0H_0: FPR =α= \alphathe oracle is calibrated if α\alpha lies in the cluster-bootstrap interval
Fisher exact test [59]RQ2H0H_0: two classes have equal rejection probability; H1H_1: they differper-mutant test; Bonferroni over the 3 class pairs (p<0.0167p < 0.0167)
Cochran–Mantel–Haenszel test [43]RQ2H0H_0: equal rejection probability within every cell (common odds ratio 1); H1H_1: common odds ratio ≠1\ne1stratified by cell, so it conditions on the shared calibration window
Cell-level sign-flip permutation testRQ2H0H_0: the per-cell difference of class rejection rates is symmetric around 0treats the cell as the unit of analysis; the most conservative test (exact up to 16 cells, 200,000 random sign patterns otherwise)
Brier score [60]RQ2(accuracy of predicted probabilities)mean squared difference between predicted rejection probability and outcome; lower is better, compared with a constant predictor
Mann–Whitney U [61] with Cliff’s δ\delta [62]RQ2H0H_0: the normalized divergences D/τD/\tau of two classes have the same distribution∣δ∣<0.147|\delta| < 0.147 negligible, <0.33< 0.33 small, <0.474< 0.474 medium, otherwise large [63]
Cohen’s κ\kappa [64]RQ3(agreement beyond chance)0.61–0.80 substantial, >0.80> 0.80 almost perfect agreement [65]
Sobol indices [66, 67]parameters(variance decomposition)S1S_1: share of output variance due to one factor alone; STS_T: share including all its interactions

Table 7. Statistical procedures. pp: probability of a result at least as extreme as the observed one if H0H_0 holds; a small pp is evidence against H0H_0.

The Sobol indices are computed exactly by functional ANOVA on complete factorial grids, so they carry no Monte Carlo error.

Parameter Selection

The parameters of NAO fall into four groups with different roles, and only the first is chosen to control false positives. The statistical parameter, the level α\alpha, sets the nominal false-positive rate. The resource parameters, the calibration size KK and the shot count NN, set sensitivity and execution cost. The numerical parameter, the bootstrap size BB, sets how accurately the quantile is estimated; it costs classical computation only (B=4000B = 4000 in the main false-positive evaluation of RQ1 and in the hardware analyses, 3000 in most other experiments, and 2000 in the sparsity, scaling, attribution and sentinel experiments; every value is recorded with its results, and RQ1 measures its effect). The *drift-handling parameters, the homogeneity level αdrift=0.01\alpha_{\mathrm{drift}} = 0.01 with the widening factor k=4k = 4 and the sentinel’s mm and αs\alpha_s, trade unnecessary recalibration against protection from drift and are evaluated in RQ5. This section selects α\alpha, KK and NN. Their roles follow from § 3.2 and § 3.3: α\alpha sets the false-positive rate, while KK and NN enter only through 1/nref+1/ntest1/n_{\mathrm{ref}} + 1/n_{\mathrm{test}} and therefore govern sensitivity. To check this before fixing values, a factorial study was run on separate seeds from all main experiments: α∈{10−2,10−3,10−4}×K∈{1,4,8,16}×N∈{1024,4096,16384}\alpha\in\{10^{-2}, 10^{-3}, 10^{-4}\} \times K \in\{1, 4, 8, 16\} \times N \in\{1024, 4096, 16384\} (36 configurations) on five benchmarks (Bell, GHZ-3, QAOA-3, Bell-8, QAOA-8) at medium noise, eight windows each, with the non-equivalent injected faults as positives. For comparison, the same analysis was run for the SPC envelope on a 5×5×55 \times5 \times5 grid (K∈{4,8,12,16,20}K \in\{4, 8, 12, 16, 20\}, k∈{2,…,6}k \in\{2, \ldots, 6\}, τmin⁡∈{0.005,0.01,0.02,0.03,0.05}\tau_{\min} \in\{0.005, 0.01, 0.02, 0.03, 0.05\}; 125 configurations) on all nine benchmarks at five noise levels with five windows per cell and paired executions. Table 8 reports the indices.

OracleFactorFPR S1S_1FPR STS_TRecall S1S_1Recall STS_T
NAOα\alpha0.5580.8950.0430.175
NAOKK0.0180.2950.0620.231
NAONN (shots)0.0240.3270.6390.880
SPC envelopeKK0.0830.3080.0000.000
SPC envelopekk0.2910.7000.0060.008
SPC envelopeτmin⁡\tau_{\min}0.1540.5320.9920.994

Table 8. Variance-based sensitivity of the false-positive rate and recall to oracle parameters (configuration level, exact functional ANOVA on the complete grid). S1S_1: first-order index; STS_T: total index. For NAO, the FPR is governed by α\alpha and recall by NN, as the null and power models predict; for the SPC envelope, recall is governed almost entirely by the floor τmin⁡\tau_{\min}.

For NAO the analysis confirms the analytic roles. The FPR is dominated by α\alpha (ST=0.895S_T = 0.895); the remaining variance of the FPR is interaction with sampling noise of the rate estimate itself, since every configuration’s FPR lies between 0 and 0.019. Recall is dominated by NN (ST=0.880S_T = 0.880). No parameter dominates both responses, which is the property a tester needs: the false-positive rate can be chosen without affecting sensitivity, and sensitivity can be bought with shots without affecting the false-positive rate. For the SPC envelope, by contrast, recall depends almost only on the floor τmin⁡\tau_{\min} (S1=0.992S_1 = 0.992): the envelope’s sensitivity is set by a constant chosen in advance, and only 75 of its 125 configurations have zero false positives. False positives occur in 76% of the configurations with τmin⁡≤0.01\tau_{\min} \le0.01, in 48% of those with τmin⁡=0.02\tau_{\min} = 0.02 and in none with τmin⁡≥0.03\tau_{\min} \ge0.03, where soundness is bought with sensitivity: the mean recall of sound configurations falls from 0.801 at τmin⁡=0.02\tau_{\min} = 0.02 to 0.691 at τmin⁡=0.05\tau_{\min} = 0.05. The full NAO grid is listed in Table 29.

Table 29. Parameter study of NAO (§ 4.8): false-positive rate (160 correct runs per configuration) and recall on non-equivalent faults (232 fault tests per configuration), five benchmarks at medium noise, eight windows. K=1K = 1 is NAO-1.

α\alphaKKN=1024N = 1024N=4096N = 4096N=16384N = 16384
FPRRecallFPRRecallFPRRecall
10−210^{-2}10.01250.9700.00001.0000.01251.000
10−210^{-2}40.01870.9960.01251.0000.01251.000
10−210^{-2}80.00630.9910.00631.0000.00631.000
10−210^{-2}160.01250.9910.00001.0000.01871.000
10−310^{-3}10.00000.9400.00631.0000.00001.000
10−310^{-3}40.00000.9660.00001.0000.00001.000
10−310^{-3}80.00000.9780.00001.0000.00001.000
10−310^{-3}160.00630.9830.00001.0000.00001.000
10−410^{-4}10.00000.9480.00001.0000.00001.000
10−410^{-4}40.00000.9780.00001.0000.00001.000
10−410^{-4}80.00000.9740.00631.0000.00001.000
10−410^{-4}160.00000.9610.00001.0000.00001.000

caption:

The values used in all main experiments follow from this analysis.

  • α=10−3\alpha= 10^{-3} for the primary oracle. The level is a design choice whose consequences are measured. A campaign that judges TT correct programs raises αT\alpha T false alarms in expectation, each costing an investigation or an attribution run: ten per 1000 tests at 10−210^{-2}, one at 10−310^{-3}. The price of a stricter level is sensitivity: at N=4096N = 4096 and K=8K = 8, the minimum divergence detected with 80% power is 0.00077–0.00396 at 10−210^{-2}, 0.00107–0.00501 at 10−310^{-3} and 0.00136–0.00591 at 10−410^{-4} for 4 to 256 outcomes, 27–39% larger from 10−210^{-2} to 10−310^{-3} and 18–27% from 10−310^{-3} to 10−410^{-4}; in the grid, recall at 4096 shots is 1.000 at all three levels. At 10−210^{-2}, the single-reference variant is liberal (0.0140 in RQ1). At 10−410^{-4}, the threshold is the largest of B=4000B = 4000 null draws, so the tail quantile is poorly resolved, and the rate is hard to assess: 9,000 correct runs would show 0.9 expected false positives, and observing none would only bound the rate below about 3.3×10−43.3 \times10^{-4} for independent runs. 10−310^{-3} is therefore the strictest evaluated level whose threshold is adequately estimated and whose rate can be assessed within this study’s budget. α=10−2\alpha= 10^{-2} is reported alongside in RQ1.

  • K=8K = 8: at fixed power, the minimum detectable divergence is proportional to 1/(KN)+1/N1/(KN) + 1/N, so K=8K = 8 obtains 87.5% of the reduction that an unlimited calibration would obtain over K=1K = 1, whereas K=16K = 16 would add about 6 percentage points for twice the calibration cost. In the grid, K=4K = 4 and K=8K = 8 have zero false positives at all three shot counts at α=10−3\alpha= 10^{-3}, and K=8K = 8 has the higher recall at 1024 shots (0.978 versus 0.966).

  • N=4096N = 4096: the FPR does not depend on NN (Table 8), so NN is chosen for sensitivity. In the grid, recall at α=10−3\alpha= 10^{-3} and K=8K = 8 rises from 0.978 at 1024 shots to 1.000 at 4096 and stays there at 16,384, which costs four times as much. The FPR does depend on NN through the shots per observed outcome: 4096 shots provide 16 per outcome for a full 8-qubit register, the width of the core benchmarks, which is where the sparsity study shows the nominal level to hold.

The SPC envelope is evaluated at K=12K = 12, k=4k = 4 and τmin⁡=0.02\tau_{\min} = 0.02, a sound configuration of its grid whose recall (0.804) is within 0.002 of the best sound configuration with k=4k = 4. The best sound configuration of the whole grid (K=20K = 20, k=5k = 5, τmin⁡=0.005\tau_{\min} = 0.005) reaches recall 0.858 on the grid’s executions; it is selected in hindsight on the evaluation data and is reported only for comparison.

Results

Each subsection reports the core benchmarks first, then the additional analyses and the QASMBench programs, and the IBM Quantum hardware last. Unless stated otherwise, NAO pools K=8K = 8 calibration runs at α=10−3\alpha= 10^{-3}, and all executions use 4096 shots. Secondary tables and a figure are in Appendix § 13.

RQ1: Soundness of the Noise-Aware Oracle

Synthetic noise. Table 9 and Figure 3 report the false-positive rate of every oracle over 9,000 executions of correct programs (nine benchmarks, five noise levels, 40 windows of five runs per cell). NAO’s rate is 0.0011 at α=10−3\alpha= 10^{-3} and

OracleFP / testsFPR95% CI
NAO (α=10−3\alpha= 10^{-3})10/90000.0011[0.0004, 0.0018]
NAO (α=10−2\alpha= 10^{-2})87/90000.0097[0.0078, 0.0117]
NAO-1 (α=10−3\alpha= 10^{-3})11/90000.0012[0.0006, 0.0021]
NAO-1 (α=10−2\alpha= 10^{-2})126/90000.0140[0.0112, 0.0170]
SPC envelope3/90000.0003[0.0000, 0.0008]
QOIN-style filter800/90000.0889[0.0761, 0.1028]
Fixed τ=0.1\tau= 0.13200/90000.3556[0.3333, 0.3778]
Fixed τ=0.05\tau= 0.053611/90000.4012[0.3784, 0.4238]
Fixed τ=0.02\tau= 0.025391/90000.5990[0.5760, 0.6213]
Fixed τ=0.01\tau= 0.015886/90000.6540[0.6317, 0.6750]
Fixed τ=0.005\tau= 0.0056630/90000.7367[0.7158, 0.7558]

Table 9. RQ1: false-positive rate on correct programs (9 benchmarks, 5 noise levels, 40 calibration windows of 5 runs per cell). CI: 95% cluster bootstrap over windows. For NAO and NAO-1, the target is the nominal level α\alpha; for the baselines, lower is better.

Figure 3. RQ1: false-positive rate of each oracle on 9,000 correct runs (point: observed rate; bar: 95% cluster-bootstrap interval over calibration windows; log scale). The dashed and dotted lines mark the nominal levels α=10−3\alpha= 10^{-3} and 10−210^{-2}. An arrow marks an interval whose lower bound is 0, which a log scale cannot show (SPC envelope: 3 false positives, interval [0, 0.0008]).

0.0097 at 10−210^{-2}, and both cluster-bootstrap intervals contain the nominal level. NAO-1 is calibrated at 10−310^{-3} and slightly liberal at 10−210^{-2} (0.0140), because one reference run underestimates the support of the output distribution and therefore the threshold.

Neither programs nor windows hide liberal behavior. Per program, NAO’s rate lies between 0 and 0.003 at α=10−3\alpha= 10^{-3} and between 0.005 and 0.013 at 10−210^{-2}, and no exact binomial test against α\alpha is significant (smallest p=0.080p = 0.080; Table 30). Per window, false positives do not cluster: at α=10−3\alpha= 10^{-3}, 10 of 1,800 windows contain one and none contains two (8.96 and 0.02 expected for independent runs), and at 10−210^{-2}, 85 contain one and one contains two (86.5 and 1.8 expected). NAO-1 at 10−210^{-2} is the exception: it is liberal on the three eight-qubit benchmarks (0.018–0.027, p≤0.017p \le0.017), and 19 windows contain two or more false positives, because one run of a wider distribution misestimates the threshold for the whole window. Pooling removes this error.

Table 30. False positives per core benchmark over 1,000 correct runs (five noise levels, 200 calibration windows; RQ1), with 95% Wilson intervals. For NAO and NAO-1 the target is α\alpha.

BenchmarkNAO, α=10−3\alpha= 10^{-3}NAO-1, α=10−3\alpha= 10^{-3}NAO, α=10−2\alpha= 10^{-2}NAO-1, α=10−2\alpha= 10^{-2}
Bell0 [0.0000, 0.0038]2 [0.0005, 0.0073]13 [0.0076, 0.0221]12 [0.0069, 0.0209]
GHZ-32 [0.0005, 0.0073]1 [0.0002, 0.0056]12 [0.0069, 0.0209]8 [0.0041, 0.0157]
QFT-30 [0.0000, 0.0038]3 [0.0010, 0.0088]10 [0.0054, 0.0183]11 [0.0062, 0.0196]
QAOA-31 [0.0002, 0.0056]0 [0.0000, 0.0038]9 [0.0047, 0.0170]14 [0.0084, 0.0234]
VQE-31 [0.0002, 0.0056]1 [0.0002, 0.0056]12 [0.0069, 0.0209]9 [0.0047, 0.0170]
Bell-80 [0.0000, 0.0038]1 [0.0002, 0.0056]5 [0.0021, 0.0117]20 [0.0130, 0.0307]
GHZ-82 [0.0005, 0.0073]3 [0.0010, 0.0088]11 [0.0062, 0.0196]27 [0.0186, 0.0390]
QAOA-83 [0.0010, 0.0088]0 [0.0000, 0.0038]10 [0.0054, 0.0183]18 [0.0114, 0.0283]
HWE-61 [0.0002, 0.0056]0 [0.0000, 0.0038]5 [0.0021, 0.0117]7 [0.0034, 0.0144]

caption:

Every fixed threshold is unsound, from 35.6% false positives at τ=0.1\tau= 0.1 to 73.7% at τ=0.005\tau= 0.005, because the divergence of a correct run from the ideal output grows with the noise and the circuit. The thresholds 0.05 and 0.1 are sound at very low and low noise but reject 22–34% of the correct runs at medium noise and 67–89% at the two highest levels (Table 31): a fixed threshold is sound only above a noise-dependent floor that is not known in advance. The QOIN-style filter reduces the rate to 8.9%; it is sound up to medium noise and rejects 22% at the two highest levels, where a per-outcome multiplicative correction no longer undoes the noise. The SPC envelope has the lowest rate (0.0003), the expected behavior of a conservative threshold whose cost in sensitivity RQ2 and RQ4 quantify.

Table 31. False-positive rate on correct programs by noise level (1,800 runs per level; RQ1). For NAO and NAO-1 the target is α\alpha; for the baselines, lower is better.

OracleVery lowLowMediumHighVery high
NAO (α=10−3\alpha= 10^{-3})0.00110.00060.00220.00060.0011
NAO (α=10−2\alpha= 10^{-2})0.01000.00830.01000.01220.0078
NAO-1 (α=10−3\alpha= 10^{-3})0.00060.00170.00280.00110.0000
NAO-1 (α=10−2\alpha= 10^{-2})0.00780.01390.02110.01670.0106
SPC envelope0.00000.00000.00060.00060.0006
QOIN-style filter0.00000.00000.00000.22220.2222
Fixed τ=0.005\tau= 0.0050.48280.58170.84110.88890.8889
Fixed τ=0.01\tau= 0.010.33440.38000.77780.88890.8889
Fixed τ=0.02\tau= 0.020.11280.33330.77110.88890.8889
Fixed τ=0.05\tau= 0.050.00000.00000.33720.78000.8889
Fixed τ=0.1\tau= 0.10.00000.00000.22220.66670.8889

caption:

Thresholds. The bootstrap threshold of NAO-1 agrees with the closed form of Equation (3) to within 12% in all 45 cells, and pooling lowers it by 28–46% (Table 32). The SPC envelope sits at its floor τmin⁡=0.02\tau_{\min} = 0.02 in 36 of the 45 cells, 14 to 25 times above the NAO threshold on the five small benchmarks, which explains both its low false-positive rate and its lower sensitivity. The dispersion ratio, the observed mean pairwise divergence among calibration runs divided by its null expectation, is 0.986 (median over cells; range 0.825–1.084): on a stationary simulator, run-to-run variability is the finite-shot variability of the null model.

CellNAO-1χ2\chi^2 formNAOSPC envelope
Bell / medium0.00140.00140.00080.0200
GHZ-3 / medium0.00220.00210.00120.0200
HWE-6 / medium0.00920.00910.00520.0200
Bell-8 / medium0.01910.01760.01200.0200
QAOA-8 / medium0.03020.02910.01720.0308
GHZ-8 / high0.03060.02840.01820.0323

Table 32. RQ1: median acceptance thresholds at α=10−3\alpha= 10^{-3} for selected cells (divergence scale). χ2\chi^2: closed form of Equation (3) for NAO-1.

Extended suite. On the 32 QASMBench programs (Table 10), NAO’s rate is 0.0013 over 8,000 correct runs (interval [0.0005, 0.0021]), and no program exceeds 5%. The fixed thresholds reject 50–82% of the correct runs and are unsound on 28 to 30 programs; the QOIN-style filter has 5.4% false positives and the noise-calibrated global threshold 12.7%, each concentrated on a few programs. Soundness therefore holds on programs not used to design the oracle, including programs with partial measurement and up to 1024 possible outcomes.

OracleFPRUnsound programs ↓Recall ↑Precision ↑F1 ↑
NAO0.00130/320.7760.9980.873
NAO-10.00090/320.7600.9990.863
SPC envelope0.00000/320.6601.0000.795
Fixed τ=0.01\tau= 0.010.818130/320.9950.5530.711
Fixed τ=0.05\tau= 0.050.623030/320.9760.6150.754
Fixed τ=0.1\tau= 0.10.503128/320.9050.6470.754
QOIN-style filter0.05443/320.5490.9110.685
Noise-calibrated global threshold0.12656/320.5290.8100.640

Table 10. Extended suite of 32 QASMBench programs (RQ1 and RQ4): 8,000 correct runs and 163 non-equivalent injected faults, five noise levels, ten windows per program and level; all oracles judge the same executions. Unsound programs: programs with FPR > 0.05. ↑: higher is better; ↓: lower is better; for NAO and NAO-1 the FPR target is α=10−3\alpha= 10^{-3}.

Device snapshots. Under the fixed compilation policy, NAO is sound on all three snapshots (Table 11; FPR ≤ 0.003). Under the random policy, layout and routing change between runs, so correct runs are no longer samples of one distribution: NAO-1 then reaches 10–11% false positives, whereas NAO stays at ≤ 0.005 because its homogeneity test flags the over-dispersed calibrations (20–25% of the windows, against 1% under the fixed policy; 73–93% on QAOA-3, whose layouts produce visibly different outputs) and widens the threshold. The fixed thresholds reach 14–80% false positives.

Table 11. RQ1: false-positive rate on device-snapshot backends (15 windows of 5 correct runs per benchmark; Bell, GHZ-3, QFT-3, QAOA-3 and VQE-3 on all backends, and Bell-8, GHZ-8 and QAOA-8 on FakeKolkataV2). Drift: fraction of windows in which the homogeneity test of NAO rejects. Lower FPR is better.

BackendCompilationNAONAO-1SPCτ=0.005\tau=0.005τ=0.05\tau=0.05τ=0.1\tau=0.1Drift
FakeManilaV2fixed0.0000.0000.0000.8000.2000.0000.01
random0.0050.1070.0000.8000.2000.0000.24
FakeLagosV2fixed0.0030.0000.0000.8000.8000.6000.01
random0.0050.1070.0000.8000.8000.6000.20
FakeKolkataV2fixed0.0020.0000.0000.6280.1370.0000.01
random0.0020.1000.0000.6500.1420.0120.25

Table 11.

Coherent noise. Under coherent over-rotation errors, the reference experiences the same errors as the program under test. NAO rejects 1 of 1,800 correct runs (interval [0, 0.0017]), and the homogeneity test flags at most 5% of the windows of any benchmark. The fixed thresholds reject 46.6% (τ=0.01\tau= 0.01), 38.8% (0.02) and 5.5% (0.05) of the correct runs, and τ=0.1\tau= 0.1 is sound but detects only 82% of the faults. On 2,000 non-equivalent fault tests, NAO has recall 0.981, precision 0.999 and F1 0.990, against 0.979 for the SPC envelope and at most 0.922 for the fixed thresholds. NAO also rejects 59 of 640 runs of equivalent faults, because an inserted gate brings its own over-rotation: the noise-amplifying effect of RQ2 in coherent form.

Monte Carlo stability of the bootstrap threshold. At α=10−3\alpha= 10^{-3} and B=4000B = 4000, only about four bootstrap draws lie beyond the threshold. For one window of each core benchmark at low, medium and high noise (27 cells), the threshold was re-estimated 30 times for each BB from a fixed plug-in estimate and compared with a reference null of 400,000 draws (Table 12). At B=4000B = 4000, the implied false-positive rate averages 0.00101, but its 5th–95th percentile range is 0.17–2.37 times the nominal level: the Monte Carlo error averages out over windows, which is why RQ1 observes 0.0011, but a single window’s rate can be well below or more than twice the nominal level. At B=40,000B = 40,000, the range narrows to 0.66–1.44 times α\alpha at about three seconds per window for 256 outcomes, negligible against the QPU time of a calibration; testers who need the level per window should use the larger BB. Experiments with B=2000B = 2000 or 3000 lie between the rows for 1000 and 4000.

Table 12. RQ1: Monte Carlo stability of the bootstrap threshold at α=10−3\alpha= 10^{-3} (27 cells: nine core benchmarks at low, medium and high noise; 30 re-estimations per cell and BB; reference null of 400,000 draws). Implied FPR: the false-positive rate of an estimated threshold under the reference null. Target: α=0.001\alpha= 0.001.

BBRelative s.d. of threshold, median (max)Implied FPR, meanImplied FPR, 5th–95th percentile range
10000.107 (0.204)0.001010.00001–0.00447
40000.047 (0.079)0.001010.00017–0.00237
40,0000.014 (0.025)0.001020.00066–0.00144

Table 12.

Calibration under sparse supports. We distinguish the possible outcomes of an mm-qubit register (2m2^m), the occupied outcomes (πx>0\pi_x > 0) and the outcomes observed in a run. Table 13 tests the threshold where shots per outcome are few, by sampling only: test and reference runs are multinomial samples of one distribution with d=8d = 8 to 16,384 occupied outcomes and three shapes (flat, skewed and head-tail), so every rejection is a false positive. Down to 16 shots per occupied outcome, the rate stays close to the nominal level for all shapes: at α=10−2\alpha= 10^{-2}, 0.005–0.017 for NAO and 0.002–0.023 for NAO-1. At 4 shots per occupied outcome it becomes liberal, depending on the shape: 0.015 (flat), 0.098 (skewed) and 0.264 (head-tail) at α=10−2\alpha= 10^{-2}, and 0.062 for head-tail at 10−310^{-3}. With many outcomes, the null distribution is narrow relative to its mean, so a small bias of the plug-in estimate moves the rate by several times α\alpha. With fewer shots than outcomes, the unseen-outcome correction overshoots and the oracle becomes conservative for the skewed and head-tail shapes but stays liberal for the flat shape (0.062 at 4096 outcomes).

Table 13. RQ1: observed false-positive rate under sparse supports (N=4096N = 4096 shots; 1000 correct test runs per cell; NAO pools K=8K = 8 calibration runs, NAO-1 uses one). Flat: π∼Dirichlet⁡(1,…,1)\pi\sim\operatorname{Dirichlet}(1,\ldots,1); skewed: Zipf with exponent 1.1; head-tail: four outcomes with 80% of the mass (weights 1,12,13,141,\frac{1}{2},\frac{1}{3},\frac{1}{4}) and a uniform tail. N1: NAO-1. Target: the nominal level α\alpha.

ddN/dN/dNAON1NAONAON1NAONAON1NAO
85120.0050.0140.0420.0060.0130.0480.0090.0110.051
64640.0080.0090.0460.0170.0170.0610.0100.0020.062
256160.0100.0180.0460.0080.0140.0490.0080.0230.042
102440.0150.0540.0750.0980.0580.2810.2640.0060.514
409610.0620.1780.2140.0000.0000.0000.0000.0000.000
163840.250.0430.0670.1500.0000.0000.0000.0000.0000.000

Table 13.

A tester does not know the occupied support, so the boundary must be stated in a quantity computable from the reference. The Good–Turing unseen mass of the reference does not separate calibrated from liberal windows; the shots per observed outcome of one run of PP (shots divided by the distinct outcomes observed) does. In every window with 16 or more, across all three shapes, NAO’s rate is 0.0003–0.0005 at α=10−3\alpha= 10^{-3}, 0.009 at 10−210^{-2} and 0.046–0.052 at 0.05 (1,800 windows); in windows with fewer than 8, it is 0.024–0.063 at 10−210^{-2}, liberal or, at the largest supports, conservative (1,800 windows); no window fell between 8 and 16. In these experiments the nominal level therefore held with at least about 16 shots per observed outcome and was not reliably controlled below 8. This is an empirical operating criterion for the evaluated shapes, shot budget (N=4096N = 4096, K=8K = 8) and bootstrap size (B=2000B = 2000), not a certificate of calibration; a tester can check it after the first calibration run. For a full register it means 8 qubits at 4096 shots, but concentrated outputs meet it at larger widths: the three QASMBench programs with 9 or 10 qubits produced one false positive in 750 correct runs, and GHZ circuits remain sound up to 16 qubits (RQ4), whereas QAOA-12, which occupies almost all 4096 outcomes, does not. Below the boundary, more shots or the marginal comparison of RQ4 are needed.

IBM Quantum hardware. In the probe on ibm_fez, the divergences of the test runs from the reference (≤0.0015\leq0.0015) stayed below the closed-form NAO-1 thresholds at both levels, whereas a fixed threshold of 0.005 against the ideal output rejected all six runs, because a correct run lay 0.019–0.035 from the ideal output. In H1 on ibm_kingston, NAO and NAO-1 rejected none of the 30 correct runs of GHZ-3, QAOA-3 and Bell-8 (Wilson 95% upper bound 0.11). In HM (Table 14), NAO rejected none of the 20 correct runs of the small programs, none of the 16 of the larger programs and none of the 16 runs of the fixed real-bug programs, whereas τ=0.005\tau= 0.005 and 0.02 rejected 10 and 4, 16 and 12, and 13 and 4; on Bell-8 and GHZ-8, a correct run lay 0.061–0.085 from the ideal output. The threshold τ=0.05\tau= 0.05 was sound on the small programs but not on the larger ones. QAOA-8 occupied all 256 outcomes, the boundary of 16 shots per observed outcome, and NAO remained sound on it. The homogeneity test flagged no calibration on hardware. These counts cannot estimate a rate of order 10−310^{-3}; they test whether the contrast between the self-baseline and the ideal reference observed in simulation holds on the device.

OracleSmall programsLarger programsReal bugs
Correct ↓\downarrowFaults ↑\uparrowNANN ↓\downarrowCorrect ↓\downarrowFaults ↑\uparrowFixed ↓\downarrowBuggy ↑\uparrow
NAO0/2023/234/100/50/1619/200/1616/16
NAO-11/2023/234/101/50/1619/200/1616/16
SPC envelope0/2023/230/100/5––––
Fixed τ=0.005\tau= 0.00510/2023/235/103/516/1620/2013/1616/16
Fixed τ=0.02\tau= 0.024/2023/232/101/512/1619/204/1616/16
Fixed τ=0.05\tau= 0.050/2021/231/100/58/1618/200/1616/16

Table 14. Mini campaign HM on ibm_kingston: rejected / tested runs per oracle. Small programs: Bell, GHZ-3, QFT-3, QAOA-3 and VQE-3 (first job). Larger programs: Bell-8, GHZ-8, QAOA-8 and HWE-6. Real bugs: the eight observable Bugs4Q bugs (runs of the fixed and of the buggy programs). Faults: non-equivalent injected faults and semantic mutants. NA, NN: noise-amplifying and noise-neutral mutants; NA rejections have no preferred direction (Section 5.2). ↑\uparrow: higher is better; ↓\downarrow: lower is better; –: not evaluated.

Answer to RQ1

Under the evaluated conditions, the noise-aware oracle attains its nominal false-positive rate on average over windows: 0.0011 at α=10−3\alpha= 10^{-3} and 0.0097 at 10−210^{-2} over 9,000 correct runs, and 0.0013 over 8,000 correct runs of 32 QASMBench programs, none above 5%. It remains sound under coherent errors (1 of 1,800) and on device snapshots, including under recompilation, where its homogeneity test prevents the 10% rate of the single-reference variant. Fixed thresholds produce 36–74% false positives, a QOIN-style filter 8.9% and, on the QASMBench programs, a globally calibrated threshold 12.7%. The level held whenever one reference run had about 16 or more shots per observed outcome, an empirical criterion a tester can check; below 8 it was not reliably controlled (up to 0.264 at α=10−2\alpha= 10^{-2}), and the range between was not sampled. On an IBM Heron device, NAO rejected none of 82 correct runs of 17 programs, while fixed thresholds rejected up to all correct runs of the eight-qubit programs.

RQ2: Equivalent Mutants, Taxonomy and Mutation Score

Equivalent mutants. Table 15 reports the fraction of raw operator draws that are equivalent to the original. On ∣0⋯0⟩\lvert0 \cdots0 \rangle, 69.7% of T-gate, 48.1% of CNOT, 28.2% of rotation and 24.3% of X insertions are equivalent: a T gate before measurement changes only a relative phase unless later gates convert it into a population change, and a CNOT whose control is ∣0⟩\lvert0 \rangle is the identity. Over four inputs, the CNOT rate falls to 20.9%; the other rates barely change. Subtractive operators rarely produce equivalent mutants (0–9.7%). Placement is decisive: the same insertions placed after the final measurement are equivalent in 1350/1350 draws, against 636/1350 before it. Among the injected faults, 16 of 66 applicable injections are equivalent on ∣0⋯0⟩\lvert0 \cdots0 \rangle, 12 of them because they change the unitary but not the measured output.

Table 15. RQ2: fraction of raw operator draws that are equivalent mutants (exact statevector labels; 100 draws per operator and applicable benchmark). ∣0⋯0⟩|0 \cdots0\rangle: fraction of draws whose ideal output equals the original’s on the all-zero input state, the input of all noisy executions; 4 inputs: fraction equal on all four tested inputs (the all-zero state and three random computational-basis states). The noise-amplifying and noise-neutral operators are equivalent by design; for semantic operators, a lower rate means fewer draws are wasted.

Class (family)OperatorDraws∣0⋯0⟩|0 \cdots0\rangle4 inputs
Semantic (additive)insert_t9000.6970.697
insert_cnot9000.4810.209
insert_rotation9000.2820.267
insert_x9000.2430.243
Semantic (subtractive)remove_gate9000.0970.066
remove_1q_gate9000.0000.000
rotation_to_identity4000.0000.000
Noise-amplifyinginverse_pair_1q, cx_pair, rotation_pair27001.0001.000
Noise-neutralcommute_swap, cx_to_h_cz_h13001.0001.000

Mutation score. Table 16 reports the rejection rate of each class (with Wilson intervals in Figure 7). NAO rejects 93.8% of the semantic mutants (NAMS, Definition 3.4), NAO-1 91.9% and the SPC envelope 78.9%. Every semantic mutant is non-equivalent, so the remaining 6.2% are genuine misses; they concentrate at high noise (NAMS 0.978, 0.974 and 0.863 at low, medium and high noise) and on small changes such as insert_t (0.738) and rotation_to_identity (0.826; Table 33). The original program passed in every cell.

Table 16. RQ2: rejection rate by taxonomy class (27 cells: 810 semantic mutants and 405 of each semantics-preserving class). For the semantic class this is the noise-aware mutation score (NAMS, higher is better); for the two semantics-preserving classes it is a false-rejection rate (lower is better).

OracleSemantic (NAMS ↑\uparrow)Noise-amplifying ↓\downarrowNoise-neutral ↓\downarrow
NAO760/810 = 0.93884/405 = 0.2072/405 = 0.005
NAO-1744/810 = 0.91975/405 = 0.1855/405 = 0.012
SPC envelope639/810 = 0.78926/405 = 0.0640/405 = 0.000
ClassOperatornnRejected
Semanticinsert_cnot1080.935
insert_rotation1590.925
insert_t420.738
insert_x960.979
remove_gate1651.000
remove_1q_gate1710.965
rotation_to_identity690.826
Noise-amplifyingcx_pair1560.538
inverse_pair_1q1350.000
rotation_pair1140.000
Noise-neutralcommute_swap1170.000
cx_to_h_cz_h2880.007

Table 33. Rejection rate of mutants by operator under NAO (RQ2). For semantic operators higher is better; for semantics-preserving operators lower is better.

Figure 7. RQ2: fraction of mutants rejected per taxonomy class and oracle (810 semantic mutants and 405 of each semantics-preserving class; 27 cells). Error bars: 95% Wilson intervals; the value above each bar is the observed rate. For the semantic class the rate is the mutation score (higher is better); for the semantics-preserving classes it is a false-rejection rate.

Noise-neutral mutants are almost never rejected (2/405). Noise-amplifying mutants are rejected in 20.7% of cases, increasingly with noise (0.148, 0.222, 0.252) and almost only through cx_pair (53.8%, against 0% for single-qubit and rotation pairs). These rejections are not statistical errors: a CNOT pair adds two-qubit gate error, so the mutant’s noisy distribution genuinely differs, and a sensitive differential oracle detects it. They are, however, false alarms with respect to the program’s semantics, the case that attribution (RQ3) must resolve. The SPC envelope rejects fewer (6.4%) only because it is less sensitive to any difference.

Taxonomy. The taxonomy is supported if the three classes have different rejection probabilities under the oracle. Table 17 tests every pair at three levels of conservativeness (§ 4.7): per-mutant Fisher tests, the CMH test stratified by cell, which conditions on the shared calibration window, and the cell-level sign-flip test, which treats each cell as one observation.

Table 17. RQ2: tests of the taxonomy under NAO. Rates: rejection rates of the two classes compared. CMH: Cochran–Mantel–Haenszel test stratified by cell. Cells: cells in which the first class has a higher / equal / lower rejection rate; sign-flip: cell-level permutation test. δ\delta: Cliff’s delta on D/τD/\tau with 95% cluster-bootstrap interval over cells. Bonferroni level for three pairs: 0.0167.

NoiseContrastRatesFisher ppCMH ppCells >/=/<> / = / <Sign-flip ppδ\delta
lowsemantic vs amplifying0.978 / 0.1482.5×10−712.5 \times10^{-71}1.1×10−651.1 \times10^{-65}9/0/00.0040.98 [0.96, 1.00]
semantic vs neutral0.978 / 0.0002.6×10−1012.6 \times10^{-101}5.2×10−825.2 \times10^{-82}9/0/00.0040.99 [0.98, 1.00]
amplifying vs neutral0.148 / 0.0008.9×10−78.9 \times10^{-7}4.3×10−64.3 \times10^{-6}4/5/00.1250.23 [0.15, 0.34]
mediumsemantic vs amplifying0.974 / 0.2227.9×10−607.9 \times10^{-60}1.2×10−561.2 \times10^{-56}9/0/00.0040.94 [0.90, 0.99]
semantic vs neutral0.974 / 0.0005.2×10−1005.2 \times10^{-100}3.3×10−813.3 \times10^{-81}9/0/00.0040.99 [0.97, 1.00]
amplifying vs neutral0.222 / 0.0003.0×10−103.0 \times10^{-10}8.6×10−98.6 \times10^{-9}6/3/00.0310.27 [0.15, 0.38]
highsemantic vs amplifying0.863 / 0.2521.8×10−341.8 \times10^{-34}3.1×10−383.1 \times10^{-38}9/0/00.0040.75 [0.67, 0.84]
semantic vs neutral0.863 / 0.0152.6×10−692.6 \times10^{-69}7.4×10−637.4 \times10^{-63}9/0/00.0040.94 [0.88, 0.98]
amplifying vs neutral0.252 / 0.0152.3×10−92.3 \times10^{-9}1.4×10−81.4 \times10^{-8}7/2/00.0160.25 [0.13, 0.36]
pooledamplifying vs neutral0.207 / 0.0059.8×10−259.8 \times10^{-25}1.7×10−211.7 \times10^{-21}17/10/02.5×10−52.5 \times10^{-5}0.24 [0.18, 0.31]

Table 17.

The semantic class differs from both semantics-preserving classes in all 27 cells, with large effects (δ≥0.75\delta\ge0.75), CMH p<10−30p < 10^{-30} at every noise level, and the minimum attainable cell-level pp (0.0039). The noise-amplifying and noise-neutral classes differ under the CMH test at every level (p≤4.3×10−6p \le4.3 \times10^{-6}) and at the cell level when the 27 cells are pooled (p=2.5×10−5p = 2.5 \times10^{-5}; higher rate in 17 cells, equal in 10, lower in none). Per level, the cell-level test is significant only at high noise (p=0.016p = 0.016), because at low noise only 4 of 9 cells reject any noise-amplifying mutant. The effect is small (δ=0.23\delta= 0.23–0.27, intervals excluding zero and reaching at most 0.38), as expected of a second-order effect that becomes visible only where the added gates contribute enough error. The three classes are therefore distinguishable, with the separation between the two semantics-preserving classes weaker and noise-dependent.

Physical criterion after compilation. The tests above use the classes as assigned by operator. Table 18 checks the physical criterion of Definition 3.3 for the 615 noise-amplifying and 615 noise-neutral mutants of the core and QASMBench programs, compiled as in the experiments for three targets: the synthetic-noise backend, a CNOT-native IBM snapshot (FakeKolkataV2) and a CZ-native IBM Heron snapshot (FakeKingston). The check counts compiled operations; it establishes exposure to additional gate error, which the rejection rates and the dose study measure. On the synthetic backend, every noise-amplifying mutant adds operations and no noise-neutral mutant adds a two-qubit gate, but cx_to_h_cz_h keeps its Hadamards in 173 of 283 mutants, which are therefore weakly noise-amplifying. They do not drive the contrasts: none of the 159 core noise-neutral executions that add no operation is rejected, and with the class restricted to verified mutants the contrast with the noise-amplifying class remains significant on the QASMBench programs at every level (CMH p≤8.8×10−11p \le8.8 \times10^{-11}, sign-flip p≤2.7×10−4p \le2.7 \times10^{-4}) and on the core benchmarks when pooled (CMH p=1.4×10−4p = 1.4 \times10^{-4}). On the IBM targets, virtual RzR_z gates make ZZ, SS and TT pairs add operations only where barriers prevent merging, and a few noise-neutral reorderings are routed differently. Class membership is therefore a property of the mutant and the target and should be verified on the target. In the archived hardware circuits, 16 of the 18 noise-amplifying mutants of H1 add physical operations and none of its 12 noise-neutral mutants does; HM and H3 used only physical pairs. Because every engine is seeded identically, the single-qubit pairs drawn in the mutation analysis are YY, ZZ, SS and TT pairs.

Table 18. RQ2: semantics-preserving mutants whose compiled circuit contains more physical operations than the compiled original (core benchmarks and QASMBench programs; in parentheses: of these, mutants with additional two-qubit gates). Synthetic: the Aer backend of the synthetic noise model, on which every gate is noisy. CX-native: FakeKolkataV2; CZ-native: FakeKingston; on both, RzR_z is virtual. Expected: all for noise-amplifying operators, none for noise-neutral operators.

ClassOperatorSyntheticCX-nativeCZ-native
Noise-amplifyingcx_pair234/234228/234234/234
inverse_pair_1q, YY76/7676/7676/76
inverse_pair_1q, ZZ, SS, TT129/12915/12936/129
rotation_pair176/176176/176176/176
Noise-neutralcommute_swap0/2286/228 (6)10/228 (6)
cx_to_h_cz_h173/283 (0)28/283 (9)13/283 (9)
x_to_h_z_h0/1040/1047/104 (0)

Table 18.

The two semantic families do not separate: their rejection rates are 0.978 and 0.978 at low noise, 0.970 and 0.978 at medium noise, and 0.815 and 0.911 at high noise (p=0.033p = 0.033, not significant after correction; δ=−0.11\delta= -0.11). Additive and subtractive operators are families within one class.

Noise-amplification dose. Whether a noise-amplifying mutant is rejected should depend on how much it changes the noisy output distribution, which depends on the error it adds and on the program. Table 19 confirms this: rejection rises with the number of inserted pairs and with noise, from 0.000 for one single-qubit pair to 0.789 for four CNOT pairs. The power model of Equation (4), applied to each mutant’s true noisy divergence, predicts these rejections (Brier score 0.028 over 1,620 mutants, against 0.246 for a constant predictor; mutants predicted below 0.1 are rejected in 0.9% of cases, those above 0.9 in 98.5%). The program matters as well: CNOT pairs are rejected in 98–100% of cases on Bell, GHZ-3, Bell-8 and GHZ-8, whose concentrated outputs are visibly blurred by added error, but never on QFT-3 and in 2% on QAOA-8, whose nearly uniform outputs depolarizing error barely changes. The weak separation between the semantics-preserving classes is therefore a property of noise amplification, not a defect of the taxonomy: the oracle rejects a noise-amplifying mutant as often as its added error is detectable at the shot budget.

PairskkLow ObservedLow PredictedMedium ObservedMedium PredictedHigh ObservedHigh Predicted
CNOT10.4220.3590.6330.6600.7220.708
20.4890.5010.7330.7530.7560.769
40.6440.6710.7890.8170.7890.805
Single-qubit10.0000.0020.0000.0050.0110.009
40.0110.0100.1440.1120.2440.251
160.1670.1990.6110.5650.6560.671

Table 19. RQ2: rejection of noise-amplifying mutants by dose (kk inserted inverse pairs; 90 mutants per row, nine benchmarks). Observed: rejection rate under NAO. Predicted: mean rejection probability from the power model applied to each mutant’s noisy divergence (estimated from 200,000 shots). Lower observed rates are better from the tester’s view, but the oracle is correct to detect a real distributional change; the question is whether its rejections are predictable.

Extended suite. On the 32 QASMBench programs, the original program failed NAO in one of 96 cells (Linear-solver at high noise), which is excluded. NAO rejects 77.2% of the 2,850 semantic mutants (93.5%, 80.5% and 56.8% at low, medium and high noise), 15.7% of the noise-amplifying and 0.3% of the noise-neutral mutants; the SPC envelope 60.8%, 1.3% and 0%. With 32 cells per level, every contrast is significant at every noise level under both the CMH test (p≤8×10−14p \le8 \times10^{-14}) and the cell-level test (p≤1.5×10−4p \le1.5 \times10^{-4}). The contrast between the semantics-preserving classes is again small (δ=0.21\delta= 0.21–0.32), whereas the semantic class separates from the noise-amplifying class with δ=0.95\delta= 0.95, 0.78 and 0.51 at low, medium and high noise.

IBM Quantum hardware. On ibm_kingston, NAO rejected 4 of 18 noise-amplifying and none of 12 noise-neutral mutants in H1, and 4 of 10 and none of 5 in HM: the ordering of the simulation.

What the evidence supports. The evidence supports three claims to different degrees. The classes are well defined: semantic membership is decided exactly for every mutant, and physical membership is checkable on any target by compiled gate counts. The classes behave differently in the evaluated data: strongly for semantic against semantics-preserving mutants, and weakly but consistently for noise-amplifying against noise-neutral mutants, also with verified classes. The classes predict behavior on programs not used to design them: the QASMBench programs reproduce every contrast, the power model predicts individual rejections, and the ordering recurs on hardware. This third claim rests on depolarizing noise, IBM device snapshots and one IBM device.

Answer to RQ2

Equivalent mutants are frequent: 24–70% of raw draws of the gate-insertion operators are equivalent on ∣0⋯0⟩\lvert0 \cdots0 \rangle, and every insertion after the final measurement is. With exact labels, NAO kills 93.8% of the semantic mutants, against 78.9% for the SPC envelope. The three classes are distinguishable: semantic mutants separate from both semantics-preserving classes in every cell with large effects, and noise-amplifying from noise-neutral mutants conditionally on the window at every noise level, more weakly because the separation depends on the dose of added error and on the program, as the power model predicts (Brier score 0.028). The separation persists with classes verified on compiled circuits, but membership depends on the target: on IBM devices, virtual ZZ, SS and TT pairs add no physical operation. Additive and subtractive operators form one class. On an IBM Heron device, NAO rejected 8 of 28 noise-amplifying and none of 17 noise-neutral mutants.

RQ3: Bug–Noise Attribution

Attribution is invoked only after the oracle rejects a program, so the operationally relevant cases are the 597 that NAO rejects: 543 bugs and 54 noise effects, all noise-amplifying mutants. Table 20 gives their confusion matrices by noise level. At low and medium noise, linear ZNE labels 372 of 377 rejected bugs Bug (recall 0.98–0.99, precision 0.984–1.000) and 27 of 30 rejected noise effects Noise (90%); quadratic ZNE labels 28 correctly (93%) at slightly lower bug recall. At high noise, noise recall falls to 3/24 (linear) and 12/24 (quadratic): the folded circuits at scales 3 and 5 approach the maximally mixed distribution, the extrapolation leaves a residual difference between the programs, and the null of the decision step, which contains sampling variation only, does not cover it (§ 3.6). Over all 597 cases, κ\kappa is 0.564 (linear) and 0.599 (quadratic; Table 21), reflecting both the high-noise failures and the imbalance of the subset.

MethodNoiseTPFNFPTNBug rec.Bug prec.Noise rec.
ZNE linearlow1863090.9841.0001.000
medium18623180.9890.9840.857
high155112130.9340.8810.125
ZNE quadraticlow1827090.9631.0001.000
medium18262190.9680.9890.905
high1481812120.8920.9250.500

Table 20. RQ3: attribution of the cases rejected by NAO, by noise level (positive class: Bug). TP/FN: bugs labeled Bug/Noise; FP/TN: noise effects labeled Bug/Noise. Higher is better in the last three columns.

CasesMethodAccuracyκ\kappaBug prec.Bug rec.Noise rec.Executions
AllZNE linear0.9200.8390.9410.9110.9314
ZNE quadratic0.9150.8300.9560.8870.9506
Re-execution triage0.8270.6621.0000.6841.0006a6^{a}
TriggeredZNE linear0.9330.5640.9560.9710.5564
ZNE quadratic0.9250.5990.9730.9430.7416
Re-execution triage0.7540.3281.0000.7291.0006a6^{a}

Table 21. RQ3: attribution against exact labels. Triggered: cases rejected by NAO (543 bugs, 54 noise), on which attribution is used in practice. All cases: every case regardless of the oracle verdict (582 bugs, 480 noise). Higher is better in every column. Executions: per attributed case. a^{a}Requires a scalable noise model and a noiseless simulator; not executable on hardware.

For comparison with procedures evaluated on every case, Table 21 also reports all 1,062 cases regardless of the oracle verdict: linear ZNE reaches accuracy 0.920 and κ=0.839\kappa= 0.839, and the quadratic variant is equivalent (κ=0.830\kappa= 0.830) at 50% more executions. Re-execution triage never labels a noise case as a bug but misses 31.6% of the bugs (κ=0.662\kappa= 0.662), because its fixed tolerance exceeds the divergence of many semantic changes, and it needs a noise model and a noiseless simulator.

Choice of extrapolation. On identical executions (Table 34), linear extrapolation has the highest agreement on all cases (κ=0.854\kappa= 0.854) and quadratic extrapolation the highest noise recall on triggered cases (0.759). Exponential extrapolation, which should remove the bias between circuits that decay at different rates, does not help: dividing by the estimated decay factor amplifies sampling noise and widens the null. On the device snapshots, all three reach κ≥0.883\kappa\ge0.883. We keep linear extrapolation as the default and recommend the quadratic variant when false bug reports cost more than six executions per case.

DataExtrapolationAccuracyκ\kappaBug rec.Noise rec.
Synthetic, alllinear0.9270.8540.9140.944
quadratic0.9170.8340.8870.954
exponential0.9120.8240.9120.912
Synthetic, triggeredlinear0.9320.5930.9670.603
quadratic0.9240.6150.9410.759
exponential0.9150.4770.9610.483
Snapshots, alllinear0.9590.9180.9520.967
quadratic0.9410.8830.9050.984
exponential0.9620.9230.9570.967

Table 34. RQ3: attribution by extrapolation method on identical executions (counts at s=1,3,5s = 1, 3, 5 stored once per case). Synthetic: the 1,062 cases of Table 21, executed again so that every method judges the same counts; NAO’s trigger is re-evaluated on these executions, so 603 cases are triggered here (58 noise effects) against 597 (54) in Table 21. Snapshots: FakeManilaV2 and FakeLagosV2, 392 cases. Higher is better.

IBM Quantum hardware. In H3 and in HM on ibm_kingston, linear ZNE attributed 10 semantic and 10 noise-amplifying mutants each, with accuracy 0.900 and κ=0.800\kappa= 0.800 in both: every semantic mutant was labeled Bug, and 8 of 10 noise-amplifying mutants Noise; the four errors are noise-amplifying mutants of Bell and GHZ-3 whose extrapolated distance (0.086–0.152) exceeded the null quantile. To check the folding, the archived transpiled circuits of every attribution pair were compared at s=3s = 3 and s=1s = 1: in 44 of 45 pairs with two-qubit gates, the folded circuit has exactly three times the two-qubit gates and 2.98–3.03 times the first-order error budget ∑gngeg\sum_g n_g e_g. The exception, a GHZ-3 mutant of HM, was routed differently (2.40 times the two-qubit gates) and has the largest extrapolated distance of all HM cases. Matching counts validate the folding construction, not the effective noise scale, which idle time, crosstalk and coherent errors can change; on hardware, attribution depends on the transpiler preserving the folded structure, which a tester can check before trusting a verdict.

Answer to RQ3

Zero-noise extrapolation attributes the deviations the oracle reports without a noiseless simulator, at four executions per case, but reliably only where its extrapolation model holds. At low and medium noise, it labels 98–99% of the rejected bugs Bug and 90% (quadratic: 93%) of the rejected noise effects Noise. At high noise, where folding saturates the output, it labels most rejected noise effects Bug (noise recall 0.125; quadratic 0.500), and exponential extrapolation does not remedy this. Over all cases, κ=0.839\kappa= 0.839 (device snapshots: 0.918); on an IBM Heron device, κ=0.800\kappa= 0.800 in H3 and HM, and the one folded circuit whose compiled structure fell short of the nominal factor produced the largest misattribution. ZNE attribution is a conditional diagnostic, not a general identifier of causes.

RQ4: Fault Detection, Power and Scaling

Recall, precision and F1. Table 22 reports the confusion matrix on the 50 non-equivalent injected faults (2,500 tests) and 1,350 matched correct runs. NAO detects 88.8% of the faults with precision 0.999 and F1 0.941; NAO-1 detects 87.7% and the SPC envelope 80.2% (F1 0.890). The fixed thresholds reach higher recall (up to 0.985) only by rejecting correct runs as well (65.6% at τ=0.01\tau= 0.01), and their F1 (0.830–0.861) is below NAO’s. They also reject 25–47% of the 800 equivalent fault tests, against 1/800 for NAO.

OracleTPFNFPTNRecall ↑\uparrowPrecision ↑\uparrowF1 ↑\uparrowFPR ↓\downarrow
NAO2221279213480.8880.9990.9410.0015
NAO-12192308113490.8771.0000.9340.0007
SPC envelope2004496013500.8021.0000.8900.0000
Fixed τ=0.01\tau= 0.012462388864640.9850.7350.8420.6563
Fixed τ=0.02\tau= 0.022431698035470.9720.7520.8480.5948
Fixed τ=0.05\tau= 0.0523031975448060.9210.8090.8610.4030
Fixed τ=0.1\tau= 0.121153854808700.8460.8150.8300.3556

Table 22. RQ4: detection of non-equivalent injected faults (positives, 2,500 tests) and matched correct runs (negatives, 1,350). TP/FN: detected/missed faults; FP/TN: rejected/accepted correct runs. ↑\uparrow: higher is better; ↓\downarrow: lower is better.

Where faults are missed. NAO detects 98% of the faults from very low to medium noise, 90.0% at high and 60.2% at very high noise (Figure 4). The misses at the two highest levels are faults whose effect noise has erased: both distributions approach the maximally mixed one, and their divergence falls below what 4096 shots resolve. By injector, recall ranges from 0.77 (wrong_angle) to 0.97 (swap_qubits; Table 35), so small changes are the hardest to detect. The SPC envelope loses most at high noise (0.704 and 0.498), where its floor exceeds the divergence of many faults.

Figure 4. RQ4: recall on non-equivalent injected faults by synthetic noise level (η=0.0005\eta= 0.0005, 0.001, 0.005, 0.02, 0.05; 500 fault tests per level). The fixed threshold compares with the ideal output; its rising recall at high noise comes with rejecting most correct runs (Table 22).

InjectornnNAONAO-1SPCFixed τ\tau=0.05
drop_last_gate4000.790.780.690.83
extra_x4000.930.910.861.00
phase_flip2000.780.730.540.66
remove_entanglement4000.930.930.931.00
swap_qubits3000.970.970.900.97
wrong_angle2000.770.760.550.85
wrong_basis3500.920.910.860.94
wrong_target2500.960.960.911.00

Table 35. Recall on non-equivalent injected faults by injector (RQ4). nn: fault tests. Higher is better.

Analytic power model. Figure 5 compares the power model of Equation (4) with simulation. On synthetic distributions with 4 to 256 outcomes (panel a), predicted and observed detection probability differ by 0.011 on average over 56 points, at most 0.14 at 256 outcomes, where the chi-square approximation is least accurate. On the injected faults (panel b), faults predicted to be detected with probability above 0.9 are detected in 99.8% (NAO-1) and 100% (NAO) of the tests, and those predicted below 0.1 in 1.5% and 0.8%. By noise level, NAO’s mean predicted and observed recall differ by at most 0.013, and by number of occupied outcomes by at most 0.007; the Brier score is 0.006, against 0.099 for a constant predictor. The model has no fitted parameters, and each fault’s divergence is estimated from separate 200,000-shot executions, so this is an out-of-sample check, limited to the evaluated faults, programs and noise levels. The model predicts whether a fault of known size is detected, not which faults a program contains; it serves to plan shot budgets (§ 6.5). Pooling K=8K = 8 runs reduces the minimum detectable divergence at 80% power by 44%, from 0.0019 to 0.0011 for four outcomes and from 0.0089 to 0.0050 for 256 (Table 36).

Figure 5. RQ4: (a) detection probability predicted by the noncentral chi-square model (lines) and observed in simulation (markers) on synthetic distributions with dd outcomes; (b) predicted versus observed recall on the 2,500 injected-fault tests, binned by predicted power; the dashed line is perfect agreement (y=xy = x).

Outcomes ddNAO-1NAO (K=8K = 8)
40.001900.00107
160.002970.00167
640.004970.00280
2560.008910.00501

Table 36. RQ4: minimum detectable Jensen–Shannon divergence at 4096 shots, 80% power and α=10−3\alpha= 10^{-3} (Equation (4)). Lower is better.

Noise-calibrated global threshold. A threshold calibrated on a noise model of the device and shared by all programs, as proposed by Fortz et al. [34], does not remove the problem of fixed thresholds. On the same executions in a separate run of the fault-detection design (Table 37), it has a false-positive rate of 0.108, recall 0.602 and F1 0.725, against 0.0007, 0.888 and 0.940 for NAO. Its false positives concentrate on Bell-8 (47%) and GHZ-8 (50%), whose distance from the ideal output exceeds the 87.5th percentile over programs, and its recall falls from 0.96 at very low to 0.23 at very high noise, because the threshold must grow to accommodate the noisiest programs. The threshold is also computed from the programs it is applied to. Leaving each program out of the calibration in turn, on fresh executions, raises its false-positive rate from 0.097 to 0.221 on the core benchmarks and from 0.124 to 0.145 on the QASMBench programs, with unchanged recall: Bell-8 and GHZ-8 are then rejected in 99–100% of their correct runs. NAO uses no statistic across programs.

OracleRecall ↑Precision ↑F1 ↑FPR ↓Equivalent rejected ↓
NAO0.8881.0000.9400.00070/800
Noise-calibrated global threshold0.6020.9120.7250.108149/800

Table 37. RQ4: NAO and a noise-calibrated global threshold (after Fortz et al. [34]) on identical executions (2,500 non-equivalent fault tests, 1,350 correct runs, 800 equivalent fault tests). This is a separate run of the design of Table 22 with its own seeds, so NAO’s values differ from that table by sampling variation (here 1 false positive, there 2). ↑: higher is better; ↓: lower is better.

Ablation. Table 23 adds the components of NAO one at a time on identical executions (a separate run, so row F differs from Table 22 by sampling variation). On synthetic noise, replacing the ideal reference by one self-baseline run (A → B) lowers the false-positive rate from 0.599 to 0.192, but a fixed threshold remains unsound, because the finite-shot variation between two runs depends on the program and the noise level. The SPC envelope (C) removes the false positives at the price of recall (0.802). The null-quantile threshold (D) keeps the rate nominal (0.0013) and raises F1 from 0.890 to 0.934, the largest gain of any component. Pooling (E) raises recall to 0.886. The drift test (F) changes no verdict on synthetic noise, where calibrations are homogeneous, but is decisive under recompilation: without it, D and E reject 8.3% and 12.1% of the correct runs, because the variation between compilations is not part of the finite-shot null, and pooling makes this worse by narrowing the null. With it, the rate falls to 0.009 at recall 0.961, and F has the highest F1 (0.976). B and C are sound under recompilation, because the change between compilations of these small programs stays below 0.02, but lose 4–5 points of recall; whether a constant threshold is sound depends on the setting and cannot be known before it is measured.

Table 23. RQ4: ablation of NAO. Each row adds one component to the row above, and all rows judge identical executions. Synthetic noise: nine core benchmarks at five noise levels (450 windows; 2,250 correct runs, 2,500 non-equivalent fault tests). Snapshots, recompilation: five small benchmarks on three device snapshots under the random compilation policy (150 windows; 750 correct runs, 750 non-equivalent fault tests). ↑: higher is better; ↓: lower is better.

ReferenceThresholdSynthetic noiseSnapshots, recompilation
FPR ↓Recall ↑F1 ↑FPR ↓Recall ↑F1 ↑
AIdeal (noiseless)Fixed τ=0.02\tau= 0.020.59870.9700.7730.46670.988
B1 self-baseline runFixed τ=0.02\tau= 0.020.19160.8380.8340.00000.920
C1 self-baseline runSPC envelope0.00000.8020.8900.00000.915
D1 self-baseline runNull quantile (NAO-1)0.00130.8780.9340.08270.977
E8 pooled runsNull quantile0.00090.8860.9390.12130.979
F8 pooled runsNull quantile + drift test (NAO)0.00090.8860.9390.00930.961

Table 23.

Real bugs. Table 24 reports the ten real bugs from Bugs4Q. On the eight that change the ideal output on ∣0⋯0⟩\lvert0 \cdots0 \rangle, NAO detects 96.4% of the buggy runs across five noise levels and two snapshots, with precision 0.994, F1 0.979 and a false-positive rate of 0.0014 on the fixed programs; all its misses occur at very high noise. The SPC envelope detects 89.6% (F1 0.945), and the fixed thresholds detect 96–98% but reject 42–77% of the runs of the fixed programs (F1 0.41–0.54). ZNE attribution labels 160 of 163 attributed rejections of observable bugs Bug.

Two bugs do not change the ideal output on ∣0⋯0⟩\lvert0 \cdots0 \rangle: Terra #5580 exchanges controls and targets of two controlled-RzR_z gates, which changes only phases, and Terra #5098 misspecifies a control state that does not fire on this input. NAO nevertheless rejects 82% of their runs, because the buggy circuits compile to different physical gate sequences whose noisy outputs genuinely differ, the effect seen for noise-amplifying mutants. ZNE labels 16 of the 36 attributed rejections Noise; for #5098, a BUG verdict is correct on other inputs.

Detection on device snapshots. On the device snapshots under the fixed policy, NAO detects 96.6% of the semantic mutants and non-equivalent faults (n=381n = 381) and rejects 2.2% of the semantics-preserving mutants (n=325n = 325); the SPC envelope detects 85.8% (1.5%), and τ=0.05\tau= 0.05 detects 89.8% but rejects 33.5%.

Scaling. Table 25 extends the evaluation to GHZ circuits of up to 16 qubits and QAOA circuits of up to 12 qubits at medium noise. Full-distribution NAO is sound on every GHZ circuit, whose output concentrates on few outcomes at any width, and detects every fault. On QAOA-12, whose output occupies almost all 4096 outcomes with about one shot each, it rejects 2 of 32 correct runs (0.062), because the unseen-outcome correction and the chi-square approximation are no longer accurate. Marginal NAO, which compares all two-qubit marginals, is sound on every circuit at recall

Table 25. RQ4: scaling at η=0.005\eta= 0.005 and 4096 shots. Shots/outcome: 4096/2n4096/2^n. Full: NAO on the full distribution; marginal: NAO on all two-qubit marginals (Bonferroni). FPR: lower is better; recall: higher is better.

CircuitQubitsShots/outcomeFPR ↓Recall ↑
FullMarginalSPCFullSPC
GHZ8160.0000.0000.0001.0001.000
GHZ1040.0000.0000.0001.0001.000
GHZ1210.0000.0000.0001.0001.000
GHZ140.250.0000.0000.0001.0001.000
GHZ160.060.0000.0000.0001.0001.000
QAOA8160.0000.0000.0001.0001.000
QAOA1040.0000.0000.0001.0000.975
QAOA1210.0620.0000.0001.0000.700

Table 25.

1.000; the SPC envelope is sound but loses recall on QAOA-12 (0.700). Full-distribution comparison is therefore limited by the ratio of shots to occupied outcomes rather than by the qubit count.

Extended suite. On the 163 non-equivalent faults of the QASMBench programs (Table 10), NAO has recall 0.776, precision 0.998 and the highest F1 (0.873). Recall is lower than on the core benchmarks because the suite contains deeper programs: it is 0.99–1.00 at very low and low noise and 0.39 at very high noise, and at medium noise NAO detects 6% of the fault runs of HHL-7 (depth 551) and none of DNN-8 (192 two-qubit gates), whose noisy outputs are nearly uniform, whereas shallow programs such as Cat-state-4 keep recall 0.89–1.00 even at high noise. The SPC envelope reaches F1 0.795, the QOIN-style filter 0.685 and the global threshold 0.640.

IBM Quantum hardware. In H1, NAO and NAO-1 detected all 78 semantic mutants and non-equivalent faults of GHZ-3, QAOA-3 and Bell-8 (Wilson 95% lower bound 0.95). In HM (Table 14), NAO detected all 23 faults and semantic mutants of the small programs, 19 of 20 of the larger programs and all 16 runs of the buggy real-bug programs; every fixed threshold either detected fewer or rejected correct runs as well. The one miss is a mutant of HWE-6 whose ideal divergence, 8.5×10−58.5 \times10^{-5}, is 33 times smaller than the minimum detectable divergence for 64 outcomes (0.0028), so the power model predicts it.

Answer to RQ4

Soundness does not cost sensitivity: NAO detects 88.8% of non-equivalent faults with precision 0.999 and the highest F1 of all oracles (0.941). A threshold calibrated globally on the device’s noise model reaches F1 0.725 with 10.8% false positives, and more on programs held out from its calibration. The ablation attributes the result to the oracle’s components: the self-baseline reference removes most false positives, the null-quantile threshold gives the largest gain in F1, and the drift test keeps the oracle sound under recompilation (0.009 against 0.121). The ranking holds on the QASMBench programs (F1 0.873) and on real bugs (96.4% of observable runs detected, F1 0.979). Misses concentrate at the two highest noise levels and on small changes, and the power model predicts detection accurately enough to plan shot budgets. Full-distribution comparison was sound on GHZ circuits up to 16 qubits and failed near one shot per occupied outcome (QAOA-12), where marginal comparison remained sound. On an IBM Heron device, NAO detected 120 of 121 faults and semantic mutants, missing one the power model predicts to be undetectable, and all runs of eight real bugs.

RQ5: Drift and Cost

Drift after calibration. Figure 6a reports the false-positive rate when the noise level at test time is η0(1+r)\eta_0(1+r) after calibration at η0=0.005\eta_0 = 0.005. A 10% increase is tolerated (FPR ≤0.016\le0.016); a 25% increase raises the rate of NAO to 0.27–0.66 on GHZ-3, QAOA-3 and Bell-8, and a 50% increase to 0.98–1.00. This is expected of an oracle calibrated to a nominal level: after drift, a correct program no longer has the reference’s output distribution, and the oracle detects the difference. The SPC envelope is insensitive to drift on the small benchmarks only because its floor is far above their divergences, and QAOA-8 is least affected because its threshold is large relative to the change.

Figure 6. RQ5: (a) false-positive rate of NAO when the noise level at test time is η0(1+r)\eta_0(1+r) after calibration at η0=0.005\eta_0 = 0.005, without recalibration (solid) and with a one-run sentinel at αs=10−3\alpha_s = 10^{-3} followed by recalibration (dotted); (b) fraction of calibration windows in which the homogeneity test detects drift that occurs within the eight calibration runs. 25 windows per point. Panel (a) uses executions separate from those of Table 26, so its values differ from the table by sampling variation.

Recalibration policies. Table 26 compares recalibration policies on paired executions. A one-run sentinel at αs=10−3\alpha_s = 10^{-3} restores soundness after a 50% drift but not after a 25% drift (FPR 0.17–0.26), because one run at a strict level has limited power. Two sentinel runs pooled and tested at αs=10−2\alpha_s = 10^{-2} detect every 50% drift and, on GHZ-3, QAOA-3 and Bell-8, every 25% drift, and keep the false-positive rate of the subsequent tests at or below 0.008 for every benchmark and drift level. The price is unnecessary recalibration: in at most 4% of drift-free windows, and in up to 24% under a 10% drift, which does not yet harm the tests. Every policy with m≥2m \ge2 keeps the rate at or below 0.04. Co-scheduled calibration is sound by construction and costs KK executions per session.

BenchmarkrrNoneSentinel (1,10−3)(1, 10^{-3})Sentinel (2,10−2)(2, 10^{-2})Co-scheduled
GHZ-30.000.0000.000 (0.00)0.000 (0.04)0.000
0.100.0000.000 (0.00)0.000 (0.16)0.000
0.250.4640.264 (0.44)0.000 (1.00)0.000
0.500.9920.008 (1.00)0.008 (1.00)0.008
QAOA-30.000.0000.000 (0.00)0.000 (0.00)0.000
0.100.0240.008 (0.04)0.008 (0.24)0.000
0.250.6240.168 (0.76)0.000 (1.00)0.000
0.501.0000.000 (1.00)0.000 (1.00)0.000
Bell-80.000.0080.008 (0.00)0.008 (0.00)0.000
0.100.0000.000 (0.00)0.000 (0.04)0.000
0.250.3200.176 (0.32)0.000 (1.00)0.000
0.501.0000.000 (1.00)0.000 (1.00)0.000
QAOA-80.000.0000.000 (0.00)0.000 (0.00)0.000
0.100.0000.000 (0.00)0.000 (0.00)0.000
0.250.0000.000 (0.00)0.000 (0.04)0.000
0.500.1280.112 (0.12)0.000 (1.00)0.000

Table 26. RQ5: false-positive rate of NAO after the noise level changes from η0=0.005\eta_0 = 0.005 to η0(1+r)\eta_0 (1 + r) between calibration and testing, under recalibration policies evaluated on the same executions (25 windows of 5 test runs per row). Sentinel (m,αs)(m, \alpha_s): mm runs of PP pooled and tested at level αs\alpha_s before testing; recalibration rate in parentheses. Co-scheduled: calibration in the same session as the tests. Lower FPR is better.

Drift during calibration. When the noise drifts linearly within the eight calibration runs (Figure 6b), the homogeneity test detects r=0.5r = 0.5 in every window on GHZ-3, QAOA-3 and Bell-8 and r=0.25r = 0.25 in 32–56% of windows; on QAOA-8, it detects r=1.0r = 1.0 in 84%. At r=0r = 0, it flags 0–4% of windows, consistent with αdrift=0.01\alpha_{\mathrm{drift}} = 0.01.

Cost. Calibration is a fixed cost per (program, backend) pair (Table 28): NAO needs K=8K = 8 executions, NAO-1 one, the SPC envelope 13 and the QOIN-style filter Kc=5K_c = 5 plus a noiseless simulation. A mutation campaign of 100 mutants costs 108 executions with NAO, 87% more than an (unsound) fixed threshold, and 112 with a two-run sentinel every 50 tests, plus eight executions per recalibration. On ibm_kingston, the 128 executions of the first HM job used 140 seconds of QPU time, about 1.1 seconds per 4096-shot execution including overhead (H1: 1.09 seconds), so calibration costs about 9 seconds per (program, backend) pair, a sentinel about 2 seconds, and the 100-mutant campaign about two minutes. Attribution adds four executions per rejection to be explained.

IBM Quantum hardware. Campaign H2 executed eight calibration sessions of GHZ-3 and QAOA-3 on ibm_kingston over 26.3 hours. Within a session, the runs were homogeneous: the homogeneity test flagged none of the 16 calibrations. Between sessions, they were not. Judging each later session against each earlier calibration (28 ordered pairs per program), a stale calibration of GHZ-3 rejected 34% of the correct runs less than six hours later, 57% at twelve to twenty-four hours and all of them after a day, and the divergence between sessions reached 3.5 times the NAO threshold. QAOA-3, executed on other qubits, moved by at most 0.54 times its threshold, and its stale calibrations rejected 4% of the correct runs. To evaluate the sentinel, the unit of analysis is an ordered pair of sessions (56 pairs): the first two runs of the later session form the sentinel, tested at αs=10−2\alpha_s = 10^{-2} against the earlier calibration, and its remaining six runs are the protected test runs (336). In 27 pairs the stale calibration rejected at least one protected run, and the sentinel fired in 22 of them; in the other 29, it fired in 9, causing unnecessary recalibration. Without the sentinel, 92 of the 336 protected runs (27.4%) are rejected; with it, and recalibration in the same session when it fires, 6 (1.8%). Pairs that share a session are not independent, so these counts describe this campaign rather than estimate rates.

In HM, the second job (24.6 hours after the first) was judged against the first job’s calibrations. GHZ-3 had drifted (divergence 0.0019 against a threshold of 0.0013), and its stale calibration rejected all three correct runs (3 of 15 overall), while the other benchmarks moved by 0.0004 or less. The two-run sentinel fired on GHZ-3 and on QFT-3 and VQE-3, which had barely moved. Since the allowance did not permit full recalibration, the two sentinel runs served as the new reference, and one correct run was rejected (VQE-3). The fourth job (48.1 hours) applied the policy as specified, with an eight-run recalibration: the stale calibration again rejected all three runs of GHZ-3, the sentinel fired on GHZ-3 and on three benchmarks that had moved less than their thresholds, and with the recalibrated references no correct run was rejected (0 of 15). All 9 faults were detected under every policy in both jobs. Drift between days is therefore real on this device, invalidated one calibration in five of both days, and is removed by the sentinel policy at the cost of unnecessary recalibrations, the trade-off of the lenient level αs=10−2\alpha_s = 10^{-2} measured in simulation.

Answer to RQ5 Without recalibration, NAO tolerates a 10% change of the noise level between calibration and testing but not a 25% change, because it is sensitive enough to detect the changed output distribution. A two-run sentinel at αs=10−2\alpha_s = 10^{-2} before each batch detects such drift and keeps the false-positive rate at or below 0.008 under every evaluated drift, at the cost of recalibrating in up to 4% of drift-free batches; co-scheduled calibration is sound by construction. The homogeneity test detects strong drift within the calibration window. Calibration adds K=8K = 8 executions per program and backend, 8% of a 100-mutant campaign. On an IBM device, drift is real and program-dependent: calibrations of GHZ-3 went stale within hours, whereas QAOA-3 was barely affected. The sentinel caught 22 of 27 stale calibrations that produced false positives, reducing them from 27.4% to 1.8% of the protected runs, and in HM the full policy removed all false positives, at the cost of some unnecessary recalibrations.

Discussion

Self-Baseline Calibration as an Oracle Reformulation

Conventional NISQ correctness oracles retain the noiseless ideal distribution as their reference and pursue robustness by acting on the comparison: a fixed-threshold oracle widens the tolerance, and a noise-filtering oracle in the style of QOIN estimates and removes the hardware noise before comparing with the ideal. Both continue to ask how far a run lies from an output the device cannot produce. Self-baseline calibration changes the reference: the oracle asks whether a run lies within the variability that the reference program itself exhibits under repeated execution on the same backend.

The evaluation separates the design decisions this involves, and the ablation (Table 23) measures each on identical executions. The choice of reference removes most false positives: every oracle that compares against the ideal output produces false positives that grow with the noise or concentrate on particular programs, whereas the oracles that compare against the self-baseline are sound in the evaluated settings. The reference alone is not sufficient, since a fixed threshold against one self-baseline run still rejects a fifth of the correct runs on synthetic noise. The choice of threshold decides whether the false-positive rate is known in advance and how sensitive the oracle is: the SPC envelope shares the reference but is set by its floor, which costs recall and mutation score, whereas the null-quantile threshold turns the false-positive rate into a nominal level and makes recall predictable from the power model. Pooling adds sensitivity and stabilizes the threshold across windows, and the drift test is inactive on homogeneous calibration runs but indispensable under recompilation. Table 27 summarizes the three paradigms.

Fixed or global thresholdNoise filtering (QOIN-style)Self-baseline calibration (NAO)
Referencenoiseless ideal outputideal output, after noise removalpooled executions of the reference program on the same backend
Requiresclassical simulation of the ideal output; a noise model for global calibrationKcK_c correct runs and the ideal outputKK runs of the reference program; no ideal output, noise model or learned model
Thresholdchosen by the tester, or one per noise model for all programsfixed on the corrected output(1−α)(1-\alpha) quantile of the finite-shot null distribution
FPR known in advancenonoyes (α\alpha)
ObservedFPR 0.36–0.74, F1 0.830–0.861 (fixed); FPR 0.108, F1 0.725 (global)FPR 0.089FPR 0.0011 at α=10−3\alpha=10^{-3}; recall 0.888; F1 0.941

Table 27. Three oracle paradigms for NISQ correctness testing. The noise-filtering column is a reproducible, ML-free stand-in for QOIN’s paradigm; QOIN itself learns the filter through transfer learning [8]. FPR: false-positive rate over 9,000 correct runs (RQ1); recall and F1 on non-equivalent injected faults (RQ4).

What the Oracle Decides, and What It Cannot

The oracle decides whether P′P' behaves like PP on the backend; it does not decide whether PP is correct. The expected output is not needed, because the reference’s own noisy behavior on the same backend is the specification. If the reference is faulty, the oracle accepts versions that share its faults and rejects versions that fix it, so a faulty reference is detected only by tests of the reference itself, such as property-based or specification-based tests on small instances [11, 15], or by comparison with the ideal output where it can be computed. Within the differential setting, the oracle distinguishes noise from a fault only in a statistical sense. A rejection can still be caused by noise, because the noise changed (RQ5) or because the program under test interacts differently with it (RQ2); these are the cases for which the sentinel policy and the attribution procedure are intended.

Sensitivity and Operating Regimes

An oracle that accepts every run has no false positives and detects nothing, so soundness alone is not evidence of usefulness. The trade-off of NAO is explicit: lowering α\alpha raises the threshold, and Equation (4) gives the recall that is lost. Its misses are not caused by a conservative threshold: at the two highest noise levels, noise has removed most of the faults’ effect from the output distribution, and the fixed thresholds that still detect these faults do so by rejecting most correct runs as well. Depth reduces sensitivity as width does, because noise accumulates over more gates, as the deeper QASMBench programs show. The applicability of full-distribution comparison is governed by the ratio between the shot budget and the outcomes a program actually occupies rather than by its qubit count: concentrated outputs remain testable at 16 qubits, whereas outputs spread over all outcomes leave the evaluated regime near one shot per outcome, where marginal comparison remains an option. The evaluated range, up to 16 qubits and two circuit families in the scaling study, is too narrow to establish this as a general law.

Beyond Quantum

The principle underlying the oracle, to judge correctness against a baseline that the system establishes about itself rather than against a fixed external tolerance, is not specific to quantum execution. Flaky tests in continuous integration [19, 20], machine-learning inference affected by numerical nondeterminism [22] and performance-regression testing all judge a noisy signal against a tolerance fixed in advance. The three ingredients developed here, a repeated-execution baseline, the null distribution of the comparison statistic under “same behavior”, and a homogeneity check of the baseline, have natural analogues wherever the sampling distribution of the correctness signal can be modeled or bootstrapped. We state this as a structural observation, not a validated result: we have not evaluated the mechanism outside the quantum setting.

Implications for Practice

  1. Use the noise-aware oracle for regression and mutation testing against a trusted reference. Choose α\alpha from the number of tests and the acceptable number of false alarms: at α=10−3\alpha= 10^{-3}, a campaign of 1000 tests of correct programs is expected to raise one false alarm.

  1. Use NAO by default; use NAO-1 only on a stable, compiled-once backend. NAO-1 saves seven executions per program and backend, but it cannot detect drift or recompilation and is less sensitive.

  1. Compile once. Transpile the reference and every program under test with a fixed seed onto one initial layout; recompilation between runs changes the noise the program experiences.

  1. Plan the shot budget with the power model. When the size of the faults to detect is not known, fix the smallest divergence whose detection matters, for example the ideal divergence of the smallest change considered a regression, computed by statevector simulation on a reduced instance, and solve Equation (4) for NN at K=8K = 8 and the desired power. Where no such divergence can be named, report the minimum detectable divergence of the available budget as the campaign’s sensitivity, so that a passing test is read as “no change larger than this”. After the first calibration run, check that it has about 16 or more shots per observed outcome; otherwise add shots or compare marginals.

  1. Check the baseline before each batch. The need to recalibrate cannot be predicted from the elapsed time alone. Before each batch of tests, execute the reference twice and test the pooled runs at αs=10−2\alpha_s = 10^{-2}; recalibrate if the test rejects. Where the platform allows it, submit the calibration runs in the same job as the tests.

(6) Label mutants before running them. Exact statevector labels are cheap at the widths at which mutation analysis is usually performed, and they make the mutation score interpretable.

(7) Attribute rejections of semantics-preserving changes, at low and medium noise. When a refactoring that should preserve semantics is rejected, ZNE attribution (four executions) indicates whether the rejection is a noise effect or a fault; at high noise, treat a Bug verdict with caution, check that the folded circuits keep their structure, or use the quadratic variant.

(8) Develop on device snapshots. Under fixed compilation, the snapshot results agree with the synthetic-noise results, so test suites can be developed on free, reproducible fake backends before hardware time is committed.

Implications for Research

State the false-positive rate of an oracle. A test oracle for stochastic programs should state the false-positive rate it is designed for and show that it attains it. Reporting only that an oracle produced no false positives does not distinguish a calibrated oracle from an insensitive one; reporting recall, precision and F1 alongside does.

Label mutants exactly and place them before measurement. Mutation scores of quantum programs are interpretable only if equivalent mutants are excluded, which exact labels make possible at moderate widths.

Validate taxonomies conditionally on shared calibrations. When mutants share a calibration, tests that treat them as independent overstate significance; stratified and cell-level tests give the honest strength of a taxonomy’s support. Classes defined by the mechanism of a change should also be validated against a quantitative account of that mechanism, such as the dose-response model used here, and, where the mechanism depends on compilation, membership should be checked on the compiled circuits of the target.

Threats to Validity

We organize threats following the classification of Wohlin et al. [68]: internal, construct, external and conclusion validity.

Internal Validity

Stochastic execution. Finite shot counts introduce sampling variance in every reported quantity. All experiments are driven by recorded seeds (mutant generation, simulator sampling, transpilation and bootstrap), so rerunning the experiments with these seeds reproduces every reported number exactly; the main rates are reported with confidence intervals, and the package reports intervals for all of them.

Parameter selection. The parameters of NAO were chosen from the null and power models and a factorial study on separate seeds (§ 4.8) that used five of the nine core benchmarks at one noise level, so it is not independent of them; the 32 QASMbench programs, which played no role in any design decision, reproduce the results with the same parameters. Moreover, the false-positive rate is set by α\alpha through the null model rather than by tuning, and every grid configuration with 4096 or more shots has recall 1.000, so the chosen configuration is not a selected optimum.

Shared calibrations. Tests in the same calibration window share a reference and a threshold. RQ1 and RQ4 use 40 and 10 independent windows per cell, and the false-positive rates of RQ1 carry cluster-bootstrap intervals over windows; RQ2 conditions on the window with a stratified test and additionally treats cells as units (§ 4.6).

Compilation. The transpiler can cancel inserted gates, change the layout between runs, or route programs under test differently from the reference. Inserted gate pairs are separated by barriers, programs under test are placed on the layout of the reference, and the effect of recompilation is measured explicitly (RQ1, random policy).

Implementation. The implementation includes 23 unit tests covering the threshold computation, the equivalence labels, the mutation operators, the folding and extrapolation used for attribution and the statistical procedures. The bootstrap threshold is checked against the closed-form chi-square threshold in every cell of RQ1.

Construct Validity

Computational-basis observability. The oracle compares measured output distributions, so a change that leaves the measured distribution unchanged on the tested input cannot be detected, whatever the distance or threshold (§ 3.1). Twelve of the injected faults change the unitary but not the output on ∣0⋯0⟩|0 \cdots0\rangle; they are reported separately and excluded from recall. Measurement in additional bases [36] would make such faults observable and is compatible with the threshold construction, which only requires a multinomial count vector.

Choice of distance. We use the Jensen–Shannon divergence [51], which is symmetric, bounded and well defined with zero counts. The null-quantile construction applies to any statistic whose null distribution can be bootstrapped from the reference counts; we do not claim that the Jensen–Shannon divergence is the most powerful choice.

Correctness of references. The differential setting assumes that the reference is correct. For injected faults and mutants this holds by construction. For the real bugs, the fixed program from Bugs4Q is the reference; a fixed program can itself be imperfect (the fixed phase-estimation program of Stack Exchange 15, for example, does not produce a sharp phase peak), but the oracle’s verdict concerns only whether the buggy and fixed programs behave alike.

Equivalence labels. Equivalence is defined with respect to the measured output distribution on the tested inputs. A mutant that is equivalent on these inputs may be non-equivalent on others; the four-input census shows that this matters for CNOT insertions and rarely for other operators. The labels require statevector simulation and are therefore available only at widths that can be simulated.

Attribution labels and noise scaling. The ground truth of attribution is the exact ideal output on the tested input, so a change of the unitary invisible on that input counts as Noise; such changes are kept out of the attribution cases. The procedure assumes that folding realizes the nominal scale factors and that extrapolation removes the difference noise causes. On ibm_kingston, the folded circuits had the nominal gate counts for all but one archived pair, which was misattributed, but gate counts do not capture idle errors, crosstalk or coherent errors. Saturation at high noise violates the second assumption, and other noise structures, such as coherent errors, which folding amplifies differently, may violate it as well.

Mutants and faults as proxies for real faults. Mutation scores and detection rates on injected faults are proxies for the detection of real faults. In classical software, mutant detection correlates with real fault detection, but not perfectly [16]; whether the same holds for quantum programs is not established, and we do not claim that the reported detection rates transfer unchanged to real defects. The ten real bugs from Bugs4Q are consistent with the injected-fault results (§ 5.4), but they are too few to estimate the correlation between mutation scores and real-fault detection.

Taxonomy operationalization. The classes are operationalized by the operators of Table 2. On the core benchmarks, the separation between the semantics-preserving classes is driven by cx_pair, since a single pair of single-qubit gates adds too little error to be detected; operators that add more physical operations may separate more strongly. Operators are proxies for the physical criterion, which depends on the target (Table 18); the contrasts hold with classes verified on the synthetic backend, but a study on another target should verify membership there. Every engine used the same seed, so operator draws are correlated across programs, and only YY, ZZ, SS and TT single-qubit pairs were drawn; HH and XX pairs were not evaluated. The dose study uses depolarizing noise, under which uniform outputs are immune to added error; under non-unital noise such as amplitude damping, the programs that react to noise amplification may differ.

Baselines. The QOIN-style filter is a stand-in for the noise-filtering strategy, not a reimplementation of QOIN’s transfer-learning pipeline [8]; a learned filter may achieve a lower false-positive rate. The noise-calibrated global threshold follows Fortz et al. [34] on our program sets at 4096 shots; its percentile is computed over the programs it is applied to, which favors it (on held-out programs its rate is higher), and a different program set or shot count changes its threshold, though not the fact that one threshold must serve all programs. Other attribution methods, such as noise fingerprinting [39], were not compared.

External Validity

Benchmarks. The nine core benchmarks cover state preparation, an arithmetic kernel and variational algorithms with fixed parameters, up to 8 qubits; the extended suite adds 32 QASMBench programs from 2 to 10 qubits across arithmetic, error correction, oracle algorithms, phase estimation, linear solvers, simulation, machine learning and variational circuits, and the scaling study extends two families to 16 qubits. Neither covers programs with mid-circuit measurement, reset or classical control, or parameterized circuits whose parameters change between versions; QASMBench programs with these features were excluded. Only ten real bugs from Bugs4Q met the inclusion criteria, so the real-bug results are evidence for the injected-fault results rather than a separate estimate of detection rates.

Noise. The main synthetic model is depolarizing with symmetric readout error; the coherent model reproduces the main results at two error strengths with a fixed error axis only. Device snapshots add realistic error rates, relaxation and coupling maps, but not drift, crosstalk or non-Markovian effects, and RQ5 models drift as a change of the error level only. Hardware drift was measured on one device for a few programs over up to two days and was strongly program-dependent (GHZ-3 against QAOA-3), so it cannot establish how often recalibration is needed in general.

Hardware. The hardware evaluation covers IBM Heron devices of one technology. The campaigns are sized to fit free IBM Quantum allowances: with 82 correct runs and 121 faults and semantic mutants in total, they confirm or refute the direction of the simulation results, but their rates carry wide intervals and are not estimates of the device’s false-positive rate or recall. Devices of other vendors and modalities (trapped ions, neutral atoms) have different noise structures.

Scope of the evidence. Taken together, the results establish the oracle’s behavior within the evaluated program families, shot budgets, noise models (depolarizing, coherent over-rotation and device snapshots), calibration procedures and IBM Heron devices. They do not establish universal false-positive control or detection performance across quantum workloads and devices; generalization to programs with mid-circuit measurement or classical control, to other hardware modalities and to noise processes with drift, crosstalk or non-Markovian memory remains an open empirical question.

Scope of the generality claim. Every experiment concerns quantum programs. The relevance of self-baseline calibration to other non-deterministic software (§ 6.4) is a motivated hypothesis, not an evaluated claim.

Conclusion Validity

Statistical power. The soundness result rests on 9,000 correct runs in 1,800 independent calibration windows, enough to distinguish an FPR of 10−310^{-3} from 2×10−32 \times10^{-3} only roughly; the cluster-bootstrap interval at α=10−3\alpha= 10^{-3}, [0.0004, 0.0018], quantifies this. On the core benchmarks, the per-noise-level cell-level taxonomy tests have nine cells each and therefore limited power; the extended suite, with 32 cells per level, resolves this for all class contrasts.

Approximate level. The level α\alpha is met only approximately (§ 3.2). The boundary of about 16 shots per observed outcome held for the three evaluated shapes, but the size of the violation below it depends strongly on the shape, other shapes may move it, no window fell between 8 and 16, and the study used one shot budget (N=4096N = 4096, K=8K = 8) and one bootstrap size (B=2000B = 2000). At B=4000B = 4000, the rate of an individual window lies between about 0.17 and 2.37 times the nominal level, and smaller BB varies more; every reported rate is an average over windows, and claims about a single window require a larger BB.

Multiple testing. The taxonomy is tested on three class pairs at three noise levels and pooled; Bonferroni correction over the three pairs is applied within each level. With correction over all twelve tests, the CMH results remain significant (p≤4.3×10−6p \le4.3 \times10^{-6}), and the cell-level result for the semantics-preserving classes is significant only when pooled.

Single research team. All experiments were designed, executed and analyzed by the authors. To mitigate this, all results are produced by scripts from recorded seeds, every number reported in the tables is regenerated from the raw per-execution records, and the raw hardware data, including job identifiers and transpiled circuits, are archived.

Conclusion

We presented a noise-aware testing framework for NISQ quantum software and evaluated it on nine core benchmarks, 32 QASMBench programs and ten real bugs under five depolarizing and two coherent-error noise levels, three device-snapshot backends, circuits of up to 16 qubits and IBM Quantum hardware, against fixed-threshold, envelope-based, QOIN-style, noise-model-calibrated and re-execution baselines.

The results support four claims. First, the noise-aware oracle, built on self-baseline calibration with a null-quantile threshold, approximately attains its nominal false-positive rate in the evaluated settings in which calibration and test runs are samples of one distribution and one run of the reference provides about 16 or more shots per observed outcome; this criterion is empirical, not a universal guarantee. Under these conditions it controls false positives where fixed thresholds against the ideal output are unsound, without a loss of sensitivity: it has the highest F1 of all evaluated oracles on the core and QASMBench programs, detects most runs of observable real bugs, and its power model predicts its recall. Second, equivalent mutants are frequent among quantum mutation operators, and exact statevector labels make mutation scores interpretable. Third, the three-class taxonomy is distinguishable under the oracle, strongly between semantic and semantics-preserving mutants and more weakly between noise-amplifying and noise-neutral ones, a difference the power model explains and that persists on compiled circuits. Fourth, zero-noise extrapolation attributes rejections to faults or noise without a noiseless simulator, reliably at low and medium noise. An ablation measures the contribution of each design decision, and the hardware campaigns on an IBM Heron device confirm soundness, detection and the need for a drift sentinel. The evaluation also shows where the approach stops working: drift between calibration and testing causes false positives unless it is detected before testing, full-distribution comparison is not reliably calibrated with too few shots per observed outcome, and attribution of noise effects fails at high noise.

The contribution beyond the specific framework is a change in how correctness oracles are built for software whose behavior is visible only through noise: instead of selecting a tolerance, derive the threshold from the system’s own observed variability and state the false-positive rate it implies. We conjecture, but do not demonstrate, that the same principle is useful for other non-deterministic software.

Future Work

  1. Hardware at scale and across vendors. Extend the hardware campaigns to more devices, larger programs and other qubit modalities (trapped ions, neutral atoms), and characterize how often recalibration is needed in practice.

  1. Continuous drift monitoring. Complement the sentinel by continuous monitoring, for example by sequential tests on reference executions interleaved with the tests or by live noise fingerprinting of the device [39], to reduce unnecessary recalibration.

  1. Sample-efficient representations. Generalize the null-quantile threshold from two-qubit marginals to observables and classical-shadow estimators, whose estimation cost does not grow with the output support, to extend sound testing beyond the shots-per-outcome limit.

  1. Phase-sensitive oracles. Combine the threshold construction with measurement in multiple bases [36] to detect faults that do not change the computational-basis distribution.

  1. Mutation scores and real faults. Combine exact labels with the wider program set and distance metrics of robust mutation analysis under noise [34], and study whether quantum mutation scores correlate with the detection of real quantum faults.

  1. Direct comparison with QOIN. Reproduce QOIN’s learned noise filter on the same benchmarks for a like-for-like comparison.

Declarations

Competing interests. The authors declare no competing interests.

Funding. This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Acknowledgments

We acknowledge the use of IBM Quantum services for this work. The views expressed are those of the authors and do not reflect the official policy or position of IBM or the IBM Quantum team.

References

A Derivation of the Null Distribution

Let p^\hat{p} and q^\hat{q} be the empirical distributions of n1n_1 and n2n_2 independent multinomial draws from π\pi over dd outcomes with πx>0\pi_x > 0, and m=(p^+q^)/2m = (\hat{p} + \hat{q})/2. With natural logarithms, a second-order Taylor expansion of f(t)=tln⁡tf(t) = t \ln t around mxm_x gives

JS⁡e(p^,q^)=12∑x[p^xln⁡p^xmx+q^xln⁡q^xmx]≈18∑x(p^x−q^x)2mx≈18∑x(p^x−q^x)2πx,\operatorname{JS}_{e}(\hat{p},\hat{q}) = \frac{1}{2} \sum_x \left[\hat{p}_x \ln\frac{\hat{p}_x}{m_x} + \hat{q}_x \ln\frac{\hat{q}_x}{m_x}\right] \approx\frac{1}{8} \sum_x \frac{(\hat{p}_x-\hat{q}_x)^2}{m_x} \approx\frac{1}{8} \sum_x \frac{(\hat{p}_x-\hat{q}_x)^2}{\pi_x},

since m→πm \to\pi. Under H0H_0, n1n2/(n1+n2)(p^−q^)\sqrt{n_1 n_2/(n_1+n_2)}(\hat{p}-\hat{q}) converges to a Gaussian vector with covariance diag⁡(π)−ππT\operatorname{diag}(\pi)-\pi\pi^{\mathsf{T}}, so the Pearson-type sum n1n2n1+n2∑x(p^x−q^x)2/πx\frac{n_1n_2}{n_1+n_2}\sum_x(\hat{p}_x-\hat{q}_x)^2/\pi_x converges to χd−12\chi^2_{d-1}, [18, 43]. With D=JS⁡e/ln⁡2D = \operatorname{JS}_{e}/\ln2 (base 2) and n1n2/(n1+n2)=1/(1/n1+1/n2)n_1n_2/(n_1+n_2) = 1/(1/n_1 + 1/n_2), this yields Equation (2). Under a fixed alternative with true divergence δ\delta (base 2), the same expansion gives a noncentral χd−12(λ)\chi^2_{d-1}(\lambda) with λ=8ln⁡2 δ/(1/n1+1/n2)\lambda= 8\ln2\ \delta/(1/n_1 + 1/n_2), which yields Equation (4). The approximation requires that most outcomes have several expected counts; RQ1 (sparse supports) and RQ4 (scaling) show where it breaks down.

B Supplementary Results

The main text summarizes the following tables and figure.

References

  1. [1]John Preskill. 2018. Quantum computing in the NISQ era and beyond. Quantum 2 (2018), 79.DOI
  2. [2]Kishor Bharti, Alba Cervera-Lierta, Thi Ha Kyaw, Tobias Haug, Sumner Alperin-Lea, Abhinav Anand, Matthias Degroote, Hermanni Heimonen, Jakob S Kottmann, Tim Menke, et al. 2022. Noisy intermediate-scale quantum algorithms. Reviews of Modern Physics 94, 1 (2022), 015004.DOI
  3. [3]Mir Mohammad Yousuf and Shabir Ahmad Sofi. 2026. A systematic exploration of quantum software engineering in the NISQ era: Methods, lifecycle practices, and a taxonomy of challenges. Neurocomputing (2026), 132809.DOI
  4. [4]Zhonghao Pan, Yang Feng, Zhiyuan Li, Yunxin Liu, and Yuanchun Li. 2023. Understanding the Impact of Quantum Noise on Quantum Programs. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 426–437. doi:10.1109/SANER56733.2023.00047DOI
  5. [5]Matteo Paltenghi and Michael Pradel. 2024. A survey on testing and analysis of quantum software. arXiv preprint arXiv:2410.00650 (2024).arxiv.org/abs/2410.00650
  6. [6]Neilson CL Ramalho, Erico A da Silva, Higor A de Souza, and Marcos Lordello Chaim. 2025. Quantum Testing in the Wild: A Case Study with Qiskit Algorithms. In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 780–785.
  7. [7]Yuechen Li, Minqi Shao, Jianjun Zhao, and Qichen Wang. 2026. A methodological analysis of empirical studies in quantum software testing. ACM Transactions on Software Engineering and Methodology (2026).DOI
  8. [8]Asmar Muqeet, Tao Yue, Shaukat Ali, and Paolo Arcaini. 2024. Mitigating noise in quantum software testing using machine learning. IEEE Transactions on Software Engineering (2024).arxiv.org/abs/2306.16992
  9. [9]Zhenyu Cai, Ryan Babbush, Simon C Benjamin, Suguru Endo, William J Huggins, Ying Li, Jarrod R McClean, and Thomas E O’Brien. 2023. Quantum error mitigation. Reviews of Modern Physics 95, 4 (2023), 045005.DOI
  10. [10]Kristan Temme, Sergey Bravyi, and Jay M Gambetta. 2017. Error mitigation for short-depth quantum circuits. Physical review letters 119, 18 (2017), 180509.DOI
  11. [11]Antonio García de la Barrera, Ignacio García-Rodríguez de Guzmán, Macario Polo, and Mario Piattini. 2023. Quantum software testing: State of the art. Journal of Software: Evolution and Process 35, 4 (2023), e2419.
  12. [12]Javier Ibarra Vaganzo, Daniel Arias Alamo, Alfredo Cuzzocrea, and Pablo Garcia Bringas. 2025. AssertsQ: A Quantum Assertion Tool for Quantum Software Debugging. (2025).
  13. [13]Matteo Paltenghi and Michael Pradel. 2023. MorphQ: Metamorphic testing of the Qiskit quantum computing platform. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2413–2424.DOI
  14. [14]Yipeng Huang and Margaret Martonosi. 2019. Statistical assertions for validating patterns and finding bugs in quantum programs. In Proceedings of the 46th International Symposium on Computer Architecture. 541–553.DOI
  15. [15]Gabriel Pontolillo, Asmar Muqeet, Shaukat Ali, and Mohammad Reza Mousavi. 2025. From Ideal to Noisy: Adapting Property-Based Testing for Real-World Noisy Quantum Computers. In 2025 IEEE International Conference on Quantum Computing and Engineering (QCE), Vol. 1. IEEE, 405–416.DOI
  16. [16]Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. Are Mutation Scores Correlated with Real Fault Detection? A Large Scale Empirical Study on the Relationship Between Mutants and Real Faults. In Proceedings of the 40th International Conference on Software Engineering. 537–548. doi:10.1145/3180155.3180183DOI
  17. [17]Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Józala. 2014. Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Transactions on Software Engineering 40, 1 (2014), 23–42.
  18. [18]Timothy R. C. Read and Noel A. C. Cressie. 1988. Goodness-of-Fit Statistics for Discrete Multivariate Data. Springer, New York.
  19. [19]Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering. 643–653.DOI
  20. [20]Owain Parry, Gregory M Kaphammer, Michael Hilton, and Phil McMinn. 2021. A survey of flaky tests. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1 (2021), 1–74.DOI
  21. [21]Jonathan Bell, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tiffany Yung, and Darko Marinov. 2018. DeFlaker: Automatically detecting flaky tests. In Proceedings of the 40th international conference on software engineering. 433–444.
  22. [22]Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering 48, 1 (2020), 1–36.arxiv.org/abs/1906.10742
  23. [23]Jianjun Zhao. 2020. Quantum software engineering: Landscapes and horizons. arXiv preprint arXiv:2007.07047 (2020).arxiv.org/abs/2007.07047
  24. [24]Jiyuan Wang, Qian Zhang, Guoqing Harry Xu, and Miryung Kim. 2021. QDiff: Differential testing of quantum software stacks. In 2021 36th IEEE/ACM International conference on automated software engineering (ASE). IEEE, 692–704.DOI
  25. [25]Sergio Segura, Gordon Fraser, Ana B Sanchez, and Antonio Ruiz-Cortés. 2016. A survey on metamorphic testing. IEEE Transactions on software engineering 42, 9 (2016), 805–824.DOI
  26. [26]Daniel Fortunato, José Campos, and Rui Abreu. 2022. QMutPy: A mutation testing tool for quantum algorithms and applications in Qiskit. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 797–800.DOI
  27. [27]Daniel Fortunato, José Campos, and Rui Abreu. 2022. Mutation testing of quantum programs: A case study with Qiskit. IEEE Transactions on Quantum Engineering 3 (2022), 1–17.DOI
  28. [28]Eñaut Mendizulze, Shaukat Ali, Paolo Arcaini, and Tao Yue. 2021. Muskit: A mutation analysis tool for quantum software testing. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1266–1270.
  29. [29]Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2022. Quito: a coverage-guided test generator for quantum programs. In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering. 1237–1241. doi:10.1109/ASE56224.2021.9678798DOI
  30. [30]Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2024. QuCAT: A Combinatorial Testing Tool for Quantum Software. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering. 2066–2069. doi:10.1109/ASE56229.2023.00062DOI
  31. [31]Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2022. QuSBT: search-based testing of quantum programs. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 173–177. doi:10.1145/3510454.3516839DOI
  32. [32]Shaukat Ali, Paolo Arcaini, Xinyi Wang, and Tao Yue. 2021. Assessing the Effectiveness of Input and Output Coverage Criteria for Testing Quantum Programs. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST 2021). 13–23. doi:10.1109/ICST49551.2021.00014DOI
  33. [33]Daniel Fortunato, José Campos, and Rui Abreu. 2024. Gate Branch Coverage: A Metric for Quantum Software Testing. In Proceedings of the 1st ACM International Workshop on Quantum Software Engineering: the Next Evolution, QSE-Ne 2024. 15–18. doi:10.1145/3663531.3664753DOI
  34. [34]Sophie Fortz, Eñaut Mendizlue Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi. 2026. Robust Mutation Analysis of Quantum Programs Under Noise. arXiv preprint arXiv:2605.13279 (2026). https://arxiv.org/abs/2605.13279
  35. [35]Aahmik Virani, Devraj, Anirudh Suresh, Lei Zhang, and M. V. Panduranga Rao. 2025. Distinguishing Quantum Software Bugs from Hardware Noise: A Statistical Approach. ArXiv abs/2507.20475 (2025).arxiv.org/abs/2507.20475
  36. [36]Noah H Oldfield, Christoph Laaber, Tao Yue, and Shaukat Ali. 2025. Faster and better quantum software testing through specification reduction and projective measurements. ACM Transactions on Software Engineering and Methodology 34, 7 (2025), 1–39.arxiv.org/abs/2405.15450
  37. [37]Noah H Oldfield, Christoph Laaber, and Shaukat Ali. 2025. Bloch Vector Assertions for Debugging Quantum Programs. arXiv preprint arXiv:2506.18458 (2025).arxiv.org/abs/2506.18458
  38. [38]Jiaming Ye, Shangzhou Xia, Fuyuan Zhang, Paolo Arcaini, Lei Ma, Jianjun Zhao, and Fuyuki Ishikawa. 2023. QuraTest: Integrating quantum specific features in quantum program testing. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1149–1161.DOI
  39. [39]Avner Bensoussan, Elena Chackarova, Karine Even-Mendoza, Sophie Fortz, Vasileios Klimis, and Mohammad Reza Mousavi. 2026. Live Noise Fingerprinting for Discrepancy Analysis in Quantum Software Engineering. In 2026 IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, 749–754.DOI
  40. [40]Douglas C Montgomery. 2020. Introduction to statistical quality control. John wiley & sons.
  41. [41]Ewan S Page. 1954. Continuous inspection schemes. Biometrika 41, 1/2 (1954), 100–115.DOI
  42. [42]Michele Basseville, Igor V Nikiforov, et al. 1993. Detection of abrupt changes: theory and application. Vol. 104. Prentice hall Englewood Cliffs.
  43. [43]Alan Agresti. 2013. Categorical Data Analysis (3 ed.). John Wiley & Sons, Hoboken, NJ.
  44. [44]Meng Wang, Swamit Tannu, and Prashant J Nair. 2025. Accelerating Simulation of Quantum Circuits under Noise via Computational Reuse. In Proceedings of the 52nd Annual International Symposium on Computer Architecture. 1539–1553.DOI
  45. [45]Thomas Grurl, Jürgen Fuß, and Robert Wille. 2022. Noise-aware quantum circuit simulation with decision diagrams. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 42, 3 (2022), 860–873.
  46. [46]Ang Li, Samuel Stein, Sriram Krishnamoorthy, and James Ang. 2023. Qasmbench: A low-level quantum benchmark suite for nisq evaluation and simulation. ACM Transactions on Quantum Computing 4, 2 (2023), 1–26.DOI
  47. [47]Naoto Sato and Ryota Katsube. 2024. Locating Buggy Segments in Quantum Program Debugging. In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results. 26–31. doi:10.1145/3639476.3639761DOI
  48. [48]Gabriel Joseph Pontillo and Mohammad Reza Mousavi. 2024. Delta Debugging for Property-Based Regression Testing of Quantum Programs. In Proceedings of the 5th ACM/IEEE International Workshop on Quantum Software Engineering. 1–8. doi:10.1145/3643667.3648219
  49. [49]Yuechen Li, Hanyu Pei, Linzhi Huang, Beibei Yin, and Kai-Yuan Cai. 2024. Automatic repair of quantum programs via unitary operation. ACM Transactions on Software Engineering and Methodology 33, 6 (2024), 1–43.
  50. [50]Siwei Tan, Liquang Lu, Debin Xiang, Tianyao Chu, Congliang Lang, Jintao Chen, Xing Hu, and Jianwei Yin. 2025. HornBro: Homotopy-Like Method for Automated Quantum Program Repair. Proc. ACM Softw. Eng. 2, FSE (2025). doi:10.1145/3715751DOI
  51. [51]Jianhua Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Transactions on Information theory 37, 1 (1991), 145–151.DOI
  52. [52]Bradley Efron and Robert J. Tibshirani. 1993. An Introduction to the Bootstrap. Chapman & Hall, New York.
  53. [53]Anne Chao. 1984. Nonparametric Estimation of the Number of Classes in a Population. Scandinavian Journal of Statistics 11, 4 (1984), 265–270.
  54. [54]I. J. Good. 1953. The Population Frequencies of Species and the Estimation of Population Parameters. Biometrika 40, 3–4 (1953), 237–264.DOI
  55. [55]Tudor Gurguia-Tiron, Yousef Hindy, Ryan LaRose, Andrea Mari, and William J. Zeng. 2020. Digital Zero Noise Extrapolation for Quantum Error Mitigation. In 2020 IEEE International Conference on Quantum Computing and Engineering (QCE). IEEE, 306–316.
  56. [56]Pengzhan Zhao, Xiongfei Wu, Zhuo Li, and Jianjun Zhao. 2023. Bugs4Q: A Benchmark of Existing Bugs to Enable Controlled Testing and Debugging Studies for Quantum Programs. Journal of Systems and Software 205 (2023), 111805.DOI
  57. [57]Christopher A. Field and Alan H. Welsh. 2007. Bootstrapping Clustered Data. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 69, 3 (2007), 369–390.DOI
  58. [58]Edwin B Wilson. 1927. Probable inference, the law of succession, and statistical inference. J. Amer. Statist. Assoc. 22, 158 (1927), 209–212.DOI
  59. [59]Ronald A. Fisher. 1922. On the Interpretation of χ² from Contingency Tables, and the Calculation of P. Journal of the Royal Statistical Society 85, 1 (1922), 87–94.DOI
  60. [60]Glenn W. Brier. 1950. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review 78, 1 (1950), 1–3.doi.org/10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
  61. [61]Henry B. Mann and Donald R. Whitney. 1947. On a Test of Whether One of Two Random Variables Is Stochastically Larger than the Other. The Annals of Mathematical Statistics 18, 1 (1947), 50–60.DOI
  62. [62]Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological bulletin 114, 3 (1993), 494.DOI
  63. [63]Jeanine Romano, Jeffrey D. Kromrey, Jesse Coraggio, and Jeff Skowronek. 2006. Appropriate Statistics for Ordinal Level Data: Should We Really Be Using t-test and Cohen’s d for Evaluating Group Differences on the NSSE and Other Surveys? In Annual Meeting of the Florida Association of Institutional Research.
  64. [64]Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 (1960), 37–46.DOI
  65. [65]J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174.pubmed.ncbi.nlm.nih.gov/843571
  66. [66]Ilya M. Sobol'. 2001. Global Sensitivity Indices for Nonlinear Mathematical Models and Their Monte Carlo Estimates. Mathematics and Computers in Simulation 55, 1–3 (2001), 271–280.DOI
  67. [67]Andrea Saltelli, Marco Ratto, Terry Andres, Francesca Campolongo, Jessica Cariboni, Debora Gatelli, Michaela Saisana, and Stefano Tarantola. 2008. Global Sensitivity Analysis: The Primer. John Wiley & Sons, Chichester.
  68. [68]Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in Software Engineering. Springer, Berlin, Heidelberg.

Paper details

vlnc:W_npj8bap8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents