betaFeedback

WorksComputer Science

Causality-Based Fault Localization

Abstract

We introduce Causality-Based Fault Localization (CBFL), a novel approach to fault localization grounded in the theory of actual causality. We use the notion of causal responsibility to formalize and define the contribution of a given location in the source code under the context of a failing test. We also utilize a distance metric to prioritize minimal fixes and penalize broken tests, leading to more interpretable and fixable fault localization. We implemented CBFL in a tool, SoftReX, and our evaluation results demonstrate that SoftReX outperforms state-of-the-art Mutation-Based Fault Localization (MBFL) tools based on Mean Reciprocal Rank (MRR) and Top-1 by 16% on both metrics, when using the same set of mutants. We also propose a sophisticated mutation generation strategy based on the structure and representation of the program to improve the accuracy of our ranked list, which outperforms state-of-the-art methods by 2.6x in terms of required mutants evaluated while remaining competitive in performance.

Causality-Based Fault Localization

Introduction

Motivation. Software debugging remains one of the most challenging and costly activities in software engineering [40]. As software systems continue to grow in scale and complexity, and with the increase in LLM-generated commits, the hard problem of identifying the root causes of failures is only becoming increasingly difficult. Finding the error is often harder than fixing: establishing where errors originate often dominates the effort required to correct the problem [22]. Fault Localization (FL), the task of identifying the program elements responsible for a failure, is widely recognized as one of the most time-consuming and expensive stages of debugging and has consequently been the subject of extensive research for more than two decades. Despite decades of research and substantial progress in automated FL, accurately identifying the source of an observed failure remains a major challenge. Recent studies continue to highlight the effort associated with FL and the limitations of existing techniques, motivating ongoing work on more effective debugging support [40, 22].

State of the art. Among automated FL techniques, Spectrum-Based Fault Localization (SBFL) has emerged as one of the most widely studied and influential paradigms [1]. SBFL analyzes program spectra—information about which program elements are executed during a particular run, such as statement or branch coverage collected from passing and failing executions—and assigns each element a suspiciousness score based on its statistical association with observed failures [19, 4]. Numerous ranking metrics have been proposed, including Tarantula [19], Muse [28], and DStar [39], with empirical studies demonstrating that SBFL substantially reduces the search space developers must inspect during debugging. More recently, Mutation-Based Fault Localization (MBFL) has extended this approach by replacing observation with active intervention, generating mutants and analyzing their impact on test outcomes to infer fault relevance. Techniques such as Metallaxis [32] have shown that mutation information can improve localization accuracy over traditional spectrum-based approaches, while frameworks such as Major [20] enabled efficient large-scale mutation analysis.

Limitations of SBFL and MBFL. Despite their practical effectiveness, both MBFL and SBFL remain fundamentally associational: program elements are ranked according to correlations with failures across executions rather than their causal contribution to the observed failure. But the question being asked is causal: what and where is the cause of the failure? As a result of its associational nature, highly ranked program elements that are not necessarily causes of the failure remain under investigation. In addition to this wasted effort, current methods struggle to distinguish between cases that only causal methods can address: over-determination (multiple faults each sufficient to produce a failure), pre-emption (one fault causing an observed failure before another fault occurs), and error masking (the effects of a fault being hidden or corrected before reaching the output). This raises a fundamental question: to what extent can FL benefit from causal reasoning, rather than statistical association alone?

Contribution. In this paper, we formalize FL as a causal analysis problem. We use the theory of actual causality [18] to construct Structural Causal Models (SCMs) reflecting the underlying structure of the program. In our formulation, the states of individual code elements act as variables, and mutations serve as targeted interventions. By observing the test suite executions of a program under these interventions, we are able to quantify the causal contribution or degree of responsibility [9] of any code element toward a test failure. Crucially, our SCM allows us to navigate masked and overdetermined faults by evaluating interventions, while still admitting efficient computation and more accuracy.

To make this theoretical framework computationally tractable on real-world software, we introduce SoftRex, a causality-based FL technique. Rather than relying on exhaustive, brute force mutation generation, SoftRex allows for some sampling of mutants or employs a hierarchical search over the Concrete Syntax Tree (CST) to dynamically iterate over potential minimal sets of code elements to mutate–contingency sets. To ensure practical accuracy, SoftReX regularizes its base causal responsibility scores by penalizing interventions that break originally passing tests, passing-to-failing (P2F) penalization, and weighing them against the syntactic or semantic distance of the mutation.

We evaluate SoftReX on the Defects4J benchmark [21], a standard benchmark for reproducible bugs, against 8 state-of-the-art MBFL techniques. Our empirical results demonstrate that shifting from correlation to causal responsibility fundamentally improves FL accuracy. Specifically, SoftReX improves Mean Reciprocal Rank (MRR) by 16% and successfully locates 31 more bugs in the Top-10 position compared to traditional baselines like BLMu. In summary, this paper makes the following contributions:

  1. Theoretical Framework: We formalize CBFL – an extension of MBFL as an actual causality problem.

  2. Heuristics: We introduce heuristics to compute the approximate degree of responsibility for faults using distance weights and P2F penalization.

  3. Algorithm: We present an algorithm and its implementation in a tool SoftReX, a scalable, adaptive, hierarchical, iterative search-based algorithm that generates mutants to approximate actual causality.

  4. Empirical Evaluation: We provide a comprehensive evaluation of SoftReX on real-world Java bugs demonstrating the superiority of our approach over traditional statistical suspiciousness metrics.

Availability. All SoftReX’s code and experimental results are available at [6].

Background

In this section, we provide a brief, simplified overview of actual causality and MBFL. For the full exposition of actual causality, see Appendix § 11.

Actual Causality

Actual causality [16, 18] provides a formal framework for reasoning about the causes of observed outcomes. It differs from type causality, formalized by Pearl [33], which deduces general causal relationships between variables from existing data and is forward-looking. In contrast, the study of actual causality is backward-looking, “what caused a certain test to fail?”, which is more suitable for analyzing faults in software. The framework describes the world in terms of variables and their values. A causal model CMCM consists of a set of variables V\mathcal{V} and a set of structural equations that capture dependencies between these variables. Many applications of actual causality [30, 24] use depth-2 causal models, in which a set of independent input variables determines an outcome variable, see Figure 1. We adopt this approach as well. In these models, causal relations are examined through interventions that assign alternative values to selected input variables and assess their effects on the outcome.

A representation of a depth-2 causal model

Figure 1. A representation of a depth-2 causal model used to identify the effect of input variables s1,s2,…,sns_1,s_2,\ldots,s_n on outcome OO. In our setting, each sis_i is a code element in the program and OO an enumeration of execution outcomes.

Informally, an actual cause of an observed outcome in a depth-2 causal model is a minimal subset of input variables whose values, if changed, alter the outcome. Following [17], we refer to parts of causes as causes. We use the definition of degree of responsibility, first introduced in [9], which quantifies causal influence. Informally, the responsibility of an input variable XX for outcome OO is 1/∣X⃗∣1 / |\vec{X}|, where X⃗\vec{X} is a smallest actual cause containing XX. If there is no actual cause containing XX, the responsibility of XX is 0. We refer to this subset, X⃗−{X}\vec{X} - \{X\}, as the contingency of XX and denote its size as kk, that is, k=∣X⃗−{X}∣k = |\vec{X} - \{X\}|.

Mutation-Based Fault Localization (MBFL)

MBFL relies on creating changes to the source code—known as mutants—which can be analyzed by observing the changes in test execution behavior to expose potential faults in the program. Formally, let a buggy target program P\mathcal{P} be represented by a finite set of discrete code elements: S={s1,s2,…,sn}S = \{s_1,s_2,\ldots,s_n\}, where each code element has a location and can be of any size. The granularity of these elements, ranging from variable expressions and abstract syntax tree nodes to full statements, defines the spatial resolution of the fault localization process, as the final responsibility and suspiciousness scores are attributed directly to these elements in SS.

To identify the program elements responsible for the failure, MBFL techniques systematically apply mutations to the elements to produce a set of program variants or mutants denoted by MM. We formalize this process via a mapping function ℓ:M→2S\ell: M \to2^S, which maps each mutant m∈Mm \in M to the non-empty subset of code elements it modifies. This formalization allows for mutation strategies where a single mutant alters multiple locations simultaneously, i.e., ∣ℓ(m)∣≥1|\ell(m)| \ge1.

Per code element s∈Ss \in S, we define Ms⊂MM_s \subset M as the set of all mutants modifying ss:

Ms={m∈M∣s∈ℓ(m)}.M_s = \{m \in M \mid s \in\ell(m)\}.

By evaluating the outcomes of test executions of the mutants in MsM_s, we can systematically measure the impact of ss on the overall behavior of P\mathcal{P}.

The execution behavior of P\mathcal{P} and its mutants is evaluated against a test suite T=F∪PT = F \cup P, where FF is the set of failing tests and PP is the set of passing tests.

For any test t∈Tt \in T and program variant P′P^{\prime}, let out(P′,t)∈{pass,fail,exc,timeout}out(P^{\prime},t) \in\{pass, fail, exc, timeout\} denote the execution outcome. This allows us to observe discrete shifts in test execution trajectories, such as transitioning from the original failure to a passing state (passpass), a process timeout (timeouttimeout), or a new exception.

To evaluate the suspiciousness of a code element, traditional MBFL constructs a statistical ranking based on execution outcomes. Following the established notion by Wang et al. [37], for a given m∈Msm \in M_s applied at location ss, we define the following transitions:

  • f(m)f(m): the number of tests that originally failed but now pass on mm.

  • p(m)p(m): the number of tests that originally passed but now failed on mm.

  • nf(m)nf(m): the number of tests originally failed that not affected by mm.

  • np(m)np(m): the number of tests originally passed that are unaffected by mm.

  • f′(m)f^{\prime}(m): the number of tests that originally failed and now the test output changes after applying mm.

Using these frequencies, traditional MBFL formulas compute a statistical suspiciousness score, sus(m)sus(m), for each mutant. Table 1 presents the formulas of the established baseline methods evaluated in this study, where α\alpha adjusts the weighting of the terms. BLMu incorporates μ\mu that denotes the number of tests that change the outcome of the failing test on any mutant over the number of test that transition from passing to failing.

MethodFormulaMethodFormula
Barinel [1, 3]1−p(m)p(m)+f(m)1-\frac{p(m)}{p(m)+f(m)}BLMu [37]α⋅f(m)+βf′(m)−μ⋅p(m)\alpha\cdot f(m)+\beta f^{\prime}(m)-\mu\cdot p(m)
DStar2 [39]f(m)p(m)+nf(m)\frac{f(m)}{p(m)+nf(m)}Muse [28]f(m)−α⋅p(m)f(m)-\alpha\cdot p(m)
Jaccard [29]f(m)2f(m)+nf(m)\frac{f(m)}{2f(m)+nf(m)}Op2 [29]f(m)−p(m)p(m)+f(m)+1\frac{f(m)-p(m)}{p(m)+f(m)+1}
Metallaxis [32]f′(m)∣F∣(f′(m)+p(m))\frac{f^{\prime}(m)}{\sqrt{\lvert F\rvert(f^{\prime}(m)+p(m))}}Tarantula [19]f(m)f(m)+nf(m)f(m)f(m)+nf(m)+p(m)p(m)+np(m)\frac{\frac{f(m)}{f(m)+nf(m)}}{\frac{f(m)}{f(m)+nf(m)}+\frac{p(m)}{p(m)+np(m)}}

Table 1. Suspiciousness formulas for fault localization methods (for m∈Msm \in M_s).

As multiple mutants can be generated at a single location ss, MBFL methods aggregate them by calculating the average over all mutant scores. The final suspiciousness ranking for code element ss is therefore computed as:

Susp⁡(s)=1∣Ms∣∑m∈Mssus(m)\operatorname{Susp}(s)=\frac{1}{\lvert M_s\rvert}\sum_{m\in M_s} sus(m)

Traditional MBFL methods fail to distinguish between a mutant that leaves a failure completely unchanged and one that completely changes the execution path or the type of failure.

Theoretical Foundation of Causality-Based Fault Localization

To move beyond correlation, we introduce the use of actual causality for FL. The goal is to determine whether a code element ss is an actual cause of a test f∈Ff \in F failing. Specifically, we define a causal model CMfCM_f for a failed test ff, where its input variables correspond to the code elements within the coverage of ff. In this formulation, interventions on these variables are equivalent to mutations. In addition, CMfCM_f includes an additional variable, OO, corresponding to the execution outcome of the—possibly intervened/mutated—program. This results in a depth-2 causal model (Figure 1), where each code element is causally independent, meaning that each element can be mutated without altering others.

Informally, a code element ss is a cause of a test failure f∈Ff \in F if an intervention on ss, applied via some mutation mm, alters the test’s execution behavior. However, in real-world settings, single interventions rarely suffice; faults are often masked by secondary errors and may require simultaneous modifications across the program to be revealed. The benefits of an explicit causal model allow evaluating under contingencies—sets of other code elements that are intervened upon to reveal the causal effect of ss.

Unlike traditional approaches that only look for a complete repair (fail →\to pass), we consider any deviation from the original failure trajectory to be potential evidence of a causal link. We formalize this using an indicator of outcome for a failing test f∈Ff \in F and mutation mm:

R(m,f)=1{out(P,f)=fail∧out(m,f)≠out(P,f)}.R(m,f)=1\{out(P,f)=\mathrm{fail}\land out(m,f)\ne out(P,f)\}.

Thus, for a code element ss to have a responsibility greater than 0 for the failed test ff, our causal model requires that there exists some mutant m∈Mm \in M such that s∈ℓ(m)s \in\ell(m) and R(m,f)=1R(m,f)=1.

Practical Causal Relaxation for Software

While formally rigorous, computing exact actual causes across arbitrary programs is computationally intractable due to the size of the space of multi-point code interventions. To make causal FL practical, we approximate the degree of responsibility, as defined in Definition 2, by combining the following: encoding behavior shifts, representative mutant sampling, side-effect penalization, and interventional cost.

Outcome Sensitive Responsibility. Rather than exhaustively searching for a minimal contingency size kk, we estimate the responsibility of an intervention for a failing test f∈Ff \in F by considering the outcome change. Specifically, in the case that there is a partial shift in the failing test’s trajectory, such as a modified stack trace, a different exception type, or a process timeout. This signals that mm perturbs the causal chain of the bug even if it does not resolve it: out(m,f)≠out(P,f)∧out(m,f)≠passout(m,f)\ne out(P,f)\land out(m,f)\ne\mathrm{pass}, where mm is the mutation applied at the location ss. We scale the responsibility so that in these scenarios the code element ss can be at most 0.5. For example, in a scenario where a mutant mm alters the execution behavior of a failing test without achieving full repair, but does so by requiring a minimum of k=3k = 3 concurrent interventions, the assigned responsibility scales accordingly. We evaluate the resulting responsibility at ss using the upper estimate:

Respf(s)=0.50.5+k=0.50.5+3=0.53.5≈0.143.\mathrm{Resp}_{f}(s) = \frac{0.5}{0.5 + k} = \frac{0.5}{0.5 + 3} = \frac{0.5}{3.5} \approx0.143.

However, if k=0k = 0 and there is partial shift, we assign responsibility of 0.5 to ss. As kk must denote the smallest number of additional interventions needed to induce this shift, this example illustrates how the framework, by default, scales to penalize highly entangled, multi-point causal dependencies.

Mutant Sampling. Evaluating every possible multi-code intervention across a whole program is computationally intractable.1 Instead, we estimate the responsibility of a code element s∈Ss \in S over an evaluated sample of representative mutations, MsM_s. Our causal responsibility score is agnostic to how MsM_s is generated; it remains fully computable given an arbitrary set of mutants, including the exhaustive single-point mutations generated by tools like Major [20].

While sampling sacrifices exact theoretical completeness, it yields a practical approximation of the causal responsibility of ss. If no mutant in MsM_s alters the failure trajectory, we cannot definitively rule out that a more complex, undiscovered multi-point intervention might do so. However, if any mutant MsM_s successfully triggers a behavior shift, the causal link exists and the element’s true responsibility is guaranteed to be at least the value computed.

Therefore, the accuracy of the approximation to the actual causal responsibility relies on the diversity of the mutation operator applied. By utilizing a comprehensive suite of operators—such as statement deletions, literal value replacement, and relational and conditional operator replacements—we maximize the interventional coverage of the code element, ensuring a high-quality, practical approximation of the true causal responsibility of ss.

Pass-to-Fail (P2F) Penalization. A fault-repairing intervention should not break non-faulty program behavior captured by the test suite. If an intervention mm alters the outcomes of originally passing tests PP, its responsibility is scaled by a validity ratio:

V(m)=1.0−∣{t∈P∣out(m,t)≠pass}∣∣P∣(1)V(m) = 1.0 - \frac{\left|\{t \in P \mid out(m,t) \ne pass\}\right|}{|P|} \tag*{(1)}

This penalizes destructive mutations that induce widespread incorrect behavior. Incorporating passing-test behavior is a foundational axiom in traditional MBFL. MUSE, for example, subtracts the number of mutants that have a passing-to-failing test transitions for a given code element, whereas Metallaxis incorporates this into its denominator. Our framework explicitly isolates this as a credibility bound on a causal claim, using only the relevant mutant rather than all mutants for code element ss. This addresses how to deal with common mutants that satisfy execution change of the failing test, however change behavior of existing logic, preserving the distinction between true root-cause repair and coincidental anomaly.

Distance Weighting. To penalize complex, large or disproportionate interventions, we regularize the responsibility with an intervention weight dist⁡(m,s)\operatorname{dist}(m,s). Massive alterations may change the test outcome simply by removing an execution path rather than exposing the specific fault. The distance-based weighting function captures either the syntactical structural change or semantic embedding distance (e.g., CodeBERT representational shift) between the original code at location ss and mutant mm. For the structural distance heuristic, the cost scales according to the size of the target code element ss, see § 6.2 for specific details. Alternatively, the semantic distance quantifies the representational shift between the mutated and original code embeddings, penalizing semantically destructive intervention regardless of the physical size.

The final ranked score for a code element ss for a failing test ff is the maximum weighted responsibility across its mutants:

Rank⁡f∗(s)=max⁡m∈Ms(Resp⁡f(m)⋅V(m)1+λ⋅dist⁡(s,m)),(2)\operatorname{Rank}^{*}_{f}(s) = \max_{m \in M_s} \left(\frac{\operatorname{Resp}_{f}(m)\cdot V(m)}{1+\lambda\cdot\operatorname{dist}(s,m)}\right), \tag*{(2)}

where λ>0\lambda> 0 is a hyperparameter balancing causal responsibility against intervention complexity. Each failing test represents a distinct causal scenario; we aggregate these individual location rankings across all the failing tests to produce a global FL score, enabling direct evaluation against standard correlation based MBFL techniques.

SoftReX Algorithm

In this section, we introduce the SoftReX search algorithm, which operationalizes our theoretical framework by dynamically generating causal interventions. Computing the exact minimal contingency set kk across an entire program is an O(2∣S∣)O(2^{|S|}) combinatorial problem. Further, the space of syntactical mutations for any single code element can be extremely large; an exact exhaustive search is computationally unfeasible. To address this, SoftReX employs a hierarchical, iterative, adaptive syntax-tree-based search that isolates the smallest blocks of the program without requiring an exhaustive search.

Algorithm 1 accepts a failing test ff, which drives the search alongside a broader test suite TT. First, SoftReX records the baseline test outcomes BB and restricts the search to source files covered by the driving test ff, using per-test coverage report. The covered files are then searched independently: for each file, SoftReX parses the source into a CST and partitions the file into coarse groups of mutable CST blocks (G\mathcal{G}). For each group, we generate batches of mutated variations (up to nn trials per group) by replacing the targeted block with a type-compatible simple value, apply⁡(P,m)\operatorname{apply}(\mathcal{P},m)—for example, substituting a block whose return type is INT with 1, 0, or MAX_INT. If a mutation yields an uncompilable

Algorithm 1 SoftRex Search

Require: Program PP, failing test ff, test suite TT, trials per group nn, min region size ss Ensure: Mutant/test outcome matrix AA.

  1. B←[ ]B \leftarrow[\ ] ▷ Initialize baseline results to empty

  2. ∀t∈T:B[t]←out(P,t)\forall t \in T : B[t] \leftarrow out(P,t) ▷ Record baseline test outcomes

  3. G←InitialGroups⁡(P.CST⁡)\mathcal{G} \leftarrow\operatorname{InitialGroups}(P.\operatorname{CST}) ▷ Initial groups of mutable CST blocks

  4. A←[ ]A \leftarrow[\ ]

  5. for G∈GG \in\mathcal{G} do

  6. M←GenBatches⁡(G,n)\mathcal{M} \leftarrow\operatorname{GenBatches}(G,n)

  7. for m∈Mm \in\mathcal{M} do

  8. P′←apply⁡(P,m)P' \leftarrow\operatorname{apply}(P,m) ▷ Replace block with type-compatible simple value

  9. if P′P' cannot be evaluated then

  10. continue

  11. for t∈Tt \in T do

  12. ot←out(P′,t)o_t \leftarrow out(P',t) ▷ Evaluate each test in the suite on the mutant

  13. Am,t←(ot=B[t])A_{m,t} \leftarrow(o_t = B[t]) ▷ Record comparison in outcome matrix

  14. if out(P′,f)=pass⁡∨out(P′,f)≠B[f]out(P',f) = \operatorname{pass} \lor out(P',f) \ne B[f] then

  15. if (∣G∣>s)(|G| > s) then

  16. G←G∪Subdivide⁡(G)G \leftarrow G \cup\operatorname{Subdivide}(G) ▷ Recursively subdivide block group

  17. return AA

or malformed variant that cannot be evaluated, it is discarded (line 9). This filtering step ensures that only valid, executable interventions are considered in our analysis. For each valid mutant, we evaluate the entire test suite to identify changes in either the success of the individual tests or the specific errors they return (lines 11–13). If the outcome of the driving test ff changes, it indicates that the mutated region warrants further exploration. Consequently, provided the block size exceeds the minimum threshold (∣G∣>s|G| > s), we subdivide that block and recursively continue the search (line 16). Ultimately, these evaluations construct the outcome matrix AA. This matrix directly provides the execution states necessary to compute both the passing-test validity penalty and the base responsibility defined in §3.

Evaluation Methodology

To assess our causality-based framework’s effectiveness and the SoftRex search algorithm’s practical viability, we conduct a comprehensive empirical evaluation. Our study is designed to answer the following Research Questions (RQs):

RQ1. How does SoftRex’s causality-based ranking formula compare against state-of-the-art MBFL formulas when using a fixed mutant sample?

RQ2. To what extent do (a) penalty regularization and (b) structural/semantic distance regularization individually improve causal ranking accuracy?

RQ3. To what extent does guiding the mutation generation improve FL accuracy and efficiency over standard mutation strategies?

Experiment Design

We structure our evaluation into two distinct experiments. Experiment 1. We isolate the effectiveness of SoftReX’s causal ranking formula from its heuristic mutation search algorithm. We replace SoftReX’s search with Major [20] to systematically generate single-line mutants via mutation analysis across the target program by applying our SoftReX’s scoring to this static mutant matrix. This allows for a direct comparison against existing MBFL formulas. Experiment 2 focuses on evaluating the whole pipeline, integrating our hierarchical search (Algorithm 1). This experiment assesses whether the targeted sampling of mutations can achieve effective fault localization without the immense cost of exhaustive mutation testing.

Configurations and Metrics

Configurations. To answer our RQs, we evaluate 8 state-of-the-art (SoA) baselines and 6 configurations of SoftReX, and additional hyperparameter variations for SoftReX with λ∈[0.05,0.1,0.25,0.5,1,2,3]\lambda\in[0.05, 0.1, 0.25, 0.5, 1, 2, 3]. For the 8 baseline methods, we re-implemented the standard MBFL scoring formulas as outlined in Table 1. These include established MBFL approaches, namely Metallaxis [32], BLMu [37] and Muse [28], alongside SBFL formulas mapped to different mutation outcomes: Tarantula [19], Barinel [1, 3], Dstar2 [39], Jaccard [40], and Op2 [29]. We avoid direct comparison to SBFL techniques to ensure a strictly fair comparison. To rigorously assess our causal framework, we conduct an ablation study across 6 SoftReX configurations for each experiment. We evaluate the foundational causal formula as SoftReXBase that directly aligns the rank to the raw degree of responsibility, Respf(m)Resp_f(m). To isolate the individual contributions of regularization mechanisms, we evaluate the P2F penalization independently as SoftReXPenalty and the distance weighting independently as SoftReXDist. We use two distance measures: one based on structural cost changes SoftReXDist and another that incorporates CodeBERT’s [14] cosine embedding distances, SoftReXDistSem. The structural distance depends on the element type of ss and whether the element was deleted, replaced or modified, with further details provided in the appendix. For Experiment 2, where mutations can span multiple lines, we compute this weighting by averaging the distance weights across all affected elements ss. Finally, the complete SoftReX is represented by the configurations where both P2F penalization and distance weighting mechanisms are fully active (SoftReXPenalty+DistSem, SoftReXPenalty+Dist).

Evaluation Metrics. We compare the rankings produced by SoftReX against SoA (Table 1) by measuring the Mean Average Rank (MAR), Mean Reciprocal Rank (MRR), and Top-1, Top-3, Top-5, and Top-10 accuracy. MAR represents the average rank of all bugs per approach. Top-NN defines the number of bugs in which the faulty code elements are located at top NN position of the ranking given by the approach.

Benchmark and Environment

We utilize Defects4J [21] (commit 8c16da8), a collection of reproducible bugs in Java programs for which the location of the real fault and the corresponding test triggering the error before the fix are known. Experiment 1 evaluates 827 bugs across 17 projects: JacksonXML (JXml), Cli, JacksonCore (JCore), Compress, Jsoup, Codec, JacksonDatabind (JDB), Math, Collections, Csv, JxPath, Gson, Lang, Chart, Time, Closure and Mockito; 27 bugs were excluded due to Out-of-Memory or Timeout exceptions. Experiment 2 is performed on a representative subset (JacksonXml, Gson Csv, Math) chosen via uniform random sampling. We selected projects spanning different scales and failure densities: Math provides a large, failure dense stress test; CSV is more compact, but contains several non-trivial bugs. Gson and JacksonXml provide sparser serialization focused targets. Full project and execution details are in the Appendix.

Experiments were executed on anonymized for submission HPC cluster within a 64-bit Linux environment managed by the Sun Grid Engine (SGE) scheduler. Jobs were submitted to the generic CPU queue, configured to utilize 4 cores and 32 GB of RAM. The max timeout for each job was 48 hours, with no experiment reaching it, and 600 seconds per task.

Results and Evaluation

RQ1 - Baseline Comparison

We evaluate SoftReX against 8 established baseline methods over a static mutant matrix. Figure 2 presents aggregated global FL performance across all projects. Our SoftReX with the specific configuration SoftReXPenalty+DistSem, where λ=0.05\lambda= 0.05 outperforms all baselines across every ranking threshold. We successfully isolate 103 bugs at exactly Top-1, outperforming the strongest MBFL method, BLMu, with 89 bugs. Further, the performance gap widens as NN increases, with SoftReX finding 301 bugs in the Top-10 compared to BLMu’s 270, demonstrating a higher overall reliability and less noise. SoftReX achieves a global MRR of 0.349, establishing a clear margin over traditional methods. BLMu and Barinel have MRRs of 0.301 and 0.222, respectively, while Op2 severely underperforms with 0.113. The relative improvement in MRR (≈15%\approx15\%) demonstrates that the causal-based method provides a less noisy ranking. While our SoftReX noticeably improves Top-N and MRR, we observe a marginal trade-off in MAR, where BLMu scores slightly better (45.58 vs. SoftReX’s 48.13).

Top-N bugs located with different thresholds across all projects

Figure 2. Top-NN bugs located with different thresholds aggregated over all projects evaluated (N∈{1,3,5,10}N \in\{1,3,5,10\}).

Project-Level Analysis. We further break down the performance across the 17 Defects4j projects. Compared with the best performing baseline MBFL technique per project, SoftReX improves average MRR by +0.0670. Examining the number of bugs located at various ranking thresholds (Top-N) reveals SoftReX’s consistent superiority across diverse project domains. Naturally, projects with a larger number of evaluated bugs, such as Math and Closure, yield higher Top-N counts across all evaluated methods. However, SoftReX has an advantage over SoA, irrespective of project scales. We observe only a few isolated cases where a traditional MBFL approach outperforms SoftReX on MRR: Cli and JacksonXml.

To verify the robustness of the final eq. (2), we conducted a sensitivity analysis on the weight parameter λ\lambda detailed in the Appendix. The results indicate SoftReX’s stability across a wide range of values while avoiding overfitting, whereas BLMu’s hyperparameter drastically change the results entirely. We used λ=0.05\lambda= 0.05 for our evaluation as it empirically provides the optimal balance.

RQ2 - Ablation Study of softrex

To understand how individual components contribute to SoftReX’s performance, we evaluate 4 structural ablations of the proposed eq. (2): the base responsibility (SoftReXBase), the integration of the P2F penalization, the distance weighting, and the fully combined formula as shown in eq. (2).

Base Causal Responsibility. The base responsibility configuration achieves an average MRR of 0.269, beating the other approaches and placing it just below BLMu. Beyond MRR, the causal formulation shows its strength: SoftReXBase located 284 bugs in the Top-10 and achieved an overall MAR of 45.903. In both these metrics, SoftReXBase outperforms every traditional baseline evaluated, except BLMu on MAR. This shows that computing pure causal responsibility is very effective at isolating the general faulty neighborhood.

P2F penalization. Applying the penalization (SoftReXPenalty) yields a significant performance leap, increasing MRR to 0.335. This supports our hypothesis that penalizing mutants that break previously passing tests is crucial for filtering out pseudo-fixes that satisfy the change in observed behavior by reducing the program’s overall logic. This filtering effect is very visible in the Top-N metric as Top-1 increases by 50% (from 50 to 75 bugs) and Top-10 peaks at 311 bugs over all configurations. This shows the constraint pushes the true root cause to the top of the ranking.

Distance Weighting. Utilizing both heuristic and semantic distance weighting independently also strengthens the base responsibility calculation. Specifically,

CodeBERT semantic distance pushes Top-1 accuracy to 93. The heuristic distance does not improve to the same magnitude as semantic distance, yet has the lowest (best) MAR out of the evaluated methods. The base responsibility often assigns the exact same responsibility scores to multiple statements, and an effective tie-breaking mechanism is required. The distance measurements resolve ties by favoring minimal shifts over massive changes, achieving high precision even without considering any outcomes from passing tests.

Combined: softrex. The fully combined configuration achieves best Top-1 and MRR performance across our evaluations, successfully locating 103 bugs at Top-1 and reaching an MRR of 0.349 at λ=0.05\lambda= 0.05. This shows that our weighting mechanisms are orthogonal and work well together. The validity penalty acts as a filter to eliminate any fake fixes, while the distance weighting ensures the true root cause is pushed to the top of the ranking. Furthermore, substituting the basic syntactic cost heuristic (SoftRexPenalty+Dist) with deep semantic embeddings (SoftRexPenalty+DistSem) yields a 27% relative increase in Top-1 accuracy (from 81 to 103 bugs), confirming that modeling semantic intent is far superior to simple structural change cost when resolving ties in the ranking.

Project-Level Analysis. Figure 3 breaks down the ablation study at the project level with respect to MRR. softrex base configuration consistently establishes the lower bound for MRR across nearly all 17 projects. Applying the P2F penalization or the semantic distance weighting isolation produces significant but project-dependent improvements. For example, the P2F penalization yields notable gains over Mockito, Compress, Cli and JDB, and semantic distance independently dominates on projects like JxPath and JXml. The fully combined configuration shows the best stability; this is especially visible in projects like Gson and Csv. Lastly, the project-wise breakdown confirms that semantic embeddings consistently provide substantially stronger, more reliable ranking signals than basic syntactic structural costs across the majority of the Defects4J dataset.

Ablation study bar chart showing Mean Reciprocal Rank (MRR) for each Defects4J project, with configurations softrex base, + penalty, + dist heuristic, + pen + dist heuristic, + dist semantic, and + pen + dist semantic

Figure 3. Ablation Study: Mean Reciprocal Rank (MRR) for each project evaluated.

RQ3 - Guided Mutation Generation Effectiveness

While Experiment 1 establishes the theoretical superiority of our ranking formula under a static mutant pool, Experiment 2 evaluates the end-to-end SoftReX pipeline utilizing our hierarchical search algorithm. We evaluate each approach over a uniform random sample of bug IDs from 4 representative projects.

Table 2 compares performance on matched bug sets. Unlike exhaustive mutation baselines, SoftReX employs a targeted search, reducing evaluated mutants by ≈2.6×\approx2.6\times while preserving competitive localization accuracy. For instance, in larger codebases like Math, SoftReX exhibits a minor trade-off in Top-1 accuracy (locating 7 bugs to Major’s 8), yet it matches or surpasses the exhaustive baseline across higher thresholds (Top-3, Top-5, and Top-10) and achieves a notably higher MRR on both Math and Csv. Conversely, smaller or sparsely populated target sets (such as Gson and JacksonXml) reflect the inherent challenges of dynamic sampling under strict search boundaries. Overall, these findings demonstrate that guided hierarchical mutation generation provides an effective balance between FL precision and computational feasibility. Additionally, across all tested bugs, however, SoftReX search matches or beats all other MBFL approaches in Top-N accuracy, while using on average 2.6×2.6\times fewer mutants evaluated than Major.

ProjectExperimentTop-1Top-3Top-5Top-10MARMRRMutants
CsvMajor (1)235612.410.2942743
SoftReX (2)345582.500.347958
GsonMajor (1)13555.420.482372
SoftReX (2)000087.000.018191
JacksonXmlMajor (1)002326.50.1441423
SoftReX (2)0011514.330.0381003
MathMajor (1)81415178.180.22619565
SoftReX (2)715171981.670.2477126

Table 2. Project-level comparison on matched bug subsets from Major (Experiment 1.), and SoftReX search (Experiment 2.).

Discussion

Our causal responsibility is approximated using sampled mutants rather than exhaustive interventions. Our validity ratio and distance weighting introduce modeling assumptions; we mitigate this through a comprehensive ablation study and a hyperparameter sensitivity analysis on λ\lambda, which confirmed performance stability. Finally, we applied standard FL metrics (MAR, MRR, and Top-N), though alternative metrics could yield different perspectives. Focusing solely on program structure while omitting data flow captures static architectural or control-flow dependencies, but it limits the model’s ability to represent runtime values, state propagation, or actual execution. Integrating both inside the SCM has been left to future work. We evaluated multiple Defects4J projects, supporting generalization across Java codebases. Although our underlying SoftReX’s is language independent, its generalization to other programming languages remains to be validated in the future.

Future Work. Beyond the controlled benchmark evaluation presented here, we will study SoftReX in the wild and on other benchmarks. Systems can include substantial setup logic, low-level implementation code, external libraries, and complex code, creating many program elements that correlate with a failure without necessarily causing it. These settings provide a useful test of whether CBFL addresses better FL. Additionally, we will extend this work and investigate how causal-model granularity affects fault ranking, comparing source-code lines with alternative program representations such as Abstract Syntax Tree (AST) nodes and program dependence graph nodes. In particular, a depth-2 causal model is likely a simplification of the actual causal structure in a software object. Combining SBFL and causal discovery could lead to significant gains in FL accuracy and localization efficiency. Finally, future work will investigate the generalization of SoftReX beyond Java programs, extending to other widely used programming languages such as Python, C/C++, and JavaScript.

Related Work

Fault Localization (FL) is essential to automated debugging, serving to identify program elements responsible for failures [42, 36, 37]. Effective FL reduces the search space for manual debugging and is used for automated program repair.

Spectrum-based Fault Localization (SBFL). SBFL remains the baseline for automated debugging due to its efficiency and ease of deployment. Implementations such as the SFLKit workbench [35] utilize execution traces to rank program elements based on their correlation with failure, while later approaches incorporate input data tracking to refine these rankings [11]. Correlation-based metrics such as Ochiai and Tarantula [19, 2] struggle in the presence of multiple faults due to masking effects [1], and their effectiveness is further reduced by coincidental correctness [27]. Recent techniques attempt to isolate failure-inducing inputs [25], but correlation-based approaches have a limited ability to distinguish real causes from mere correlations.

Mutation-based Fault Localization (MBFL). MBFL refines SBFL by introducing active interventions, namely, injecting code changes (mutants) and observing their effect on the test outcomes. This provides stronger evidence of fault relevance than passive observation, and frameworks such as Major [20] have standardized mutant generation for Java programs. The evidential gains obtained by MBFL come at the cost of substantial computational overhead and the equivalent-mutant problem.

Wang et al. [37] conduct a systematic investigation into different MBFL formulae and introduce a new method: BLMu. Although BLMu performs well on Defects4J, it requires a number of hyper-parameters, in particular two weighting coefficients on passing and failing tests respectively, α\alpha and β\beta.

SoftReX addresses a fundamental limitation of traditional MBFL approaches by replacing heuristic suspiciousness calculations with a formal notion of causal responsibility, weighted by semantic distance. Consequently, SoftReX provides not only evidence that a mutation is relevant to a failure, but an explanation of the degree to which that mutation is responsible for causing it.

Causality in Software Testing. Baah et al. [7] were the first to formulate fault localization as a causal inference problem, although their work primarily adjusted execution spectra. More recent approaches employ explicit counterfactual reasoning. CounterFault [34] models faults by asking whether alternative variable values would have changed the outcome of execution. Further work integrated value- and predicate-based causal inference to isolate the causes of failure [26]. While these approaches represent a significant step forward, they predominantly address data causality (the state of program variables) rather than structural causality (the program logic itself). Clark et al. [10] argue for using causal inference to test scientific software. In short, existing techniques either provide scalable statistical rankings without causal guarantees or causes in terms of values of the variables (and not the program code).

Semantic Distance and Program Repair. FL is often a precursor to Automated Program Repair (APR), although the objectives differ. FL ranks suspicious elements, whereas APR seeks locations amenable to patch generation. Yang et al. [42] show that highly ranked FL locations are not necessarily the most repairable, motivating metrics that account for repair cost and complexity. There is a substantial body of work on repair techniques based on semantic distance and similarity [31, 23, 38, 15, 13, 5, 12]. Despite that, semantic distance has not been used for FL, and a potential connection of causal responsibility to semantic significance of the corresponding code is unexplored.

Large Language Models in Localization. LLMs have introduced new paradigms for FL [41]. Unfortunately, they remain susceptible to non-determinism and hallucination [8], hence reinforcing the need for deterministic, formally grounded FL techniques that provide interpretable and reproducible results.

Statistical approaches provide scalable localization but lack causal guarantees; causal approaches reason about execution states rather than source-code responsibility; mutation-based approaches perform interventions but rely on heuristic ranking; and semantic-distance techniques quantify code similarity without contributing to causal attribution. Consequently, existing FL methods cannot formally determine which source-code changes are responsible for a failure, nor the degree of that responsibility. This gap motivates the development of causally grounded FL techniques that identify not only which code is a cause of a failure, but also how strongly particular source-code changes contribute. SoftReX closes this gap by unifying intervention-based mutation analysis, actual causality, causal responsibility, and semantic distance within a single FL framework.

Conclusions

We introduced a new paradigm, CBFL, for fault localization, based on actual causality. This paradigm, CBFL, addresses the shortcomings of the existing statistical approaches by computing actual causes of faults and ranking them according to their degree of causal influence on the fault. The problem of computing causes is intractable; hence we presented an algorithm and a number of heuristics for practical approximate computation of the ranking. We implemented this approach in a tool softRex, a causality-based fault-localization technique that uses responsibility to rank fault locations causally. We evaluated softRex on Defects4J using a common set of mutants and showed that it outperforms state-of-the-art MBFL techniques by 15% in MRR and 16% in Top-1. We further showed that guided mutation generation improves localization efficiency—in terms of mutants required to be evaluated—by 2.6×2.6 \times over standard mutation strategies. These results show the potential of causal analysis for more accurate and actionable FL.

Actual Causality

The framework of actual causality [16, 18] describes the world in terms of variables and their values. A causal model MM consists of a set of variables V\mathcal{V} and a set of structural equations that capture dependencies between these variables. Some salient features of causal models can be represented by a causal network: a directed acyclic graph whose nodes are the variables of MM and whose directed edges capture the existence and direction of causal dependence.

A context u⃗\vec{u} is an assignment to the variables of MM that have no incoming edges in the causal network—that is, their values do not depend on the values of other variables. u⃗\vec{u} determines, through the equations of MM, the values of all variables in V\mathcal{V}. Together, a causal model MM and a context u⃗\vec{u}, which specifies the particular situation being analyzed, form a causal setting (M,u⃗)(M,\vec{u}). We write (M,u⃗)⊨φ(M,\vec{u}) \models\varphi if a Boolean formula φ\varphi holds in the causal setting (M,u⃗)(M,\vec{u}).

Causal analysis is performed through interventions. An intervention X⃗←x⃗\vec{X} \leftarrow\vec{x} assigns values x⃗\vec{x} to a subset X⃗⊆V\vec{X} \subseteq\mathcal{V}, replacing the corresponding equations with these assignments. We write (M,u⃗)⊨[X⃗←x⃗]φ(M,\vec{u}) \models[\vec{X} \leftarrow\vec{x}]\varphi if φ\varphi holds after this intervention. Interventions, therefore, allow us to reason about the effect of changing selected variables on the outcome. With that, we can define actual causes:

Definition 1 (Actual cause [17, 18]). X⃗=x⃗\vec{X} = \vec{x}, a subset of V\mathcal{V} and their valuation, is an actual cause of φ\varphi in the causal setting (M,u⃗)(M,\vec{u}) if the following three conditions hold:

AC1. (M,u⃗)⊨(X⃗=x⃗)(M,\vec{u}) \models(\vec{X} = \vec{x}) and (M,u⃗)⊨φ(M,\vec{u}) \models\varphi.

AC2. There is an alternative setting x⃗′\vec{x}' of the variables in X⃗\vec{X}, a (possibly empty) set W⃗\vec{W} of variables in V−X⃗\mathcal{V} - \vec{X}, and a setting w⃗\vec{w} of the variables in W⃗\vec{W} such that (M,u⃗)⊨W⃗=w⃗(M,\vec{u}) \models\vec{W} = \vec{w} and (M,u⃗)[X⃗←x⃗′,W⃗←w⃗]⊨¬φ(M,\vec{u})[\vec{X} \leftarrow\vec{x}', \vec{W} \leftarrow\vec{w}] \models\neg\varphi.

AC3. X⃗\vec{X} is minimal, i.e., there is no strict subset X⃗′\vec{X}' of X⃗\vec{X} such that X⃗′=x⃗′′\vec{X}' = \vec{x}'' can replace X⃗=x⃗′\vec{X} = \vec{x}' in AC2, where x⃗′′\vec{x}'' is the restriction of x⃗′\vec{x}' to the variables in X⃗′\vec{X}'.

Essentially, AC1 states that the cause, X⃗=x⃗\vec{X} = \vec{x}, and the outcome, φ\varphi indeed happened in context u⃗\vec{u} (this is actual causality). AC2 says that for X⃗=x⃗\vec{X} = \vec{x} to be a cause of φ\varphi, there must be an assignment x⃗′\vec{x}' for X⃗\vec{X} such that if we clamp X⃗=x⃗′\vec{X} = \vec{x}' and W⃗=w⃗\vec{W} = \vec{w}, φ\varphi no longer holds. AC3 states that there should be no unnecessary variables in X⃗=x⃗\vec{X} = \vec{x}. A variable XX in an actual cause X⃗\vec{X} is called a part of a cause. Following [17], we often refer to parts of causes as causes.

In this paper, we use the definition of degree of responsibility, first introduced in [9], which quantifies the measure of causal influence.

Definition 2 (Responsibility [9]). The degree of responsibility of an endogenous variable XX for φ\varphi in (M,u⃗)(M,\vec{u}) is defined as 1/(∣X⃗∣+∣W⃗∣)1/(|\vec{X}| + |\vec{W}|), where X⃗\vec{X} is an actual cause for φ\varphi containing XX that minimizes ∣X⃗∣+∣W⃗∣|\vec{X}| + |\vec{W}|, and W⃗\vec{W} is its respective contingency. If there is no actual cause containing XX, its degree of responsibility is 0.

Experimental Details

Table 3 details the heuristic distance weights applied during Experiment 2 to approximate structural mutation distances based on AST node types.

Mutation Type of mmTarget Element ss TypeStructural Distance
DeletionFunction1.0
DeletionAssignment / Control Flow / Statements0.8
ReplacementConditional0.6
ReplacementArithmetic / Relational0.4
ModificationLiteral Value / Variable0.2

Table 3. Heuristic structural distance function.

Detailed Results

Table 4 summarizes the experimental metadata across the 17 projects, detailing the average number of mutants generated, test suite sizes, and the number of failing and passing test within the test suite for both experiments. This

Exp.ProjectMutantsTotal TestsFailedPassed
1Chart570.251819.793.751816.04
Cli175.28263.561.69261.87
Closure520.196910.363.246907.12
Codec707.83447.062.39444.67
Collections222.4613149.891.3613148.54
Compress490.43434.431.53432.89
Csv230.50197.811.50196.31
Gson1249.00986.201.93984.27
JacksonCore1904.23356.192.04354.15
JacksonDatabind240.051619.641.181618.45
JacksonXml237.17152.172.00150.17
Jsoup563.15453.601.56452.04
JxPath286.45346.681.68345.00
Lang810.161814.181.971812.21
Math847.512508.551.672506.88
Mockito32.741137.312.971134.34
Time508.363916.842.643914.20
2Csv87.09193.551.55192.00
Gson95.50852.001.00851.00
JacksonXml167.17152.172.00150.17
Math134.452505.301.532503.77

Table 4. Average test suite execution and mutant generation statistics across evaluated Defects4J projects for Experiment 1 (exhaustive) and Experiment 2 (heuristic search). This is with respect to the versions ran for each experiment.

Table 5 and Table 8 present the complete global fault localization metrics and ablation study data. Furthermore, Table 8 (Sensitivity Analysis) demonstrates the robustness of the λ\lambda parameter, confirming that SoftReX’s semantic distance weighting remains stable across a wide range of values without overfitting.

Exp.MethodTop-1Top-3Top-5Top-10MARMRR
Major (1)softRex10320324430148.1320.349
Major (1)BLMu8916821127045.5780.304
Major (1)Barinel7112214017681.5750.222
Major (1)Metallaxis409312218561.6520.181
Major (1)MUSE6812013918768.6090.224
Major (1)Tarantula6711313217181.7050.213
Major (1)DStar26312314017681.5710.218
Major (1)jaccard5911313117081.7510.204
Major (1)Op223445611776.2180.113
softRex (2)softRex10192325135.6480.223
softRex (2)BLμ714192488.8400.163
softRex (2)Barinel59101199.4880.103
softRex (2)Metallaxis29142192.6600.112
softRex (2)Muse59101198.0250.103
softRex (2)Tarantula69101199.4940.108
softRex (2)DStar236101299.5000.083
softRex (2)Jaccard69101199.5060.107
softRex (2)Op25891099.2280.097

Table 5. Global fault localization effectiveness across both experimental setups.

λ\lambdaTop-1Top-3Top-5Top-10MARMRRλ\lambdaTop-1Top-3Top-5Top-10MARMRR
0.0510320324430148.1320.3490.058119624830948.3060.331
0.110220424729948.1710.3470.18119624830948.3080.330
0.2510020224529748.2910.3430.258119624830948.3080.330
0.59919924229748.4030.3410.58119624830948.3080.330
19819624329648.5660.33918119624830948.3080.330
29819624129348.6760.33927819824630748.4760.327
39719624129448.6940.33837519624430648.8990.321

Table 8. (a) softRexPenalty+DistSem; (b) softRexPenalty+Dist

To ensure complete transparency regarding project-level variance, we provide absolute Top-NN accuracy counts (N∈{1,3,5,10}N \in\{1, 3, 5, 10\}) for all baseline methods across every evaluated project (Table 9–Table 12).

ConfigurationTop-1Top-3Top-5Top-10MARMRR
softRexBase5016421528445.9030.269
softRexPenalty7519525931147.6050.335
softRexDist (Heuristic)6517422628945.2600.297
softRexDistSem (CodeBERT)9319323729248.7840.331
softRexPenalty+Dist8119624830948.3060.331
softRexPenalty+DistSem10320324430148.1320.349

Table 6. Ablation Study - Experiment 1.

ConfigurationTop-1Top-3Top-5Top-10MARMRR
softRexBase35711207.6230.084
softRexPenalty6131721133.4570.160
softRexDist (Heuristic)35711207.6230.084
softRexDistSem (CodeBERT)10172127198.5740.208
softRexPenalty+Dist6131721133.4570.160
softRexPenalty+DistSem10202426135.6480.223

Table 7. Ablation Study - Experiment 2.

ProjectSoftReXBarinelMUSEDStar2TarantulaMetallaxisJaccardOp2BLMu
Chart522320212
Cli343136245
Closure1400000000
Codec222130112
Collections212212213
Compress655553526
Csv411110101
Gson111111103
JacksonCore211110101
JacksonDatabind12898969214
JacksonXml000001001
Jsoup13111171189415
JxPath631411111
Lang71111111128111
Math1715131212412416
Mockito433334204
Time533332324

Table 9. Project-level comparison of Top-1 accuracy (absolute number of bugs located).

ProjectSoftReXBarinelMUSEDStar2TarantulaMetallaxisJaccardOp2BLMu
Chart545441444
Cli116775115411
Closure3700000010
Codec445340434
Collections433332315
Compress1411101210610314
Csv544432314
Gson311114114
JacksonCore444534417
JacksonDatabind23161515151216424
JacksonXml000002002
Jsoup20161515151513621
JxPath1065654518
Lang18141415141113219
Math28232223211022826
Mockito855556529
Time955553526

Table 10. Project-level comparison of Top-3 accuracy (absolute number of bugs located).

ProjectSoftReXBarinelMUSEDStar2TarantulaMetallaxisJaccardOp2BLMu
Chart645441446
Cli127787137413
Closure4901000030
Codec556442435
Collections533332315
Compress1613131313613417
Csv744432326
Gson511114114
JacksonCore554545418
JacksonDatabind27181818171417628
JacksonXml200003003
Jsoup21181819172116929
JxPath10777757110
Lang20151515151714224
Math322726262413251034
Mockito128688108312
Time1055554527

Table 11. Project-level comparison of Top-5 accuracy (absolute number of bugs located).

ProjectSoftReXBarinelMUSEDStar2TarantulaMetallaxisJaccardOp2BLMu
Chart765654548
Cli13111111101510917
Closure55050010152
Codec867663667
Collections533332316
Compress20141614141114719
Csv855555528
Gson532336325
JacksonCore7555585110
JacksonDatabind412521252526251237
JacksonXml300003003
Jsoup282028201926191737
JxPath11878878111
Lang29201920192318831
Math343135313024301844
Mockito151413141414141214
Time12555575211

Table 12. Project-level comparison of Top-10 accuracy (absolute number of bugs located).

References

  1. [1]Abreu, R., Zoeteweij, P., van Gemund, A.J.C.: On the accuracy of spectrum-based fault localization. In: TAICPART-MUTATION ’07. p. 89–98. IEEE, USA (2007)
  2. [2]Abreu, R., Zoeteweij, P., Gemund, A.J.C.v.: An evaluation of similarity coefficients for software fault localization. In: PRDC ’06. p. 39–46. IEEE, USA (2006)
  3. [3]Abreu, R., Zoeteweij, P., Gemund, A.J.C.v.: Spectrum-based multiple fault localization. In: ASE ’09. p. 88–99. IEEE, USA (2009)
  4. [4]Abreu, R., Zoeteweij, P., Golsteijn, R., van Gemund, A.J.C.: A practical evaluation of spectrum-based fault localization. Journal of Systems and Software (2009)
  5. [5]Alon, U., Zilberstein, M., Levy, O., Yahav, E.: code2vec: learning distributed representations of code. Proc. ACM Program. Lang. 3(POPL) (Jan 2019)arxiv.org/abs/1803.09473
  6. [6]Anonymous: Artifact of softrex (Oct 2026), https://doi.org/10.5281/zenodo.23354390
  7. [7]Baah, G.K., Podgurski, A., Harrold, M.J.: Causal inference for statistical fault localization. In: ISSTA ’10. pp. 73–84. ACM, NY, USA (Jul 2010)DOI
  8. [8]Chen, F., Yang, G., Lee, C., Jie, Z., Chen, Y.: Measure twice, locate once: Mitigating hallucinations in llm-based agents for repository-scale fault localization. ACM Trans. Softw. Eng. Methodol. (Jul 2026)DOI
  9. [9]Chockler, H., Halpern, J.Y.: Responsibility and blame: A structural-model approach. Journal of Artificial Intelligence Research 22, 93–115 (2004)
  10. [10]Clark, A.G., Foster, M., Prifling, B., Walkinshaw, N., Hierons, R.M., Schmidt, V., Turner, R.D.: Testing causality in scientific modelling software. ACM Trans. Softw. Eng. Methodol. 33(1) (Nov 2023)arxiv.org/abs/2209.00357
  11. [11]DeMott, J.D., Enbody, R.J., Punch, W.F.: Systematic bug finding and fault localization enhanced with input data tracking. Computers & Security 32, 130–157 (Feb 2013)DOI
  12. [12]Eberlein, M., Smytzek, M., Steinhöfel, D., Grunske, L., Zeller, A.: Semantic debugging. In: ESEC/FSE 2023. p. 438–449. ACM, NY, USA (2023)
  13. [13]Falleri, J.R., Morandat, F., Blanc, X., Martinez, M., Monperrus, M.: Fine-grained and accurate source code differencing. In: ASE ’14. p. 313–324. ACM, NY, USA (2014)
  14. [14]Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al.: Codebert: A pre-trained model for programming and natural languages. In: Findings of the association for computational linguistics: EMNLP 2020 (2020)arxiv.org/abs/2002.08155
  15. [15]Feser, J., Dillig, I., Solar-Lezama, A.: Metric program synthesis. arXiv (2022)arxiv.org/abs/2206.06164
  16. [16]Halpern, J.Y., Pearl, J.: Causes and explanations: a structural-model approach. Part I: causes. British Journal for Philosophy of Science 56(4), 843–887 (2005)
  17. [17]Halpern, J.Y.: A modification of the Halpern–Pearl definition of causality. In: Proceedings of the International Joint Conferences on Artificial Intelligence. pp. 3022–3033. AAAI Press (2015)DOI
  18. [18]Halpern, J.Y.: Actual causality. the MIT press, Cambridge (Mass.) (2016)
  19. [19]Jones, J.A., Harrold, M.J.: Empirical evaluation of the tarantula automatic fault-localization technique. In: ASE ’05. p. 273–282. ACM, NY, USA (2005)DOI
  20. [20]Just, R.: The major mutation framework: efficient and scalable mutation analysis for java. In: ISSTA 2014. p. 433–436. ACM, NY, USA (2014)DOI
  21. [21]Just, R., Jalali, D., Ernst, M.D.: Defects4J: a database of existing faults to enable controlled testing studies for java programs. In: ISSTA 2014. p. 437–440. ACM, NY, USA (2014)DOI
  22. [22]Kang, S., An, G., Yoo, S.: A quantitative and qualitative evaluation of llm-based explainable fault localization. Proc. of the ACM on Software Engineering (2024)DOI
  23. [23]Ke, Y., Stolee, K.T., Le Goues, C., Brun, Y.: Repairing programs with semantic code search (t). In: ASE ’15. pp. 295–306. IEEE (2015)DOI
  24. [24]Kelly, D.A., Chockler, H., Kroening, D., Blake, N., Ramaswamy, A., Navaratnarajah, M., Shivakumar, A.: You only explain once (2023), https://arxiv.org/abs/2311.14081
  25. [25]Kirschmer, L., Soremekun, E., Zeller, A.: Isolating faults in failure-inducing inputs (2 2021). https://doi.org/10.18420/SE2021_20
  26. [26]Küçük, Y., Henderson, T.A., Podgurski, A.: Improving fault localization by integrating value and predicate based causal inference techniques. In: ICSE ’21. pp. 649–660. IEEE (2021)arxiv.org/abs/2102.06292
  27. [27]Masri, W., Assi, R.A.: Prevalence of coincidental correctness and mitigation of its impact on fault localization. ACM Trans. Softw. Eng. Methodol. 23(1), 1–28 (2014)DOI
  28. [28]Moon, S., Kim, Y., Kim, M., Yoo, S.: Ask the Mutants: Mutating Faulty Programs for Fault Localization. In: ICST ’14. pp. 153–162 (Mar 2014), iSSN: 2159-4848DOI
  29. [29]Naish, L., Lee, H.J., Ramamoohanarao, K.: A model for spectra-based software diagnosis. ACM Trans. Softw. Eng. Methodol. 20(3) (Aug 2011)DOI
  30. [30]Navaratnarajah, M., Martin, S.A., Kelly, D.A., Blake, N., Chockler, H.: 3d rex: Causal explanations in 3d neuroimaging classification. arXiv preprint arXiv:2502.12181 (2025)arxiv.org/abs/2502.12181
  31. [31]Nguyen, H.D.T., Qi, D., Roychoudhury, A., Chandra, S.: Semfix: Program repair via semantic analysis. In: 2013 35th International Conference on Software Engineering (ICSE). pp. 772–781. IEEE (2013)
  32. [32]Papadakis, M., Le Traon, Y.: Metallaxis-FL: Mutation-based fault localization. Softw. Test. Verif. Reliab. 25(5–7), 605–628 (Aug 2015)
  33. [33]Pearl, J.: Causality. Cambridge University Press (2009)
  34. [34]Podgurski, A., Küçük, Y.: CounterFault: Value-based fault localization by modeling and predicting counterfactual outcomes. In: ICSME ’20. pp. 382–393. IEEE (2020)DOI
  35. [35]Smytzek, M., Zeller, A.: Sfkit: A workbench for statistical fault localization. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 1701–1705 (2022)DOI
  36. [36]Soremekun, E., Kirschner, L., Böhme, M., Zeller, A.: Locating faults with program slicing: an empirical analysis. Empirical Software Engineering 26(3), 51 (2021)arxiv.org/abs/2101.03008
  37. [37]Wang, B., Wei, J., Chen, M., Chen, C., Lin, Y., Zhang, J.M.: A Systematic Exploration of Mutation-Based Fault Localization Formulae. Softw. Test. Verif. Reliab. 35(1), e1905 (Jan 2025)DOI
  38. [38]Wang, T., Roychoudhury, A.: Automated path generation for software fault localization. In: Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering. pp. 347–351 (2005)DOI
  39. [39]Wong, W.E., Debroy, V., Gao, R., Li, Y.: The dstar method for effective software fault localization. IEEE Transactions on Reliability 63(1), 290–308 (2014)DOI
  40. [40]Wong, W.E., Gao, R., Li, Y., Abreu, R., Wotawa, F.: A survey on software fault localization. IEEE Transactions on Software Engineering 42(8), 707–740 (2016)DOI
  41. [41]Wu, Y., Li, Z., Zhang, J.M., Papadakis, M., Harman, M., Liu, Y.: Large Language Models in Fault Localization. arXiv (Oct 2023)
  42. [42]Yang, D., Qi, Y., Mao, X., Lei, Y.: Evaluating the usage of fault localization in automated program repair: an empirical study. Frontiers of Computer Science 15(1), 151202 (2021)DOI

Paper details

vlnc:W_ya7c5sbg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents