Noisy Issues, Noisy Results: Mislabeling in Information Retrieval-based Bug Localization

Introduction

Issue reports are critical artifacts in software engineering projects, serving as documented records of bugs, enhancement requests, and other development tasks [1]. These reports are generally managed through Issue Tracking Systems (ITSs), where users and contributors can submit, track, and resolve issues. Among various types of issue reports, bug reports are particularly important as unresolved bugs can compromise the reliability, performance and security of software systems [2]. However, modern software projects often receive a large volume of bug reports, making it challenging for developers to manage them. For example, the Eclipse project received over 4,400 bug reports in 2009 alone [3]. Addressing each report requires developers to reproduce the bug, identify its root cause and locate the relevant source code entities, a process that is both time-consuming and labor-intensive [4].

To assist developers in this process, various Information Retrieval-based Bug Localization (IRBL) techniques have been proposed. These approaches formulate bug localization as a document retrieval task, where the bug report serves as a query and the source files as documents. Given a bug report, these techniques generate a ranked list of source files based on their likelihood of containing the fault [5]. The goal is to rank the buggy files as high as possible. To calculate the similarity between bug reports and the source files, existing techniques use approaches ranging from traditional models such as the Vector Space Model (VSM) [6] and Topic Modeling [7, 8] to advanced methods based on Deep Learning [5, 9, 10, 11, 12, 13, 14] and Large Language Models (LLMs) [15, 16, 17].

To evaluate the effectiveness of these techniques, several benchmark datasets have been constructed from ITSs. These datasets generally collect fixed issues labeled "bugs", assuming that these labels are correct [18]. However, a study by Herzig et al. found that approximately 33.8% of the bug reports are misclassified (i.e., these issues do not actually describe bugs) [19]. This finding raises concerns about the quality of IRBL benchmark datasets and the validity of performance evaluations conducted using them. Since IRBL techniques are designed for bug reports, the inclusion of non-bug issues may introduce bias into their evaluation [1].

Despite the importance of this problem, only a few studies have examined the impact of mislabeled issues on IRBL performance. Kochhar et al. first examined this problem using VSM-based bug localization [1, 2], while Widyasari et al. subsequently investigated two traditional approaches, VSM and BugLocator [20]. However, as summarized in Table I, these studies have several limitations. First, they focused on traditional IRBL techniques, leaving the impact of mislabeling on more advanced deep learning and LLM-based approaches (e.g., DreamLoc [5], GenLoc [17]) largely unexplored. Second, their analyses were conducted on small datasets and did not extend to established IRBL benchmark datasets (e.g., Ye et al. dataset [21], Bench4BL [18]). Third, their evaluations were limited to intra-technique comparisons (i.e., analyzing how the performance of individual techniques changes between noisy and clean datasets) without performing inter-technique comparisons, which assess how mislabeled issues affect the comparative performance of different techniques. Finally, they overlooked the partially overlapping nature of the data in which the clean dataset is a subset of the original noisy dataset. Consequently, they applied statistical tests that assume data independence, which may lead to misleading conclusions. As a result, the validity of prior findings remains uncertain.

Comparison Criteria	Kochhar et al.	Widyasari et al.	Our Study
Established IRBL Benchmarks	✗	✗	✓
DL/LLM Techniques	✗	✗	✓
Intra-Technique Comparison	✓	✓	✓
Inter-Technique Comparison	✗	✗	✓
Partial-Overlap Aware Analysis	✗	✗	✓

In this study, we address these limitations through a comprehensive empirical investigation of the prevalence of mislabeled issues and their impact on IRBL performance. We extend prior work by examining established IRBL benchmarks and modern localization techniques, evaluating the effects of mislabeling on both individual and comparative performance, and employing statistical methods that account for dataset overlap. In particular, we investigate the following two research questions:

RQ1. How prevalent are mislabeled issues in IRBL benchmark datasets?

RQ2. How do mislabeled issues impact the performance of IRBL techniques?

To address RQ1 (Section II), we investigate five IRBL benchmark datasets, including three widely used datasets (Zhou et al. [3], Ye et al. [21] and Bench4BL [18]) and two recently introduced datasets (GHRB (GitHub Recent Bugs) [22] and BeetleBox [23]). Together, these benchmarks contain more than 55,000 issue reports spanning over a decade of IRBL research. To identify mislabeled issues at scale, we develop GenTag, an LLM-based issue classifier that achieves a weighted F1-score of 90% while maintaining an affordable cost of $0.0003 per issue. Applying GenTag to these datasets reveals that mislabeled reports are widespread, accounting for 29.01% to 44.01% of the issues across the five benchmarks, with an average of 34.82%, highlighting persistent limitations in benchmark label quality.

To address RQ2 (Section III), we evaluate six representative IRBL techniques, namely VSM [1], BugLocator [3], BLUiR [24], BRTracer [25], DreamLoc [5], and GenLoc [17]. These techniques span traditional retrieval, heuristic-enhanced, deep learning-based, and LLM-based approaches. Using the widely adopted Ye et al. dataset [21], we investigate the effects of mislabeled issues in both evaluation and historical/training data. We conduct intra-technique comparisons to assess how mislabeling affects individual techniques and inter-technique comparisons to examine its influence on their relative performance. Our statistical analyses account for the overlapping nature of the evaluated datasets. Our results show that removing mislabeled issues from the evaluation data consistently reduces the measured performance of all six techniques, with statistically significant differences. In contrast, cleaning historical/training data has varying effects depending on the technique. Furthermore, although mislabeling does not change the overall ordering of techniques, it influences the magnitude of performance differences between them. These findings highlight the importance of dataset quality for fair and accurate evaluation in IRBL research.

In summary, this paper makes the following contributions:

• Large-scale analysis of issue mislabeling: We investigate the prevalence of issue mislabeling across five IRBL benchmarks comprising over 55,000 issue reports using a validated LLM-based classifier.

• Impact on intra-technique comparisons: We systematically evaluate the effects of mislabeling in evaluation and historical/training data on the performance of six IRBL techniques, spanning traditional, deep learning-based, and LLM-based approaches.

• Impact on inter-technique comparisons: We examine how issue mislabeling affects the relative performance and statistical comparisons among IRBL techniques.

• Replication package: We release a replication package [26] to facilitate reproducibility and support future research.

Prevalence of Mislabeled Issues in IRBL

Identification of Mislabeled Issues

To assess the prevalence of mislabeled issues in IRBL datasets, an automated classifier is required to distinguish between bug and non-bug reports, as manual labeling is time-consuming [27] and does not scale to large benchmarks (e.g., over 20,000 reports) [19]. Existing approaches based on machine learning, deep learning or BERT-like models (e.g., RoBERTa, SetFit) [28]–[33] rely on manually verified data for training or fine-tuning, which is often unavailable and therefore limits their applicability. Recently, LLM-based classifiers [34], [35] have been introduced to reduce this dependency. However, these models are generally provided with context about only a limited set of categories (e.g., BUG, FEATURE, DOCUMENTATION, QUESTION) and prompted to choose among them. When the true label is absent from the prompt, the LLM is forced to map the issue to one of the available labels. For example, issues related to REFACTORING or BUILD_SYSTEM could be misclassified as BUG, which could inflate bug counts and bias subsequent analyses. This limitation underscores the need for incorporating a broader taxonomy into LLM prompts to enable more accurate and fine-grained issue classification.

Motivated by this limitation, we design GenTag (Generative AI-based Issue Tagger), an LLM-powered classifier designed to perform fine-grained issue classification ¹. GenTag leverages the taxonomy and guidelines proposed by Herzig et al., which define 13 comprehensive issue categories such as BUG, RFE (Request for Enhancement), and REFACTORING [19]. Based on these guidelines, GenTag instructs an LLM to classify each issue report into one of the predefined issue categories.

The classification process begins with a prompt that tells an LLM to act as an expert software maintainer, as role-playing strategies have been shown to improve the reasoning skills of LLMs [36]. The prompt then includes the 13 issue categories along with their descriptions. However, since the original Herzig et al. guidelines were designed for human annotators, we refined the guidelines through prompt engineering to make them more suitable for LLM-based classification. We applied two types of modifications:

• Text Reduction: To reduce token usage and simplify the prompt, we condensed complex definitions and removed redundant ones. Specifically, we shortened the description of

the IMPROVEMENT category by consolidating Herzig et al.’s detailed nine cases into four broader subcategories, focusing on feature enhancements and error handling, exceptions, performance or compatibility improvements. Additionally, we removed definitions for categories such as BACKPORT (backports), CLEANUP (code cleanups), SPEC (changes to specification rather than documentation or code) and TASK (general development tasks), as we found that these definitions provided little added value for LLM classification. These adjustments reduced token consumption and inference cost without degrading classification performance.

• Text Enhancement: To improve the precision of the classification, we clarify certain category definitions. For BUILD_SYSTEM, we added explicit examples of configuration files (e.g., pom.xml, build.xml) to help identify them better. We also revised the vague definition of the TEST category, which originally covered all "issues regarding test cases". Our refined definition explicitly included issues related to test coverage, testing frameworks, test case initialization, as well as typos or improvements in test methods or test logic. Additionally, we clarified to the LLM that test-related changes addressing broken or malfunctioning tests intended to restore correct behavior should be classified as bugs, not as test issues. This distinction helped to ensure that bug-fixing changes in the test code were correctly labeled as BUG rather than mistakenly categorized under TEST. These enhancements provided clearer guidance and improved the LLM’s classification performance.

The prompt then tells the LLM to classify an issue in the most appropriate category by providing the title and description of the issue. The prompt additionally asks the LLM to provide a brief explanation of its decision, as previous work has shown that self-explaining prompts significantly improve LLM comprehension [37]. The complete prompt used by GenTag is provided in our replication package [26].

While GenTag performs fine-grained classification into 13 categories, our ultimate goal is to analyze the impact of mislabeled issues on bug localization. Since bug localization techniques operate on a binary distinction between bug-related and non-bug-related reports, we map the fine-grained categories into two classes: BUG and NOT BUG. The BUG category remains unchanged, whereas all remaining categories (e.g. RFE, REFACTORING, and DOCUMENTATION) are labeled as NOT BUG. This abstraction allows us to preserve the benefits of fine-grained annotation while aligning the final labels with the requirements of bug localization evaluation.

Since LLM outputs are inherently non-deterministic, we repeat the classification process three times and select the final label through majority voting. This approach mitigates potential variability in LLM predictions and ensures more stable results [38].

Validation of the Identification Process

Baseline Techniques, Dataset and Evaluation Metrics: To evaluate the performance of GenTag, we compared it against a state-of-the-art LLM-based approach, called GPT-like model, that classifies an issue into categories such as bug, feature, documentation and question under a zero-shot setting [34]. In addition, we evaluated a variant of GenTag (GenTag-Herzig) that directly follows the original issue classification guidelines proposed by Herzig et al. to assess the impact of prompt engineering.

For evaluation, we used Herzig et al.’s dataset [19], which consists of 5,591 hand-labeled issue reports from three projects (HTTPClient, Jackrabbit and Lucene-Java). Classification performance was measured using standard metrics, namely precision, recall and F1-score. Following prior work [32], we report both class-specific scores for the BUG and NOT BUG categories as well as weighted averages to account for class imbalance. The weighted metrics are computed using the proportion of instances belonging to each class.

Experiment Setup: GenTag uses GPT-4o mini by OpenAI due to its cost-effectiveness ($0.15 per 1M input tokens and $0.60 per 1M output tokens) [39]. We set the model’s temperature to 0 to reduce variability and facilitate reproducibility [40]. For comparison, we adapted the replication package of the GPT-like model to support Herzig’s dataset and used the same underlying LLM as GenTag to ensure a fair comparison. Following the original study [34], we executed the GPT-like model three times to account for the non-deterministic nature of LLMs and reported the average results.

Result: Table II compares the performance of GenTag against two baselines: the GPT-like model [34], a state-of-the-art LLM-based issue classifier, and GenTag-Herzig, a variant of GenTag that adheres strictly to Herzig et al.’s original classification guidelines. Since GenTag serves as a supporting tool rather than the primary focus of this paper, we do not present the detailed results of each individual classification trial in the manuscript due to space limitations. The results of all three GenTag trials, together with the agreement analysis across trials, are available in the replication package.

Approach	Bug	Bug	Bug	Not-bug	Not-bug	Not-bug	Weighted	Weighted	Weighted
	prec	rec	F1	prec	rec	F1	prec	rec	F1
GPT-like model	0.61	0.98	0.76	0.99	0.67	0.80	0.86	0.78	0.78
GenTag-Herzig	0.91	0.65	0.76	0.84	0.96	0.90	0.86	0.85	0.85
GenTag	0.83	0.88	0.86	0.94	0.90	0.92	0.90	0.90	0.90

Results show that GenTag obtains the highest F1-score for both bug (F1 = 0.86) and not-bug (F1 = 0.92) issues as well as the highest weighted metrics (precision = 0.90, recall = 0.90 and F1 = 0.90). In contrast, the GPT-like model baseline achieves exceptionally high precision on the not-bug class (0.99), but this comes at the expense of recall (0.67), resulting in a lower F1-score (0.80). Similarly, GenTag-Herzig attains high precision on the bug class (0.91), yet its lower recall (0.65) reduces overall effectiveness (0.76). These findings highlight the importance of maintaining balanced precision and recall across classes, a balance that GenTag successfully achieves without compromising either metric.

A sample case is shown in Figure 1, which describes a simple spelling mistake (“Trasaction” vs. “Transaction”). Since this issue does not cause runtime or memory errors or require semantic changes to fix faulty functionality, it is categorized as REFACTORING during manual labeling. In this case, GenTag correctly classified the issue as refactoring, whereas the GPT-like model baseline misclassified it as a bug due to its restricted label space which does not include a REFACTORING category. This example shows how GenTag’s fine-grained taxonomy helps prevent issues from being mislabeled as bugs, thereby yielding more accurate classifications for downstream IRBL studies.

In addition to achieving strong performance, GenTag is highly cost-effective. Running it on 5,591 issues from Herzig et al.’s dataset costs only $1.56 (approximately $0.0003 per issue), making it feasible to apply to large-scale IRBL datasets.

Analysis of IRBL Benchmark Datasets

To assess the prevalence of mislabeled issues, we applied GenTag to five established IRBL datasets. These include three widely used datasets: Zhou et al. [3], containing 3,479 bug reports from four open-source Java projects; Ye et al. [21], comprising over 22,000 bug reports collected from six large open-source Java projects; and Bench4BL [18], containing 10,019 bug reports from 51 open-source Java projects. We also included two recently introduced datasets: GHRB (GitHub Recent Bugs) [22], containing 131 recent bug reports from 16 Java projects; and BeetleBox [23], a large multi-language dataset comprising 19,146 bug reports from 29 open-source projects across five programming languages (Java, C++, Python, Go and JavaScript) $^{2}$.

To further validate GenTag, we manually labeled a stratified random sample of 382 issue reports drawn from these five datasets. The sample size corresponds to a 95% confidence level and a 5% margin of error, and the number of sampled issues from each dataset was proportional to its share of the combined corpus. Two authors independently categorized these issues based on Herzig et al.’s guidelines [19] and resolved discrepancies through discussion to reach a consensus. Inter-rater reliability, measured by Krippendorff’s Alpha, was 0.842, which indicates strong agreement among raters [41]. The consensus labels were then compared with GenTag’s predictions to assess its performance.

Table III presents the aggregated performance of GenTag on the manually labeled issues. Dataset-specific precision, recall and F1-scores are available in the replication package. GenTag achieves a weighted F1-score of 0.93, demonstrating strong generalization beyond its original evaluation dataset. These results confirm that GenTag can be reliably applied to IRBL datasets.

Class	Precision	Recall	F1-Score	Support
Bug	0.96	0.93	0.95	262
Not Bug	0.85	0.93	0.89	120
Weighted	0.93	0.93	0.93	382

Prevalence of Mislabeled Issues

Table IV reveals a consistent presence of mislabeled issues across all IRBL datasets. Across the five datasets, the proportion of non-bug reports ranges from 29.01% to 44.01%, with an average of 34.82%, indicating that roughly one-third of the issues labeled as bugs are actually non-bug reports. This finding aligns with the prior study by Herzig et al. [19], which reported a similar proportion of mislabeled issues; however, their analysis focused on several open-source projects rather than IRBL benchmark datasets. Our analysis extends this observation by showing that IRBL datasets proposed both before and after the study of Herzig et al. are affected by mislabeled issues. Notably, even recently introduced benchmarks such as GHRB and BeetleBox contain a substantial proportion of non-bug reports. This pattern across datasets spanning more than a decade highlights that the assumption of clean ground truth in IRBL datasets does not always hold, and such noise has the potential to skew the evaluation of localization techniques. Therefore, it is important to examine whether dataset cleaning is necessary, depending on the extent to which mislabeled issues impact the evaluation of IRBL techniques.

Dataset	Year	No. of Issues	Bug Reports (%)	Non-bug Reports (%)
Zhou et al.	2012	3,479	55.99%	44.01%
Ye et al.	2015	22,747	65.42%	34.58%
Bench4BL	2018	10,019	69.29%	30.71%
GHRB	2024	131	70.99%	29.01%
BeetleBox	2025	19,146	64.21%	35.79%

Impact of Mislabeled Issues on Performance of IRBL Techniques

IRBL Techniques

To investigate the impact of mislabeled issues, we first defined selection criteria to guide the choice of IRBL techniques. Our goal is to ensure that the selected approaches are reproducible, scalable and aligned with the scope of our study. In particular, we considered techniques that (1) have a publicly available and functional implementation, (2) complete the localization of each bug report within 80 seconds, as followed in prior studies [42], [43], (3) perform file-level localization, and (4) support Java projects, since our benchmarks consist of large-scale Java systems and the majority of IRBL techniques and datasets in the literature are Java-based [5], [14], [21].

Based on these criteria, we selected six representative IRBL techniques that cover different methodological paradigms, including traditional information retrieval model, heuristic-enhanced approach, deep learning-based technique, and recent LLM-driven method. A brief overview of each technique is provided below:

Vector Space Model (VSM): It is one of the earliest IRBL approaches and has served as the basis for many state-of-the-art techniques [1]. In VSM, the bug report and the source files are represented as vectors of weighted terms, where the weights are computed using the Term Frequency-Inverse Document Frequency (TF-IDF) scheme. Next, the similarity between the bug report and each source file is measured using cosine similarity.

BugLocator: It extends VSM by using a revised Vector Space Model (rVSM) that accounts for file length, assigning higher scores to longer files since they are more bug-prone [3]. Additionally, it leverages historical bug data by identifying previously fixed bugs similar to the current report and boosts the rank of files involved in those fixes (SimiScore). Finally, files are ranked by combining the rVSM-based similarity and the SimiScore using a weighted sum.

BLUiR: Building on the limitations of earlier models, BLUiR enhances bug localization by leveraging the structure of source code [24]. It extracts elements like class names, method names, and comments from each file. Bug reports are similarly processed and treated as structured queries. Using a BM25-based retrieval model, BLUiR computes relevance scores by matching bug report sections with corresponding code elements. Lastly, files are ranked based on aggregated scores from all field combinations, giving more weight to terms appearing in multiple regions.

BRTracer: It integrates two key heuristics for bug localization: segmentation and stack-trace analysis [25]. To reduce noise from large files, BRTracer splits each source file into segments and selects the segment most similar to the bug report to represent the file. Additionally, it analyzes stack traces in bug reports to identify directly mentioned files and their dependencies, assigning boost scores based on their presence and position in the trace. These enhancements are combined with a revised VSM-based score to produce a final ranking.

DreamLoc: It is a deep learning-based IRBL framework that uses a Wide and Deep architecture [5]. The Wide component is a linear model that combines software-specific features such as VSM similarity, cyclomatic complexity, and bug-fix history. The Deep component uses a fine-grained relevance matching model, leveraging word embeddings and an attention mechanism to compare bug reports with tokenized code snippets. A fusion layer combines both components to produce a final score for each file.

GenLoc: GenLoc combines semantic retrieval with LLM-driven iterative code exploration to identify buggy files [17]. At first, it retrieves semantically relevant files for a given bug report using embedding-based similarity. Next, it leverages an LLM to analyze a bug report and further inspect these retrieved files, or explore additional files as needed. To enable this exploration, GenLoc provides the LLM with a set of functions (e.g., retrieving method signatures or method bodies) that support iterative analysis about which files are likely related to the reported bug.

Collectively, these techniques reflect the evolution of IRBL research. VSM and BugLocator represent foundational models; BLUiR and BRTracer incorporate structural and heuristic enhancements; DreamLoc exemplifies deep learning-based solutions; and GenLoc reflects the emerging paradigm of LLM-based bug localization. This diversity allows us to comprehensively assess how mislabeled issues affect techniques with varying dependencies on textual content, structural information, and learning mechanisms.

For DreamLoc and GenLoc, we used the original implementations. For BugLocator, BLUiR, and BRTracer, and VSM, we used the implementations provided by [18] and [20], respectively. Several other IRBL techniques (e.g., [11, 14, 15]) were excluded because they lacked publicly available implementations, had unresolved dependency issues, or exhibited scalability limitations. Detailed inclusion and exclusion decisions are provided in the replication package [26].

Dataset, Configurations, and Evaluation Metrics

We evaluated the six techniques on the Ye et al. dataset [21], which remains the most widely used benchmark in IRBL research [44] and continues to serve as the evaluation dataset for recent techniques [17, 23, 45]. Moreover, the size of the dataset (over 22,000 bug reports) makes it suitable for training DreamLoc, a deep learning–based approach. The other datasets used in RQ1 are less suitable for evaluating techniques that rely on training (i.e., DreamLoc) or historical bug-fix data (i.e., BugLocator, BRTracer). The GHRB dataset is relatively small, while the Bench4BL and Zhou et al. datasets contain several projects with fewer than 100 bug reports, making it difficult to obtain sufficient project-level training or historical data. Similarly, BeetleBox is primarily designed for cross-project bug localization scenarios (i.e., training on one project and testing on different projects), which does not align with the design of these approaches [23]. Hence, the Ye et al. dataset provides the most appropriate setting for evaluating the impact of mislabeled issues on all prior techniques.

Following prior work [14, 46], we chronologically divided the original dataset into 60% historical/training data and 40% test data based on issue creation time. Earlier reports were used as historical or training information, while later reports were reserved for evaluation to avoid future-information leakage [21, 47].

We prepared three configurations of the Ye et al. dataset to systematically investigate the impact of mislabeled issues in evaluation and historical/training data: Original (ORIG), Test-Cleaned (TC), and Fully-Cleaned (FC). In the ORIG configuration, both historical/training and test partitions retain all issue reports, including those identified as non-bugs by GenTag. In the TC configuration, non-bug reports are removed only from the test partition, while the historical/training partition remains unchanged. Finally, in the FC configuration, non-bug reports are removed from both partitions. The same chronological split was maintained across all three configurations, with non-bug reports removed from the corresponding partitions according to each configuration. Thus, TC and FC share the same set of bug reports for evaluation but differ in the historical/training information available to the techniques.

These configurations enable us to separately examine two sources of dataset contamination. First, we compare ORIG and TC to assess how mislabeled reports in the test set influence measured localization performance. This comparison is conducted for all six IRBL techniques. Second, we compare TC and FC to investigate the impact of mislabeled reports in historical/training data while keeping the evaluation set fixed. This comparison applies only to techniques that leverage historical bug-fix information or require training data, namely BugLocator, BRTracer, and DreamLoc. For VSM, BLUiR, and GenLoc, which do not use such information, cleaning the historical/training partition has no effect on their inputs. Therefore, we exclude these techniques from the TC–FC comparison.

Each technique was evaluated under its applicable configurations. For GenLoc, we repeated the experiments three times to account for the inherent variability of LLM outputs. In addition to these intra-technique comparisons, we performed inter-technique comparisons to investigate how mislabeled issues affect the relative performance of different IRBL approaches. Specifically, we compared pairs of techniques on ORIG and TC using the same evaluation reports within each configuration. We used the following two evaluation metrics to assess localization performance across these comparisons:

Accuracy@k: It measures the percentage of bugs for which at least one of the buggy files is ranked within the top-$k$ positions ($k = 1, 5, 10$) of the ranked list [48]. A higher Accuracy@k value indicates a better performance of the bug localization technique. Similar to other studies [5], [13], [49], $k$ is set to 1, 5 and 10.

Mean Reciprocal Rank (MRR): This metric calculates the average reciprocal rank of the first correctly identified buggy file across all bug reports [48]. For each bug report, the reciprocal rank is defined as the inverse rank of the first correct buggy file. MRR is calculated as follows:

where $N$ denotes the total number of bug reports. $\operatorname{rank}_{i}$ refers to the rank of the first correct buggy file for the $i$-th bug report. The higher the value of MRR, the better the technique.

We selected these two metrics to capture both ranking quality and practical utility. MRR emphasizes the importance of returning the first relevant result as early as possible, which is crucial for minimizing developer effort [44]. Although multiple files may be associated with a single bug report, identifying the first correct file is especially important, as it often serves as a starting point for discovering additional faulty files through manual inspection or complementary debugging techniques [50]. On the other hand, Accuracy@k is highly practical for real-world scenarios where developers typically examine the top few candidates rather than inspecting an entire ranked list [44].

Statistical Test

To determine whether the observed performance differences are statistically significant, we conducted statistical analyses using the reciprocal rank (RR) values of individual issue reports. We first examined the normality of the RR distributions using the Anderson–Darling test, which is considered one of the most powerful tests for assessing normality [51]. Since the distributions were not normally distributed, we employed non-parametric statistical tests. The choice of statistical test depended on the relationship between the observations being compared. For all analyses, we used a significance level of 0.05 [52].

Intra-technique comparison: We conducted two types of intra-technique comparisons to examine the impact of mislabeled issues in the test set and historical/training data separately.

(i) ORIG vs. TC: To assess the impact of mislabeled test issues, we compared each technique’s performance between the Original (ORIG) and Test-Cleaned (TC) configurations. Since TC is a subset of ORIG, the two samples are partially overlapping: genuine bug reports appear in both configurations and form paired observations, whereas non-bug reports appear only in ORIG and form unpaired observations. Consequently, conventional non-parametric tests are not directly applicable. The Wilcoxon signed-rank test requires fully paired observations [53], while the Mann–Whitney U test assumes fully independent samples [54].

To address this, we applied the non-parametric partially overlapping samples test statistic, as proposed in [55]. There are two variants of this test statistic: one assumes $(T_{\mathrm{RNK1}})$ equal variances and another $(T_{\mathrm{RNK2}})$ does not. To determine the appropriate choice, we conducted the Brown and Forsythe test, a robust version of Levene’s test to assess the equality of variances between the noisy and clean datasets [56]. Based on the results of the Brown–Forsythe test, we applied $T_{\mathrm{RNK1}}$ when the equal-variance assumption was satisfied and $T_{\mathrm{RNK2}}$ otherwise.

These tests first pool all observations from both samples into a single dataset and rank them in ascending order using the RT-1 procedure [57]. In cases of tied ranks, the median rank is assigned. Next, these rank values are used to calculate the means $(\bar{X}_{1}$ and $\bar{X}_{2})$ and variances $(S_{1}^{2}$ and $S_{2}^{2})$ for the ORIG and TC data. Finally, these values are used in equation 2 and 4 for calculating $T_{\mathrm{RNK1}}$ and $T_{\mathrm{RNK2}}$ respectively.

The test statistic ($T_{\mathrm{RNK1}}$ or $T_{\mathrm{RNK2}}$) is referenced against the $t$-distribution with degrees of freedom $\nu_2$, calculated using equation 5.

In the equations, $\bar{X}_1$, $\bar{X}_2$ Means of ORIG and TC data respectively $n_a$, $n_b$ Number of unpaired observations exclusive to ORIG and TC data respectively. In our case, $n_b$ is 0 as there is no issue exclusive to TC data $n_c$ Number of paired observations (bug issues) $n_1$ Total number of observations in ORIG data ($n_1 = n_a + n_c$) $n_2$ Total number of observations in TC data ($n_2 = n_b + n_c$) $S_1^2$, $S_2^2$ Sample variances of ORIG data and TC data (with Bessel’s correction) $r$ Spearman’s rank correlation coefficient for the $n_c$ paired observations

For effect size, we calculated Cohen’s d using equation 7, as suggested by [58].

where $t$ denotes the test-statistic ($T_{\mathrm{RNK1}}$ or $T_{\mathrm{RNK2}}$) and $\nu$ is the degrees of freedom ($\nu_2$). We interpreted the effect size using thresholds from prior work [1]: $d < 0.2$ (trivial); $0.2 \le d < 0.5$ (small); $0.5 \le d < 0.8$ (medium); $0.8 \le d < 1.3$ (large); and $d \ge1.3$ (very large).

To assess the robustness of the ORIG–TC findings to random classification errors in GenTag’s predictions, we complemented the statistical tests above with a Monte Carlo simulation [59]. GenTag may occasionally misclassify issues by labeling bug reports as non-bugs or non-bug reports as bugs. We simulated both types of errors by randomly flipping issue labels. Specifically, we varied the false-negative and false-positive rates independently over five levels (1%, 3%, 5%, 8% and 10%), resulting in 25 (5×5) error-rate combinations. The upper bound of 10% was selected because it exceeds GenTag’s observed classification error (weighted F1-score and accuracy of 0.93 on our manually validated sample). For each scenario, we generated 1,000 perturbed datasets, recomputed the MRR changes and repeated the partially overlapping samples test, resulting in 25,000 perturbed datasets and 150,000 technique-level evaluations.

(ii) TC vs. FC: To assess the impact of mislabeled historical/training issues, we compared the Test-Cleaned (TC) and Fully-Cleaned (FC) configurations for BugLocator, BRTracer, and DreamLoc. Unlike ORIG–TC, TC and FC contain exactly the same evaluation reports, with differences arising only from the historical/training data used by the techniques. Therefore, their per-issue RR values form fully paired samples. We applied the Wilcoxon signed-rank test [52] to evaluate whether the performance differences were statistically significant. We additionally quantified the magnitude of these differences using rank-biserial correlation [60], as defined in equation 8.

where $W_+$ and $W_-$ denote the sums of the positive and negative signed ranks respectively. The rank-biserial correlation $rb$ ranges from $-1$ to $1$. We interpret effect sizes following thresholds from prior literature [61]: $|rb| < 0.1$ (trivial); $0.1 \le|rb| < 0.3$ (small); $0.3 \le|rb| < 0.5$ (medium); $|rb| \ge0.5$ (large).

Inter-technique comparison: To investigate whether mislabeled test issues affect the relative performance of IRBL techniques, we performed pairwise comparisons among the six techniques separately on ORIG and TC. Within each configuration, every pair of techniques was evaluated on the same set of issue reports, producing paired RR observations. Therefore, we employed the Wilcoxon signed-rank test to determine whether the differences between techniques were statistically significant. To control the family-wise error rate arising from multiple pairwise comparisons, we applied the Holm–Bonferroni correction [62] separately to the 15 comparisons within each configuration, using a significance level of 0.05. We additionally computed rank-biserial correlation, as defined in equation 8, to quantify the magnitude and direction of the differences. By comparing the statistical significance and effect sizes obtained on ORIG and TC, we assessed whether removing mislabeled issues changes the relative performance differences among IRBL techniques and the magnitude of such changes.

Result Analysis of IRBL Approaches

Intra-technique Performance: We present the intra-technique performance of the evaluated IRBL approaches under the three dataset configurations: Original (ORIG), Test-Cleaned (TC), and Fully-Cleaned (FC). ORIG corresponds to the benchmark as originally released, TC removes non-bug reports only from the test partition, and FC removes non-bug reports from both the historical/training and test partitions. This design enables us to distinguish the effect of mislabeled issues in the evaluation set from their effect in historical or training data.

We first compare ORIG and TC to examine how mislabeled issues in the test set affect the measured performance of IRBL techniques. Table V and Table VI present the corresponding Accuracy@k and MRR results, respectively. Across all six approaches, removing non-bug reports from the test set consistently reduces measured performance across all metrics. The reduction is most pronounced for VSM, whose Accuracy@1 decreases by 29.26%. The remaining techniques also exhibit reductions in Accuracy@1: 10.31% for BugLocator, 9.40% for BLUiR, 10.02% for BRTracer, 6.46% for DreamLoc, and 5.40% for GenLoc. These declines are notably larger than those reported in prior studies [2, 20], which observed only minimal reductions in Accuracy@1 of classical techniques after removing mislabeled issues. This difference suggests that the impact of mislabeled issues may be more pronounced when IRBL techniques are evaluated on a large-scale and diverse benchmark such as the one considered in our study. The reductions at early cutoff values are also practically relevant, as 83.01% of practitioners consider a bug successfully localized when the correct file appears within the top five ranked results [63]. Therefore, even moderate reductions in early-ranked localization performance can affect the practical usefulness of IRBL techniques.

Technique	Accuracy@1			Accuracy@5			Accuracy@10
	ORIG	TC	Diff (%)	ORIG	TC	Diff (%)	ORIG	TC	Diff (%)
VSM	3.52%	2.49%	-29.26%	8.53%	6.62%	-22.39%	12.17%	9.62%	-20.95%
BugLocator	23.37%	20.96%	-10.31%	45.32%	42.11%	-7.08%	55.57%	52.57%	-5.40%
BLUiR	25.33%	22.95%	-9.40%	48.19%	45.24%	-6.12%	58.11%	54.89%	-5.54%
BRTracer	24.95%	22.45%	-10.02%	47.96%	44.81%	-6.57%	58.07%	54.98%	-5.32%
DreamLoc	28.62%	26.77%	-6.46%	54.52%	52.35%	-3.98%	64.74%	63.23%	-2.33%
GenLoc	40.55%	38.36%	-5.40%	61.24%	58.49%	-4.49%	67.00%	64.41%	-3.87%

Technique	ORIG MRR	TC MRR	Diff (%)	p-value	Effect size
VSM	0.069	0.054	-21.74%	<0.001	0.26
BugLocator	0.341	0.314	-7.92%	<0.001	0.22
BLUiR	0.362	0.335	-7.46%	<0.001	0.23
BRTracer	0.360	0.333	-7.50%	<0.001	0.22
DreamLoc	0.406	0.387	-4.68%	<0.001	0.15
GenLoc	0.493	0.469	-4.87%	<0.001	0.17

A similar pattern emerges for MRR. The differences in reciprocal rank between ORIG and TC are statistically significant for all six techniques, with effect sizes ranging from 0.15 to 0.26. Unlike prior studies [2], [20], which reported statistically insignificant differences for VSM, we observe a statistically significant and substantial reduction (21.74%) in its MRR. BugLocator, BLUiR, BRTracer, DreamLoc, and GenLoc likewise exhibit statistically significant reductions. Notably, even the recent approaches evaluated in our study, including the deep learning-based DreamLoc and LLM-based GenLoc, are affected by mislabeled test issues.

To better understand why VSM exhibits the largest relative decline in MRR, we first consider the nature of the MRR metric. As illustrated in Figure 2, reciprocal rank decreases sharply at early positions. For example, moving the first correct file from rank 1 to rank 5 causes a substantially larger reduction in reciprocal rank than moving it from rank 40 to rank 60. Consequently, a technique that loses a considerable number of early-ranked successful predictions after test-set cleaning can experience a pronounced decline in MRR.

We therefore examine the contribution of bug and non-bug reports to successful predictions at ranks 1–5. Table VII shows that VSM disproportionately benefits from non-bug reports. Although non-bug reports constitute only about one-third of the benchmark, they account for 53% of VSM’s correctly localized top-1 cases, 51% at rank 2, and approximately half of its successful predictions across ranks 3–5. In contrast, genuine bug reports contribute the majority of early-ranked successful predictions for all other techniques.

Technique	Type	Rank
		@1	@2	@3	@4	@5
VSM	bug	47	49	51	50	51
VSM	non-bug	53	51	49	50	49
BugLocator	bug	59	60	61	61	62
BugLocator	non-bug	41	40	39	39	38
BLUiR	bug	60	61	61	62	62
BLUiR	non-bug	40	39	39	38	38
BRTracer	bug	60	60	61	62	62
BRTracer	non-bug	40	40	39	38	38
DreamLoc	bug	62	63	63	63	64
DreamLoc	non-bug	38	37	37	37	36
GenLoc	bug	63	63	63	63	63
GenLoc	non-bug	37	37	37	37	37

successful predictions for all other techniques. At rank 1, bug reports account for 59% of BugLocator’s, 60% of BLUiR’s and BRTracer’s, 62% of DreamLoc’s, and 63% of GenLoc’s successful predictions. This disproportionate contribution of non-bug reports helps explain why VSM experiences a substantially larger performance reduction when such reports are excluded from the test set.

Manual examination of representative cases provides further insight into this pattern. We observed that non-bug reports often contain terms such as UI labels, method names, or field identifiers that closely match source-code tokens, making them comparatively easy for lexical retrieval techniques such as VSM to localize. Figure 3 illustrates such a non-bug issue that requests a UI label change from "Data Field" to a more accurate alternative. Because the report directly mentions a term appearing in the relevant source code, VSM assigns a high rank to the associated file. When such non-bug reports are removed from the evaluation set, VSM loses a substantial portion of its early-ranked successful predictions, contributing to its larger decline in both Accuracy@k and MRR.

Although the observed effect sizes are generally small, the differences in measured localization performance can still have meaningful practical implications. File-level bug localization is inherently coarse-grained, requiring developers to manually inspect candidate files to identify the faulty method, statement, or expression [64]. When the correct buggy file is ranked lower even by a few positions, developers must examine more files, each of which demands time and cognitive effort to understand. Consequently, even moderate rank degradations translate into greater debugging effort. Moreover, degradation in bug localization can affect downstream tasks such as Automated Program Repair (APR) [65, 66]. When the rank of the correct buggy file degrades, APR tools may generate invalid patches by modifying irrelevant parts of the codebase, thereby reducing their effectiveness [67, 68]. For example, an APR tool that relies on GenLoc and is calibrated based on its performance on the original noisy dataset may fail in practice if that performance does not hold on cleaner data. In such cases, the correct file may fall outside the top-ranked candidates, causing the APR tool to apply transformations to unrelated files and produce invalid patches. Therefore, reliable evaluation of bug localization techniques is important not only for fair benchmarking but also for downstream software maintenance tasks that depend on localization quality.

To assess whether the observed ORIG–TC differences are robust to random classification errors in GenTag’s predictions, we conducted the Monte Carlo analysis described in Section III-C. Across all 25,000 simulated datasets, the performance decrease remained consistent, and statistical significance ($p < 0.05$) was maintained in 100% of trials for every technique across all considered error-rate combinations. The 95% confidence intervals for the relative MRR change were narrow with both the lower and upper bounds remaining negative for every technique: VSM [-22.17%, -16.52%], BugLocator [-8.31%, -6.10%], BLUiR [-7.70%, -5.60%], BRTracer [-7.92%, -5.79%], DreamLoc [-5.18%, -3.45%], and GenLoc [-5.23%, -3.64%]. These results indicate that the observed reduction in measured performance after test-set cleaning is robust to plausible GenTag classification errors.

We next compare TC and FC to examine the effect of mislabeled issues in historical or training data. Table VIII and Table IX present the corresponding Accuracy@k and MRR results, respectively. This comparison applies only to techniques that rely on such information, namely BugLocator, BRTracer, and DreamLoc. Unlike the consistent reductions observed when cleaning the test set, the effect of cleaning historical/training data varies considerably across techniques. BugLocator exhibits minor decreases across all metrics, whereas BRTracer shows slight improvements or unchanged performance, indicating that both techniques are relatively insensitive to the removal of mislabeled historical reports. In contrast, DreamLoc experiences a more pronounced decline, with Accuracy@1 decreasing by 12.10% and MRR by 6.98%. One possible explanation is that DreamLoc, as a deep learning-based approach, benefits from a larger volume of training data, even when some reports are mislabeled. Removing these reports reduces the available training data, potentially limiting the model’s ability to learn useful relevance patterns. These results show that, unlike test-set noise, whose removal consistently lowers measured performance across all evaluated techniques, the effect of mislabeled issues in historical or training data is technique-dependent.

Technique	Accuracy@1			Accuracy@5			Accuracy@10
	TC	FC	Diff (%)	TC	FC	Diff (%)	TC	FC	Diff (%)
BugLocator	20.96%	20.81%	-0.72%	42.11%	41.58%	-1.26%	52.57%	52.09%	-0.91%
BRTracer	22.45%	22.82%	1.65%	44.81%	44.81%	0.00%	54.98%	55.21%	0.42%
DreamLoc	26.77%	23.53%	-12.10%	52.35%	47.40%	-9.46%	63.23%	58.49%	-7.50%

Technique	TC MRR	FC MRR	Diff (%)	p-value	Effect size
BugLocator	0.314	0.311	-0.96%	<0.001	0.141
BRTracer	0.333	0.334	0.30%	<0.001	0.216
DreamLoc	0.387	0.360	-6.98%	<0.001	-0.121

Since TC and FC are evaluated on exactly the same set of genuine bug reports, we apply the Wilcoxon signed-rank test to the paired reciprocal-rank observations. The results indicate statistically significant differences for the three history-dependent techniques. However, the magnitude and direction of the changes differ across approaches, reinforcing that mislabeled historical/training data does not have a uniform effect on IRBL techniques. The underlying reasons for these differences may depend on how individual approaches exploit historical or training information and are beyond the scope of this study.

Overall, our results reveal two distinct effects of mislabeled issues. First, mislabeled reports in the test set systematically inflate the measured performance of all evaluated IRBL techniques, although the magnitude of this inflation varies across approaches. Second, mislabeled reports in historical or training data have a technique-dependent effect: BugLocator and BRTracer remain largely stable after cleaning, whereas DreamLoc exhibits a more pronounced reduction. These findings demonstrate the importance of distinguishing between test-set contamination and historical/training-set contamination when assessing the impact of mislabeled issues on IRBL evaluations.

Inter-technique Performance: To investigate whether mislabeled issues influence the comparative evaluation of IRBL techniques, we perform pairwise comparisons using the Original (ORIG) and Test-Cleaned (TC) configurations. We select TC rather than Fully-Cleaned (FC) based on our intra-technique findings, which show that removing mislabeled historical/training reports has negligible effects on BugLocator and BRTracer, while reducing DreamLoc’s performance. These findings suggest that retaining historical/training reports, including those classified as non-bugs, may provide useful information for certain techniques. Furthermore, comparing ORIG and TC allows us to isolate the impact of mislabeled evaluation reports without introducing technique-dependent effects from cleaning historical/training data. We therefore compare the reciprocal rank (RR) values of each technique pair on ORIG and TC. Table X presents the results of the Wilcoxon signed-rank tests, including Holm-adjusted p-values and rank-biserial correlation effect sizes. A positive effect size indicates that the second technique outperforms the first.

Technique 1	Technique 2	Original p-value	Original Significant	Original Effect Size	Original Interpretation	Test-Cleaned p-value	Test-Cleaned Significant	Test-Cleaned Effect Size	Test-Cleaned Interpretation
VSM	BugLocator	<0.001	Yes	0.87	large	<0.001	Yes	0.88	large
VSM	BLUiR	<0.001	Yes	0.86	large	<0.001	Yes	0.88	large
VSM	BRTracer	<0.001	Yes	0.87	large	<0.001	Yes	0.89	large
VSM	DreamLoc	<0.001	Yes	0.89	large	<0.001	Yes	0.91	large
VSM	GenLoc	<0.001	Yes	0.74	large	<0.001	Yes	0.71	large
BugLocator	BLUiR	<0.001	Yes	0.09	trivial	<0.001	Yes	0.09	trivial
BugLocator	BRTracer	<0.001	Yes	0.17	small	<0.001	Yes	0.19	small
BugLocator	DreamLoc	<0.001	Yes	0.22	small	<0.001	Yes	0.26	small
BugLocator	GenLoc	<0.001	Yes	0.35	medium	<0.001	Yes	0.34	medium
BLUiR	BRTracer	0.060	No	-0.03	trivial	0.243	No	-0.02	trivial
BLUiR	DreamLoc	<0.001	Yes	0.16	small	<0.001	Yes	0.20	small
BLUiR	GenLoc	<0.001	Yes	0.28	small	<0.001	Yes	0.27	small
BRTracer	DreamLoc	<0.001	Yes	0.16	small	<0.001	Yes	0.20	small
BRTracer	GenLoc	<0.001	Yes	0.31	medium	<0.001	Yes	0.30	small
DreamLoc	GenLoc	<0.001	Yes	0.16	small	<0.001	Yes	0.13	small

The results show that the statistical significance of pairwise differences remains consistent after removing mislabeled issues. Among the 15 technique pairs, 14 exhibit statistically significant differences ($p < 0.001$) on both ORIG and TC. The only exception is BLUiR versus BRTracer, for which the difference remains statistically insignificant ($p = 0.060$ on ORIG and $p = 0.243$ on TC). Although the statistical significance outcomes remain unchanged, the effect sizes vary across dataset configurations, indicating that mislabeled issues influence the magnitude of performance differences between techniques.

The most pronounced differences are observed between VSM and the remaining approaches. On ORIG, the effect sizes range from 0.74 to 0.89, indicating large differences in favor of the other techniques. After cleaning, these effect sizes increase for four of the five comparisons: VSM versus BugLocator (0.87 to 0.88), BLUiR (0.86 to 0.88), BRTracer (0.87 to 0.89), and DreamLoc (0.89 to 0.91). This pattern suggests that removing mislabeled issues further widens the relative performance gap between VSM and these approaches. These findings are consistent with our intra-technique analysis, where VSM exhibits the greatest sensitivity to mislabeled test reports.

Among the more advanced techniques, the changes are generally smaller but reveal different patterns. The effect sizes for DreamLoc compared with BugLocator, BLUiR, and BRTracer increase from 0.22 to 0.26, 0.16 to 0.20, and 0.16 to 0.20, respectively, suggesting that DreamLoc’s relative advantage becomes more pronounced on the cleaned dataset. Conversely, the effect sizes for GenLoc decrease slightly when compared with VSM (0.74 to 0.71), BugLocator (0.35 to 0.34), BLUiR (0.28 to 0.27), BRTracer (0.31 to 0.30), and DreamLoc (0.16 to 0.13), suggesting that GenLoc’s relative advantage becomes slightly less pronounced after removing mislabeled issues. Nevertheless, GenLoc maintains a statistically significant advantage over all other techniques on both datasets.

Overall, these results demonstrate that mislabeled issues can influence the comparative evaluation of IRBL techniques. Although removing mislabeled issues does not change the statistical significance or direction of the pairwise comparisons in our evaluation, it alters the magnitude of the observed differences, sometimes widening and sometimes narrowing the gap between techniques. Consequently, conclusions about the extent to which one IRBL technique outperforms another may depend on the quality of the evaluation dataset, highlighting the importance of using correctly labeled bug reports for fair and reliable comparisons.

Discussion

This study highlights a critical yet underexamined issue in IRBL research: the widespread inclusion of mislabeled issues in benchmark datasets and the resulting implications for the reliability and practical utility of IRBL techniques.

Our analysis of five IRBL benchmarks spanning more than a decade indicates that issue mislabeling remains a persistent problem, even in recently introduced datasets. This raises concerns about the common practice of treating issue-tracker labels as reliable ground truth. In particular, including non-bug reports in evaluation datasets may lead to overly optimistic estimates of bug localization effectiveness, as our results suggest that these reports tend to be easier to localize than genuine bug reports. These findings highlight the importance of validating issue labels during benchmark construction to ensure the reliability of IRBL performance evaluations.

Importantly, our findings diverge from earlier studies [2, 20] that reported only minor or statistically insignificant changes in performance (e.g., Accuracy@1, MRR), whereas our results show consistent and statistically significant performance degradation across all techniques when evaluated on clean data. This discrepancy is likely due to our use of a larger benchmark dataset, suggesting that the impact of mislabeled issues becomes more pronounced in realistic, large-scale evaluation settings.

A key insight from our study is that the impact of mislabeling depends on both the localization technique and the role of the affected data. Techniques relying primarily on lexical similarity, such as VSM, appear particularly sensitive to mislabeled evaluation reports because non-bug issues may contain explicit textual references to relevant source-code elements. Although techniques incorporating structural information, historical evidence, or learning-based representations exhibit smaller performance changes, they are not immune to evaluation noise. The impact of mislabeling also depends on whether it occurs in evaluation or historical/training data. While removing mislabeled reports from evaluation data provides a more representative assessment of localization performance on genuine bugs, removing them from historical/training data does not consistently improve performance and may even reduce it. This suggests that some reports, despite being unsuitable for evaluating bug localization, may still provide useful information for learning or historical matching. Therefore, dataset cleaning should distinguish between ensuring valid evaluation ground truth and preserving potentially useful historical/training information.

Our inter-technique analysis further demonstrates that mislabeled issues can influence the magnitude of performance differences between approaches, even when their overall ranking and statistical significance remain unchanged. This distinction is important because conclusions about the superiority of a technique often depend not only on whether it outperforms a baseline but also on the extent of that improvement. Thus, evaluations using mislabeled reports may exaggerate or underestimate the relative advantages of different IRBL techniques. These findings underscore the importance of using correctly labeled evaluation datasets and accounting for statistical dependencies when comparing localization approaches.

Finally, mislabeled issues have practical implications beyond evaluation. Performance degradation in bug localization can translate into lower-quality rankings, which prior work has shown to increase developers’ code inspection effort [69] and negatively affect downstream tasks such as automated program repair [70]. Overall, our findings emphasize that dataset quality is a fundamental factor in IRBL research and must be carefully considered to ensure fair and meaningful evaluation.

Threats to Validity

This section presents potential ways in which the validity of the study may be compromised.

Internal Validity: A primary threat lies in the correctness of our issue classification. To address this, we evaluated our classifier against the manually labeled dataset by Herzig et al., a widely used ground truth in prior studies [1, 71, 72]. Additionally, we evaluated GenTag on a manually labeled stratified sample drawn from the five benchmark datasets and performed a Monte Carlo sensitivity analysis that showed the observed performance degradation remained statistically significant under all simulated random label perturbations. However, GenTag’s classification errors may be associated with issue characteristics that also influence localization performance. Investigating the impact of such systematic errors is beyond the scope of this study and remains an avenue for future work. To reduce LLM non-determinism, we set the temperature to zero and repeated experiments three times. Although this study isolates the impact of mislabeled issues, other dataset-level biases such as tangling (i.e., incorrect ground truth) and localization hints may also influence bug localization performance. Since both the noisy and clean datasets originate from the same benchmarks, these biases are expected to remain consistent across conditions and thus do not threaten the comparative validity of our conclusions, though they may slightly affect the absolute performance values. Future work could systematically examine the joint and individual effects of these biases, e.g., by developing automated methods to detect and mitigate tangling and hint-related noise.

Construct Validity: Construct validity concerns whether the evaluation metrics used accurately capture the performance of IRBL techniques. We mitigate this by adopting Accuracy@k and MRR, which are widely used metrics in IRBL literature [1, 48].

External Validity: To assess the prevalence of mislabeled issues, we investigated five benchmarks spanning from 2012 to 2025. To analyze the impact of mislabeled issues, we used the widely adopted Ye et al. dataset [21], which contains over 22,000 reports from diverse, large-scale Java projects. While this dataset is a standard benchmark [14], our findings may not generalize to industrial systems, or software written in other languages. Although we examine six representative IR-based techniques that span a range of methodologies and time frames, our results may not hold for all IRBL approaches. Further studies are required to generalize our findings to other techniques, datasets and ecosystems beyond Java.

Related Work

In the literature, there are four categories of work related to bug localization: Spectrum-Based Bug Localization (SBBL), Information Retrieval-based Bug Localization (IRBL), issue localization and issue resolution. SBBL techniques rely on execution traces from passing and failing test cases to identify suspicious program elements [73, 74, 75, 76, 77, 78]. Issue localization considers a broader range of issue types (e.g., feature requests, refactoring tasks and performance improvements) and aims to identify fine-grained code locations [79, 80, 81, 82, 83]. Issue resolution focuses on generating patches to fix reported issues [66, 84, 85, 86, 87, 88, 89, 90, 91]. Among all issue types, bug reports remain particularly important since unresolved bugs can compromise the reliability, performance and security of software systems [2]. Consequently, IRBL focuses specifically on identifying buggy source files given a bug report using textual information from issue trackers. Since IRBL benchmarks assume that issues labeled as bugs correspond to actual bug reports, mislabeled issues can significantly affect the validity of IRBL evaluation. Therefore, we focus on analyzing the impact of mislabeled issues on the performance of IRBL techniques and primarily discuss prior work in IRBL.

Existing IRBL related work can be divided into two categories: (i) novel IRBL techniques, and (ii) empirical studies analyzing them.

Novel IRBL techniques fall into three main categories by methodology. The first category [3, 8, 24, 25, 64, 92, 93, 94, 95, 96, 97, 98, 99] uses traditional IR models such as Latent Dirichlet Allocation (LDA) and VSM. The second category [5, 9, 10, 11, 12, 13, 14, 23, 47, 50, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113] consists of learning-based models (i.e., machine learning and deep learning models). The third category consists of reasoning-enhanced approaches [15, 16, 17, 114], where LLMs are used as a reasoning component to analyze candidate files.

Several empirical studies have examined the effectiveness and limitations of IRBL techniques and datasets. Some [18, 115, 116, 117] focus on evaluating techniques under different configurations of models or datasets, such as comparisons across multiple projects and languages. Other studies [1, 2, 20, 118, 119, 120] highlight dataset-related issues, including mislabeled issues, incorrect ground truth and their impact on IRBL performance. In contrast, our work presents a comprehensive analysis of the impact of mislabeled issues by evaluating six representative IRBL techniques spanning the evolution of IRBL (from traditional IR models to deep learning and LLM-based approaches), conducting both intra- and inter-technique performance analyses and accounting for dataset overlap through appropriate statistical testing.

Conclusion and Future Work

This study investigates the prevalence of mislabeled issues in established benchmark datasets and their impact on IRBL techniques. Our analysis of three popular benchmark datasets, Ye et al., Bench4BL, and Zhou et al., as well as two recent datasets, GHRB and BeetleBox, shows that mislabeled issues remain widespread, accounting for 29.01%–44.01% of reports across datasets, with an average of 34.82%.

To assess the impact of these mislabeled issues, we analyzed the performance of six representative IR-based bug localization techniques, VSM, BugLocator, BLUiR, BRTracer, DreamLoc, and GenLoc, using the original, test-cleaned, and fully-cleaned configurations of the Ye et al. dataset. Our results show that removing mislabeled issues from evaluation data consistently reduces the measured performance of all six techniques, with Accuracy@1 reductions ranging from 5.40% to 29.26% and statistically significant MRR reductions ranging from 4.68% to 21.74%. Although mislabeling does not change the overall ordering of techniques or the statistical significance of pairwise comparisons, it influences the magnitude of their performance differences. This highlights the importance of dataset curation in IRBL research to ensure fair evaluation and development of reliable debugging tools.

In the future, we will examine the impact of mislabeled issues on more fine-grained localization approaches, such as method-level [97, 104, 121] and change-set-based [64, 109, 110] techniques. Additionally, we will analyze projects written in other programming languages to assess whether the observed impact of mislabeled issues remains consistent across different development ecosystems.

References