WorksComputer Science
Hybrid Source Code Change Impact Analysis
Abstract
Change impact analysis (CIA) helps developers identify code entities impacted by their initial changes. While several approaches for CIA were proposed, they still struggle with either low precision or low recall, thereby increasing manual effort for developers to verify the results. Furthermore, limitations in the used datasets and experimental setups can bias the results reported by many CIA approaches. In this paper, we first address the dataset limitations by manually labeling 1,931 changed code entities stemming from 263 commits of 22 open-source Java projects. Next, we introduce two variants of katIA, our code change impact analysis approach that combines static impact analysis with the reasoning capabilities of large language models (LLMs). The first variant uses a single prompt (katIA-SP) for refining an initial static impact set while the second variant uses three specialized agents (katIA-MA). The evaluation with our curated dataset shows that katIA-SP with Gemini 2.5 Flash achieved an F1-score of 22.4% when matching impacted entities by their fully qualified names (FQN) and 27.0% when matching them by their class and entity names (CE). The latter shows an improvement of 2.0 percentage points over the state-of-the-art. This score is further improved by katIA-MA which, using Gemini 2.5 Flash, yields an FQN-matching F1-score of 29.1% and a CE-matching F1-score of 33.3%. While these improvements come at the cost of increased median generation time from 0.570 s to 11.214 s and median token usage from 1,235 to 6,462 tokens, the improved performance justifies the additional cost. The comparison of three LLMs using katIA-SP finds that Gemini 2.5 Flash is the most consistent model. These results indicate that combining static impact analysis with LLM-based refinement, particularly with the multi-agent variant, can improve code change impact analysis.
Introduction
Software systems continuously evolve through changes to their source code [1]. Whenever software developers modify a piece of source code, they need to understand the impact of these changes on other parts of the system to ensure code functionality and quality. For example, changing a method’s sorting criterion may affect callers that rely on the previous ordering of the returned data, requiring updates to preserve the behavior.
According to Lehnert [2], developers face two fundamental questions when changing software: (1) which software entities need to be modified to implement the intended change (seed), and (2) which additional entities may be affected as a consequence of this modification (impacted). Change impact analysis (CIA) aims to identify such affected entities of a software system [2]. In practice, change impact analysis can support software maintenance tasks, such as bug fixing and new feature development. Kretsou et al. Kretsou et al. [3] found that developers benefit from knowledge about co-changing code parts, because identifying affected code parts often requires more effort than implementing the intended change itself. However, understanding the full impact of code changes still comes with a high workload and insufficient tool support, particularly during code review [4]. Furthermore, Yadavally and Nguyen Yadavally and Nguyen [5] found that existing change impact analysis approaches often struggle with low precision or low recall, especially because not all code changes and their impacts are structurally connected.
Existing CIA approaches rely on different sources of information, including syntactic dependency analysis of source code changes [6], co-evolution information [7], and, more recently, transformer-based code representations [8]. Motivated by the improvements achieved with large language models (LLMs) in software engineering [9, 10, 11, 12, 13], research has begun to investigate whether code change impact analysis can benefit from them, too. For example, Yadavally and Nguyen Yadavally and Nguyen [5] presented RIPPLE, a two-phase LLM-based approach to predict the change impact from the changes in one method and the corresponding Jira bug report.
While RIPPLE is a promising approach, several aspects of its methodology and evaluation can be improved. First, determining the main change, which Yadavally and Nguyen Yadavally and Nguyen [5] refer to as the seed edit location, they chose the structurally most central method as the seed change, which is an overly simplified heuristic. Other approaches, such as ATHENA [8], treat each changed method once as the seed change and the remaining changed methods as impacted entities. We argue that such heuristics might bias the evaluation of CIA approaches. Second, the dataset ALEXANDRIA [8] used by both RIPPLE and ATHENA does not cover added methods, changed fields and classes, and it does not account for language-specific features, such as method overloading in Java. For instance, RIPPLE matches impacted methods only by the source file name and the simple method name without parameters. To address these issues, we propose the dataset CONSTANTINOPLE, which extends ALEXANDRIA with field and class changes. CONSTANTINOPLE comprises a total of 1,931 changed code entities. Each entity is represented by its fully qualified name and manually labeled as either seed, impacted, or non-related.
Next, we introduce katIA, a hybrid approach that first uses static code analysis to provide an initial static impact set which katIA then refines using LLMs. For the LLM-based refinement, we explore two variants, first the refinement with a single prompt called katIA-SP and second, the refinement with three specialized agents, called katIA-MA. A preliminary version of CONSTANTINOPLE and initial results for katIA are presented in an extended abstract [14], whereas this paper presents dataset construction, both variants of katIA, and the evaluation in detail.
Compared to RIPPLE, katIA-SP and katIA-MA both receive the seed changes, the initial static impact set, and the bug report, whereas the change planner in RIPPLE receives only the bug report and the seed changes, and the reasoner receives the change intent, change plan and the candidate set within the dependence clusters (i.e., initial impact set). Furthermore, katIA does not rely on evolutionary coupling but focuses on the static code relationships of the seed changes. Finally, we do not restrict katIA-SP and katIA-MA to only refine the impact set within the clusters computed from the candidate set, but allow them to extend the initial impact set.
We used CONSTANTINOPLE to evaluate the two variants of katIA. First, we studied the performance and consistency of katIA-SP to answer RQ1: What is the performance of katIA-SP using static analysis and a single prompt to analyze the code change impact? We experimented with three LLMs and measured the performance with precision, recall, and F1-score. To compare with the state-of-the-art, we computed these metrics using CE-matching, based on class and entity names, and additionally using FQN-matching, based on fully qualified names. Furthermore, we repeated each experiment three times to measure the consistency of results. When using CE-matching, katIA-SP achieved F1-scores of 27.0% with Gemini 2.5 Flash and 28.5% with Qwen3 Coder Plus. Gemini 2.5 Flash was the most consistent model, producing 99.62% consistent results. Furthermore, katIA-SP with Gemini 2.5 Flash and Qwen3 Coder Plus achieved F1-score improvements of 2.0 and 3.5 percentage points over the F1-score reported for RIPPLE [5].
Second, we investigated whether decomposing the refinement task across three specialized agents improves katIA-SP to answer RQ2: To what extent does katIA’s multi-agent variant (katIA-MA) improve change impact analysis compared to katIA-SP?
For this experiment, we used only Gemini 2.5 Flash because of its high consistency and decent performance. The results show a further increase in precision, recall, and F1-score under FQN-matching, the latter reaching 29.1%. Under CE-matching, recall and F1-score increased, the latter reaching 33.3%, while precision decreased slightly. Compared to RIPPLE, the CE-matching F1-score is 8.3 percentage points higher.
Overall, our results show that combining static code analysis with LLM-based refinement, especially in a multi-agent variant, can improve code change impact analysis. However, despite these improvements, change impact analysis remains a challenging problem.
In summary, this paper makes the following contributions:
An extended, more detailed version of the ALEXANDRIA dataset, named CONSTANTINOPLE, for evaluating code change impact analysis approaches, comprising manually labeled method, field, and class changes.
A detailed analysis of the manual labeling process.
katIA, our hybrid code change impact analysis approach that combines static code analysis with LLM-based refinement, considering a single prompt (katIA-SP) and a multi-agent variant (katIA-MA).
An evaluation of both variants and comparison to the state-of-the-art approach RIPPLE.
The remainder of the paper is structured as follows: Section II discusses scientific work. Section III explains our dataset labeling process and presents the descriptive statistics of CONSTANTINOPLE. § 4 presents our approach, including static code change impact set construction and LLM-based refinement. § 5 reports our experimental setup and results, Section VI discusses the results and threats to validity, and Section VII concludes the paper. The supplementary material of this work can be found online2.
Related Work
This section discusses related work on change impact analysis and points out the research gaps addressed in this paper.
Change Impact Analysis
Several approaches to support developers in analyzing the impact of code changes were proposed. Ren et al. [6] designed and implemented CHIANTI, a change impact analysis tool for Java. CHIANTI receives two application versions and works on the atomic changes within them. The resulting change impact is the set of tests potentially affected by the changes performed. Furthermore, the authors reported the affecting changes, i.e., the changes that could alter test execution. Their empirical evaluation revealed that an average of 52% of the unit tests of Daikon were affected.
Lehnert [2] proposed a taxonomy of impact analysis to help practitioners navigate and filter the variety of existing impact analysis approaches. The taxonomy comprises eight classification criteria, including scope, granularity, analysis style, and the available experimental results. Gethers et al. [15] proposed an approach for change request impact analysis that integrates history-based mining with dynamic analysis and information retrieval. They evaluated their approach on four open-source projects, yielding statistically significant improvements over the information retrieval-based impact analysis approach (IRCR). Dit et al. [16] presented IMPACTMINER, which implements the approach proposed by Gethers et al. [15].
Hanam et al. [17] addressed the problem of code change understanding in the context of code review by proposing semantic change impact relations and their tool SEMCIA which was evaluated on 2,000 JavaScript commits from three Node.js projects. They were able to decrease the number of false positives and the size of change impact sets. Finally, they conducted a user study where they found that their tool positively impacted both the speed and accuracy of the developers’ code review tasks. Huang et al. [18] proposed an approach to improve traditional change impact analysis by mapping historical change patterns from similar classes in other projects to boost the prediction accuracy of existing CIA tools. Zhou et al. [7] used evolutionary coupling to improve impact predictions. Evolutionary coupling refers to relationships where code entities tend to frequently change together, similarly to the co-evolution of production and test code. The authors discussed twelve types of software relationships that define the edges between code entities and can be used to model impact dependencies.
Yan et al. [8] proposed ATHENA, a transformer-based ranked impact set prediction approach using program dependence graphs. They evaluated ATHENA using ALEXANDRIA, a dataset based on the set of untangled commits from Herbold et al. [19]. As mentioned before, the dataset contains no clear indication of the seed change. In their evaluation, each changed method of a commit was treated as a query method, i.e., seed change, once, while the other methods formed the corresponding ground-truth impact set. This led to 4,405 impact analysis tasks in the largest of the three different evaluation settings, which were evaluated using ranked set metrics.
Yadavally and Nguyen [5] proposed RIPPLE, an intent-aware, two-phase LLM-based approach to predicting impact sets from seed method changes and Jira bug reports. For the evaluation, they created a dataset comprising the changed methods of 100 commits from the ALEXANDRIA dataset represented by their source-file and method names. For each commit, they chose the method with the most direct and indirect dependencies, including call and class-member dependencies, as the seed change for their impact analysis task. Using GPT-4o, RIPPLE outperformed ATHENA achieving an average precision of 28.2%, recall of 36.3%, and F1-score of 25.0% for matching the source file name and the simple name of the method. Finally, they report the results from an ablation study, demonstrating the importance of incorporating both the change intent and the LLM planning phase into their method.
Addressing the shortcomings of simple heuristics for determining the seed change, Stengg et al. [20] proposed a dataset extension of ALEXANDRIA, for which they labeled the seed-impact relationships of changed code entities. They evaluated the capabilities of the models GPT-5 and GPT-5-mini for CIA, obtaining median F1-scores of 0.0%, which shows the limitations of approaches that rely solely on simple prompts and LLMs.
Research Gap
As mentioned before, existing CIA approaches, such as RIPPLE [5] or ATHENA [8], rely on datasets, such as ALEXANDRIA, that have several limitations. First, they consist of Java method changes only and ignore new methods and changes of other code entities such as fields and classes. Second, they use simplified heuristics to determine the seed change [5] or use each method as the seed change once [8]. Third, they do not fully support method overloading. We argue that all three shortcomings can notably bias the results. In our dataset CONSTANTINOPLE presented in the next section, we address these shortcomings and lay the foundation for a less biased evaluation of change impact analysis approaches.
Furthermore, we propose katIA, a hybrid approach combining static code analysis and LLMs. Similar to RIPPLE, katIA begins with a static analysis phase to produce an initial candidate set. For RIPPLE, this set restricts LLM reasoning to a fixed dependency-enhanced candidate set, while katIA treats the initial static impact set as a starting point and uses the LLM to both filter existing entities and identify additional impacted entities. Furthermore, katIA relies solely on static code relationships, eliminating the need for evolutionary coupling and project history. We evaluate two variants of katIA, katIA-SP, which uses a single prompt for impact set refinement, and katIA-MA, which uses three agents for impact set refinement.
Code Change Impact Dataset
This section presents the motivation, construction, and descriptive statistics of our dataset CONSTANTINOPLE. It extends ALEXANDRIA, a benchmark for impact analysis published by Yan et al. [8]. ALEXANDRIA is based on a set of manually untangled commits proposed by Herbold et al. [19]. Yan et al. state that there are other datasets using manually untangled commits. However, these datasets either contain a limited number of commits or perform untangling at the file or commit level rather than the method level, which limits the applicability for method-level impact analysis. The ALEXANDRIA dataset contains 910 commits from 25 open-source Java projects, which is substantially more than other benchmarks so far.
Limitations of Alexandria
Despite being used for the evaluation of various state-of-the-art CIA approaches such as ATHENA [8] and RIPPLE [5], we found that ALEXANDRIA has four main limitations, namely (1) limited entity coverage, (2) missing method signatures, (3) missing newly added methods, and (4) undefined seed changes.
Limited entity coverage. It only covers method changes but ignores changes to other program entities, such as fields and classes. This limits the practical usability of the approaches as developers not only change methods.
Missing method signatures. Method changes are represented by the method names without their parameters, which implies that approaches cannot precisely resolve overloaded methods. Missing added methods. The dataset does not include newly added methods. Consequently, the approaches using it are limited to methods that already exist in the corresponding parent commit.
Undefined seed changes. Yan et al. [8] state that the ALEXANDRIA dataset provides no clear indication of the seed change. Instead, in their approach each changed method of a commit is treated as a seed change once, while the remaining methods form the corresponding impact set. Yadavally and Nguyen [5] also used ALEXANDRIA but chose a different approach for determining the seed change. They used the method with the most direct and indirect call and class-member dependencies as the seed change. Both approaches solely rely on heuristics to identify the seed changes.
To address these limitations, we extended the ALEXANDRIA dataset in two ways. First, in addition to method changes, we extracted changes to fields and classes, as well as newly added methods. For each entity, we used its fully qualified name3, e.g. org.example.A.b(), including parameter types for methods to handle method overloading. Second, we manually labeled each changed code entity as either a seed, an impacted, or a non-related change.
Dataset Construction
Change impact analysis (CIA) aims to identify the software entities impacted by a change [2]. Accordingly, we annotated every changed production code entity either as seed, impacted, or non-related.
Seed. A seed change is a code change made to fulfill the requirements derived from the corresponding Jira bug report. The set of all seeds for a bug report constitutes the bug-fixing change as defined by Rodríguez-Pérez et al. [21]. Unlike the heuristics used in RIPPLE or ATHENA, a single commit may contain multiple seeds when several changed entities are involved in addressing the reported bug.
Impacted. Impacted changes are triggered by the seed changes, not by the requirements of the reported bug. They are consequences, i.e., they do not directly address the bug report but become necessary because of the seed changes. For example, if a seed change modifies a method’s sorting criterion, its callers that rely on the previous ordering of the returned data must be modified to preserve the behavior.
Non-related. Code changes that are neither seed nor impacted are non-related changes. These changes occur within the same tangled commit [22] but have no clear origin in or relationship to the bug report. Although Herbold et al. [19] untangled the dataset, we still found a portion of tangled changes during our dataset construction.
To classify changed code entities into these categories, we grouped the data of ALEXANDRIA by repository, commit hash, and parent commit hash. This resulted in 918 commits. In contrast, the 910 commits reported by the original authors correspond to grouping by repository and parent commit hash only. We then extracted a list of changed files per commit and
excluded test files based on standard directory and naming conventions. Finally, we filtered the commits to retain only those that modified between one and five Java source files for our dataset construction. We selected this range because at least one changed file is required to contain changed entities, while an upper bound of five changed files keeps manual labeling feasible. In the original dataset, only 94 of the 918 commits are not within this range. Similar to Yadavally and Nguyen [5], we extended our dataset with the issue ID from the corresponding commit message to link the commits to their Jira bug report. The majority of the bug reports (82.87%) refer to a single commit, while the maximum number of commits for a single bug report is eight. The latter occurs only once while 111 bug reports (14.62%) are referenced by exactly two commits, 13 by three, and five by four commits. We excluded commits referencing the same bug report from the dataset only if they contained identical code changes or a subset of changes of another commit in the dataset. Changes to comments, imports, and annotations were excluded. Also note that commits which did not contain at least one seed change and one impacted change were skipped, adhering to the criteria used in other works.
To label the changed code entities, we implemented a helper tool based on Spoon [23] to extract the changed source code entities per commit. Furthermore, its user interface helps evaluators to inspect the changed code entities and the corresponding bug report. The labeled code entities are exported into a CSV file, where each row corresponds to a commit containing metadata and one JSON object per change label with further information, such as the fully qualified name. All evaluators used this tool for labeling.
The labeling process was performed by the first author and two co-authors, all of them having several years of experience in Java and code analysis. The labeling comprised three phases. We initially sampled 420 commits to use 20 commits for the calibration phase and the remaining 400 for the actual dataset labeling phases. The calibration phase, where each evaluator labeled the same 20 commits, established a common understanding of the labeling guidelines and reduced ambiguities before the main labeling process. The shared phase produced the first 100 labeled commits of the dataset, for which the main author calculated the agreement among the evaluators to refine the guidelines and ensure consistent labeling. Finally, the individual phase was used to extend the dataset, as each evaluator labeled 100 individual commits.
Calibration Phase
Calibration Phase: First, each evaluator received the same set of 20 randomly sampled commits to be labeled independently. After that, they discussed their findings to establish common labeling guidelines and practices to define a comprehensive labeling strategy for further labeling.
Shared Phase
Shared Phase: In the shared labeling phase, each evaluator labeled the same 100 randomly sampled commits that were not part of the calibration set. The first author collected the results from all evaluators and analyzed their pairwise overlap. Majority vote was applied, and we discussed labels with strong disagreements. The first author also improved the guidelines
with the learnings of this phase.
Since each evaluator selected a label for each code entity, we measured the pairwise agreement using Jaccard set overlap similar to prior work on agreement for set-valued annotations [24]. We report the averages per evaluator pair and label in Table I. During this shared phase, the overall set overlap was 56%, with average pairwise overlaps ranging from 52% to 60%. The similar values among the author pairs suggest that differences did not occur between a specific evaluator pair. Furthermore, these values are not interpreted as intercoder agreement, but as analysis of the set overlaps within the shared labeling process. An overlap of 100% would mean that the sets reported by both authors were identical, whereas missing or additional code entities per label category result in lower values. We can observe that the agreement for non-related code changes is substantially higher than for seed or impacted changes. This indicates that it is easier to choose changes that neither contribute to a bug fix nor result from it than to distinguish between seed and impacted code entities. The distinction between seed and impacted code entities is not always clear from the changes alone but needs context and developer interpretation.
| Evaluator Pair | Seed | Impacted | Non-related | Overall |
| A–C | 50% | 51% | 64% | 55% |
| A–B | 51% | 55% | 74% | 60% |
| B–C | 48% | 47% | 62% | 52% |
| Average | 50% | 51% | 67% | 56% |
Table I. Jaccard set similarities per evaluator pair and label of the shared phase.
One evaluator documented all label discussions and the final decisions for previously undecided labels. This resulted in the first 100 annotated commits of our dataset, comprising 71 labeled and 29 skipped commits, as they did not contain at least one seed change and one impacted change and therefore were excluded from our dataset. All evaluators agreed that previous discussions were valuable as preparation for the individual labeling phase and helped resolve ambiguities as the distinction between seed and impacted entities may allow different interpretations.
Individual Phase
In the individual labeling phase, each evaluator labeled another set of 100 commits that were in neither of the two previous labeling phases. The goal of this phase was to extend the dataset.
Constantinople Dataset
Table II shows the descriptive statistics of the labeled dataset CONSTANTINOPLE. It comprises 263 commits and 1,931 changed code entities across 22 projects. The number of changed code entities is computed as the sum of the entities labeled as seed changes (#S), impacted changes (#I), and non-related changes (#NR). For each commit, the dataset also contains the commit hash of the current commit and its parent,
| Project | #C | #BR | #S | #I | #NR |
| ant-ivy | 56 | 55 | 109 | 276 | 14 |
| commons-bcel | 3 | 3 | 4 | 37 | 5 |
| commons-beanutils | 5 | 5 | 9 | 16 | 1 |
| commons-codec | 2 | 2 | 3 | 36 | 0 |
| commons-compress | 19 | 14 | 337 | 109 | 2 |
| commons-configuration | 21 | 20 | 32 | 69 | 5 |
| commons-dbcp | 3 | 3 | 5 | 13 | 0 |
| commons-digester | 4 | 3 | 6 | 15 | 0 |
| commons-io | 9 | 9 | 12 | 44 | 3 |
| commons-jcs | 10 | 10 | 17 | 36 | 1 |
| commons-lang | 11 | 11 | 24 | 65 | 1 |
| commons-math | 29 | 28 | 58 | 147 | 13 |
| commons-net | 18 | 17 | 23 | 63 | 0 |
| commons-scxml | 8 | 8 | 14 | 39 | 9 |
| commons-validator | 5 | 4 | 22 | 31 | 0 |
| commons-vfs | 10 | 10 | 30 | 58 | 6 |
| deltaspike | 2 | 2 | 2 | 5 | 2 |
| giraph | 18 | 18 | 40 | 92 | 5 |
| gora | 5 | 5 | 11 | 17 | 33 |
| openjnp | 12 | 12 | 23 | 45 | 0 |
| parquet-mr | 12 | 12 | 16 | 109 | 6 |
| systemml | 1 | 1 | 1 | 5 | 0 |
| Total | 263 | 252 | 498 | 1327 | 106 |
Table II. Descriptive statistics of the labeled dataset Constantinople. #C refers to the number of commits, #BR the number of bug reports in Jira, #S, #I, and #NR list the number of changed code entities labeled as seed, impacted, and non-related, respectively.
the number of changed production code files and code entities, the total number of added and deleted lines of code, the commit message, the Jira bug report key, title, description, and the labeled entities represented by their fully qualified names. CONSTANTINOPLE is available in the supplementary material.
During the phases of dataset construction, the evaluators labeled a total of 420 randomly sampled commits. The 20 samples used during the calibration phase were not considered for the final dataset.
Among the 400 commits, they identified 37 bug reports for which multiple commits referenced the same Jira bug report key. For ten of these 37 bug reports, the corresponding commits contained no identical code changes, indicating that they addressed different aspects of the same bug report. For 20 bug reports, the commits contained identical code changes, while for seven bug reports, a later commit contained a subset of the code changes introduced by an earlier commit. To avoid duplicates, we retained only the first occurrence of each such commit and excluded the remaining occurrences from the dataset, resulting in 20 commits being removed from our final dataset. As an additional consistency check, we compared the labels of these commits. If commits contained a subset of the changes of another, we restricted this comparison to the shared code changes. We found that the code changes for 18 bug reports received identical labels, while four received partially identical labels with overlapping selected labels. The
caption: Overview of the katIA approach that, given the seed changes of a commit, first constructs an initial static code change impact set that is then refined either with a single prompt (katIA-SP) or three agents (katIA-MA).
commits of the remaining five bug reports were assigned different labels.
During the labeling process, the evaluators identified 117 commits that did not contain at least one seed change and one impacted change. Similar to prior work that uses labeling heuristics to create sets containing at least one seed and one impacted entity, we excluded these 117 commits from the final version of CONSTANTINOPLE. Together with the previously discussed 20 commits that reference the same bug reports, 137 of the 400 commits were excluded from CONSTANTINOPLE, resulting in 263 labeled commits that we use for the evaluation of katIA.
katIA Approach
fig:1 provides an overview of our proposed approach katIA. katIA consists of two phases. In the first phase, given a set of seed changes, static analysis is used to compute the initial static code change impact set. This set consists of code entities connected to the seed changes through usage relationships. For example, for a method, this set contains all its callers and callees. In the second phase, the initial static code change impact set is refined by using LLMs. The second phase exists in two variants, either using a single prompt (katIA-SP) or three agents (katIA-MA). In the following, we describe each phase of katIA in detail.
Phase 1: Static Impact Set Construction
To construct the initial static code change impact set, katIA first clones the corresponding project and checks out the respective parent commit. Next, it uses the static code analysis framework Spoon [23] to parse the project and to identify the corresponding code entity of each given seed change. If a seed change is absent from the checked-out commit because the entity is newly added in the evaluated commit, the static analysis cannot resolve the corresponding code entity. Consequently, the analysis returns an empty impact set for that entity. Next, depending on the type of the code entity, katIA statically analyzes the dependencies to other code entities. katIA supports classes including interfaces, methods including constructors, initializer blocks, and fields. Concerning fields, katIA extracts read, write, and read-write relations, i.e., the static impact set for a field consists of a list of methods that access it. Concerning methods, katIA extracts all callers and callees. Classes can be used, extended, or implemented by other classes. katIA extracts incoming and outgoing dependencies, as it identifies the classes that use, extend, or implement the seed class, and the classes that the seed uses, extends, or implements. The initial static impact set of the seed changes in a commit consists of all dependent code entities identified by katIA. Currently, the static parser of katIA does not consider indirect dependencies between code entities, dynamic analysis during runtime, reflection-based accesses, or external libraries. Consider an example where methods , , and were changed, assuming that the seed change is in method . Method calls method , whereas calls . The corresponding initial static code change impact set computed by katIA contains org.example.A.b() and org.example.A.c().
Phase 2: Impact Set Refinement
LLMs have been widely used to address software engineering challenges [25]. Regarding code change impact analysis, a recent approach of Yadavally and Nguyen [5] called Ripple showed that a combination of dependency analysis in the first phase and LLMs in the second phase can improve the performance of code change impact analysis. Motivated by Ripple, katIA also uses LLMs in the second phase to refine the initial static code change impact set computed by our static analysis. We provide two variants of the second phase: first, using a single prompt (katIA-SP), and second, using three specialized agents (katIA-MA).
Single Prompt Refinement
We carefully constructed the prompt according to the findings and guidelines of recent software engineering research [26, 27, 28]. The prompt template can be found in the supplementary material. It starts with a system prompt that tells the LLM that it is an expert in Java and in analyzing the impact of source code changes on other code entities in a project’s production code. Such impersonation has been shown to enhance the performance of LLMs [29]. The prompt then presents general background information. It states that the model receives information about a set of changes, called seed changes, that were made to fix a bug reported in Jira and an initial static impact set containing the fully qualified names of code entities statically connected to the seed changes. This general information is followed by the specific change impact analysis task. This task tells the model that an impacted code entity must be created or modified as a consequence of the seed changes. Additionally, this part states when to remove or add a code entity to the initial static impact set and the final goal of the task, which is to refine the initial set to only contain impacted code entities. Next, the prompt contains all inputs (Jira bug report, seed changes, and initial static code change impact set), followed by explicit instructions on the output format and three example outputs.
Multi-Agent Refinement
2) Multi-Agent Refinement: The goal of katIA-MA is to improve katIA-SP by decomposing the single prompt into three separate subtasks. All agent and task definitions can be found in the supplementary material. Each subtask corresponds to a separate state of the change impact analysis process and is assigned to one specific agent.
Analysis Agent. This agent is tasked to interpret the seed changes and the Jira bug report to extract evidence relevant to identifying non-seed Java code entities that likely need modification or creation as a consequence of the seed changes. It produces evidence for each entity of the initial static code change impact set, including their fully qualified name, an assessment of whether the entity should be modified as a consequence of the seed changes, and the supporting evidence. Similarly, it is tasked to produce this evidence for code entities that need to be created as a consequence of the seed changes.
Refinement Agent. The structured evidence of the previous agent is provided to the refinement agent together with the seed changes and the initial static code change impact set. Similar to the task description of katIA-SP, this agent is instructed to remove existing code entities from the set and add missing ones when sufficient evidence was provided by the previous agent.
Reporting Agent. The reporting agent is provided with the description of the expected output format and example outputs adhering to this format. This agent is responsible for reporting the output in the expected format. It is instructed to not alter the set received from the previous agent as its only responsibility is reporting the refined impact set in the expected output format.
Evaluation
This section first presents the experimental setup used to evaluate katIA-SP and katIA-MA, followed by the results for our two research questions. RQ1 evaluates the single prompt refinement variant katIA-SP, while RQ2 evaluates the multi-agent refinement variant katIA-MA.
Experimental Setup
We measured the performance and consistency of katIA-SP considering three different LLMs. Second, we used the LLM that provided the best balance between performance and result consistency to evaluate katIA-MA. We also compared the resource usage using this LLM in both variants of katIA.
Table III lists the three LLMs used to evaluate katIA-SP. We selected a diverse set of models from three different model families, including both general-purpose and code-focused models. These API-hosted models are optimized mainly for either inference efficiency or coding tasks while keeping inference costs low. We accessed all models through OpenRouter4 and pinned requests to a single provider per model to improve reproducibility. Based on token throughput per second, latency, uptime, and cost, we selected OpenAI for GPT-5.4 Mini, Alibaba Cloud Int. for Qwen3 Coder Plus, and Google Vertex (EU) for Gemini 2.5 Flash. This setup makes the evaluation independent of local computing resources.
| Model | Context | OpenRouter Identifier | Provider |
| GPT-5.4 Mini | 400K | openai/gpt-5.4-mini | OpenAI |
| Qwen3 Coder Plus | 1M | qwen/qwen3-coder-plus | Alibaba Cloud Int. |
| Gemini 2.5 Flash | 1M | google/gemini-2.5-flash | Google Vertex (EU) |
Table III. Selected LLMs, their context size, their OpenRouter identifiers, and pinned providers.
Despite their advantages, LLMs have the fundamental problem of non-determinism, also for code generation [30]. To assess the consistency of the results while keeping the computational cost manageable, we repeated each experiment for each model and variant three times. Throughout all experiments, the temperature parameter was set to 0 to maximize output determinism.
Similar to previous approaches, such as Ripple [5], we used set-based evaluation metrics to measure the performance because katIA outputs unordered impact sets. For a commit, we computed precision, recall, and F1-score, based on the ground truth impact sets from our curated dataset and the refined impact set output by katIA. Specifically, for each commit and metric, we computed the median across the three repetitions for each model. We then calculated the macro-averages across all commits for both FQN-matching and CE-matching. CE-matching only considers the declaring class and the code entity’s simple name without any parameters or further information. For example, the code entity with the fully qualified name org.example.A.b gives the class-entity name A.b. We included CE-matching to allow the comparison with Ripple [5], as they represented their ground truth values as sourceFileSimpleName.methodSimpleName. Note that our CE-matching favored Ripple, because we considered class name and entity name, therefore also inner classes, while Ripple only considers top-level classes. We do not report ranking-based metrics (e.g. Hit@k), as our approach predicts an unordered set of impacted entities.
Evaluation of katIA-SP
Table IV lists the results of the three models and initial static impact set as an intermediate step, evaluated with both FQN-matching and CE-matching. We can observe that the two models with 1M context size, Gemini 2.5 Flash and Qwen3 Coder Plus outperform GPT-5.4 Mini for all metrics. CE-matching consistently outperforms FQN-matching, indicating that the LLMs produce approximate results well but struggle to accurately identify complete fully qualified names, potentially because they lack access to the source code repository. For FQN-matching, the single prompt with all three LLMs improves precision compared to the impact set obtained only with static analysis. GPT-5.4 Mini increases precision by 2.9 percentage points, while the other models more than double the static analysis precision of 15.8%. Recall decreases by 3.7 percentage points for GPT-5.4 Mini and by 0.4 percentage points for Gemini 2.5 Flash, whereas Qwen3 Coder Plus improves the recall by 5.1 percentage points. Despite these differences, the single prompt approach with the three LLMs achieves higher F1-scores than the static analysis.
| Configuration | FQN-matching | CE-matching | ||||
| Prec | Rec | F1 | Prec | Rec | F1 | |
| Static analysis | 15.8% | 23.5% | 14.6% | 16.9% | 24.9% | 15.6% |
| katIA-SP (GPT-5.4 Mini) | 18.7% | 19.8% | 15.8% | 22.1% | 22.9% | 18.5% |
| katIA-SP (Gemini 2.5 Flash) | 31.9% | 23.1% | 22.4% | 39.9% | 26.7% | 27.0% |
| katIA-SP (Qwen3 Coder Plus) | 32.2% | 28.6% | 25.2% | 36.4% | 32.3% | 28.5% |
| katIA-MA (Gemini 2.5 Flash) | 32.9% | 33.8% | 29.1% | 37.8% | 38.0% | 33.3% |
| Ripple (GPT-4o)† | – | – | – | 28.2% | 36.3% | 25.0% |
Table IV. Macro-averaged precision (Prec), recall (Rec), and F1-score for FQN-matching and CE-matching of the static analysis, katIA-SP across the three models, katIA-MA using Gemini 2.5 Flash, and Ripple†.
For CE-matching, all models substantially improve precision over the static analysis, with Gemini 2.5 Flash achieving the highest precision of 39.9%. Recall also improves for all models except GPT-5.4 Mini, which exhibits a decrease of 2.0 percentage points compared to the static analysis. Overall, the single prompt F1-scores for two of the three LLMs are slightly higher than those achieved by RIPPLE, which reached a precision of 28.2%, a recall of 36.3%, and an F1-score of 25.0%.
In addition to their predictive performance, we evaluated the consistency of the results obtained by the three LLMs. For each sample, we compared the impact sets from the three repetitions, irrespective of their order, and counted the number of commits for which all three impact sets were identical. GPT-5.4 Mini was the least consistent model, producing consistent results for only 57 out of the 263 commits. Qwen3 Coder Plus produced consistent results for 110 out of 263 samples. Gemini 2.5 Flash achieved the highest consistency with consistent results for 262 commits, whereas the only inconsistent commit was when the model produced an empty impact set for two of the three repetitions, while the third repetition’s result identified one correct impacted entity and missed the other three. The consistency analysis shows that
LLMs differ not only in predictive performance but also in result consistency, reinforcing recommendations for empirical studies using LLMs in software engineering [31].
We answer RQ1: What is the performance of katIA-SP using static analysis and a single prompt to analyze the code change impact? as follows: Compared to the static analysis results, all evaluated LLMs produce an impact set with higher precision and F1-scores. The best performance is achieved by Qwen3 Coder Plus, resulting in an F1-score of 25.2% for FQN-matching and 28.5% for CE-matching. Two of our three evaluated models outperform Ripple in terms of precision and F1-score. Qwen3 Coder Plus returned only 41.83% consistent results, while Gemini 2.5 Flash achieved comparable results of 22.4% and 27.0% F1-score, with 99.62% consistent results. GPT-5.4 Mini performed worst with a CE-matching F1-score of 18.5% and only 21.67% consistent results.
Evaluation of katIA-MA
Despite performing best, Qwen3 Coder Plus produced highly inconsistent results. Therefore, we used only Gemini 2.5 Flash to evaluate katIA-MA. The respective performance is depicted in the penultimate row in Table IV. Similar to the experiment with katIA-SP, we performed three repetitions and used the median result per commit to compute the performance metrics. As with katIA-SP, CE-matching achieves better performance across all metrics. Furthermore, katIA-MA with Gemini 2.5 Flash produced consistent answers across all samples.
For CE-matching, recall and F1-score increase from 26.7% and 27.0% to 33.3%, whereas precision decreases from 39.9% to 37.8% compared to katIA-SP with Gemini 2.5 Flash. katIA-MA outperforms Ripple for all metrics in the CE-matching evaluation. For instance, katIA-MA’s F1-score is 33.3% while Ripple in its best setting achieved an F1-score of 25%. Compared to katIA-SP, katIA-MA improves all FQN-matching metrics, increasing precision from 31.9% to 32.9%, recall from 23.1% to 33.8%, and F1-score from 22.4% to 29.1%. Overall, we observe that katIA-MA particularly improves the recall while reaching similar or slightly better precision than the other approaches.
In addition to performance metrics and result consistency, we report token usage, cost, and generation time obtained from the respective OpenRouter logs. Table V lists the median and quartiles ([Q1, Q3]) across the 263 commits for katIA-SP and katIA-MA, both using Gemini 2.5 Flash and the median values among the three repetitions. In comparison to katIA-SP, katIA-MA increased median token usage by 5.2×, median generation time by 19.7×, and costs by 13.0× observed using the pricing options available at the time of the experiments (July 2026). It shows that katIA-MA comes with higher token usage, and thus, higher cost and generation time due to its three model calls instead of katIA-SP’s single prompt approach. Despite that increase, both time and cost are within an acceptable range while improving both performance and result consistency.
| Variant | Generation Time (s) | Token Usage (×10^3) | OpenRouter Cost (US cents) |
| katIA-SP | 0.570 [0.444, 0.738] | 1.235 [1.023, 1.774] | 0.048 [0.036, 0.064] |
| katIA-MA | 11.214 [8.298, 14.301] | 6.462 [4.953, 8.836] | 0.626 [0.472, 0.807] |
Table V. Resource usage of katIA-SP and katIA-MA with the model Gemini 2.5 Flash. Median and the first and third quartiles ([Q1, Q3]) were extracted across all 263 commits using their median over the three respective repetitions. All values were extracted from the corresponding OpenRouter logs.
With these results, we answer RQ2: To what extent does katIA’s multi-agent variant (katIA-MA) improve change impact analysis compared to katIA-SP? as follows: katIA-MA outperforms the single prompt approach katIA-SP and achieves higher performance metrics than the approach RIPPLE. Compared to katIA-SP and using Gemini 2.5 Flash, katIA-MA increases the FQN-matching F1-score from 22.4% to 29.1% and the CE-matching F1-score from 27.0% to 33.3%. Despite the increased performance, katIA-MA needs substantially more resources than katIA-SP. Nevertheless, these values stay within a reasonable range.
Discussion
This section discusses potential implications of the results and threats to the validity of our dataset CONSTANTINOPLE, approach katIA, and evaluation.
Implications
Our results show that the combination of static code analysis and LLMs benefits code change impact analysis. The consistent increase in precision across the experiments indicates that the LLMs primarily act as filters of the initial static impact set. The static analysis includes entities connected through program dependencies, even when they are not relevant for the given seed changes. By additionally considering the seed changes, their specific source code changes, and the corresponding bug report, LLMs can remove candidates that are structurally related but unlikely to need modification as a consequence of the seed changes. However, such filtering introduces a trade-off between precision and recall as filtering false positives increases precision, but it may also wrongly remove impacted entities and thereby reduce the recall. The results achieved by katIA-MA indicate that performing impact set refinement across three specialized agents, each responsible for exactly one task, achieves similar precision but higher recall. To exactly map the performance and consistency improvements to the agentic setup, an ablation study needs to be performed, as our results do not report whether the improvement is due to the setup, decomposed prompt, structured intermediate results, or additional model calls.
In addition to predictive performance, our experiments demonstrated that result consistency is an important criterion when evaluating LLMs for code change impact analysis. While Qwen3 Coder Plus and Gemini 2.5 Flash both achieved comparably high F1-scores, the results of Qwen3 Coder Plus were less consistent, reducing the practical usability of the approach.
Despite improving the state-of-the-art in code change impact analysis, the absolute performance values indicate that it remains a challenging problem. Even katIA-MA, which performed best in our experiments, missed a large number of impacted code entities, highlighting the complexity of code change impact analysis.
Threats to Validity
We discuss threats to external and internal validity, covering both the generalizability of our findings and potential bias in our study design and execution.
External Validity
External Validity: Our curated dataset CONSTANTINOPLE faces several potential threats to external validity. First, the projects contained in the dataset comprise only open-source Apache Java projects. Hence, our results might not generalize to other programming languages or projects outside the Apache ecosystem. This limitation stems from using the set of projects used in ALEXANDRIA, a well-established dataset for code change impact analysis. While we cannot remove this limitation, we improved the data quality of ALEXANDRIA by proposing our curated dataset CONSTANTINOPLE. Another threat is that the considered projects are not uniformly represented in the dataset, as we randomly sampled commits from the projects rather than enforcing an equal number of commits per project. This imbalance might bias the results towards the projects with a large number of commits. However, Table II shows that the majority of projects have between 5 and 20 commits, and only one project has 56 commits.
We randomly sampled 400 commits from the 918 in ALEXANDRIA and finally obtained a dataset that contained 263 manually labeled commits. This provides a margin of error of 5.11% within a 95% confidence level, which we deem acceptable. We excluded commits without both seed and impacted code entities as keeping them in our dataset might introduce a selection bias. Similar works also followed this criterion, as they always considered a minimum of two changed methods and then applied their heuristics to determine the respective seed change. Furthermore, we considered only commits with one to five changed production code files, which does not cover large commits. However, in the original dataset, 824 of 918 commits are within this range, and only 94 are larger commits. Therefore, our restriction excludes a relatively small proportion of commits which we consider acceptable.
We also did not include files other than Java production code files in our analysis. We excluded both test files and configuration files. Consequently, our approach cannot identify impacts outside the production code. Providing information on Java test files or meta-files, such as XML files, might improve the detection of change impact.
Finally, we performed the experiments with three LLMs and the results might not generalize to all LLMs. Many LLMs exist, and the rapid developments reveal new versions frequently. We mitigated this threat by selecting models from three different model families, including both general-purpose and code-focused models.
Internal Validity
One potential threat to internal validity concerns the labeling of the commits. We mitigated this threat by performing the labeling with three experts and in three phases: a calibration phase, a shared phase, and an individual labeling phase. The first two phases were used to evaluate and refine the rules and guidelines then used for labeling the commits in the individual phase. We considered the informal agreement in the shared phase sufficient to proceed with the individual labeling. Since there was no dedicated overlap in the individual phase, we did not measure an agreement for this share of data. However, the previous labeling phases helped to reduce potential errors.
The prompt is the fundamental part of any LLM call, and therefore represents a potential source of bias. To mitigate the risk of introducing a prompt bias, we carefully designed all prompts used in this study according to findings and guidelines of recent software engineering research [25], [26], [31].
In addition, LLMs have many hyperparameters that can lead to different results. We tested all LLMs using their default hyperparameter settings and a temperature parameter set to 0, to mitigate this threat. Furthermore, we addressed LLM non-determinism in our experiments by performing three repetitions per experiment to assess consistency in the results. We also addressed potential model version drift by providing the model identifier and provider to improve reproducibility.
Another threat to internal validity is the comparability of our work to recent LLM-based code change impact analysis approaches, such as RIPPLE [5]. We performed a best-effort comparison by aligning our evaluation with their more coarse-grained matching strategy. We compared their reported average performance metrics with an approximation of ours, using simple class and entity names rather than file and entity names. This favors RIPPLE because our entity-name matching also considers inner classes, while the file name used by RIPPLE does not. Furthermore, we provide an evaluation based on FQN-matching, which also addresses method overloading. Finally, RIPPLE considers only method changes and one seed change, whereas katIA also considers field and class changes, newly added methods, and multiple seed changes, which, in our opinion, represents a more realistic setting because not all bugs are fixed by changing one method.
Conclusion and Future Work
Code change impact analysis is a fundamental activity in software development. However, this activity is still often performed manually by software developers because existing approaches struggle with either low precision or low recall. Moreover, appropriate benchmarks are missing to thoroughly evaluate change impact analysis approaches.
Addressing the problem with the benchmark, we first presented CONSTANTINOPLE, a curated dataset consisting of 263 commits from 22 open-source Java projects and a total of 1,931 manually labeled changed source code entities. By labeling the changed entities as either seed, impacted, or non-related, we provide a more realistic ground truth for code change impact analysis, compared to prior work [5], [8].
Next, we presented our approach katIA that combines static impact analysis with the reasoning capabilities of LLMs. Regarding the integration of LLMs into our approach, we studied two variants, one using a single prompt called katIA-SP and one using three specialized agents called katIA-MA. Using CONSTANTINOPLE, we evaluated both variants, and, in addition to the classical performance metrics, precision, recall, and F1-score, we also considered the consistency of the results by repeating each experiment three times.
The results for katIA-SP show that the LLM refines the initial static impact set. It improves the FQN-matching F1-score from 14.6% to 22.4% with Gemini 2.5 Flash and to 25.2% with Qwen3 Coder Plus when considering the fully qualified name of code entities. Furthermore, it achieved CE-matching F1-scores 2.0 and 3.5 percentage points higher than the score reported for RIPPLE [5]. Gemini 2.5 Flash produced consistent results and was used for evaluating katIA-MA. The results for katIA-MA using Gemini 2.5 Flash showed a further increase in precision, recall, and F1-score under FQN-matching, with the latter reaching 29.1%. Under CE-matching, the F1-score reached 33.3%. Overall, the results show that katIA-MA with Gemini 2.5 Flash further improves the performance while keeping inference time and costs low and that the consistency of predictions should be considered and reported by software engineering studies involving LLMs.
Despite these improvements, code change impact analysis remains a challenging problem. In future work, we plan to further improve katIA-MA by providing access to the full source code and equipping it with specific tools [32]. We also plan to extend CONSTANTINOPLE with more commits from more projects and enrich our labels with change-type information based on the taxonomy of Fluri et al. [33]. Furthermore, future work should investigate the factors that caused katIA-MA’s performance improvement through ablation studies and analyze the conversation logs for further information about the contribution of the individual agents and their patterns.
Acknowledgment
We used OpenAI Codex to assist with coding, literature search, and polishing of the prompts and the paper. The authors reviewed and refined the generated material and take full responsibility for the final manuscript.
References
- [1]M. M. Lehman, “Programs, life cycles, and laws of software evolution,” Proceedings of the IEEE, vol. 68, no. 9, pp. 1060–1076, 1980.DOI
- [2]S. Lehnert, “A taxonomy for software change impact analysis,” in Proceedings of the 12th International Workshop on Principles of Software Evolution and the 7th Annual ERCIM Workshop on Software Evolution, ser. WIPSE-EVOL ’11. New York, NY, USA: Association for Computing Machinery, 2011, p. 41–50. [Online]. Available: https://doi.org/10.1145/2024445.2024454
- [3]M. Kretsou, E.-M. Arvanitou, A. Ampatzoglou, I. Deligiannis, and V. C. Gerogiannis, “Change impact analysis: A systematic mapping study,” Journal of Systems and Software, vol. 174, p. 110892, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S016412122030282XDOI
- [4]A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges of modern code review,” in 2013 35th International Conference on Software Engineering (ICSE), 2013, pp. 712–721.
- [5]A. Yadavally and T. N. Nguyen, “From seed to scope: Reasoning to identify change impact sets,” in Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering, ser. ICSE ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 400–412. [Online]. Available: https://doi.org/10.1145/3744916.3773265
- [6]X. Ren, F. Shah, F. Tip, B. G. Ryder, and O. Chesley, “Chianti: a tool for change impact analysis of java programs,” in Proceedings of the 19th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, ser. OOPSLA ’04. New York, NY, USA: Association for Computing Machinery, 2004, p. 432–448. [Online]. Available: https://doi.org/10.1145/1028976.1029012
- [7]D. Zhou, J. Zhang, P. Yu, and W. Guo, “Enhancing change impact prediction by integrating evolutionary coupling with software change relationships,” in Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ser. ESEM ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 49–60. [Online]. Available: https://doi.org/10.1145/3674805.3686668
- [8]Y. Yan, N. Cooper, K. Moran, G. Bavota, D. Poshyvanyk, and S. Rich, “Enhancing code understanding for impact analysis by combining transformers and program dependence graphs,” Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/36433770
- [9]A. Khare, S. Dutta, Z. Li, A. Solko-Breslin, R. Alur, and M. Naik, “Understanding the effectiveness of large language models in detecting security vulnerabilities,” in 2025 IEEE Conference on Software Testing, Verification and Validation (ICST), 2025, pp. 103–114.arxiv.org/abs/2311.16169
- [10]L. Fruntke and J. Krinke, “Automatically fixing dependency breaking changes,” Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3729366
- [11]C. S. Xia and L. Zhang, “Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2024. New York, NY, USA: Association for Computing Machinery, 2024, p. 819–831. [Online]. Available: https://doi.org/10.1145/3650212.3680323
- [12]Z. Yang, C. Gao, Z. Guo, Z. Li, K. Liu, X. Xia, and Y. Zhou, “A roadmap for modern code review: Challenges and opportunities,” ACM Trans. Softw. Eng. Methodol., vol. 35, no. 9, Aug. 2026. [Online]. Available: https://doi.org/10.1145/3800963
- [13]C. Macho, K. Stengg, and M. Pinzger, “Agent-based dependency-related build repair,” in 2026 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2026, pp. 1–6.DOI
- [14]K. Stengg, C. Macho, and M. Pinzger, “katIA: Hybrid source code change impact analysis,” in Proceedings of the 1st International Workshop on Agentic AI for Next-Generation Software Development, ser. AgenticDev ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 177–178. [Online]. Available: https://doi.org/10.1145/3843282.3844420
- [15]M. Gethers, B. Dit, H. Kagdi, and D. Poshyvanyk, “Integrated impact analysis for managing software changes,” in 2012 34th International Conference on Software Engineering (ICSE), 2012, pp. 430–440.
- [16]B. Dit, M. Wagner, S. Wen, W. Wang, M. Linares-Vásquez, D. Poshyvanyk, and H. Kagdi, “ImpactMiner: a tool for change impact analysis,” in Companion Proceedings of the 36th International Conference on Software Engineering, ser. ICSE Companion 2014. New York, NY, USA: Association for Computing Machinery, 2014, p. 540–543. [Online]. Available: https://doi.org/10.1145/2591062.2591064
- [17]Q. Hanam, A. Mesbah, and R. Holmes, “Aiding code change understanding with semantic change impact analysis,” in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2019, pp. 202–212.
- [18]Y. Huang, J. Jiang, X. Luo, X. Chen, Z. Zheng, N. Jia, and G. Huang, “Change-patterns mapping: A boosting way for change impact analysis,” IEEE Transactions on Software Engineering, vol. 48, no. 7, pp. 2376–2398, 2022.DOI
- [19]S. Herbold, A. Trautsch, B. Ledel, A. Aghamohammadi, T. A. Ghaleb, K. K. Chahal, T. Bossenmaier, B. Nagaria, P. Makendonski, M. N. Ahmadabadi, K. Szabados, H. Spieker, M. Madeja, N. Hoy, V. Lenarduzzi, S. Wang, G. Rodríguez-Pérez, R. Colomo-Palacios, R. Verdecchia, P. Singh, Y. Qin, D. Chakraborti, W. Davis, V. Walunj, H. Wu, D. Marcílio, O. Alam, A. Aldaeej, I. Amit, B. Turhan, S. Eismann, A.-K. Wickert, I. Malavolta, M. Sulír, F. Fard, A. Z. Henley, S. Kourtzanidis, E. Tuzun, C. Treude, S. M. Shamsabadi, I. Pashchenko, M. Wyrich, J. Davis, A. Serebrenik, E. Albrecht, E. U. Aktas, D. Strüber, and J. Erbel, “A fine-grained data set and analysis of tangling in bug fixing commits,” Empirical Software Engineering, vol. 27, no. 6, p. 125, Jul. 2022. [Online]. Available: https://doi.org/10.1007/s10664-021-10083-5
- [20]K. Stengg, C. Macho, and M. Pinzger, “A dataset and preliminary study of using gpt-5 for code-change impact analysis,” 2025. [Online]. Available: https://arxiv.org/abs/2512.19481
- [21]G. Rodríguez-Pérez, G. Robles, A. Serebrenik, A. Zaidman, D. M. Germán, and J. M. Gonzalez-Barahona, “How bugs are born: a model to identify how bugs are introduced in software components,” Empirical Software Engineering, vol. 25, no. 2, pp. 1294–1340, Mar. 2020. [Online]. Available: https://doi.org/10.1007/s10664-019-09781-y
- [22]M. Dias, A. Bacchelli, G. Gousios, D. Cassou, and S. Ducasse, “Untangling fine-grained code changes,” in 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), 2015, pp. 341–350.arxiv.org/abs/1502.06757
- [23]R. Pawlak, M. Monperrus, N. Petitprez, C. Noguera, and L. Seinturier, “Spoon: A library for implementing analyses and transformations of java source code,” Software: Practice and Experience, vol. 46, no. 9, pp. 1155–1179, 2016. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/spe.2346DOI
- [24]R. Passonneau, “Measuring agreement on set-valued items (masi) for semantic and pragmatic annotation,” in Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC), May 2006. [Online]. Available: https://doi.org/10.7916/D8PV6TTT
- [25]Y. Sasaki, H. Washizaki, J. Li, N. Yoshioka, N. Ubayashi, and Y. Fukazawa, “Landscape and taxonomy of prompt engineering patterns in software engineering,” IT Professional, vol. 27, no. 1, pp. 41–49, 2025.
- [26]G. Wang, Z. Sun, S. Ye, Z. Gong, Y. Chen, Y. Zhao, Q. Liang, and D. Hao, “Do advanced language models eliminate the need for prompt engineering in software engineering?” ACM Trans. Softw. Eng. Methodol., vol. 35, no. 8, Jul. 2026. [Online]. Available: https://doi.org/10.1145/3771933
- [27]Q. Ma, W. Peng, C. Yang, H. Shen, K. Koedinger, and T. Wu, “What should we engineer in prompts? training humans in requirement-driven llm use,” ACM Trans. Comput.-Hum. Interact., vol. 32, no. 4, Aug. 2025. [Online]. Available: https://doi.org/10.1145/3731756
- [28]L. Fan, J. Liu, Z. Liu, D. Lo, X. Xia, and S. Li, “Exploring the capabilities of llms for code-change-related tasks,” ACM Trans. Softw. Eng. Methodol., vol. 34, no. 6, Jul. 2025. [Online]. Available: https://doi.org/10.1145/3709358
- [29]L. Salewski, S. Alanzi, I. Rio-Torto, E. Schulz, and Z. Akata, “In-context impersonation reveals large language models' strengths and biases,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 72 044–72 057. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/file/e3fe7b34ba4f37f8df39cb12a9713f41-Paper-Conference.pdfarxiv.org/abs/2305.14930
- [30]S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, “An empirical study of the non-determinism of chatgpt in code generation,” ACM Trans. Softw. Eng. Methodol., vol. 34, no. 2, Jan. 2025. [Online]. Available: https://doi.org/10.1145/3697010
- [31]S. Baltes, F. Angerer, C. Arora, M. M. Barón, C. Chen, L. Böhme, F. Calefato, N. Ernst, D. Falessi, B. Fitzgerald, D. Fucci, J. He, C. Treude, M. Kalinowski, S. L. L. Zambase, D. Russo, M. Lungu, C. M. Montes, L. Prechelt, P. Ralph, R. van Tonder, and S. Wagner, “Guidelines for Empirical Studies in Software Engineering involving Large Language Models,” Empirical Software Engineering, 2026. [Online]. Available: https://arxiv.org/abs/2508.15503
- [32]T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raaileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 68 539–68 551. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/file/d842425e4bf79ba039352da0f658a906-Paper-Conference.pdfarxiv.org/abs/2302.04761
- [33]B. Fluri, M. Würsch, M. Pinzger, and H. Gall, “Change distilling: Tree differencing for fine-grained source code change extraction,” IEEE Transactions on Software Engineering, vol. 33, no. 11, pp. 725–743, 2007.DOI