betaFeedback

WorksComputer Science

Beyond Benchmarks: Evaluating Large Language Models and Agentic Patterns for Static Code Analysis Warning Classification

Abstract

Context: Static Code Analysis (SCA) tools support software verification and validation, reducing the time and cost of code reviews. However, the best SCA tools in the practitioner community generate a high volume of false positives, which developers tend to ignore, leading to genuine warnings being overlooked. Accordingly, studies increasingly adopt Large Language Models (LLMs) to identify and reduce false positives. Existing studies have produced promising results on public datasets, but underperformed on unseen datasets. Moreover, researchers do not leverage the full potential of Agentic patterns in LLMs or incorporate established code quality standards for warning classification. Objectives: This study utilises four manually annotated datasets to evaluate the performance of three contemporary LLMs for SCA warning classification in real-world applications, and incorporates Agentic patterns and established coding standards to enhance models' performance, addressing gaps identified in existing studies. Method: Three LLMs were selected using the LiveCodeBench benchmark, and the Structured Chain-of-Thought (SCoT) prompting technique was used to evaluate these LLMs. Furthermore, we incorporated Agentic patterns, such as Debating and Self-Consistency, to potentially improve performance in SCA warning classification. Results: Our results show that the performance of different LLMs varies with the characteristics of the dataset, ranging from 0.52 to 0.96 F1-Score for the best models. Additionally, the use of Agentic patterns does not improve LLMs' performance in most scenarios. Conclusions: SCA warning classification using LLMs works best only with certain warning types. We recommend evaluating LLMs in real-world applications rather than using publicly available datasets/benchmarks to understand their full utility.

Introduction

Static Code Analysis (SCA) tools play a crucial role in software verification and validation. However, according to existing studies, SCA tools exhibit high false positive1 warning rates, wasting developers’ time and effort in adjudication [1, 2]. Unfortunately, an excessive number of false-positive warnings may lead developers to ignore all warnings [1, 3]. Thus, considerable effort is invested to reduce false positives by classifying or ranking SCA warnings [1, 4]. Nonetheless, these efforts do not fully leverage recent AI advancements, particularly Large Language Models (LLMs), for classifying SCA warnings [4, 5, 6]. LLMs are trained on extensive collections of natural language and software source code, enabling them to perform tasks such as code understanding and summarisation, debugging, code generation, test case generation, and documentation [7, 8, 9]. By utilising those capabilities, several previous studies utilised LLMs for SCA warning classification [2, 4, 6, 10]. However, multiple gaps in these studies prevent these models from being practically applied in software engineering.

For instance, as summarised in Appendix A, existing studies observed good classification performance on publicly available datasets. However, many frontier LLMs may already have been trained on these publicly available datasets, so these LLMs may classify SCA warnings by memorising what they have already seen rather than generalising knowledge to classify them [6, 11, 12]. Despite higher performance in existing studies, implementing these approaches to classify real-world SCA warnings2 may yield unsatisfactory results. Conversely, different companies and software project teams may classify SCA warnings differently as true or false positives according to their individual preferences [13, 14]. Some teams may use established standards, while others might use a different classification based on their personal experience. Therefore, software developers must choose the appropriate standards based on their preferences over the suggestions generated by LLMs. Furthermore, advanced prompting and decoding techniques such as “Debating” and “Self-Consistency” require further evaluation across different real-world datasets [15]. Therefore, it is worth considering advanced techniques to improve SCA warning classification performance.

To address these gaps, two research questions were defined. The first research question was defined to address the first two gaps, using our own manually annotated SCA warning datasets, incorporating defined standards to classify warnings [16, 17]. The second research question was defined to address the third gap outlined in this section, aiming to improve the performance of SCA warning classification by integrating Agentic Patterns. The defined RQs are as follows,

  1. What is the performance of pre-trained models in classifying SCA warnings using established coding standards?

  1. How do the Agentic Patterns (Debating and Self-Consistency) elevate the performance of the pre-trained best models in classifying warnings generated by SCA tools?

Our study makes several contributions to academia and the practitioner community as follows.

  • We evaluated the three best LLMs based on the LiveCodeBench benchmark3 across our four datasets, which have not been seen by LLMs before. Thus, we are the first to systematically demonstrate LLMs’ performance on unseen datasets.

  • We demonstrate how to incorporate company/individual coding standards when classifying SCA warnings by LLMs. This facilitates the practitioner community in adopting their defined standards for SCA warning classification rather than using standards that LLMs have seen before.

  • We demonstrated that, despite using advanced prompting and decoding techniques, such as Self-Consistency and Debating, the performance of SCA warning classification does not improve when the dataset is unfamiliar to the LLM.

The rest of the study is organised as follows. § 2 discusses the background of the study, and § 3 reviews the existing studies related to the use of LLMs to classify SCA warnings. § 4 outlines the datasets used in our study. Furthermore, § 5 documents the methodology followed, and § 6 summarises the results obtained. We then discuss our results and their implications in § 7. In § 8, we outline potential threats to the validity of our outcomes. Finally, § 9 concludes the work and highlights potential areas for future research. Moreover, we provide a replication package containing the raw experimental results and the complete source code used to analyse the LLMs, enabling full reproducibility of our findings [18].

Background

Performance metrics used in this study

Several performance metrics, such as MCC, F1-Score, Balanced Accuracy and the time taken to analyse an SCA warning were utilised to evaluate the performance of LLMs in this study. The Matthews Correlation Coefficient (MCC) is a comprehensive performance metric for evaluating binary classifiers, especially when the dataset is imbalanced [19]. The MCC uses all four categories of the confusion matrix: true positives, true negatives, false positives, and false negatives to measure the performance, making it valuable when working with imbalanced datasets. On the other hand, metrics such as accuracy can be deceptive and yield misleading insights into the performance of SCA warning classifiers. A MCC score of +1 indicates perfect predictions, while zero signifies performance no better than random guessing. Moreover, the MCC score of -1 represents complete disagreement between predictions and actual outcomes. In summary, MCC reliably indicates model effectiveness, serving as an essential tool for researchers and practitioners in machine learning and statistical analysis [20].

The F1-Score is a statistical performance metric used to evaluate a binary classification model. It is the harmonic mean of Precision and Recall, balancing the two metrics. Here, Precision measures the accuracy of positive predictions, while Recall (or Sensitivity) measures the model's ability to identify all relevant instances [21, 22]. The F1-Score varies from 0 to 1, with 1 representing perfect Precision and Recall, and 0 indicating the lowest performance. It is a valuable metric in situations with class-imbalance, as it accounts for both false positives and false negatives [22].

Balanced Accuracy is a better performance metric than accuracy, as accuracy can be misleading when one class dominates the dataset. However, Balanced Accuracy solves this issue by averaging Sensitivity and Specificity. Sensitivity measures how effectively the model detects the positive class, and specificity assesses how well the model identifies the negative class. Consequently, Balanced Accuracy measures the performance of the binary classification model in terms of correctly classifying both true positives and false positives, making it especially helpful for assessing models on imbalanced datasets, where one class may dominate the other [20]. By giving equal attention to both classes, Balanced Accuracy ensures that the model performs well across all categories, not just the majority class [20].

The time taken to analyse an SCA warning is the average time per analysis. This metric measures the latency of generating a prediction for an SCA warning. It helps select suitable LLMs and decoding methods to develop SCA warning classifiers, particularly when dealing with large datasets or real-time systems. This measure helps balance performance in terms of metrics such as F1-Score, Balanced Accuracy, and MCC with the limitations of available computational resources. Consequently, classifiers that take less time to analyse an SCA warning would be preferred.

Purposes of building a classifier for SCA warnings

A comprehensive literature study was conducted in § 3. However, none of the existing studies considered the potential purposes of building a classifier for SCA warnings. Instead, existing studies have only tried to evaluate their approaches using the percentage of false positive warning reduction [2, 4, 5, 10, 23, 24, 25]. Therefore, we identified four primary goals for this study to classify warnings generated by SCA tools. Additionally, we selected suitable performance metrics to evaluate these goals (Refer to § 2.1).

The first goal is to maximise the overall performance of SCA warning classification. We can assess the overall performance of LLMs in classifying SCA warnings using metrics such as accuracy, which offers a solid understanding of how effectively all warnings are classified. However, the datasets used in this study are biased towards actual or false positive warnings. Therefore, relying solely on accuracy provides less reliable insights into overall performance. As a result, we employed the MCC performance metric to evaluate the effectiveness of the SCA warning classification task, since this study involved binary classifications using LLMs.

The second goal of using LLMs to classify SCA warnings is to detect and reduce false positive warnings as much as possible. This is the most frequently discussed goal in existing studies [2, 4, 6, 10]; however, other goals also exist that can be achieved in similar studies. This helps to lessen developers’ frustration with identifying false positive warnings. However, LLMs that identify as many false positive warnings as possible should not cause actual warnings to be misclassified as false positive warnings. In our dataset, false positive warnings are labelled as “1” and actual warnings as “0”. Therefore, we need to find an approach that delivers excellent performance on a metric that maximises true positives (False positive warnings in our dataset) and minimises false positives (Actual warnings in our dataset) and false negatives (False positive warnings misclassified as actual warnings). In this context, reducing false positives is mandatory, and ideally, it would be preferable to have zero false positives in an ideal situation. Because a model would classify actual warnings as non-actionable, developers will ignore these warnings without further inspection. Conversely, reducing false negatives is also important, as it would prevent false positive warnings from being misclassified as actual warnings, thereby saving the efforts of developers from unnecessary investigation. However, minimising false negatives is less critical than reducing false positives. Considering these factors, the F1-Score is the most suitable performance metric for the second goal. The F1-Score is the harmonic mean of Precision and Recall [11]. When the F1-Score reaches 1.0, all false positive warnings are correctly identified as false positives, and all warnings classified as false positive warnings are actually false positives. In this ideal scenario (F1-Score = 1.0), developers can disregard all the identified false-positive warnings and focus on the remaining ones. Therefore, if the aim is to optimise the classifier for this goal, the objective is to find the best approach or LLMs with the highest F1-Score. Consequently, developers can eliminate all warnings classified as positives, though some actionable warnings might also be removed, especially if the F1-Score is not perfect.

Practically, what developers need is to identify actionable warnings. For this, we can identify false positive warnings similar to the second goal and remove them from the entire set of warnings to obtain actionable warnings. However, the second goal was created to maximise the false positive warnings, which include some actionable warnings. Therefore, we defined the third goal to identify the maximum possible actual warnings with a minimum number of false positive warnings, where the loss of actual warnings is minimised. However, false positive warnings may still occur. Considering the confusion matrix, the aim is to maximise true negatives while minimising false positives and false negatives. In this context, Balanced Accuracy (BA) can be employed as a performance metric, taking into account both the True Positive Rate (TPR) and the True Negative rate (TNR) [20]. Using TPR and TNR makes BA especially valuable in situations where the classes are imbalanced. In the ideal scenario where the BA value reaches 1.0, the classifier accurately identifies all actual warnings. Moreover, we do not use Accuracy as a performance metric for this purpose, as our datasets are imbalanced towards either false positives or true positives. In such a scenario, BA is better for class-imbalanced datasets [20]. That said, developers can reduce the time spent on investigating false positive warnings.

In addition to the other goals discussed in this section, the time required to classify a single SCA warning is a practical consideration when implementing the classification model in a production environment. Therefore, we tracked the time taken to classify SCA warnings for each dataset and LLMs separately. All the goals discussed in this section are summarised in Table 1.

Goal IDPurpose of model buildingMetric used
G1Maximise the overall model performanceMCC
G2Correctly detect false positive warnings as much as possible to minimise false warningsF1-Score
G3Correctly identify the maximum possible actual warnings with minimum false positive warningsBalanced Accuracy
G4Reduce the time taken to process an SCA warningTime in seconds

Table 1. Goals for model building

LLMs optimisation with fine-tuning, hyperparameter optimisation and prompt engineering

LLMs can be optimised using various approaches, including fine-tuning, hyperparameter optimisation, and prompt engineering [26, 27, 28]. These different optimisation approaches focus on different aspects of the model. Model fine-tuning involves specialising LLMs for specific tasks, such as source code generation, summarisation, and solving maths questions [29]. However, fine-tuning demands substantial resources and large specialised datasets [30]. Therefore, we did not incorporate fine-tuning for our study. The second method for optimising LLMs is hyperparameter optimisation, which can improve LLM performance without altering the models. Several hyperparameters are relevant when prompting LLMs, including Temperature, Top Probability (top_p), Frequency Penalty, and Presence Penalty [31]. The temperature parameter primarily controls the randomness of the output [31]. Top Probability (top_p) restricts the sampling pool of tokens, limiting the allowed tokens for the next predicted word [31]. Since this study utilises LLMs for binary classification, setting the temperature to a lower level of randomness (temp = 0.2) and setting top_p between 0.75 and 1 (top_p = 0.95 in this study) is advantageous for making the response more factual rather than being creative [31]. Moreover, the Frequency Penalty discourages repeated tokens and encourages more novel outputs [31]. Conversely, the presence penalty reduces the model from generating tokens that have already been generated, promoting diversity in the generated output [31]. Because the model output is expected to include the predicted label (true positive or false positive), a reason for the decision, and any missing details, we set the max_tokens parameter to 16384, because it matches a common 16k context window of LLMs, giving the model room to generate long, uninterrupted outputs without being cut off4 [32]. It should be noted that only the temperature, top_p, and max_tokens hyperparameters significantly influence models’ performance, while other hyperparameters will have a minimal effect on SCA warning classification [31].

Similar to hyperparameter optimisation, prompt engineering is also used to enhance the performance of LLMs without altering internal model parameters [33]. It can be described as the process of designing or shaping natural language instructions to achieve desired behaviours and outputs from pre-trained language models [34]. Instead of updating model weights through fine-tuning, prompt engineering allows seamless integration of pre-trained models into downstream tasks by providing task-specific instructions, called prompts, that guide the model's response generation [33]. Several techniques have been developed for prompt engineering, including Zero-shot and Few-shot learning, as well as Chain-of-Thought (CoT).

According to Google's prompt engineering guidelines5, the Zero-shot prompting technique involves instructing an LLM with a direct request without providing any examples [33]. With this approach, we must depend solely on the LLM’s internal knowledge, unless the relevant information is provided with the prompt. However, we can utilise One-shot, Few-shot, or Multi-shot prompting methods to present one or more examples of desired input-output pairs to the LLM, enhancing its understanding of the specific task and resulting in more accurate responses [33]. Moreover, the CoT prompting technique encourages models to generate intermediate reasoning steps before arriving at a final answer, significantly improving performance on complex reasoning tasks [34]. With CoT, the original problem is decomposed into smaller sub-problems, enabling each to be solved effectively. The results of each sub-problem are then combined to produce the final solution. There are automatic versions of CoT as well, by adding a string similar to “Let’s think step-by-step” to the prompt, which leads the LLM along a CoT generation path without user intervention [33].

Furthermore, there are other prompting techniques, including Self-Consistency and Generated Knowledge [34]. The Self-Consistency prompt engineering method generates multiple reasoning paths and selects the most consistent answer, enhancing the reliability of model outputs [34]. Moreover, the Generated Knowledge approach involves prompting the model to first generate relevant background knowledge before answering the main query, thereby improving factual accuracy [34]. Other prompting techniques are also noted in other works [33, 34, 35]. These do not train the underlying parameters of the LLM, but instead modify the input data and the way calculations are performed in the LLM, resulting in significantly different outcomes. Therefore, the selection of the prompting technique is vital [33].

LLMs are algorithms that predict the next token based on the decoding methods used. There are multiple decoding methods available in LLMs to generate outputs, such as Greedy Search, Beam Search, Temperature Sampling and Top-p Sampling (Nucleus Sampling) [36]. Greedy Search selects the next token with the highest probability. However, this can lead to a local optimal solution, missing the global optimal solution. To address this issue, the Beam Search approach is introduced. Beam Search improves the Greedy Search by keeping track of the kk most probable sequences (Beams) at each step. It explores a broader range of possibilities than greedy search. However, it can still get stuck in local optima and lacks diversity in open-ended generation tasks. In contrast, the Temperature Sampling technique adjusts the sharpness of the entire probability distribution over tokens using a scalar parameter called temperature (T). Higher temperatures (e.g., T > 1) flatten the distribution, increasing the likelihood of selecting lower-probability tokens, leading to more diverse and creative outputs. Lower temperatures (e.g., T < 1) make the distribution sharper, favouring high-probability tokens and resulting in more deterministic and focused text. Moreover, Top-p Sampling (Nucleus Sampling) is an adaptive decoding method that selects the smallest set of tokens whose cumulative probability exceeds a predefined threshold p (e.g., 0.9). This allows the number of considered tokens to vary dynamically based on the probability distribution's shape, providing flexibility and generally producing more natural-sounding text than fixed top-k sampling.

This study aims to classify SCA warnings into actionable and non-actionable categories. For that, the LLM should understand the source code (or the relevant slice of it) and the SCA warning and be able to identify the relevant coding standards to produce the correct response. Since this is a classification task and the output should be factual, the recommended hyperparameters were used to maximise the LLM's performance, with Temperature Sampling and Nucleus Sampling decoding methods. Additionally, for this task, information such as coding standards and rules defined by SCA tools is involved. Fine-tuning LLMs with this information is a good option; however, for that, there should be a significantly large dataset [30]. Therefore, we use existing fine-tuned models with a large set of open-source coding repositories, which improved source code understandability. Consequently, in this study, we aim to improve the performance of SCA warning classification through optimising prompt engineering.

Providing external knowledge using RAG and CAG

Retrieval Augmented Generation (RAG) is utilised to supply external information to the LLM model. In RAG, data relevant to a specific task is searched from a vector database, pre-processed, and presented with the LLM prompt. RAG assists LLMs in accessing current and factual data, particularly for the task. LLMs trained with data available before the cut-off date and data generated after that date are excluded from the LLM [37]. Consequently, LLMs can produce inaccurate results, and their outputs are outdated. To address this, RAG is employed. However, during the data search stage, RAG might retrieve irrelevant data, increase the token size of the prompt and reduce the accuracy of the generated results [38]. To mitigate this issue, we used Context Augmented Generation (CAG) [39], where only the related data is retrieved based on the type of the SCA warning assessed. Consequently, only the relevant data will be fed to the LLM along with a slice of the applicable source code and the SCA warning message.

AST graphs and Data dependency graphs

Abstract Syntax Tree (AST) is used to represent the structure of software source code in a tree-like form, where each node in AST represents a particular construct, such as classes, methods, and variables [40]. AST represents only the structural and content-related details of the source code, while discarding any additional details, such as comments. Information that gets transferred to the AST includes:

  • The variable types and their declaration locations

  • The order and definition of executable statements

  • The left and right components of binary operations

  • Identifiers and their assigned values

AST graphs are used in this study to extract program slices from source code where the relevant lines are spread throughout, and the relationship between these lines depends on their structure. In contrast, a data dependency graph is a directed graph showing the dependencies between data items in source code [41]. For example, within source code, a variable definition with an initial value creates data dependencies through the modification of that variable and its subsequent use. In this study, data dependency graphs are utilised to extract code slices from source code, where relevant lines are dispersed throughout the code and their relationships depend on the data or identifiers/variables.

Related work

In prior studies, several efforts have been made to classify SCA warnings using LLMs, and this section reviews these approaches. Kharkar et al. [23] used approaches, such as a feature-based model utilising handcrafted features, DeepInferEnhance (a fine-tuned CodeBERTa model on Java and C# functions), and GPT-C (a variant of the GPT-2 code completion model in a Zero-shot setting) to classify SCA warnings. Using a dataset consisting of 539 null dereference warnings from open-source and proprietary projects, they demonstrate that DeepInferEnhance improves Precision by 15.13% and GPT-C achieves the best Precision gain (+17.47%) without additional training. Additionally, the approach generalises to resource leak warnings, where GPT-C improved Precision by 5.56%. In this study, Kharkar et al. [23] demonstrated the potential for reducing the false positive rate in the SCA warning dataset. However, the primary drawback of this study is that the Recall value decreases from 100% to 83.7% in GPT-C when compared to the baseline.

Ge et al. [24] systematically examined the feasibility of using Pre-Trained Models (PTMS), which are trained on billions of code tokens for SCA warning classification. They assessed five representative PTMs, including CodeBERT, GraphCodeBERT, CodeT5, UniXcoder, and CodeGPT. These PTMs encompass encoder-only, decoder-only, and encoder-decoder architectures. The evaluation was conducted on a dataset comprising over 10,000 SpotBugs warnings from 10 large-scale Java projects. All studied PTMs significantly surpass the State-Of-The-Art (SOTA) deep learning-based Actionable Warning Identification (AWI) approaches, such as CNN and LSTM. PTMs enhanced performance in terms of AUC by a margin of 9.85% to 21.12% over the SOTA baseline models. The CodeBERT model (an encoder-only architecture) achieved the best performance with an AUC of 70.77%. The study indicates that encoder-only PTMs are most suitable for this code classification task compared to encoder-decoder or decoder-only models. The authors of the study provided several practical guidelines to improve actionable warning identification systems, including reducing the context window by minimising the size of source code using techniques such as program slicing, and enhancing AWI by training domain-specific PTMs and fine-tuning these PTMs. Additionally, Ge et al. [24] suggest using a training dataset with diverse patterns SCA warnings to improve AWI performance. This will help to train the AWIs with diverse patterns.

Flynn and Klieber [5] presented an approach for leveraging LLMs, such as GPT-4, to automate the adjudication of Static Analysis (SA) alerts. The authors developed a model where an LLM-based tool ingests source code and SA alerts, then queries the LLM to output adjudicated true positives with a trace, false positives with a proof sketch, or “uncertain”. To mitigate the limitation of the context window, they employed several approaches. One approach is using traditional SCA tools to produce the required information, and another is using the LLM itself to generate function summaries. Additionally, they instructed the LLM in the prompt to ask for the needed information to prevent the limitation of the context window. Furthermore, they used the LLM to create preconditions to avoid a bad state in a function, then had the LLM check whether callers satisfy the precondition. Although the paper did not provide statistical results, it demonstrated several examples of GPT-4 successfully performing complex tasks related to alert adjudication, with historical LLM chat records.

Apostolidis et al. [25] proposed a novel approach to reduce non-actionable warnings generated by CppCheck. They analysed the same software code using both CppCheck and fine-tuned CodeBERT LLM to predict vulnerabilities. If CodeBERT predicted a function as vulnerable, CppCheck warnings within that function were retained as actionable; otherwise, they were classified as non-actionable warnings. They could improve the Precision (from 5.88% to 96.20% in the Big-Vul dataset and from 8.71% to 47.67% in the ReVeal dataset), while Recall decreased from 46.90% to 40.66% on Big-Vul and from 6.88% to 5.93% on ReVeal. Thus, higher Precision was achieved while some genuine detections were lost. However, in this study, SCA warning details were not incorporated into the LLM for its predictions, and CodeBERT classified functions rather than individual warnings. A vulnerable function may contain both actionable and non-actionable warnings, while an incorrect prediction that a function is safe may remove actionable warnings. Additionally, the datasets were collected from public commits linked to known vulnerabilities. Such labels do not confirm the correctness of every SCA warning in those functions. Therefore, Apostolidis et al. [25] provide evidence for selecting functions for review, but more limited evidence for classifying individual warnings.

The previously mentioned attempts do not completely harness the full potentials of LLMs in classifying SCA warnings. These attempts utilised Small Language Models or pre-trained transformers to classify SCA warnings. Additionally, some of them used SCA tools and LLM as an ensemble to analyse code independently, without cooperating with each other [25]. Wen et al. [2] introduced LLM4SA, the first LLM-based SCA warning classifier that analyses both source code and SCA warning messages with an LLM. They used LLMs such as, ChatGPT and Llama-2 and SCA tools such as CppCheck, FB Infer, and Clang Static Analyser. To overcome the token window limitation when analysing large codebases, they employed static program-dependence traversal to extract the relevant slice of the source code, providing sufficient context related to bugs [42]. Additionally, they improved prompting through Few-shot prompting and CoT. Their evaluation of the Juliet benchmark programs, three embedded operating systems and 11 real-world C/C++ projects showed that LLM4SA achieved high Precision (81.13%) and Recall (94.64%) for a total of 9,547 bug warnings. Additionally, they experimented with their approach on real-world applications, and the results were lower than the Juliet benchmark. However, the embedded-system evaluation contained mainly false warnings. Of the 749 warnings, only 13 warnings were actionable. The reported 100% Recall therefore provides limited evidence that actionable warnings would be preserved in other projects. In contrast, approximately 84% of the Juliet warnings were actionable, making high Precision easier to obtain even when some false warnings were retained. These different class distributions limit comparisons between the reported scores. They discussed possible data leakage but did not establish whether the evaluated code had been used in LLM training. If the LLM had seen the same code before, its performance could partly reflect memorisation rather than classification of unfamiliar warnings. Additionally, the study covered six bug categories using category-specific examples, so the results do not establish similar performance for other warning types. LLM4SA incorporated the bug report generated by the SCA tools to provide contextual information. However, they did not use any software source code quality standards or the relevant description of the warning given by the SCA tool in the official documentation. Without this contextual information, the LLM will classify SCA warnings based on its internal knowledge, which might be outdated or incorrect. Moreover, Svärd and Teräs [10] proposed a Large Language Model Assisted Static Analysis (LLMAS) prototype that integrates an LLM (Google Gemini) with an SCA tool (SpotBugs) to identify and reduce false positive warnings generated by this SCA tool. They used 10,056 test cases from the Juliet Java 1.3 dataset, covering 14 of the dataset’s 112 Common Weakness Enumerations (CWEs). Svärd and Teräs [10] benchmarked LLMAS against the standalone SpotBugs, and the results showed that LLMAS improved Precision by 13.50%, from 0.3236, while maintaining the same Recall (0.4914). The selected CWEs also had unequal numbers of test cases, so common warning types may have a greater influence on the overall results. Juliet is also synthetic, and names such as “good” and “bad” may reveal the intended labels. These clues and possible training-data overlap may make the results more favourable than those obtained on unfamiliar real-world code. However, using LLMAS added significant latency, with execution times rising from 7.2 minutes (for SpotBugs) to 4.86 hours (for both SpotBugs and LLMAS), which may limit its use in Continuous Integration/Continuous Development (CI/CD) pipelines when rapid feedback is required.

In contrast, Chen et al. [4] identified several drawbacks in existing studies, including (1) imprecision, where LLM4SA utilises overly broad code snippets such as entire function bodies, cluttering inputs with irrelevant code and overwhelming the LLM, leading to inaccurate assessments, longer processing times, and higher token costs; and (2) incompleteness, where LLM4SA misses essential code context, such as definitions of global variables or external functions, which are vital for accurate data and control flow analysis. To address these issues, Chen et al. [4] proposed LLM4FPM, a lightweight and efficient framework for SCA warning classification. The primary contribution of this study is to extract more precise and complete source code context for the LLM. They identified several challenges in extracting the most relevant code context, including (1) insufficient fidelity or accuracy of the existing program slicer; (2) missing some dependencies in Joern's Code Property Graph (CPG) [43]; and (3) inefficient inter-file dependency extractors. To tackle these, they introduced two components: eCPG-Slicer and Find All Referenced Files (FARF). The eCPG-Slicer performs line-level, precise code slicing by extending Joern's CPG and creating a structured report. Moreover, FARF identifies all necessary dependent files for a given warning, providing the eCPG-Slicer with complete inputs, thereby eliminating the need for costly whole-program analysis. The code slice extracted using eCPG-Slicer, and the SCA warning report are combined into a structured format and fed to a lightweight, open-source LLM (Qwen 2.5-32B). The LLM is guided using specialised techniques such as Expert Prompting [44], CoT reasoning, Few-shot examples, and a Self-Consistency mechanism (majority voting) to determine whether a warning is classified as a “real bug” or a “false alarm”. The results from this study demonstrated that providing precise and complete context significantly improves SCA warning classification using LLMs. Additionally, LLM4FPM outperformed the current SOTA method, achieving an overall F1-Score of 98.92% on the Juliet dataset. However, approximately 85% of the evaluated warnings were actionable. Retaining most warnings can therefore give a high F1-Score for actionable warnings, so this score also needs to be considered alongside false positive removal. The evaluation covered seven CWEs using one LLM and specialised prompts, limiting the evidence for other warning types and models. The authors also acknowledged possible training-data overlap with Juliet, so performance on unseen code remains uncertain. Additionally, LLM4FPM was used to relabel the noisy D2A dataset, with 43 of 50 manually checked examples from its libtiff subset labelled correctly (86%). This small sample from one project does not establish the accuracy of the complete relabelled dataset. Furthermore, testing on eight real-world open-source C/C++ projects demonstrated that the framework successfully reduced false positives by over 85% in the manually checked sample. The system is also effective, with an average inspection time of 4.7 seconds per warning, showing its potential for integration into CI/CD pipelines under the tested hardware and execution settings. Moreover, they evaluated their approach with eight real-world code repositories as well. They analysed those code repositories with SCA tools and selected 50 samples randomly from the warnings generated by SCA tools. Two human experts confirmed that 43 of the samples were false positives. They could not annotate the other 7 warnings. Out of the 43 confirmed warnings 37 cases are classified as false positives by their approach. Since the manually checked real-world sample contained no confirmed actionable warnings, this evaluation does not show whether the approach would incorrectly remove actionable warnings. The seven unresolved cases also limit confidence in its performance on warnings that are difficult to annotate. Therefore, our study conducted an in-depth analysis on this using four manually annotated datasets considering actionable and non-actionable warnings.

On the other hand, Iranmanesh et al. [6] proposed ZeroFalse to reduce false positive warnings while preserving coverage. They experimented with 10 LLMs to classify SCA warnings, which belong to 10 CWEs, generated by CodeQL. They used CWE-specific guidance within a common prompt template to help LLMs understand the context better. Their best model achieved 0.912 in terms of F1-Score for the OWASP Java benchmark and 0.955 on the OpenVuln dataset. The OWASP evaluation contained 1,449 actionable warnings and 525 false positives, with no false positives for two CWEs. Results for these two categories therefore do not show how well the approach rejects false warnings. According to Iranmanesh et al. [6], CWE specialised prompts improved results for most models compared with prompts without information regarding CWEs. However, some models performed worse, so the benefit of this guidance depends on the model and dataset. Additionally, they found that reasoning models outperformed other LLMs. However, their OWASP evaluation covered only 10 CWEs, so the results do not establish similar performance for other warning types. Extending the approach requires preparing and validating guidance for those types. Furthermore, they used a common program slicing approach for different CWEs, but did not evaluate whether warning-specific slice would further improve the results.. Moreover, proprietary LLMs achieved the highest F1-Scores, although some open-weight models also performed well with this approach. Grok-4 and Gemini-2.5 pro were the best models for OWASP benchmark results, and GPT-5 and Grok-4 performed best in OpenVuln benchmark results. In this study, OpenVuln contained 58 warnings from seven open-source Java projects, including 23 actionable warnings and 35 false positives. The small sample limits conclusions about other projects. Its source code and known fixes were publicly available, so the study does not establish that they were unseen by the LLMs. Prior exposure could make the results less representative of unfamiliar code. Labels were assigned by comparison with known fixes, which may not identify other faults in the same code. Therefore, agreement with these labels does not confirm that every rejected warning is non-actionable. Their experiments shows that the OpenVuln dataset underperforms when compared to the OWASP dataset for multiple LLMs. Additionally, they mentioned the necessity of real-world dataset which is not trained for LLMs. In our study, we show that our four datasets provide a good benchmark for evaluating LLMs as they are not trained on LLMs and have different false positive percentages. On the other hand, for practical implementation, the confidentiality of the source code should be considered. To prevent sharing source code with external parties, approaches that work on open-source models, which can be deployed in company environments, should be used. Consequently, there is a necessity for better approaches which consistently produce exceptional results using open-source models.

Appendix A summarised the LLMs, datasets and the reported performance in existing studies and overall. Existing studies have used unnecessarily large source code or slices from the source code to feed LLMs, expecting better performance in SCA warning classification. However, with the increase in input token size, the throughput of the SCA warning classification system decreases, due to the transfer of more data and an increase in processing time in the LLM. Additionally, the performance of LLMs decreased with the rise in input token size, as shown by Levy [38], mainly due to hallucination in LLMs. LLMs also hallucinate due to the lack of necessary information, then attempt to guess the answer with limited data [45]. Moreover, sufficient effort was not invested in effectively using coding standards to judge SCA warnings accurately. Therefore, in this study, we aim to eliminate this gap by providing all the necessary information for LLM to classify SCA warnings.

In summary, existing studies demonstrated the potential for reducing the false positive warning rate in SCA warning datasets [4, 5, 6, 23, 25]. These studies showed that LLMs work better than existing SOTA [24]. However, in our study, we identified that LLMs produce lower performance for datasets which is not seen by the LLM previously. Moreover, only some existing studies utilised the full potential of LLMs, evaluating both source code and SCA warning messages to identify actionable warnings from the set of warnings generated by SCA tools. Studies that used both source code and SCA warning messages as input data achieved promising results in identifying actionable warnings [2, 4, 6, 10]. Additionally, we embedded established source code quality standards with LLM prompts in classifying SCA warnings, as software engineering companies and teams may need to use their own standards rather than using the standards trained by LLMs. Therefore, in our study, we used source code, warning description and established source code quality standards which are relevant for the warning type (SCA rules that warnings were generated).

Moreover, some of the existing studies used different approaches for program slicing to fit the source code and warning messages into the limited token window [2, 4]. However, Chen et al. [4] identified two issues with existing program slices used as inputs to LLMs, such as imprecision and incompleteness. By addressing these two issues, Chen et al. [4] could harness the full potential of LLMs for SCA warning classification. However, in our previous study, manually annotating four SCA warning datasets, we identified that the approach needed for program slicing differs based on the relevant SCA rule [46]. Therefore, we defined several program slicing approaches for individual SCA rules. Moreover, Chen et al. [4] used multiple prompting techniques such as Expert Prompting [44], CoT reasoning, Few-shot examples, and a Self-Consistency mechanism (majority voting) and achieved near-perfect results (99%) on the Juliet public dataset and correctly relabelled 43 of 50 manually checked examples from the libtiff subset of D2A (86%). On the other hand, we used the Structured Chain of Thought (SCoT) prompting technique for this study, as it facilitates the embedding of relevant source code quality standards to the prompt and guide the LLM to generate intended results following specified procedures and rules. Furthermore, we used Agentic patterns such as Self-Consistency and Debating to potentially improve the performance of warning classifiers, extending the body of evidence.

Furthermore, Iranmanesh et al. [6] experimented with 10 LLMs such as Grok-4, Gemini-2.5, GPT-5 and Grok-4 to classify SCA warnings and found that the use of warning information in prompts improved results for most models. Additionally, they found that reasoning models generated higher performance in SCA warning classification. . Extending this approach requires additional CWE-specific guidance, and the study did not evaluate whether customised program slicing for each warning type would improve the results. However, SCA tools introduce and deprecate warning types from time to time [46]. Therefore, . maintaining CWE-specific guidance may require additional effort as warning types change. In our study, we use a common prompt template with the relevant coding standards for each warning type. However, the program slicing approach should be defined based on the warning type. Furthermore, as a next step, we plan to automate this process through an expert model as mentioned in § 9.

Overall, we found several gaps in existing studies which are addressed in the current study. Existing studies did not use sufficient contextual information for SCA warning classification. We used software source code quality standards and the description of the warning given by the SCA tool in the official documentation [2, 10]. We provided this information with each SCA warning classification request. Additionally, we experimented with our approach on real-world datasets that have not been seen by LLMs. Moreover, we tried to optimise the results using better prompting techniques, adding more context and using Agentic AI patterns. Furthermore, we evaluated an open-source model which may be deployed using an on-premise server, which eliminates privacy concerns.

Datasets

Four SCA warning datasets6 [46] were created by analysing two source code datasets (the Stack Overflow dataset collected by Meldrum et al. [47] and the Apache Tomcat source code downloaded from GitHub on 8th December 2023) using two SCA tools (PMD and SonarQube).

The reason for choosing the Stack Overflow and Apache Tomcat datasets is that these two source code datasets have been utilised in multiple existing studies and are recognised as reliable testbeds [48, 49, 50, 51, 52, 53]. Moreover, the Stack Overflow dataset consists of code snippets, whereas Apache Tomcat included full-length source code, thereby creating diversity across the four SCA warning datasets. Additionally, PMD and SonarQube are the leading SCA tools within the software engineering practitioner community and have the lowest false positive rates [54, 55, 56]. Therefore, PMD and SonarQube were used to scan the two source code datasets, generating the four SCA warning datasets. As a result, the four datasets contain SCA warnings from both code snippets and full-length source codes. The dataset summary is shown in Table 2, where the number of specific rules of SCA tools that generated warnings (number of rules triggered in SCA tools for the source code) is indicated as “# of rules found”. Here, a considerably large number of warnings (indicated as “# of alarms” in Table 2) were produced by each SCA tool through the analysis of each code dataset. Furthermore, due to the significant resources required for manually annotating the entire datasets, four sample datasets were created randomly [57], with a 95% confidence level and a 5% margin of error. Thereafter, the sample size of each SCA rule was determined proportionally to the number of warnings in the population dataset [57, 58]. When the proportion of the sample size for a given rule was a fraction, it was rounded up, increasing the total number of samples, which is slightly above the 95% level. The sample sizes for each dataset are listed as “Sample size” in Table 2.

Dataset NameSCA toolCode Dataset# of rules found# of alarmsSample size# of false positives in the sample# of rules generated false positivesFalse positive percentage# of samples for evaluationFalse positive percentage in the final dataset
Dataset 1PMDSOD18880788531981318.45%13970.50%
Dataset 2PMDAT197176293536721613.43%11761.54%
Dataset 3SonarQubeSOD310838906463054.64%15020.00%
Dataset 4SonarQubeAT28112852463144146.97%29814.77%

Table 2. Summary of datasets

Following the defined coding standards from our previous study [16], SCA warnings in the sample datasets were manually reviewed to identify whether they were true positives or false positives. The number of false positive warnings for each dataset is listed as “# of false positives in the sample” in Table 2. Furthermore, the number of rules that produced false positive warnings is indicated as “# of rules generated false positive” in Table 2, and the false positive percentage for each dataset is displayed as “False positive percentage” in Table 2. It is worth noting that most of the rules did not produce any false positive warnings, and the associated warnings were removed, resulting in a dataset size referred to as “# of samples for evaluation” in Table 2. The final false positive percentages of these evaluation datasets are listed as “False positive percentage in the final dataset” in Table 2. These datasets were then used to evaluate LLM models and generate results for each research question as described in this study.

After selecting samples randomly, the sample dataset 1 consists of 531 samples, covering all the rules available in the population dataset [59]. Sample Dataset 1 consists of 98 false positives, which were generated by 13 SCA rules out of a total of 188 rules. Similarly, sample Dataset 2 consists of 536 samples, including 72 false positives arising from 16 SCA rules out of 197 rules, which generated all the SCA warnings in dataset 2. Sample Dataset 3 contains 646 random samples from Dataset 3, including 30 false positives from 5 SCA rules out of a total of 310 rules, which generated all SCA warnings for Dataset 3. Finally, sample Dataset 4 consists of 631 samples, which are randomly selected from Dataset 4, including 44 false positives from 14 SCA rules out of 281 rules that generated all the SCA warnings for dataset 1. Moreover, the standards used to review and identify false positives manually in all the datasets are shown in Table 3, including the reason for selecting them for our study.

IdEstablished StandardsReason for selecting the standard
01https://wiki.sei.cmu.eduWiki page maintained by Carnegie Mellon University and used as a standard for SonarQube
02Book: Clean Code.Used as a standard for SonarQube
03Book: Effective Java by Joshua Bloch.Used as a standard for SonarQube
04Book: Mastering Regular Expressions by Jeffrey Friedl.Used as a standard for SonarQube
05Book: Refactoring: Improving the Design of Existing Code by Martin Fowler, Kent Beck, John Brant, William Opdyke, and Don Roberts.Used as a standard for SonarQube
06Google Java Coding Style Guide.Used as a standard for SonarQube and A popular and well-accepted standard [60, 61]
07https://cwe.mitre.org/data/definitionsUsed as a standard for SonarQube and A popular and well-accepted standard [62, 63]
08Java Doc.Java official user guide
09Book: Object-Oriented Metrics in Practice: Using Software Metrics to Characterize, Evaluate, and Improve the Design of Object-Oriented Systems by Michele Lanza and Radu Marinescu.Used as a reference for PMD
10Paper: Linguistic Antipatterns - What They Are and How Developers Perceive Them.[64]
11Principle of Least Astonishment.[65]
12SOLID principles.Used as a standard for SonarQube
13Transitive property of equality.Used as a standard for SonarQube
14https://owasp.orgA popular and well-accepted standard [60, 66]

Table 3. Established standards

SOD - Stack Overflow dataset, AT - Apache Tomcat

The sample datasets were manually annotated, with false positive warnings marked as “1” and true positive warnings marked as “0”. These datasets were manually annotated in accordance with defined coding standards, with support from a software tool we developed to display all relevant information on one page. This mitigates mistakes that can occur while manually cross-checking source code, warning descriptions generated by SCA tools, rules, and standards. As we discussed in Deshapriya et al. [46], Datasets 1 and 2 were created using warnings generated by PMD and Datasets 3 and 4 were created using warnings generated by SonarQube. Results from Deshapriya et al. [46] show that the majority of the false positives are generated by a few SCA rules, and those rules are common between Datasets 1 and 2 as well as between Datasets 3 and 4. Therefore, we conducted reliability checks only in Datasets 1 and 3. To conduct reliability checking, a sample of 10% of SCA warnings from Datasets 1 and 3 were selected randomly, and two authors reviewed these samples together [67, 68, 69]. We found three data disagreements out of 115 (51 in Dataset 1 and 64 in Dataset 3) at the beginning of the reliability assessment process. The Inter-rater agreement was calculated using Cohen’s Kappa between the authors [70], which was initially found to be 0.6547, indicating substantial agreement between the authors based on the categories provided by Landis and Koch [71]. After conducting the reliability check, all the authors were discussed on the samples which has disagreements and complete agreements (100%) among all the authors on the 115 samples were reached after a round of discussions [70]. Additionally, we reviewed all labels in all four datasets based on the discussion to improve overall annotation accuracy.

Moreover, all rules with no false positives were excluded from the datasets to ensure the fairness of the analysis. In Table 2, the difference between “# of rules found” and “# of rules generated false positive” represents the number of rules removed that did not produce any false positive warnings. This led to a further reduction in the size of the dataset, as shown in Table 2 under “# of samples for evaluation”. Otherwise, including only true warnings for most SCA rules would impose unnecessary overhead on performance analysis. Additionally, for simplicity of explanation, all sample datasets used in the analysis are referred to as Dataset 1, Dataset 2, Dataset 3, and Dataset 4 in other sections of this paper.

Methodology

This section outlines the methodology used to create SCA warning classifiers using LLMs. In building these classifiers, only the required slice of source code that is relevant for the SCA warnings was extracted, and the best LLMs were selected to evaluate these SCA warning classifiers. Moreover, program slicing approaches and methods for selecting the best LLM models for our study, along with LLM decoding techniques, are introduced in this section.

Program Slicing is an important step in feeding source code into an LLM for classifying SCA warnings [4, 6]. This process reduces the size of the source code fed into the LLM and ensures that the fed slices are sufficient for the LLM to classify SCA warnings [4, 6]. Program Slicing helps to speed up the classification process and minimise cost by reducing the LLM token count. After the program slicing step, prompts were designed with the program slices, warning messages and source code quality standards related to the rules that generated SCA warnings. The prompt structure is shown in Appendix § 12.7. We then feed the generated prompts for each SCA warnings to the LLMs and extract results (True Positive/False Positive) from the LLM. We then evaluate these LLMs against four manually annotated datasets [46].

Program Slicing

The datasets were manually reviewed and annotated to determine whether these SCA warnings are true positives or false positives. While manually annotating, we found that only a portion of the source code is relevant for deciding actionable warnings. Additionally, we identified that the Program Slicing approach should be based on the rules of SCA warnings, and warnings generated for different SCA rules require different Program Slicing techniques. Some of the warnings could be annotated by investigating only a few nearby source code lines, and we do not need to check other blocks of the source code (i.e., the issue is localised to a few nearby code lines) [73]. In contrast, some warnings require examining source code lines across different code blocks; however, there is a relationship between these code lines in terms of either data dependency or execution path, which can be identified using Data Dependency graphs or AST graphs, respectively. Therefore, irrelevant code lines exist between relevant code lines, which are unnecessary for determining the SCA warning and lead to inefficient use of the context window in the prompt [38, 74]. Consequently, we developed several program slicing approaches for each SCA rule, adapting existing Program Slicing methods, which are shown in Table 4.

Program Slicing ApproachDescriptionInspiration
ISSUE_LINE_ONLYExtract the issue line onlyYang et al. [78] and Li et al. [75]
ONE_LINE_UPExtract the issue line and the previous line from the source codeYang et al. [78] and Li et al. [75]
ONE_LINE_DOWNExtract the issue line and the following line from the source codeYang et al. [78] and Li et al. [75]
AST_BACKWARDExtract source code lines related to the backward AST graph from the issue lineTufano et al. [76], Yerramreddy et al. [74], Li et al. [75]
AST_FORWARDExtract source code lines related to the forward AST graph from the issue lineTufano et al. [76], Yerramreddy et al. [74], Li et al. [75], and Salimi et al. [77]
AST_FORWARD_BACKWARDExtract source code lines related to the backward AST graph and the forward AST graph from the issue lineTufano et al. [76], Yerramreddy et al. [74], Li et al. [75], and Salimi et al. [77]
DATA_DEPENDENCYExtract source code lines which have data dependencies on the issue lineTufano et al. [76], Yerramreddy et al. [74], Li et al. [75], and Salimi et al. [77]

Table 4. Source code extraction approaches

Figure 1.A demonstrates source code which generated an SCA warning for the rule “CommentRequired” of PMD. This SCA warning only required one code line to determine actionable SCA warnings. In such a situation, the “ISSUE_LINE_ONLY” approach was used, as shown in Table 4. In contrast, SonarQube Rule Code - “java:S1451” (Track lack of copyright and license headers) requires several code lines to determine whether the warning is a true positive or a false positive. In such scenarios, we used a few lines of code from the issue line as the slice of code. For these cases, either “ONE_LINE_UP” (the issue line and one previous line) or “ONE_LINE_DOWN” (the issue line and one following line) is used.

Code examples for warning types based on how code slices are extracted

Figure 1. Code examples for warning types based on how code slices are extracted. A. Type 1 warnings: Only a few lines are needed to determine whether it is a true positive or a false positive. B. Type 2 warnings: Numerous code lines dispersed throughout the source code are necessary to determine whether it is a true positive or false positive.

On the other hand, there are some warnings where the related issue lines are dispersed throughout the source code file. Therefore, identifying the relevant issue lines is not straightforward for this type of warning. The issue type – “LawOfDemeter” belongs to this kind of warning (as shown in Figure 1.B), where multiple lines of related source code are scattered throughout the source code. Therefore, following existing studies [74, 75, 76, 77], several program slicing approaches, including “AST_BACKWARD”, “AST_FORWARD”, “AST_FORWARD_BACKWARD”, and “DATA_DEPENDENCY”, were employed when there is a code flow dependency (Data Dependency or AST) related to the classification of these kinds of warnings.

Furthermore, “AST_BACKWARD” was used to extract source code lines with AST backward dependencies, while “AST_FORWARD” was used to extract source code lines with AST forward dependencies. If source code lines with both dependencies were necessary, “AST_FORWARD_BACKWARD” was employed. Moreover, “DATA_DEPENDENCY” was utilised to extract code lines with data dependencies. AST graphs and data dependency graphs are explained in § 2.5.

Table 4 presents a summary of the Program Slicing approaches used in this study, along with the sources of their inspiration. The Programme Slicing methods for each PMD warning type are presented in Appendix § 12.2, while those for each SonarQube warning type are in Appendix § 12.3. A script was developed to execute the ONE_LINE_UP, ONE_LINE_DOWN, and ISSUE_LINE_ONLY slicing techniques. However, other types of Program Slicing required existing software to identify AST graphs and data dependency graphs (refer to § 2.4). In such cases, existing studies used JOANA and Joern for program slicing [45, 46]. However, since JOANA only supports Java bytecode and older JVM versions (JDK 7 and 8) [47], Joern was employed to identify AST and data dependency graphs.

All the extracted slices were manually evaluated by the first author, and a sample of 10% from each slicing approach which is distributed over all four datasets were manually examined by the second and third authors to ensure the reliability of our Program Slicing approaches. During our manual evaluation, we did not find any missing slice that were required to determine whether the warning is true positive or false positive. Given the total agreement (100%), we did not formally compute reliability using established methods.

After extracting all the necessary line numbers, which are essential for determining whether the warning is a true positive or a false positive, the corresponding lines of source code were extracted, creating a program slice of the source code that represents the required source code to classify the warning. The resulting code snippets were used for classifying SCA warnings.

LLM model selection and prompting

To select LLM models for our study to evaluate, we used the LiveCodeBench leaderboard. It is based on a benchmark created by Jain et al. [79]. It is a contamination-free evaluation benchmark, where the problems/data used to evaluate the LLMs are not part of the LLMs’ training process. LLMs are AI models which respond to a given query. These models are trained on a large dataset; however, it is vital to train them with data that are relevant to specific tasks [80, 81]. This study aims to classify SCA warnings into true positives or false positives using LLMs. Therefore, several factors should be satisfied by the LLM to classify SCA warnings correctly. First, the LLM should understand the patterns and structures of source code. Therefore, model fine-tuning with software source code is vital. However, since fine-tuned

LLMs with source code are already available, it is not necessary to fine-tune models in this study.

Since the LLMs are expected to behave in a deterministic manner for this study (Refer to § 2.3), hyperparameter optimisation may not provide significant advantages. Therefore, we selected the hyperparameters based on recommendation from existing studies to improve the possibility of the LLM behaving in a deterministic manner in most situations [15, 31, 82]. Other aspects such as prompt engineering are considered in this study, as they might offer improvements in the performance of SCA warning classification when using LLMs [33, 34, 83, 84, 85, 86]. Therefore, we used the SCoT technique to design prompts for LLMs [84]. We selected five random SCA warnings from each dataset (20 warnings altogether) and refined the prompt structure iteratively. We could reach up to 80% accuracy for the randomly selected SCA warnings. Once we fixed the prompting structure, we did not modify the structure based on the SCA rules. However, we modified the prompts for Debating Agentic pattern and iteratively optimised using ten randomly selected SCA warnings (five warnings from each rule). To evaluate our approach, we selected the top two proprietary LLMs and the top open-source LLMs from this leaderboard, as seen in Table 5 to assess the generalisability of the results of the best LLM model in the leaderboard across proprietary and open-source models. Furthermore, we accessed the leaderboard on 7th October 2025, and the LLMs have been evaluated with 579 problems within the time frame from the May 2024 to the May 2025 to create the leaderboard.

LLM modelCommentRQs used for
O4-Mini (o4-mini-2025-04-16)Proprietary model, Same version as the sourceRQ1 and RQ2
Gemini-2.5-Pro (Through VertexAI)Proprietary model, Same version as the sourceRQ1 and RQ2
OpenReasoning-Nemotron-32B (nvidia/OpenReasoning-Nemotron-32B)Open-source model, Same version as the sourceRQ1 and RQ2

Table 5. LLMs used in this study

To evaluate the LLMs' ability to classify SCA warnings as true positives or false positives, we designed prompts using the SCoT technique [84]. In SCoT, we structure the prompts so that the procedure to follow to classify SCA warnings is abstractly defined. A sample of the SCoT prompt is shown in Appendix § 12.7, where the role and task are defined in the first section of the prompt. In the second section of the prompt, input parameters are defined, and in the third section, the decision criteria are defined. In the decision criteria, we defined the requirements for distinguishing between a true positive and a false positive separately. Additionally, we defined the analysis steps to be followed, how to handle uncertainty, and confidence scales. Then the final section of the prompt defined the output format of the results generated by the LLM and the essential instructions to follow when analysing SCA warnings and generating predictions.

We provide several pieces of information related to SCA warnings, including the relevant program slice of the software source, the warning message, and a description of the rule associated with the SCA warning, as documented in the official SCA tool guide. The rule description is modified for more clarity or in accordance with established standards if any contradictions are found. The modified rules and their descriptions are shown in Appendix § 12.8.

We provided all the necessary information for classifying SCA warnings when prompting the LLM. Furthermore, details of the information provided to LLMs are provided in Table 6 with examples. Moreover, we used recommended hyperparameters such as temperature =0.2= 0.2 and top_p =0.95= 0.95 for similar tasks [31]. O4-mini LLM do not support temperature and top_p hyperparameters, thus, when we use O4-mini LLM, we set “high” to the reasoning effort, as the same value was used when benchmarking O4-mini via the LiveCodeBench.

Information supplied with the input promptDescriptionExample
Program SliceSlice of the relevant source codepublic void run() { // issue_line final Runnable runnable = new Runnable() {
SCA warning messageThe warning message generated by the SCA toolPublic method and constructor comments are required
SCA tool’s rule descriptionThe description of the relevant rule in the official documentation of the SCA tool - this is modified from the original description to address any contradictions with established coding standards and to improve understandabilityAccording to Clean Code book, comments are not required for classes, Constructors, Enum, fields/variables, methods. But for main methods, public non-static methods and API endpoints need comment which explain what is done by the method

Table 6. Details of information provided with input prompts

LLM decoding techniques and Measures for the RQs

We evaluated all the LLMs as described in § 5.2. All the information mentioned in Table 6 are provided with the prompt for RQ1. O4-Mini, Gemini-2.5-pro, and OpenReasoning-Nemotron-32B LLM models are evaluated for answering RQ1, where we used the SCoT prompting technique. Furthermore, existing studies did not meaningfully evaluated Agentic patterns in LLMs for SCA warning classification [4, 15]. Consequently, we applied LLM agentic patterns for answering RQ2, including Self-Consistency and Debating patterns, as decoding techniques to evaluate LLMs [15]. Moreover, in RQ2, we evaluated LLMs for classifying SCA warnings using Agentic patterns with SCoT and measured their performance to determine whether Agentic patterns improve warning classification. The procedure we followed for the Debating pattern is shown in Figure 2. We set the temperature to 0.7 for both the Prosecutor Agent and the Defender Agents, and to 0.4 for the Critic Agent [15, 31, 82]. Additionally, we set the temperature to 0.2 for the Arbiter Agent. Furthermore, we used 0.95 as top_p for every agent [15, 31, 82]. Moreover, the procedure we followed for the Self-Consistency pattern is shown in Figure 3, where we used 0.7 for the temperature hyperparameter and 0.95 for top_p following the original paper for the Self-Consistency [82]. Additionally, because O4-mini LLM lacks support for temperature and top_p hyperparameters, we consistently set reasoning effort to high, matching the value used during the O4-mini benchmarking in LiveCodeBench. Moreover, results for each research question are reported in MCC, F1-Score, BA, and time taken to classify a SCA warning to understand the usability of each LLM in terms of the purpose of building classifiers as described in the § 2.2. Furthermore, the methodologies followed in answering each research questions are summarised in Table 7.

Debating Agentic pattern with Structured CoT request for RQ2

Figure 2. Debating Agentic pattern with Structured CoT request for RQ2

Figure 3. Self-Consistency Agentic pattern with Structured CoT request for RQ1

Research QuestionMethodology
RQ1: What is the performance of pre-trained models in classifying SCA warnings using established coding standards?Each LLM model was separately evaluated with four datasets. The instances that false positives were correctly/incorrectly identified by LLMs using SCoT prompting approach are collected individually. Time taken to analyse SCA warnings were also recorded and cost was also calculated in relevant scenarios. These results were reported and analysed in Section 6.1. Moreover, Cochran's Q test was conducted among LLMs for each dataset separately to check if the performance difference between each LLM is significant. Additionally, Pairwise McNemar tests were conducted to identify which LLMs produced significantly better performance among the datasets.
RQ2: How do the Agentic Patterns (Debating and Self-Consistency) elevate the performance of the best pre-trained models in classifying warnings generated by SCA tools?We identified the rules which generated the most incorrect results in RQ1. Of the rules, most incorrect results generated from each SCA tool were selected and examined further in RQ2. Similar to RQ1, LLMs were evaluated using Agentic patterns such as Self-Consistency and Debating. The performance of each LLM for each dataset, time taken to classify SCA warnings and costs are reported and analysed in Section 6.2. Moreover, Cochran's Q test was conducted among LLMs for each dataset separately to check if the performance difference between each LLM is significant. Additionally, Pairwise McNemar tests were conducted to identify which LLMs produced significantly better performance among datasets. Furthermore, Cochran's Q tests were conducted among different runs in the Self-consistency approach to understand the impact of higher temperature values for LLMs’ performance variability.

Table 7. Methodology followed to prepare data to answer RQs

Results

RQ1: What is the performance of pre-trained models in classifying SCA warnings using established coding standards?

Our four datasets, discussed in § 4, were used to evaluate the selected LLM models from § 5.2. These LLMs classified the SCA warnings in our datasets independently of the manually assigned labels, and later, the predicted labels were compared with the manually assigned labels to assess the performance of our approach. We used the SCoT methodology as the prompting technique. The results for the top 57 rules which generated the most SCA warnings are shown in Table 8, Table 9, Table 10 and Table 11 for Datasets 1, 2, 3 and 4, respectively. The complete results for each rule and dataset are shown in Appendix § 12.4, Appendix § 12.5, and Appendix § 12.6 for Datasets 1, 2, and 4, respectively. Moreover, Table 10 shows the complete results for Dataset 3.

LLMCommentRequiredUseUtilityClassLawOfDemeterSingularFieldUnusedAssignment
OpenReasoning102/312/09/12/02/0
Gemini97/812/09/11/11/1
O4_mini101/412/09/12/02/0

Table 8. Correct and incorrect number of warnings for the top five rules in Dataset1

LLMCommentRequiredLawOfDemeterJUnitTestsShouldIncludeAssertCloseResourceUselessParentheses
OpenReasoning38/1422/232/22/01/2
Gemini39/1326/193/12/01/2
O4_mini41/1121/242/32/01/2

Table 9. Correct and incorrect number of warnings for the top five rules in Dataset2

LLMSource code should be indented consistentlyMagic numbers should not be usedequals(Object obj) should test the argument's typeLazy initialization of "static" fields should be "synchronized"Classes should not have too many fields
OpenReasoning104/128/141/01/00/1
Gemini103/228/141/01/01/0
O4_mini103/227/151/01/00/1

Table 10. Correct and incorrect number of warnings for each rule in Dataset3

LLMSource code should be indented consistentlyMagic numbers should not be usedTrack lack of copyright and license headersMember variable visibility should be specifiedArrays and lists should not be copied using loops
OpenReasoning223/2914/72/71/50/1
Gemini245/711/101/81/51/0
O4_mini248/47/148/10/61/0

Table 11. Correct and incorrect number of warnings for the top five rules in Dataset4

The performance of each LLM for each dataset is shown in Figure 4 in terms of performance metrics, including Precision, Recall, MCC, F1-Score, BA, and average time taken to analyse a warning. Figure 4 show that Dataset 1 achieved the best performance across all metrics. Moreover, the OpenReasoning model performed best across most datasets and most performance metrics. However, Gemini was the best LLM for Dataset 2 in terms of Precision, Recall, MCC, F1-Score, and BA, and O4-mini was best for Dataset 4 in terms of Precision, MCC and F1-Score. Moreover, OpenReasoning models took significantly (5.8-7.1 times slower than the median) longer to analyse warnings than other LLMs for all the datasets. As described in § 5, we deployed the OpenReasoning model on a locally hosted single-GPU server. Therefore, the performance of the OpenReasoning models in terms of time to produce results is noteworthy.

Performance of SCA warning classification using SCoT for all the warning types

Figure 4. Performance of SCA warning classification using SCoT for all the warning types

Each SCA warning in each dataset was fed into the LLM to predict the label, and the predicted label was compared with the actual label. The labels are binary, therefore, to compare the results, Cochran’s Q test was conducted [87, 88], and the results are shown in Table 12. First, we conducted these statistical tests on each dataset, and then on all SCA warnings across all datasets. Therefore, the last row of the Table 12 shows the statistical test results for the collection of all SCA warnings across all datasets. Table 12 demonstrates the accuracy of each LLM for each dataset, Q statistic values, and the p-values. The significant p values (p < 0.05) are bolded. According to Table 12, for Datasets 1, 4 and All, at least one LLM produced significantly better performance over other LLMs.

DatasetAccuracyQ statisticp value
OpenReasoningGeminiO4-mini
10.94960.89210.949610.66670.0048
20.60680.67520.63252.51280.2847
30.89330.89330.88000.40000.8187
40.83220.89260.909414.39340.0007
All0.83100.85650.86517.09090.0289

Table 12. Cochran's Q test results for all datasets among LLMs (Significant values were bolded)

To identify the exact LLMs which generated significantly better performance, Pairwise McNemar tests with Holm-Bonferroni Correction were executed [89]. These results are shown in Table 13 where O4-mini is seen to be significantly better than Gemini on Dataset 1. Additionally, Gemini and O4-mini are significantly better than OpenReasoning LLM for Dataset 4, and O4-mini is better than OpenReasoning for all the datasets collectively. Consequently, the LLMs show no significant performance differences on Datasets 2 and 3; in other scenarios, O4-mini performs best.

DatasetGemini vs O4-MiniOpenReasoning vs GeminiOpenReasoning vs O4-Mini
P ValueAdjusted_alphaP ValueAdjusted_alphaP ValueAdjusted_alpha
10.0133280.0166670.0433080.0250000.6170750.050000
40.4042480.0500000.0141380.0250000.0020660.016667
All0.5319710.0500000.1018760.0250000.0164890.016667

Table 13. Pairwise McNemar test results among LLMs with Holm-Bonferroni Correction(Significant values were bolded)

We selected few Machine Learning (ML) approaches as baseline for our study as ML approaches were popular in academia for SCA warning classification before LLMs were introduced [74, 78, 90, 91]. We used four Code Representation Learning (CRL) models (Code2Vec, CodeBERT, FastText and Word2Vec) to generate numerical representations of source code program slices to ML algorithms. We utilised six ML algorithms such as BGRU, BiLSTM, GRU, LSTM, Random Forest, and TextCNN). Figure 5 shows the results for ML approaches trained on the same datasets from our previous study [92]. Datasets were trained with multiple ML models 10 times each and graphs in Figure 5 show boxplots for F1-Score values for each iteration. According to the Figure 5, the best ML model is varies based on the dataset. However, some ML models performed better than LLM approaches evaluated on the same datasets shows that LLM models produce lower results, mainly for Datasets 2-4.

Box plots for ML models trained and evaluated on the same datasets

Figure 5. Box plots for ML models trained and evaluated on the same datasets

RQ2: How do the Agentic Patterns (Debating and Self-Consistency) elevate the performance of the best pre-trained models in classifying warnings generated by SCA tools?

RQ1 focused on identifying the performance of the three best frontier LLM models in classifying SCA warnings. The results for the top 5 rules which generated the highest number of warnings are shown in the Table 8 to Table 11. According to these results, the “LawOfDemeter” rule produced the most incorrect results in PMD, while the “Magic numbers should not be used” rule generated the most incorrect results for SonarQube. Therefore, for RQ2, further investigations were conducted only for warnings generated by these rules.

In RQ2, we examined the use of Agentic patterns proposed by Gullí [15]. We used Self-Consistency and Debating patterns to improve the warning classification task, and the results are compared with SCoT in Figure 6. According to Figure 6, results generated for our analysis show that although the use of Agentic patterns yielded lower performance, the O4-mini LLM model achieved higher performance in some scenarios, such as classifying warnings generated by the SonarQube rule “Magic numbers should not be used,” as measured by Precision, MCC, F1-Score, and BA. Furthermore, according to Figure 6, the use of Agentic patterns required more time to analyse a warning, as it needed multiple LLM calls for each SCA warning. Agentic patterns seem unnecessary for most scenarios, and further investigation is required in order to identify practical applications where they are useful.

(a) Precision

SCoT vs Agentic vs Self-Consistency in Recall

(b) Recall

Figure 6.

Figure 6 (PDF p. 27). Performance of SCoT, Agentic AI and Self-consistency implementations for selected warning types

Cochran’s Q tests were conducted for all issueType_LLM combinations among different prompting techniques, and the results are shown in Table 14 [87, 88]. Table 14 shows the accuracy of SCA warning classification in each prompting technique and Q statistic values, and p values for each issueType_LLM combination. The significant p values (p < 0.05) are bolded in Table 14, where LOD_Gemini and OpenReasoning (for both issue types collectively) and Gemini (for both issue types collectively) have at least one significant prompting technique. However, for other IssueType_LLM combinations, all the prompting techniques perform without any significant performance difference.

IssueType_LLMAccuracyQ statisticP value
SCoTDebatingSelf-Consistency
LOD_OpenReasoning0.50910.32730.38184.64710.0979
LOD_Gemini0.63640.32730.600015.23530.0005
LOD_O4-mini0.54550.52730.49090.51850.7716
Magic_OpenReasoning0.66670.55560.57143.30770.1913
Magic_Gemini0.61900.60320.61900.11760.9429
Magic_O4-mini0.53970.66670.65084.07140.1306
OpenReasoning (LOD and Magic)0.59320.44920.48317.90000.0193
Gemini (LOD and Magic)0.62710.47460.610211.4510.0033
O4-mini (LOD and Magic)0.54240.60170.57631.34550.5103

Table 14. Cochran's Q test results for all issueType_LLM among different prompting techniques

To identify the exact prompting technique which generated significantly better performance, Pairwise McNemar tests with Holm-Bonferroni Correction were conducted for the IssueType_LLM [89]. The results are shown in Table 15, where for LOD_Gemini and Gemini (LOD and Magic), both SCoT and Self-Consistency perform significantly better than Debating. OpenReasoning (LOD and Magic) shows a sign of the existence of a significant prompting technique, according to Cochran's Q test. However, after applying the Holm-Bonferroni correction for the pairwise McNemar test, there is no prompting technique that performs significantly better than the other techniques.

IssueType_LLMSCoT vs DebatingDebating vs Self-ConsistencySCoT vs Self-Consistency
P ValueAdjusted_alphaP ValueAdjusted_alphaP ValueAdjusted_alpha
LOD_Gemini0.0020760.0166670.0093300.0250000.7728300.050000
OpenReasoning0.0170730.0166670.6510770.0500000.0311410.025000
Gemini0.0087120.0166670.0177060.0250000.8230630.050000

Table 15. Pairwise McNemar test results among different prompting techniques with Holm-Bonferroni Correction

On the other hand, we ran the same set of SCA warnings five times to conduct a Self-Consistency approach. Given the use of 0.7 as the temperature value for the LLM calls, we used the results from individual LLM calls made for the Self-Consistency approach to validate the potential impact of the higher temperature value on the overall performance variation of the LLMs in different runs. We conducted Cochran's Q tests for each dataset and each LLM. The results are shown in Table 16, where p values for every scenario are higher than 0.05, indicating that there are no significant performance differences between runs.

Dataset 1 (Q Statistic/P Value)Dataset 2 (Q Statistic/P Value)Dataset 3 (Q Statistic/P Value)Dataset 4 (Q Statistic/P Value)
Gemini2.285/0.6837.081/0.1312.800/0.5915.846/0.210
O4-Mini4.666/0.3233.593/0.4632.086/0.7197.076/0.131
OpenReasoning6.352/0.1745.090/0.2785.492/0.2401.166/0.883

Table 16. Cochran's Q test results for LLM performance among different runs in the Self-consistency approach

Moreover, we conducted hyper-parameter tuning for a small set of temperature (T) values such as 0.0, 0.7, 1.0 and 2.0 for self-consistency for the OpenReasoning model. During the experiments, T = 2.0 could not generate final answers within the given maximum number of tokens allowed (16,384). A summary of results for other temperature values is shown in Table 17, where it can be seen that T = 0.7 is the best temperature value for our study overall. While T = 1.0 also performed well, T = 0.7 performed best.

DatasetT = 0.0T = 0.7T = 1.0
Dataset 15/106/107/10
Dataset 221/4526/4520/45
Dataset 318/4216/4215/42
Dataset 410/2111/2111/21

Table 17. Hyperparameter tuning results for each dataset for multiple temperature values

We used three LLMs for our analysis and we paid O4-mini through the OpenAI platform. Moreover, we used the Gemini 2.5 pro Api through a free Vertex AI account8, where we did not pay for the api calls. Additionally, our OpenReasoning-Nemotron-32B model was deployed in our university servers, where it is difficult to estimate the cost of LLM calls. Therefore, we conducted a cost analysis for the O4-mini model to get an understanding of the cost of each approach we experimented with in this paper. O4-mini LLM model cost 0.009134 USD on average for SCoT where only one LLM call was made for one SCA warning. Additionally, the cost to classify one SCA warning using Self-consistence was 0.045668 USD and 0.036535 USD for using Debating. SCoT generates only one LLM request per SCA warning, whereas five are generated for Self-Consistency and four for the Debating approach. On the other hand, Self-Consistency and Debating approaches did not generate significant performance improvements, according to Figure 6. According to our analysis, investing more in Agentic patterns is not cost effective.

Discussion and implications

RQ1: What is the performance of pre-trained models in classifying SCA warnings using established coding standards?

RQ1 examined the performance of the best pre-trained LLMs for SCA warning classification. To classify SCA warnings, we used relevant source code slices, warning messages generated by the SCA tool and the rule descriptions applicable to the generated warning type. Moreover, we checked the rule descriptions against the established coding standards and, if there was any contradiction, modified the descriptions accordingly. All three LLMs achieved good results on Dataset 1 across all performance metrics, including MCC, F1-Score, BA, and time taken to analyse a warning. However, all the LLMs generated lower results for the other datasets. Datasets 1 and 2 consist of warnings generated by PMD, while Datasets 3 and 4 consist of warnings generated by SonarQube. Furthermore, even within Datasets 1 and 2 (and Datasets 3 and 4), the number of warnings per SCA rule varies. Therefore, the choice of LLMs for SCA warning classification is based on the performance metrics and designated goals (Refer to § 2.2). According to § 3, most of the existing studies do not incorporate necessary contextual information, such as SCA rule description and established standards for SCA warnings classification [2, 4, 25]. Without the contextual information, LLM will use their internal knowledge, which may be incorrect or outdated. Additionally, the standards for defining warnings are true positives or false positives, depending on the individual preferences. Therefore, it is vital for the SCA warning classifier to support defining individual preferences instead of using internal knowledge of LLMs. Reviewing existing studies summarised in § 3 [4, 6, 10], none of these studies attempted to provide defined code standards (possibly different based on the software organisation or the team) when prompting LLMs so that the warning classification tasks are performed based on the defined standards. We observed in our previous study that warning classification is based on defined standards [46]. Therefore, it is vital to provide coding standards with the LLM prompts. However, existing studies did not address this requirement. Moreover, existing studies primarily rely on publicly available datasets to evaluate LLMs in SCA warning classification. However, the threat of using publicly available datasets for this designated purpose is that LLMs might have incorporated them into their training, and LLMs might memorise and retrieve labels for SCA warnings from their internal knowledge rather than predicting them. However, when these LLMs are applied to real-world datasets in a real software project, they produce lower results. This aspect is evaluated by multiple existing studies [2, 4, 6]. However, none of them experimented on multiple datasets with different false positive percentages. Only Chen et al. [2, 4] tried to identify the real threat of the usage of publicly available datasets to evaluate LLMs in classifying SCA warnings.

Furthermore, none of the existing studies conducted any experiments for both ML models and LLMs to compare the performance among ML and LLM technologies. We experimented with several ML models on the same dataset that is used to evaluate LLMs in this study. According to the results, ML models perform better than LLM models. Even simpler models such as Random Forest performed better than LLMs. These Random Forest models can train on commodity hardware without any costly GPUs and produces results faster than LLMs tend to respond. Therefore, we can conclude that ML models are better than LLM models in classifying SCA warnings at present.

In this study, we address gaps identified in existing studies satisfactorily. We used our own dataset [16, 17], which was not used to train LLMs, and source code datasets used to create warning datasets are align with real world code. Thus, our results show the genuine performance of the real applications in the practitioner community. Even though existing studies show that LLMs generate near-perfect results [2, 4, 6, 25], the reality is different when unseen data is used for modelling. A summary of SCA warning classification performance is shown in Appendix A, where our approach achieved better results even on real-world datasets compared to existing studies that used them to evaluate their approaches. Moreover, LLMs generate a description of their decision, which makes SCA warning classification with LLMs more explainable than using other ML approaches blindly.

RQ2: How do the Agentic Patterns (Debating and Self-Consistency) elevate the performance of the pre-trained best models in classifying warnings generated by SCA tools?

The performance of the LLMs for our datasets was worse than existing similar studies (§ 3). Therefore, we sought to improve performance by applying some Agentic patterns to our approach. Consequently, we selected a warning type from each SCA tool and applied these Agentic patterns only to those warning types. The results show that the performance did not improve, even using Agentic patterns, except for warnings generated by the “Magic numbers should not be used” rule in SonarQube with O4-mini LLM. There can be two possible reasons for this. The first reason could be that we use a higher temperature (0.7) for Agentic patterns as recommended in the original study to increase the creativity of the generated results [15, 82]. Although, this temperature value is not recommended for tasks such as code generation and understanding related tasks [31], existing studies generated near-perfect results for publicly available benchmark datasets in classifying SCA warnings. Since LLMs may have been trained on these datasets, these existing studies generated near-perfect results while LLMs may generate worse results for unseen datasets. However, the O4-mini LLM does not support temperature or top_p hyperparameters. Therefore, we used the “reasoning effort” hyperparameter. We used “high” for both RQ1 and RQ2. Moreover, O4-mini reduced the performance for LOD when using Agentic patterns and improved the performance for the “Magic numbers should not be used” rule. Therefore, the performance of LLMs may be dependent on the rule type as well. However, for most LLMs, using Agentic patterns in our datasets is unnecessary.

Additionally, we use SCoT when designing prompts for LLMs. Therefore, there is little room for LLMs’ performance to improve or deteriorate, even though they use these Agentic patterns. Comparing our study with existing studies reveals that only a few have used a self-consistency-agentic approach. However, none of the existing studies used a Debating approach. In our study, we showed that using Agentic patterns for SCA warning classification is not beneficial, as it incurs a higher cost due to multiple LLM API calls, takes longer to process each warning, and does not significantly improve performance. Additionally, when Self-Consistency was added, the performance of SCA warning classification deteriorated in most scenarios.

Implications for SCA tool developers

SCA tool developers need to reduce false positive warnings generated by their tools. For that, they can use LLMs to identify false-positive warnings generated by their SCA tools and common patterns that cause them to issue false positives. Consequently, by identifying patterns, tool developers can improve SCA tool performance. On the other hand, the goal of SCA tool developers is to identify all issues in source code, even if they generate false positive warnings. However, users of the SCA tool (the practitioner community) may need to identify as many warnings as possible while minimising false positives. Therefore, it is vital to introduce a sensitivity parameter that allows SCA tool users to adjust based on their needs. Additionally, SCA tool developers can publish statistics on false positive rates for each rule. Therefore, users can deactivate rules that generate a high number of false positive warnings if those warnings are not critical to their software projects.

Implications for the Practitioner community

The purpose of creating the SCA warning classifier is discussed in § 2.2, and according to that, there are multiple goals. Moreover, goals vary depending on individual preferences and use case. For instance, one might need to reduce false positives as much as possible, even though the classifier removes some genuine warnings as well. Another situation might be to identify the maximum number of genuine warnings, while minimising false warnings as much as possible. One might need to optimise overall model performance, while another may need to consider the trade-off between model performance and the time required to process warnings. Therefore, the practitioner community should use appropriate performance metrics to evaluate LLMs against their requirements, rather than rely solely on a few popular metrics such as accuracy and F1-Score. Furthermore, practitioner communities can host open-source LLMs on their own servers or use proprietary services such as ChatGPT, Gemini, and Claude. According to results generated from our study (refer to § 6), proprietary services are faster than self-hosted services, depending on the performance of the hosted hardware. However, there is a potential risk to data privacy, and the practitioner community should also consider this.

Additionally, it is useful to experiment with different LLMs, structured prompts and prompting techniques, decoding techniques and hyperparameters to find the best combination for individual use cases, as most cutting-edge technologies/approaches may not be the best for some use cases. The results are summarised in § 6 show that the performance of the classifier depends on the characteristics of the dataset, the purpose of building the classifier (refer to § 2), and the decoding technique used. Therefore, it is necessary to experiment with the variables mentioned earlier to find the best combination for each use case and dataset.

Coding standards are not universal. Standards can depend on the software engineering company, client organisation, project team and the programming language and framework used. We identified some defined standards in our previous study; however, those should be tailored to different use cases. Additionally, standards evolve over time. Therefore, coding standards should be configurable in SCA warning classifiers to cater for this requirement. Moreover, the practitioner community can use the description generated by LLM to determine whether the SCA warning is a true positive or a false positive, rather than relying solely on the label generated by the classifier. Using the description, the developer can understand whether the SCA warning is a true positive or a false positive.

The results show that the model performance depends on the LLM, dataset and performance metric. Therefore, practitioners should select performance metrics based on their goals (Refer to § 2.2). Furthermore, practitioners should evaluate LLMs on their private datasets as their performance is heavily dependent on the characteristics of the dataset. Moreover, LLMs provide description with the final answer (whether a SCA warning is a true positive or false positive). The description provides good understanding of the reason for the classification, providing model explainability. Results from this study shows that ML models provide better performance. However, explainability of ML models are lower, requiring separate consideration (e.g., perhaps via the use of SHAP or LIME). Therefore, the explainability aspect of a model should also be considered when selecting technologies to create warning classification models. On the other hand, Agentic patterns did not improve the performance for our datasets. Therefore, practitioners should not assume that Agentic patterns may always improve the performance outcomes. The final decision should be made based on experiments conducted on their datasets.

Implications for academia

Most existing studies evaluating LLMs on public datasets/benchmarks suggest that LLMs perform perfectly in SCA warning classification. However, the LLM may have been trained on a given publicly available dataset/benchmark, giving LLMs the opportunity to memorise the label for a specific SCA warning. Consequently, when the same approach is applied to real-world use cases, LLMs may perform worse. Therefore, academia should create benchmarks to evaluate SCA warning classification using LLMs which do not incorporate the LLM’s training data, similar to LiveCodeBench. Furthermore, the use of high-quality evaluation datasets that mirror the characteristics of real-world use cases is vital. Otherwise, the evaluation results are not compatible with the real-world use cases. Conversely, academia can report evaluation results across datasets with varying characteristics, such as different false positive rates and SCA tools. Additionally, it is vital for academia to report their experiment results in multiple performance metrics, so that the community can choose SCA warning classification approaches suitable for their use cases, reflecting their goal of warning classification (refer to § 2.2). Moreover, most existing studies only experiment with specific LLMs and prompting techniques to classify SCA warnings. However, academia can focus on hyperparameter optimisation and model fine-tuning, specialising in warning classification, leading to improved model performance. Additionally, to speed up experiments, academia can first test their approaches on small datasets to evaluate different technologies and methodologies, thereby accelerating the refinement of their methods. This will speed up similar studies, as it takes academia a long time to experiment with LLMs across large datasets. Consequently, we recommend our datasets to the academic community for their experiments, as they are relatively small yet mirror real-world scenarios. These datasets were created by analysing publicly available source code datasets with PMD and SonarQube. Moreover, results show that LLMs produce lower performance for unseen datasets. In this study, we did not evaluate the reason for LLMs producing lower results. We suggest that future research should focus on conducting deeper qualitative analyses in this regard.

Threats to validity

This section discusses the potential threats to the validity of this study following some existing studies [2, 24, 25, 93].

Internal threats to validity

As pointed out by Croft et al. [94], AI/ML model evaluation depends on the quality of the dataset used. If the dataset is incorrectly classified, the evaluation results differ from the actual performance statistics. Therefore, it is vital to use a high-quality dataset. According to Lin et al. [95], the best approach to creating such a high-quality database is manual annotation rather than automated methods. On the other hand, publicly available datasets might have been used to train existing frontier LLMs. Consequently, using publicly available datasets to evaluate LLMs for SCA warning classification can be ineffective. To mitigate these issues, we created four datasets and annotated them manually, as explained in § 4. These datasets were manually annotated following defined standards. Additionally, we checked the reliability of the data annotation as described in § 4, where complete agreement (100%) among the authors were reached after a round of discussion, attesting to the strong reliability of our datasets development. Moreover, the reliability check was conducted for only Datasets 1 and 3, following results generated by Deshapriya et al. [46]. However, this could be a treat for generalising results for Datasets 2 and 4, and thus, our results should be interpreted accordingly.

We used the recommended hyperparameters from existing studies. We used 0.7 for the temperature hyperparameter for LLM calls that required creativity, and 0.2 for those that required factual behaviour. However, we did not experiment with our approaches with multiple hyperparameters. Therefore, the generated results may not be optimal, and there can be room for improvement in terms of model performance. Consequently, our approach with the current hyperparameters should achieve near-optimal performance, based on recommendations [27]. Additionally, we selected the temperature value based on existing papers. However, the O4-mini model does not accept configurable temperature values. We used “high” for the reasoning effort value in this regard. Moreover, we used SCoT and Self-Consistency as prompting strategies. However, there are some other prompting strategies, such as few-shot exemplars and ReAct [96, 97], and different prompting strategies may provide better performance. Furthermore, we used the LiveCodeBench benchmark to choose LLMs. However, this benchmark evaluates LLMs’ ability to generate source code, while we use LLMs for source code identification. Therefore, there may be better LLMs specifically suited to SCA warning classification. However, to generate source code, LLMs should understand the source code. Therefore, the best LLM models in LiveCodeBench benchmark should be suitable for our study. Moreover, our prompt templates used SCoT as the prompting technique. According to existing studies [84], SCoT provides better results. However, the performance of SCoT may differ depending on the exact structure of the prompt template, and there can be enhancements we can implement to help the LLM produce better results.

External threats to validity

There were a limited number of SCA rules in four datasets. Therefore, experimenting with a limited number of warning types can compromise the generalisability of the results. However, we did not filter rules; we selected all rules that generated at least one false positive warning. Therefore, we selected most of the warning types which might generate false positives for Java source code by PMD and SonarQube. Hence, the threat to generalisability of our results should be minimal when consider these two tools. Even though we used all the warning types (Rules) for the analysis in RQ1, we used only two rules, which generated the most incorrect results for RQ2. However, different warning types may yield distinct performance patterns, and our conclusion from RQ2, such as the Agentic patterns’ inability to improve performance, may not generalise.

Conclusion threats to validity

The methodology and dataset used in this study heavily influence its conclusions, and the results may change depending on the characteristics of the dataset used. Therefore, we used four datasets with different characteristics to mitigate this threat. According to the results section, the performance of different LLMs varies depending on the dataset used. Consequently, the conclusion is based on results for multiple datasets. Therefore, this threat is minimal. Moreover, the datasets we used are biased towards a single label, and we did not modify them to address the class-imbalance problem in the datasets. However, we used performance metrics such as F1-Score and BA to mitigate this issue. On the other hand, outputs generated by LLMs are not deterministic. Therefore, the results of this study can vary over time, making it hard to reproduce them. However, to mitigate the issue, we mentioned the exact version of LLM we used in this study, potentially helping with replicability for the same LLMs.

Conclusion and future work

Conclusion:

Existing studies evaluated several LLMs using various prompting techniques to leverage their potential for classifying SCA warnings [2, 6, 10]. They addressed some existing gaps, such as overly broad code snippets, missing context, and issues with prompting techniques. However, existing studies do not fully exploit the potential of LLMs through Agentic patterns. Moreover, most of these studies used publicly available datasets to evaluate their approaches. Although these approaches generated promising results, the datasets might have been part of the training datasets used to train the LLMs in these studies, and these LLMs might have memorised the labels for each SCA warning. Consequently, most of these approaches are not valid for practical use cases. Furthermore, most existing studies do not use defined coding standards. The practitioner community adopts individual standards for their software source code, and LLMs should adhere to these standards when classifying SCA warnings.

To address these three gaps in existing studies, we created two research questions. We identified the three best LLMs using the LiveCodeBench benchmark and used SCoT as the prompting technique. We evaluated the selected three LLMs across our four warning datasets, and the results show that performance varies depending on the LLM and dataset used. Additionally, locally hosted LLMs can generate better results than proprietary LLMs, potentially reducing cost for LLM tokens and preserving data privacy. Moreover, our approach performs better than existing studies that used non-publicly available datasets for their evaluations. Additionally, the use of Agentic patterns does not improve SCA warning classification performance. Overall, our results show that current LLMs achieve good performance on certain warning types, and publicly available datasets do not reliably evaluate LLMs in SCA warning classifications.

Future work:

Software developers identify true positive warnings and eliminate false positive warnings with higher accuracy. Considering the process they follow, there can be different approaches taken in future studies. For instance, multiple software developers may collectively discuss warnings generated by SCA tools and may reflect on the unique knowledge gathered through discussions of other similar warnings. Once they find a different warning, they may discuss it with their team. This approach blends a single-agent system and a multi-agent system, and the single agent should be able to decide when to work alone (single-agent system) and when to work with a team (multi-agent system). Additionally, the system should preserve the unique knowledge it gathered through collectively analysing warnings. One characteristic of such a system would be that developers in the real world bring diverse experiences and unique expertise. Their experience and knowledge will be distinctive from each other. To implement that, we can use multiple LLMs from different vendors. Consequently, these systems will be cost-effective, as the SCA warning classification can be performed individually (single-agent system), and if they need help, they can work together (multi-agent system) to share relevant knowledge.

However, there are some challenges in such systems. To successfully implement such a system, LLMs should be able to self-assess their output and provide a confident score that reflects their accuracy. However, existing LLMs suffer from a calibration issue, producing incorrect results with higher confidence scores [98, 99]. Therefore, it is hard for LLMs to determine whether they can address the warning individually or need to work together with other LLMs. Furthermore, LLMs have limited capability to generalise knowledge gathered through solving a warning task itself in real-time to use the knowledge for other similar situations. While solving these two issues, we are planning to create such a system and evaluate this system using our datasets to benchmark with other approaches.

We also noticed that different LLMs generated incorrect results for a unique set of warnings, where one LLM may generate incorrect results for some set of warnings while another LLM generates correct results for the same set of warnings. Therefore, in future studies, we will examine patterns of incorrect results and their causes for each LLM separately. Through this analysis, we can understand which LLM is suitable for which warning types and how to mitigate the common patterns to reduce the number of incorrect results and improve their performance.

On the other hand, we used 0.7 as the temperature value for the self-consistency agentic approach, as recommended by existing studies [82]. However, after these existing studies were conducted, some LLM providers have extended the temperature range to 0-2 (previously 0-1). Since the temperature value is an absolute value, the current behaviour is as expected, consistent with existing studies. However, we plan to extend the existing experiments on the Self-Consistency approach to temperatures between 1 and 2.

Author Contributions Lakmal Deshapriya: Conceptualisation, Software, Data curation, Methodology, Writing-Original draft preparation.

Sherlock A. Licorish: Conceptualisation, Data curation, Supervision, Writing-Reviewing and Editing, Statistics Validation.

Brendon J. Woodford: Data curation, Supervision, Writing-Reviewing and Editing.

Grant Dick: Writing-Reviewing and Editing.

Funding Not applicable.

Data Availability: We used four manually annotated SCA warning datasets in our study, previously published by Deshapriya et al. [16]. Additionally, our replication package is available at https://doi.org/10.5281/zenodo.18820913, which contains the raw experimental results and the complete source code used to analyse the LLMs, ensuring full reproducibility of our findings.

Declarations

Ethical Approval Not applicable.

Informed Consent Not applicable.

Conflicts of Interest No conflict of interest.

Clinical Trial Number Not applicable.

References

Representation Stability,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 10459-10525.

Appendices

Summary of the datasets and LLMs used in existing studies and our study

StudyProgramming LanguageDatasetLLMPerformance metricPerformance
Apostolidis et al. [25]C/C++Big-VulFinetuned CodeBERTPrecision0.9620
Apostolidis et al. [25]C/C++Big-VulFinetuned CodeBERTRecall0.4066
Apostolidis et al. [25]C/C++Big-VulFinetuned CodeBERTF1-Score0.5716
Apostolidis et al. [25]C/C++Big-VulFinetuned CodeBERTFalse Positive rate0.0010
Apostolidis et al. [25]C/C++ReVealFinetuned CodeBERTPrecision0.4767
Apostolidis et al. [25]C/C++ReVealFinetuned CodeBERTRecall0.0593
Apostolidis et al. [25]C/C++ReVealFinetuned CodeBERTF1-Score0.1056
Apostolidis et al. [25]C/C++ReVealFinetuned CodeBERTFalse Positive rate0.0071
Wen et al. [2]C/C++Juliet Test Suites 1.3ChatGPTAccuracy0.7452-0.8225 (for different SCA tools – CppCheck, CSA and Infer)
Wen et al. [2]C/C++Juliet Test Suites 1.3ChatGPTPrecision0.7539-0.8846 (for different SCA tools – CppCheck, CSA and Infer)
Wen et al. [2]C/C++Juliet Test Suites 1.3ChatGPTRecall0.8822-0.9960 (for different SCA tools – CppCheck, CSA and Infer)
Wen et al. [2]C/C++Juliet Test Suites 1.3Llama-2Accuracy0.6603-0.7516 (for different SCA tools – CppCheck, CSA and Infer)
Wen et al. [2]C/C++Juliet Test Suites 1.3Llama-2Precision0.8170-0.9029 (for different SCA tools – CppCheck, CSA and Infer)
Wen et al. [2]C/C++Juliet Test Suites 1.3Llama-2Recall0.7480-0.9066 (for different SCA tools – CppCheck, CSA and Infer)
Wen et al. [2]C/C++Real-world applicationsChatGPT (Zero-shot)Precision0.2154-0.5694 (for different SCA tools – CppCheck, CSA and Infer)
Wen et al. [2]C/C++Real-world applicationsChatGPT (Zero-shot)Recall0.5714-0.8667 (for different SCA tools – CppCheck, CSA and Infer)
Wen et al. [2]C/C++Real-world applicationsChatGPT (Zero-shot)True Positive rate0.0144-0.0376 (for different SCA tools – CppCheck, CSA and Infer)
Wen et al. [2]C/C++Real-world applicationsChatGPT (Few-shot)Precision0.3974-0.6581 (for different SCA tools – CppCheck, CSA and Infer)
Wen et al. [2]C/C++Real-world applicationsChatGPT (Few-shot)Recall0.9167-0.9333 (for different SCA tools – CppCheck, CSA and Infer)
Wen et al. [2]C/C++Real-world applicationsChatGPT (Few-shot)True Positive rate0.0349-0.0556 (for different SCA tools – CppCheck, CSA and Infer)
Svärd and Teräs [10]JavaJuliet Test Suits 1.3Gemini Flash Lite 2.0Precision0.3673
Svärd and Teräs [10]JavaJuliet Test Suits 1.3Gemini Flash Lite 2.0Recall0.4914
Chen et al. [4]C/C++Juliet Test Suites 1.3Qwen2.5-32bAccuracy0.9490-0.9960
Chen et al. [4]C/C++Juliet Test Suites 1.3Qwen2.5-32bPrecision0.9460-1.0000
Chen et al. [4]C/C++Juliet Test Suites 1.3Qwen2.5-32bRecall0.9950-1.0000
Chen et al. [4]C/C++Juliet Test Suites 1.3Qwen2.5-32bF1-Score0.9720-0.9980
Chen et al. [4]C/C++Real-world applicationQwen2.5-32bRecall0.8605
Iranmanesh et al. [6]JavaOpenVulndeepseek-r1F1-Score0.6500
Iranmanesh et al. [6]JavaOpenVulndeepseek-r1Precision1.0000
Iranmanesh et al. [6]JavaOpenVulndeepseek-r1Recall0.4900
Iranmanesh et al. [6]JavaOpenVulndeepseek-r1-distillF1-Score0.6600
Iranmanesh et al. [6]JavaOpenVulndeepseek-r1-distillPrecision0.8300
Iranmanesh et al. [6]JavaOpenVulndeepseek-r1-distillRecall0.5400

Table 18.

gemini-2.5-proF1-Score0.3700
Precision1.0000
Recall0.2300
gpt-5F1-Score0.9600
Precision1.0000
Recall0.9100
gpt-oss-120bF1-Score0.8500
Precision0.8800
Recall0.8300
gpt-oss-20bF1-Score0.9000
Precision0.8700
Recall0.9400
grok-4F1-Score0.9200
Precision1.0000
Recall0.8600
mixtral-8x7b-instructF1-Score0.7700
Precision0.8100
Recall0.7400
o4-miniF1-Score0.8800
Precision0.8400
Recall0.9100
qwen3-235b-a22bF1-Score0.7900
Precision1.0000
Recall0.6600
OWASPdeepseek-r1F1-Score0.7910
Precision0.9720
Recall0.6670
deepseek-r1-distillF1-Score0.7120
Precision0.9410
Recall0.5730
gemini-2.5-proF1-Score0.9100
Precision0.9780
Recall0.8510
gpt-5F1-Score0.8110
Precision0.8060
Recall0.8150
gpt-oss-120bF1-Score0.8970
Precision0.9210
Recall0.8740
gpt-oss-20bF1-Score0.8840
Precision0.8690
Recall0.9000
grok-4F1-Score0.9120
Precision0.9760
Recall0.8550
mixtral-8x7b-instructF1-Score0.2170
Precision0.5550
Recall0.1350
o4-miniF1-Score0.8720
Precision0.9070
Recall0.8400
qwen3-235b-a22bF1-Score0.8610
Precision0.9640
Recall0.7780
Our studyJavaPMD_StackOverflowGeminiBalanced Accuracy0.9093
F1-Score0.9189
MCC0.7714
Precision0.9770
Recall0.8673
Average Time in Seconds12.083
O4-miniBalanced Accuracy0.9572
F1-Score0.9634
MCC0.8862
Precision0.9892
Recall0.9388
Average Time in Seconds5.9261
OpenReasoningBalanced Accuracy0.9691
F1-Score0.9681
MCC0.9060
Precision1.0000
Recall0.9381
Average Time in Seconds86.341
PMD_Apache_TomcatGeminiBalanced Accuracy0.6507
PMD_Apache_TomcatGeminiF1-Score0.7432
PMD_Apache_TomcatGeminiMCC0.3014
PMD_Apache_TomcatGeminiPrecision0.7432
PMD_Apache_TomcatGeminiRecall0.7432
PMD_Apache_TomcatGeminiAverage Time in Seconds16.442
PMD_Apache_TomcatO4-miniBalanced Accuracy0.5974
PMD_Apache_TomcatO4-miniF1-Score0.7152
PMD_Apache_TomcatO4-miniMCC0.1980
PMD_Apache_TomcatO4-miniPrecision0.7013
PMD_Apache_TomcatO4-miniRecall0.7297
PMD_Apache_TomcatO4-miniAverage Time in Seconds11.1661
PMD_Apache_TomcatOpenReasoningBalanced Accuracy0.6111
PMD_Apache_TomcatOpenReasoningF1-Score0.6857
PMD_Apache_TomcatOpenReasoningMCC0.2191
PMD_Apache_TomcatOpenReasoningPrecision0.7059
PMD_Apache_TomcatOpenReasoningRecall0.6667
PMD_Apache_TomcatOpenReasoningAverage Time in Seconds150.4156
SonarQube_StackOverflowGeminiBalanced Accuracy0.7708
SonarQube_StackOverflowGeminiF1-Score0.6800
SonarQube_StackOverflowGeminiMCC0.6374
SonarQube_StackOverflowGeminiPrecision0.8500
SonarQube_StackOverflowGeminiRecall0.5667
SonarQube_StackOverflowGeminiAverage Time in Seconds17.7994
SonarQube_StackOverflowO4-miniBalanced Accuracy0.7500
SonarQube_StackOverflowO4-miniF1-Score0.6400
SonarQube_StackOverflowO4-miniMCC0.5883
SonarQube_StackOverflowO4-miniPrecision0.8000
SonarQube_StackOverflowO4-miniRecall0.5333
SonarQube_StackOverflowO4-miniAverage Time in Seconds7.2911
SonarQube_StackOverflowOpenReasoningBalanced Accuracy0.8583
SonarQube_StackOverflowOpenReasoningF1-Score0.7500
SonarQube_StackOverflowOpenReasoningMCC0.6847
SonarQube_StackOverflowOpenReasoningPrecision0.7059
SonarQube_StackOverflowOpenReasoningRecall0.8000
SonarQube_StackOverflowOpenReasoningAverage Time in Seconds120.69
SonarQube_Apache_TomcatGeminiBalanced Accuracy60.746
SonarQube_Apache_TomcatGeminiF1-Score0.4286
SonarQube_Apache_TomcatGeminiMCC0.4194
SonarQube_Apache_TomcatGeminiPrecision0.7059
SonarQube_Apache_TomcatGeminiRecall0.3077
SonarQube_Apache_TomcatGeminiAverage Time in Seconds22.57584
SonarQube_Apache_TomcatO4-miniBalanced Accuracy11.4418
SonarQube_Apache_TomcatO4-miniF1-Score0.5263
SonarQube_Apache_TomcatO4-miniMCC0.5281
SonarQube_Apache_TomcatO4-miniPrecision0.8333
SonarQube_Apache_TomcatO4-miniRecall0.3846
SonarQube_Apache_TomcatO4-miniAverage Time in Seconds10.2443
SonarQube_Apache_TomcatOpenReasoningBalanced Accuracy5.31446
SonarQube_Apache_TomcatOpenReasoningF1-Score0.4444
SonarQube_Apache_TomcatOpenReasoningMCC0.3520
SonarQube_Apache_TomcatOpenReasoningPrecision0.3922
SonarQube_Apache_TomcatOpenReasoningRecall0.5128
SonarQube_Apache_TomcatOpenReasoningAverage Time in Seconds131.0500

Table 44.

Program Slicing Approaches for PMD Rules

PMD RulesProgram Slicing ApproachMotivation for selected program slicing approach
AvoidArrayLoopsAST_FORWARDTo classify the warning, Issue line (loop definition) is required. Additionally, array calls are connected with AST forward relationships. Therefore “AST_FORWARD” was used.
AvoidDeeplyNestedIfStmtsAST_FORWARD_BACKWARDIf the “IF” statement is deeply nested, they are connected with AST graph with their parent AST node (IF). To classify warnings from this rule, all nested IF statements should be extracted. Therefore the parent AST node of the issue line and all Children were extracted using “AST_FORWARD_BACKWARD”.
AvoidThrowingNewInstanceOfSameExceptionDATA_DEPENDENCYTo classify warnings generated by this rule, the new exception and the original exception should be connected with a data dependency. Therefore “DATA_DEPENDENCY” is used for warnings generated by this SCA rule.
CloneMethodMustBePublicAST_BACKWARDTo classify warnings generated by this rule, the issue line and it’s respective class definition should connect with the AST relationship. Since the analysis starts from the issue line and we do not need any deeper AST nodes, we use “AST_BACKWARD” to extract class definition to check whether it implements Cloneable. We are then able to check whether the method (issue line) is public.
CloneMethodMustImplementCloneableAST_BACKWARDTo classify warnings generated by this rule, the issue line and it’s respective class definition should connect with the AST relationship. Since the analysis starts from the issue line and we do not need any deeper AST nodes, we use “AST_BACKWARD” to extract class definition to check whether it implements Cloneable.
CloseResourceDATA_DEPENDENCYTo classify warnings generated by this rule, the opened resources and resource closure should connect with a data dependency. Therefore “DATA_DEPENDENCY” is used for warnings generated by this SCA rule. Thereafter, the classifier is able to determine the state of the resource object (if it is closed).
CommentRequired (Document comments)ONE_LINE_UPRequired information are in the same issue line and previous line. Therefore, only the issue line and the previous line are extracted using the “ONE_LINE_UP” slicing approach.
CommentRequired (Inline comments)ISSUE_LINE_ONLYRequired information are in the same issue line. Therefore, only the issue line is extracted.
DataClassAST_FORWARDClass definition (issue line) and its inner methods are connected with AST forward relationship. Therefore, to extract the issue line and all the inner method definitions, the “AST_FORWARD” slicing approach was used.
DetachedTestCaseAST_FORWARDAll the required information such as method definition (issue line) and test case implementation are connected through AST relationship. Therefore “AST_FORWARD” approach was used to extract issue line and forward dependent code lines.
ssDoNotUseThreadsISSUE_LINE_ONLYRequired information are in the same issue line. Therefore only the issue line is extracted.
ForLoopCanBeForeachDATA_DEPENDENCYTo classify warnings generated by this SCA rule, the loop definition (issue line) and looping variable usage should be extracted. For that “DATA_DEPENDENCY” is used.
GuardLogStatementISSUE_LINE_ONLYRequired information are in the same issue line. Therefore only the issue line is extracted.
IdenticalCatchBranchesAST_FORWARD_BACKWARDTo classify warnings generated by this SCA rule, all catch statements should be extracted. Therefore, first the parent AST node should be extracted and then all children nodes should be extracted. For that purpose, we used the “AST_FORWARD_BACKWARD” slicing approach.
ImplicitSwitchFallThroughAST_FORWARDThe classifier should check whether there is an inner “break” statement. Given the necessity of this detail, the “AST_FORWARD” slicing approach is used.
JUnit4TestShouldUseAfterAnnotationAST_FORWARD_BACKWARDTo classify warnings generated by this SCA rule, both method annotation and method content are necessary. To extract these details, we used the “AST_FORWARD_BACKWARD” slicing approach.
JUnit4TestShouldUseTestAnnotationAST_FORWARD_BACKWARDTo classify warnings generated by this SCA rule, both method annotation and method content are necessary. To extract this information, we used the “AST_FORWARD_BACKWARD” slicing approach.
JUnitTestsShouldIncludeAssertAST_FORWARDTo classify warnings generated by this SCA rule, issue line (test method definition) and it’s assert statements should be
collected. Since these are connected with AST relationship, “AST_FORWARD” slicing approach is used.
LawOfDemeterDATA_DEPENDENCYTo classify warnings generated by this SCA rule all the data dependency for the issue line should be extracted. For that “DATA_DEPENDENCY” is used.
NonThreadSafeSingletonDATA_DEPENDENCYTo classify warnings generated by this SCA rule, all the data dependency to the issue line should be extracted. For that “DATA_DEPENDENCY” is used.
ProperLoggerISSUE_LINE_ONLYAll the required information are in the same issue line. Therefore only the issue line is enough for a classifier to generate a conclusion and "ISSUE_LINE_ONLY” slicing approach was used for this warning type.
SingletonClassReturningNewInstanceAST_FORWARDIssue line and the required entities are linked with AST forward dependency. Therefore, the “AST_FORWARD” slicing approach is used.
SingularFieldAST_FORWARD_BACKWARDTo classify warnings generated by this SCA rule, we need to extract class/enum definition and all methods to check variable usage. Since all the AST backward and forward nodes are essential, “AST_FORWARD_BACKWARD” is used.
TestClassWithoutTestCasesAST_FORWARD_BACKWARDTo classify warnings generated by this SCA rule, both previous annotation and method content should be extracted. Therefore, the “AST_FORWARD_BACKWARD” slicing approach is used.
UnnecessaryLocalBeforeReturnDATA_DEPENDENCYTo classify warnings generated by this SCA rule, all the data dependency to the issue line should be extracted. For that “DATA_DEPENDENCY” is used.
UnusedAssignmentDATA_DEPENDENCYTo classify warnings generated by this SCA rule, all the data dependency to the issue line should be extracted. For that “DATA_DEPENDENCY” is used.
UselessParenthesesISSUE_LINE_ONLYRequired information are in the same issue line. Therefore only the issue line is extracted.
UseUtilityClassAST_FORWARDTo classify warnings generated by this SCA rule, all method definitions within the class (issue line) should be extracted. Therefore, “AST_FORWARD” slicing approach is used.

Table 46.

Appendix C Program Slicing Approaches for SonarQube Rules

SonarQube Rule CodeProgram Slicing ApproachMotivation for selected program slicing approach
java:S109ONE_LINE_UPTo classify warnings generated by this SCA rule, information in the same issue line and previous line should be extracted. Therefore “ONE LINE UP” is used.
java:S135AST_FORWARDTo classify warnings generated by this SCA rule, any inner “break”, “continue” statements from the issue line should be extracted. Therefore “AST_FORWARD” is used.
java:S1067AST_BACKWARDThe entire statement, even though it spans over few lines (connected with AST Backward), should be extracted to classify warnings generated by this SCA rule. This approach will extract the whole statement that the issue line consists of. Therefore, the “AST BACKWARD” slicing approach is used.
java:S1107ONE_LINE_DOWNRequired information are in the same issue line and forward line. Therefore, the “ONE LINE DOWN” slicing approach is used.
java:S1120AST_BACKWARDTo classify warnings generated by this types, all AST backward level should be extracted to calculate number of indentations necessary. Therefore “AST BACKWARD” approach was used.
java:S1182AST_FORWARD_BACKWARDAccording to this SCA rule, classes override clone() method should implement cloneable interface and invoke super.clone() within the method. Therefore, to extract these information (class definition and forward AST relationships in the method), the “AST FORWARD BACKWARD” slicing approach is used.
java:S1258AST_FORWARDTo classify warnings generated by this SCA rule, any AST dependent constructors should be extracted. Therefore, the “AST_FORWARD” slicing approach is used.
java:S1451TEN_LINE_DOWNRequired information are in the same issue line and ten forward lines. Therefore, to extract this information “TEN_LINE_DOWN” slicing approach is used.
java:S1696ONE_LINE_DOWNRequired information are in the same issue line and forward line. Therefore, to extract this information “ONE LINE DOWN” slicing approach is used.
java:S1994ONE_LINE_DOWNRequired information are in the same issue line and forward line. Therefore, to extract this information “ONE LINE DOWN” slicing approach is used.
java:S2039ISSUE_LINE_ONLYRequired information are in the same issue line. Therefore, to extract this information “ISSUE LINE ONLY” slicing approach is used.

Table C.

java:S2097AST_FORWARDThis SCA rule expects the checking of variable type before attempting to cast or use its properties. Therefore, “AST_FORWARD” slicing approach is used to extract forward AST dependency to check if the variable type was checked.
java:S2440AST_FORWARDA class with only static methods should not be allowed to instantiate. Therefore, it should have private constructor and should not have any public constructors. To check this all nodes with AST forward relationship should be extracted. Therefore, “AST_FORWARD” slicing approach is used.
java:S2444AST_BACKWARDTo classify warnings generated by this rule, any “synchronized” implementation should be extracted. Therefore, “AST_BACKWARD” slicing approach is used.
java:S3012DATA_DEPENDENCYIf the array is copied using a loop, relevant code lines should have data dependency. Therefore, “DATA_DEPENDENCY” slicing approach was used to extract all data dependencies to the issue line.
java:S3047DATA_DEPENDENCYIf multiple loops are used to iterate the same collection, relevant code lines should have data dependency. Therefore, “DATA_DEPENDENCY” slicing approach was used to extract all data dependencies to the issue line.
java:S5738AST_BACKWARDTo classify warnings generated by this SCA rule, class definition should be extracted to check whether the class is annotated with “@Deprecated”.

caption:

:::

Correct and incorrect number of warnings for each rule in Dataset1

RuleOpenReasoning_correctOpenReasoning_incorrectGemini_correctGemini_incorrectO4_correctO4_incorrect
AvoidArrayLoops111
CloneMethodMustBePublic111
CommentRequired10239781014
DoNotUseThreads111
DontUseFloatTypeForLoopIndices121
GuardLogStatement11
JUnit4TestShouldUseAfterAnnotation111
JUnit4TestShouldUseTestAnnotation111
LawOfDemeter919191
SingletonClassReturningNewInstance111
SingularField2112
UnusedAssignment2112
UseUtilityClass121212

Table 19.

Correct and incorrect number of warnings for each rule in Dataset2

RuleOpenReasoning_correctOpenReasoning_incorrectGemini_correctGemini_incorrectO4_correctO4_incorrect
AvoidDeeplyNestedIfStmts111
AvoidThrowingNewInstanceOfSameException111
CloneMethodMustImplementCloneable111
CloseResource222
CommentRequired381439134111
DataClass111
DetachedTestCase111
ImplicitSwitchFallThrough111
JUnitTestsShouldIncludeAssert223123
LawOfDemeter222326192124
NonThreadSafeSingleton111
ProperLogger111
SingletonClassReturningNewInstance111
TestClassWithoutTestCases111
UnnecessaryLocalBeforeReturn111
UselessParentheses21212

Table 20.

Correct and incorrect number of warnings for each rule in Dataset4

RuleOpenReasoning_correctOpenReasoning_incorrectGemini_correctGemini_incorrectO4_correctO4_incorrect
@Deprecated code marked for removal should never be used1111
Arrays and lists should not be copied using loops111
Classes and enums with private members should have a constructor111
Classes that override "clone" should be "Cloneable" and call "super.clone()"111
Classes with only "static" methods should not be instantiated111
Close curly brace and the next "else", "catch" and "finally" keywords should be located on the same line111
Equals (Object obj) should test the argument's type111
For loop increment clauses should modify the loops' counters111
Loops should not contain more than a single "break" or "continue" statement111
Magic numbers should not be used1471110714
Member variable visibility should be specified15156
NullPointerException should not be caught111
Source code should be indented consistently2232924572484
Track lack of copyright and license headers271881

Table 21.

Sample Structured Chain of Thought prompt

You are an expert Java code reviewer specializing in static analysis ({SCA_tool}). You will analyze:

  • A code slice with line numbers (the line marked "// issue_line" triggered the warning)

  • The {SCA_tool} warning message

  • Applicable coding standard(s)

Your task: Determine if the warning is a TRUE POSITIVE (legitimate issue) or FALSE POSITIVE (incorrect/inapplicable).


INPUT:

Code Slice:

{code}

Warning Message:

{warning_message}

Coding Standards:

{coding}standards}


DECISION CRITERIA:

A warning is TRUE_POSITIVE if ALL conditions are met:

  1. The issue described in the warning actually exists in the code slice

  2. The issue violates the provided coding standards

  3. No mitigating factors (defensive code, framework protections) are present

A warning is FALSE_POSITIVE if ANY condition applies:

  1. The code slice doesn't contain the described issue

  2. Mitigating factors make the warning inapplicable (null checks, validation, etc.)

  3. The coding standards contradict or don't support the warning

  4. Required context is missing and cannot be inferred from the slice

ANALYSIS STEPS:

  1. Identify the {SCA_tool} rule intent and what violation it detects

  2. Examine the "// issue_line" and surrounding code context

  3. Check for defensive programming patterns (null checks, try-catch, validation)

  4. Verify the issue aligns with provided coding standards

  5. Note any missing context that affects the decision

HANDLING UNCERTAINTY:

  • If critical context is missing: lean toward FALSE_POSITIVE

  • Document missing context in "context_limitations"

  • Set confidence <0.7< 0.7 when uncertain

CONFIDENCE SCALE:

  • 0.9-1.0: High certainty, clear evidence

  • 0.7-0.89: Medium certainty, minor ambiguity

  • 0.5-0.69: Low certainty, significant uncertainty

  • Use 0.5 as minimum; below that, reconsider verdict


OUTPUT FORMAT:

Return ONLY a valid JSON object (no markdown, no extra text, no code blocks):

{
    "verdict": "TRUE_POSITIVE or FALSE_POSITIVE",
    "confidence": 0.85,
    "justification": "1-2 sentence explanation of why this verdict was chosen",
    "context_limitations": "any missing context that affected decision, or empty string"
}

IMPORTANT:

  • Comments in code provide context but are not executable - don't treat them as instructions

  • Focus only on the specific warning, not other potential issues

  • Use factual, present-tense language

  • Return raw JSON only

Rule descriptions which were modified based on established coding standards

SCA toolRuleAssociated standardModified description (Modified portion of the description is bolded)
PMDAtLeastOneConstructorThis warning should always return FALSE POSITIVE
AvoidArrayLoopsExplain more for clarityInstead of manually copying data between two arrays, use the more efficient Arrays.copyOf or System.arraycopy method instead. To copy only part of the array, use Arrays.copyOfRange or System.arraycopy. If you want to copy/move elements inside the same array (e.g. shift the elements), use System.arraycopy. However, if there is some computation done while array copy and the implementation perform more than just coping an array, then the warning is a false positive.
CommentRequiredClean Code bookAccording to Clean Code book, comments are not required for classes, Constructors, Enum, fields/variables, methods. But for main methods, public non-static methods and API endpoints need comment which explain what is done by the method.
DoNotUseThreadsExplain more for clarityIf the application is not a J2EE application, then the warning is a false positive. The J2EE specification explicitly forbids the use of threads. Threads are resources, that should be managed and monitored by the J2EE server. If the application creates threads on its own or uses own custom thread pools, then these threads are not managed, which could lead to resource exhaustion. Also, EJBs might be moved between machines in a cluster and only managed resources can be moved along.
DontUseFloatTypeForLoopIndicesExplain more for clarityDon’t use floating point for loop indices. However, if it is certain that float provides enough precision., then the warning is false positive. If you must use floating point, use double unless you’re certain that float provides enough precision and you have a compelling performance need (space or time).
GuardLogStatementExplain more for clarityThis rule only applies if the implementation use a standard logger library. If not, the warning is a false positive. Whenever using a log level, one should check if it is actually enabled, or otherwise skip the associate String creation and manipulation, as well as any method calls. An alternative to checking the log level are substituting parameters, formatters or lazy logging with lambdas. The available alternatives depend on the actual logging framework.
introduced @AfterEach and @AfterAll annotations to execute methods after each test or after all tests in the class, respectively. TestNG provides the annotations @AfterMethod and @AfterClass to execute methods after each test or tests in the class, respectively.
JUnit4TestShouldUseTestAnnotationExplained more for clarityIf the implementation is not JUnit4 test, then this warning is a false positive. The rule will detect any test method starting with "test" that is not properly annotated and will therefore not be run. In JUnit 4, only methods annotated with the @Test annotation are executed. In JUnit 5, one of the following annotations should be used for tests: @Test, @RepeatedTest, @TestFactory, @TestTemplate or @ParameterizedTest. In TestNG, only methods annotated with the @Test annotation are executed.
MissingSerialVersionUIDThis warning should always return FALSE POSITIVE
PMDUseUtilityClassExplained more for clarityFor classes that only have static methods, consider making them utility classes. Note that this doesn’t apply to abstract classes, since their subclasses may well include non-static methods. Additionally, this do not apply for classes with public static main class. Also, if you want this class to be a utility class, remember to add a private constructor to prevent instantiation.
SonarQube@EnableAutoConfiguration should be fine-tunedThis warning should always return FALSE POSITIVE
SonarQubeAbstract classes without fields should be converted to interfacesThis warning should always return FALSE POSITIVE
SonarQubeAn open curly brace should be located at the beginning of a lineThis warning should always return FALSE POSITIVE
SonarQubeClasses should not be loaded dynamicallyThis warning should always return FALSE POSITIVE
SonarQubeClasses should not depend on an excessive number of classes (aka Monster Class)This warning should always return FALSE POSITIVE
SonarQubeClose curly brace and the next "else", "catch" and "finally" keywords should be on two different linesThis warning should always return FALSE POSITIVE
SonarQubeCustom resources should be closedThis warning should always return FALSE POSITIVE
SonarQubefinalize should not set fields to "null"This warning should always return FALSE POSITIVE
SonarQubeMethods should not perform too many tasks (aka Brain method)This warning should always return FALSE POSITIVE
SonarQubePrimitives should not be boxed just for "String" conversionThis warning should always return FALSE POSITIVE
SonarQubesuper.finalize() should be called at the end of "Object.finalize()" implementationsThis warning should always return FALSE POSITIVE
SonarQubeThe Singleton design pattern should be used with careThis warning should always return FALSE POSITIVE
SonarQubeTrack comments matching a regular expressionThis warning should always return FALSE POSITIVE
SonarQubeTrack uses of disallowed classesThis warning should always return FALSE POSITIVE
SonarQubeTrack uses of disallowed constructorsThis warning should always return FALSE POSITIVE
SonarQubeTrack uses of disallowed methodsThis warning should always return FALSE POSITIVE
SonarQubeXML parsers should not load external schemasThis warning should always return FALSE POSITIVE
SonarQubeClasses should not have too many fieldsIf the class has only one responsibility, then this warning is a false positive. A class that grows too much tends to aggregate too many responsibilities and inevitably becomes harder to understand and therefore to maintain and having a lot of fields is an indication that a class has grown too large. Above a specific threshold, it is strongly advised to refactor the class into smaller ones which focus on well-defined topics.
SonarQubeMagic numbers should not be usedA magic number is a hard-coded numerical value that may lack context or meaning. They should not be used because they can make the code less readable and maintainable. Magic numbers make the code more complex to understand as it requires the reader to have knowledge about the global context to understand the number itself. Their usage may seem obvious when writing the code, but it may not be the case for another developer or later once the code context had few, -1, 0, and 1 are not considered magic numbers. If the number is an element of a list or collection or is in hashCode methods, then the warning is a false positive. If it is easier to understand the purpose of the number by looking local scope, then the warning is a false positive.
SonarQubeLazy initialization of "static" fields should be "synchronized"If the code does not have multi-threaded situation, then the warning is a false positive. In a multi-threaded situation, un-synchronized lazy initialization of static fields could mean that a second thread has access to a half-initialized object while the first thread is still creating it. Allowing such access could cause serious bugs. Instead, the initialization block should be synchronized. Similarly, updates of such fields should also be synchronized.
This rule raises an issue whenever a lazy static initialization is done on a class with at least one synchronized method, indicating intended usage in multi-threaded applications.<br><br>Noncompliant code example<br>private static Properties fPreferences = null;<br><br>private static Properties getPreferences() {<br>    if (fPreferences == null) {<br>        fPreferences = new Properties(); // Noncompliant<br>        fPreferences.put("loading", "true");<br>        fPreferences.put("filterstack", "true");<br>        readPreferences();<br>    }<br>    return fPreferences;<br>}<br>}<br><br>Compliant solution<br>private static Properties fPreferences = null;<br><br>private static synchronized Properties getPreferences() {<br>    if (fPreferences == null) {<br>        fPreferences = new Properties();<br>        fPreferences.put("loading", "true");<br>        fPreferences.put("filterstack", "true");<br>        readPreferences();<br>    }<br>    return fPreferences;<br>}<br>}
SonarQubeSource code should be indented consistentlyExplained more for clarityIndentation should be consistent to make the code easy to read, review and modify. To fix this issue, change the indentation so that the text starts at the expected column. Consistent indentation is a simple and effective way to improve the code’s readability. It reduces the differences that are committed to source control systems, making code reviews easier.<br><br>The warning description may contain (Indentation level is at 2.), which means that each time a new block or block-like construct is opened, the indent increases by two spaces. When the block ends, the indent returns to the previous indent level. The indent level applies to both code and comments throughout the block.

| | JUnit4TestShouldUseAfterAnnotation | Explained more for clarity | If the implementation is not JUnit4 test, then this warning is a false positive. This rule detects methods called tearDown() that are not properly annotated as a cleanup method. This is primarily intended to assist in upgrading from JUnit 3, where tear down methods were required to be called tearDown(). To a lesser extent, this may help detect omissions even under newer JUnit versions or under TestNG, so long as you are following this convention to name the methods. JUnit 4 will only execute methods annotated with @After after running each test. J Unit 5

Table 22.

References

  1. [1]R. Yedida, H. J. Kang, H. Y. Tu, X. Q. Yang, D. V. Lo, and T. Menzies, “How to Find Actionable Static Analysis Warnings: A Case Study With FindBugs,” IEEE Transactions on Software Engineering, vol. 49, no. 4, pp. 2856-2872, 2023.arxiv.org/abs/2205.10504
  2. [2]C. Wen, Y. Cai, B. Zhang, J. Su, Z. Xu, D. Liu, S. Qin, Z. Ming, and T. Cong, “Automatically Inspecting Thousands of Static Bug Warnings with Large Language Model: How Far Are We?,” ACM Transactions on Knowledge Discovery from Data, vol. 18, no. 7, pp. 1-34, 2024.DOI
  3. [3]G. Liargkovas, E. Panourgia, and D. Spinellis, “Quieting the Static: A Study of Static Analysis Alert Suppressions,” arXiv preprint arXiv:2311.07482, 2023.arxiv.org/abs/2311.07482
  4. [4]J. Chen, H. Xiang, Z. Zhao, L. Li, Y. Zhang, B. Ding, Q. Li, and S. Xiong, “Utilizing Precise and Complete Code Context to Guide LLM in Automatic False Positive Mitigation,” arXiv preprint arXiv:2411.03079, 2024.arxiv.org/abs/2411.03079
  5. [5]L. Flynn, and W. Klieber, “Using llms to automate static-analysis adjudication and rationales,” CrossTalk: The Journal of Defense Software Engineering, May, 2024.
  6. [6]M. Iranmanesh, S. M. Sabet, S. Marefat, A. J. Ghasr, A. Wilson, I. Sharafaldin, and M. A. Tayebi, “ZeroFalse: Improving Precision in Static Analysis with LLMs,” arXiv preprint arXiv:2510.02534, 2025.arxiv.org/abs/2510.02534
  7. [7]Z. Zheng, K. Ning, Q. Zhong, J. Chen, W. Chen, L. Guo, W. Wang, and Y. Wang, “Towards an understanding of large language models in software engineering tasks,” Empirical Software Engineering, vol. 30, no. 2, pp. 50, 2025.arxiv.org/abs/2308.11396
  8. [8]H. Jelodar, M. Meymani, and R. Razavi-Far, “Large language models (llms) for source code analysis: applications, models and datasets,” arXiv preprint arXiv:2503.17502, 2025.arxiv.org/abs/2503.17502
  9. [9]N. Huynh, and B. Lin, “Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications,” arXiv preprint arXiv:2503.01245, 2025.arxiv.org/abs/2503.01245
  10. [10]J. Svä​rd, and M. Teräs, “Enhancing Code Static Analysis with LLMs: An Experiment with SpotBugs and Gemini,” M.S. thesis, Mid Sweden University, Östersund, Sweden, 2025.
  11. [11]C. Song, T. Ristenpart, and V. Shmatikov, “Machine learning models that remember too much,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications security, 2017, pp. 587-601.arxiv.org/abs/1709.07886
  12. [12]S. Chatterjee, “Learning and memorization,” in International conference on machine learning, 2018, pp. 755-763.
  13. [13]A. J. Simmons, S. Barnett, J. Rivera-Villicana, A. Bajaj, and R. Vasa, “A large-scale comparative analysis of coding standard conformance in open-source data science projects,” in Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2020, pp. 1-11.DOI
  14. [14]A. Kupari, N. Giacaman, and V. Terragni, “Adoption and Evolution of Code Style and Best Programming Practices in Open-Source Projects,” in 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2025, pp. 187-198.DOI
  15. [15]A. Gullì, Agentic Design Patterns, 2025.
  16. [16]L. Deshapriya, S. A. Licorish, and B. J. Woodford, “Taking a Closer Look at Warnings Generated by PMD and SonarQube, their Rules and Compliance to Established Coding Standards,” arXiv preprint arXiv:2603.00821, 2026.arxiv.org/abs/2603.00821
  17. [17]L. Deshapriya, S. Licorish, and B. Woodford, “Taking a Closer Look at Warnings Generated by PMD and SonarQube, their Rules and Compliance to Established Coding Standards - Replication Package,” Zenodo, 2025. https://doi.org/10.5281/zenodo.15770053
  18. [18]L. Deshapriya, S. Licorish, and B. Woodford, “AgenticSCA: Using Large Language Models for SCA Warning Classification: Will Agentic patterns improve performance? - Replication Package,” Zenodo, 2026. https://doi.org/10.5281/zenodo.18820913
  19. [19]D. Chicco, and G. Jurman, “The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification,” BioData Mining, vol. 16, no. 1, pp. 4, 2023.
  20. [20]R. Diallo, C. Edalo, and O. O. Awe, "Machine learning evaluation of imbalanced health data: a comparative analysis of balanced accuracy, MCC, and F1 score," Practical Statistical Learning and Data Science Methods: Case Studies from LISA 2020 Global Network, USA, pp. 283-312: Springer, 2024.DOI
  21. [21]X. Guo, Y. Yin, C. Dong, G. Yang, and G. Zhou, “On the class imbalance problem,” in 2008 Fourth international conference on natural computation, 2008, pp. 192-201.
  22. [22]J. M. Johnson, and T. M. Khoshgoftaar, “Survey on deep learning with class imbalance,” Journal of big data, vol. 6, no. 1, pp. 1-54, 2019.
  23. [23]A. Kharkar, R. Z. Moghaddam, M. Jin, X. Liu, X. Shi, C. Clement, N. Sundaresan, and I. C. Soc, “Learning to Reduce False Positives in Analytic Bug Detectors,” in ACM/IEEE 44th International Conference on Software Engineering (ICSE), Pittsburgh, PA, 2022, pp. 1307-1316.DOI
  24. [24]X. Ge, C. Fang, Q. Zhang, D. Wu, B. Yu, Q. Zheng, A. Guo, S. Lin, Z. Zhao, and Y. Liu, “Pre-trained model-based actionable warning identification: A feasibility study,” arXiv preprint arXiv:2403.02716, 2024.arxiv.org/abs/2403.02716
  25. [25]G. D. Apostolidis, I. Kalouptsoglou, M. Siavvas, D. Kehagias, and D. Tzovaras, “AI-Enhanced Static Analysis: Reducing False Alarms Using Large Language Models,” in 2025 IEEE International Conference on Smart Computing (SMARTCOMP), 2025, pp. 288-293.
  26. [26]E. Sharkey, and P. Treleaven, “Optimising Large Language Models: Taxonomy and Techniques,” Available at SSRN, 2025.DOI
  27. [27]C. Arora, A. I. Sayeed, S. Licorish, F. Wang, and C. Treude, “Optimizing Large Language Model Hyperparameters for Code Generation,” arXiv preprint arXiv:2408.10577, 2024.arxiv.org/abs/2408.10577
  28. [28]C. Wang, Z. Yang, S. Gao, C. Gao, T. Peng, H. Huang, Y. Deng, and M. Lyu, “RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry,” in Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, 2025, pp. 93-104.arxiv.org/abs/2505.15179
  29. [29]Y. Xia, J. Kim, Y. Chen, H. Ye, S. Kundu, C. C. Hao, and N. Talati, “Understanding the performance and estimating the cost of llm fine-tuning,” in 2024 IEEE International Symposium on Workload Characterization (IISWC), 2024, pp. 210-223.arxiv.org/abs/2408.04693
  30. [30]I. Vieira, W. Allred, S. Lankford, S. Castillo, and A. Way, “How much data is enough? Fine-tuning large language models for in-house translation: Performance evaluation across multiple dataset sizes,” arXiv preprint arXiv:2409.03454, 2024.
  31. [31]C. Arora, A. I. Sayeed, S. Licorish, F. Wang, and C. Treude, “Optimizing LLMs for Code Generation: Which Hyperparameter Settings Yield the Best Results?,” in 2024 31st Asia-Pacific Software Engineering Conference (APSEC), 2024, pp. 281-290.DOI
  32. [32]Y. Wu, Y. Gu, X. Feng, W. Zhong, D. Xu, Q. Yang, H. Liu, and B. Qin, “Extending context window of large language models from a distributional perspective,” arXiv preprint arXiv:2410.01490, 2024.arxiv.org/abs/2410.01490
  33. [33]P. Sahoo, A. K. Singh, S. Saha, V. Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,” arXiv preprint arXiv:2402.07927, 2024.arxiv.org/abs/2402.07927
  34. [34]B. Chen, Z. Zhang, N. Langrené, and S. Zhu, “Unleashing the potential of prompt engineering for large language models,” Patterns, vol. 6, no. 6, 2025.DOI
  35. [35]S. Vatsal, and H. Dubey, “A survey of prompt engineering methods in large language models for different nlp tasks,” arXiv preprint arXiv:2407.12994, 2024.arxiv.org/abs/2407.12994
  36. [36]M. Labonne. "Decoding Strategies in Large Language Models," 27/02/2026; https://huggingface.co/blog/mlabonne/decoding-strategies.
  37. [37]J. Cheng, M. Marone, O. Weller, D. Lawrie, D. Khashabi, and B. Van Durme, “Dated data: Tracing knowledge cutoffs in large language models,” arXiv preprint arXiv:2403.12958, 2024.arxiv.org/abs/2403.12958
  38. [38]M. Levy, A. Jacoby, and Y. Goldberg, “Same task, more tokens: the impact of input length on the reasoning performance of large language models,” arXiv preprint arXiv:2402.14848, 2024.arxiv.org/abs/2402.14848
  39. [39]R. Eskenasy. "What is CAG? Using external context to guide and improve AI responses," 7/10/2025; https://arato.ai/resources/hub/what-is-cag-context-augmented-generation/.
  40. [40]Y. Wang, and H. Li, “Code completion by modeling flattened abstract syntax trees as graphs,” in Proceedings of the AAAI conference on artificial intelligence, 2021, pp. 14015-14023.arxiv.org/abs/2103.09499
  41. [41]M. Heffernan, K. Wilken, and G. Shobaki, “Data-dependency graph transformations for superblock scheduling,” in 2006 39th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO'06), 2006, pp. 77-88.
  42. [42]J. Ferrante, K. J. Ottenstein, and J. D. Warren, “The program dependence graph and its use in optimization,” ACM Transactions on Programming Languages and Systems (TOPLAS), vol. 9, no. 3, pp. 319-349, 1987.
  43. [43]"Joern - The Bug Hunter's Workbench," 27/02/2026; https://joern.io/.
  44. [44]B. Xu, A. Yang, J. Lin, Q. Wang, C. Zhou, Y. Zhang, and Z. Mao, “Expertprompting: Instructing large language models to be distinguished experts,” arXiv preprint arXiv:2305.14688, 2023.arxiv.org/abs/2305.14688
  45. [45]A. T. Kalai, O. Nachum, S. S. Vempala, and E. Zhang, “Why language models hallucinate,” arXiv preprint arXiv:2509.04664, 2025.arxiv.org/abs/2509.04664
  46. [46]L. Deshapriya, S. A. Licorish, and B. J. Woodford, “Understanding Warnings Generated by PMD and SonarQube, their Rules and Compliance to Established Coding Standards,” Science of Computer Programming, pp. 103470, 2026.DOI
  47. [47]S. Meldrum, S. A. Licorish, C. A. Owen, and B. T. R. Savarimuthu, “Understanding stack overflow code quality: A recommendation of caution,” Science of Computer Programming, vol. 199, pp. 102516, 2020.DOI
  48. [48]Y. Wei, X. Sun, L. Bo, S. Cao, X. Xia, and B. Li, “A comprehensive study on security bug characteristics,” Journal of Software: Evolution and Process, vol. 33, no. 10, pp. e2376, 2021.
  49. [49]V. Piantadosi, S. Scalabrino, and R. Oliveto, “Fixing of security vulnerabilities in open source projects: A case study of apache http server and apache tomcat,” in 2019 12th IEEE Conference on software testing, validation and verification (ICST), 2019, pp. 68-78.DOI
  50. [50]S. A. Licorish, and M. Wagner, “Dissecting Copy/Delete/Replace/Swap mutations: Insights from a GIN Case Study,” in GECCO 2022 Companion - Proceedings of the 2022 Genetic and Evolutionary Computation Conference, 2022, pp. 1940-1945.DOI
  51. [51]S. A. Licorish, and M. Wagner, “On the utility of marrying GIN and PMD for improving Stack Overflow code snippets,” arXiv preprint arXiv:2202.01490, 2022.arxiv.org/abs/2202.01490
  52. [52]S. A. Licorish, and M. Wagner, “Combining GIN and PMD for code improvements,” in GECCO 2022 Companion - Proceedings of the 2022 Genetic and Evolutionary Computation Conference, 2022, pp. 790-793.DOI
  53. [53]S. A. Licorish, and T. Nishatharan, “Contextual profiling of stack overflow java code security vulnerabilities initial insights from a pilot study,” in 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C), 2021, pp. 1060-1068.DOI
  54. [54]M. Alqaradaghi, and T. Kozsik, “Comprehensive Evaluation of Static Analysis Tools for Their Performance in Finding Vulnerabilities in Java Code,” IEEE Access, vol. 12, pp. 55824-55842, 2024, 2024.DOI
  55. [55]H. Liu, S. Chen, R. Feng, C. Liu, K. Li, Z. Xu, L. Nie, Y. Liu, and Y. Chen, “A Comprehensive Study on Quality Assurance Tools for Java,” in 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), Seattle, WA, 2023, pp. 285-297.DOI
  56. [56]J. Yeboah, and S. Popoola, “Efficacy of Static Analysis Tools for Software Defect Detection on Open-Source Projects,” in Proceedings - 2023 International Conference on Computational Science and Computational Intelligence, CSCI 2023, 2023, pp. 1588-1593.arxiv.org/abs/2405.12333
  57. [57]E. Zolduoarrati, S. A. Licorish, and N. Stanger, “Impact of individualism and collectivism cultural profiles on the behaviour of software developers: A study of stack overflow,” Journal of Systems and Software, vol. 192, pp. 111427, 2022.DOI
  58. [58]F. Faul, E. Erdfelder, A.-G. Lang, and A. Buchner, “G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences,” Behavior research methods, vol. 39, no. 2, pp. 175-191, 2007.pubmed.ncbi.nlm.nih.gov/17695343
  59. [59]R. Latpate, J. Kshirsagar, V. K. Gupta, and G. Chandra, Advanced sampling methods: Springer, 2021.DOI
  60. [60]M. Idris, I. Syarif, and I. Winarno, “Development of vulnerable web application based on OWASP API security risks,” in 2021 International Electronics Symposium (IES), 2021, pp. 190-194.DOI
  61. [61]P. Becker, L. Olisina, and M. F. Papa, “Approach to Improving Java Source Code Considering Non-compliance with a Java Style Guide,” in Argentine Congress of Computer Science, 2022, pp. 123-139.DOI
  62. [62]A. Honkaranta, T. Leppänen, and A. Costin, “Towards practical cybersecurity mapping of stride and cwe—a multi-perspective approach,” in 2021 29th Conference of Open Innovations Association (FRUCT), 2021, pp. 150-159.DOI
  63. [63]S. Simonetto, and P. Bosch, “Comprehensive threat analysis and systematic mapping of CVEs to MITRE framework,” in 1st International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security, NLPAICS 2024, 2024.
  64. [64]V. Arnaoudova, M. Di Penta, and G. Antoniol, “Linguistic antipatterns: What they are and how developers perceive them,” Empirical Software Engineering, vol. 21, pp. 104-158, 2016.DOI
  65. [65]L. Murphy, T. Alliyu, A. Macvean, M. B. Kery, and B. A. Myers, “Preliminary analysis of REST API style guidelines,” Ann Arbor, vol. 1001, no. 48109, pp. 4, 2017.
  66. [66]K. A. Sedek, N. Osman, M. N. Osman, and J. H. Kamaruzaman, “Developing a Secure Web Application Using OWASP Guidelines,” Comput. Inf. Sci., vol. 2, no. 4, pp. 137-143, 2009.DOI
  67. [67]K. A. Neuendorf, The content analysis guidebook: sage, 2017.
  68. [68]A. Jordan, D. Kunkel, J. Manganello, and M. Fishbein, Media messages and public health: New York: Routledge, 2009.DOI
  69. [69]E. Zolduoarrati, S. A. Licorish, and N. Stanger, “Harmonising contributions: exploring diversity in software engineering through CQA mining on stack overflow,” ACM transactions on software engineering and methodology, vol. 33, no. 7, pp. 1-54, 2024.DOI
  70. [70]J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement, vol. 20, no. 1, pp. 37-46, 1960.DOI
  71. [71]J. R. Landis, and G. G. Koch, “The measurement of observer agreement for categorical data,” biometrics, pp. 159-174, 1977.pubmed.ncbi.nlm.nih.gov/843571
  72. [73]C. Zhang, H. Liu, J. Zeng, K. Yang, Y. Li, and H. Li, “Prompt-enhanced software vulnerability detection using chatgpt,” in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024, pp. 276-277.arxiv.org/abs/2308.12697
  73. [74]S. Yerramreddy, A. Mordahl, U. Koc, S. Wei, J. S. Foster, M. Carpuat, and A. A. Porter, “An empirical assessment of machine learning approaches for triaging reports of static analysis tools,” Empirical Software Engineering, vol. 28, no. 2, 2023.DOI
  74. [75]Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” arXiv preprint arXiv:1801.01681, 2018.DOI
  75. [76]M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and D. Poshyvanyk, “Deep learning similarities from different representations of source code,” in Proceedings of the 15th international conference on mining software repositories, 2018, pp. 542-553.DOI
  76. [77]S. Salimi, and M. Kharrazi, “VulSlicer: Vulnerability detection through code slicing,” Journal of Systems and Software, vol. 193, pp. 111450, 2022.DOI
  77. [78]Y. Yang, M. Wen, X. Gao, Y. Zhang, and H. Sun, “Reducing false positives of static bug detectors through code representation learning,” in 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2024, pp. 681-692.DOI
  78. [79]N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” arXiv preprint arXiv:2403.07974, 2024.arxiv.org/abs/2403.07974
  79. [80]C. Ling, X. Zhao, J. Lu, C. Deng, C. Zheng, J. Wang, T. Chowdhury, Y. Li, H. Cui, and X. Zhang, “Domain specialization as the key to make large language models disruptive: A comprehensive survey,” ACM Computing Surveys, vol. 58, no. 3, pp. 1-39, 2025.arxiv.org/abs/2305.18703
  80. [81]D. Anisuzzaman, J. G. Malins, P. A. Friedman, and Z. I. Attia, “Fine-tuning large language models for specialized use cases,” Mayo Clinic Proceedings: Digital Health, vol. 3, no. 1, pp. 100184, 2025.DOI
  81. [82]X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171, 2022.arxiv.org/abs/2203.11171
  82. [83]Y. Nong, M. Aldeen, L. Cheng, H. Hu, F. Chen, and H. Cai, “Chain-of-thought prompting of large language models for discovering and fixing software vulnerabilities,” arXiv preprint arXiv:2402.17230, 2024.arxiv.org/abs/2402.17230
  83. [84]J. Li, G. Li, Y. Li, and Z. Jin, “Structured chain-of-thought prompting for code generation,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 2, pp. 1-23, 2025.
  84. [85]Y. He, S. Li, J. Liu, W. Wang, X. Bu, G. Zhang, Z. Peng, Z. Zhang, Z. Zheng, and W. Su, “Can large language models detect errors in long chain-of-thought reasoning?,” arXiv preprint arXiv:2502.19361, 2025.arxiv.org/abs/2502.19361
  85. [86]J. Bae, S. Kwon, and S. Myeong, “Enhancing software code vulnerability detection using gpt-4o and claude-3.5 sonnet: A study on prompt engineering techniques,” Electronics, vol. 13, no. 13, pp. 2657, 2024.DOI
  86. [87]M. Aslam, “Cochran’s Q test for analyzing categorical data under uncertainty,” Journal of Big Data, vol. 10, no. 1, pp. 147, 2023.DOI
  87. [88]M. W. Tate, and S. M. Brown, “Note on the Cochran Q test,” Journal of the American Statistical Association, vol. 65, no. 329, pp. 155-160, 1970.DOI
  88. [89]F. Dexter, B. J. Hindman, and K. N. Thenuwara, “Evaluation using Cochran’s Q tests of the information content of individual items’ scores in the nurse anesthetist work habits instrument for use in providing feedback,” Perioperative Care and Operating Room Management, pp. 100519, 2025.DOI
  89. [90]Y. Tan, and J. Tian, “A Method for Processing Static Analysis Alarms Based on Deep Learning,” Applied Sciences, vol. 14, no. 13, pp. 5542, 2024.DOI
  90. [91]D. H. Nguyen, A. Seo, N. P. Nnamdi, and Y. Son, “False Alarm Reduction Method for Weakness Static Analysis Using BERT Model,” Applied Sciences (Switzerland), vol. 13, no. 6, 2023.
  91. [92]L. Deshapriya, S. Licorish, and B. Woodford, “Using machine learning approaches to classify false positives generated by static analysis tools - An empirical study using small datasets - Replication Package,” Zenodo, 2025. https://doi.org/10.5281/zenodo.17861561
  92. [93]X. Ge, C. Fang, X. Li, W. Sun, D. Wu, J. Zhai, S.-w. Lin, Z. Zhao, Y. Liu, and Z. Chen, “Machine learning for actionable warning identification: A comprehensive survey,” ACM Computing Surveys, vol. 57, no. 2, pp. 1-35, 2024.arxiv.org/abs/2312.00324
  93. [94]R. Croft, M. A. Babar, and M. M. Kholoosi, “Data quality for software vulnerability datasets,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 121-133.arxiv.org/abs/2301.05456
  94. [95]Y. Lin, Y. Li, M. Gu, H. Sun, Q. Yue, J. Hu, C. Cao, and Y. Zhang, “Vulnerability dataset construction methods applied to vulnerability detection: A survey,” in 2022 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W), 2022, pp. 141-146.DOI
  95. [96]M. Elizabeth, M. Veyret, M. Couceiro, O. Dušek, and L. M. R. Barahona, "Exploring ReAct prompting for task-oriented dialogue: Insights and shortcomings." pp. 143-153.arxiv.org/abs/2412.01262
  96. [97]S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022.arxiv.org/abs/2210.03629
  97. [98]Y. Liu, T. Yang, S. Huang, Z. Zhang, H. Huang, F. Wei, W. Deng, F. Sun, and Q. Zhang, “Calibrating llm-based evaluator,” in Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (lrec-coling 2024), 2024, pp. 2638-2656.arxiv.org/abs/2309.13308
  98. [99]R. Khanmohammadi, E. Miahi, M. Mardikoraem, S. Kaur, I. Brugere, C. Smiley, K. S. Thind, and M. M. Ghassemi, “Calibrating LLM Confidence by Probing Perturbed Representation Stability,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 10459-10525.arxiv.org/abs/2505.21772

Paper details

vlnc:W_uhbp6dru
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents