betaFeedback

WorksComputer Science

Using machine learning approaches to classify false-positives generated by static analysis tools - An empirical study using small datasets

Abstract

Static Code Analysis (SCA) tools are vital for software development to reduce the cost and time required for manual code reviews. It was established in our previous study that the best tools reported in the software engineering community return high false-positive and false-negative rates when used for code analysis. Several approaches have been taken in existing studies to identify false-positive alarms generated by SCA tools, in helping to enhance the utility of these tools. However, existing studies reveal several limitations, chiefly among which is the tendency to focus on a limited number of alarm types or failing to evaluate a large breadth of Machine-Learning (ML) techniques comprehensively. This study has addressed this opportunity. We identified the best Code Representation Learning (CRL) techniques and ML algorithms for identifying false-positives, where results show that performance outcomes vary based on the characteristics of the training datasets and the goal of the model building. Given the evidence established in this study, we recommend further investigations targeting specific configurations of advanced CRLs and ML Algorithms, including simple ML models.

Introduction

Static Code Analysis (SCA) tools are essential to software development. They enable developers to detect software code quality issues, such as vulnerabilities and performance issues in source code. SCA tools analyse source code without execution, improving security and overall source code quality [1]. However, SCA tools are often plagued by high false-positive and false-negative rates, requiring developers to sift through numerous false alarms, diminishing their productivity and focus on resolving critical issues [1, 2].

Existing studies have explored various approaches to identify false-positive and false-negative warnings1. For instance, Pujar et al. [2] used their D2A dataset to train several Machine-Learning (ML) approaches using selected features of warnings and source code. As a result, four ML approaches were selected as best. Moreover, Yang et al. [3] used several ML approaches to train ML models2. They used the source code and the warning message as the input for ML model, and they found that ML approaches can provide good results. Moreover, Code Representation Learning techniques (CRLs) [3] and some ML approaches stand out from those existing approaches in terms of their performance towards classifying SCA warnings. In recent studies, Deep Learning (DL) approaches have been dominant where they were used to train ML models for classification tasks in various domains. However, according to Yedida et al. [4], simple ML approaches like the Random Forest (RF) algorithm perform better than DL approaches for SCA warning classification tasks.

While advancements in technologies such as CRLs and ML algorithms3 have introduced sophisticated solutions, existing studies often focus on datasets with specific warning types using limited CRLs and ML algorithms [5, 6, 7, 8]. Therefore, we have conducted experiments on two public datasets using two popular SCA tools which were identified as less prone to false-positives and false-negatives by a previous study [9]. From our experiments summarised in § 4, we have identified the warning types that were mainly prone to false-positives. Extending our previous study, in the current work, we conducted an empirical investigation to explore the best CRLs and ML algorithms for identifying false-positives generated by PMD and SonarQube. Our study identifies performance patterns in ML models across various warnings using a relatively small dataset compared to existing studies. While it is challenging for most researchers to find a large, annotated dataset for their experiments, this study demonstrates that the software engineering industry can achieve good performance even with small datasets, provided the dataset is of high quality and the right ML models are employed.

In § 3, we have identified several drawbacks in previous studies, including failure to evaluate ML algorithms and CRLs that have emerged as the most effective in existing related studies, advanced and novel ML approaches that were introduced recently were not evaluated to classify SCA warnings, and most studies have experimented on limited code violation types. Addressing these issues contribute significantly to the body of evidence aimed at evaluating SCA tools.

In fact, existing studies evaluated specific CRLs and ML algorithms to find false-positive warnings generated by SCA tools [5, 10, 11, 12], however, ML algorithms like BiLSTM have not been studied for SCA warning classification. Therefore, it is necessary to benchmark these technologies against those previously evaluated. In evaluating CRLs and ML algorithms, we found four CRLs and six ML algorithms produced the best results [3, 8, 12, 13, 14]. However, none of the existing studies evaluate all the technologies, in performing a detailed comparative evaluation. Therefore, it is worth considering the selected CRLs and ML algorithms that have proven best in existing studies for classifying SCA warnings in a comprehensive study to identify false-positives. This study provides an evaluation of the performance and behavioural patterns of wide range of CRLs when used for modelling across several ML algorithms for multiple datasets. We created four warning datasets, each with distinct features. Our analysis started with two different source code datasets: the Stack Overflow dataset, which includes code snippets, and the Apache Tomcat dataset, which contains complete source code files. We then used two SCA tools, including PMD and SonarQube, to analyse each source code dataset. This process produced four SCA warning datasets. Each warning dataset has unique features based on the type of source code (snippets versus full-length files) and the specific analysis capabilities of each SCA tool.

In performing our evaluation and addressing the need for comprehensive assessment [3, 8, 12, 13, 14], the first Research Question (RQ1) was developed to identify the impact of different CRLs on the performance of the ML algorithm in classifying false-positive SCA warnings. Source codes and SCA warning messages were embedded using CRLs, and ML algorithms were used to build ML models.

RQ1: How does the performance of ML algorithms vary with the use of different CRLs in SCA warning classification?

As outlined in § 2, the performance of ML models in identifying false-positive warnings depends on the capabilities of the algorithms employed. While we have identified the most effective ML algorithms used in previous studies [3, 5, 8, 11, 12, 13, 14], none of these studies evaluated a wide breadth of these ML algorithms in a single investigation. It is essential to conduct a comprehensive evaluation of the best-performing ML algorithms to explore how they perform under varying contexts. Thus, the second Research Question (RQ2) was formulated to address this gap.

RQ2: How does SCA warning classification performance vary with the use of different ML algorithms?

The performance of the ML model depends on the practical purpose of the model building [15], which is further discussed in the § 2. We identified four possible purposes for building models in classifying SCA warnings and determined the most appropriate performance metrics to evaluate ML models for each purpose. There can also be variations in performance patterns depending on the metrics used for evaluation [15], which needs consideration. To address these gaps, the third research question has been formulated.

RQ3: How does SCA warning classification performance vary depending on the purpose of building ML models?

For our study, we utilised four warning datasets with different characteristics as explained in the § 4. For RQ1 and RQ2, the evaluation was conducted on all datasets collectively to gain a general overview of the performance trends of CRLs and ML algorithms. However, variations in performance patterns may occur in ML models depending on the characteristics of the trained datasets. Therefore, to examine how the choice of dataset influences the performance of the selected machine learning models, the fourth research question has been formulated.

RQ4: How does SCA warning classification performance vary with the use of different datasets?

As noted above, this study used four datasets to answer four research questions. The datasets are summarised in § 4, and the methodology for model building is documented in § 5. In this regard, the main contributions of this study are as follows.

  • Our evidence provides an opportunity for the community to select the best CRLs and ML algorithms from existing studies, and our datasets provide benchmarks data for future studies to use.

  • We demonstrate that ML models can train with small datasets when they are of high quality, providing suggestions for those faced with data scarcity issues.

  • We introduce multiple program-slicing approaches by adapting existing approaches for false-positive detection experiments. Here we provide support for software fault localisation efforts more generally.

  • We provide a replication package to support follow-up investigations. Our package includes detailed results from modelling along with optimised hyperparameters.

The rest of the study is organised as follows. § 2 discusses the background of the study. § 3 reviews existing studies related to building ML models to classify SCA warnings and § 4 outlines the datasets we used for our study. § 5 documents the methodology followed in our study and § 6 summarises the results generated in our study. We then discuss our results and their implications in § 7. In § 8, we outline potential threats to the validity of our outcomes. § 9 concludes the work and outlines potential areas for future research. Moreover, we provide a replication package that includes the appendices and training datasets used in this study [16].

Background

Performance metrics used in this study

Several performance metrics were used to evaluate ML models experimented with in this study, including MCC, F1-Score, Balanced Accuracy and Computational Complexity. The Matthews Correlation Coefficient (MCC) is a balanced metric used to evaluate the performance of binary classifiers, particularly when dealing with imbalanced datasets. It is frequently utilised to assess binary classification models and is computed using the following formula [17]:

MCC=TP×TN−FP×FN(TP+FP)(TP+FN)(TN+FP)(TN+FN)MCC = \frac{TP \times TN - FP \times FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}

Where,

True Positives (TP) = Number of correctly predicted positive instances.

True Negatives (TN) = Number of correctly predicted negative instances.

False Positives (FP) = Number of incorrectly predicted positive instances.

False Negatives (FN) = Number of incorrectly predicted negative instances.

The MCC is a key metric for assessing the performance of binary classification models. It accounts for all four categories of the confusion matrix: true positives, true negatives, false-positives, and false-negatives. This comprehensive approach makes MCC especially valuable when working with imbalanced datasets, where metrics like accuracy can be deceptive. A MCC score of +1 indicates perfect predictions, zero signifies performance no better than random guessing, and -1 represents complete disagreement between predictions and actual outcomes. In summary, MCC is a reliable indicator of model effectiveness, making it a crucial tool for researchers and practitioners in machine learning and statistical analysis [18].

The F1-Score is a statistical measure used to evaluate the performance of a binary classification model. It is the harmonic mean of Precision and Recall, providing a balance between the two metrics. Here, Precision measures the accuracy of positive predictions, while Recall (or sensitivity) measures the ability of the model to identify all relevant instances [19, 20]. The formula for F1-Score is:

F1 ⁣− ⁣Score=2×Precision×RecallPrecision+Recall\mathrm{F1\!-\!Score} = 2 \times\frac{\mathrm{Precision} \times\mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}

Where,

Precision=TPTP+FP\mathrm{Precision} = \frac{TP}{TP + FP}
Recall/Sensitivity=TPTP+FN\mathrm{Recall/Sensitivity} = \frac{TP}{TP + FN}

The F1-Score ranges from 0 to 1, where 1 indicates perfect Precision and Recall, and 0 indicates the poorest performance. It is beneficial as a measure when the class distribution is imbalanced, and the evaluation needs to consider both false-positives and false-negatives [20].

Balanced Accuracy is a metric used to assess the performance of a classification model, particularly when the classes are unequally distributed. It provides a more meaningful assessment than traditional accuracy, which can be misleading if one class dominates. Balanced Accuracy is calculated by averaging Sensitivity and Specificity. The formula is:

Balanced Accuracy=Sensitivity+Specificity2\mathrm{Balanced\ Accuracy} = \frac{\mathrm{Sensitivity} + \mathrm{Specificity}}{2}

Where,

Specificity=TNTN+FP\mathrm{Specificity} = \frac{TN}{TN + FP}

In this formula, Sensitivity (Recall) is the ratio of true positives to the sum of true-positives and false-negatives, indicating how effectively the model detects the positive class. Specificity is the ratio of true negatives to the sum of true-negatives and false-positives, assessing how well the model identifies the negative class. In summary, Balanced Accuracy considers both the true positive and true negative rates, making it particularly useful for evaluating models on imbalanced datasets where one class may dominate the other [18]. By treating both classes equally, Balanced Accuracy helps ensure that the model performs well across all categories, rather than just on the majority class [18].

Computational Complexity in ML model building refers to the amount of computational resources required to train and evaluate a model. This includes time complexity (how an algorithm's running time scales with the input data's size) and space complexity (the amount of memory needed to store data and intermediate computations). It involves analysing how different algorithms might perform as the dataset size or the model's complexity increases [21]. For example, some algorithms perform well with small datasets but become inefficient as datasets grow larger. Understanding Computational Complexity aids in choosing suitable algorithms and tools for model development, particularly when dealing with large datasets or real-time systems. It helps balance model performance with the limitations of available computational resources. In this study, we constructed each model in a similar environment (refer to § 5.3) and recorded the time required for training. The information regarding Computational Complexity can be used when selecting ML models to classify SCA warnings and identify false positive warnings. In our study, we measured the time taken to train each ML model, excluding the time required to create the numerical vector by embedding the source code and SCA warning description. Consequently, models with lower computational cost would be preferred.

Purposes of building ML models in classifying SCA warnings

We conducted a comprehensive literature study and summarised the outcome in § 3. It is observed that existing studies did not consider the potential purposes of building ML models to classify SCA warnings. Instead, existing studies have tried to optimise ML models with random performance metrics without focusing on these purposes [10, 11, 12, 22]. Therefore, we identified four main goals for this study in classifying warnings generated by SCA tools, and suitable performance metrics to measure these goals.

The first model-building goal would be to maximise the overall model performance. Therefore, the model will accurately identify actual and false-positive warnings as much as possible, reducing incorrect predictions. For this scenario, metrics such as accuracy provide a good understanding of the performance of the ML models and how well all the warnings are classified. However, the datasets used in this study are biased towards actual or false-positive warnings. Therefore, the accuracy performance metric provides less reliable insights regarding model performance. Consequently, we used the MCC performance metric to identify the effectiveness of ML models, as in this study, we performed binary classifications for the model building.

The second goal of the model building is to detect false-positive warnings as much as possible to minimise false warnings. This helps to reduce developers’ frustration with identifying false-positive warnings. However, optimising ML models to identify as many false-positive warnings as possible should not result in actual warnings being misclassified as false-positive warnings. In our dataset, false-positive warnings are labelled as “1” and actual warnings are labelled as “0”. Therefore, we need to optimise ML models using a performance metric that maximises true positives and minimises false-positives and false-negatives. In this scenario, reducing false-positives is mandatory, as it will classify actual warnings as non-actionable warnings, and developers will eliminate these warnings without further investigation. In contrast, reducing false-negatives is also essential, as it will classify false-positive warnings as actual warnings. If the count of this case is higher, further investigation of false-negatives will waste the developer’s time.

However, minimising false-negatives is less critical compared to minimising false-positives. Considering all these requirements, the F1-Score performance metric is the most suitable for the second goal of model building. As noted above, F1-Score is the harmonic mean of Precision and Recall performance metrics [22]. When the F1-Score equals 1.0, all the false-positive warnings are accurately classified as false-positive warnings, with no misses and no errors. Therefore, in the ideal situation (F1-Score = 1.0), developers can disregard all the identified false-positive warnings and focus on the remaining warnings. Hence, if the goal of model building is to optimise the model with the second goal, the objective is to find the best model (s) with the best F1-Score. Therefore, developers can eliminate all the warnings classified as positives. However, the eliminated warnings can include some actionable warnings, especially when the F1-Score is not perfect.

The third goal of the model building is to identify the maximum possible actual warnings with a minimum number of false-positive warnings, where the loss of actual warnings is minimised. However, false positive warnings may still occur. Considering the confusion matrix, the goal is to maximise true negatives while minimising false-positives and false-negatives. Similar to the second goal, minimising false-positives is more important than reducing false-negatives. In this context, Balanced Accuracy can be utilised as a performance metric, where both the True Positive Rate (TPR) and the True Negative Rate (TNR) are taken into account [18]. Usage of TPR and TNR makes Balanced Accuracy particularly useful in situations where the classes are imbalanced. In the ideal situation where the Balanced Accuracy value equals 1.0, all actual warnings are identified by the ML model (s). That said, developers may reduce the time spent identifying false-positive warnings. On the other hand, the same goal can be achieved by identifying false-positive warnings as determined in the second goal using F1-Score performance metrics and removing those from the whole set of warnings. However, while finding ML models with perfect F1-Score values is challenging, striving for ML models with the best Balanced Accuracy value would be a practically feasible solution.

The fourth goal would be to reduce Computational Complexity. This will reduce the model's training time, allowing us to fine-tune ML models quickly. Consequently, models will generate higher performance. For this goal, Computational Complexity performance metrics can be used [21]. All the goals discussed in this section are summarised in Table 1. Additionally, we summarise the results generated from this study in § 6, in terms of the performance metrics discussed in § 2.1. We discussed these results in § 7.

Goal IDPurpose of model buildingMetric used
G1Maximise the overall model performanceMCC
G2Correctly detect false-positive warnings as much as possible to minimise false warningsF1-Score
G3Correctly identify the maximum possible actual warnings with minimum false-positive warningsBalanced Accuracy
G4Faster training and evaluation of ML models with limited computational resources, so that the models can be fine-tuned for better performanceComputational Complexity

Table 1. Goals for model building

AST graphs and data dependency graphs

Only a slice of source code is relevant for the warning generated by the SCA tool, and identifying the relevant slice is critical for the performance of the classification model [11, 12, 23, 24]. To extract the relevant slice of the source code, several approaches are used [11, 12, 23, 24]. An Abstract Syntax Tree (AST) is a tree-like representation of software source code that illustrates its structure [25]. Each node in this tree signifies a particular construct (classes, methods, variables, etc.) found in the source code. During the conversion of source code into an AST, only the structural and content-related details of the source code are transferred, while any additional details, such as comments, are discarded. Information that gets transferred to the AST includes:

  • The variable types and their declaration locations

  • The order and definition of executable statements

  • The left and right components of binary operations

  • Identifiers and their assigned values

In this study, AST graphs are employed to extract code slices from source code where the relevant lines are dispersed throughout the source code, and the relationship between those lines depends on their structure.

In contrast, a data dependency graph is a directed graph illustrating the dependencies between data items in source code [26]. For instance, within source code, for a variable definition with an initial value, the modification of that variable and the utilisation of the variable's value present data dependencies. In this study, data dependency graphs are employed to extract code slices from source code, where the relevant lines are scattered throughout the code, and the relationship between these lines relies on their data or identifiers/variables.

Related work

CRLs and ML algorithms used to classify SCA warnings

SCA tools generate a high false-positive rate, which wastes time developers’ use for finding and eliminating false alarms [1]. Therefore, efforts have been invested towards automating the task of eliminating false warnings and identifying actual warnings, emphasising the importance of classifying SCA alarms into true and false-positives [6, 13]. Existing studies have taken different approaches, such as using extracted features as input to train ML models, hybrid approaches requiring human involvement, ranking the warning based on the likelihood of the alarm being true positive and using ML approaches on CRLs [11, 27, 28, 29]. We review the studies in this regard in this section.

ML model building using extracted features

Previous studies have utilised features extracted from SCA alarms and/or source code rather than directly incorporating source code as a training feature for ML models. Despite certain limitations, including the challenges in extracting all relevant features and uncertainty about whether the extracted features adequately represent the source code, using these features as input simplifies ML models, making it easier to fine-tune them [4, 30]. Alikhashashneh et al. [31] evaluated four ML approaches, SVM, KNN, RF, and Repeated Incremental Pruning to Produce Error Reduction (RIPPER) to classify warnings generated by SCA tools. They used software engineering metrics (source code metrics) like “codeline_count” and “Cyclomatic” (Cyclomatic complexity) to train the models, where RF was seen to be the best performer on average compared to the other examined ML techniques. According to their study, the most critical software engineering metrics were the “CountInput”, “Knots”, “CountOutput”, “CountPath”, and “Cyclomatic”; and these metrics were essential to predict the behaviour of SCA tools against a given piece of source code. Moreover, they found that reducing source code complexity, coupling, and usage of global variables in source code reduce false-positive and false-negative warnings generated by SCA tools. Cheirdari and Karabatis [32] also experimented with classifying warnings and found that features like “Author”, “vulnerability ID”, and “CWE ID” are significant when classifying SCA warnings. Overall, outcomes here show that Alikhashashneh et al. [31] found technical source code features to be correlated with SCA warning classification, while Cheirdari and Karabatis [32] identified personal identifiers to be more significant.

Berman et al. [33] experimented with using active learning to identify false-positives in SCA warnings while reducing the need for a large, labelled dataset. They used RF and lasso logistic regression with three active learning algorithms (uncertainty, query by committee, and random selection). Random Forest produced the best accuracy, while Query by Committee active learning returned the best results in terms of requiring the minimum number of labelled training examples to classify SCA alerts correctly. However, these authors did not use source code directly to train the model. Instead, they used features extracted from the alerts generated by static code analysis tools, which may not fully represent source code [4, 30], invoking curiosity about the potential for other approaches to represent SCA warnings (e.g., CRLs).

Qing et al. [34] evaluated six ML models (Naïve Bayes (NBayes), Logistic Regression, K-Nearest Neighbour (KNN), RF, Decision Tree (DT) and Support Vector Machine (SVM)) to classify SCA warning in a dataset created by Wang et al.[35] using 91 warning features, where it was found that out of those features, only 25 were relevant. This study revealed that Logistic Regression performed best in terms of false-positive rate when classifying SCA warnings. Yang et al. [27] used the “Golden Set” features suggested by Wang et al. [35] to train several ML models to classify SCA warnings. Wang et al. [35] identified 23 features, such as warning context in method and warning context in file, related to software source code and SCA warnings, which can be used to classify the SCA warnings into actionable/non-actionable warnings. They referred to these features as the “Golden Set”. The method presented by Levina and Bickel [36] was used to reduce the feature dimensionality of the SCA warnings. They used several ML algorithms to train ML models for the classification task, including Deep Neural Network (DNN), Convolutional Neural Network (CNN), DNN_weighted (similar to the DNN with a weighted strategy), RF, DT, and Linear SVM. In this latter study, Linear SVM performed best for the dataset used. Additionally, the authors recommended Linear SVM for classifying SCA warnings if the dataset is low-dimensional.

Flynn et al. [37] proposed a novel approach to use a test suit (Juliet C/C++ test suit) as a dataset for classifying warnings generated by SCA tools. They extracted metadata from the dataset and used RF (H2O.ai implementation of RF), Lasso (the glmnet implementation of lasso-regularized logistic regression), LightGBM and XGBoost for the classification. In this work, LightGBM generated the best results in terms of performance metrics such as Accuracy, Recall , and AUPRC (Area Under the Precision–Recall Curve) for classifying SCA warnings.

Yang et al. [38] proposed a human-in-the-loop active learning [39] approach to identify actionable warnings from those generated by SCA tools. They used features extracted from both software source code and SCA warnings. Additionally, they used active learning with SVM, RF, and Decision Tree algorithms for warning classification and Active learning with SVM produced the best results in terms of AUC (Area Under the Curve).

Kharkar et al. [40] proposed a Transformer-based ML approach to identify false-positives generated from analytic bug detectors. GPT-C produced a 17.5% Precision improvement on top of the baseline for null pointer warnings and demonstrated generalizability to other bug types like resource leaks. Experiments used feature-based and neural models, including DeepInferEnhance (a fine-tuned CodeBERTa model) and GPT-C (a pre-trained generative model based on GPT-2). Results showed that GPT-C achieves the highest Precision improvements, while DeepInferEnhance offers better Recall, showcasing a trade-off between the models.

Kang et al. [30] used SVM and K-Nearest Neighbours (KNN to classify SCA warnings using “Golden Set” features proposed by Wang et al. [35]. They found that SVM is better for the task. However, they found that existing Golden features underperformed with ML models, highlighting the significant limitations in existing approaches. Therefore, according to Kang et al. [30], the Golden features are not suitable for SCA warning classification tasks. Yedida et al. [4] extend the work done by Kang et al. [30] where they identified that global treatments perform worse than localised treatments for complex input data. Therefore, the authors introduced four local treatments and showed that they improved the results generated by Kang et al. [30]. While performing localised treatments may improve the ability to identify patterns in source code, recent CRL technologies also find patterns efficiently [41, 42, 43]. In fact, it was noted that training using older technologies, like feedforward networks, produce superior performance to more recent DL and CodeBERT models [41, 42, 43]. Simpler models may also run faster, allowing researchers the opportunity at fine-tuning more rapidly.

Pujar et al. [2] experimented with several ML approaches to classify their SCA warning dataset – D2A. They labelled the dataset using a novel approach called Differential Dataset Analysis and made the dataset publicly available. Using hand-crafted features as the input, they experimented with 13 well-known ML models, namely XGBoost [44], LightGBM [45], CatBoost [46] and the scikit-learn [47] implementations of RF, DTs, Extra Trees4, K-means clustering, Gradient Boosting, Ada Boost, Linear Classifiers, Gaussian NBayes, Multinomial NBayes, and Complement NBayes. Based on the performance of those ML algorithms in terms of their AUC and F1-Score, the four best models – RF, Extra Trees, LightGBM, and CatBoost were selected for further analysis with Voting and Stacking ensembles.

The studies above show that training on features extracted from SCA warnings and/or software source code produced promising results despite limitations identified by Kang et al. [30] and Yedida et al. [4]. According to Kang et al. [30] and Yedida et al. [4], the extracted features may not fully represent source code and SCA warnings to the ML models. To address these issues, researchers have employed CRLs to represent source code to train ML models for classifying SCA warnings [42, 43, 48]. Usage of CRLs to represent source code eliminates effort required to extract features from both source code and SCA warnings and enables the utilisation of the latest advanced CRLs, which may represent relationships within source code and SCA warning message effectively [41, 42, 43].

ML models that used CRLs to represent source code

CRLs were used to represent source code and SCA warnings to ML models, and reduce the effort required to extract features from source code and warning messages [41, 42, 43]. Koc et al. [49] used NBayes and Long Short-Term Memory (LSTM) ML models to classify SCA warnings using a balanced dataset. They extracted relevant slices of source code to feed the models as input, where LSTM produced the best results in terms of Precision, Recall and Accuracy. Lee et al. [10] used a CNN ML model on 10k historic static analysis alarms generated by six static analysis alarm types, which developers annotated. The dataset had about 10% of false-positive warnings, where the average Precision across all six checkers was 79.72%, showing that CNN models may achieve good performance for this task. Code lines were extracted from the code as lexical tokens and tokenised to feed those tokens to the Word2Vec embedding technique (using a skip-gram model) that was trained using source code. A CNN model was then trained on the vector resulting in the outcomes. However, the authors also reported trivial overfitting, correlation between model quality and data quality and difficulty in integrating the classifier into the Continues Integration (CI) pipeline. While these outcomes are interesting, model loss graphs and ROC graphs could have provided more concrete tests of the degree to which the models were overfitted. Additionally, there was need to use other datasets to help with supporting the inference reported.

Koc et al. [11] experimented with four families of ML approaches (Hand-engineered features, Bag of words, Recurrent Neural Networks, and Graph Neural Networks) to classify static code analysis warnings generated by three SCA tools. They used Word2Vec as the CRL technique and a class-balanced dataset for model training. In this study, LSTM outperformed the other ML models. Gated Graph Neural Networks (GGNNs) also generated promising results. The authors used two methods to represent the code: program splicing and AST. The authors experimented with transformations on source code to train with ML models and found that LSTM with “Extracting English Words from Identifiers” transformation produced the best results. However, this may not be necessary in all such scenarios as contextual CRLs such as CodeBERT is able to find relationships between words and return word vectors based on the context [3].

Tanwar et al. [12] used an Ensemble Learning model with XGBoost, RF, and Neural Network to classify static code analysis alarms. They used historic user commits as the dataset, which had approximately 12.17% of false-positive alarms in the training and validation datasets. Moreover, the model was setup in the development pipeline, and eventually, the developers labelled the alarms. Additionally, a pre-trained Code2Vec model was used for code embedding. Nagaraj et al. [5] used four ML models to classify the warnings generated by an SCA tool into true positives and false-positives. They used a Word2Vec word embedding model trained on their corpus of source code. They evaluated RF, SVM, XGBoost, and an Ensemble ML model, finding that the RF model generated the best results in terms of model Accuracy. This study demonstrates that even simple models are capable of identifying false-positive warnings.

Hegedűs and Ferenc [14] used Word2Vec as a CRL, which was trained using a public dataset to train Code2Vec. They evaluated DTs, RF, NBayes and Neural Network (NN) ML approaches with 224,484 real-world warnings, which had about 79% false positive warnings from 160 SonarQube rules. They found that the RF algorithm produced the best results in terms of Accuracy, F1-Score, Precision, and MCC values. Nguyen et al. [6] presented a novel approach to identify and reduce false alarms in static code analysis tools by combining the BERT deep learning model and a Decision Tree classifier. They used Byte Pair Encoding for word embedding and used a dataset with about 85.9% false-positive percentage, achieving 96% Precision, 92% Accuracy and 94% F1-Score for their BERT-based line-level weakness analysis model. The Alarm Classification model which used DT classifier also achieved 96% F1-Score, 0.89 AUC, and 89% Accuracy. Their proposed models used the Juliet test suite C/C++ 1.3 version as the dataset.

Yerramreddy et al. [8] experimented on three classes of ML techniques, traditional models (such as NBayes and Bayesian networks), LSTM, and Graph Neural Networks, to classify alarms generated by three SCA tools (FindSecBugs, CBMC, and JBMC) into true positives and false-positives. They used similar code pre-processing techniques to Koc et al. [11] and found that LSTM with an “Extracting English Words from Identifiers” transformation produced the best results in terms of Accuracy, Precision and Recall performance metrics, similar to the finding of Koc et al. [11].

Tan and Tians [13] experimented with four ML models to identify false-positive alarms, and Defects4J was used as the dataset, where CNNs produced the best Accuracy and Precision, and BiRNNs had the best Recall. They benchmarked their methods with Lee et al. [10] and Yang et al. [27], finding that their approach was superior when taking average Precision and Recall values into account. Yang et al. [3] explored various CRL techniques to understand the semantics of warnings generated by SCA tools. They used scratch models (Word2Vec, FastText, GloVe, and ELMo) and pre-trained models (BERT, RoBERT, and CodeBERT) as CRL techniques. Also, the authors used various ML models (MLP, LSTM, GRU, BiLSTM, BGRU, and TextCNN) for training, finding that CRL models can efficiently differentiate buggy code snippets (median F1-Score of 87.3% for binary classification and 77.4% for multi-class classification) and identify false-positive warnings (F1-Score of best performer being 75.6%). ML models with CRLs such as Word2Vec, FastText, GloVe, and CodeBERT generated higher performance than with other CRLs in binary classifications and when the training dataset contained 36% of bugs. The authors proposed a novel approach called PRISM (PRIoritize Static warnings based on aggregating multiple CRL Models) to prioritise static warnings based on aggregating multiple CRL models to reduce the false-positives generated by existing SCA tools. For CRL models, FastText outperformed other scratch models in terms of both effectiveness and stability (F1-Score) when combined with a Recurrent Neural Network (RNN) CodeBERT and RoBERTa outperformed BERT and performed better in terms of F1-Score when combined with BRNN and TextCNN. For NN models, RNN models perform better than other NN models in terms of F1-Score, but MLP gains performed poorest, showing it may not be ideal for static bug detection.

A summary of the CRLS and ML algorithms that produced high performance in identifying false-positive warnings in relevant existing studies is provided in Table 21 and Table 22, respectively. Here it is noted that various CRL technologies are used for source code prediction studies, including CodeBERT, Word2Vec, Code2Vec, FastText, and GloVe. Among these CRLs, the majority of studies have employed Word2Vec as the CRL. Additionally, CodeBERT and FastText have also been utilised in recent research to represent source code in ML models. To support modelling, various ML algorithms have been employed, including BGRU, GRU, TextCNN, BiLSTM, LSTM, and RF, with LSTM and RF being used most. Given the disconnected manner in which these approaches have been assessed, it is timely to evaluate these technologies in a single study to understand their performance patterns across various datasets.

PaperPublished YearCodeBERTWord2VecCode2VecCodeBert CPPFastText
[13]20241
[3]202411
[8]20231
[7]2023
[6]20231
[5]20221
[40]2022
[14]20221
[23]20221
[12]20211
[24]20211
[11]20191
[10]20191

Table 21. CRL techniques used in recent papers

Paper ID12345678910111213
Paper Title[13][3][8][7][6][5][40][14][23][12][24][11][10]
Published Year2024202420232023202320222022202220222021202120192019
ChatGPT1
BGRU11
GRU1
TextCNN1
DT model2
LSTM-ext11
LSTM21
GPT-C based on GPT-21
DeepInferEnhance based on CodeBERTa22
Random forest11
CNNs2321
BiLSTM121
CNN+BiGRU1
BERT1
Ensemble (XGBoost + Random Forest + Neural Network)1
Graph Neural Networks22
MLP32
Attention mechanism3
XGBoost3
Ensemble (Random forest + XGBoost + SVM)2
SVM4
NBayes2
NeuralNet2

Table 22. Machine Learning techniques used in recent papers

Ranking and prioritising SCA warnings

Given that some actual warnings may be classified as false-positives when using binary classification, by prioritising warnings based on the likelihood of the warning being true positive, developer may not miss any warning, allowing them to start fixing issues ranked highest. Therefore, ranking algorithms could be useful [24], and are explored in this section.

Ribeiro et al. [28] used an Ensemble Learning method using Adaboost and multiple DTs to rank warnings generated by three SCA tools: Clang Static Analyzer, CPPCheck, and Frama-C. They used only the warning details as the input for model training, where it was shown that the Ensemble Learning method was able to prioritise actual warnings. Ngo et al. [24] introduced DEFP, a novel approach for ranking static analysis warnings to improve efficiency in identifying actual vulnerabilities. Their approach used Word2Vec for code representation and BiLSTM to train the model. DEFP analyses SCA warnings and prioritises them according to the probability of them being True Positives (TPs). By prioritising warnings that are more probable to represent actual vulnerabilities, DEFP is said to reduce the effort wasted on false-positives, thereby enhancing developer productivity.

Sun et al. [50] proposed an interactive ranking algorithm (IRA) based on developers’ vote for warning reports, where IRA performed better than AWARE [51, 52]. The study showed the possibility of using human developers’ votes to rank SCA warnings. However, the algorithm used in this study did not use any ML model to rank SCA warnings. Siavvas et al. [53] proposed a ML-based technique for prioritising and classifying security-related static analysis alerts based on alerts’ criticality, considering information retrieved from alerts, a vulnerability prediction model, and user feedback. They used DT, NBayes, SVMs, MLP, and RF to classify warnings, finding that MLP provided the best results in terms of Accuracy, Precision and Recall performance metrics. Vu and Vo [23] introduced VULRG, a novel method to rank warnings based on their likelihood of being TPs. VULRG used Word2Vec for word embedding, and CNN and BiGRU were used for model training. They benchmarked VULRG with DEFP and found that VULRG is better than DEFP in terms of Precision and Recall performance metrics in terms of identifying the number of true positive warnings found in top-kk% warnings for all kk values they have investigated (kk= 5, 10, 20, 50, and 60).

With the latest advancement of Large Language Models (LLMs) like BERT, GPT-3, GPT-2, and LLaMA, researchers also classify SCA warnings using LLMs [7]. The best advantage of using those models is that they do not require large dataset for training [7]. Li et al. [7] used ChatGPT to identify false-positives for Use Before Initialization (UBI) Bug suggestions generated by the UBITect bug detection tool [54]. They achieve 80% Accuracy and 94% for soundness and completeness with ChatGPT-4, finding that false-positives can be identified by asking carefully constructed simple questions about function-level behaviours or function summaries from ChatGPT-4. Furthermore, this study showed the ability to classify SCA warnings using LLMs.

Datasets used in existing studies for ML model building

Recent studies have devoted considerable effort to creating warning datasets for ML model development. For instance, Tan and Tian [13] utilised a real bug database known as Defects4J [55] to train several ML models. Yang et al. [3] employed three distinct datasets: a bug dataset developed by Yang et al. [3], Defects4J [55] and a dataset created by Liu et al. [56]. Furthermore, Liu et al. [56] annotated warnings by analysing GitHub commit history. In addition, Yerramreddy et al. [8] used four warning datasets comprising three Java and one C dataset. These datasets were generated using the analysis results of JBMC and CBMC SCA tools. All the datasets utilised by Yerramreddy et al. [8] covered only specific aspects of source code quality, such as vulnerabilities and bugs. Li et al. [7] utilised a UBI (Bug dataset to conduct experiments with ChatGPT, assessing whether bugs were real or false-positive.

Nguyen et al. [6] employed the Juliet test suite (for C/C++) to train ML models. The dataset comprised a collection of test cases organised under 118 distinct CWEs.5 Nagaraj et al. [5] established a security bug dataset by analysing 2,740 Java files with the FindSecBugs SCA tool. Furthermore, Kharkar et al. [40] compiled a dataset containing 539 null dereference warnings produced by analysing seven Java repositories with the Infer SCA tool, labelled by experienced software developers. Furthermore, Hegedus and Ferenc [14] created a warning dataset by collecting 9,958 GitHub Java projects and analysing them using SonarQube. They analysed commit histories to annotate the dataset. Vu and Vo [23] utilised a dataset that was also employed by Ngo et al. [24]. In total, there were 6,620 warnings across 10 popular open-source C/C++ programs. The dataset consisted of two categories of vulnerability types: Buffer Overflow and Null Pointer Dereference. Additionally, Tanwar et al. [12] created a dataset of 10 CWE categories and labelled them using commit history analysis. Koc et al. [11] utilised two datasets to train their models, the OWASP benchmark and a dataset of warnings generated by the FindSecBugs SCA tool for 14 real-world programs. The first dataset was synthetic, while the second dataset consisted of bugs labelled manually. Lee et al. [10] utilised 9871 alarms, which were either resolved or labelled as false-positives at Samsung Electronics. The dataset comprised warnings generated by their in-house static analysis checkers, covering six types of warnings: Resource Handle Leak, Double Free, Null Pointer Dereference After Null, Tainted Loop Termination Condition, Unintentional Fall Through, and Unreachable Code.

The datasets used to train ML models were not diverse, as only a limited number of warning types were included [7, 8, 12, 40]. Consequently, none of these datasets covered all aspects of source code quality [57]. Therefore, it is emphasised that building ML models on datasets that encompass various aspects of source code quality would yield generalisable results.

Overall, in this Section we identified that existing studies focus on identifying ML models in terms of performance metrics, such as Precision, Recall and F1-Score. However, existing studies do not completely address the practical purposes of building ML models, as explained in § 2.2. In our study, we address this gap in addition to evaluating ML models’ performance based on a range of factors, including characteristics of training datasets, CRL, and the ML algorithm used to train the model. For our study, we selected the best-performing CRLs: Word2Vec, Code2Vec, CodeBERT, and FastText, as well as the best-performing ML techniques: LSTM, BiLSTM, GRU, BGRU, RF, and TextCNN, to evaluate all the combinations of those approaches (4 * 6 = 24) with four datasets. These datasets have distinct characteristics, and we evaluated all combinations of ML approaches with these datasets to identify performance patterns across different CRLs, ML techniques, and characteristics of datasets. We optimised the hyperparameters of these models and selected the best ones based on their performance on the validation dataset. According to existing studies, identifying the relevant source code slice significantly influences the performance of the ML model [24, 58]. Therefore, we selected the most suitable code-slicing framework from some frameworks used in previous studies for this purpose [11, 24].

Datasets

The Stack Overflow dataset collected by Meldrum et al. [57] and the Apache Tomcat source code downloaded from GitHub on 8th December 2023 were analysed using two SCA tools (PMD and SonarQube), making four SCA warning datasets. The original Stack Overflow and Apache Tomcat datasets were selected as these were used in multiple existing studies and are validated as good testbeds [59, 60, 61, 62, 63, 64]. Furthermore, PMD and SonarQube were chosen as they were the leading SCA tools within the software engineering practitioner community and were noted for having the lowest false-positive rate [65, 66, 67]. Therefore, the four created datasets consist of SCA warnings from code snippets (Stack Overflow) and full source code (Apache Tomcat) for two SCA tools.

A summary of the datasets is presented in Table 3, where warnings were generated from specific rules of SCA tools (Denoted as “# of rules found” in Table 3), with a significantly large number of warnings being generated for each code dataset by each SCA tool. The number of warnings is denoted as “# of alarms” in Table 3. Since manually annotating the whole dataset would demand exorbitant resources, four sample datasets were created randomly [68]. The sample size was calculated with a 95% confidence level and a 5% error rate, and the numbers of samples were divided into each SCA rule proportional to the number of warnings in the population dataset [68, 69]. The number of samples for each dataset is presented as “Sample size” in Table 3. Following the defined standard from our previous study, each SCA warnings in those sample datasets were manually examined to determine whether the warnings were true or false-positives. Moreover, the number of false-positives for each dataset is shown as “# of false-positives in the sample” in Table 3. Furthermore, the number of rules that generated false-positives is denoted as “# of rules generated false-positive” in Table 3 and the false-positive percentage for each dataset is shown as “False-positive percentage” in Table 3. It is noted that certain rules did not produce any false-positive warnings, where all warnings associated with those rules are removed, and the resulting dataset size is referred to as “# of samples for model building” in Table 3. The final false-positive percentages for each dataset are denoted as “False-positive percentage in the final dataset” in Table 3 and these datasets were then used to train and validate the ML models described in this study.

Dataset NameSCA toolCode Dataset# of rules found# of alarmsSample size# of false-positives in the sample# of rules generated false-positiveFalse-positive percentage# of samples for model buildingFalse-positive percentage in the final dataset
Dataset 1PMDSOD18880788531981318.45%13970.50%
Dataset 2PMDAT197176293536721613.43%11761.54%
Dataset 3SonarQubeSOD310838906463054.64%15020.00%
Dataset 4SonarQubeAT28112852463144146.97%29814.77%

Table 3. Summary of datasets

For sample Dataset 1, 531 samples were randomly selected with a 95% confidence level covering all the rules available in the population dataset [71]. Out of these samples, 98 were false-positives. Those false-positives were generated by 13 SCA rules out of 188, the total number of rules that generated warnings. Similarly, sample Dataset 2 was created by selecting 536 samples from Dataset 2, and there were 72 false positive samples from 16 SCA rules out of 197, the overall count of rules that generated warnings. Sample Dataset 3 was created using 646 random samples selected from Dataset 3, and 30 false-positive warnings were found from 5 SCA rules out of 310, the total number of rules that generated warnings. Finally, sample Dataset 4 was created using 631 randomly selected samples from Dataset 4. The sample Dataset 4 consisted of 44 false-positives from 14 SCA rules out of a total number of 281 rules that generated warnings. In order to classify the warnings into true or false positives, 14 defined standards were used. The defined standards are shown in Table 2 with the reason for selecting the standards for our study.

IdEstablished StandardsReason for selecting the standard
01https://wiki.sei.cmu.eduWiki page maintained by Carnegie Mellon University and used as a standard for SonarQube
02Book: Clean CodeUsed as a standard for SonarQube
03Book: Effective Java by Joshua BlochUsed as a standard for SonarQube
04Book: Mastering Regular Expressions by Jeffrey FriedlUsed as a standard for SonarQube
05Book: Refactoring: Improving the Design of Existing Code by Martin Fowler, Kent Beck, John Brant, William Opdyke, and Don RobertsUsed as a standard for SonarQube
06Google Java Coding Style GuideUsed as a standard for SonarQube and A popular and well-accepted standard [72, 73]
07https://cwe.mitre.org/data/definitions/Used as a standard for SonarQube and A popular and well-accepted standard [74, 75]
08Java DocJava official user guide
09Book: Object-Oriented Metrics in Practice: Using Software Metrics to Characterize, Evaluate, and Improve the Design of Object-Oriented Systems by Michele Lanza and Radu MarinescuUsed as a reference for PMD
10Paper: Linguistic Antipatterns - What They Are and How Developers Perceive Them[76]
11Principle of Least Astonishment[77]
12SOLID principlesUsed as a standard for SonarQube
13Transitive property of equalityUsed as a standard for SonarQube
14https://owasp.orgA popular and well-accepted standard [78, 79]

Table 2. Established Standards

SOD - Stack Overflow dataset, AT - Apache Tomcat

The sample datasets were annotated so that false-positive warnings were annotated as “1” and true positive warnings as “0”. Finally, all the rules without any false-positives were removed from the ML model for better learning on the dataset. In Table 3, the number of rules identified by the SCA tools is noted under “# of rules found” and the number of rules that generated false-positive warnings is indicated under “# of rules generated false-positive”. The difference between these figures represents the number of rules removed, as these rules did not produce any false-positive warnings. However, this caused the size of the dataset to be further reduced, as shown in Table 3 under “# of samples for model building”. Otherwise, having only true warnings for most of the SCA rules will be an unnecessary overhead for model performance.

Consequently, the model might start training on the rule names of warnings (e.g. “CommentDefaultAccessModifier”) to predict their label and predict “0” (true) for the SCA warnings belonging to the SCA rules that do not have any false-positive warnings. Thus, warnings from SCA rules that did not have any false-positive samples were removed from the sample sets that were used for model building. On the other hand, as shown in Table 3, all the sample datasets are highly class imbalanced, and to mitigate the class-imbalance problem, several steps were taken, as discussed in the § 5. Additionally, for simplicity of explanation, in other sections of this paper, all the sample datasets prepared for model building were named Dataset 1, Dataset 2, Dataset 3, and Dataset 4.

Methodology

Selecting CRLs and ML algorithms to train models

An informal literature review was conducted using Google Scholar, limited to publications from 2019 to 2024 using the following search string:

("false positive" OR "false positives") source code (learning OR predict OR training OR classify OR classifier OR classification) "static analysis"

As a result of the searches, fifty papers relevant to our study were found by reading their titles and abstracts. These papers were then read entirely, where thirteen relevant papers were selected, which aligns with our study. These studies used CRLs to represent source code and evaluated ML models, in finding false-positive SCA warnings. Referring to the existing papers, four CRLs and six ML algorithms were identified based on their popularity and novelty. These CRLs and ML algorithms are provided in Table 21 and Table 22 respectively. In Table 21 and Table 22, “1” refers to the best approach and “2” refers to the second-best approach, and so on.

As shown in Table 21, most existing papers (8 out of 13) used Word2Vec as a CRL for their studies, so Word2Vec was selected for our study. CodeBERT and FastText were also chosen as most recent studies used them [80]. Code2Vec performs better than Word2Vec for similar tasks [81, 82] and was added to the list. Therefore, Word2Vec, CodeBERT, FastText, and Code2Vec were selected as CRL techniques.

As shown in Table 22, most existing studies used LSTM (3 out of 13) and RF (2 out of 13) to train models to classify static code warnings. However, Yang et al. [3] experimented on several ML algorithms and found that BiLSTM performs better than LSTM. Therefore, both BiLSTM and LSTM are selected for our study. Furthermore, BGRU, TextCNN, and GRU were chosen from recent papers due to their novelty, as these technologies have not been extensively explored in previous studies. Hence, LSTM, BiLSTM, RF, BGRU, GRU, and TextCNN were chosen as the ML algorithms for our study.

Our study evaluates a significant number of different CRLs and ML algorithms. We evaluate CRLs from pretrained models (Code2Vec and CodeBERT) as well as CRLs trained on our datasets (Word2Vec and FastText). Furthermore, we assess CRLs that encompass both static CRLs (Word2Vec, Code2Vec, and FastText) and contextual CRLs (CodeBERT) [3]. Moreover, we examine both simple machine learning algorithms, such as RF and DL models (BiLSTM, LSTM, BGRU, GRU, and TextCNN). Additionally, we assess recently introduced technologies (Code2Vec and BiLSTM) that have not been extensively examined for ML model building to identify false-positive warnings. Moreover, none of the existing studies reviewed thoroughly investigate the impact of various CRLs on ML model performance in detecting false-positive warnings.

In total, we have chosen four CRLs and five ML algorithms to train our four datasets aimed at classifying SCA warnings and identifying false-positive SCA warnings. Furthermore, this study evaluates several ML algorithms and models across different performance metrics, including those optimised for Computational Complexity and minimising false-positives (in RQ2 and RQ3). We assessed four datasets with unique properties to uncover potential patterns in ML model performance related to identifying false-positive warnings (RQ1, RQ2, and RQ3). The training process for building the ML models on each dataset is depicted in Figure 1 and discussed in the § 5.3.

ML model training for a dataset

Figure 1. ML model training for a dataset

Data preparation and representation

Program slicing

As shown in Table 4, multiple program slicing approaches were used based on the warning types. While manually annotating the warnings in our previous study, we found two warning types based on how the warning was annotated. In the first type, the warning can be annotated by investigating only a few nearby code lines and does not need to check other blocks of the source code (i.e., the issue is localised to a few nearby code lines) [3], and those types of warnings can be annotated using the issue line or a few code lines near the issue line. In the second type, the issue is globalised to multiple source code blocks. Furthermore, among the lines of source code pertinent to the reported warning, there may be lines of source code that are irrelevant to the issue raised [8]. Thus, for the second type of warning, it is required to check the issue lines and other related issue lines that are far away from the issue line. Sometimes, those related lines were in different scopes of source code.

Program Slicing ApproachDescriptionInspiration
ISSUE_LINE_ONLYExtract the issue line only[3, 83]
ONE_LINE_UPExtract the issue line and the previous line from the source code[3, 83]
ONE_LINE_DOWNExtract the issue line and the following line from the source code[3, 83]
AST_BACKWARDExtract source code lines related to the backward AST graph from the issue line[8, 83, 84]
AST_FORWARDExtract source code lines related to the forward AST graph from the issue line[8, 83, 84, 85]
AST_FORWARD_BACKWARDExtract source code lines related to the backward AST graph and the forward AST graph from the issue line[8, 83, 84, 85]
DATA_DEPENDENCYExtract source code lines which have data dependencies to the issue line[8, 83, 84, 85]

Table 4. Source code extraction approaches

One example of the first warning type (“CommentRequired”) is shown in Figure 2.A, returned by PMD to identify comments associated only with a specific code line (“inline comments”), which requires only the issue line to be classified. In such a situation, the “ISSUE_LINE_ONLY” approach was used. In contrast, SonarQube Rule Code - “java:S1451” (Track lack of copyright and license headers) requires multiple lines to understand the issue. In such scenarios, either the previous or the following line of code, along with the line where the issue was reported, needs to be extracted. For these cases, either “ONE_LINE_UP” (the issue line and one previous line) or “ONE_LINE_DOWN” (the issue line and one following line) is used.

Figure 2. Code examples for warning types based on how code slices are extracted

In the second type of warning, the related issue lines are scattered over the source code file, therefore, extracting the related issue lines is not straightforward for this type of warning. For example, the issue type – “LawOfDemeter” is a second type (as shown in Figure 2.B), where multiple lines of the source code affect the issue. Therefore, following existing studies [8, 83, 84, 85], several techniques including “AST_BACKWARD”, “AST_FORWARD”, “AST_FORWARD_BACKWARD”, and “DATA_DEPENDENCY” were used when there is a code flow dependency (Data dependency or AST) when it comes to classifying the second types of warning. Furthermore, “AST_BACKWARD” was used to extract source code lines with AST backward dependencies, while “AST_FORWARD” was used to extract source code lines with AST forward dependencies. If source code lines with both dependencies were required, “AST_FORWARD_BACKWARD” was used. Moreover, “DATA_DEPENDENCY” was used to extract code lines with data dependencies. AST graphs and data dependency graphs are explained in § 2.3.

Table 4 shows the summary of program slicing approaches used in this study, and the source of their inspiration. The programme slicing approaches used for each PMD warning type are shown in Appendix § 11, and those used for each SonarQube warning type are shown in Appendix § 12. A simple script was created to perform the ONE_LINE_UP, ONE_LINE_DOWN, and ISSUE_LINE_ONLY slicing. However, other program slicing types required existing software to identify AST and data dependency graphs (refer to § 2.3). For such scenarios, existing studies used JOANA6 and Joern7 for program slicing [11, 24]. However, since JOANA only supports Java byte code and older versions (JDK 7 and 8) of JVM [86], Joern was used to identify AST and data dependency graphs.

After extracting all the necessary line numbers to classify the warning as either a true positive or false-positive, the corresponding lines of source code were extracted based on the previously identified line numbers. The related warning description was also added to the end of the issue line as a comment (example is shown in Figure 3). This ensured that the CRLs could easily create relationships between the issue and the issue description, because of their proximity [87, 88, 89]. The resulting code snippets were used for ML model building.

Figure 3. sample code slice with issue description

Data preparation

First, relevant code slices were extracted, as explained in § 5.2.1. However, the prepared data cannot be fed directly to ML algorithms as they expect a numerical vector representation of the dataset [3, 8, 13]. Therefore, data were transformed using the selected CRLs: Word2Vec, CodeBERT, FastText, and Code2Vec. Since pre-trained Word2Vec, and FastText were not trained on source code, our two complete, unprocessed raw source code datasets (Stack Overflow dataset and Apache Tomcat dataset) were used to train these embedding models following Nagaraj et al. [5]. The pre-trained models of CodeBERT and Code2Vec were used in their original form in this study, since those models were trained using Java source code. For CodeBERT, the "microsoft/codebert-base"8 pre-trained model was used and for Code2Vec, the “Java-large”9 pre-trained model was used.

The prepared dataset was tokenised, splitting text data into multiple tokens, as CRLs generate vector representations for individual tokens. For example, the code snippet “Object foo = "x"; // issue_line” can be split into multiple tokens such as “Object”, “foo”, “=”, “"”, “x”, “"”, “;”, “//”, “issue_line”. This task was facilitated by the “AutoTokenizer” from the “transformers” Python package, where CodeBERT was used as the source CRL. The Python “pygments” package was used for other RF ML models. In existing studies, the “javalang” Python package is used [3, 8, 14, 90]. However, the “javalang” Python package does not support extracting Java comments, which is required for most scenarios. Therefore, the “pygments” Python package was used. For all the other ML models, “Tokenizer” from “tensorflow.keras.preprocessing.text” Python package was used. The reason behind using multiple tokenizers for different models is that different models require different token formats [13, 14]. For RF models, a flattened vector of word embeddings was needed. However, for other deep learning models, the word embedding vector was created from the embedding matrix in its first (Embedding) layer. CodeBERT has its own tokenizer, and therefore, this default tokenizer was used for all the CodeBERT models.

To use the input vector for the ML models, the input vector size of every code snippet should be standardised to the same length [10, 13, 23, 24]. Therefore, for DL models (models generated by BGRU, GRU, BiLSTM, LSTM, and TextCNN ML algorithms), the input sequence is padded to the length of the largest code snippet size using their tokenizers. Since CodeBERT only supports up to 512-word size, smaller code snippets were padded, and larger code snippets were truncated using its tokenizer. For the other RF models (models with Word2Vec, FastText, and Code2Vec CRLs), the numerical vectors of each code snippet were padded to the size of the largest code snippet vector size by adding zeros at the end [91]. For RandomForest models, except for the CodeBERT-Random_Forest model, this was done after creating vectors using CRLs. Furthermore, padding/truncating was performed first for deep learning models, and then the vector was created using CRLs.

Thereafter, the tokens were vectorised using each CRL with their default parameters. The resulting vector was two-dimensional for the DL models, and considering all data items (all code snippets for all the SCA warnings), it was three-dimensional. In contrast, the input vector was converted to a two-dimensional vector (flattened) for RF, as it does not accept higher dimensional vectors. Then, the resulting vectors, which represent the code slice and the SCA warning description, were fed to the ML algorithm to train the model using each ML algorithm – LSTM, BiLSTM, RF, BGRU, GRU, and TextCNN.

Model training

All four sample datasets were prepared as described in Section 4.3, and the resulting numerical vector was used as the input vector for each ML model. Each dataset was then split into two parts. The first 80% was used for model building, and the remaining 20% was used to test the final model [8, 91]. To perform splitting, “stratify” parameters were enabled to handle the class imbalanced dataset [92].

Hyperparameter optimisation was done using Optuna [93]. The first 80% of the dataset was used to train the models using the Five-Fold cross-validation technique. The stratified version of KFold was employed, as it efficiently split the datasets despite the class-imbalanced nature of the data [92]. In addition, the SMOTE oversampling technique was also used to resolve the class imbalanced problems [94]. SMOTE creates a new dataset using the existing one by oversampling the data points with less frequent labels, ensuring that the resulting dataset does not suffer from class imbalance issues.

When each model was trained, the same dataset needed to be split into training and testing datasets. However, to ensure the data splitting process always generated the same results, a random constant (“42”) was used for the random state. The exact number was also used for random seeds in “numpy.random.seed()”, “random.seed()”, and “tensorflow.random.set_seed()”. This ensures consistency of using the same data items for each phase across all ML models.

The early stopping technique, with ‘val_loss' as the monitoring value and number 10 as the patience value, was used to train the models [91]. The early stopping technique helps to reduce the chance of overfitting the models [95, 96, 97, 98, 99]. Each model trained with 100 epochs and other hyperparameters was optimised using the Optuna [93] with the TPE sampler10 for 100 iterations for each model. The TPE sampler was the default sampling technique to find the best parameters from the hyperparameter space, and according to Hanif et al. [100], Optuna with the TPE sampler provides the highest performance. The training dataset was split into five folds [91, 92], and for each of the five iterations, four folds were used to train the model and the other fold was used to validate the model. The validation F1-Score was calculated in each iteration, and the mean value was also calculated. The goal of the optimisation process was to maximise the mean F1-Score value. The F1-Score was used for optimisation as the approach is proven to be efficient in minimising the effect of class imbalance problems [101, 102]. The hyperparameters used and the defined ranges for those parameters are shown in Table 5. According to Table 5, the hyperparameters employed for LSTM, BiLSTM, BGRU, and GRU remain consistent. Furthermore, RF and TextCNN possess their own distinct hyperparameters in contrast to other models.

ML AlgorithmHyperparametersData typeDefined value ranges
LSTM, BiLSTM, BGRU, GRUn_model_layersInt1 - 3
learning_rateFloat (Log)1e-5 - 1e-2
batch_sizeInt4 - 32
n_dense_layersInt1 – 3
model_units_l{1… n_model_layers}Int16 - 256
dense_units_l{1… n_dense_layers}Int16 - 128
l2_reg_l{1… n_dense_layers}Float (Log)1e-5 - 1e-3
Random Forestn_estimatorsInt50 - 300
max_depthInt3 – 20
min_samples_splitInt2 - 10
min_samples_leafInt1 - 10
max_featuresCategorical'sqrt' or 'log2'
bootstrapCategoricalTrue or False
max_samples (only if bootstrap = True)Float0.5 - 1.0
TextCNNlearning_rateFloat (Log)1e-5 - 1e-2
batch_sizeInt4 - 32
n_dense_layersInt1 – 3
model_l2_regFloat (Log)1e-5 - 1e-3
n_filtersInt16 - 256
kernel_size_{1 … 3}Int2, 5
dropout_rate_1Float0.1 - 0.5
dropout_rate_1Float0.1 - 0.5
dense_units_l{1 … n_dense_layers}Int16 - 128
dense_l2_reg_l{1 … n_dense_layers}Float (Log)1e-5 - 1e-3

Table 5. Hyperparameters with defined ranges

Models were trained using Optuna with the hyperparameter ranges outlined Table 5. However, model training requires a considerable amount of time. Therefore, parameter optimisation was conducted on two servers, and the server configurations are presented in Table 6. Additionally, all ML models were evenly distributed across the two servers, as shown in Table 7.

Server IDNumber of CPU coresRAM (GB)GPU device
Server 132128GPU A100 80GB
Server 232128GPU A100 40GB

Table 6. Server configurations

Random ForestGRUBGRULSTMBiLSTMTextCNN
Word2VecServer 1Server 0Server 1Server 0Server 1Server 0
Code2VecServer 0Server 1Server 0Server 1Server 0Server 1
CodeBERTServer 1Server 0Server 1Server 0Server 1Server 0
FastTextServer 0Server 1Server 0Server 1Server 0Server 1

Table 7. Model trained environment

The resulting optimised hyperparameters are shown in Appendix C in our replication package11. All the models with the best hyperparameters were retrained on server one sequentially after the initial training process. This ensures consistency across the performance metrics. The same training and testing datasets were used, as was the case in the hyperparameter optimisation phase. While training the ML model, 80% of the training dataset was used to train the model, and 20% was used to validate the model. The validation portion of the training dataset was internally used by the early stopping technique to decide the stopping point [95, 96, 97, 98, 99]. Each model was trained 10 times, and the evaluation metrics were collected separately. The final evaluation matrices are shown in Appendix U in our replication package12 for all datasets. We evaluated the ML models using MCC, F1-Score, Balanced Accuracy and Computational Complexity (Training time). In this context, the Computational Complexity is measured as the time taken to train an ML model, excluding the time needed to generate the numerical vector representation using CRLs. Therefore, the Computational Complexity of CRLs was not included. Additionally, model loss curves for the best trial are shown in Appendix M13, Appendix O14, Appendix Q15, and Appendix S16 in our replication package, and ROC curves for the best trial are shown in Appendix N17, Appendix P18, Appendix R19, and Appendix T20 in our replication package. Furthermore, the analysis of the results is discussed in § 6.

Design Of deep learning ML algorithms

The designs of the deep learning models are described in this section. The design of sequential models (GRU [103], BGRU [104], LSTM [105], and BiLSTM [105]) is shown in Figure 4.A and these sequential models were designed with a linear topology (a plain stack of layers). In contrast, TextCNN [91] was designed with non-linear topology as shown in Figure 4.B. Those designs were created following Chollet [91], Gupta et al. [106], Zhang and Wallace [107], Zhang et al. [108], and Zaremba [109].

Design of Sequential Models

(a) Design of Sequential Models

Design of TextCNN Models

(b) Design of TextCNN Models

Figure 4.

Image not extracted

Figure 4 (PDF p. 23). Designs of ML Models

As shown in Figure 4.A, for the sequential models, the first layer is the embedding layer, where the word embedding technique is integrated. The following main layers are a variable number (n_model_layer) of layers between 1 and 3, decided by the Optuna hyperparameter optimiser. The Main layer is dependent on the model. For example, in GRU models, the Main layer is a GRU layer, while in the BGRU model, the primary layer is a Bi-Directional GRU layer. Likewise, for LSM, it is a LSTM layer, and for BiLSTM, it is a Bi-Directional LSTM layer. The following Dense layers are the variable number (n_dense_layers) layers, where “Relu” is used as the activation function. Finally, there is an output Dense layer, which uses “Softmax” as the activation function.

As shown in Figure 4.B, similar to sequential models, TextCNN also used the Embedding layer to integrate the word embedding technique into the model. Then, there are three parallel one-dimensional convolution layers and a global max pool layer. The three global max-pooling layers are connected to a dropout layer, as shown in Figure 4.B. Later, similar to the sequential model, there are dense layers with similar configurations.

All the DL models used AdamW as the optimiser. It is an improved variant of the Adam Optimiser, which prevents overfitting and allows for a more stable convergence compared to Adam [110]. AdamW was used with “categorical_crossentropy” as the loss function and F1-Score (with 0.5 thresholds) as a metric [111, 112]. The learning rate of the optimiser was selected using hyperparameter optimisation using Optuna, as described in § 5.3.

Measures for the RQs

We trained the ML models as detailed in the § 5.3. The four datasets were represented in numerical vector format by source CRLs: Code2Vec, CodeBERT, FastText, and Word2Vec, and those 16 (4×44 \times4) representations (vectors) were trained using six ML algorithms: BGRU, BiLSTM, GRU, LSTM, RF, and TextCNN (visualised in Figure 1). The resulting 96 (4×4×64 \times4 \times6) ML models were trained while tuning their hyperparameters using Optuna, and their best hyperparameters are shown Appendix C in our replication package. Using the best hyperparameters, models were retrained and tested with the testing dataset 10 times each, producing 960 results. We organised these into a CSV file. The raw data is uploaded to the online repository. Table 8 outlines the data in the CSV file, where each row represents the results of each iteration of the ML model training. Each iteration of the model training involved a code representation technique and an ML algorithm, and each row of the CSV file consists of these details in addition to the dataset ID, iteration number and relevant performance values, as shown in Table 8. Additionally, the training time refers to the duration needed to train the numerical vector representation of the source code and the SCA warning description, excluding the time spent embedding textual inputs such as the source code and the SCA warning. The methodologies followed in answering each research question are outlined in Table 9. RQ1 evaluates the performance pattern of ML algorithms that vary based on the use of different code representation techniques, while RQ2 examines the performance pattern of ML models that vary based on the use of different ML Algorithms. Moreover, RQ3 assesses that the performance of ML models vary depending on the purpose of building ML models (refer to § 2.2), while RQ4 examines how the performance of ML models varies with the use of different datasets for each purpose of ML model building. Following these methodologies, we used the Python language to analyse this CSV file and answer the research questions. Additionally, we used statisy.app21, an online free app to conduct Kruskal-Wallis tests [113, 114, 115].

Column nameDescription
DatasetDataset ID (e.g. 1)
ModelModel name (e.g. – Code2Vec_LSTM)
CRLCode representation technique (e.g. Code2Vec)
ML_AlgoML Algorithm (e.g. - LSTM)
SetModel training iteration number (e.g. 1)
MCCPerformance metric
F1-ScorePerformance metric
Balanced AccuracyPerformance metric
Training TimePerformance metric

Table 8. Structure of the result CSV file

Research QuestionMethodology
RQ1All four datasets were collectively analysed across different ML algorithms, independent of the datasets used for training and testing the models. For each ML algorithm, the Kruskal-Wallis test, followed by Dunn's post hoc test with a Bonferroni correction, was conducted to evaluate the impact of different CRLs on model performance. The same statistical procedure was applied separately to each performance metric, including MCC, F1-Score, Balanced Accuracy, and Computational Complexity.
RQ2The four datasets are jointly analysed to evaluate the effect of different ML algorithms on model performance. For each performance metric, the Kruskal-Wallis test is applied, followed by Dunn's post hoc tests with a Bonferroni correction to identify pairwise differences.
RQ3To determine the best-performing ML models, the results from all datasets were aggregated. For each performance metric, the Kruskal-Wallis test was employed, followed by Dunn's post hoc tests with Bonferroni correction analyses to identify statistically significant pairwise differences between models.
RQ4The same tests done for RQ3 are conducted for each dataset separately.

Table 9. Methodology followed to prepare data to answer RQs

The datasets were introduced in § 4. The focus of this study is to identify false-positive warnings. Therefore, false-positive warnings were annotated as “1”, and actual/accurate warnings were annotated as “0”. Additionally, the datasets suffer from a class-imbalanced problem, as noted in § 3, which makes the accuracy metric biased towards the majority class [116, 117]. Therefore, following Tanha et al. [116] and He and Ma [117], MCC, F1-Score, and Balanced Accuracy values were used to evaluate the models’ performance. Even though this study focuses on identifying false-positive warnings, reducing misclassifications of actual warnings (True Negatives in our research) can be a practical objective. The Balanced Accuracy values were also calculated to better understand the models' performance for such situations. The usage of these performance metrics for the purpose of model building is discussed in the § 2.2.

Results

RQ1: How does the performance of ML algorithms vary with the use of different CRLs in SCA warning classification?

Software source code and SCA warnings were embedded using CRLs to convert the textual inputs into a numerical vector, making the input compatible with ML algorithms. Different CRLs embed textual inputs through various techniques and generate unique numerical vectors. Therefore, this research question examines the impact of utilising various CRLs for each ML algorithm in classifying SCA warnings.

The performance of ML models for each CRL and ML algorithm, measured by the MCC metric, is shown in Figure 5. According to Figure 5, the performance of the ML algorithm varies depending on the CRL used. Word2Vec performs best in BGRU, BiLSTM, GRU, and LSTM, based on median MCC values. However, Word2Vec performs worst in TextCNN, based on median MCC values. The wide boxplots in Figure 5 indicate that the performance of ML models fluctuates randomly when they are trained and tested over time. We conducted Kruskal-Wallis tests between each CRL for all the ML algorithms separately. Table 11 summarises the mean ranks of each CRLs, and the best values are highlighted in bold. Table 11 also shows that Word2Vec performs best in BGRU, BiLSTM, GRU, and LSTM, and worst in TextCNN. Furthermore, according to Table 11, FastText is the best in RF, and CodeBERT is the best in TextCNN. Additionally, the Kruskal-Wallis test results are presented in Table 10, which reveal significant differences only in CRLs for LSTM and TextCNN (p < 0.05). Moreover, the Dunn's post hoc tests with a Bonferroni correction comparing CRLs for each ML algorithm are shown in Table 12, indicating that in LSTM, Word2Vec significantly outperforms FastText (p < 0.05). Still, there is no significant performance difference between the other combinations. According to Table 12, CodeBERT outperforms both FastText and Word2Vec in TextCNN (p < 0.05), and there are no significant performance differences between the other combinations. Thus, RQ1 reveals that some ML algorithms experience a performance impact depending on the CRL used. However, for most (four out of six) ML algorithms, there are no significant differences in model performance based on the CRL used to generate numerical vectors.

MCC distribution by ML algorithm and CRL

Figure 5. Performance of ML models in each CRL and ML algorithm in MCC

ML ModelBGRUBiLSTMGRULSTMRFTextCNN
Chi25.977.694.5312.464.8910.10
df333333
p0.1130.0530.2100.0060.1800.018

Table 10. Kruskal-Wallis Test results between CRLs for each ML algorithm for the MCC performance metric

CRLBGRUBiLSTMGRULSTMRFTextCNN
Code2Vec75.8072.4082.3571.1384.2576.28
CodeBERT71.8670.2073.8364.7874.25100.46
FastText78.9983.7673.3590.1091.7573.01
Word2Vec95.3595.6492.4896.0071.7572.25

Table 11. Mean rank of CRLs for each ML algorithm for the MCC performance metric (higher is better)

CRL pairsBG RUBiL STMG RULS TMR FText CNNBG RUBiL STMG RULS TMR FText CNN
Code2Vec - CodeBERT3.942.208.526.3510.0024.191.0001.0001.0001.0001.0000.116
Code2Vec - FastText3.1911.369.0018.977.503.261.0001.0001.0000.4011.0001.000
Code2Vec - Word2Vec19.5523.2410.1224.8712.504.030.3530.1481.0000.0981.0001.000
CodeBERT - FastText7.1213.560.4825.3217.5027.451.0001.0001.0000.0870.5230.048
CodeBERT - Word2Vec23.4925.4418.6531.2225.0028.210.1390.0840.4290.0151.0000.038
FastText - Word2Vec16.3611.8719.125.9020.000.760.6831.0000.3871.0000.3041.000

Table 12. Dunn's post hoc test with a Bonferroni correction comparing CRLs for each ML algorithm for the MCC performance metric

Similar analyses are performed for F1-Score, Balanced Accuracy, and Computational Complexity performance metrics, and the results are presented in Appendix D in our replication package22 (from Appendix D.§ 1 to Appendix D.§ 7). According to Appendix D, the best CRL for each ML algorithm in terms of F1-Score and Balanced Accuracy is similar to that of the MCC. However, the performance pattern varies in terms of Computational Complexity, and CodeBERT surpasses other CRLs when paired with BGRU, BiLSTM, GRU, and LSTM in terms of Computational Complexity. Word2Vec remains the best choice for RF, and Code2Vec is the most suitable for TextCNN in terms of Computational Complexity. There is a significant difference (p<0.05p<0.05) between CRLs in GRU and LSTM for the F1-Score. Furthermore, CRLs in BiLSTM and LSTM are significantly different (p<0.05p<0.05). Conversely, CRLs across all ML algorithms show significant differences (p<0.05p<0.05) in terms of Computational Complexity.

RQ2: How does SCA warning classification performance vary with the use of different ML algorithms?

ML algorithms such as BGRU, RF, and LSTM are used to develop ML models that identify patterns in training datasets. The way patterns are identified varies depending on the algorithm used. Therefore, it is worthwhile to explore the impact of ML algorithms on SCA warning classification.

The performance patterns in classifying SCA warnings using MCC, F1-Score, Balanced Accuracy, and Computational Complexity are shown in Figure 6. According to Figure 6, the performance of the model depends on the ML Algorithms used. We conducted Kruskal-Wallis tests between ML algorithms for MCC, F1-Score, Balanced Accuracy and Computational Complexity and the results are shown in Table 13. According to Table 13, there are significant differences (p<0.05p<0.05) in model performance between ML Algorithms in each performance metric. Moreover, Table 14 demonstrates the mean rank of ML algorithms for each performance metric. According to the mean ranks shown in Table 14, Randon Forest is the best in MCC, F1-Score and Computational Complexity. However, BiLSTM is the best in Balanced Accuracy. However, according to the Dunn's post hoc tests with a Bonferroni correction results, which are shown in Table 15, RF significantly outperforms the baseline (GRU) in terms of MCC and F1-Score. Furthermore, BiLSTM, RF, and TextCNN outperform the baseline (GRU) in Balanced Accuracy. Moreover, in Computational Complexity, RF performs significantly better (p<0.05p<0.05) compared to all other ML Algorithms. Moreover, according to Table 15, most of the ML Algorithms performed significantly differently (p<0.05p<0.05, except BGRU and BiLSTM, and GRU and LSTM) in terms of Computational Complexity).

Boxplots of MCC, F1-Score, Balanced Accuracy, and Training Time by ML Algorithm

Figure 6. Performance of ML models based on the ML algorithm used

MetricsMCCF1-ScoreBalanced AccuracyTraining Time
Chi213.5318.8619.83686.53
df5555
p0.0190.0020.001<.001

Table 13. Kruskal-Wallis Test results between ML algorithms for each performance metric

ML AlgorithmMCCF1-ScoreBalanced AccuracyTraining Time
BGRU445.55453.52444.15726.59
BiLSTM498.81474.22514.48696.70
GRU430.74431.06409.08569.91
LSTM482.95488.53497.08551.50
RF526.75556.41512.5680.66
TextCNN498.20479.27505.65257.63

Table 14. Mean rank of ML algorithms for each performance metric (higher is better for MCC, F1-Score and Balanced Accuracy; lower is better for Training Time)

ML Algorithms pairsTest statistic MCCTest statistic F1-ScoreTest statistic Balanced AccuracyTest statistic Training TimeBonferroni Adjusted Significance MCCBonferroni Adjusted Significance F1-ScoreBonferroni Adjusted Significance Balanced AccuracyBonferroni Adjusted Significance Training Time
BGRU - BiLSTM-53.26-20.71-70.3429.88110.3471
BGRU - GRU14.8122.4635.07156.67111<.001
BGRU - LSTM-37.40-35.01-52.93175.09111<.001
BGRU - RF-81.20-102.89-68.42645.920.1310.0130.408<.001
BGRU - TextCNN-52.65-25.75-61.50468.95110.706<.001
BiLSTM - GRU68.0643.16105.40126.790.4210.010.001
BiLSTM - LSTM15.86-14.3117.41145.20111<.001
BiLSTM - RF-27.94-82.181.92616.0410.1191<.001
BiLSTM - TextCNN0.61-5.058.83439.07111<.001
GRU - LSTM-52.21-57.47-87.9918.4110.9520.0671
GRU - RF-96.01-125.35-103.48489.250.0290.0010.013<.001
GRU - TextCNN-67.46-48.21-96.57312.280.44110.027<.001
LSTM - RF-43.80-67.88-15.49470.8410.4261<.001
LSTM - TextCNN-15.259.26-8.57293.87111<.001
RF - TextCNN28.5577.146.91-176.9710.1911<.001

Table 15. Dunn's post hoc tests with a Bonferroni correction comparing ML algorithms for each performance metric

RQ3: How does SCA warning classification performance vary depending on the purpose of building ML models?

We introduced some potential purposes of ML model building in the context of SCA warning classification in § 2.2. In this subsection, we examine the performance of ML model variations in relation to the purpose of building ML models. The performance pattern of the ML model reflecting the purpose of the building is shown in Figure 5, Appendix D.§ 1, Appendix D.§ 2 and Appendix D.§ 3. The purpose of the model building is discussed in § 2.2. To identify the best-performing ML models for each purpose (related to specific performance metrics), Kruskal-Wallis tests were conducted for each performance metric, including MCC, F1-Score, Balanced Accuracy, and Computational Complexity. According to Table 16, in all performance metrics, at least one model significantly outperforms the other ML models (p<0.05p < 0.05). Furthermore, the mean ranks generated through the analysis shown in Table 17 indicate that CodeBERT_TextCNN is the best in MCC, and FastText_RF is the best in F1-Score. Moreover, according to Table 17, Word2Vec_LSTM is the best in Balanced Accuracy, and Word2Vec_RF is the best in Computational Complexity. Additionally, the Dunn's post hoc tests with a Bonferroni correction are shown in Appendix E in our replication package23.

MetricsMCCF1-ScoreBalanced AccuracyTraining Time
Chi258.4466.0181.91871.93
Df23232323
P<.001<.001<.001<.001

Table 16. Kruskal-Wallis Test results between ML models for each performance metric

Groups (n = 40)MCCF1-ScoreBalanced AccuracyTraining Time
Code2Vec_BGRU419.45428.60445.30739.05
Code2Vec_BiLSTM449.69415.14479.75818.51
Code2Vec_GRU442.91429.98457.60552.3
Code2Vec_LSTM429.55418.28461.08534.91
Code2Vec_RF530.63527.00520.6352.20
Code2Vec_TextCNN482.46458.96504.11237.23
CodeBERT_BGRU388.86413.05362.45468.80
CodeBERT_BiLSTM437.84431.66407.30402.35
CodeBERT_GRU394.50415.03322.06393.68
CodeBERT_LSTM382.06401.16332.64323.84
Groups (n=40n = 40)MCCF1-ScoreBalanced AccuracyTraining Time
CodeBERT_RF509.13552.13515.50138.85
CodeBERT_TextCNN625.30578.11585.45286.14
FastText_BGRU441.84442.88444.48878.00
FastText_BiLSTM517.42475.70569.29802.95
FastText_GRU391.93339.84406.05673.00
FastText_LSTM546.58555.65578.45653.83
FastText_RF582.00627.63585.5093.13
FastText_TextCNN440.48459.00461.81238.36
Word2Vec_BGRU532.05529.54524.36820.50
Word2Vec_BiLSTM590.28574.39601.60763.00
Word2Vec_GRU493.64539.40450.61660.68
Word2Vec_LSTM573.61579.03616.14693.43
Word2Vec_RF485.25518.88428.6338.48
Word2Vec_TextCNN444.56421.00471.23268.81

Table 17. Mean rank of ML models for each performance metric (higher is better for MCC, F1-Score and Balanced Accuracy; lower is better for Training Time)

According to Appendix E in our replication package, CodeBERT_TextCNN performs significantly better (p<0.05p<0.05) than the baselines (CodeBERT_LSTM, CodeBERT_BGRU, and FastText_GRU) in MCC. Furthermore, in F1-Score, CodeBERT_TextCNN, FastText_RF, Word2Vec_BiLSTM, and Word2Vec_LSTM perform significantly (p<0.05p<0.05) better than FastText_GRU. Moreover, in Balanced Accuracy, Word2Vec_BiLSTM and Word2Vec_LSTM are significantly better than CodeBERT_BGRU. Furthermore, CodeBERT_TextCNN, FastText_BiLSTM, FastText_LSTM, FastText_RF, Word2Vec_BiLSTM, and Word2Vec_LSTM are significantly better than the baseline of both CodeBERT_GRU and CodeBERT_LSTM in terms of Balanced Accuracy. In terms of Computational Complexity, there are no significant differences (p<0.05p<0.05) between CodeBert_LSTM, CodeBERT_TextCNN, FastText_TextCNN, Word2Vec_TextCNN, Code2Vec_TextCNN, CodeBERT_RF, and FastText_RF. Additionally, there are no significant differences (p<0.05p < 0.05) between the RF models. However, Word2Vec_RF, and Code2Vec_RF models are significantly better than CodeBert_LSTM and CodeBERT_TextCNN. Moreover, Word2Vec_RF is slightly better than Code2Vec_RF, according to Table 17. However, there are no significant differences between these models, as noted in Appendix E in our replication package. Consequently, outcomes here suggest that the optimal ML model varies depending on specific performance metrics, and the best choice of model also changes based on the intended purpose.

RQ4: How does SCA warning classification performance vary with the use of different datasets?

In answering previous research questions, we analysed the results disregarding the dataset used to train and test ML models. In contrast, this section examines how the choice of dataset influences the performance of the selected machine learning models. The performance patterns for each performance metric and dataset are shown in Appendix F24, Appendix G25, Appendix H26 and Appendix I27 in our replication package. According to these figures, we can observe distinct performance patterns between the datasets. Therefore, we conducted Kruskal-Wallis tests between datasets for each performance metric. According to the test results shown in Table 18, there are significant differences (p<0.05p<0.05) in performance between each dataset for each performance metric. Therefore, post hoc tests were conducted, and the results are shown in Table 19. The post hoc test results (Table 19) indicate that there are significant differences in performance between each dataset in all the performance metrics.

MetricsMCCF1-ScoreBalanced AccuracyTraining Time
Chi2518.47510.76455.9221.98
df3333
p<.001<.001<.001<.001

Table 18. Kruskal-Wallis Test results between datasets

Dataset pairMCCF1-ScoreBalanced AccuracyTraining TimeMCCF1-ScoreBalanced AccuracyTraining Time
1 - 2550.56503.27506.43-13.33<.001<.001<.0011
1 - 3333.87485.84231.95-90.10<.001<.001<.0010.002
1 - 4168.53335.2694.40-89.99<.001<.0010.0010.002
2 - 3-216.69-17.43-274.48-76.77<.0011<.0010.015
2 - 4-382.03-168.01-412.03-76.66<.001<.001<.0010.015
3 - 4-165.34-150.58-137.550.11<.001<.001<.0011

Table 19. Dunn's post hoc tests with a Bonferroni correction comparing performance of datasets

According to Table 19, there are no significant differences between Dataset 1 and Dataset 2, nor between Dataset 3 and Dataset 4 in Computational Complexity. Also, there are no significant difference between Dataset 2 and Dataset 3 in terms of F1-Score. However, there are substantial performance differences (p<0.05p<0.05) between other combinations of dataset pairs in each performance metrics. To identify ML model performance patterns within each dataset, we then conducted Kruskal-Wallis tests comparing each ML model separately across performance metrics and datasets, and the results are shown in Appendix J in our replication package28. The mean rankings for ML models are presented in Appendix K in our replication package29 (for Dataset 1 and Dataset 2) and Appendix L in our replication package30 (for Dataset 3 and Dataset 4). The best mean ranking values are highlighted in bold. According to Appendices M and N, the best-performing ML model depends on the dataset used. Moreover, according to Appendix K in our replication package and Appendix L in our replication package, the best ML model for each dataset and performance metric varies, and the best ML models for different scenarios are shown in Table 20. According to Table 20, the best-performing ML model varies based on the datasets. Moreover, RF models with either Word2Vec or Code2Vec CRLs are best in terms of Computational Complexity for all the datasets. Additionally, FastText CRL with ML algorithms such as TextCNN, BiLSTM, BGRU, and LSTM are the best in Dataset 1, Dataset 2, and Dataset 3 in terms of MCC, F1-Score, and Balanced Accuracy. For Dataset 4, Code2Vec_RF performs the best across all performance metrics. Outcomes here suggest that the performance of SCA warning classification depends on the characteristics of the dataset used to train and test ML models.

DatasetsMCCF1-ScoreBalanced AccuracyTraining Time
Dataset 1FastText_TextCNNFastText_TextCNNFastText_BiLSTMWord2Vec_RF
Dataset 2FastText_BGRUFastText_LSTMFastText_BGRUWord2Vec_RF
Dataset 3FastText_RFFastText_RFFastText_RFCode2Vec_RF
Dataset 4Code2Vec_RFCode2Vec_RFCode2Vec_RFCode2Vec_RF

Table 20. Best ML models for each dataset and performance metric

Discussion and implications

Existing studies have shown that SCA tools generate a significant amount of false-positive warnings, which wastes developers’ time, as it takes considerable effort to identify actual warnings that need to be fixed [1, 2]. Therefore, several efforts have already been invested to identify false-positive warnings, either by classifying SCA warnings or by ranking SCA warnings [3, 6, 8, 13, 50, 52, 53]. Moreover, there are a few attempts using LLMs for identifying false-positive SCA warnings [7]. However, these existing studies do not entirely address practical problems associated with classifying SCA warnings, such as understanding the potential purposes of building ML models to classify SCA warnings. Instead, existing studies have tried to optimise ML models with a few performance metrics without focusing on these purposes. Therefore, we identified four primary goals in Section 2.2 which are applicable to classify SCA warnings and now discuss our research questions below.

RQ1: How does the performance of ML algorithms vary with the use of different CRLs in SCA warning classification?

For RQ1, all the datasets were analysed together across different ML algorithms, including Word2Vec, Code2Vec, FastText, and CodeBERT, independent of the datasets used for the analysis. For each ML algorithm, the Kruskal-Wallis test followed by Dunn's post hoc test with a Bonferroni correction was performed to evaluate model performance across different CRLs (Word2Vec, Code2Vec, FastText, and CodeBERT). The same statistical analysis was conducted separately for each performance metric.

Findings from RQ1 show that the variant of CRL affects the performance of the ML algorithm, where the optimal CRL varies between different ML algorithms. For MCC, F1-Score, and Balanced Accuracy, Word2Vec excels with BGRU, BiLSTM, GRU, and LSTM. Additionally, FastText is the most effective for RF, and CodeBERT is the most effective for TextCNN in terms of MCC, F1-Score, and Balanced Accuracy. Furthermore, in terms of Computational Complexity, CodeBERT outperforms other CRLs in BGRU, BiLSTM, GRU, and LSTM. For Computational Complexity, Word2Vec is best suited for RF, while Code2Vec is optimal for TextCNN.

We found that there were significant performance differences between each CRL only for a few ML algorithms, indicating that we can use any CRL with other ML algorithms without compromising substantial performance. For example, only LSTM and TextCNN showed significant performance differences in the MCC metric between CRLs, indicating that any CRL would perform similarly to BGRU, BiLSTM, GRU, and RF without any notable differences in MCC. Conversely, for optimal model performance in MCC, Word2Vec should be combined with LSTM, and CodeBERT should be paired with TextCNN.

These findings indicate that the effectiveness of a CRL varies considerably across different performance metrics depending on the underlying ML algorithm, rather than being consistent. Word2Vec’s strong performance with RNN-based models demonstrates that dense token-level embeddings are effective for sequential modelling approaches [103, 118, 119, 120]. Conversely, FastText embeddings seem to complement tree-based models, such as RF, likely because of the additional subword information that enhances feature diversity [121, 122]. Similarly, CodeBERT’s pretrained representations of both natural language and source code benefit CNN-based models, which are adept at extracting semantic features from dense vectors [123, 124]. Conversely, Code2Vec consistently underperformed across all ML algorithms in this study. This is likely because Code2Vec is designed to learn from AST paths, which capture structural information but may not align well with the token-sequence–based classifiers used here [81, 103, 120]. Therefore, these results highlight that the interaction between the algorithm and CRL must be carefully considered, as not all code-specific representations automatically lead to improved classification performance.

Practically, these findings imply that the practitioner community may prefer lighter CRLs (in terms of training CRLs with their own source code), such as Word2Vec or FastText, when using traditional or recurrent models, reserving CodeBERT for CNN-based architectures despite its increased computational cost (considering the computational cost involved in embedding source code and SCA warning description with CodeBERT).

RQ2: How does SCA warning classification performance vary with the use of different ML algorithms?

For RQ2, all datasets are analysed collectively to evaluate the impact of different ML algorithms on model performance. For each performance metric, the Kruskal-Wallis test is conducted, followed by Dunn's post hoc tests with a Bonferroni correction to identify pairwise differences. The results for RQ2 show that the performance of SCA warning classification varies according to both the ML algorithm and the evaluation metric. RF achieves the best results in terms of MCC and F1-Score while also requiring the lowest Computational Complexity. BiLSTM, however, perform best in terms of Balanced Accuracy, although the performance difference between BiLSTM and RF in this metric was not statistically significant. Therefore, RF may be considered the most suitable algorithm for general-purpose model building. Overall, the findings indicate that selecting an ML algorithm significantly influences the performance of SCA warning classification. In contrast, results from RQ2 demonstrate that the choice of evaluation metric has a comparatively minor effect.

The better performance of RF in terms of MCC and F1-Score can be attributed to the ensemble nature of the algorithm, which effectively reduces overfitting and performs reliably in cases of class imbalance [122, 125]. Its lower Computational Complexity also highlights the efficiency of traditional ML approaches in settings with limited data. In contrast, BiLSTM’s advantage in Balanced Accuracy highlights the strength of bidirectional sequence modelling in capturing contextual dependencies, which seems particularly beneficial for warnings in the minority class [104, 105, 126]. The underperformance of GRU and LSTM compared to BGRU and BiLSTM suggests that unidirectional models are less effective at modelling the bidirectional dependencies inherent in source code [104, 105, 126]. Similarly, the relatively good results of TextCNN indicate that it is sufficient to extract local features in most warnings to capture the sequential and semantic characteristics of SCA warnings and source code [124]. Additionally, it has benefited from advanced program slicing techniques and appending SCA warning messages near the issue line.

Overall, these findings demonstrate that the choice of algorithm has a significant influence on classification outcomes, with RF generally outperforming other algorithms. Consequently, the results suggest that the practitioner community may prefer to use RF, as it provides strong performance with lower Computational Complexity.

RQ3: How does SCA warning classification performance vary depending on the purpose of building ML models?

For RQ3, we determine the best-performing ML models, and for this purpose, the results from all datasets were aggregated. For each performance metric, the Kruskal-Wallis test was employed, followed by Dunn's post hoc tests with a Bonferroni correction analyses to identify statistically significant pairwise differences between models. Outcomes from RQ3 reveal significant differences between ML models (CRLs paired with ML algorithms) within each performance metric. According to the findings, CodeBERT_TextCNN performs best in terms of MCC, while FastText_RF is the top performer for F1-Score. Additionally, Word2Vec_LSTM performs best in terms of Balanced Accuracy, while Word2Vec_RF excels in terms of Computational Complexity. Comparing the results from RQ1 and RQ2, the results generated for RQ3 differ slightly. According to RQ1 and RQ2, FastText_RF was the best ML model in terms of MCC, F1-Score, and Balanced Accuracy. However, in RQ3, FastText_RF is the best only in F1-Score. Considering all these results show that model performance depends on the variant of both the CRL and ML algorithms used to build the model.

CodeBERT captures rich semantic information from source code, while TextCNN excels at detecting local patterns [123, 124]. Therefore, CodeBERT_TextCNN may achieve the best overall performance in SCA warning classification. FastText represents words (or tokens) using subword information, enabling it to generate useful embeddings even for rare or unseen tokens, such as uncommon variable names in code [121]. Moreover, RF is an ensemble of DTs, with each tree learning a different perspective of the data; the forest then takes a majority vote [122]. This improves its ability to recognise a broader range of source code patterns and effectively identify nonlinear relationships, making RF suitable for detecting false-positive warnings. As a result, the FastText_RF combination generally achieves a good balance between Precision and Recall, thereby boosting the F1-Score.

Word2Vec maps tokens into dense vectors that retain semantic similarity, grouping warnings with similar wording (e.g., “unused variable” vs. “redundant variable”) together in the vector space [118, 127]. This provides the model with a compact yet meaningful representation of SCA warnings [118, 127]. Additionally, LSTM is designed to learn dependencies in sequences, making it more effective at tracking sequential information than ML algorithms like RF [105, 119, 120, 122]. This may help capture minority classes more accurately, improving the handling of class-imbalanced datasets and increasing Balanced Accuracy. Since both Word2Vec and RF are lightweight, with low Computational Complexity, Word2Vec_RF could achieve the best performance in terms of computational efficiency. Therefore, CodeBERT_TextCNN can be used if overall performance is the priority, FastText_RF for detecting false-positive warnings, and Word2Vec_LSTM for identifying actual warnings with minimal false-positives. Word2Vec_RF is also suitable for faster training and evaluation of ML models with limited computational resources, allowing for fine-tuning to improve performance.

RQ4: How does SCA warning classification performance vary with the use of different datasets?

Each dataset possesses unique characteristics, as outlined in § 4. Datasets 1 and 2 (PMD) have substantially higher false-positive percentages (70.5% and 61.54%), whereas Datasets 3 and 4 (SonarQube) have much lower false-positive percentages (20.00% and 14.77%). In terms of source code length, Datasets 1 and 3 were derived from code snippets, while Datasets 2 and 4 come from the Apache Tomcat full-length source code. Consequently, the unique characteristics of the datasets motivated RQ4 to examine performance variability across ML models and datasets. Therefore, for each dataset separately, the Kruskal-Wallis test was employed to compare each ML model, followed by Dunn's post hoc tests with a Bonferroni correction to identify statistically significant pairwise differences between models. Kruskal-Wallis tests indicate significant differences in performance across datasets.

The experimental results demonstrate a clear relationship between dataset characteristics and the performance of the ML models. Datasets 1 and 2, which were generated using PMD and contain a high percentage of false-positive warnings, required more sophisticated deep learning architectures to achieve satisfactory performance. For example, the best-performing models across most metrics for Dataset 1 were FastText-based deep learning variants (TextCNN and BiLSTM), while Dataset 2 also favoured FastText-based recurrent architectures (BGRU and LSTM). The majority of the warnings in Datasets 1 and 2 consist of simple warnings which require only investigating a single line of source code. Related rules for these warnings include CommentRequired, MethodArgumentCouldBeFinal and NoPackage. Therefore, FastText and Word2Vec are efficient in representing such simple source codes [121]. Additionally, FastText and Word2Vec are trained with our source code datasets, which boosts their efficiency in representing source codes related to warnings in Datasets 1 and 2. Further, because of a higher false-positive percentage, there are more variations of patterns in the SCA warnings in Datasets 1 and 2, even though source codes related to these warnings are simple. Therefore, these datasets have benefited from the complex pattern identification capability of DL algorithms [103, 104, 105, 119, 120, 126].

In contrast, Datasets 3 and 4, generated by SonarQube with significantly lower false-positive percentages, exhibited different trends. For Dataset 3, the classical ML algorithm (RF) with FastText embeddings (FastText_RF) consistently outperformed DL variants in MCC, F1-Score, and Balanced Accuracy, suggesting that when warnings have fewer false-positives, simpler models may suffice. Similarly, the Code2Vec_RF model outperformed on Dataset 4 across all performance metrics considered in this study, further supporting the fact that, with a lower false-positive percentage, classical ML algorithms such as RF perform best. Furthermore, in both Datasets 3 and 4, the majority of the warnings were generated for the SonarQube rule “Source code should be indented consistently” and “Magic numbers should not be used”. However, their proportions (number of warnings for the rule “Source code should be indented consistently” divided by the number of warnings for the rule “Magic numbers should not be used”) differ. For Dataset 3, this proportion is 2.5, while in Dataset 4, it is 12. Additionally, in Dataset 4, there are warnings for more variety of rules. Therefore, Dataset 3 can be represented with simpler word embedding models such as FastText, while Dataset 4 requires complex CRL models such as Code2Vec [81].

Another key observation relates to training efficiency. For both Dataset 1 and Dataset 2, Word2Vec with RF achieved the best Computational Complexity. In contrast, for Datasets 3 and 4, Code2Vec with RF achieve the best Computational Complexity. Out of the six ML algorithms used in this study, RF is the fastest ML algorithm for training ML models. Meanwhile, Word2Vec is the fastest and simplest among the four CRLs used in the study. Thus, RF is the best ML Algorithm for every dataset in terms of Computational Complexity. Datasets 1 and 2 may have many SCA warnings, which only requires a single issue line to classify SCA warnings. Therefore, for these datasets, Word2Vec provides the best performance in terms of Computational Complexity. However, Datasets 3 and 4 might require more complex and semantic source code representations. As a result, Code2Vec may produce more compatible code representations than Word2Vec and FastText for Datasets 3 and 4. Therefore, Code2Vec_RF provided the best performance for Datasets 3 and 4 in terms of Computational Complexity. Moreover, comparing dataset characteristics with classification results shows that source code length does not significantly impact performance, likely due to the use of improved code-slicing techniques that extract only the relevant portions of the source code. Consequently, higher false-positive percentages appear to necessitate more sophisticated deep learning architectures to compensate for noise. In comparison, datasets with lower false-positive percentages can be effectively modelled with simpler approaches.

The best CRLs (or the only CRL used for their study) in each existing study are shown in Table 21 and the best ML algorithms in each existing study are shown in Table 22. According to Table 21, Word2Vec is the best in many existing studies. However, according to Table 22, we cannot identify any best ML algorithm from existing studies. Considering the existing studies discussed in § 3, none of the existing studies have comprehensively analysed many ML models with combinations of various CRLs and ML Algorithms, in terms of various perspectives, such as multiple performance metrics, model-building goals, and multiple datasets with unique characteristics. Therefore, we have selected four CRLs and six ML algorithms used in existing studies and evaluated these using four SCA warning datasets. Results generated from our study show that the best CRL and ML algorithm is based on the characteristics of the dataset and the purpose of the model building. Nagaraj et al. [5] identified that even simple models can produce significant performance, and our study yields the same results as those of Nagaraj et al. [5]. Moreover, some studies only used specific warning types [5, 6, 7, 8]. However, in our study, we used all the warning types which generated false-positive warnings for our two source code datasets by PMD and SonarQube. Furthermore, existing studies optimised ML models without considering the purpose of model building [10, 11, 12, 21]. The majority of them labelled false-positives of SCA warnings as “1” and “0” for actual warnings, and they optimised their models with F1-Score. However, as explained in § 2.2, these models are optimised to find false-positive warnings, but they do not completely cater to all the identified purposes. To address this gap, we identified four possible purposes of model building and identified the best performance metrics to achieve these purposes. We then analysed the performance of ML models based on these performance metrics to find the best CRLs, ML Algorithms and ML models for each purpose of model building. Additionally, we utilised four SCA warning datasets to identify the performance patterns of ML models in relation to the characteristics of the datasets. However, existing studies discussed in § 3 did not evaluate the model performance against the characteristics of the datasets. Furthermore, most of the existing studies discussed in § 3, used one simple program slicing approach for every SCA warning type, such as using the issue line or using some lines above and below the issue line. In contrast, we employed multiple program slicing approaches, depending on the type of SCA warning. Consequently, this study makes a significant contribution to the body of scientific knowledge.

Implications for tool developers

SCA tools developers need to enhance SCA tools to reduce false-positive warnings. Therefore, they can employ ML models optimised with F1-Score performance metric to identify false-positive SCA warnings and detect common patterns of such warnings. Recognising these patterns will help tool developers decrease the occurrence of false-positives. Developers aim to identify all issues in source code, even if it results in false positive warnings. In contrast, users may prefer fewer false-positives and are less concerned about missing some real warnings. Consequently, they can implement a sensitivity parameter, allowing users to select the desired behaviour of the tools. This enables users to choose between reducing false-positives and maximising the detection of actual warnings.

Implications for the Practitioners community

ML models can be used to filter false positive SCA warnings. However, results generated from this study show that models do not perform perfectly. Therefore, classifying SCA warnings require ML models that are selected based on specific goals, as described in § 2.2. For example, the practitioner community might need to maximise the overall model performance. Consequently, they can use ML models with the highest MCC values, which provide a model with the best overall performance. Additionally, MCC is a more robust performance metric when the dataset exhibits class-imbalance. On the other hand, they can use ML models with the best F1-Score values, if they need to correctly detect false-positive warnings as much as possible to minimise false warnings. Conversely, they can use ML models optimised with Balanced Accuracy metric if they need to correctly identify the maximum possible actual warnings with the minimum number of false-positive warnings. Finally, if they need to train and evaluate ML models quickly with limited computational resources, they can utilise models with the most efficient Computational Complexity. Therefore, they can fine-tune the models for better performance. Additionally, in this study, models are trained while optimising F1-Score. However, training can be done based on the purpose of model building.

Furthermore, results generated through this study show that the performance of the ML model depends on the CRL used, the ML algorithm employed, the characteristics of the dataset and the purpose of model building. Word2Vec and FastText can be used for SCA warnings, which involve a few nearby issue lines in the source code where there are no complex relationships between several source code regions. In contrast, when complex and semantic code representation is required to classify SCA warnings, Code2Vec, and CodeBERT can be utilised due to their ability to represent complex code effectively. The selection of the ML algorithm is based mostly on the complexity of the SCA warnings. If there are more false-positives in the source code, or the complexity of the SCA warning is higher, then deep learning ML algorithms can be used. On the other hand, if the warnings are simple, then using simpler models such as RF is sufficient. Therefore, the practitioner community can utilise CRLs and ML algorithms, based on the characteristics of the dataset, rather than relying on these technologies solely for novelty and advanced methodologies. Moreover, the use of performance metrics tailored to the task's purpose is also vital, rather than relying on common metrics such as Accuracy and F1-Score, which have been used in previous studies [6, 8, 13, 40]. Furthermore, employing hyperparameter optimisation methods such as Optuna, along with advanced program slicing approaches based on the SCA rule, will significantly enhance the model's performance.

Implications for the research community

Many existing studies primarily focus on experimenting with ML algorithms, overlooking the importance of evaluating CRLs for classifying false-positive SCA warnings [5, 6, 13]. However, we identified in this study that variants of both CRLs and ML Algorithms are essential. Therefore, it is vital to experiment with both CRLs and ML Algorithms to classify SCA warnings. Moreover, we identified that sometimes simple ML Algorithms, such as RF, perform well with the right CRL. It will be worth experimenting with multiple CRLs using simple ML Algorithms, such as RF and DTs, as these models have lower Computational Complexity. Therefore, it is faster to fine-tune these models.

Furthermore, we assessed our datasets, which were created by two SCA tools and exhibit different false-positive warning percentages. We found in our study that model performance depends on the traits of the datasets. Therefore, we urge academia to evaluate model performance using high-quality datasets and to make these datasets publicly accessible. Our previous research contributed to the academic community by making the datasets, which were used in this study, publicly available. Thus, the research community can assess and compare these results with other CRLs and ML Algorithms. Moreover, multiple performance metrics for ML model evaluation are emphasised. Additionally, it is vital to evaluate ML models based on the model-building goals as well as characteristics of the datasets and multiple performance metrics, as discussed in the Section 7, as finding the best ML model with the best Accuracy or F1-Score is not always enough for the practitioner community.

Moreover, it would be a good idea to train a small sample of the dataset to find the optimal combination of CRLs and ML algorithms and optimise hyperparameters, thereafter, training on the whole dataset to build the final model. This reduces the overhead of training multiple ML models with a large dataset. Ultimately, results show that a well-trained ML model can yield effective results even with a relatively small dataset. However, program slices of source code should be correctly extracted based on the warning type, and multiple technologies (CRL and ML model) need to be experimented with to find the best model for the dataset. Moreover, introducing more robust program slicing techniques based on the SCA rules/Warning types will contribute to academia.

Threats to validity

In this section, the potential threats to the validity of this study are discussed following previous guidelines [128].

Internal Threats to Validity

According to Croft et al. [129], the quality of the ML model depends on the quality of the dataset on which the ML model is built. In addition, the highest quality dataset can be created using manual annotation [130]. In our previous study, four manually annotated datasets were created and introduced in § 4. The datasets were created by manual annotation according to defined standards. However, there may be incorrect annotations in those datasets due to errors during the annotation process (e.g., due to incomplete or misunderstood standards). To minimise errors, all authors discussed the standards, and several software tools were developed to assist in the annotation process. Additionally, all four datasets were rechecked at least once. Additionally, 10% of Dataset 1 and Dataset 3 were selected randomly, and two authors reviewed these samples together [131, 132, 133]. There were three data points out of 115 (51 in Dataset 1 and 64 in Dataset 3) that recorded disagreements at the beginning of the reliability assessment process. However, complete agreement on the 115 samples was reached after a round of discussions [134], attesting to the strong reliability of our datasets development [135].

The performance of ML models heavily relies on the hyperparameters used to train each ML model. Therefore, we used Optuna optimiser with TPE sampler to identify the best hyperparameters for each ML model. However, there is still a possibility that an alternative configuration could produce a better result. Moreover, there can be a bias in the results generated towards SCA tools. To mitigate that, we used two SCA tools. However, there can still be some bias, which we did not identify while analysing the results. This may affect the generalisability of the results for other SCA tools.

We used four CRLs: Word2Vec, Code2Vec, FastText, and CodeBERT. Code2Vec and CodeBERT are pre-trained with large sets of publicly available source code datasets. In contrast, Word2Vec and FastText are trained on non-source code text data. Therefore, following previous studies, we trained both Word2Vec and FastText with our two source code datasets (Stack Overflow dataset and Apache Tomcat dataset) [3, 5]. While training some CRLs with our source code datasets might bias the results in favour of those CRLs, we used the entire code dataset for training to minimise bias. Nonetheless, there remains a risk of producing biased results.

We used four datasets with sizes ranging from 117 to 298, which are relatively small compared to existing studies. Although we did not identify any overfitting issues in the results, there may still be some effect on the results.

While splitting the datasets into training and testing datasets, we did not consider the SCA rules of warnings. Consequently, for specific SCA rules, all warnings associated with the rule may be present solely in the training dataset, with none in the testing dataset, or vice versa. Therefore, the dataset will be significantly different based on the random seed used when splitting the dataset, which may cause different results to be generated by the ML model. However, we used the same random seed for each dataset to be consistent across all the ML models. Therefore, all the ML models that needed to be trained with the same dataset from our four datasets were trained and tested using the same portions of the dataset.

External Threats to Validity

There were warnings for a limited number of SCA rules in the datasets, which may cause generalisability issues. However, we did not filter any warnings based on the SCA rules, and we selected all the warnings which belonged to SCA rules that generated false-positive warnings. Therefore, the results should be relevant to any other dataset containing warnings generated for Java source code by PMD or SonarQube SCA tools.

We used CRLs and ML Algorithms that were the most effective in existing studies. However, there may be more advanced technologies that have not yet been evaluated and could be more effective than those we assessed in this study. To address this, we considered some recently introduced technologies that have not been extensively studied in existing studies, such as Code2Vec CRL and BiLSTM ML Algorithm.

Conclusion Threats to Validity

The conclusion made from this study is based on the defined methodology and the training datasets used. However, the conclusion might vary based on training datasets with different characteristics and alternative methods. To mitigate this, we used four datasets with various characteristics, and we used recent technologies/methodologies to evaluate ML models. These methodologies have been used in recent studies in a similar domain.

Conclusion and future work

Conclusion: We conducted a comprehensive literature study to identify gaps in existing studies in identifying false-positives from SCA warnings. As a result, we identify several gaps in existing studies. One of the limitations is that existing studies do not comprehensively evaluate multiple CRLs and do not assess the impact of CRL performance on the overall ML model performance. Additionally, existing studies do not evaluate SCA warnings from multiple SCA tools to identify any possible performance patterns in the overall model performance changes based on the SCA tool used. Furthermore, they do not evaluate their ML models in multiple datasets with diverse characteristics. Moreover, existing studies do not consider model-building goals and multiple performance metrics to evaluate ML models under different model building goals, such as, maximising the overall model performance, correctly identifying false-positive warnings as much as possible to minimise false warnings, correctly detecting the maximum possible actual warnings with minimum false warnings, and faster training and evaluating ML models with limited computational resources so that models can be fine-tuned for better performance.

In our study, we fill these gaps through four research questions. RQ1 shows that the performance of the ML algorithm depends on the CRL used to represent the source code, and the best CRL for each ML algorithm is dependent on the performance metric used to evaluate the classification of SCA warnings. Moreover, RQ2 reveals that the performance of SCA warning classification depends on the ML algorithm used, and the best algorithm also varies based on the performance metric used to evaluate the warning classification. Furthermore, RQ3 shows that the performance of warning classification depends on the purpose of ML model building (related to performance metrics) and varies based on the utilised CRL, ML Algorithm, and the performance metric. Moreover, RQ4 revealed that the performance of warning classification is also dependent on the characteristics of the dataset used to train and test the model.

In this study, we evaluated multiple CRLs along with multiple ML algorithms and their individual impact on the overall performance of the model. In addition, we assessed these combinations of CRLs and ML Algorithms using numerous datasets with diverse characteristics in terms of various performance metrics. Additionally, these datasets were created by analysing two source code datasets using two SCA tools to identify any patterns in the used SCA tools and the overall performance of ML models. Overall, the best CRLs, ML Algorithms, and ML models (performance of false-positive warning classification) vary based on the characteristics of the training dataset and the goal of the model building. However, we did not identify any dependency on the SCA tools or the characteristics of the source code (code snippets or full-length source code) on the model performance. This is because of the use of advanced program slicing techniques, which adopt existing methodologies and customise them for each SCA warning rule.

The results generated in this study provide recommendations for SCA tool developers, the practitioner community, and the research community. Tool developers can train and use ML models to identify false positive warning patterns and reduce false-positive warnings. Additionally, introducing a sensitivity parameter can allow tool users to choose between having fewer false-positives or having actual warnings as much as possible. The practitioner community may select ML models based on their intention (refer to the purpose of building ML model in § 2.2), as ML models do not produce perfect results. Additionally, training CRLs, such as Word2Vec and FastText, with source code that was analysed using the SCA tools, and training ML models multiple times, is vital to achieve the optimal ML model. Furthermore, the variant of CRL used to train the model affects the performance of the ML model. Therefore, it is emphasised that the research community should evaluate various CRLs as well as numerous ML algorithms. Additionally, the research community can consider factors such as Computational Complexity or interpretability in addition to other performance metrics such as MCC and F1-Score. Moreover, simple ML algorithms such as RF can at times also provide good performance compared to DL models. Therefore, it is necessary to evaluate simple models as well. Additionally, it is vital to report ML model performance in conjunction with the characteristics of the dataset, as the performance of the ML model is dependent on the characteristics of the dataset. Considering the ML model-building goals are also vital, as optimising ML models with commonly used performance metrics such as Accuracy, the F1-Score is not always applicable.

Future Work: In this study, we used several program slicing techniques. However, the quality of the program slice may affect the overall performance of the ML model. Therefore, we will conduct further research on the program slicing technique to enhance the overall performance of the model. Moreover, we employed machine learning models for this study to classify SCA warnings. However, we will research the possibility of using LLMs to enhance our outcomes, as utilising LLMs for a similar study may reduce the need for large training datasets.

Appendices

Program Slicing Approaches for PMD Rules

PMD RulesProgram Slicing Approach
AvoidArrayLoopsAST_FORWARD
AvoidDeeplyNestedIfStmtsAST_FORWARD_BACKWARD
AvoidThrowingNewInstanceOfSameExceptionDATA_DEPENDENCY
CloneMethodMustBePublicAST_BACKWARD
CloneMethodMustImplementCloneableAST_BACKWARD
CloseResourceDATA_DEPENDENCY
CommentRequired (Document comments)ONE_LINE_UP
CommentRequired (Inline comments)ISSUE_LINE_ONLY
DataClassAST_FORWARD
DetachedTestCaseAST_FORWARD
DoNotUseThreadsISSUE_LINE_ONLY
ForLoopCanBeForeachDATA_DEPENDENCY
GuardLogStatementISSUE_LINE_ONLY
IdenticalCatchBranchesAST_FORWARD_BACKWARD
ImplicitSwitchFallThroughAST_FORWARD
JUnit4TestShouldUseAfterAnnotationAST_FORWARD_BACKWARD
JUnit4TestShouldUseTestAnnotationAST_FORWARD_BACKWARD
JUnitTestsShouldIncludeAssertAST_FORWARD
LawOfDemeterDATA_DEPENDENCY
NonThreadSafeSingletonDATA_DEPENDENCY
ProperLoggerISSUE_LINE_ONLY
SingletonClassReturningNewInstanceAST_FORWARD
SingularFieldAST_FORWARD_BACKWARD
TestClassWithoutTestCasesAST_FORWARD_BACKWARD
UnnecessaryLocalBeforeReturnDATA_DEPENDENCY
UnusedAssignmentDATA_DEPENDENCY
UselessParenthesesISSUE_LINE_ONLY
UseUtilityClassAST_FORWARD

Table 23.

Program Slicing Approaches for SonarQube Rules

SonarQube Rule CodeProgram Slicing Approach
java:S109ONE_LINE_UP
java:S135AST_FORWARD
java:S1067AST_BACKWARD
java:S1107ONE_LINE_DOWN
java:S1120AST_BACKWARD
java:S1182AST_FORWARD_BACKWARD
java:S1258AST_FORWARD
java:S1451TEN_LINE_DOWN
java:S1696ONE_LINE_DOWN
java:S1994ONE_LINE_DOWN
java:S2039ISSUE_LINE_ONLY
java:S2097AST_FORWARD
java:S2440AST_FORWARD
java:S2444AST_BACKWARD
java:S3012DATA_DEPENDENCY
java:S3047DATA_DEPENDENCY
java:S5738AST_BACKWARD

Table 24.

References

  1. [1]V. R. L. de Mendonca, C. L. Rodrigues, F. A. A. de MN Soares, and A. M. R. Vincenzi, “Static analysis techniques and tools: A systematic mapping study,” 2013.
  2. [2]S. Pujar, Y. H. Zheng, L. Buratti, B. Lewis, Y. C. Chen, J. Laredo, A. Morari, E. Epstein, T. N. Lin, B. Yang, and Z. Su, “Analyzing source code vulnerabilities in the D2A dataset with ML ensembles and C-BERT,” EMPIRICAL SOFTWARE ENGINEERING, vol. 29, no. 2, MAR, 2024.DOI
  3. [3]Y. Yang, M. Wen, X. Gao, Y. Zhang, and H. Sun, “Reducing false positives of static bug detectors through code representation learning,” in 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2024, pp. 681-692.DOI
  4. [4]R. Yedida, H. J. Kang, H. Y. Tu, X. Q. Yang, D. V. Lo, and T. Menzies, “How to Find Actionable Static Analysis Warnings: A Case Study With FindBugs,” IEEE Transactions on Software Engineering, vol. 49, no. 4, pp. 2856-2872, 2023.
  5. [5]A. Nagaraj, B. Sinha, M. Sood, Y. Mathur, S. Gupta, and D. Sitaram, “Learning Algorithms in Static Analysis of Web Applications,” arXiv preprint arXiv:2210.07465, 2022.
  6. [6]D. H. Nguyen, A. Seo, N. P. Nnamdi, and Y. Son, “False Alarm Reduction Method for Weakness Static Analysis Using BERT Model,” Applied Sciences (Switzerland), vol. 13, no. 6, 2023.DOI
  7. [7]H. Li, Y. Hao, Y. Zhai, and Z. Qian, "Assisting static analysis with large language models: A chatgpt experiment." pp. 2107-2111.
  8. [8]S. Yerramreddy, A. Mordahl, U. Koc, S. Wei, J. S. Foster, M. Carpuat, and A. A. Porter, “An empirical assessment of machine learning approaches for triaging reports of static analysis tools,” Empirical Software Engineering, vol. 28, no. 2, 2023.DOI
  9. [9]L. Deshapriya, S. A. Licorish, and B. J. Woodford, “Understanding Warnings Generated by PMD and SonarQube, their Rules and Compliance to Established Coding Standards,” Science of Computer Programming, pp. 103470, 2026.arxiv.org/abs/2603.00821
  10. [10]S. Lee, S. Hong, J. Yi, T. Kim, C.-J. Kim, and S. Yoo, "Classifying false positive static checker alarms in continuous integration using convolutional neural networks." pp. 391-401.DOI
  11. [11]U. Koc, S. Wei, J. S. Foster, M. Carpuat, and A. A. Porter, "An empirical assessment of machine learning approaches for triaging reports of a Java static analysis tool." pp. 288-299.DOI
  12. [12]A. Tanwar, H. Manikandan, K. Sundaresan, P. Ganesan, S. Kumar Chandrasekaran, and S. Ravi, “Assessing Validity of Static Analysis Warnings using Ensemble Learning,” Arxiv, Apr 21, 2021.arxiv.org/abs/2104.11593
  13. [13]Y. Tan, and J. Tian, “A Method for Processing Static Analysis Alarms Based on Deep Learning,” Applied Sciences, vol. 14, no. 13, pp. 5542, 2024.
  14. [14]P. Hegedus, and R. Ferenc, “Static Code Analysis Alarms Filtering Reloaded: A New Real-World Dataset and its ML-Based Utilization,” IEEE ACCESS, vol. 10, pp. 55090-55101, 2022.DOI
  15. [15]M. Sokolova, and G. Lapalme, “A systematic analysis of performance measures for classification tasks,” Information processing & management, vol. 45, no. 4, pp. 427-437, 2009.
  16. [16]L. Deshapriya, S. Licorish, and B. Woodford, "Using machine learning approaches to classify false positives generated by static analysis tools - An empirical study using small datasets - Replication Package," Zenodo, 2025.DOI
  17. [17]D. Chicco, and G. Jurman, “The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification,” BioData Mining, vol. 16, no. 1, pp. 4, 2023.DOI
  18. [18]R. Diallo, C. Edalo, and O. O. Awe, "Machine learning evaluation of imbalanced health data: a comparative analysis of balanced accuracy, MCC, and F1 score," Practical Statistical Learning and Data Science Methods: Case Studies from LISA 2020 Global Network, USA, pp. 283-312: Springer, 2024.DOI
  19. [19]X. Guo, Y. Yin, C. Dong, G. Yang, and G. Zhou, “On the class imbalance problem,” in 2008 Fourth international conference on natural computation, 2008, pp. 192-201.
  20. [20]J. M. Johnson, and T. M. Khoshgoftaar, “Survey on deep learning with class imbalance,” Journal of big data, vol. 6, no. 1, pp. 1-54, 2019.
  21. [21]S. A. Cook, “An overview of computational complexity,” ACM Turing award lectures, pp. 1982, 2007.
  22. [22]G. Naidu, T. Zuva, and E. M. Sibanda, "A review of evaluation metrics in machine learning algorithms." pp. 15-25.
  23. [23]T. T. Vu, and H. D. Vo, "Using Multiple Code Representations to Prioritize Static Analysis Warnings."
  24. [24]K. T. Ngo, D. T. Do, T. T. Nguyen, and H. D. Vo, "Ranking Warnings of Static Analysis Tools Using Representation Learning." pp. 327-337.DOI
  25. [25]Y. Wang, and H. Li, “Code completion by modeling flattened abstract syntax trees as graphs,” in Proceedings of the AAAI conference on artificial intelligence, 2021, pp. 14015-14023.arxiv.org/abs/2103.09499
  26. [26]M. Heffernan, K. Wilken, and G. Shobaki, “Data-dependency graph transformations for superblock scheduling,” in 2006 39th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO'06), 2006, pp. 77-88.DOI
  27. [27]X. Yang, J. Chen, R. Yedida, Z. Yu, and T. Menzies, “Learning to recognize actionable static code warnings (is intrinsically easy),” Empirical Software Engineering, vol. 26, pp. 1-24, 2021.
  28. [28]A. Ribeiro, P. Meirelles, N. Lago, and F. Kon, "Ranking warnings from multiple source code static analyzers via ensemble learning."
  29. [29]E. A. Alikhashashneh, R. R. Raje, and J. H. Hill, "Using Machine Learning Techniques to Classify and Predict Static Code Analysis Tool Warnings."
  30. [30]H. J. Kang, K. L. Aw, D. Lo, and I. C. Soc, "Detecting False Alarms from Automatic Static Analysis Tools: How Far are We?," International Conference on Software Engineering. pp. 698-709, 2022.DOI
  31. [31]E. Alikhashashneh, R. Raje, and J. Hill, “Using Software Engineering Metrics to Evaluate the Quality of Static Code Analysis Tools,” in 2018 1ST INTERNATIONAL CONFERENCE ON DATA INTELLIGENCE AND SECURITY (ICDIS 2018), 2018, pp. 65-72.DOI
  32. [32]F. Cheirdari, and G. Karabatis, "Analyzing False Positive Source Code Vulnerabilities Using Static Analysis Tools." pp. 4782-4788, 2019.
  33. [33]M. Berman, S. Adams, T. Sherburne, C. Fleming, and P. Beling, "Active learning to improve static analysis." pp. 1322-1327.
  34. [34]M. Qing, X. Feng, J. Luo, W. Huang, J. Zhang, P. Wang, Y. Fan, X. Ge, and Y. Pan, "A machine learning-based static analysis warning prioritization." pp. 685-690.DOI
  35. [35]J. Wang, S. Wang, and Q. Wang, "Is there a" golden" feature set for static warning identification? an experimental evaluation." pp. 1-10.
  36. [36]E. Levina, and P. Bickel, “Maximum likelihood estimation of intrinsic dimension,” Advances in neural information processing systems, vol. 17, 2004.
  37. [37]L. Flynn, W. Snavely, and Z. Kurtz, "Test Suites as a Source of Training Data for Static Analysis Alert Classifiers." pp. 100-108.arxiv.org/abs/2105.03523
  38. [38]X. Yang, Z. Yu, J. Wang, and T. Menzies, “Understanding static code warnings: An incremental AI approach,” Expert Systems with Applications, vol. 167, pp. 1, 2021 Apr 01 2021-03-20, 2021.arxiv.org/abs/1911.01387
  39. [39]B. Settles, “Active learning literature survey,” 2009.
  40. [40]A. Kharkar, R. Z. Moghaddam, M. Jin, X. Liu, X. Shi, C. Clement, N. Sundaresan, and I. C. Soc, “Learning to Reduce False Positives in Analytic Bug Detectors,” in ACM/IEEE 44th International Conference on Software Engineering (ICSE), Pittsburgh, PA, 2022, pp. 1307-1316.DOI
  41. [41]B. Wang, A. Wang, F. Chen, Y. Wang, and C.-C. J. Kuo, “Evaluating word embedding models: Methods and experimental results,” APSIPA transactions on signal and information processing, vol. 8, pp. e19, 2019.
  42. [42]S. Liu, P.-T. Bremer, J. J. Thiagarajan, V. Srikumar, B. Wang, Y. Livnat, and V. Pascucci, “Visual exploration of semantic relationships in neural word embeddings,” IEEE transactions on visualization and computer graphics, vol. 24, no. 1, pp. 553-562, 2017.DOI
  43. [43]M. A. Rodriguez, and P. Merlo, "Word associations and the distance properties of context-aware word embeddings." pp. 376-385.
  44. [44]T. Chen, and C. Guestrin, "Xgboost: A scalable tree boosting system." pp. 785-794.DOI
  45. [45]G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” Advances in neural information processing systems, vol. 30, 2017.
  46. [46]A. V. Dorogush, V. Ershov, and A. Gulin, “CatBoost: gradient boosting with categorical features support,” arXiv preprint arXiv:1810.11363, 2018.arxiv.org/abs/1810.11363
  47. [47]F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, and V. Dubourg, “Scikit-learn: Machine learning in Python,” the Journal of machine Learning research, vol. 12, pp. 2825-2830, 2011.
  48. [48]H. Aman, S. Amasaki, T. Yokogawa, and M. Kawahara, "A Survival Analysis-Based Prioritization of Code Checker Warning: A Case Study Using PMD," 4th IEEE/ACIS International Conference on Big Data, Cloud Computing and Data Science Engineering, BCD 2019, R. Lee, ed., Springer Verlag, 2020, pp. 69-83.DOI
  49. [49]U. Koc, P. Saadatpanah, J. S. Foster, and A. A. Porter, "Learning a Classifier for False Positive Error Reports Emitted by Static Code Analysis Tools." pp. 35-42, 2017.DOI
  50. [50]L. Sun, W. Lin, S. Shu, and L. Li, "An Interactive Ranking Algorithm for Program Static Analysis." pp. 242-249.
  51. [51]S. S. Heckman, “Adaptively ranking alerts generated from automated static analysis,” XRDS: Crossroads, The ACM Magazine for Students, vol. 14, no. 1, pp. 1-11, 2007.DOI
  52. [52]S. E. Smith, L. Williams, and J. Xu, "Expediting Programmer AWAREness of Anomalous Code." pp. 4.49-4.50.
  53. [53]M. Siavvas, I. Kalouptsoglou, D. Tsoukalas, and D. Kehagias, "A self-adaptive approach for assessing the criticality of security-related static analysis alerts." pp. 289-305.
  54. [54]Y. Zhai, Y. Hao, H. Zhang, D. Wang, C. Song, Z. Qian, M. Lesani, S. V. Krishnamurthy, and P. Yu, "UBITect: a precise and scalable method to detect use-before-initialization bugs in Linux kernel." pp. 221-232.
  55. [55]R. Just, D. Jalali, and M. D. Ernst, "Defects4J: A database of existing faults to enable controlled testing studies for Java programs." pp. 437-440.
  56. [56]K. Liu, D. Kim, T. F. Bissyandé, S. Yoo, and Y. Le Traon, “Mining fix patterns for findbugs violations,” IEEE Transactions on Software Engineering, vol. 47, no. 1, pp. 165-188, 2018.
  57. [57]S. Meldrum, S. A. Licorish, C. A. Owen, and B. T. R. Savarimuthu, “Understanding stack overflow code quality: A recommendation of caution,” Science of Computer Programming, vol. 199, pp. 102516, 2020.DOI
  58. [58]X. Cheng, H. Wang, J. Hua, G. Xu, and Y. Sui, “Deepwukong: Statically detecting software vulnerabilities using deep graph neural network,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 30, no. 3, pp. 1-33, 2021.DOI
  59. [59]S. A. Licorish, and M. Wagner, “Combining GIN and PMD for code improvements,” in GECCO 2022 Companion - Proceedings of the 2022 Genetic and Evolutionary Computation Conference, 2022, pp. 790-793.
  60. [60]S. A. Licorish, and T. Nishatharan, “Contextual profiling of stack overflow java code security vulnerabilities initial insights from a pilot study,” in 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C), 2021, pp. 1060-1068.DOI
  61. [61]S. A. Licorish, and M. Wagner, “On the utility of varying GIN and PMD for improving Stack Overflow code snippets,” arXiv preprint arXiv:2202.01490, 2022.arxiv.org/abs/2202.01490
  62. [62]S. A. Licorish, and M. Wagner, “Dissecting Copy/Delete/Replace/Swap mutations: Insights from a GIN Case Study,” in GECCO 2022 Companion - Proceedings of the 2022 Genetic and Evolutionary Computation Conference, 2022, pp. 1940-1945.DOI
  63. [63]V. Piantadosi, S. Scalabrino, and R. Oliveto, “Fixing of security vulnerabilities in open source projects: A case study of apache http server and apache tomcat,” in 2019 12th IEEE Conference on software testing, validation and verification (ICST), 2019, pp. 68-78.DOI
  64. [64]Y. Wei, X. Sun, L. Bo, S. Cao, X. Xia, and B. Li, “A comprehensive study on security bug characteristics,” Journal of Software: Evolution and Process, vol. 33, no. 10, pp. e2376, 2021.
  65. [65]M. Alqaradaghi, and T. Kozsik, “Comprehensive Evaluation of Static Analysis Tools for Their Performance in Finding Vulnerabilities in Java Code,” Ieee Access, vol. 12, pp. 55824-55842, 2024, 2024.
  66. [66]H. Liu, S. Chen, R. Feng, C. Liu, K. Li, Z. Xu, L. Nie, Y. Liu, and Y. Chen, “A Comprehensive Study on Quality Assurance Tools for Java,” in 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), Seattle, WA, 2023, pp. 285-297.DOI
  67. [67]J. Yeboah, and S. Popoola, “Efficacy of Static Analysis Tools for Software Defect Detection on Open-Source Projects,” in Proceedings - 2023 International Conference on Computational Science and Computational Intelligence, CSCI 2023, 2023, pp. 1588-1593.
  68. [68]E. Zolduoarrati, S. A. Licorish, and N. Stanger, “Impact of individualism and collectivism cultural profiles on the behaviour of software developers: A study of stack overflow,” Journal of Systems and Software, vol. 192, pp. 111427, 2022.
  69. [69]F. Faul, E. Erdfelder, A.-G. Lang, and A. Buchner, “G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences,” Behavior research methods, vol. 39, no. 2, pp. 175-191, 2007.
  70. [71]R. Latpate, J. Kshirsagar, V. K. Gupta, and G. Chandra, Advanced sampling methods: Springer, 2021.
  71. [72]P. Becker, L. A. Olsina Santos, and M. F. Papa, "Strategy for improving source code compliance to a style guide."
  72. [73]P. Becker, L. Olsina, and M. F. Papa, “Approach to Improving Java Source Code Considering Non-compliance with a Java Style Guide,” in Argentine Congress of Computer Science, 2022, pp. 123-139.DOI
  73. [74]A. Honkaranta, T. Leppänen, and A. Costin, “Towards practical cybersecurity mapping of stride and cwe—a multi-perspective approach,” in 2021 29th Conference of Open Innovations Association (FRUCT), 2021, pp. 150-159.DOI
  74. [75]S. Simonetto, and P. Bosch, “Comprehensive threat analysis and systematic mapping of CVEs to MITRE framework,” in 1st International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security, NLPAICS 2024, 2024.
  75. [76]V. Arnaoudova, M. Di Penta, and G. Antoniol, “Linguistic antipatterns: What they are and how developers perceive them,” Empirical Software Engineering, vol. 21, pp. 104-158, 2016.
  76. [77]L. Murphy, T. Alliyu, A. Macvean, M. B. Kery, and B. A. Myers, “Preliminary analysis of REST API style guidelines,” Ann Arbor, vol. 1001, no. 48109, pp. 4, 2017.
  77. [78]K. A. Sedek, N. Osman, M. N. Osman, and J. H. Kamaruzaman, “Developing a Secure Web Application Using OWASP Guidelines,” Comput. Inf. Sci., vol. 2, no. 4, pp. 137-143, 2009.DOI
  78. [79]M. Idris, I. Syarif, and I. Winarno, “Development of vulnerable web application based on OWASP API security risks,” in 2021 International Electronics Symposium (IES), 2021, pp. 190-194.DOI
  79. [80]S. Chand, S. K. Pandey, J. Horkoff, M. Staron, M. Ochodek, and D. Durisic, "Comparing Word-Based and AST-Based Models for Design Pattern Recognition." pp. 44- 48.
  80. [81]U. Alon, M. Zilberstein, O. Levy, and E. Yahav, “code2vec: Learning distributed representations of code,” Proceedings of the ACM on Programming Languages, vol. 3, no. POPL, pp. 1-29, 2019.
  81. [82]Z. Ding, H. Li, W. Shang, and T.-H. P. Chen, “Can pre-trained code embeddings improve model performance? Revisiting the use of code embeddings in software engineering tasks,” Empirical Software Engineering, vol. 27, no. 3, pp. 63, 2022.
  82. [83]Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” arXiv preprint arXiv:1801.01681, 2018.DOI
  83. [84]M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and D. Poshyvanyk, “Deep learning similarities from different representations of source code,” in Proceedings of the 15th international conference on mining software repositories, 2018, pp. 542-553.
  84. [85]S. Salimi, and M. Kharrazi, “VulSlicer: Vulnerability detection through code slicing,” Journal of Systems and Software, vol. 193, pp. 111450, 2022.DOI
  85. [86]K. I. f. T. . "JOANA (Java Object-sensitive ANalysis) - Information Flow Control Framework for Java," 02/12/2024; https://pp.ipd.kit.edu/projects/joana/.
  86. [87]T. Mikolov, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781, vol. 3781, 2013.arxiv.org/abs/1301.3781
  87. [88]J. Pennington, R. Socher, and C. D. Manning, "Glove: Global vectors for word representation." pp. 1532-1543.
  88. [89]O. Levy, and Y. Goldberg, “Neural word embedding as implicit matrix factorization,” Advances in neural information processing systems, vol. 27, 2014.
  89. [90]S. Kang, J. Yoon, N. Askarbekkyzy, and S. Yoo, “Evaluating diverse large language models for automatic and general bug reproduction,” IEEE Transactions on Software Engineering, 2024.
  90. [91]F. Chollet, Deep learning with Python: simon and schuster, 2021.
  91. [92]E. G. Jaroy, G. T. Risa, I. N. Farstad, R. Emblem, and R. Ougland, “A recall-optimised machine learning framework for small data improves risk stratification for Hirschsprung's disease,” Informatics in Medicine Unlocked, pp. 101530, 2024.
  92. [93]T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, "Optuna: A next-generation hyperparameter optimization framework." pp. 2623-2631.
  93. [94]N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: synthetic minority over-sampling technique,” Journal of artificial intelligence research, vol. 16, pp. 321-357, 2002.
  94. [95]L. Prechelt, "Early stopping-but when?," Neural Networks: Tricks of the trade, pp. 55-69: Springer, 2002.
  95. [96]X. Ying, "An overview of overfitting and its solutions." p. 022022.
  96. [97]L. Rice, E. Wong, and Z. Kolter, "Overfitting in adversarially robust deep learning." pp. 8093-8104.
  97. [98]B. Sabiri, B. El Asri, and M. Rhanoui, "Mechanism of Overfitting Avoidance Techniques for Training Deep Neural Networks." pp. 418-427.
  98. [99]M. K. Anam, S. Defit, H. Haviluddin, L. Efrizoni, and M. B. Firdaus, “Early Stopping on CNN-LSTM Development to Improve Classification Performance,” Journal of Applied Data Sciences, vol. 5, no. 3, pp. 1175-1188, 2024.
  99. [100]S. Hanifi, A. Cammarono, and H. Zare-Behtash, “Advanced hyperparameter optimization of deep learning models for wind power prediction,” Renewable Energy, vol. 221, pp. 119700, 2024.
  100. [101]M. Sarma, and S. Chatterjee, “Multistage diagnosis of Alzheimer’s disease from clinical data using ‘Deep Ensemble Learning’,” Journal of Artificial Intelligence and Autonomous Intelligence, vol. 1, no. 2, pp. 9, 2024.
  101. [102]S. N. A. Mohd Johari, S. Khairunniza-Bejo, A. R. Mohamed Shariff, N. A. Husin, M. M. Mohd Masri, and N. Kamarudin, “Effect of datasets size on the machine learning performance of the bagworm, Metisa plana (Walker) infestation using UAV remote sensing,” Journal of Plant Diseases and Protection, vol. 132, no. 1, pp. 1-17, 2025.
  102. [103]R. Dey, and F. M. Salem, "Gate-variants of gated recurrent unit (GRU) neural networks." pp. 1597-1600.
  103. [104]H. M. Lynn, S. B. Pan, and P. Kim, “A deep bidirectional GRU network model for biometric electrocardiogram classification based on recurrent neural networks,” IEEE Access, vol. 7, pp. 145395-145405, 2019.
  104. [105]R. L. Abduljabbar, H. Dia, and P.-W. Tsai, “Unidirectional and bidirectional LSTM models for short-term traffic prediction,” Journal of Advanced Transportation, vol. 2021, no. 1, pp. 5589075, 2021.
  105. [106]M. Gupta, S. K. Verma, and P. Jain, "Detailed study of deep learning models for natural language processing." pp. 249-253.
  106. [107]Y. Zhang, and B. Wallace, “A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification,” arXiv preprint arXiv:1510.03820, 2015.arxiv.org/abs/1510.03820
  107. [108]G. Zhang, T. Zeng, Z. Dai, and X. Dai, “Prediction of CRISPR/Cas9 single guide RNA cleavage efficiency and specificity by attention-based convolutional neural networks,” Computational and structural biotechnology journal, vol. 19, pp. 1445-1457, 2021.
  108. [109]W. Zaremba, “Recurrent neural network regularization,” arXiv preprint arXiv:1409.2329, 2014.arxiv.org/abs/1409.2329
  109. [110]P. Zhou, X. Xie, Z. Lin, and S. Yan, “Towards understanding convergence and generalization of AdamW,” IEEE transactions on pattern analysis and machine intelligence, 2024.
  110. [111]O. El Azzouzzy, T. Chanyour, and S. Jai Anda l ouss i, “Transformer-Based Models for Sentiment Analysis of Youtube Video Comments,” Available at SSRN 5071111.
  111. [112]I. Loshchilov, and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017.arxiv.org/abs/1711.05101
  112. [113]E. Ostertagova, O. Ostertag, and J. Kováč, “Methodology and application of the Kruskal-Wallis test,” Applied mechanics and materials, vol. 611, pp. 115-120, 2014.
  113. [114]T. Sueyoshi, and S. Aoki, “A use of a nonparametric statistic for DEA frontier shift: the Kruskal and Wallis rank test,” Omega, vol. 29, no. 1, pp. 1-18, 2001.
  114. [115]Y. Chan, and R. P. Walmsley, “Learning and understanding the Kruskal-Wallis one-way analysis-of-variance-by-ranks test for differences among three or more independent groups,” Physical therapy, vol. 77, no. 12, pp. 1755-1761, 1997.
  115. [116]J. Tanha, Y. Abdi, N. Samadi, N. Razzaghi, and M. Asadpour, “Boosting methods for multi-class imbalanced data classification: an experimental review,” Journal of Big data, vol. 7, pp. 1-47, 2020.
  116. [117]H. He, and Y. Ma, “Imbalanced learning: foundations, algorithms, and applications,” 2013.
  117. [118]Y. Goldberg, and O. Levy, “word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method,” arXiv preprint arXiv:1402.3722, 2014.arxiv.org/abs/1402.3722
  118. [119]R. C. Staudemeyer, and E. R. Morris, “Understanding LSTM--a tutorial into long short-term memory recurrent neural networks,” arXiv preprint arXiv:1909.09586, 2019.arxiv.org/abs/1909.09586
  119. [120]A. Sherstinsky, “Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network,” Physica D: Nonlinear Phenomena, vol. 404, pp. 132306, 2020.
  120. [121]P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” Transactions of the association for computational linguistics, vol. 5, pp. 135-146, 2017.
  121. [122]A. Parmar, R. Katariya, and V. Patel, "A review on random forest: An ensemble classifier," pp. 758-763.
  122. [123]Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, and D. Jiang, “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155, 2020.arxiv.org/abs/2002.08155
  123. [124]L. Gong, and R. Ji, “What does a TextCNN learn?,” arXiv preprint arXiv:1801.06287, 2018.arxiv.org/abs/1801.06287
  124. [125]B. Ghojogh, and M. Crowley, “The theory behind overfitting, cross validation, regularization, bagging, and boosting: tutorial,” arXiv preprint arXiv:1905.12787, 2019.arxiv.org/abs/1905.12787
  125. [126]M. Schuster, and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE transactions on Signal Processing, vol. 45, no. 11, pp. 2673-2681, 1997.
  126. [127]K. W. Church, “Word2Vec,” Natural Language Engineering, vol. 23, no. 1, pp. 155-162, 2017.
  127. [128]X. Zhou, Y. Jin, H. Zhang, S. Li, and X. Huang, "A map of threats to validity of systematic literature reviews in software engineering," pp. 153-160.
  128. [129]R. Croft, M. A. Babar, and M. M. Kholoosi, “Data quality for software vulnerability datasets,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 121-133.
  129. [130]Y. Lin, Y. Li, M. Gu, H. Sun, Q. Yue, J. Hu, C. Cao, and Y. Zhang, “Vulnerability dataset construction methods applied to vulnerability detection: A survey,” in 2022 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W), 2022, pp. 141-146.
  130. [131]K. A. Neuendorf, The content analysis guidebook: sage, 2017.
  131. [132]A. Jordan, D. Kunkel, J. Manganello, and M. Fishbein, Media messages and public health: New York: Routledge, 2009.
  132. [133]E. Zolduoarrati, S. A. Licorish, and N. Stanger, “Harmonising contributions: exploring diversity in software engineering through CQA mining on stack overflow,” ACM transactions on software engineering and methodology, vol. 33, no. 7, pp. 1-54, 2024.
  133. [134]J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement, vol. 20, no. 1, pp. 37-46, 1960.
  134. [135]L. Marks, Y. Zou, and A. E. Hassan, “Studying the fix-time for bugs in large open source projects,” in Proceedings of the 7th International Conference on Predictive Models in Software Engineering, 2011, pp. 1-8.

Paper details

vlnc:W_smsxjc4d
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents