betaFeedback

WorksComputer Science

SBVUL: Statement-Based Vulnerability Detection

Abstract

Many recent approaches for vulnerability detection use large language models (LLMs) to classify functions as vulnerable or benign, leaving developers to locate the exploitable statements. Furthermore, while these approaches have shown performance improvements, they still fall short in distinguishing vulnerable from patched real-world functions. This paper presents SBVul, an agentic approach that detects exploitable statements in C/C++ functions. In Phase~1, an LLM agent explores the function and its repository and identifies candidate statements. In Phase~2, a second agent analyzes the control and data flow of each candidate and confirms it, rules it out, or marks it uncertain. We created a manually verified ground truth of exploitable statements for 200 pairs of the PrimeVul Paired test set and ran SBVul with the open-weight models Qwen 3.8 and DeepSeek V4.1 Flash and Anthropic's Sonnet 5.5 and Opus 5.5. Phase~1 identifies ground-truth statements in 56–77% of the vulnerable functions, and Phase~2 confirms such a statement with an explanation accepted by an LLM judge in 26–73%. However, all models also confirm candidates in 23–61% of the patched functions. Across vulnerable and patched functions, the highest statement-level precision is 0.315 at a recall of 0.605 obtained with Opus 5.5 (xhigh). At the function level, SBVul with Opus 5.5 classifies 42.0% of the pairs correctly, compared with 33.8% reported for the best previous approach.

Introduction

To detect software vulnerabilities, static analysis tools encode vulnerability patterns as rules or as taint flows from sources to sinks [23, 41], and code property graphs make such patterns queryable [40]. For C/C++, however, a recent comparison found that static analyzers produced tens of thousands of warnings on programs containing about one hundred known bugs [13]. Deep learning-based detectors instead learn vulnerability patterns from labeled functions or statements [19, 12, 30], but their performance drops sharply on unrealistic, deduplicated data, partly because they learn artifacts associated with noisy labels rather than the vulnerabilities themselves [5, 8, 4, 28].

Most recent approaches build on large language models (LLMs) [31]. They fine-tune LLMs to reason about vulnerabilities [35, 36], retrieve security knowledge from earlier vulnerabilities [10, 43], provide structural context from abstract syntax trees, control-flow and data-flow graphs, or code property graphs [32, 18], or let multiple agents debate or explore the code with tools [38, 34]. On the PrimeVul Paired test set [9], which pairs each vulnerable C/C++ function with its patched version, GPT-3.5 and GPT-4 perform close to random guessing, and recent approaches still classify only a minority of the pairs correctly. For example, VulTriager [38] classifies 81 of the 435 pairs (18.6%) correctly, VulTriage [32] 147 pairs (33.8%), and VulInstruct [43] 17.2% of the pairs when its explanation must also match the actual vulnerability.

Two limitations reduce the usefulness of these approaches for developers. First, almost all of them classify a function as vulnerable or benign. Even when they explain their decision, as required by VulInstruct [43] and CORRECT [21], the developer still has to locate the statements that make the function exploitable. Second, the few approaches that report vulnerable statements, such as LineVul [12] and the agentic pipeline of SecVulEval [1], take the lines changed by the fixing commit as their ground truth. A fix, however, often adds a check before the exploitable statement and leaves the statement itself unchanged, so this ground truth rewards identifying the changed lines rather than the exploitable statements.

We address these limitations with SBVUL, an agentic approach that detects exploitable statements in C/C++ functions and explains why they are exploitable. These statements perform the operations an attacker can exploit, such as an unchecked memcpy. Following GPTLens [15] and VulAgent [34], SBVUL splits the detection into two phases. In Phase 1, an LLM agent explores the function and its repository with file and text search tools and proposes candidates, each with its line number(s) and an explanation. In Phase 2, a second agent investigates the control and data flow of every candidate and confirms it, rules it out, or marks it uncertain. Based on these results, SBVUL classifies a function as vulnerable if Phase 2 confirms at least one candidate. We implemented two versions of SBVUL, one with LangChain1 for open-weight models and one with the Claude Code Agent SDK2 for Anthropic’s frontier models.

For the evaluation of SBVUL, we created a ground truth for 200 randomly selected pairs from the PrimeVul Paired test set [9]. We prompted an LLM to identify exploitable state-

This research was funded in whole or in part by the Austrian Science Fund (FWF) 10.55776/P36698. For open access purposes, the author has applied a CC BY public copyright license to any author accepted manuscript version arising from this submission. The research was supported by the Austrian ministries BMIMI, BMWET and the State of Upper Austria in the frame of the SCC COMET competence center INTEGRATE (FFG 892418). ments using the CVE description, the commit message, and the diff, and manually verified the results for every vulnerable function. A confirmed verdict counts as a true positive only if its line number matches a ground-truth statement and an LLM judge accepts its explanation. Every confirmed verdict on a patched function counts as a false positive. We ran SBVUL with Qwen 3.8, DeepSeek V4.1 Flash, Sonnet 5.5, and Opus 5.5 to answer three research questions:

  1. RQ1: What is the performance of SBVUL in finding candidates for exploitable statements in C/C++ functions?

  2. RQ2: What is the performance of SBVUL in confirming the exploitable statements of C/C++ functions?

  3. RQ3: What is the performance of SBVUL in distinguishing vulnerable from patched C/C++ functions?

Phase 1 identifies candidates that match at least one ground-truth statement in 56–77% of the vulnerable functions (RQ1). Phase 2 confirms at least one ground-truth statement with an accepted explanation in 26–73% of the vulnerable functions, with Opus 5.5 performing best, followed by DeepSeek V4.1, Sonnet 5.5, and Qwen 3.8. At the same time, all models also confirm candidates in 23–61% of the patched functions, hence the verdicts alone separate vulnerable from patched code only partly (RQ2). With Opus 5.5, the classification based on confirmed Phase 2 verdicts exceeds the reported results of previous approaches: SBVUL correctly classifies 42% of the pairs, compared with 33.8% for VulTriage and 18.6% for VulTrial (RQ3). Under the lenient CORRECT criterion, which also requires an accepted explanation, SBVUL with Opus 5.5 classifies 75.5% of the pairs correctly, compared with 17.2% for VulInstruct (RQ3).

In summary, this paper makes the following contributions:

  • SBVUL, an agentic approach that detects exploitable statements in C/C++ functions in two phases and derives the function-level classification from the confirmed statements.

  • Two implementations to run SBVUL with open-weight models and with Anthropic’s frontier models.

  • A ground truth of manually verified exploitable statements for 200 pairs of the PrimeVul Paired test set.

  • An evaluation of SBVUL with four LLMs and different reasoning efforts at statement and function level.

  • A replication package with all tools, the ground truth, the instructions, and all results.3

Related Work

We review related work on LLM-based vulnerability detectors, followed by approaches to detect vulnerabilities at statement or line level.

LLM-based Vulnerability Detection

Ding et al. introduced PrimeVul [9], a vulnerability dataset that merges and relabels existing datasets such as Big-Vul [11] and DiverseVul [7], with more accurate labels, de-duplication, a chronological split, and a paired test set of vulnerable and patched functions. They evaluated code language models and LLMs on it, among them GPT-3.5 and GPT-4, and showed that these models perform close to random guessing on the paired test set. Other studies [31, 10] confirm that LLMs hardly distinguish vulnerable from patched code.

Most recent approaches therefore add knowledge or context about the function under analysis. The evaluation framework CORRECT [21] supplies context on callees, types, and globals, enabling LLMs to reach an accuracy of up to 0.67. An LLM judge checks whether their explanations match the ground truth. VulInstruct [43] retrieves security specifications extracted from earlier patches and reaches an F1-score of 0.45 and 17.2% correct pairs on the PrimeVul Paired test set under the CORRECT criterion. VulTriage [32] verbalizes AST, CFG, and DFG information and generic CWE patterns and classifies 147 of 435 pairs (33.8%) correctly with GPT-4o, without checking the explanation. LLMxCPG [18] fine-tunes one model to generate Joern4 queries that slice the code property graph and another to classify the slices, reaching an F1-score of 0.62 for memory-safety CWEs on 40 PrimeVul pairs. Other approaches fine-tune LLMs to reason about vulnerabilities. ReVD [35] reaches a VP-Score (correct minus reversed pairs) of 16.1% on PrimeVul, and VULPO [20] correctly classifies 22.6% of pairs from PrimeVul and SecVulEval.

Agentic detectors divide the task among several agents. VulTrial [38] lets four GPT-4o agents debate a code snippet and classifies 81 of 435 (18.6%) pairs correctly. Like SBVUL, GPTLens [15] separates candidate identification by an auditor from assessment by a critic. The critic ranks candidates, and the approach correctly classifies 44 of 435 pairs (10.1%) with GPT-4o [38]. Phoenix [33] reports 64.4% correct pairs with 7–14B models, but synthesizes its behavioral contract from both functions of the same test pair, their CVE description, and the commit message, hence the ground truth enters the specification. VulnAgent-R2 [26] reaches an F1-score of 0.39 on the imbalanced PrimeVul test set (3% vulnerable functions) against 0.70 on DiverseVul. Further agentic detectors validate hypotheses against project context (VulAgent [34]), build causal knowledge graphs (CLEAR [42]), or orchestrate static analysis (Argus [22]). Unlike SBVUL, most of these approaches classify functions without locating the exploitable statements, and they rely on fine-tuning, a retrieval corpus, or context extracted in advance.

Statement-Level Detection

Static application security testing (SAST) tools, such as Flawfinder [37], Cppcheck [25], Semgrep [29], Infer [3], the Clang Static Analyzer [24], and CodeQL [2], report warnings on statements based on rules or semantic analyses. On 112 memory-safety bugs in C/C++, the best of five analyzers, CodeQL, detected 31 bugs, but only 0.24% of its reports corresponded to a known bug, and most analyzers had to be integrated into the build [13]. Learning-based detectors use trained models to identify statements. IVDetect [19] identifies statements that explain a function-level prediction, while LineVul [12] ranks lines within functions predicted as vulnerable. LineVD [14] classifies statements with a graph neural network, and SLVD-CC [6] classifies nodes in value-flow slices of compilable code. Their ground truth consists of changed lines or synthetic testcase annotations. IVDetect and LineVul evaluate localization only on vulnerable functions, while SLVD-CC was evaluated only on a synthetic dataset. SecVulEval [1] comes with a five-agent pipeline whose context agent extracts the definitions of the symbols a function uses. A vulnerable function counts as detected if the pipeline names, with a correct explanation, at least one statement that the fixing commit deletes or adds or a sink statement. The best configuration reaches an F1-score of 0.24 on 1,000 vulnerable and patched functions. LLM-based localization has also been studied from crash evidence (T2L [39]) and for coding agents at repository level (VulnGym [16] and the Vulnerability Localization Benchmark [27]).

SBVUL differs from these detectors in four respects. First, it needs no training and no compilation, and its candidates are its primary output rather than an explanation of a function-level decision. Second, our ground truth consists of manually verified exploitable statements instead of the changed lines. Third, the agents of SBVUL retrieve the context with tools, and Phase 2 assigns a verdict to every candidate. Fourth, we count true and false positives per statement over the vulnerable and the patched functions, and a true positive also needs an explanation accepted by an LLM judge.

SBVul Approach

Figure 1 shows an overview of SBVUL, which consists of two analysis phases followed by an optional classification step. In Phase 1, SBVUL analyzes the source code of a given function, identifies potentially exploitable statements, and outputs them each with the line number and a short explanation. Phase 2 verifies each candidate from Phase 1 by a detailed analysis of its control and data flow and outputs a verdict for it (confirmed, ruled out, or uncertain) together with a short explanation. Based on the verdicts, SBVUL derives the classification of the function as vulnerable or benign. The split into two phases is motivated by previous work, such as GPTLens [15] and VulAgent [34], that found such a split to reduce the number of false positives while improving precision of the detection. Phases 1 and 2 run as two separate agents that receive the same system prompt but share no conversation history. Both agents investigate the code with filesystem and text search tools that list directories, find files by name, search file contents, and read exact line ranges. In the following, we describe each phase in more detail.

Overview of SBVUL

Figure 1. Overview of SBVUL. Phase 1 identifies exploitable statements. Phase 2 gathers evidence with tools to confirm them, rule them out, or mark them uncertain. Based on the Phase 2 verdicts, Classification labels a function vulnerable or benign.

Phase 1 - Candidate Identification

Listing 1 shows an excerpt of the system prompt’s general and Phase 1 instructions. The agent receives the target function’s name, source code, and file path. It starts with the function and extends its analysis to callers and callees. For each caller, it analyzes whether the arguments can be derived from untrusted input, such as network data or file contents, and are thus controllable by an attacker. For each callee, it traces where the arguments come from and where the return value is used, searching for exploitable statements, such as memory copies and command execution, that these values may reach without a check. Besides the memory-safety patterns, such as unchecked copies, integer arithmetic on sizes and indices, use-after-free, uninitialized reads, and format strings, the instructions list logic-level patterns. These are missing validation, assertions that attacker-controlled input can trigger, type confusion, missing authorization or completeness checks, and stale references. The agent is instructed to report potentially exploitable statements outside these patterns. It stops when it finds no further candidates or reaches the tool-call limit.

You are a highly specialized C/C++ vulnerability detection
expert [...] . Your task is to determine whether a given C/C++
function [...] contains an exploitable vulnerability.

## Phase 1 - Explore & Hunt for Candidate Statements
[...] Start by reading and reasoning about the target
function’s source above. Analyze:
- Structure & call graph: find its callers and see what it
actually does with the arguments [...]
- Taint sources & sinks: search for how the target function’s
parameters/return value are produced upstream [...] and
search for dangerous sink calls [...]

Combine tool findings with direct code reading [...]. In
particular pay attention to:
- Memory operations: malloc, realloc, free, memcpy, [...]
- Integer arithmetic used as a size or index [...]
- Stale references across a side-effecting call: [...]

The line you report for each candidate must be the statement
whose execution actually performs the unsafe operation
[...] Output a JSON array – one object per candidate:
[{"line": <file line number as integer>, "code": <brief code
snippet>, "reason": <...>}]

Listing 1. Excerpt of the system’s prompt general and Phase 1 instructions for identifying exploitable statements.

Phase 2 - Candidate Verification

Listing 2 shows an excerpt of the Phase 2 instructions. The agent receives the candidate list from Phase 1 and assigns each candidate the verdict confirmed, ruled out, or uncertain. While Phase 1 explores the code broadly, Phase 2 seeks evidence to determine whether a candidate is exploitable. The agent examines the surrounding control and data flow, guards, and argument validation in callers. It reads type and macro

## Phase 2 - Confirm Each Candidate Statement
The orchestrator will provide the candidate list from Phase
1 and ask you to confirm or rule out each one using the
available tools. For each candidate, investigate by:
- Reading the exact lines around the candidate again in
context (`read_lines`) [...]
- Searching for other call sites of the function [...]
- [...]
A verdict must rest on evidence you actually gathered, not
an assumption [...] If you haven’t verified that assumption,
the candidate is `uncertain` [...]
For each candidate record whether a vulnerability is
confirmed, ruled out, or uncertain.
Stop investigating a candidate as soon as the evidence is
conclusive. Output a JSON array [...]

Listing 2. Excerpt of the system’s prompt Phase 2 instructions for verifying the candidates from Phase 1. definitions to establish relevant buffer sizes and layouts, and traces allocations and bounds checks across adjacent code.

Every verdict must rest on evidence gathered with tools. If that evidence neither confirms nor rules out a candidate, the agent marks it uncertain. For a memcpy, for example, the agent must trace the origin of the copy size and explain how an attacker could influence it. For a logic-level candidate, it must cite a specific control or data flow path. An empty tool result alone provides no evidence in either direction.

The optional classification step labels a function as vulnerable if Phase 2 confirms at least one candidate, and as benign otherwise. Uncertain verdicts do not count as positive predictions because Phase 2 could not establish exploitability. This step adds no analysis of its own, in contrast to detectors that predict the class directly, such as VulTrial [38] and VulTriage [32], or that let an LLM judge the code against a specification, such as Phoenix [33].

Data Preparation

In this section, we first describe the selection of the test dataset and then the curation of the ground truth on statement level.

Test Dataset

We used the PrimeVul Paired test set [9], which contains 435 pairs of a vulnerable function and its patched version. We selected this dataset because it is de-duplicated and split chronologically, and its labels reach an accuracy comparable to human-verified benchmarks. Furthermore, previous studies, such as VulTrial [38] and VulTriage [32], evaluated their detectors on it, allowing comparison with published results.

For each pair, our checkout tool fetched a shallow copy of the repository containing only the parent and fixing commits. The checkout succeeded for 386 pairs and failed for 49 pairs, mainly because the repository had moved from GitHub to another git server. We manually fixed the URLs of 28 pairs and skipped the remaining 21 pairs. From the resulting 414 pairs, we randomly selected 200 to keep the runtime and cost of evaluating several configurations and LLMs manageable. To prevent a few large projects from dominating the sample, we selected at most 10 pairs per project. Without this limit, TensorFlow (72 pairs), the Linux kernel (68 pairs), and Vim (38 pairs) would make up 43% of the 414 pairs. In our sample, the largest project contributes 9 pairs.

Table 1 shows descriptive statistics of the sampled 200 pairs. Our dataset covers 104 C/C++ projects, 45 types of common weaknesses (CWEs), most of them contained in .c files, with a median number of lines of code (LOC) of 96 and maximum 2,411. About two thirds of the vulnerabilities (CVEs) were reported in 2021 and 2022 and the oldest date back to 2008.

Most frequent CWEsCWE-787 (38), CWE-125 (27), CWE-703 (16), CWE-476 (14)
#CVEs from 2021 or 2022134
#c/.cpp/.cc/.h files162 / 19 / 18 / 1
LOC per vulnerable function (median/mean/max)96 / 211 / 2,411

Table 1. Descriptive statistics of the 200 sampled pairs from 104 projects.

Ground Truth

For statement-level evaluation, we annotated the vulnerable functions in the 200 pairs with their exploitable statements. Our helper tool prompts an LLM to locate statements that perform the exploited operation, such as an out-of-bounds memcpy or array access, an unsafe free, an unchecked size computation, or a branch taken without authorization. Unlike SBVUL, the tool receives the CWE type, the CVE number and description, the commit message, the names of the vulnerable function and its source file, the diff, and the source code annotated with line numbers. For each function, the tool outputs the line numbers of the exploitable statements and a rationale explaining why they are exploitable. A statement spanning several lines is listed with all of its lines. We ran this tool with Opus 5.5 and high reasoning effort.

Two authors then individually verified the output for the first 20 functions against the source code, the diff, and the commit message. In a meeting, they discussed every function for which one of them questioned the output and decided on each statement. Next, each author individually verified the remaining 180 functions, and in a second meeting they discussed all differences until they reached consensus. In total, the authors revised the ground truth of six functions, mainly by replacing a guard with the statement that performs the exploited operation. They removed four pairs whose vulnerable function contains no exploitable statement, because the vulnerability is a missing statement whose effect becomes visible only in other

Statements1234568–910–14
Functions1184614114223

caption: NUMBER OF EXPLOITABLE STATEMENTS PER VULNERABLE FUNCTION.

:::

functions, and replaced them with randomly drawn pairs for which the previously described process was repeated.

tbl:2 shows that the ground truth comprises 380 statements on 460 lines. Only 37% of the 380 ground-truth statements are modified by the patch. In 113 of the 200 vulnerable functions (56%), the patch leaves all ground-truth statements unchanged, and 58 patches only insert code, typically a check before an unchanged exploitable statement. Using lines added or deleted by the patch as ground truth would therefore reward identifying the fix rather than the exploitable statement.

Experimental Setup

This section presents the experimental setup used to answer the research questions. First, we describe the two implementations of SBVUL and their configurations used in our runs followed by the metrics used to evaluate it.

Agents Implementation

We implemented two versions of SBVUL, one with LangChain (LC) and one with the Claude Code Agent SDK (CC), which use the same instructions and equivalent read-only tools on the same export of the source code.

LangChain

The implementation with LangChain uses a custom LLM-tool loop to handle timeouts, context length exceed errors, and malformed tool calls, and to enforce the tool-call limit. These errors particularly affected open-weight models in our test runs. When an error is encountered, the agent returns the results gathered so far. For instance, if an LLM call with reasoning effort medium times out, it retries the call with reasoning effort none. The filesystem5 and ripgrep6 MCP servers provide the tools for directory listing, file-name search, and content search. In addition, SBVUL provides a tool to read exact line ranges from source files.

With this setup, we used the open-weight models Qwen3.8-27B-FP8 (Qwen 3.8), served with vLLM 0.27.1, and DeepSeek V4.1 Flash 1017 (DeepSeek V4.1), accessed via DeepSeek’s OpenAI-compatible API. At the time of our experiments (August to October 2026), Qwen 3.8 (27B parameters, 256k-token context) was among the strongest open-weight models of its size on the Artificial Analysis Intelligence Index (score 34)7, and DeepSeek V4.1 (552B parameters, 16B active, 1M-token context) performed comparably to large commercial models (score 45) at a lower cost per task8. We performed all experiments in a virtual machine with an 8-core AMD CPU, 62 GB of main memory, and NVIDIA H100 NVL with 94 GB, and Ubuntu 22.04.5 LTS. vLLM served Qwen 3.8 with the maximum context length, prefix caching, and the recommended settings for the model9. We used greedy decoding (temperature 0, seed 42) with top-k 15, min-p 0.02, and a repetition penalty of 1.02. Qwen 3.8 does not bound its thinking, therefore we capped medium reasoning effort at 4,096 thinking tokens. Based on several test runs on our hardware, we set the timeouts of an LLM call to 90 s and wrap-up call to 180 s, the timeout of a tool call to 30 s, and the maximum number of tool calls to 80 per phase.

DeepSeek V4.1 offers three reasoning efforts low, high, and max, and none disables thinking. We limited the output of a DeepSeek call, including thinking, to 16,000 tokens, because very long thinking exceeds the call timeout and increases the cost. We kept the same timeouts as for Qwen 3.8, except for the runs with max effort for which we increased the timeouts to 300 s and 600 s respectively. DeepSeek’s API ignores the seed and, in thinking mode, the temperature, so runs with DeepSeek V4.1 are not exactly reproducible.

Claude Code Agent SDK

The agents implemented with the Claude Code Agent SDK run in a Docker container to better restrict internet and filesystem access. The agents see only the vulnerable or patched version of the source code exported from the code repository. Each agent runs as a separate Claude Code session, and Claude Code’s own loop executes the tool calls. Our instructions replace Claude Code’s default system prompt and differ from those of the LangChain implementation only in two respects. First, they name Claude Code’s tools Read, Grep, and Glob. Second, the agent passes the output format as a JSON schema instead of in a wrap-up prompt. We limited a session to 80 turns, each of which may contain several tool calls, and to 1,800 s per function and phase.

Metrics

We evaluated the output of SBVUL in the two phases with statement-level and function-level metrics against our ground truth. For Phase 2, we additionally evaluated the explanation of each verdict against the ground truth with an LLM judge.

LLM Judge

As found in previous work, such as COR-RECT [21] and VulInstruct [43], matching a ground-truth statement does not establish that a detector identified the vulnerability for the right reason. We therefore use an LLM judge to verify whether the explanation of a confirmed verdict in Phase 2 matches the reasoning in the ground truth. We use Sonnet 5.5 with high effort as judge for all models. To check for self-preference, we also judged the verdicts of Sonnet 5.5 and Opus 5.5 with DeepSeek V4.1, which changes their true positives (TPs) by at most four and their hit rates by at most one percentage point. The LLM judge runs once per function with at least one confirmed or uncertain verdict, without tools and without the source code. It receives the ground-truth line numbers and rationale, together with the line number and explanation of each confirmed or uncertain verdict. It then checks whether each explanation matches the root cause, independently of the line number. For a patched function, the judge instead decides whether the explanation points at the vulnerability that the patch fixed. Ruled-out verdicts claim no vulnerability and are not judged.

Statement-Level Metrics

As in previous work by Hin et al. [14] and Chen et al. [6], each ground-truth statement is one instance. Note, a statement can span multiple lines of code and every remaining line in the function is a separate instance. In Phase 1, a candidate statement is a true positive (TP) if it is contained in the ground truth, and a false negative (FN) otherwise. A candidate statement not contained in the ground-truth is a false positive (FP), and every remaining line in the function is a true negative (TN). In a patched function, every candidate is an FP and every other line in the function is a TN.

In Phase 2, a statement in a vulnerable function is a TP if it has a confirmed verdict contained in the ground truth and the LLM judge accepts the verdict’s explanation, and an FN otherwise. Every other confirmed verdict is an FP, including one with a rejected explanation. In the patched function, each confirmed verdict is an FP. From these counts, we compute precision (P), recall (R), F1-score, and the Matthews correlation coefficient (MCC).

Function-Level Metrics

The hit rate (Hit) is the share of vulnerable functions with at least one TP. Cand is the average number of Phase 1 candidates per function. In Phase 2, ≥ 1 Conf is the share of functions with at least one confirmed verdict, whether or not it matches the ground truth. For the classification, we use the pair-wise metrics of PrimeVul [9]. A pair is pair-correct (P-C) if the vulnerable function is classified as vulnerable and the patched function as benign, pair-vulnerable (P-V) if both are classified as vulnerable, pair-benign (P-B) if both are classified as benign, and pair-reversed (P-R) otherwise.

Following CORRECT [21] and SecVulEval [1], we also check whether a classification is correct for the right reason (RR). A vulnerable function classified as vulnerable counts as a TP only if it has a Phase 2 TP. Otherwise, it counts as an FN. Every patched function classified as vulnerable is an FP. A pair is P-CRR if it is P-C and its vulnerable function has a Phase 2 TP. For a comparison with CORRECT and VulInstruct [43], we also report the metrics under the lenient CORRECT criterion (C). A vulnerable function classified as vulnerable is a TP if the judge accepts at least one of its verdicts regardless of its line number, and a patched function classified as vulnerable is an FP only if at least one of its verdicts points at the fixed vulnerability. Our right-reason criterion (RR) differs from CORRECT, which checks explanations against the CVE description and diff rather than a verified ground truth. It differs from SecVulEval, which accepts at most two unrelated statements per function. One TP suffices for a function to count as correct for the right reason, while the statement-level metrics also penalize false confirmed verdicts.

We count the functions with a hard timeout (HT), i.e., an LLM call that failed even after the retry, and with a soft timeout (ST), i.e., a call that succeeded when retried with reasoning effort none. Finally, we measured the runtime, tool calls, tokens, and estimated cost per function. We provide the latter two measurements in the replication package.

Results

In this section, we present the results to answer the research questions. All results are provided in the replication package.

Results and Answer to RQ1

tbl:III shows the Phase 1 results for the best configurations obtained with the LangChain (LC) and Claude Code (CC) agents. For Phase 1, we aim at a high hit rate and at a high recall for vulnerable functions.

Qwen 3.8 (none) reaches a hit rate of 56% (111 functions) and a recall of 0.455 with 173 TP statements. Although it is the best Phase 1 configuration for Qwen 3.8, all other models outperform it in hit rate and recall. DeepSeek V4.1 (high) reaches a hit rate of 73% (145 functions) and a recall of 0.613. Note that DeepSeek V4.1 offers no medium effort, and its high effort corresponds to the medium effort of Sonnet and Opus 5.5. Sonnet 5.5 reaches a hit rate of 69% (138 functions) and a recall of 0.568. Its hit rate exceeds that of Qwen 3.8 but is slightly lower than that of DeepSeek V4.1. Opus 5.5 (medium) performs best. It reaches the highest hit rate of 77% (153 functions), matches the most ground-truth statements (250, recall 0.658) and has the best precision (0.291), F1-score (0.403), and MCC (0.430) with on average 4.3 candidates per vulnerable function.

Every model also proposes candidates in nearly all patched functions (3.9–6.1 per function), which are all FPs. Considering both labels, precision therefore drops to 0.087–0.138. Qwen 3.8 has the fewest FPs on patched functions (772), followed by Opus 5.5 (957). Sonnet 5.5 and Opus 5.5 hardly use the tools, with on average 0.3 and 2.3 tool calls per function, compared to 34.5 by Qwen 3.8 and 33.4 by DeepSeek V4.1. With Claude Code, Sonnet 5.5 and Opus 5.5 process a function in 9–20 s on average, compared to 106 s and 185 s for Qwen 3.8 and DeepSeek V4.1.

a) Impact of the reasoning effort: We ran the two versions of SBVUL with various reasoning efforts. For Qwen 3.8, increasing the reasoning effort to medium does not improve the hit rate (55%) and recall (0.450) and improves precision only moderately (0.212 to 0.261 on the vulnerable functions), but raises the runtime from 103 s to 399 s per function and causes 26 soft timeouts.

We tested DeepSeek V4.1 with none and max reasoning effort on a random sample of 49 pairs and compared it to the results with high effort. Max effort raises the hit rate of DeepSeek V4.1 from 73% to 78% and the recall from 0.645 to 0.699, but lowers the overall precision from 0.124 to 0.103 and uses 2.6 times as many tokens. Increasing the reasoning effort therefore does not pay off for DeepSeek V4.1. Disabling thinking lowers both the hit rate to 59% and the precision to 0.049.

For Sonnet 5.5, on the same 49 pairs, high effort raises the hit rate from 67% to 78%, the recall from 0.602 to 0.763, and the overall precision from 0.071 to 0.110 compared to medium effort, but at 6.7 times the runtime and 4.7 times the cost. Opus 5.5 with medium effort reaches a similar hit rate (76%) and recall (0.720) on these pairs with a higher overall precision (0.147), at 41% of the cost and a quarter of the runtime. On the first 30 of these pairs, xhigh effort further raises the hit rate of Opus 5.5 from 80% to 97% and the recall from 0.797 to 0.922, but lowers the overall precision at 4.7 times the runtime.

b) Differences between CWE types: We compare the hit rates per CWE type of the four configurations in Table III. DeepSeek V4.1, Sonnet 5.5, and Opus 5.5 achieve their highest hit rates on out-of-bounds accesses (75 functions with CWE-787, -125, -119, or -120), with 84%, 76%, and 87%; Qwen 3.8 reaches 65–68%. NULL pointer dereferences (CWE-476, 14 functions) reach similar hit rates (71–86% for DeepSeek V4.1, Sonnet 5.5, and Opus 5.5, 50–79% for Qwen 3.8). The hit rates for CWE-703, improper check or handling of exceptional conditions (16 functions), are 31–38% for Qwen 3.8, 69% for DeepSeek V4.1, 50% for Sonnet 5.5, and 56% for Opus 5.5. These defects center on missing or incorrect checks and involve more ground-truth statements (2.3 per function vs. 1.9 for out-of-bounds accesses). Of the five resource management weaknesses (CWE-400, -401, and -772), every configuration finds the same single function, and of the reachable assertions (CWE-617, 6 functions), each configuration finds at most three, while all miss three. The seven access-control and authentication weaknesses (CWE-269, -276, -284, -287, -288, and -522) are missed by Qwen 3.8 and, except for one, by Sonnet 5.5, whereas Opus 5.5 finds five and DeepSeek V4.1 two. Missing releases, checks, or permissions can make it difficult to identify a single exploitation point. Overall, 28 of the 200 vulnerable functions are missed by every configuration.

ModelEffortLabelHitTPFPFNPRF1MCCCandHT/STTCTime
Qwen 3.8LC_{LC}nonevul56%1736432070.2120.4550.2890.3024.122 / 032.6103 s
pat--772-----3.861 / 036.4109 s
all-17314152070.1090.4550.1760.2163.993 / 034.5106 s
DeepSeek V4.1LC_{LC}highvul73%2336461470.2650.6130.3700.3954.450 / 731.7168 s
pat--1020-----5.100 / 935.0202 s
all-23316661470.1230.6130.2040.2684.770 / 1633.4185 s
Sonnet 5.5CC_{CC}mediumvul69%21610501640.1710.5680.2620.3016.37-0.39 s
pat--1220-----6.10-0.310 s
all-21622701640.0870.5680.1510.2146.24-0.39 s
Opus 5.5CC_{CC}mediumvul77%2506101300.2910.6580.4030.4304.33-1.818 s
pat--957-----4.79-2.723 s
all-25015671300.1380.6580.2280.2954.56-2.320 s

Table III. Best Phase 1 results on the 200 pairs per model and reasoning effort for the vulnerable (vul) and patched (pat) functions.

c) Answer to RQ1: In Phase 1, SBVUL identifies ground-truth statements in 56–77% of the vulnerable functions, with a recall of 0.45–0.66. Opus 5.5 performs best with a hit rate of 77%, followed by DeepSeek V4.1, Sonnet 5.5, and Qwen 3.8. Higher reasoning effort raises hit rate and recall for DeepSeek V4.1, Sonnet 5.5, and Opus 5.5, with mixed effects on precision, but at 2.6–6.7 times the tokens, cost, or runtime. Patched functions receive about as many candidates as vulnerable ones, so the candidates require further verification.

Results and Answer to RQ2

Table IV shows the best results of Phase 2 with the Phase 1 candidates from Table III. We aim for high hit rate, and in addition to high recall also high precision.

ModelEffortLabelHit≥ 1 ConfTPFPFNPRF1MCCHT/STTCTime
Qwen 3.8LC_{LC}mediumvul26%42%921202880.4340.2420.3110.3200 / 1014.7197 s
pat-23%-104-----0 / 914.1178 s
all--922242880.2910.2420.2640.2630 / 1914.4188 s
DeepSeek V4.1LC_{LC}highvul57%84%1772972030.3730.4660.4150.4110 / 225.0117 s
pat-61%-360-----0 / 325.8136 s
all--1776572030.2120.4660.2920.3100 / 525.4126 s
Sonnet 5.5CC_{CC}mediumvul43%59%1391972410.4140.3660.3880.384-5.227 s
pat-37%-211------5.126 s
all--1394082410.2540.3660.3000.301-5.227 s
Opus 5.5CC_{CC}mediumvul66%89%2152131650.5020.5660.5320.529-4.631 s
pat-52%-266------4.934 s
all--2154791650.3100.5660.4000.415-4.733 s
xhighvul73%94%2302201500.5110.6050.5540.552-13.2107 s
pat-54%-281------14.4134 s
all--2305011500.3150.6050.4140.433-13.8121 s

Table IV. Phase 2 results on the 200 pairs per model and reasoning effort for the vulnerable (VUL) and patched (PAT) functions.

Qwen 3.8 (medium) reaches a hit rate of only 26% (52 functions), although its Phase 1 candidates hit 56% of the vulnerable functions. It has the fewest FPs on patched functions of all configurations (104). DeepSeek V4.1 reaches a hit rate of 57% (113 functions). However, it confirms candidates in 61% of patched functions, the most of any model, compared with 84% of vulnerable functions. Sonnet 5.5 lies between Qwen 3.8 and DeepSeek V4.1 with a hit rate of 43%. Its hit rate of 66% (131 functions) clearly exceeds DeepSeek V4.1’s 57%, although its Phase 1 hit rate is only 4 percentage points higher (77% vs. 73%). It outperforms DeepSeek V4.1 and Sonnet 5.5 in hit rate, precision, recall, F1-score, and MCC. With xhigh effort, Opus 5.5 achieves the best results with a hit rate of 73% (146 functions), a recall of 0.605, a precision of 0.511, and an MCC of 0.552. However, Opus 5.5 confirms candidates in 52% (medium) and 54% (xhigh) of the patched functions, with more FPs on patched functions (266 and 281) than Qwen 3.8 (104).

Considering both labels, overall precision drops by 0.12–0.20, relatively most for DeepSeek V4.1 (from 0.373 to 0.212), leaving DeepSeek V4.1 and Sonnet 5.5 with similar MCC values (0.310 and 0.301), while Opus 5.5 stays ahead (0.415 and 0.433). Compared to the rows all of Phase 1, Phase 2 raises precision by a factor of 1.7 to 2.9 for all models while recall drops by 0.05–0.21, least for Opus 5.5 (from 0.658 to 0.566 and 0.605) and most for Qwen 3.8 (from 0.455 to 0.242).

a) Impact of the reasoning effort: For Qwen 3.8, using no reasoning results in almost the same hit rate (51 vs. 52 functions) and recall (0.234 vs. 0.242) as medium effort, but in lower precision (0.324 vs. 0.434) and MCC (0.270 vs. 0.320). For DeepSeek V4.1, max effort does not improve over high effort on the first 30 of the 49 sampled pairs, but needs three times the runtime and 2.5 times the tokens, so we did not run it on the remaining pairs. For Sonnet 5.5, xhigh effort in both phases on the random sample of 49 pairs confirms a candidate on a ground-truth statement in 36 functions, against 22 with medium effort and 36 for Opus 5.5 with medium effort, at three times the cost of Opus 5.5 and four times the runtime. For Opus 5.5, xhigh effort raises the hit rate from 66% to 73% (131 to 146 functions) and MCC from 0.529 to 0.552 at almost the same precision (0.502 and 0.511), but it confirms candidates in slightly more patched functions (54% vs. 52%), at 3.7 times the runtime and 3.5 times the cost.

b) Differences between CWE types: Out-of-bounds accesses (75 pairs) reach hit rates of 33–35% for Qwen 3.8, 67% for DeepSeek V4.1, 56% for Sonnet 5.5, and 69% (medium) and 84% (xhigh) for Opus 5.5. Unlike in Phase 1, NULL pointer dereferences (14 pairs) are verified about as well or better (21–29% for Qwen 3.8, 57–86% for the others). For CWE-703 (16 pairs), Qwen 3.8 gets a TP in only one function, DeepSeek V4.1 in 56%, Sonnet 5.5 in 31%, and Opus 5.5 in 44% and 50%. Rare types remain difficult to detect, with hits in at most one of the five resource management pairs and one of the six reachable assertions. Access-control and authentication weaknesses (7 pairs) are an exception for Opus 5.5, which gets a TP in six (medium) and five (xhigh) of them, whereas Qwen 3.8 gets none, DeepSeek V4.1 two, and Sonnet 5.5 one.

c) Answer to RQ2: Phase 2 of SBVUL confirms at least one ground-truth statement with an accepted explanation in 26–73% of the vulnerable functions. Opus 5.5 performs best with a hit rate of 66% (medium) and 73% (xhigh) and a recall of 0.566 and 0.605, followed by DeepSeek V4.1 (57%, 0.466), Sonnet 5.5 (43%, 0.366), and Qwen 3.8 (26%, 0.242). Higher reasoning effort improves Opus 5.5’s hit rate but barely changes Qwen 3.8’s. Over the vulnerable and patched functions, MCC is 0.26–0.43. However, all models also confirm candidates in many patched functions (23% for Qwen 3.8, 37–61% for the others), limiting the function-level classification derived from confirmed verdicts. RQ3 examines this limitation.

Results and Answer to RQ3

Table V shows the results of the classification step based on the verdicts from Phase 2 under the three evaluation

ModelEffortLabel only P-CP-VP-BP-RPRF1MCCRight reason P-C_RRP_RRR_RRF1_RRMCC_RRCORRECT P-C_CF1_CMCC_C
Qwen 3.8LCmedium4736107100.6430.4150.5050.198350.5310.2600.3490.035540.4280.379
DeepSeek V4.1LChigh5910820130.5800.8350.6840.256420.4830.5650.521−0.0411150.7460.608
Sonnet 5.5CCmedium52657490.6130.5850.5980.215440.5370.4300.4780.061850.6160.472
Opus 5.5CCmedium78991850.6300.8850.7360.399650.5570.6550.6020.1371350.8200.680
xhigh84103850.6340.9350.7560.449730.5750.7300.6430.1971510.8680.747

Table V. Classification results on the 200 pairs per model and reasoning effort.

criteria presented in Section V-B. Label only compares the predicted labels with the ground-truth labels, regardless of the verdicts behind them. Right reason (RR) additionally requires a vulnerable function classified as vulnerable to have a Phase 2 TP. For CORRECT, which is more lenient than RR, an accepted explanation suffices regardless of the line, and a patched function classified as vulnerable counts as an FP only if one of its explanations points at the fixed vulnerability.

a) Label only pair-wise classification: Opus 5.5 classifies the most pairs correctly, P-C of 84 (42%) with xhigh and 78 (39%) with medium effort, with an MCC of 0.449 and 0.399. DeepSeek V4.1 classifies 59 pairs (29.5%) correctly, Sonnet 5.5 52 (26%), and Qwen 3.8 47 (23.5%). The models fail in opposite ways. DeepSeek V4.1, Sonnet 5.5, and Opus 5.5 classify both functions as vulnerable in most of their incorrect pairs (P-V of 65–108 pairs), whereas Qwen 3.8 classifies both as benign (P-B of 107 pairs). Reversed pairs are rare (5–13 pairs). Opus 5.5 classifies clearly more pairs correctly than DeepSeek V4.1 (78 and 84 vs. 59 pairs), whereas DeepSeek V4.1, Sonnet 5.5, and Qwen 3.8 lie close together (47–59 pairs), and xhigh effort adds only six correct pairs for Opus 5.5.

b) Correct for the right reason (RR): Correct labels do not always rest on correctly identified vulnerabilities, as also found by Li et al. [21] and Ahmed et al. [1]. For Opus 5.5, 73 of the 84 (xhigh) and 65 of 78 (medium) pairs are also correct for the right reason, for Sonnet 5.5 44 of 52 pairs, for Qwen 3.8 35 of 47 pairs, and for DeepSeek V4.1 only 42 of 59 pairs. Under the right-reason criterion, the lead of Opus 5.5 over DeepSeek V4.1 grows (73 vs. 42 pairs with xhigh effort), and the lead of DeepSeek V4.1 over Sonnet 5.5 disappears (42 vs. 44 pairs). Recall drops from 0.415–0.935 to 0.260–0.730 and precision from 0.580–0.643 to 0.483–0.575, since every patched function classified as vulnerable remains an FP. MCC falls to 0.197 (xhigh) and 0.137 (medium) for Opus 5.5 and to around zero (−0.041 to 0.061) for the other configurations, with Opus 5.5 retaining the strongest positive correlation with the labels. Under the lenient CORRECT criterion, DeepSeek V4.1, Sonnet 5.5, and Opus 5.5 classify 85–151 pairs correctly (F1-score 0.616–0.868). This criterion, however, ignores most of the confirmed verdicts on patched functions that our statement-level metrics count as FPs.

c) Comparison with previous approaches: Table VI compares SBVUL with the three state-of-the-art approaches VulTrial [38], VulTriage [32], and VulInstruct [43] based on the results reported in the corresponding papers. All three approaches were evaluated on the PrimeVul Paired test set which allows a comparison with our approach. * denotes values computed by us from the reported pair counts (VulTrial, VulTriage) or accuracy, precision, and recall (VulInstruct). With both reasoning efforts, Opus 5.5 exceeds the reported P-C, F1-score, and MCC of previous approaches. With xhigh effort, it achieves 42% correct pairs, an F1-score of 0.756, and an MCC of 0.449, compared with 33.8%, 0.691, and 0.290 for VulTriage. All configurations of SBVUL, including the open-weight models, classify more pairs correctly than VulTrial (23.5–42% vs. 18.6%) with a higher MCC (0.198–0.449 vs. 0.072). VulTriage classifies both functions as vulnerable in 47.6% of pairs, while DeepSeek V4.1 does so in 54%. DeepSeek V4.1 reaches almost the same F1-score (0.684), despite an MCC of only 0.256. Under the CORRECT criterion, all configurations outperform VulInstruct in P-C (27–75.5% vs. 17.2%) and MCC (0.379–0.747 vs. 0.083), and all except Qwen 3.8 also in F1-score (0.616–0.868 vs. 0.450).

ApproachModelLabel only P-CF1MCCCORRECT P-CF1MCC
VulTrialGPT-4o18.60.563*0.072*–––
VulTriageGPT-4o33.80.6910.290*–––
VulInstructDeepSeek-V3–––17.20.4500.083*
SBVULQwen 3.8 (med.)23.50.5050.19827.00.4280.379
SBVULDeepSeek V4.1 (high)29.50.6840.25657.50.7460.608
SBVULSonnet 5.5 (med.)26.00.5980.21542.50.6160.472
SBVULOpus 5.5 (med.)39.00.7360.39967.50.8200.680
SBVULOpus 5.5 (xhigh)42.00.7560.44975.50.8680.747

Table VI. Comparison with previous approaches on the PrimeVul Paired test set (previous approaches: all 435 pairs, SBVUL: our 200 pairs). P-C in %.

d) Answer to RQ3: SBVUL classifies 23.5–42% of the pairs correctly, with Opus 5.5 performing best and the open-weight models reaching 29.5% (DeepSeek V4.1) and 23.5% (Qwen 3.8). Of these correct pairs, 35–73 are also correct for the right reason. Opus 5.5 achieves the most such pairs 73 (36.5%) of all pairs. The larger models DeepSeek V4.1, Sonnet 5.5, and Opus 5.5 mainly err by classifying patched functions as vulnerable, while Qwen 3.8 often classifies both functions as benign.

as benign. Opus 5.5 exceeds the reported P-C, F1-score, and MCC of three recent approaches, and all configurations exceed VulTrial and VulInstruct in P-C and MCC.

Discussion

Implications

In the following, we discuss potential implications of our results on developers and the research on vulnerability detection.

a) For developers: SBVUL can support security code reviews and the triage of software vulnerabilities. With Opus 5.5 (xhigh), Phase 2 confirms 450 statements in the vulnerable functions (TP + FP in Table IV), 2.3 per function, but also 281 in the patched functions, 1.4 per function. Of these 731 confirmed statements, 230 are TPs, meaning about a third of them point a developer to the vulnerability (precision 0.315). It misses 150 of the 380 ground-truth statements (FN, recall 0.605) and finds none of them in 54 of the 200 vulnerable functions (hit rate 73%). Each verdict output by SBVUL names the statement and explains why it is exploitable, so a developer can check the statement instead of searching the function for the reason of a label. SAST tools leave developers with longer lists. In the comparison of Hassler et al., only 0.24% of the reports of the best analyzer correspond to a known bug [13], against 31.5% of the statements confirmed by Opus 5.5 (xhigh) over the vulnerable and patched functions. The settings differ, and our balanced sample contains far more vulnerable functions than real code. With 3% vulnerable functions as in PrimeVul [9], the same rates would give a precision of about 2.4%, which is still ten times higher than 0.24%. Even when running SBVUL with Qwen 3.8, the precision would be about 2.6%, because it confirms fewer statements in patched functions (104 vs. 281). However, Qwen 3.8 also finds fewer of the vulnerable statements (recall 0.242 vs. 0.605).

If the task of a developer is to detect vulnerable functions, SBVUL with Opus 5.5 (xhigh) flags 94% of the vulnerable but also 54% of the patched functions (precision 0.634 in Table V). With 3% vulnerable functions in a project, this would give a precision of about 5%, i.e., a developer would inspect about 20 flagged functions per vulnerable one. In return, SBVUL misses only 13 of the 200 vulnerable functions (recall 0.935). Under the right-reason criterion, recall drops to 0.730. In 146 of the 187 flagged vulnerable functions (hit rate 73% in Table IV), a confirmed statement points the developer to the vulnerability, and in the remaining 41, the developer still has to search the function for it. With 3% vulnerable functions, the right-reason precision of 0.575 would give about 4%, i.e., about 25 flagged functions per such vulnerable function.

b) For researchers: Our results indicate that detecting vulnerabilities at the statement level is feasible with current LLMs and makes detection checkable. With Opus 5.5 (xhigh), SBVUL confirms a ground-truth statement in 74% of the vulnerable functions, and across all models, the judge accepts 94–99% of the confirmed verdicts on ground-truth statements, so a hit almost always comes with the right reason. Verifying candidates in a second phase raises precision by a factor of 1.7–2.9 at a cost of 0.05–0.21 recall, and the statement-level verdicts also yield the best pair-wise classification (42% correct pairs, MCC 0.449). Two limitations remain. First, all models also confirm statements in 23–61% of the patched functions, i.e., they often do not recognize that the patch removed the vulnerability. This is in line with Ding et al. [9] and Steenhoek et al. [31], who found that LLMs classifying functions directly hardly distinguish vulnerable from patched code. Our results show that this limitation persists even when every verdict must be backed by evidence collected with tools. Second, the model matters more than the amount of tool use. With medium effort and only 2–5 tool calls per function in Phases 1 and 2, Opus 5.5 outperforms DeepSeek V4.1 (high), which makes 25–33 tool calls.

Threats to Validity

We identify several threats to construct, internal, and external validity of our results.

a) Construct validity: Our statement-level metrics are strict in two ways and may therefore underestimate the performance. First, a confirmed verdict counts as a TP only if it lies on a ground-truth statement, although the judge accepts the explanations of another 30–84 confirmed verdicts per configuration that name the root cause but point at a different line. Second, every confirmed verdict on a patched function counts as an FP, although a patched function may still contain other vulnerabilities. The lenient CORRECT criterion in RQ3 relaxes both rules. The LLM judge (Sonnet 5.5, high effort) belongs to the same model family as two of the evaluated models and might prefer their explanations. To check this, we also judged the verdicts of Sonnet 5.5 and Opus 5.5 with DeepSeek V4.1. The two judges agree on 96–98% of the confirmed verdicts on vulnerable functions (Cohen’s κ\kappa 0.93–0.95), and Sonnet 5.5 accepts only 2–6 more verdicts per configuration, which changes the TPs by at most four.

b) Internal validity: Using Opus 5.5 to create the ground truth may favor statements that this model considers exploitable. To mitigate this threat, the ground-truth tool received the CVE description, the commit message, and the diff, which SBVUL never sees. Furthermore, two authors verified every function and resolved all disagreements by consensus. Since most CVEs date from 2021 and 2022, before the training cutoff of all models, the models may know the vulnerabilities. However, the agents never see the CVE, the commit, or the version history, and the many confirmed verdicts on patched functions indicate that the models do not simply recall the fix. We ran each configuration once, and the APIs of Anthropic and DeepSeek offer no seed, limiting reproducibility. Differences of a few percentage points between configurations may therefore lie within this run-to-run variation. Finally, a difference between models may partly stem from our two implementations of the agents. However, both versions of SBVUL receive the same instructions only adapted to the available tools and equivalent filesystem and text search tools. To identify potential differences, we ran the LangChain Phase 1 agent with Sonnet 5.5 which reached a similar Phase 1 hit rate on 97 pairs compared to the implementation with the Claude Code Agent SDK (67% vs. 65%).

c) External validity: We evaluated 200 randomly selected pairs of the PrimeVul Paired test set with C/C++ functions only. Therefore, our results are valid for these 200 pairs, while previous approaches used all 435 pairs, which might bias the comparison with these approaches. The 200 pairs cover 104 of the 123 projects and 45 of the 57 CWEs from the original PrimeVul Paired test set. While we consider this a representative sample of real-world functions, our results remain limited to C/C++ projects. Furthermore, we performed the experiments with four models, and the results might differ for other models. We mitigated this threat by selecting two recent and highly rated open-weight models of different sizes from two providers and included two recent frontier models from Anthropic. Since new models are released frequently, we report their versions and provide all prompts, tools, and results in our replication package.

Conclusion

We presented SBVUL, an agentic approach that detects exploitable statements in C/C++ functions in two phases. An agent first identifies candidate statements, and a second agent analyzes each of them in detail and either confirms or rules it out with evidence, or marks it uncertain. On a random sample of 200 pairs of the PrimeVul Paired test set with a manually verified ground truth, Phase 2 confirms a ground-truth statement with an accepted explanation in up to 73% of the vulnerable functions. Because each confirmed verdict can be checked at its line, a function flagged as vulnerable becomes verifiable for developers.

The classification derived from these verdicts outperforms existing approaches, with up to 42% correct pairs and an MCC of 0.449 compared to 33.8% and 0.290 for VulTriage [32], the best of them. However, separating vulnerable from patched code remains challenging. Even the best-performing model, Opus 5.5, confirms at least one candidate in 54% of the patched functions, resulting in a statement-level precision of 0.315, recall of 0.605, and F1-score of 0.414 over both vulnerable and patched functions.

In future work, we plan to first extend the evaluation of our approach to non-paired vulnerable and benign functions from the PrimeVul dataset. Next, motivated by VulTriage, we plan to improve our approach by considering information on vulnerabilities from the CWE database. Furthermore, we plan to investigate the benefits of equipping our agents with static code analysis tools, such as codebadger [17].

USE OF GENERATIVE AI

We used Anthropic Claude and OpenAI GPT models to assist with writing, coding, and literature search. The authors reviewed and refined the generated material and take full responsibility for the final manuscript

References

  1. [1]M. Ahmed et al. SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detection. preprint. 2025. DOI: 10.48550/arXiv.2505.19828.DOI
  2. [2]P. Avgustinov et al. “QL: Object-oriented Queries on Relational Data”. In: 30th European Conference on Object-Oriented Programming (ECOOP). Vol. 56. LIPIcs. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, 2016, 2:1–2:25. DOI: 10.4230/LIPIcs.ECOOP.2016.2.
  3. [3]Cristiano Calcagno et al. “Moving Fast with Software Verification”. In: NASA Formal Methods (NFM). Vol. 9058. LNCS. Springer, 2015, pp. 3–11. DOI: 10.1007/978-3-319-17524-9_1.DOI
  4. [4]Partha Chakraborty et al. “Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets”. In: IEEE Trans. Software Engineering (TSE) 50.8 (2024), pp. 2163–2177. DOI: 10.1109/tse.2024.3423712.DOI
  5. [5]Saikat Chakraborty et al. “Deep Learning Based Vulnerability Detection: Are We There Yet?”. In: IEEE Trans. Software Engineering (TSE) 48.9 (2022), pp. 3280–3296. DOI: 10.1109/tse.2021.3087402.DOI
  6. [6]Yinglong Chen et al. “SLVD-CC: Enhancing Statement-Level Vulnerability Detection via Context Clarification”. In: Science of Computer Programming 252 (2026), p. 103469. DOI: 10.1016/j.scico.2026.103469.DOI
  7. [7]Yizheng Chen et al. “DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection”. In: RAID 2023: The 26th International Symposium on Research in Attacks, Intrusions and Defenses. 2023, pp. 654–668. DOI: 10.1145/3607199.360742.arxiv.org/abs/2304.00409
  8. [8]Roland Croft, M. Ali Babar, and M. Mehdi Kholoosi. “Data Quality for Software Vulnerability Datasets”. In: Proc. IEEE/ACM Int. Conf. Software Engineering (ICSE). 2023, pp. 121–133. DOI: 10.1109/icse48619.2023.00022.DOI
  9. [9]Yangxin Drumond et al. “Vulnerability Detection with Code Language Models: How Far are We?” In: Proc. IEEE/ACM Int. Conf. Software Engineering (ICSE). 2025, pp. 1729–1741. DOI: 10.1109/icse55347.2025.00038.
  10. [10]Xueying Du et al. “iscp˅ul-rag/scpl: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG”. In: ACM Trans. Software Engineering and Methodology (TOSEM) 35.10 (2026), pp. 1–25. DOI: 10.1145/3972777.
  11. [11]Jiahao Fan et al. “A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries”. In: Proc. IEEE/ACM Int. Conf. Mining Software Repositories (MSR). 2020, pp. 508–512. DOI: 10.1145/3379597.3387501.DOI
  12. [12]Michael Fu and Chakkrit Tantithamthavorn. “LineVul: a transformer-based line-level vulnerability prediction”. In: Proc. IEEE/ACM Int. Conf. Mining Software Repositories (MSR). 2022, pp. 608–620. DOI: 10.1145/3524842.3528452.DOI
  13. [13]Keno Hassler, Philipp Görz, and Stephan Lipp. A Comparative Study of Fuzzers and Static Analysis Tools for Finding Memory Unsafe in C and C++. preprint. 2025. DOI: 10.48550/arXiv.2505.22052.DOI
  14. [14]David Hin et al. “LineVD: Statement-level Vulnerability Detection using Graph Neural Networks”. In: Proc. IEEE/ACM Int. Conf. Mining Software Repositories (MSR). 2022, pp. 596–607. DOI: 10.1145/3524842.3527949.DOI
  15. [15]Sihao Hu et al. “Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives”. In: 2023 5th IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA). IEEE, 2023, pp. 297–306. DOI: 10.1109/TPS-ISA58951.2023.00044.DOI
  16. [16]Kexing Ji et al. VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection. preprint. 2026. DOI: 10.48550/arXiv.2608.02001.DOI
  17. [17]Ahmed Lekssays. “Bridging Code Property Graphs and Language Models for Program Analysis”. In: Proceedings of the 2026 IEEE/ACM 4th International Workshop on Software Vulnerability Management. 2026, pp. 33–40. DOI: 10.1145/3786165.3788441.DOI
  18. [18]Ahmed Lekssays et al. “LLMxCPG: Context-Aware Vulnerability Detection Through Code Property Graph-Guided Large Language Models”. In: 34th USENIX Security Symposium (USENIX Security 25). 2025.arxiv.org/abs/2507.16585
  19. [19]Yi Li, Shaohua Wang, and Tien N. Nguyen. “Vulnerability detection with fine-grained interpretations”. In: Proc. ACM Int. Conf. Foundations of Software Engineering (FSE). 2021, pp. 292–303. DOI: 10.1145/3468264.3468597.DOI
  20. [20]Youpeng Li et al. VULPO: Context-Aware Vulnerability Detection via On-Policy LLM Optimization. preprint. 2025. DOI: 10.48550/arXiv.2511.11896.DOI
  21. [21]Yue Li et al. Everything You Wanted to Know About LLM-based Vulnerability Detection But Were Afraid to Ask. preprint. 2025. DOI: 10.48550/arXiv.2504.13474.DOI
  22. [22]Zi Liang et al. Argus: Reorchestrating Static Analysis via a Multi-Agent Ensemble for Full-Chain Security Vulnerability Detection. preprint. 2026. DOI: 10.48550/arXiv.2604.06633.DOI
  23. [23]V. Benjamin Livshits and Monica S. Lam. “Finding Security Vulnerabilities in Java Applications with Static Analysis”. In: Proc. USENIX Security Symposium (USENIX Security). 2005, pp. 271–286.
  24. [24]LLVM. Clang Static Analyzer. https://clang-analyzer.llvm.org. 2020.
  25. [25]Daniel Marjamäki. Cppcheck: A Tool for Static C/C++ Code Analysis. https://cppcheck.sourceforge.io/. 2007.
  26. [26]Renwei Meng, Haoyi Wu, and Jingming Wang. VulnAgent-R2: Evidence-Calibrated Multi-Agent Auditing for Repository-Level Vulnerability Detection. preprint. 2026. DOI: 10.48550/arXiv.2603.13384.DOI
  27. [27]Aman Priyanshu et al. Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale. preprint. 2026. DOI: 10.48550/arXiv.2609.15939.DOI
  28. [28]Niklas Risse, Jing Liu, and Marcel Böhme. “Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection”. In: Proceedings of the ACM on Software Engineering 2.ISSTA (2025), pp. 380–410. DOI: 10.1145/3728887.DOI
  29. [29]Semgrep. Semgrep OSS. https://github.com/semgrep/semgrep. 2020.
  30. [30]Benjamin Jeremiah Steenhoek, Hongyang Gao, and Wei Le. “Dataflow Analysis-Inspired Deep Learning for Efficient Vulnerability Detection”. In: ICSE ’24: IEEE/ACM 46th International Conference on Software Engineering. 2024, pp. 1–13. DOI: 10.1145/3597503.3623345.DOI
  31. [31]Benjamin Jeremiah Steenhoek et al. To Err is Machine: Vulnerability Detection Challenges LLM Reasoning. preprint. 2024. DOI: 10.48550/arXiv.2403.17218.DOI
  32. [32]Wenxin Tang et al. VulTriage: Triple-Path Context Augmentation for LLM-Based Vulnerability Detection. preprint. 2026. DOI: 10.48550/arXiv.2605.09461.DOI
  33. [33]Yongshao Wang and Zhiqiu Huang. Security Is Relative: Training-Free Vulnerability Detection via Multi-Agent Behavioral Contract Synthesis. preprint. 2026. DOI: 10.48550/arXiv.2604.19012.DOI
  34. [34]Ziliang Wang et al. “VulAgent: Hypothesis-Validation Driven Multi-Agent Architecture for Vulnerability Detection”. In: Findings of the Association for Computational Linguistics: ACL 2026. 2026, pp. 18598–18616. DOI: 10.18653/v1/2026.findings-acl.928.DOI
  35. [35]Xin-Cheng Wen et al. “Boosting Vulnerability Detection of LLMs via Curriculum Preference Optimization with Synthetic Reasoning Data”. In: Findings of the Association for Computational Linguistics: ACL 2025. 2025, pp. 8935–8949. DOI: 10.18653/v1/2025.findings-acl.467.
  36. [36]Martin Weyssow et al. R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillation. preprint. 2025. DOI: 10.48550/arXiv.2504.04699.DOI
  37. [37]David A. Wheeler. Flawfinder. https://dwheeler.com/flawfinder/. 2001.
  38. [38]Ratnadria Widyasari et al. Let the Trial Begin: A Mock-Court Approach to Vulnerability Detection using LLM-Based Agents. preprint. 2026. DOI: 10.1145/3744916.3773256.DOI
  39. [39]Haoran Xi et al. From Trace to Line: LLM Agent for Real-World OSS Vulnerability Localization. preprint. 2025. DOI: 10.48550/arXiv.2510.02389.DOI
  40. [40]Fabian Yamaguchi et al. “Modeling and Discovering Vulnerabilities with Code Property Graphs”. In: Proc. IEEE Symposium on Security and Privacy (S&P). 2014, pp. 590–604. DOI: 10.1109/SP.2014.44.DOI
  41. [41]Fabian Yamaguchi et al. “Automatic Inference of Search Patterns for Taint-Style Vulnerabilities”. In: Proc. IEEE Symposium on Security and Privacy (S&P). 2015, pp. 797–812. DOI: 10.1109/SP.2015.54.DOI
  42. [42]Sungju Yun et al. CLEAR: Causal Context-Based Agentic Reasoning for Vulnerability Detection. preprint. 2026. DOI: 10.48550/arXiv.2608.03134.arxiv.org/abs/2608.03134
  43. [43]Hao Zhu et al. VulInstr: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specifications. preprint. 2025. DOI: 10.48550/arXiv.2511.04014.arxiv.org/abs/2511.04014

Paper details

vlnc:W_jpyz9s7d
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Anthropic Claude, GPT (multiple versions)

View statement
Looking for related papers…
Contents