When LLM Architectures Meet Smart Contract Vulnerability Detection: a Replication Study

Introduction

A smart contract is a self-executing program that runs on a blockchain platform, such as Ethereum [1]. It automatically enforces and executes the terms of an agreement when predefined conditions are met, without the need for intermediaries. Unlike traditional software, once deployed, smart contracts cannot be modified or patched, so even minor flaws can lead to severe repercussions [2].

Ensuring the security of smart contracts is critical. Well-known vulnerabilities, including Reentrancy, Timestamp Dependence, Transaction Order Dependence, Authorization via tx.origin, and Unchecked External Calls, have already led to major financial losses, most notably in the DAO hack [3]. To address these issues, static and dynamic analysis techniques have long been used by researchers, as extensively surveyed by Ramed­er et al. [4] and Iuliano et al. [5]. Recently, the focus has shifted to methods based on Large Language Models (LLMs), including VulnHunt [6], GPTScan [7], AdvSCanner [8], LLMSmartSec [9], GPTLens [10], FELLMVul [11], and SolGPT [12], demonstrating the ability of LLMs to discover vulnerabilities by leveraging carefully designed prompts. Within this line of work, the empirical study by Chen et al. [13] stands out as a reference point. They systematically evaluated three GPT model snapshots on the SmartBugs curated benchmark [14], reporting high recall coupled with low and unstable precision. However, two aspects of their setup leave open questions worth investigating. First, their investigation focuses on a single architectural family, i.e., the dense Transformer used in the GPT line, leaving it unknown whether the observed behavior is a property of LLMs in general or an artifact of that design. Second, the study relies on model snapshots scheduled for deprecation by OpenAI [15], which threatens the long-term reproducibility and practical relevance of their conclusions. Meanwhile, the architectural landscape of LLMs has diversified considerably, with Mixture-of-Experts, RNN-like, and State Space Model designs now competing with the dense Transformer at comparable scales. This paper presents an extended replication of Chen et al. [13] to verify whether their findings hold under a change in the underlying architecture. We organize the work around two research questions. The first asks whether the original results can be reproduced under controlled conditions on GPT-4 itself (exact replication). The second question asks whether the same protocol, prompt, dataset, and evaluation procedure yield the same behavior when the model is drawn from a different architectural family (differentiated replication). Therefore, we re-execute the experiment on representative models from three further families: Mixture-of-Experts, RNN-like, and, in a preliminary capacity, State Space Models. By keeping the prompt, dataset, evaluation procedure, and temperature fixed, we examine whether the behavior observed for GPT-4 transfers to a set of models with heterogeneous architectural designs. However, because the selected models also differ in scale, training data, tokenization, tuning, and cut-off, our design does not isolate architecture as the only causal factor. The results confirm Chen et al.’s findings [13] in the exact replication, including the high-recall, low-precision pattern observed for GPT-4. However, the differentiated replication shows that this pattern does not generalize uniformly across architectures. In particular, the same qualitative denseTransformer profile is observed in both GPT-4 and Gemma-3 despite a substantial scale gap, whereas DeepSeek-V2-Lite and RWKV-7 exhibit much more conservative behavior, achieving higher specificity at the cost of substantially lower recall. At the same time, the contrast between Qwen-3.5 and DeepSeek-V2-Lite shows that shared membership in the same architectural family is not, by itself, sufficient to predict detection behavior.

Paper Structure. Section II reviews related work and key concepts to understand the paper, whereas Section III describes the reference work by Chen et al. [13]. Section IV details the methods used to conduct the study, with its execution plan and procedures to ensure transparency and reproducibility, and whose results are reported in § 5 and VI, and discussed in Section VII. § 8 lists the threats to the validity of the study, addressing potential limitations and biases in the study and the measures taken to mitigate them. Finally, Section IX concludes the paper.

Background and Related Works

This section provides background on smart contracts, their vulnerabilities, an introduction to large language models and their architectures, and related work.

Smart Contracts and their Vulnerabilities

Nick Szabo introduced smart contracts in 1994 [16], but they have found new life on the Ethereum blockchain. They are self-executing programs that automatically enforce agreements between untrusted parties [17, 18]. Initially intended to automate legal contracts, they have evolved with the emergence of blockchain technology, becoming scripts that execute consistently across all nodes in a distributed ledger [19]. Developers write smart contracts in Solidity [20] or Vyper [21], two languages explicitly designed for this purpose.

In Ethereum, smart contracts execute within the Ethereum Virtual Machine (EVM), a Turing-complete, stack-based virtual machine that ensures the isolation of contract code. Each smart contract has a unique address, its own private storage, an Ether balance, and executable code. When a transaction is sent to a contract address, it activates the contract’s functions, using the provided input data and consuming gas to cover computation costs [18]. Once deployed on the blockchain, these contracts become immutable and tamper-proof [22, 23], automatically executing predefined conditions to ensure trust and reliability. Nevertheless, immutability, while ensuring transparency and trust, makes code defects permanent [23], making smart contracts susceptible to a range of security vulnerabilities stemming from coding errors and software development bugs. Among the most critical are the following:

• Reentrancy (SWC-107) [24, 25, 26, 27] is one of the most dangerous vulnerabilities, famously exploited in the DAO hack [28, 29]. It occurs when a contract invokes an external contract before updating its own state, thereby enabling recursive calls that can drain funds or disrupt the intended execution flow.

• Timestamp Dependency (SWC-116) [24, 25, 26, 27] occurs when a contract relies on the block.timestamp value to make critical decisions, such as triggering payouts, closing auctions, or generating randomness. Since miners have limited control over the block timestamp within a reasonable range, it can be exploited to manipulate contract behavior [30, 31], especially in scenarios with small time windows or financial incentives.

• Integer Overflow and Underflow (SWC-101) [24, 25, 26, 27] occur when arithmetic operations on integer variables exceed their maximum or minimum representable limits. These bugs can lead to unexpected behavior, including bypassing balance checks or incorrectly locking or unlocking tokens. Starting with Solidity 0.8.0, the compiler automatically reverts on overflow or underflow, helping prevent such vulnerabilities by default.

• Transaction-Ordering Dependence (TOD) creates unpredictability in transaction execution order, which attackers can exploit through front-running to alter the outcome of a pending transaction [32, 33, 34].

• Denial of Service (DoS) vulnerabilities can arise from excessive gas usage in external calls or from loops that consume excessive computation, rendering a contract temporarily or permanently inoperable [35].

Large Language Models and their Architecture

Large Language Models (LLMs) are advanced AI systems that can understand and generate human-like text [36] and generalize across a wide range of tasks [37, 38, 39]. They are trained on vast amounts of text data, and use deep learning with billions, sometimes trillions, of parameters to learn patterns in language. Three properties usually describe an LLM: the number of parameters, which reflects how much the model has learned to store; the context window, which bounds how much text it can consider at once; and the architecture, which determines how each token is actually processed inside the network. Below, we present a non-exhaustive taxonomy of the main architectures of modern LLMs that we employ in the study. Two aspects govern what happens when a token enters a model: (i) which parameters are activated, and (ii) which other tokens it can attend to.

• In dense Transformers, all parameters participate in processing every token, and self-attention allows each token to interact with any other token in the input. This design underlies models such as GPT-4 [40], Gemini [41], LLaMA [42], and Claude [43], explaining both their expressive power and their computational cost, which scales quadratically with input length.

• Mixture of Experts (MoE) [44] models divide their parameters into specialized subnetworks known as experts, in contrast to dense Transformers, and only activate a small subset of these subnetworks for each input token. While keeping a high total parameter count, this selective activation lowers effective compute per token. As a result, two MoE models of similar scale may behave quite

differently, as performance depends more on routing policies and expert specialization than on raw size.

• RNN-like architectures–such as RWKV [45]–do not include an attention mechanism. The input history is compressed into a compact recurrent state, which is updated at each token-processing stage. For long sequences, this yields linear-time processing, but at the expense of less direct access to earlier tokens, which are available only as long as they remain in the state.

• State Space Models (SSMs) occupy a middle ground between RNNs and Transformers, and are an active research direction [46], [47], [48]. They do not compute attention over all previous tokens but maintain a fixed-sized hidden state that is updated recurrently as each token is processed, serving as a compressed memory of the sequence seen so far. Unlike attention-based models, the model has no direct access to earlier tokens, and therefore it can handle long sequences more efficiently [49].

Beyond these architectures, the LLM landscape includes further variants, encoder-only Transformers [50], diffusion-based language models [51], sub-quadratic attention schemes [52], and reasoning models [53], [54] that incorporate multi-step deliberation directly into the generation process.

Related Work

Boi et al. [55] propose an approach for smart contract vulnerability detection by fine-tuning relatively lightweight LLMs, specifically Llama-2-7b-chat-hf and GPT-2-XL, on a curated dataset of known vulnerabilities. The fine-tuning process adapts these models to identify specific vulnerable code segments and provide targeted remediation strategies, such as recommending the SafeMath library for arithmetic issues. The fine-tuned Llama-2-7b-chat-hf achieved 82% accuracy in identifying problematic contracts, significantly outperforming GPT-2-XL’s 62% on the same test set.

Ding et al. [56] propose SmartGuard, an LLM-enhanced framework for smart contract vulnerability detection that combines semantic code retrieval, Chain-of-Thought (CoT) prompting, and in-context learning (ICL). SmartGuard retrieves the top-k code snippets most semantically similar to a given test function using CodeBERT embeddings, generates corresponding reasoning chains through a self-check architecture that leverages LLMs’ self-correcting capabilities, and assembles them into a structured prompt for vulnerability inference. The self-check mechanism iteratively prompts the model to revise incorrect predictions and, when the rethinking limit is exceeded, guides it to construct a reasoning chain from a known correct label, thereby maximizing the yield of high-quality CoTs from limited labeled data. Evaluated on the SolidiFI benchmark using GPT-3.5 Turbo, ChatGLM3-6B, and Qwen-7B, SmartGuard outperforms classical static analysis tools such as Slither, Securify, and Mythril.

Wei et al. [57] propose LLM-SmartAudit, a multi-agent conversational framework designed to automate smart contract security auditing using LLMs. Rather than relying on a single model query, the system orchestrates a team of specialized agents, a Project Manager, a Smart Contract Counselor, an Auditor, and a Solidity Programming Expert, which alternate between assistant and user roles, collaborating through structured conversations to analyze contracts, identify vulnerabilities, and produce comprehensive audit reports. The framework operates in two complementary modes: a Broad Analysis (BA) mode, based on a Thought-Reasoning prompt, and a Targeted Analysis (TA) mode, based on a Buffer-Reasoning prompt. Evaluated against ten traditional tools on a labeled dataset of 110 annotated contracts spanning ten vulnerability categories, LLM-SmartAudit in BA mode using GPT-3.5-turbo outperforms all baselines. On a real-world dataset of 6,454 contracts drawn from Code4rena audit contests, TA mode further achieves a recall of 47.55% on complex logic vulnerabilities, a category where traditional tools score zero.

Research Gap

While the aforementioned works advance smart contract vulnerability detection through fine-tuning [55], retrieval-augmented prompting [56], and multi-agent frameworks [57], they share a common limitation: none systematically investigates whether behavioral findings observed for a specific LLM remain consistent across changes in the underlying architecture. Boi et al. [55] focus on comparing individual model performance within a fine-tuning paradigm without examining cross-architectural generalization. Despite evaluating across multiple model families, SmartGuard [56] attributes its performance gains to the design of its retrieval-augmented pipeline rather than to the intrinsic transferability of vulnerability-reasoning capabilities across architectures. Similarly, LLM-SmartAudit [57], while providing evidence that performance varies across prompting strategies and GPT model generations, remains confined to the GPT family of architectures. Our work addresses this gap by systematically replicating and extending the findings of Chen et al. [13] across heterogeneous LLM architectures, without altering the detection methodology, to determine whether the reported behavioral patterns are model-specific or architecture-agnostic.

Reference Work

Chen et al. [13] conducted an empirical study to investigate the effectiveness of GPT models in detecting vulnerabilities in Solidity smart contracts, evaluating three GPT model variants on the widely used smartbugs-curated dataset [14].

The authors interacted with the GPT models via the official OpenAI API [58], targeting three specific model snapshots: gpt-3.5-turbo-0613 [59], gpt-4-2024-05-13 [60], and gpt-4-0613 [61]. The temperature parameter was set to 0.7 across all experiments to balance creativity and stability in text generation, while all remaining parameters were kept at their default values (e.g., TopP = 1, FrequencyPenalty = 0). A central methodological contribution of the study is its prompt design process. Models based on GPT architecture rely on a prompt-based learning approach [62]. Rather than manually crafting a prompt, the authors employed GPT-4 itself as a prompt optimizer [63]. Starting from a naive task description, they fed the initial prompt into the model and iteratively refined it. Multiple draft prompts were generated and manually evaluated on a sample of 20 contracts drawn from the smartbugs-curated dataset. This process converged on the following general-purpose template:

Prompt Template: You are [ROLE]. [TASK DESCRIPTION]. Think step by step, carefully. The input is [INPUT].

Two distinct roles were instantiated within this template. The first, designated as a smart contract vulnerability detector, was used to perform the actual detection task. The prompt was filled as follows:

Detection Prompt: You are a vulnerability detector for a smart contract. Here are nine common vulnerabilities. [VULS]. Check the following smart contract for the above vulnerabilities. Think step by step, carefully. The input is [INPUT].

Placeholder [VULS] was replaced with the names and aliases of the nine DASP Top 10 vulnerability categories, and [INPUT] with the source code of the contract under analysis. The second role, designated as text semantic analyzer, was invoked in a separate conversation session to normalize the free-form output of the first step into a structured binary format ([VUL]: 0/1). This two-step pipeline was motivated by an empirical finding reported by the authors: directly constraining the output format within the detection prompt degraded performance across most vulnerability types (Finding 1). The semantic analysis prompt took the following form:

Formatting Prompt: You are a semantic analyzer of text. Here are nine common vulnerabilities. [VULS]. The following text is a vulnerability detection result for a smart contract. Use 0 or 1 to indicate whether there are specific types of vulnerabilities. For example: ”Reentrancy: 1”. Think step by step, carefully. The input is [INPUT].

To validate the reliability of the semantic analysis step, the authors manually inspected the normalized output for 50 sampled contracts. Of these, 47 were entirely consistent with the original detection output, 2 exhibited omissions, and 1 showed hallucinations, confirming the overall soundness of the approach. Finally, to account for the known non-determinism of GPT models, each experiment was repeated five times on the full dataset, and the standard deviation of the F1 score across runs was reported alongside precision, recall, specificity, and F1.

Research Method

The empirical study by Chen et al. [13] relies on three specific GPT model snapshots scheduled for deprecation by OpenAI and scheduled for shutdown in 2026 [15], raising an important concern about the longevity and generalizability of their findings. If the results are tightly coupled to a specific architectural family that is becoming unavailable, their practical applicability is inherently limited. We therefore investigate whether the reported findings hold across different LLM architectural families or are an artifact of the GPT-4 architecture. Figure 1 shows the research method we followed. Specifically, the exact replication is represented on the left, while the differentiated replication is depicted on the right.

Vector (PDF)

To achieve our goal, we formulated the following research questions.

RQ$_1$. Can the results reported in the original study be replicated using the same prompt, dataset, and method?

$RQ_1$ establishes the baseline of the replication. It verifies that the original findings are stable and reproducible under controlled conditions with the same prompt, dataset, and metrics, before applying changes, thereby revealing discrepancies arising from model versioning, non-determinism, or methodological ambiguity in the original study.

RQ$_2$. Do the findings of Chen et al. hold across different LLM architectural families?

$RQ_2$ assesses transfer across selected models representing different architectural families without estimating the isolated causal effect of architecture. The evaluation conducted by Chen et al. [13] relied on GPT-4, a dense Transformer model. $RQ_2$ investigates whether their main findings, namely, high recall at the cost of low precision and substantial performance variation across vulnerability types, hold across different LLM architectural families.

Dataset

As in the original study [13], we used the SmartBugs curated dataset [14], which comprises 142 vulnerable smart contracts across nine vulnerability types described in the DASP Top 10 [64]. Smart contracts are annotated with a comment that lists the line numbers at which vulnerabilities are reported, along with their types. As in the original paper, we removed all comments.

Execution and Analysis

All experiments were conducted using the same prompts as in the reference paper and were repeated five times to mitigate model instability. To evaluate the detection results, as in the previous paper [13], we used precision, recall, F1 score, and specificity. We also calculated the standard deviation of the F1-score to quantify performance variability and evaluate the consistency of the results. To make the recall–false-positive trade-off easier to read, we used Youden’s $J$ statistic [65], defined as $J = \mathrm{TPR} - \mathrm{FPR}$, or equivalently $J = \mathrm{Sensitivity} + \mathrm{Specificity} - 1$. Since Recall, TPR, and Sensitivity coincide in this setting, $J$ summarizes in a single number how much a model improves over random guessing while balancing its ability to recover true positives against its tendency to raise false alarms. Higher values indicate a better trade-off, values near zero indicate weak discrimination, and negative values mean that false-positive behavior outweighs true-positive recovery.

Vulnerability	Chen et al. Results						Our Results
	Precision	Recall	Specificity	F1-score	F1 Std.		Precision	Recall	Specificity	F1-score	F1 Std.	Youden’s J
Access Control	0.17	0.96	0.30	0.28	0.04		0.17	0.98	0.33	0.30	0.01	0.31
Arithmetic	0.22	0.96	0.63	0.36	0.03		0.24	0.92	0.66	0.38	0.03	0.58
Bad Randomness	0.40	0.98	0.91	0.57	0.04		0.46	0.95	0.94	0.63	0.03	0.88
Denial of Service	0.07	0.97	0.42	0.13	0.02		0.07	1.00	0.38	0.12	0.01	0.38
Front Running	0.09	0.70	0.81	0.16	0.01		0.12	0.50	0.88	0.19	0.09	0.38
Reentrancy	0.34	0.97	0.41	0.50	0.03		0.40	0.97	0.57	0.56	0.01	0.54
Short Addresses	0.05	0.60	0.91	0.08	0.00		0.06	0.80	0.91	0.11	0.06	0.71
Time Manipulation	0.17	0.88	0.83	0.28	0.04		0.12	1.00	0.72	0.21	0.01	0.72
Unchecked Calls	0.54	0.93	0.68	0.68	0.05		0.49	0.95	0.42	0.65	0.01	0.37
Macro Average	0.23	0.88	0.66	0.34	0.03		0.24	0.90	0.65	0.35	0.03	0.54
Micro Average	–	–	–	–	–		0.26	0.93	0.68	0.41	–	0.61

Data Availability

The datasets and results of this paper are publicly available at https://figshare.com/s/bbb79bd8002deb1d40ac.

RQ1 - Exact Replication

RQ1 Design

The reference study [13] showed that the GPT-4 model [61] has the best performance compared to GPT-3.5-turbo and GPT-4o. To answer $RQ_{1}$, we repeated the experiment using the same model, GPT-4-0613, with the same settings: a temperature of 0.70 and all other parameters at their default values.

Similarly, we processed the output with a second prompt rather than enforcing strict formatting in the initial response, given the limitations of GPT in consistently adhering to such formats [13]. This two-step approach leverages the semantic understanding of the model, improves its reliability, ensures standardized output (e.g., “[VUL]: 0/1”), and reduces the risk of formatting errors during detection [13]. We relied on the information provided in the paper and its replication package.

RQ1 Results

The side-by-side comparison of our replication and the results reported by Chen et al. [13] provides insights into both the stability and the variability of GPT-4’s performance on the smartbugs-curated dataset. Table I shows both results. At a high level, the two experiments confirm a similar trend: GPT-4 achieves consistently high recall across most vulnerability classes but exhibits relatively low precision, resulting in modest F1 scores. Although the dataset, prompt, model snapshot, and reported decoding configuration were kept consistent with the reference study, our replication highlights several important differences, suggesting that residual differences may arise from inference non-determinism, undocumented provider-side implementation details, API behavior, or ambiguities in the original execution and post-processing procedures.

First, recall is robust across both studies, often close to or at 1.00 for classes such as denial-of-service and time manipulation, indicating that GPT-4 systematically favors sensitivity over selectivity, detecting almost all true vulnerabilities at the cost of many false positives. Interestingly, while the original study reported slightly lower recall in denial-of-service (0.97 vs. 1.00 in our results), our replication shows higher recall for time manipulation (1.00 vs. 0.88). Such discrepancies indicate that class-level estimates remain sensitive to stochastic generation and potentially undocumented execution details, even when the published protocol is closely reproduced.

Second, precision exhibits a more pronounced divergence between the two studies. For example, in reentrancy, our results yield higher precision (0.40 vs. 0.34), while in unchecked low-level calls, the original study reports stronger precision (0.54 vs. 0.49). Similarly, in bad randomness, our replication produces higher F1-scores (0.63 vs. 0.57). These shifts illustrate that, although the model’s overall tendency is consistent, its reliability in identifying specific vulnerability classes is variable, even after five rounds of execution.

Third, specificity values underscore an important trade-off. In some categories (e.g., short addresses), both studies agree on relatively high specificity. However, for others, the results diverge, i.e., front running shows lower specificity in the original (0.81) compared to our replication (0.88). These differences suggest that GPT-4’s balance between FPs and false negatives is unstable across replications.

Fourth, the FPR values reported in our replication further confirm GPT-4’s tendency to over-predict vulnerabilities. The macro- and micro-averages of 0.36 and 0.32, respectively, indicate that approximately one in three non-vulnerable contracts is incorrectly flagged. This behavior is particularly pronounced for access control (FPR = 0.67) and denial-of-service (FPR = 0.62), where the model raises a false alarm in most negative cases. Similarly, unchecked low-level calls (FPR = 0.58) and reentrancy (FPR = 0.43) exhibit higher FPRs, suggesting that GPT-4 struggles to distinguish vulnerable patterns from benign ones in these categories. In contrast, bad randomness (FPR = 0.07) and short addresses (FPR = 0.09) stand out as the classes in which the model produces the fewest false alarms, likely reflecting more pronounced syntactic or semantic patterns. These FPR disparities align with the low precision observed across most categories and reinforce the concern that GPT-4, while rarely missing a true vulnerability, does so at the expense of a high false-positive rate.

Looking at the macro averages, both sets of results are closely aligned: precision remains at 0.23–0.24, recall between 0.88–0.90, and F1-score at 0.34–0.35. This consistency indicates that, although class-level variations exist, the aggregate performance of GPT-4 is relatively reproducible.

RQ1 Summary. Our replication broadly validates the conclusions of the reference study but highlights significant variability across vulnerability classes. GPT-4 achieves high recall, making it suitable when missing vulnerabilities are unacceptable. Nevertheless, its low and unstable precision, coupled with a macro-average FPR of 0.36, raises concerns about the number of FPs, thereby hindering the practical applicability of the method.

Architectural Family	Model	Total	Active	Context
Dense Transformer	Gemma 3 12B	12B	12B	128K
Mixture of Experts	Qwen 3.5 35B a3b	35B	3B	262K
Mixture of Experts	DeepSeek-V2-Lite	16B	2.4B	32K
RNN-like	RWKV-7 13.3B	14B	14B	1M

RQ2 - Differentiated Replication

RQ2 Design

Table II reports the LLM architectural families and the selected representative model.

The dominant architecture in modern LLMs is the dense Transformer decoder, which underpins GPT-4. To assess whether the findings of Chen et al. are attributable to this architectural choice or reflect a more general behavior, we selected one representative model per major architectural family, keeping the parameter count comparable across models where possible to avoid conflating architectural effects with scale effects. Reasoning models were excluded from this analysis, because the original prompt (see RQ1) already incorporates a Chain-of-Thought pattern, and applying it to a reasoning model would conflate the effect of the prompt pattern with the model’s internal mechanism, making it impossible to attribute observed performance differences to either factor in isolation.

Gemma 3 12B was selected as the dense Transformer reference point because it belongs to the same architectural family as GPT-4. Qwen 3.5 35B a3b represents the MoE family. We include a second MoE representative, DeepSeek-V2-Lite 16B, whose total parameter count is approximately 16 billion with roughly 2.4 billion active per token. The reason for including two MoE models is that Qwen 3.5 and DeepSeek-V2-Lite come from different development lineages and differ in their expert routing strategies, allowing us to distinguish behavior that is attributable to the MoE mechanism from behavior that is vendor-specific.

Finally, RWKV-7 14B is an RNN-like architecture and is the most structurally distant model from GPT-4 in our selection.

The rationale for this selection is architectural diversity rather than performance maximization. We are not asking which model performs best, nor whether the prompt transfers effectively. We are asking whether the empirical pattern reported by Chen et al., namely, high recall coupled with low precision, emerges consistently regardless of how the underlying model processes sequences, or whether it is contingent on the Transformer attention mechanism.

The two-step prompting process is maintained to ensure consistent output formatting, and all models are run at a temperature of 0.70, as in RQ1.

We also included SSMs in our preliminary evaluation. Specifically, we evaluated Bamba 9B, a pure SSM, and Jamba 2 3B, a hybrid SSM-Transformer model that interleaves standard attention layers with state-space layers, situating architecturally between the two families (i.e., SSMs and Transformers). However, both models exhibited a systematic failure mode on the detection task. Rather than producing the expected vulnerability assessment, generation entered repetitive loops that consumed the available context window without converging to a terminating output. Consequently, the outputs could not be parsed by the formatting prompt pipeline used in RQ1 and RQ2, and the metrics could not be computed, indicating not poor detection performance but rather that the models did not produce well-formed outputs for which precision, recall, or specificity could be defined. This failure mode is consistent with reported limitations of current SSM checkpoints in long-form instruction following [66], [49]. Therefore, we report it as a negative observation rather than a quantitative comparison, and leave to future work a focused investigation of whether this failure is intrinsic to the architecture or an artifact of the current generation of instruction-tuned checkpoints.

RQ2 Results

Table III shows that findings provided by Chen et al. [13] do not uniformly generalize across different LLM architectural families. However, the degree of divergence varies substantially across architectures.

The Qwen-3.5 35B MoE model comes closest to replicating the baseline behavior, achieving a macro F1 of 0.33 and a micro F1 of 0.40, with a macro/micro FPR of 0.33, all comparable to our replication (macro F1 0.35, micro F1 0.41, macro FPR 0.36, micro FPR 0.32). It also attains a macro Youden’s J of 0.42, closer to the GPT-4 replication (0.54) than any other non-dense model, suggesting that some MoE models can approximate the dense-Transformer detection p. Still, the family does not do so uniformly.

In contrast, the Dense Transformer (Gemma-3 12B) maintains high recall (macro 0.83), consistent with the tendency of the baseline to flag aggressively, but at the cost of a substantially higher FPR (0.55 vs 0.36), indicating a more indiscriminate detection strategy that inflates false positives beyond what was observed in the original study. In particular, we observe the same qualitative pattern in the two dense Transformer LLMs, which differ by a factor of two orders of magnitude in scale. Two data points are insufficient to support any claims of scale-invariance. Still, the consistency suggests that the high-recall/low-precision profile is not due solely to scale within this family of architectures.

The most pronounced deviations occur with the DeepSeek-V2-Lite and RWKV7 models, both of which break the high-recall, low-precision pattern observed in Chen et al.’s results. These models achieve much better specificity (0.81 and 0.85, respectively) and lower FPRs (0.19 and 0.15), but at the expense of recall, which drops dramatically to 0.29 and 0.30 — less than a third of the baseline’s 0.90. Their macro Youden’s J values (0.10 and 0.15) are also far below those of GPT-4 and Qwen-3.5, making the trade-off evident in a single statistic. Across all models, certain vulnerability classes, such as Unchecked Low-Level Calls and Bad Randomness, remain relatively easier to detect. At the same time, Front-Running and Short Addresses consistently prove challenging regardless of architecture. Overall, the evidence suggests that the high-recall detection behavior reported by Chen et al. is not an inherent property of LLM-based vulnerability detection, but rather is sensitive to architectural choices, and practitioners should be cautious about assuming these findings transfer to smaller or non-transformer-based models.

The message is supported by the ROC-style scatter plot shown in Figure 2, showing that the models do not differ only in magnitude on a single metric but occupy qualitatively different operating regions. GPT-4 and Qwen occupy the upper portion of the plane, indicating that both recover a large fraction of true vulnerabilities. Still, Qwen sits slightly further to the left, indicating a somewhat better false-positive profile. Gemma-3 remains in the same high-TPR region as GPT-4 but is markedly displaced to the right, indicating its more aggressive over-flagging behavior. By contrast, DeepSeek-V2-Lite and RWKV-7 move toward the lower-left region: they reduce false alarms, but only at the cost of a substantial reduction in true-positive recovery.

The last column of Table III makes the pattern even clearer. At the macro level, GPT-4 remains the strongest reference point with a Youden’s J of 0.54, followed by Qwen-3.5 at 0.42. Gemma-3 reaches only 0.28: its high recall is partially offset by a much larger false-positive burden, which is precisely why its behavior should be read as aggressive but less selective. DeepSeek-V2-Lite and RWKV-7 fall further to 0.10 and 0.15, respectively, confirming that their higher specificity comes at the cost of a sharply reduced true-positive recovery. Looking at the per-vulnerability rows, Qwen is closest to the baseline on categories such as Arithmetic, Bad Randomness, Time Manipulation, and Unchecked Calls. In contrast, DeepSeek and RWKV rarely match that balance and even become negative in isolated cases, such as Front Running or Short Addresses, where false positives outweigh true positives. In this sense, Youden’s J provides a compact summary of the same message conveyed by the separate recall and specificity columns: the dense baseline and Qwen occupy a relatively effective operating region, Gemma shifts toward over-flagging, and DeepSeek-V2-Lite and RWKV-7 shift toward under-detection.

Architecture	Vulnerability	Precision	Recall	Specificity	F1-score	F1 Std.	Youden’s J
Dense Transformer (Gemma-3 12B)	Access Control	0.15	0.92	0.25	0.26	0.02	0.18
Dense Transformer (Gemma-3 12B)	Arithmetic	0.16	0.98	0.41	0.28	0.01	0.39
Dense Transformer (Gemma-3 12B)	Bad Randomness	0.42	0.88	0.93	0.57	0.01	0.81
Dense Transformer (Gemma-3 12B)	Denial of Service	0.05	1.00	0.09	0.09	0.00	0.09
Dense Transformer (Gemma-3 12B)	Front Running	0.03	0.75	0.30	0.06	0.02	0.05
Dense Transformer (Gemma-3 12B)	Reentrancy	0.28	1.00	0.25	0.44	0.00	0.25
Dense Transformer (Gemma-3 12B)	Short Addresses	0.00	0.00	0.80	0.00	0.00	-0.20
Dense Transformer (Gemma-3 12B)	Time Manipulation	0.10	0.93	0.70	0.19	0.02	0.63
Dense Transformer (Gemma-3 12B)	Unchecked Calls	0.46	0.99	0.31	0.63	0.02	0.30
Dense Transformer (Gemma-3 12B)	Macro Average	0.18	0.83	0.45	0.28	0.01	0.28
Dense Transformer (Gemma-3 12B)	Micro Average	0.18	0.96	0.46	0.31	—	0.42
Mixture-of-Experts (Qwen-3.5 35B a3b)	Access Control	0.16	0.62	0.51	0.25	0.03	0.13
Mixture-of-Experts (Qwen-3.5 35B a3b)	Arithmetic	0.23	0.92	0.65	0.38	0.03	0.57
Mixture-of-Experts (Qwen-3.5 35B a3b)	Bad Randomness	0.41	0.95	0.92	0.57	0.04	0.87
Mixture-of-Experts (Qwen-3.5 35B a3b)	Denial of Service	0.06	0.93	0.39	0.12	0.01	0.32
Mixture-of-Experts (Qwen-3.5 35B a3b)	Front Running	0.03	0.35	0.68	0.06	0.04	0.03
Mixture-of-Experts (Qwen-3.5 35B a3b)	Reentrancy	0.34	0.99	0.45	0.51	0.01	0.44
Mixture-of-Experts (Qwen-3.5 35B a3b)	Short Addresses	0.02	0.20	0.88	0.04	0.09	0.08
Mixture-of-Experts (Qwen-3.5 35B a3b)	Time Manipulation	0.15	0.84	0.82	0.25	0.03	0.66
Mixture-of-Experts (Qwen-3.5 35B a3b)	Unchecked Calls	0.67	0.99	0.71	0.80	0.03	0.70
Mixture-of-Experts (Qwen-3.5 35B a3b)	Macro Average	0.23	0.75	0.67	0.33	0.03	0.42
Mixture-of-Experts (Qwen-3.5 35B a3b)	Micro Average	0.26	0.90	0.67	0.40	—	0.57
Mixture-of-Experts (DeepSeek-V2-Lite 16B)	Access Control	0.17	0.69	0.50	0.27	0.04	0.19
Mixture-of-Experts (DeepSeek-V2-Lite 16B)	Arithmetic	0.11	0.13	0.87	0.12	0.08	0.01
Mixture-of-Experts (DeepSeek-V2-Lite 16B)	Bad Randomness	0.14	0.40	0.86	0.21	0.08	0.26
Mixture-of-Experts (DeepSeek-V2-Lite 16B)	Denial of Service	0.08	0.17	0.90	0.11	0.11	0.07
Mixture-of-Experts (DeepSeek-V2-Lite 16B)	Front Running	0.02	0.05	0.92	0.02	0.05	-0.03
Mixture-of-Experts (DeepSeek-V2-Lite 16B)	Reentrancy	0.31	0.51	0.66	0.38	0.05	0.18
Mixture-of-Experts (DeepSeek-V2-Lite 16B)	Short Addresses	0.00	0.00	0.90	0.00	0.00	-0.10
Mixture-of-Experts (DeepSeek-V2-Lite 16B)	Time Manipulation	0.16	0.36	0.93	0.22	0.07	0.29
Mixture-of-Experts (DeepSeek-V2-Lite 16B)	Unchecked Calls	0.42	0.31	0.75	0.36	0.08	0.06
Mixture-of-Experts (DeepSeek-V2-Lite 16B)	Macro Average	0.15	0.29	0.81	0.19	0.06	0.10
Mixture-of-Experts (DeepSeek-V2-Lite 16B)	Micro Average	0.21	0.38	0.82	0.27	—	0.20
RNN-like (RWKV7 13.3B)	Access Control	0.14	0.25	0.78	0.18	0.10	0.04
RNN-like (RWKV7 13.3B)	Arithmetic	0.28	0.40	0.88	0.33	0.07	0.28
RNN-like (RWKV7 13.3B)	Bad Randomness	0.27	0.33	0.95	0.29	0.10	0.28
RNN-like (RWKV7 13.3B)	Denial of Service	0.06	0.27	0.81	0.10	0.07	0.08
RNN-like (RWKV7 13.3B)	Front Running	0.07	0.10	0.96	0.08	0.11	0.06
RNN-like (RWKV7 13.3B)	Reentrancy	0.33	0.69	0.58	0.45	0.04	0.27
RNN-like (RWKV7 13.3B)	Short Addresses	0.07	0.20	0.97	0.10	0.22	0.17
RNN-like (RWKV7 13.3B)	Time Manipulation	0.05	0.12	0.94	0.07	0.11	0.06
RNN-like (RWKV7 13.3B)	Unchecked Calls	0.48	0.36	0.77	0.41	0.09	0.13
RNN-like (RWKV7 13.3B)	Macro Average	0.19	0.30	0.85	0.22	0.10	0.15
RNN-like (RWKV7 13.3B)	Micro Average	0.26	0.40	0.86	0.32	—	0.26

Discussion

The section discusses the implications of our findings for the research questions.

Architecture, Scale, and Active Parameter Count

A potentially important alternative explanation for our findings is scale rather than architecture. GPT-4 is vastly larger than the open models considered here, so a skeptical reader could argue that the high-recall/low-precision profile is simply what happens at sufficiently large scale, regardless of architecture. Our data do not fully rule out that possibility, but they do weaken it in one important respect. Despite being far smaller than GPT-4, Gemma-3 12B exhibits the same qualitative dense-Transformer pattern: high recall accompanied by high false-positive rates. Two data points are not enough to claim scale invariance, but they do suggest that this detection profile is not solely attributable to the dense Transformer family. The active parameter count provides additional context for interpreting these differences. RWKV-7 activates approximately 14B parameters, a budget comparable to Gemma-3’s 12B, yet exhibits a markedly different detection profile. This comparison weakens a simple active-capacity explanation for RWKV-7, although it does not isolate architecture from differences in training data, instruction tuning, and alignment. By contrast, DeepSeek-V2-Lite activates only about 2.4B parameters per token, so limited active capacity remains a plausible contributor to its substantially lower recall.

MoE Architecture: Same Family, Two Philosophies

A notable takeaway from comparing Qwen 3.5 35B a3b and DeepSeek-V2-Lite 16B is not merely that they perform differently, but that they occupy clearly different regions of the recall–false-positive trade-off despite sharing the same underlying architectural principle of sparse expert activation.

Qwen 3.5 35B a3b exhibits a consistently aggressive detection posture. Across almost every vulnerability category, its recall is notably high, indicating that it tends to flag a contract as vulnerable rather than clean. This behavior is observed across structurally distinct vulnerability types, from Reentrancy to Bad Randomness, placing Qwen relatively close to GPT-4 in ROC-style terms: high TPR paired with a comparatively elevated FPR, and a macro Youden’s J of 0.42.

DeepSeek-V2-Lite 16B exhibits the opposite tendency: it is considerably more conservative, and in some categories it is almost entirely silent. Empirically, the model combines low recall with low false-positive rates, which places it in a markedly different operating region from both Qwen and GPT-4. Its macro Youden’s J of 0.10 confirms that this is not simply a cleaner version of the same behavior, but a weaker discriminative profile overall.

As shown also in Figure 2, both models share the MoE mechanism but differ markedly in detection behavior. Qwen lies near GPT-4 in the upper-middle region of the plane, whereas DeepSeek-V2-Lite moves toward the lower-left region, making the intra-family divergence immediately apparent. Architectural family membership alone is therefore not sufficient to predict the detection profile. Differences in lineage, routing design, active parameter budget, and training signal are all plausible contributors, but our study does not isolate them individually. We can support the narrower empirical claim that two MoE models can behave qualitatively differently under the same prompting and evaluation protocol. In a security-auditing context, neither posture is inherently correct because the preferred balance depends on the cost asymmetry between false alarms and missed vulnerabilities.

Detection Scope Limitations of RNN-like Architectures.

RWKV-7 occupies a structurally distinct position in this comparison, as it is the only model that processes tokens sequentially via gated recurrent updates without any attention. This architectural distinction has observable consequences. Unlike every other model in the study, whether dense or sparse, RWKV-7 cannot directly relate distant tokens to one another. Instead, it compresses the preceding context into a fixed-size recurrent state at each step, meaning that information from earlier in a contract is subject to progressive decay and interference. For smart contract vulnerability detection, where meaningful signals often emerge from non-local interactions between a state variable and a function body defined hundreds of tokens apart, this is a structurally significant limitation.

The resulting detection pattern is consistently conservative. Across nearly all vulnerability categories, RWKV-7 exhibits high specificity paired with low recall, suggesting that the model rarely accumulates sufficient contextual evidence to commit to a positive detection. This behavior contrasts with Gemma-3, which aggressively flags vulnerability-suggestive patterns using global attention, sometimes indiscriminately. RWKV-7 does not overclaim; it largely remains silent but this conservatism is relaxed for categories such as Reentrancy and Unchecked Low Level Calls, where the problematic pattern tends to be locally concentrated within a single function body, consistent with what one would expect from a recurrent architecture: short-range dependencies are preserved reasonably well, while cross-function reasoning suffers.

Therefore, the inclusion of RWKV-7 in this study is analytically valuable beyond its raw performance. It serves as a structural control, isolating the attention mechanism’s contribution from confounding factors such as model scale or training data. The performance gap between RWKV-7 and the attention-based models is most plausibly attributable to inductive bias rather than capacity. For tasks that demand long-range relational reasoning over structured code, attention is not merely beneficial but potentially necessary. That said, the observed scope limitations do not preclude RWKV-7 from being effective in narrower settings; function-level vulnerability detection, or any formulation in which the relevant context is compact enough to be faithfully captured within the recurrent state, may be a more natural fit for this architectural family.

Threats to Validity

We discuss threats to validity following Wohlin et al. [67].

Internal Validity: A primary threat to internal validity concerns model non-determinism. Despite repeating each experiment five times and reporting the standard deviation of the F1-score, the stochastic nature of LLM inference at a temperature of 0.70 means that observed performance differences between architectures may partly reflect run-to-run variability rather than stable behavioral properties of the underlying model. Although our replication design mirrors the protocol of Chen et al. [13], we cannot fully exclude that some class-level divergences are artifacts of sampling noise rather than architectural signals.

A second threat concerns the two-step prompting pipeline. While we adopted the same formatting prompt used in the reference study to normalize model outputs, different architectures may exhibit heterogeneous tendencies to deviate from the expected output schema, leading to higher rates of parsing failures or silent omissions for some models than for others. Although this design choice was motivated by the need for methodological consistency with Chen et al. [13], it introduces a confound: differences in adherence to structured outputs across architectures may distort detection metrics independently of vulnerability reasoning capability.

A further threat relates to parameter comparability across models. Despite our effort to select models of comparable scale, the total and active parameter counts across our selected architectures are not perfectly aligned. Although we explicitly account for this limitation in our design rationale, we cannot fully disentangle the effects of architectural family and model capacity on observed performance.

Finally, we did not independently revalidate the semantic normalization stage for each model. Hence, the observed variations may partially originate from model-specific interactions with the second formatting prompt rather than from the initial vulnerability assessment only. Future replications should report normalization agreement separately for each model.

External Validity: The most significant threat to external validity is the limited scope of the benchmark dataset. Our study, like the reference work, relies exclusively on the SmartBugs curated dataset [14], which comprises 142 annotated contracts spanning nine DASP Top 10 vulnerability categories. While this dataset is widely used in the literature and enables direct comparison with Chen et al. [13], its limited size constrains the generalizability of our findings. Results may not generalize to synthetic datasets [68] or to real-world audit scenarios involving more diverse, complex, or more recent contract codebases.

A further threat concerns possible benchmark contamination. SmartBugs-curated is a public and widely used benchmark that predates the evaluated models. Some contracts, annotations, or derivative discussions may have been included in pretraining or instruction-tuning data. Removing comments from the source code prevents direct label leakage during inference, but it does not preclude prior exposure to the benchmark, which may differ across model providers and families, potentially contributing to the observed differences. Since the training corpora of most evaluated models are not fully disclosed, we cannot quantify or rule out this threat.

Construct Validity: A related concern involves the quality and consistency of ground-truth annotations in the SmartBugs curated dataset. The dataset relies on developer-provided comments to identify vulnerability locations and types. If these annotations contain errors, omissions, or ambiguities, the resulting metrics may misrepresent the model’s actual performance. We adopted the dataset as-is to preserve comparability with Chen et al. [13], but this design choice necessarily inherits any annotation noise present in the original benchmark.

Moreover, excluding reasoning models from our architectural comparison, while methodologically justified to avoid confounding the Chain-of-Thought prompt pattern with the model’s internal reasoning mechanism, limits the construct coverage of our study. Reasoning-oriented architectures are an increasingly prominent paradigm in the LLM landscape, and excluding them means that our findings speak only to a subset of the architectural space relevant to practitioners.

Conclusion Validity: A primary threat to conclusion validity is the statistical power of our experimental design. With only five repetitions per experiment and a relatively small dataset of 142 contracts, the confidence intervals around our performance estimates are wide, particularly for vulnerability classes with few positive instances, limiting our ability to draw statistically robust conclusions about inter-architecture differences, especially when observed F1-score gaps are small. In particular, we do not report bootstrap confidence intervals or formal significance tests on the macro-level comparisons, so the strength of our cross-architecture claims should be interpreted accordingly.

Finally, the selection of representative models per architectural family. Our conclusions about each architectural family are based on a single model instance. Since models within the same family may differ substantially in behavior due to training data, instruction tuning, and alignment procedures, as evidenced by the divergence between Qwen-3.5 and DeepSeek-V2-Lite, a single representative may not adequately characterize the family as a whole. Our findings should therefore be interpreted as evidence about the specific models evaluated rather than as definitive characterizations of their respective architectural families.

Conclusion and Future Work

This paper revisits the empirical study by Chen et al. [13] on LLM-based smart contract vulnerability detection to verify whether their findings hold under changes in the underlying architecture. Our results show that the detection profile reported in the original study does not transfer uniformly across architectures. Outside the dense Transformer family, the same prompt elicits qualitatively different regimes, from aggressive over-flagging to markedly conservative behavior. The same qualitative dense-Transformer pattern appears in both GPT-4 and Gemma-3 despite a large-scale gap, which weakens the interpretation that the original result is driven only by model size. Finally, behavior within a single family is not fully predictable from architectural membership alone, as the contrast between the two MoE representatives illustrates.

Several directions for future work emerge. First, our analysis considers a single representative model per architectural family. Expanding this to small cohorts would allow direct measurement of intra-family variability rather than inferring it from isolated cases. Second, the choice of benchmark deserves further attention. Although widely used, the SmartBugs dataset is relatively small and somewhat outdated; repeating the same protocol on more recent corpora would help assess whether the observed architectural effects persist under distributional shifts. Third, the limited number of runs and contract calls necessitate statistical reenforcement of the comparisons, for example, by computing bootstrap confidence intervals for macro-level metrics. Fourth, reasoning-oriented models introduce an additional complication: their internal deliberation mechanisms are not fully compatible with the Chain-of-Thought prompting strategy adopted in our experiments and therefore require a dedicated evaluation setup. Finally, the difficulties we encountered with State Space Models in our preliminary exploration suggest that a focused investigation of this family, disentangling architectural mismatch from the current state of instruction-tuned SSM checkpoints, is worthwhile.

Acknowledgment

Gerardo Iuliano is funded by the European Union – Next Generation EU, Mission 4 Component 2, CUP D42B24002220004, by the Italian Government (MUR), and by AstraKode S.r.l.

References