Large language models vary in finding smart contract security issues
When LLM Architectures Meet Smart Contract Vulnerability Detection: a Replication Study
Software Engineering
Summary
Smart contracts are computer programs on blockchains that can’t be changed once deployed, so bugs may cause financial harm. The paper studies how well different types of large language models (LLMs) can spot these bugs. It confirmed that GPT-4 often finds many potential bugs but also makes many mistakes. Other LLM architectures don’t all behave the same way, even within the same family, so knowing a model’s type alone doesn’t predict how well it detects vulnerabilities. This shows that vulnerability detection depends on more than just the model’s basic design.
What this means in practice
- •For blockchain security teams: Choose suitable AI models for automated detection of smart contract vulnerabilities beyond just GPT-based choices.
- •For ai developers in finance: Design tailored vulnerability detection tools by considering diverse LLM architectures to improve smart contract safety.
Authors
Gerardo Iuliano, Gabriele De Vito, Dario Di Nucci
Abstract
Smart contracts are immutable programs deployed on blockchain platforms, where vulnerabilities can translate directly into financial loss. Among the techniques proposed for their detection, recent work has relied increasingly on Large Language Models (LLMs) guided by carefully crafted prompts. However, the evidence accumulated so far rests almost entirely on the dense Transformer architecture behind the GPT line and on model snapshots already scheduled for deprecation, leaving open the question of whether the reported behaviors reflect a general property of LLMs or an artifact of that specific design. This paper presents an extended replication of the empirical study by Chen et al. on LLM-based smart contract vulnerability detection. We replicate the original protocol on GPT-4 as a baseline, followed by a differentiated replication, executing the protocol on representative models based on Mixture-of-Experts and RNN-like designs, together with an attempted preliminary evaluation of State Space Models. The exact replication confirms the findings of Chen et al., including the high-recall, low-precision detection pattern they report on GPT-4. Nevertheless, the differentiated replication shows that this pattern does not transfer uniformly across the evaluated models. Those from the same architectural family may also exhibit different behaviors, indicating that architectural family alone is insufficient to predict the resulting detection profile. We observe the same qualitative dense-Transformer profile in both GPT-4 and Gemma-3, despite a gap of roughly two orders of magnitude in parameter counts, whereas two Mixture-of-Experts models from different lineages exhibit qualitatively divergent behaviors, indicating that architectural family alone is insufficient to predict the detection profile.