Large language models struggle to detect hidden risks in smart contracts
The Deception Delta: Adversarial Evaluation of LLM-Based Smart Contract Bytecode Forensics
Cryptography and Security
Summary
Smart contracts are computer programs that run on blockchains, but some are designed to hide harmful actions like stealing money. The authors tested 22 language models that analyze smart contract code and found many fail to spot these hidden dangers, even when given different instructions. Some models incorrectly say contracts are safe because they accept deceptive explanations, despite describing the risks. Overall, current language models are not reliable on their own for finding these hidden problems in smart contract code.
What this means in practice
- •For blockchain security teams: Detect limitations of language models analyzing raw smart contract bytecode to avoid false security assurances from deceptive contracts.
- •For fraud detection analysts: Improve tools for identifying hidden vulnerabilities in smart contracts by understanding where LLMs fail under adversarial conditions.
Authors
Timo Schefold
Abstract
Large language models are increasingly used in blockchain forensic investigations to interpret unverified smart contract bytecode. Their robustness has not been systematically tested against contracts adversarially designed to mislead analysis. We evaluate 22 frontier models on 13 purpose-built contracts (9 deception vectors, 4 controls) across six prompt strategies, yielding 8,528 analyzable non-refusal runs against contracts with EVM-verified ground truth. A calibrated LLM-as-judge pipeline, supported by two judge-independent metrics and 50 human gold-standard labels, shows that adversarial deception reduces drain detection by 20.0 percentage points (95% CI: [17.2, 22.8]) relative to functionally matched controls. Structural camouflage via multi-hop call chains, XOR-masked selectors, and storage-loaded drain parameters resists detection across nearly all models. Beyond non-detection, we identify rationalization: models correctly describe the hidden drain mechanism but accept the contract's deceptive framing and dismiss it as benign, yielding positive but incorrect evidence of safety. Simple guard instructions provide no aggregate benefit and destabilize individual models in both directions. Structural deception is largely insensitive across the six tested prompt strategies, more consistent with a capability limitation than with a simple prompting problem. Only five models from two providers exceed 50% detection. Under our single-shot, raw-bytecode-only protocol, current LLMs are not reliable standalone forensic tools. Our central claim does not extend to multi-turn, tool-augmented, source-aware, or decompiler-in-the-loop workflows; a source-code boundary check is reported as an explicit subset analysis rather than as part of the main evaluation.