DeFiFlowBench benchmarks safety of natural language DeFi workflows
DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis
Machine LearningSoftware Engineering
Summary
DeFi workflows can sometimes authorize risky trades even if they are structurally correct. The authors created DeFiFlowBench, a test set of 207 prompts, to measure how well systems generate safe DeFi transaction sequences in natural language. They found common prompting methods led to many unsafe executions despite safety limits. To improve safety, they designed Koan-Safe, which uses intent parsing and structural checks to reduce unsafe trades significantly. Their tests show that explicit trade safety checks and execution testing are important to really ensure DeFi trade safety.
What this means in practice
- •For blockchain developers: Evaluate and improve safety checks in automatically generated decentralized finance trade sequences from text descriptions.
- •For automated trading platform teams: Incorporate intent parsing and structural repair methods to reduce unsafe trades caused by natural language order instructions.
Authors
Abhinav Rajeev Kumar, Harshit Arora, Varun Singh, Manikandan Nanjappan
Abstract
A structurally valid DeFi workflow can still authorize a costly trade. We introduce DeFiFlowBench, a benchmark of 207 team-authored prompts for natural-language DeFi workflow synthesis. It measures graph coverage, configuration completeness, and declared safety predicates, then tests supported trade configurations on a local EVM. Direct, constrained, and few-shot prompting produce 14-19 unsafe held-out executions per configuration under a fixed 5% price-impact cap. A slippage bound derived from a quote does not prevent the price impact of the order itself. We propose Koan-Safe, which combines a prompt-only intent parser, a replaceable generator, and structural repair with default safety parameters. On 75 held-out workflow prompts, its hybrid variant scores 0.67 on the static safety proxy, compared with 0.33 for the best baseline. Koan-Safe records no unsafe executions on the saved benchmark outputs. A matched-candidate ablation produces 14-17 unsafe executions when enforcement is disabled. Additional tests expose the limits of default injection: permissive existing thresholds can still authorize unsafe trades. A separately evaluated policy cap addresses this failure on a 36-case diagnostic grid. These results support explicit trade protections and execution-based evaluation, while distinguishing declared safety from a general guarantee.